diff options
Diffstat (limited to 'doc')
1069 files changed, 27163 insertions, 16134 deletions
diff --git a/doc/.vale/gitlab/BadgeCapitalization.yml b/doc/.vale/gitlab/BadgeCapitalization.yml index 6e77c3fe822..a44bcbc0a7d 100644 --- a/doc/.vale/gitlab/BadgeCapitalization.yml +++ b/doc/.vale/gitlab/BadgeCapitalization.yml @@ -10,5 +10,5 @@ link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html level: error scope: raw raw: - - '(?!\*\*\((FREE|PREMIUM|ULTIMATE)( (SELF|SAAS))?\)\*\*)' - - '(?i)\*\*\((free|premium|ultimate)( (self|saas))?\)\*\*' + - '(?!\*\*\((FREE|PREMIUM|ULTIMATE)( (SELF|SAAS|ALL) (BETA|EXPERIMENT))?\)\*\*)' + - '(?i)\*\*\((free|premium|ultimate)( (self|saas|all) (beta|experiment))?\)\*\*' diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index c081ca5a4f7..94befa9c0e6 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -59,6 +59,6 @@ swap: sub-group: "subgroup" sub-groups: "subgroups" timezone: "time zone" - utilize: "use" + utiliz(?:es?|ing): "use" we recommend: "you should" within: "in" diff --git a/doc/.vale/gitlab/Substitutions.yml b/doc/.vale/gitlab/Substitutions.yml index 10369669ca5..0d49ac583dd 100644 --- a/doc/.vale/gitlab/Substitutions.yml +++ b/doc/.vale/gitlab/Substitutions.yml @@ -14,6 +14,7 @@ swap: codequality: code quality Customer [Pp]ortal: Customers Portal disallow: prevent + '(?<!GitLab )Duo': GitLab Duo frontmatter: front matter GitLabber: GitLab team member GitLabbers: GitLab team members diff --git a/doc/.vale/gitlab/Uppercase.yml b/doc/.vale/gitlab/Uppercase.yml index 4730184b950..b13ebe2c0a8 100644 --- a/doc/.vale/gitlab/Uppercase.yml +++ b/doc/.vale/gitlab/Uppercase.yml @@ -16,6 +16,7 @@ second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)' exceptions: - ACL - AJAX + - ALL - AMI - ANSI - APAC @@ -27,6 +28,7 @@ exceptions: - ASG - AST - AWS + - BETA - BMP - BSD - CAS diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index 1e66d12adc9..575dc713b51 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -109,6 +109,7 @@ bugfixed bugfixes bugfixing Bugzilla +Buildah Buildkite buildpack buildpacks @@ -350,6 +351,7 @@ Fastly Fastzip favicon favorited +Fediverse ffaker Figma Filebeat @@ -525,6 +527,7 @@ LaunchDarkly ldapsearch Lefthook Leiningen +Lemmy libFuzzer Libgcrypt Libravatar @@ -644,6 +647,7 @@ OmniAuth onboarding OpenID OpenShift +OpenTelemetry Opsgenie Opstrace ORMs @@ -958,6 +962,7 @@ subqueried subqueries subquery subquerying +Subreddit substring substrings subtask diff --git a/doc/administration/admin_area.md b/doc/administration/admin_area.md index 1e103bb55c8..a27fc04ff8f 100644 --- a/doc/administration/admin_area.md +++ b/doc/administration/admin_area.md @@ -10,7 +10,7 @@ type: reference The Admin Area provides a web UI to manage and configure features of GitLab self-managed instances. If you are an administrator,to access the Admin Area: -- In GitLab 16.1 and later: on the left sidebar, expand the top-most chevron (**{chevron-down}**), then select **Admin Area**. +- In GitLab 16.1 and later: on the left sidebar, select **Search or go to**, then select **Admin Area**. - In GitLab 16.0 and earlier: on the top bar, select **Main menu > Admin**. NOTE: @@ -22,7 +22,7 @@ You can administer all projects in the GitLab instance from the Admin Area's Pro To access the Projects page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Projects**. 1. Select the **All**, **Private**, **Internal**, or **Public** tab to list only @@ -74,7 +74,7 @@ You can combine the filter options. For example, to list only public projects wi You can administer all users in the GitLab instance from the Admin Area's Users page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. @@ -119,7 +119,7 @@ This allows the administrator to "see what the user sees," and take actions on b You can impersonate a user in the following ways: - Through the UI: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. From the list of users, select a user. @@ -138,7 +138,7 @@ By default, impersonation is enabled. GitLab can be configured to [disable imper When using authentication providers, administrators can see the identities for a user: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. From the list of users, select a user. @@ -185,7 +185,7 @@ GitLab billing is based on the number of [**Billable users**](../subscriptions/s You must be an administrator to manually add emails to users: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Locate the user and select them. @@ -198,16 +198,16 @@ You must be an administrator to manually add emails to users: The [Cohorts](user_cohorts.md) tab displays the monthly cohorts of new users and their activities over time. -## Prevent a user from creating groups +## Prevent a user from creating top level groups -By default, users can create groups. To prevent a user from creating a top level group: +By default, users can create top level groups. To prevent a user from creating a top level group: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Locate the user and select them. 1. Select **Edit**. -1. Clear the **Can create group** checkbox. +1. Clear the **Can create top level group** checkbox. 1. Select **Save changes**. It is also possible to [limit which roles can create a subgroup within a group](../user/group/subgroups/index.md#change-who-can-create-subgroups). @@ -218,7 +218,7 @@ You can administer all groups in the GitLab instance from the Admin Area's Group To access the Groups page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Groups**. @@ -244,7 +244,7 @@ To [Create a new group](../user/group/index.md#create-a-group) select **New grou To view all topics in the GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. @@ -252,7 +252,7 @@ For each topic, the page displays its name and the number of projects labeled wi ### Search for topics -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. 1. In the search box, enter your search criteria. @@ -262,7 +262,7 @@ For each topic, the page displays its name and the number of projects labeled wi To create a topic: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. 1. Select **New topic**. @@ -282,7 +282,7 @@ Do not include sensitive information in the name of a topic. You can edit a topic's name, title, description, and avatar at any time. To edit a topic: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. 1. Select **Edit** in that topic's row. @@ -294,7 +294,7 @@ To edit a topic: If you no longer need a topic, you can permanently remove it. To remove a topic: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. 1. To remove a topic, select **Remove** in that topic's row. @@ -307,7 +307,7 @@ After a merged topic is deleted, you cannot restore it. To merge topics: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Topics**. 1. Select **Merge topics**. @@ -322,7 +322,7 @@ page. For more details, see [Gitaly](gitaly/index.md). To access the **Gitaly Servers** page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Gitaly Servers**. @@ -347,7 +347,7 @@ You can administer all runners in the GitLab instance from the Admin Area's **Ru To access the **Runners** page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Runners**. @@ -374,7 +374,7 @@ You can also filter runners by status, type, and tag. To filter: You can delete multiple runners at the same time. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Runners**. 1. To the left of the runners you want to delete, select the checkbox. @@ -405,7 +405,7 @@ You can administer all jobs in the GitLab instance from the Admin Area's Jobs pa To access the Jobs page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Jobs**. All jobs are listed, in descending order of job ID. 1. Select the **All** tab to list all jobs. Select the **Pending**, **Running**, or **Finished** @@ -431,7 +431,7 @@ The following topics document the **Monitoring** section of the Admin Area. ### System Information -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341248) in GitLab 15.2, support for relative time. "Uptime" statistic was renamed to "System started". +> Support for relative time [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341248) in GitLab 15.2. "Uptime" statistic was renamed to "System started". The **System Info** page provides the following statistics: diff --git a/doc/administration/analytics/dev_ops_reports.md b/doc/administration/analytics/dev_ops_reports.md index 1dcee5ec03d..a5a3120489d 100644 --- a/doc/administration/analytics/dev_ops_reports.md +++ b/doc/administration/analytics/dev_ops_reports.md @@ -12,7 +12,7 @@ from planning to monitoring. To see DevOps Reports: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Analytics > DevOps Reports**. diff --git a/doc/administration/analytics/index.md b/doc/administration/analytics/index.md index 6441cd866c8..5ba14222165 100644 --- a/doc/administration/analytics/index.md +++ b/doc/administration/analytics/index.md @@ -18,7 +18,7 @@ Prerequisite: To view instance-level analytics: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Analytics**, then one of the available analytics: diff --git a/doc/administration/analytics/usage_trends.md b/doc/administration/analytics/usage_trends.md index 49e82f71a3a..20b732d028d 100644 --- a/doc/administration/analytics/usage_trends.md +++ b/doc/administration/analytics/usage_trends.md @@ -19,7 +19,7 @@ Usage Trends data refreshes daily. To view Usage Trends: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Analytics > Usage Trends**. diff --git a/doc/administration/appearance.md b/doc/administration/appearance.md index c5c50d95eb6..ceb4a4a30e9 100644 --- a/doc/administration/appearance.md +++ b/doc/administration/appearance.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Several options are available for customizing the appearance of a self-managed instance of GitLab. To access these settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Appearance**. @@ -29,7 +29,7 @@ supported by many email clients. ## Favicon -By default, the favicon (used by the browser as the tab icon, as well as the CI status icon) +By default, the favicon (used by the browser as the tab icon and the CI status icon) uses the GitLab logo. This can be customized with any icon desired. It must be a 32x32 `.png` or `.ico` image. @@ -42,7 +42,7 @@ of the page to activate it in the GitLab instance. You can add a small header message, a small footer message, or both, to the interface of your GitLab instance. These messages appear on all projects and pages of the -instance, including the sign in / sign up page. The default color is white text on +instance, including the sign-in/sign-up page. The default color is white text on an orange background, but this can be customized by selecting **Customize colors**. Limited [Markdown](../user/markdown.md) is supported, such as bold, italics, and links, for @@ -55,9 +55,9 @@ the header and footer added to all emails sent by the GitLab instance. After you add a message, select **Update appearance settings** at the bottom of the page to activate it in the GitLab instance. -## Sign in / Sign up pages +## Sign-in / Sign-up pages -You can replace the default message on the sign in / sign up page with your own message +You can replace the default message on the sign-in/sign-up page with your own message and logo. You can make full use of [Markdown](../user/markdown.md) in the description. The optimal size for the logo is 128 x 128 pixels, but any image can be used (below 1 MB) @@ -69,7 +69,7 @@ to activate it in the GitLab instance. You can also select **Sign-in page**, to review the saved appearance settings: NOTE: -You can add also add a [customized hcelp message](settings/help_page.md) below the sign in message or add [a Sign in text message](settings/sign_in_restrictions.md#sign-in-information). +You can add also add a [customized help message](settings/help_page.md) below the sign-in message or add [a Sign-in text message](settings/sign_in_restrictions.md#sign-in-information). ## Progressive Web App @@ -83,7 +83,7 @@ description, and icon. To configure the PWA settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Appearance**. 1. Scroll to the **Progressive Web App (PWA)** section. diff --git a/doc/administration/audit_event_streaming/audit_event_types.md b/doc/administration/audit_event_streaming/audit_event_types.md index b69f1815394..5c27dabe378 100644 --- a/doc/administration/audit_event_streaming/audit_event_types.md +++ b/doc/administration/audit_event_streaming/audit_event_types.md @@ -9,18 +9,24 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo Please do not edit this file directly. To update this file, run: bundle exec rake gitlab:audit_event_types:compile_docs + + To make changes to the output of the rake task, + edit `tooling/audit_events/docs/templates/audit_event_types.md.erb`. ---> -# Audit event types **(ULTIMATE)** +# Audit event types **(PREMIUM ALL)** Audit event types are used to [filter streamed audit events](index.md#update-event-filters). -Every audit event is associated with an event type. The association with the event type can mean that an audit event is: +Every audit event is associated with an event type. Audit event types can allow audit events to be: + +- Saved to the database. Available in the Premium and Ultimate tier. You can retrieve audit events associated with these + types by using the audit events dashboard or the [audit events API](../../api/audit_events.md). +- Streamed to external destinations. Available in the Ultimate tier. You can stream audit events associated with these + types [to external destinations](../index.md) if a destination is set. -- Saved to database. Audit events associated with these types are retrievable by using the audit events dashboard or the [audit events API](../../api/audit_events.md). -- Streamed. Audit events associated with these types are [streamed to external destinations](../index.md) if a - destination is set. -- Not streamed. Audit events associated with these types are not streamed to external destinations even if a destination is set. +Some audit event types don't allow saving audit events to the database. Other audit event types don't allow streaming +audit events to external destinations. ## Available audit event types @@ -85,10 +91,11 @@ Every audit event is associated with an event type. The association with the eve | [`destroy_compliance_framework`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74292) | Triggered on successful compliance framework deletion | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) | | [`destroy_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74632) | Event triggered when an external audit event destination is deleted | **{check-circle}** Yes | **{check-circle}** Yes | `audit_events` | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) | | [`destroy_instance_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125846) | Event triggered when an instance level external audit event destination is deleted | **{check-circle}** Yes | **{check-circle}** Yes | `audit_events` | GitLab [16.2](https://gitlab.com/gitlab-org/gitlab/-/issues/404730) | +| [`email_confirmation_sent`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129261) | Triggered when users add or change and email address and it needs to be confirmed. | **{dotted-circle}** No | **{check-circle}** Yes | `user_profile` | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/377625) | | [`email_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114546) | Event triggered when an email is created | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.11](https://gitlab.com/gitlab-org/gitlab/-/issues/374107) | | [`email_destroyed`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114546) | Event triggered when an email is destroyed | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.11](https://gitlab.com/gitlab-org/gitlab/-/issues/374107) | -| [`environment_protected`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108247) | This event is triggered when a protected environment is created. | **{check-circle}** Yes | **{dotted-circle}** No | `environment_management` | GitLab [15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) | -| [`environment_unprotected`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108247) | This event is triggered when a protected environment is deleted. | **{check-circle}** Yes | **{dotted-circle}** No | `environment_management` | GitLab [15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) | +| [`environment_protected`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108247) | This event is triggered when a protected environment is created. | **{check-circle}** Yes | **{check-circle}** Yes | `environment_management` | GitLab [15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) | +| [`environment_unprotected`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108247) | This event is triggered when a protected environment is deleted. | **{check-circle}** Yes | **{check-circle}** Yes | `environment_management` | GitLab [15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) | | [`epic_closed_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an epic is closed by a group access token | **{check-circle}** Yes | **{check-circle}** Yes | `portfolio_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`epic_created_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an epic is created by a group access token | **{check-circle}** Yes | **{check-circle}** Yes | `portfolio_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`epic_reopened_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an epic is reopened by a group access token | **{check-circle}** Yes | **{check-circle}** Yes | `portfolio_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | @@ -141,13 +148,19 @@ Every audit event is associated with an event type. The association with the eve | [`group_shared_runners_minutes_limit_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106079) | Event triggered when a groups shared runners minutes limit is updated. | **{check-circle}** Yes | **{check-circle}** Yes | `groups_and_projects` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369324) | | [`group_two_factor_grace_period_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106079) | Event triggered when a groups two factor grace period is updated. | **{check-circle}** Yes | **{check-circle}** Yes | `groups_and_projects` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369326) | | [`group_visibility_level_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106079) | Event triggered when a groups visibility level is updated. | **{check-circle}** Yes | **{check-circle}** Yes | `groups_and_projects` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369322) | +| [`inactive_project_scheduled_for_deletion`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130699) | Triggered when inactive project is scheduled for deletion | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/423263) | | [`incident_closed_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an incident is closed using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `incident_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`incident_created_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an incident is created using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `incident_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`incident_reopened_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an incident is reopened using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `incident_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | +| [`instance_google_cloud_logging_configuration_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130663) | Triggered when Instance level Google Cloud Logging configuration is created | **{check-circle}** Yes | **{check-circle}** Yes | `audit_events` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/423038) | +| [`instance_google_cloud_logging_configuration_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131752) | Triggered when instance level Google Cloud Logging configuration is deleted. | **{check-circle}** Yes | **{check-circle}** Yes | `audit_events` | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/423040) | | [`ip_restrictions_changed`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86037) | Event triggered on any changes in the IP AllowList | **{check-circle}** Yes | **{check-circle}** Yes | `system_access` | GitLab [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) | | [`issue_closed_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an issue is closed using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `team_planning` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`issue_created_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an issue is created using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `team_planning` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | | [`issue_reopened_by_project_bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121485) | Triggered when an issue is reopened using a project access token | **{check-circle}** Yes | **{check-circle}** Yes | `team_planning` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/323299) | +| [`login_failed_with_otp_authentication`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129595) | Triggered when the login fails due to an incorrect OTP | **{check-circle}** Yes | **{check-circle}** Yes | `system_access` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/377758) | +| [`login_failed_with_standard_authentication`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129595) | Triggered when login to GitLab fails with standard authentication like password. | **{check-circle}** Yes | **{check-circle}** Yes | `system_access` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/377758) | +| [`login_failed_with_webauthn_authentication`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129595) | Triggered when login fails via WebAuthn device | **{check-circle}** Yes | **{check-circle}** Yes | `system_access` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/377758) | | [`member_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109711) | Event triggered when a membership is created | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/374112) | | [`member_destroyed`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109711) | Event triggered when a membership is destroyed | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/374112) | | [`member_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109711) | Event triggered when a membership is updated | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/374112) | @@ -162,6 +175,7 @@ Every audit event is associated with an event type. The association with the eve | [`merged_merge_request_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118793) | Audit event triggered when a merged merge request is deleted | **{dotted-circle}** No | **{check-circle}** Yes | `source_code_management` | GitLab [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/408288) | | [`merged_merge_request_deletion_started`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118793) | Audit event triggered when a merged merge request's deletion is started | **{dotted-circle}** No | **{check-circle}** Yes | `source_code_management` | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/408288) | | [`omniauth_login_failed`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123080) | Event triggered when an OmniAuth login fails | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/374107) | +| [`password_reset_failed`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129079) | Event triggered when a password reset fails for a user | **{dotted-circle}** No | **{check-circle}** Yes | `user_management` | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/377762) | | [`password_reset_requested`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114548) | Event triggered when a user requests a password reset using a registered email address | **{check-circle}** Yes | **{dotted-circle}** No | `compliance_management` | GitLab [15.11](https://gitlab.com/gitlab-org/gitlab/-/issues/374107) | | [`personal_access_token_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108952) | Event triggered when a user creates a personal access token | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/374113) | | [`personal_access_token_revoked`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108952) | Event triggered when a personal access token is revoked | **{check-circle}** Yes | **{check-circle}** Yes | `compliance_management` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/374113) | @@ -283,6 +297,7 @@ Every audit event is associated with an event type. The association with the eve | [`user_enable_admin_mode`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104754) | Event triggered on enabling admin mode | **{check-circle}** Yes | **{check-circle}** Yes | `system_access` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/362101) | | [`user_impersonation`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79340) | Triggered when an instance administrator starts or stops impersonating a user | **{check-circle}** Yes | **{check-circle}** Yes | `user_management` | GitLab [14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) | | [`user_password_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106086) | audit when user password is updated | **{check-circle}** Yes | **{check-circle}** Yes | `user_management` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369330) | +| [`user_profile_visiblity_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129149) | Triggered when user toggles private profile user setting | **{dotted-circle}** No | **{check-circle}** Yes | `user_profile` | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129149) | | [`user_rejected`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113784) | Event triggered when a user registration is rejected | **{check-circle}** Yes | **{dotted-circle}** No | `user_management` | GitLab [15.11](https://gitlab.com/gitlab-org/gitlab/-/issues/374107) | | [`user_username_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106086) | Event triggered on updating a user's username | **{check-circle}** Yes | **{check-circle}** Yes | `user_profile` | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/369329) | | [`feature_flag_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113453) | Triggered when a feature flag is created. | **{check-circle}** Yes | **{check-circle}** Yes | `feature_flags` | GitLab [15.10](https://gitlab.com/gitlab-org/gitlab/-/issues/374109) | diff --git a/doc/administration/audit_event_streaming/examples.md b/doc/administration/audit_event_streaming/examples.md index 1c9a14dbab0..b0d6190211c 100644 --- a/doc/administration/audit_event_streaming/examples.md +++ b/doc/administration/audit_event_streaming/examples.md @@ -13,7 +13,7 @@ The following sections provide examples of audit event streaming. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. -> - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. +> - `details.author_class` field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed. Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories: diff --git a/doc/administration/audit_event_streaming/graphql_api.md b/doc/administration/audit_event_streaming/graphql_api.md index e6584369e5a..768b1f03bf3 100644 --- a/doc/administration/audit_event_streaming/graphql_api.md +++ b/doc/administration/audit_event_streaming/graphql_api.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit event streaming GraphQL API **(ULTIMATE ALL)** -> - API [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. -> - API [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. -> - API [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - API [enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. +> - API [feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. > - Custom HTTP headers API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](../feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. > - Custom HTTP headers API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. > - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. @@ -16,6 +16,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - APIs for custom HTTP headers for instance level streaming destinations [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/404560) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. > - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. > - User-specified destination name API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/413894) in GitLab 16.2. +> - API [feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/417708) removed in GitLab 16.4. Audit event streaming destinations can be maintained using a GraphQL API. @@ -331,7 +332,6 @@ mutation { id googleProjectIdName logIdName - privateKey clientEmail } errors @@ -364,7 +364,6 @@ query { id logIdName googleProjectIdName - privateKey clientEmail } } @@ -397,7 +396,6 @@ mutation { googleCloudLoggingConfiguration { id logIdName - privateKey googleProjectIdName clientEmail } @@ -441,10 +439,7 @@ Streaming configuration is deleted if: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. > - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. - -FLAG: -On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. +> - Instance streaming destinations [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) in GitLab 16.4. [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/417708) removed. Manage HTTP streaming destinations for an entire instance. diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md index ae566891ac7..8a1aa9a661c 100644 --- a/doc/administration/audit_event_streaming/index.md +++ b/doc/administration/audit_event_streaming/index.md @@ -46,7 +46,7 @@ Prerequisites: To add streaming destinations to a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **Add streaming destination** and select **HTTP endpoint** to show the section for adding destinations. @@ -66,7 +66,7 @@ Prerequisites: To list the streaming destinations for a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the stream to expand it and see all the custom HTTP headers. @@ -79,7 +79,7 @@ Prerequisites: To update a streaming destination's name: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the stream to expand. @@ -88,7 +88,7 @@ To update a streaming destination's name: To update a streaming destination's custom HTTP headers: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the stream to expand. @@ -111,7 +111,7 @@ Prerequisites: To delete a streaming destination: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. 1. Select the stream to expand. @@ -120,7 +120,7 @@ To delete a streaming destination: To delete only the custom HTTP headers for a streaming destination: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. 1. Select the stream to expand. @@ -145,7 +145,7 @@ Prerequisites: To list streaming destinations and see the verification tokens: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams**. 1. Select the stream to expand. @@ -162,7 +162,7 @@ A streaming destination that has an event type filter set has a **filtered** (** To update a streaming destination's event filters: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. 1. Select the stream to expand. @@ -203,11 +203,11 @@ Prerequisites: To add Google Cloud Logging streaming destinations to a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **Add streaming destination** and select **Google Cloud Logging** to show the section for adding destinations. -1. Enter the Google Project ID, Google Client Email, Log ID, and Google Private Key to add. +1. Enter the Google project ID, Google client email, log ID, and Google private key to add. 1. Select **Add** to add the new streaming destination. #### List Google Cloud Logging destinations @@ -218,24 +218,27 @@ Prerequisites: To list Google Cloud Logging streaming destinations for a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the Google Cloud Logging stream to expand and see all the fields. #### Update a Google Cloud Logging destination +> Button to add private key [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419675) in GitLab 16.3. + Prerequisites: - Owner role for a top-level group. To update Google Cloud Logging streaming destinations to a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. 1. Select the Google Cloud Logging stream to expand. -1. Enter the Google Project ID, Google Client Email, Log ID, and Google Private Key to update. +1. Enter the Google project ID, Google client email, and log ID to update. +1. Select **Add a new private key** and enter a Google private key to update the private key. 1. Select **Save** to update the streaming destination. #### Delete a Google Cloud Logging streaming destination @@ -246,7 +249,7 @@ Prerequisites: To delete Google Cloud Logging streaming destinations to a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. 1. Select the Google Cloud Logging stream to expand. @@ -257,10 +260,7 @@ To delete Google Cloud Logging streaming destinations to a top-level group: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. > - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. - -FLAG: -On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. +> - Instance streaming destinations [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) in GitLab 16.4. [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/417708) removed. Manage HTTP streaming destinations for an entire instance. @@ -274,7 +274,7 @@ Prerequisites: To add a streaming destination for an instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. @@ -295,7 +295,7 @@ Prerequisites: To list the streaming destinations for an instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. @@ -309,7 +309,7 @@ Prerequisites: To update a instance streaming destination's name: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. @@ -319,7 +319,7 @@ To update a instance streaming destination's name: To update a instance streaming destination's custom HTTP headers: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. @@ -343,7 +343,7 @@ Prerequisites: To delete the streaming destinations for an instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select the **Streams** tab. @@ -353,7 +353,7 @@ To delete the streaming destinations for an instance: To delete only the custom HTTP headers for a streaming destination: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select the **Streams** tab. @@ -367,10 +367,7 @@ To delete only the custom HTTP headers for a streaming destination: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. > - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. - -FLAG: -On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. +> - Instance streaming destinations [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) in GitLab 16.4. [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/417708) removed. Each streaming destination has a unique verification token (`verificationToken`) that can be used to verify the authenticity of the event. This token is either specified by the Owner or generated automatically when the event destination is created and cannot be changed. @@ -384,7 +381,7 @@ Prerequisites: To list streaming destinations for an instance and see the verification tokens: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select the **Streams** tab. @@ -401,7 +398,7 @@ A streaming destination that has an event type filter set has a **filtered** (** To update a streaming destination's event filters: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select the **Streams** tab. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 9657994ba8f..736f381e9d7 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -69,7 +69,7 @@ You can view audit events from user actions across an entire GitLab instance. To view instance audit events: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. @@ -82,7 +82,7 @@ To view instance audit events: You can export the current view (including filters) of your instance audit events as a CSV file. To export the instance audit events to CSV: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. Select the available search [filters](#filter-audit-events). @@ -156,284 +156,7 @@ with additional details: ## Available audit events -You can view different events depending on the version of GitLab you have. - -### Group events - -The following actions on groups generate group audit events: - -#### Group management - -- Group name or path changed. -- Group repository size limit changed. -- Group created or deleted. -- Group changed visibility. -- User was added to group and with which [permissions](../user/permissions.md). -- Removed user from group. -- Project repository imported into group. -- [Project shared with group](../user/project/members/share_project_with_groups.md) and with which - [permissions](../user/permissions.md). -- Removal of a previously shared group with a project. -- LFS enabled or disabled. -- Membership lock enabled or disabled. -- Request access enabled or disabled. -- Roles allowed to create project changed. -- Group deploy token was successfully created, revoked, or deleted. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. -- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) - in GitLab 14.9. -- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. -- Group had a security policy project linked, changed, or unlinked. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6. -- An environment is protected or unprotected. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8. - -#### Authentication and authorization - -- User sign-in using [Group SAML](../user/group/saml_sso/index.md). -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8071) in GitLab 14.5, changes to the following - [group SAML](../user/group/saml_sso/index.md) configuration: - - Enabled status. - - Enforcing SSO-only authentication for web activity. - - Enforcing SSO-only authentication for Git and Dependency Proxy activity. - - Enforcing users to have dedicated group-managed accounts. - - Prohibiting outer forks. - - Identity provider SSO URL. - - Certificate fingerprint. - - Default membership role. - - SSO-SAML group sync configuration. -- Permissions changes of a user assigned to a group. -- 2FA enforcement or grace period changed. - -#### Compliance and security - -- Compliance framework created, updated, or deleted. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5. -- Event streaming destination created, updated, or deleted. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6. -- Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. -- Changes to streaming audit destination custom HTTP headers. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3. -- Instance administrator started or stopped impersonation of a group member. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. - -#### CI/CD - -- Shared runners minutes limit changed. -- Group CI/CD variable added, removed, or protected status changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. - -#### Code collaboration - -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge - request approvals settings: - - Prevent approval by author. - - Prevent approvals by users who add commits. - - Prevent editing approval rules in projects and merge requests. - - Require user password to approve. - - Remove all approvals when commits are added to the source branch. -- Changes to Code Suggestions. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/405295) in GitLab 15.11. - -### Project events - -The following actions on projects generate project audit events: - -#### Project management - -- Added or removed deploy keys. -- Project created, deleted, renamed, moved (transferred), changed path. -- Project changed visibility level. -- Project export was downloaded. -- Project repository was downloaded. -- Project was archived. -- Project was unarchived. -- Project had a security policy project linked, changed, or unlinked. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6. -- Project was scheduled for deletion due to inactivity. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0. -- Project deploy token was successfully created, revoked, or deleted. Introduced in GitLab 14.9. - GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/353451`. -- Failed attempt to create a project deploy token. Introduced in GitLab 14.9. - GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/353451`. -- When [strategies for feature flags](../operations/feature_flags.md#feature-flag-strategies) are changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68408) in GitLab 14.3. - -#### User management - -- User was added to project and with which [permissions](../user/permissions.md). -- Permission changes of a user assigned to a project. -- User was removed from project. -- Users and groups allowed to merge and push to protected branch added or removed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3. - -#### Access control - -- Branch protection was added, removed, or updated. -- Failed attempt to create or revoke a project access token. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9. -- Allowing force push to protected branch changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3. -- An environment is protected or unprotected. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8. -- User password required for approvals was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2. -- Project access token was successfully created or revoked. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230007) in GitLab 13.9. - -#### Code collaboration - -- Default description template for merge requests is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0. -- Merge commit message template is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0. -- Squash commit message template is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0. -- Delete source branch option by default enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Squash commits when merging is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- All discussions must be resolved enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Commit message suggestion is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Automatically resolve merge request diff discussions enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Show link to create or view a merge request when pushing from the command line enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- When merge method is updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Merge trains enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Code owner approval requirement on merge requests targeting protected branch changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3. -- Permission to modify merge requests approval rules in merge requests was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2. -- New approvals requirement when new commits are added to an MR was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336211) in GitLab 14.2. -- Added or removed users and groups from project approval groups. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2. -- Permission to approve merge requests by committers was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9. - - Message for event [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72623/diffs) in GitLab 14.6. -- Permission to approve merge requests by authors was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9. -- Number of required approvals was updated. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9. - -#### Release management - -- Release was added to a project. -- Release was updated. -- Release was deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94793/) in GitLab 15.3. -- Release milestone associations changed. - -#### CI/CD - -- Project CI/CD variable added, removed, or protected status changed. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4. -- When default branch changes for a project. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/52339) in GitLab 13.9. -- Pipelines must succeed enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Skipped pipelines are considered successful enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. -- Status check is added, edited, or deleted. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0. -- Merged results pipelines enabled or disabled. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9. - -#### Compliance and security - -- Created, updated, or deleted DAST profiles, DAST scanner profiles, and DAST site profiles. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. -- Changed a project's compliance framework. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329362) in GitLab 14.1. - -### GitLab agent for Kubernetes events - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/382133) in GitLab 15.10. - -GitLab generates audit events when a cluster agent token is created or revoked. - -### Instance events **(PREMIUM SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5, audit events for failed second-factor authentication attempt. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6, audit events for when a user is approved using the Admin Area. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6, audit events for when a user's personal access token is successfully or unsuccessfully created or revoked. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9, audit events for when a user requests access to an instance or is rejected using the Admin Area. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in - GitLab 15.1, audit events when a user's two-factor authentication is disabled. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124169) in GitLab 16.2, audit events when a user's access is locked. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124973) in GitLab 16.2, audit events when a user's access is unlocked. - -The following user actions on a GitLab instance generate instance audit events: - -#### Authentication - -- Sign-in events and the authentication type such as standard, LDAP, or OmniAuth. -- Failed sign-ins. -- Ask for password reset. -- Grant OAuth access. -- Failed second-factor authentication attempt. -- A user's personal access token was successfully or unsuccessfully created or revoked. -- A user's two-factor authentication was disabled. -- A user's access is locked. -- A user's access is unlocked. - -#### User management - -- Added SSH key. -- Added or removed email. -- Changed password. -- Started or stopped user impersonation. -- Changed username. -- User was added or deleted. -- User requests access to an instance. -- User was approved, rejected, or blocked using the Admin Area. -- User was blocked using the API. -- Administrator added or removed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323905) in GitLab 14.1. -- Removed SSH key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1. -- Added or removed GPG key. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220127) in GitLab 14.1. -- Enabled Admin Mode. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362101) in GitLab 15.7. -- All [group events](#group-events) and [project events](#project-events). -- User was unblocked using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115727) in GitLab 15.11. -- User was banned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116103) in GitLab 15.11. -- User was unbanned using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116221) in GitLab 15.11. -- User was deactivated using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117776) in GitLab 16.0. -- User was activated using the Admin Area or API. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121708) in GitLab 16.1. - -Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). - -#### Application settings **(PREMIUM ALL)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282428) in GitLab 16.2. - -When a user changes an application setting in an instance, project, or group, -that change and the user that made the change are recorded in the audit log. - -### GitLab Runner events - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335509) in GitLab 14.8, audit events for when a runner is registered. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349540) in GitLab 14.9, audit events for when a runner is unregistered. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/349542) in GitLab 14.9, audit events for when a runner is assigned to or unassigned from a project. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355637) in GitLab 15.0, audit events for when a runner registration token is reset. - -GitLab generates audit events for the following GitLab Runner actions: - -- Instance, group, or project runner is registered. -- Instance, group, or project runner is unregistered. -- Runner is assigned to or unassigned from a project. -- Instance, group, or project runner registration token is reset. - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102579) in GitLab 15.6. - -## "Deleted User" events - -Audit events created after users are deleted are created for "Deleted User". For example, if a deleted user's access to -a project is removed automatically due to expiration. - -Issue [343933](https://gitlab.com/gitlab-org/gitlab/-/issues/343933) proposes to change this behavior. +For a list of available audit events, see [Audit event types](../administration/audit_event_streaming/audit_event_types.md). ## Unsupported events diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index e8ed0eb4313..2befb01f096 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -31,7 +31,7 @@ To create a new user account with auditor access (or change an existing user): To create a user account with auditor access: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Create a new user or edit an existing one. Set **Access Level** to **Auditor**. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index cbfb4921e14..639e285ead2 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 554b3d776ac..7d1b98d16d1 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -1,6 +1,6 @@ --- type: concepts, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 6ced9f844cd..590d38bf67a 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index 4e96cdf0411..4526a74c1f8 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -1,6 +1,6 @@ --- type: index -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 9f95682fc47..0d33fca1fa2 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 388288bbe49..1d0b082a80f 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 746d1f6b7fd..d2c27fe35b4 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -27,7 +27,7 @@ Users added through LDAP: even if password authentication for Git [is disabled](../../settings/sign_in_restrictions.md#password-authentication-enabled). -The LDAP DN is associated with existing GitLab users when: +The LDAP distinguished name (DN) is associated with existing GitLab users when: - The existing user signs in to GitLab with LDAP for the first time. - The LDAP email address is the primary email address of an existing GitLab user. If the LDAP email @@ -40,9 +40,7 @@ If an existing GitLab user wants to enable LDAP sign-in for themselves, they sho ## Security -GitLab has multiple mechanisms to verify a user is still active in LDAP. If the user is no longer active in -LDAP, they are placed in an `ldap_blocked` status and are signed out. They are unable to sign in using any authentication provider until they are -reactivated in LDAP. +GitLab verifies if a user is still active in LDAP. Users are considered inactive in LDAP when they: @@ -51,7 +49,7 @@ Users are considered inactive in LDAP when they: - Are marked as disabled or deactivated in Active Directory through the user account control attribute. This means attribute `userAccountControl:1.2.840.113556.1.4.803` has bit 2 set. -Status is checked for all LDAP users: +GitLab checks LDAP users' status: - When signing in using any authentication provider. [In GitLab 14.4 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/343298), status was checked only when signing in using LDAP directly. @@ -59,6 +57,12 @@ Status is checked for all LDAP users: - When performing Git over HTTP requests using LDAP username and password. - Once per day during [User Sync](ldap_synchronization.md#user-sync). +If the user is no longer active in LDAP, they are: + +- Signed out. +- Placed in an `ldap_blocked` status. +- Unable to sign in using any authentication provider until they are reactivated in LDAP. + ### Security risks You should only use LDAP integration if your LDAP users cannot: @@ -70,10 +74,39 @@ You should only use LDAP integration if your LDAP users cannot: ## Configure LDAP -LDAP users must have a set email address, regardless of whether or not it's used +LDAP users must have an email address, regardless of whether or not it's used to sign in. -Here's an example of setting up LDAP with only the required options. +After configuring LDAP, to test the configuration, use the +[LDAP check Rake task](../../raketasks/ldap.md#check). + +### Basic configuration settings + +> The `hosts` configuration setting was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/139) in GitLab 14.7. + +The following configuration settings are available: + +| Setting | Required | Type | Description | +|--------------------|-------------|------|-------------| +| `label` | **{check-circle}** Yes | String | A human-friendly name for your LDAP server. It is displayed on your sign-in page. Example: `'Paris'` or `'Acme, Ltd.'` | +| `host` | **{check-circle}** Yes | String | IP address or domain name of your LDAP server. Ignored when `hosts` is defined. Example: `'ldap.mydomain.com'` | +| `port` | **{check-circle}** Yes | Integer | The port to connect with on your LDAP server. Ignored when `hosts` is defined. Example: `389` or `636` (for SSL) | +| `uid` | **{check-circle}** Yes | String | The LDAP attribute that maps to the username that users use to sign in. Should be the attribute, not the value that maps to the `uid`. Does not affect the GitLab username (see [attributes section](#attribute-configuration-settings)). Example: `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | +| `base` | **{check-circle}** Yes | String | Base where we can search for users. Example: `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | +| `encryption` | **{check-circle}** Yes | String | Encryption method (the `method` key is deprecated in favor of `encryption`). It can have one of three values: `'start_tls'`, `'simple_tls'`, or `'plain'`. `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. If you specify `simple_tls`, usually it's on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. | +| `hosts` | **{dotted-circle}** No | Array of strings and integers | An array of host and port pairs to open connections. Each configured server should have an identical data set. This is not meant to configure multiple distinct LDAP servers, but to configure failover. Hosts are tried in the order they are configured. Example: `[['ldap1.mydomain.com', 636], ['ldap2.mydomain.com', 636]]` | +| `bind_dn` | **{dotted-circle}** No | String | The full DN of the user you bind with. Example: `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | +| `password` | **{dotted-circle}** No | String | The password of the bind user. | +| `verify_certificates` | **{dotted-circle}** No | Boolean | Defaults to `true`. Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. If set to `false`, no validation of the LDAP server's SSL certificate is performed. | +| `timeout` | **{dotted-circle}** No | Integer | Defaults to `10`. Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. | +| `active_directory` | **{dotted-circle}** No | Boolean | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | +| `allow_username_or_email_login` | **{dotted-circle}** No | Boolean | Defaults to `false`. If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | +| `block_auto_created_users` | **{dotted-circle}** No | Boolean | Defaults to `false`. To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator . | +| `user_filter` | **{dotted-circle}** No | String | Filter LDAP users. Follows the format of [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html). GitLab does not support `omniauth-ldap`'s custom filter syntax. Examples of the `user_filter` field syntax:<br/><br/>- `'(employeeType=developer)'`<br/>- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | +| `lowercase_usernames` | **{dotted-circle}** No | Boolean | If enabled, GitLab converts the name to lower case. | +| `retry_empty_result_with_codes` | **{dotted-circle}** No | Array | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | + +Here's an example of setting up LDAP with the basic configuration settings. ::Tabs @@ -89,8 +122,18 @@ Here's an example of setting up LDAP with only the required options. 'host' => 'ldap.mydomain.com', 'port' => 636, 'uid' => 'sAMAccountName', + 'bind_dn' => 'CN=Gitlab,OU=Users,DC=domain,DC=com', + 'password' => '<bind_user_password>', 'encryption' => 'simple_tls', - 'base' => 'dc=example,dc=com' + 'verify_certificates' => true, + 'timeout' => 10, + 'active_directory' => false, + 'user_filter' => '(employeeType=developer)', + 'base' => 'dc=example,dc=com', + 'lowercase_usernames' => 'false', + 'retry_empty_result_with_codes' => [80], + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false } } ``` @@ -122,7 +165,18 @@ Here's an example of setting up LDAP with only the required options. port: 636 uid: 'sAMAccountName' base: 'dc=example,dc=com' + bind_dn: 'CN=Gitlab,OU=Users,DC=domain,DC=com' + password: '<bind_user_password>' encryption: 'simple_tls' + verify_certificates: true + timeout: 10 + active_directory: false + user_filter: '(employeeType=developer)' + base: 'dc=example,dc=com' + lowercase_usernames: false + retry_empty_result_with_codes: [80] + allow_username_or_email_login: false + block_auto_created_users: false ``` 1. Save the file and apply the new values: @@ -154,8 +208,19 @@ For more information, see 'host' => 'ldap.mydomain.com', 'port' => 636, 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', 'base' => 'dc=example,dc=com' + 'bind_dn' => 'CN=Gitlab,OU=Users,DC=domain,DC=com', + 'password' => '<bind_user_password>', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, + 'timeout' => 10, + 'active_directory' => false, + 'user_filter' => '(employeeType=developer)', + 'base' => 'dc=example,dc=com', + 'lowercase_usernames' => 'false', + 'retry_empty_result_with_codes' => [80], + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false } } ``` @@ -180,8 +245,19 @@ For more information, see host: 'ldap.mydomain.com' port: 636 uid: 'sAMAccountName' + base: 'dc=example,dc=com' + bind_dn: 'CN=Gitlab,OU=Users,DC=domain,DC=com' + password: '<bind_user_password>' encryption: 'simple_tls' + verify_certificates: true + timeout: 10 + active_directory: false + user_filter: '(employeeType=developer)' base: 'dc=example,dc=com' + lowercase_usernames: false + retry_empty_result_with_codes: [80] + allow_username_or_email_login: false + block_auto_created_users: false ``` 1. Save the file and restart GitLab: @@ -199,42 +275,6 @@ For more information about the various LDAP options, see the `ldap` setting in ::EndTabs -After configuring LDAP, to test the configuration, use the -[LDAP check Rake task](../../raketasks/ldap.md#check). - -### Basic configuration settings - -> `hosts` configuration setting [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/139) in GitLab 14.7. - -You can configure either: - -- A single LDAP server using `host` and `port`. -- Many LDAP servers using `hosts`. This setting takes precedence over `host` and `port`. GitLab attempts to use the - LDAP servers in the order specified, and the first reachable LDAP server is used. - -These configuration settings are available: - -| Setting | Description | Required | Examples | -|--------------------|-------------|----------|----------| -| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | -| `host` | IP address or domain name of your LDAP server. Ignored when `hosts` is defined. | **{check-circle}** Yes | `'ldap.mydomain.com'` | -| `port` | The port to connect with on your LDAP server. Always an integer, not a string. Ignored when `hosts` is defined. | **{check-circle}** Yes | `389` or `636` (for SSL) | -| `hosts` (GitLab 14.7 and later) | An array of host and port pairs to open connections. | **{dotted-circle}** No | `[['ldap1.mydomain.com', 636], ['ldap2.mydomain.com', 636]]` | -| `uid` | The LDAP attribute that maps to the username that users use to sign in. Should be the attribute, not the value that maps to the `uid`. Does not affect the GitLab username (see [attributes section](#attribute-configuration-settings)). | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | -| `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | -| `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | -| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'`, `'simple_tls'`, or `'plain'`. `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. If you specify `simple_tls`, usually it's on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. | -| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. If set to false, no validation of the LDAP server's SSL certificate is performed. Defaults to true. | **{dotted-circle}** No | boolean | -| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | -| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | -| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | -| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | -| `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | Some examples of the `user_filter` field syntax:<br/><br/>- `'(employeeType=developer)'`<br/>- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | -| `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | -| `retry_empty_result_with_codes` | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | **{dotted-circle}** No | `[80]` | -| `attributes` | A hash of attribute mappings to LDAP for GitLab to use ([see attributes section](#attribute-configuration-settings)). | **{dotted-circle}** No | `'attributes' => { 'username' => ['uid'], 'email' => ['mail', 'email'] },` | - ### SSL configuration settings SSL configuration settings can be configured under `tls_options` name/value @@ -1107,7 +1147,7 @@ read about [Helm LDAP secrets](https://docs.gitlab.com/charts/installation/secre ## Updating LDAP DN and email -When an LDAP server creates a user in GitLab, the user's LDAP distinguished name (DN) is linked to their GitLab account +When an LDAP server creates a user in GitLab, the user's LDAP DN is linked to their GitLab account as an identifier. When a user tries to sign in with LDAP, GitLab tries to find the user using the DN saved on that user's account. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 8cffff7b725..c5624235a76 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -167,7 +167,7 @@ may see the following message: `Access denied for your LDAP account`. We have a workaround, based on toggling the access level of affected users: -1. As an administrator, on the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. As an administrator, on the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. @@ -225,7 +225,7 @@ field contains no data: To resolve this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General**. 1. Expand both of the following: @@ -369,7 +369,7 @@ things to debug the situation. - Ensure the correct [LDAP group link is added to the GitLab group](ldap_synchronization.md#add-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Search for the user. diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 9d9ed563fe5..e95876a8b3c 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -501,7 +501,7 @@ When global group memberships lock is enabled: To enable global group memberships lock: 1. [Configure LDAP](index.md#configure-ldap). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -514,7 +514,7 @@ By default, group members with the Owner role can manage [LDAP group synchroniza GitLab administrators can remove this permission from group Owners: 1. [Configure LDAP](index.md#configure-ldap). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 8ef95872ad4..f3200aea67b 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -554,7 +554,7 @@ For your app, complete the following steps on Casdoor: ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/ruby/gitlab) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/ruby/gitlab/) for more details. Example configuration for Linux package installations (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 1662639dd29..855157321d7 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference diff --git a/doc/administration/auth/test_oidc_oauth.md b/doc/administration/auth/test_oidc_oauth.md index be0ea5c963e..eba911dcb33 100644 --- a/doc/administration/auth/test_oidc_oauth.md +++ b/doc/administration/auth/test_oidc_oauth.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md index dc0724dc561..cd65bf67638 100644 --- a/doc/administration/backup_restore/backup_gitlab.md +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -160,9 +160,9 @@ GitLab uses Redis both as a cache store and to hold persistent data for our back Elasticsearch is an optional database for advanced search. It can improve search in both source-code level, and user generated content in issues, merge requests, and discussions. The [backup command](#backup-command) does _not_ back up Elasticsearch data. Elasticsearch data can be regenerated from PostgreSQL data after a restore. It is possible to [manually back up Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html). -## Command line interface +## Command-line interface -GitLab provides a command line interface to back up your entire instance, +GitLab provides a command-line interface to back up your entire instance, including: - Database @@ -301,7 +301,7 @@ the timestamp is `1493107454_2018_04_25_10.6.4-ce`. ### Backup options -The command line tool GitLab provides to back up your instance can accept more +The command-line tool GitLab provides to back up your instance can accept more options. #### Backup strategy option @@ -435,6 +435,34 @@ sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=producti ::EndTabs +#### Create server-side repository backups + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. + +Instead of storing large repository backups in the backup archive, repository +backups can be configured so that the Gitaly node that hosts each repository is +responsible for creating the backup and streaming it to object storage. This +helps reduce the network resources required to create and restore a backup. + +1. [Configure a server-side backup destination in Gitaly](../gitaly/configure_gitaly.md#configure-server-side-backups). +1. Create a back up using the `REPOSITORIES_SERVER_SIDE` variable. See the following examples. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create REPOSITORIES_SERVER_SIDE=true +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_SERVER_SIDE=true +``` + +::EndTabs + #### Back up Git repositories concurrently > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3. diff --git a/doc/administration/backup_restore/index.md b/doc/administration/backup_restore/index.md index 824fa56f443..56f080a5f9f 100644 --- a/doc/administration/backup_restore/index.md +++ b/doc/administration/backup_restore/index.md @@ -107,7 +107,7 @@ To prepare the new server: ``` 1. Disable periodic background jobs: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Under the Sidekiq dashboard, select **Cron** tab and then @@ -188,7 +188,7 @@ To prepare the new server: 1. [Restore the GitLab backup](#restore-gitlab). 1. Verify that the Redis database restored correctly: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Under the Sidekiq dashboard, verify that the numbers diff --git a/doc/administration/backup_restore/restore_gitlab.md b/doc/administration/backup_restore/restore_gitlab.md index 784a8708de6..c7439a88661 100644 --- a/doc/administration/backup_restore/restore_gitlab.md +++ b/doc/administration/backup_restore/restore_gitlab.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Restore GitLab **(FREE SELF)** -GitLab provides a command line interface to restore your entire installation, +GitLab provides a command-line interface to restore your entire installation, and is flexible enough to fit your needs. The [restore prerequisites section](#restore-prerequisites) includes crucial @@ -287,7 +287,7 @@ Each backup archive contains a full self-contained backup, including those creat ## Restore options -The command line tool GitLab provides to restore from backup can accept more +The command-line tool GitLab provides to restore from backup can accept more options. ### Specify backup to restore when there are more than one @@ -450,3 +450,19 @@ For more information, see: - [Having different owners](https://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us). - Stack Overflow: [Resulting errors](https://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql). + +### Restoring fails due to Git server hook + +While restoring from backup, you can encounter an error when the following are true: + +- A Git Server Hook (`custom_hook`) is configured using the method for [GitLab version 15.10 and earlier](../server_hooks.md) +- Your GitLab version is on version 15.11 and later +- You created symlinks to a directory outside of the GitLab-managed locations + +The error looks like: + +```plaintext +{"level":"fatal","msg":"restore: pipeline: 1 failures encountered:\n - @hashed/path/to/hashed_repository.git (path/to_project): manager: restore custom hooks, \"@hashed/path/to/hashed_repository/<BackupTimestamp>_<GitLabVersion>-ee/001.custom_hooks.tar\": rpc error: code = Internal desc = setting custom hooks: generating prepared vote: walking directory: copying file to hash: read /mnt/gitlab-app/git-data/repositories/+gitaly/tmp/default-repositories.old.<timestamp>.<temporaryfolder>/custom_hooks/compliance-triggers.d: is a directory\n","pid":3256017,"time":"2023-08-10T20:09:44.395Z"} +``` + +To resolve this, you can update the Git [server hooks](../server_hooks.md) for GitLab version 15.11 and later, and create a new backup. diff --git a/doc/administration/broadcast_messages.md b/doc/administration/broadcast_messages.md index 78a4ec798d1..4ed80010220 100644 --- a/doc/administration/broadcast_messages.md +++ b/doc/administration/broadcast_messages.md @@ -57,7 +57,7 @@ To display messages to users on your GitLab instance, add a broadcast message. To add a broadcast message: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Messages**. 1. Select **Add new message**. @@ -86,7 +86,7 @@ If you must make changes to a broadcast message, you can edit it. To edit a broadcast message: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Messages**. 1. From the list of broadcast messages, select the edit button for the message. @@ -101,7 +101,7 @@ You can delete a broadcast message while it's active. To delete a broadcast message: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Messages**. 1. From the list of broadcast messages, select the delete button for the message. diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index 4e98423ea60..7a6316a1e50 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -18,7 +18,7 @@ CI/CD to be disabled by default in new projects by modifying the settings in: - `gitlab.rb` for Linux package installations. Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes -the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). +the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md). For self-compiled installations: diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index b7247e2251f..e291a162fb9 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -9,40 +9,38 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, the GitLab agent server (KAS) became available on GitLab.com at `wss://kas.gitlab.com`. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -The agent server is a component you install together with GitLab. It is required to +The agent server is a component installed together with GitLab. It is required to manage the [GitLab agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent). The KAS acronym refers to the former name, `Kubernetes agent server`. The agent server for Kubernetes is installed and available on GitLab.com at `wss://kas.gitlab.com`. -If you use self-managed GitLab, you must install an agent server or specify an external installation. +If you use self-managed GitLab, by default the agent server is installed and available. ## Installation options -As a GitLab administrator, you can install the agent server: +As a GitLab administrator, you can control the agent server installation: - For [Linux package installations](#for-linux-package-installations). - For [GitLab Helm chart installations](#for-gitlab-helm-chart). ### For Linux package installations -You can enable the agent server for Linux package installations on a single node, or on multiple nodes at once. +The agent server for Linux package installations can be enabled on a single node, or on multiple nodes at once. +By default, the agent server is enabled and available at `ws://gitlab.example.com/-/kubernetes-agent/`. -#### Enable on a single node +#### Disable on a single node -To enable the agent server on a single node: +To disable the agent server on a single node: 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_kas['enable'] = true + gitlab_kas['enable'] = false ``` 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). -For additional configuration options, see the **Enable GitLab KAS** section of the -[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/be52c36c243a3422ec38b7d45d459682a07e195f/files/gitlab-config-template/gitlab.rb.template#L1951). - ##### Configure KAS to listen on a UNIX socket If you use GitLab behind a proxy, KAS might not work correctly. You can resolve this issue on a single-node installation, you can configure KAS to listen on a UNIX socket. @@ -70,6 +68,9 @@ To configure KAS to listen on a UNIX socket: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). +For additional configuration options, see the **GitLab Kubernetes Agent Server** section of the +[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template). + #### Enable on multiple nodes To enable the agent server on multiple nodes: @@ -79,7 +80,6 @@ To enable the agent server on multiple nodes: ```ruby gitlab_kas_external_url 'wss://kas.gitlab.example.com/' - gitlab_kas['enable'] = true gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>' gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>' gitlab_kas['private_api_listen_address'] = '0.0.0.0:8155' @@ -88,7 +88,6 @@ To enable the agent server on multiple nodes: 'OWN_PRIVATE_API_URL' => 'grpc://<ip_or_hostname_of_this_host>:8155' } - gitlab_rails['gitlab_kas_enabled'] = true gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/' gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/' @@ -114,31 +113,7 @@ To enable the agent server on multiple nodes: ### For GitLab Helm Chart -For GitLab [Helm Chart](https://docs.gitlab.com/charts/) installations: - -1. Set `global.kas.enabled` to `true`. For example, in a shell with `helm` and `kubectl` - installed, run: - - ```shell - helm repo add gitlab https://charts.gitlab.io/ - helm repo update - helm upgrade --install gitlab gitlab/gitlab \ - --timeout 600s \ - --set global.hosts.domain=<YOUR_DOMAIN> \ - --set global.hosts.externalIP=<YOUR_IP> \ - --set certmanager-issuer.email=<YOUR_EMAIL> \ - --set global.kas.enabled=true # <-- without this setting, the agent server will not be installed - ``` - -1. To configure the agent server, use a `gitlab.kas` sub-section in your `values.yaml` file: - - ```yaml - gitlab: - kas: - # put your custom options here - ``` - -For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). +See [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). ## Kubernetes API proxy cookie diff --git a/doc/administration/credentials_inventory.md b/doc/administration/credentials_inventory.md index 482d46498e3..39cbf4e0dc8 100644 --- a/doc/administration/credentials_inventory.md +++ b/doc/administration/credentials_inventory.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto @@ -10,42 +10,45 @@ type: howto > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20912) in GitLab 12.6. > - [Bot-created access tokens not displayed in personal access token list](https://gitlab.com/gitlab-org/gitlab/-/issues/351759) in GitLab 14.9. -GitLab administrators are responsible for the overall security of their instance. To assist, GitLab -provides a Credentials inventory to keep track of all the credentials that can be used to access -their self-managed instance. +As a GitLab administrator, you are responsible for the overall security of your instance. +To assist, GitLab provides an inventory of all the credentials that can be used to access +your self-managed instance. -Use Credentials inventory to see for your GitLab instance all: +In the credentials inventory, you can view all: -- Personal access tokens (PAT). -- Project access tokens (GitLab 14.8 and later). +- Personal access tokens (PATs). +- Project access tokens (introduced in GitLab 14.8). +- Group access tokens ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102959) in GitLab 15.6). - SSH keys. - GPG keys. -You can also [revoke](#revoke-a-users-personal-access-token) and [delete](#delete-a-users-ssh-key) and see: +You can also [revoke](#revoke-a-users-personal-access-token), [delete](#delete-a-users-ssh-key), and view: - Who they belong to. - Their access scope. - Their usage pattern. -- When they expire. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. -- When they were revoked. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. - -To access the Credentials inventory: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Credentials**. +- [In GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/214809), when they: + - Expire. + - Were revoked. ## Revoke a user's personal access token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214811) in GitLab 13.4. -If you see a **Revoke** button, you can revoke that user's PAT. Whether you see a **Revoke** button depends on the token state, and if an expiration date has been set. For more information, see the following table: +You can revoke a user's personal access token. + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Credentials**. +1. By the personal access token, select **Revoke**. -| Token state | Show Revoke button? | Comments | -|-------------|---------------------|----------------------------------------------------------------------------| -| Active | Yes | Allows administrators to revoke the PAT, such as for a compromised account | -| Expired | No | Not applicable; token is already expired | -| Revoked | No | Not applicable; token is already revoked | +If a **Revoke** button is not available, the token may be expired or revoked, or an expiration date set. + +| Token state | Revoke button displayed? | Comments | +|-------------|--------------------------|----------------------------------------------------------------------------| +| Active | Yes | Allows administrators to revoke the PAT, such as for a compromised account | +| Expired | No | Not applicable; token is already expired | +| Revoked | No | Not applicable; token is already revoked | When a PAT is revoked from the credentials inventory, the instance notifies the user by email. @@ -55,10 +58,13 @@ When a PAT is revoked from the credentials inventory, the instance notifies the > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/243833) in GitLab 14.8. -The **Revoke** button next to a project access token can be selected to revoke that particular project access token. This both: +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Credentials**. +1. Select the **Project Access Tokens** tab. +1. By the project access token, select **Revoke**. -- Revokes the token project access token. -- Enqueues a background worker to delete the project bot user. +The project access token is revoked and a background worker is queued to delete the project bot user.  @@ -66,8 +72,13 @@ The **Revoke** button next to a project access token can be selected to revoke t > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225248) in GitLab 13.5. -You can **Delete** a user's SSH key by navigating to the credentials inventory's SSH Keys tab. -The instance then notifies the user. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Credentials**. +1. Select the **SSH Keys** tab. +1. By the SSH key, select **Delete**. + +The instance notifies the user.  @@ -76,11 +87,11 @@ The instance then notifies the user. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282429) in GitLab 13.10. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292961) in GitLab 13.12. -You can view all existing GPG in your GitLab instance by navigating to the +You can view all existing GPG in your GitLab instance by going to the credentials inventory GPG Keys tab, as well as the following properties: - Who the GPG key belongs to. - The ID of the GPG key. -- Whether the GPG key is [verified or unverified](../user/project/repository/gpg_signed_commits/index.md) +- Whether the GPG key is [verified or unverified](../user/project/repository/signed_commits/gpg.md).  diff --git a/doc/administration/custom_project_templates.md b/doc/administration/custom_project_templates.md index 2bbbb5649e6..a63201b721e 100644 --- a/doc/administration/custom_project_templates.md +++ b/doc/administration/custom_project_templates.md @@ -16,7 +16,7 @@ Every project in the group, but not its subgroups, can be selected when a new pr is created, based on the user's access permissions: - Public projects can be selected by any authenticated user as a template for a new project, - if all enabled [project features](../user/project/settings/index.md#configure-project-visibility-features-and-permissions) + if all enabled [project features](../user/project/settings/index.md#configure-project-features-and-permissions) except for **GitLab Pages** and **Security and Compliance** are set to **Everyone With Access**. The same applies to internal projects. - Private projects can be selected only by users who are members of the projects. @@ -33,7 +33,7 @@ To set project templates at the group level, see [Custom group-level project tem To select the group to use as the source for the project templates: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Templates**. 1. Expand **Custom project templates**. diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index b995c837136..9008b5dbc11 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -25,7 +25,7 @@ GitLab Dedicated Engineers also don't have direct access to tenant environments, To request the creation of a new GitLab Dedicated environment for your organization, you must provide the following information to your account team: - Expected number of users. -- Desired primary region: Primary AWS region in which your data is stored (do note the [list of unsupported regions](../../subscriptions/gitlab_dedicated/index.md#aws-regions-not-supported)). +- Desired primary region: Primary AWS region in which your data is stored (take note of [unavailable AWS regions](../../subscriptions/gitlab_dedicated/index.md#unavailable-aws-regions)). - Desired secondary region: Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster. - Desired backup region: An AWS region where the primary backups of your data are replicated. This can be the same as the primary or secondary region or different. - Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You get to choose the subdomain name where your instance is accessible from (for example, `customer_name.gitlab-dedicated.com`). @@ -39,13 +39,14 @@ When onboarding, you must also specify your preference for the weekly four-hour Available scheduled maintenance windows, performed outside standard working hours: -- APAC: Wednesday 1 AM - 5 AM UTC +- APAC: Wednesday 1 PM - 5 PM UTC - EU: Tuesday 1 AM - 5 AM UTC - AMER Option 1: Tuesday 7 AM - 11 AM UTC - AMER Option 2: Sunday 9 PM - Monday 1 AM UTC Consider the following notes: +- The Dedicated instance is not expected to be down the entire duration of the maintenance window. Occasionally, a small period of downtime (on the order of a few tens of seconds) can occur while compute resources restart after they are upgraded. If it occurs, this small period of downtime typically happens during the first half of the maintenance window. Long-running connections may be interrupted during this period. To mitigate this, clients should implement strategies like automatic recovery and retry. Longer periods of downtime during the maintenance window are rare, and GitLab provides notice if longer downtime is anticipated. - In case of a performance degradation or downtime during the scheduled maintenance window, the impact to [the system SLA](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/) is not counted. - The weekly scheduled maintenance window can be postponed into another window within the same week. @@ -79,20 +80,30 @@ The turnaround time for processing configuration change requests is [documented ### Encrypted Data At Rest (BYOK) -If you want your GitLab data to be encrypted at rest, the KMS keys used must be accessible by GitLab services. KMS keys can be used in two modes for this purpose: +NOTE: +To enable BYOK, you must do it during onboarding. + +You can opt to encrypt your GitLab data at rest with AWS KMS keys, which must be made accessible to GitLab Dedicated infrastructure. GitLab Dedicated only supports keys with AWS-managed key material (the [AWS_KMS](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-origin) origin type). -1. Per-service KMS keys (Backup, EBS, RDS, S3), or -1. One KMS key for all services. +For instructions on how to create and manage KMS keys, see the [AWS KMS documentation](https://docs.aws.amazon.com/kms/latest/developerguide/getting-started.html). -If you use a key per service, all services must be encrypted at rest. Selective enablement of this feature is not supported. +In GitLab Dedicated, you can use KMS keys in two ways: -The keys provided have to reside in the same primary, secondary and backup region specified during [onboarding](#onboarding). +- One KMS key for all services +- Per-service KMS keys (Backup, EBS, RDS, S3) + - Keys do not need to be unique to each service. + - All services must be encrypted at rest. + - Selective enablement of this feature is not supported. + - Keys do not need to be unique to each service. -For instructions on how to create and manage KMS keys, visit [Managing keys](https://docs.aws.amazon.com/kms/latest/developerguide/getting-started.html) in the AWS KMS documentation. +Make sure the AWS KMS keys are replicated to your desired primary, secondary, and backup region specified during [onboarding](#onboarding). -GitLab Dedicated supports only AWS managed KMS keys with KMS [as key material](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-origin). +#### Create KMS keys in AWS -To create a KMS key using the AWS Console: +To enable BYOK, indicate on your onboarding ticket that you'd like to use this functionality. +GitLab will provide you with your AWS account ID which is necessary to enable BYOK. + +After you have received the AWS account ID, create your KMS keys using the AWS Console: 1. In `Configure key`, select: 1. Key type: **Symmetrical** @@ -185,7 +196,7 @@ The last page asks you to confirm the KMS key policy. It should look similar to } ``` -Make sure the AWS KMS keys are replicated to your desired primary, secondary and backup region specified during [onboarding](#onboarding). +Make sure the AWS KMS keys are replicated to your desired primary, secondary and backup region specified during [onboarding](#onboarding). After you have created the keys, send GitLab the corresponding ARNs of each key so that GitLab can use to encrypt the data stored in your Dedicated instance. ### Inbound Private Link @@ -206,14 +217,43 @@ To enable the Inbound Private Link: ### Outbound Private Link -Outbound Private Links allow the GitLab Dedicated instance to securely communicate with services running in your VPC on AWS. This type of connection can execute [webhooks](../../user/project/integrations/webhooks.md) where the targeted services are running in your VPC, import or mirror projects and repositories, or use any other GitLab functionality to access private services. +Consider the following when using Outbound Private Links: + +- Outbound Private Links allow the GitLab Dedicated instance to securely communicate with services running in your VPC on AWS. This type of connection + can execute [webhooks](../../user/project/integrations/webhooks.md) where the targeted services are running in your VPC, import or mirror projects + and repositories, or use any other GitLab functionality to access private services. +- You can only establish Private Links between VPCs in the same region. Therefore, you can only establish a connection in the regions you selected for + your Dedicated instance. +- The Network Load Balancer (NLB) that backs the Endpoint Service at your end must be enabled in at least one of the Availability Zones to which your Dedicated tenant was + deployed. This is not the user-facing name such as `us-east-1a`, but the underlying [Availability Zone ID](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html). + If you did not specify these during onboarding to Dedicated, you must either: + - Ask for the Availability Zone IDs in the ticket you raise to enable the link and ensure the NLB is enabled in those AZs, or + - Ensure the NLB has is enabled in every Availability Zone in the region. To enable an Outbound Private Link: -1. In your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650), GitLab provides you with an IAM role ARN that connects to your endpoint service. You can then add this ARN to the allowlist on your side to restrict connections to your endpoint service. -1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service is available to GitLab Dedicated. Provide the associated `Service Endpoint Name` on the [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). -1. When creating the Endpoint service, you must provide GitLab with a [verified Private DNS Name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html#verify-domain-ownership) for your service. Optionally, if you would like GitLab Dedicated to reach your service via other aliases, you have the ability to specify a list of Private Hosted Zone (PHZ) entries. With this option, GitLab sets up a Private Hosted Zone with DNS names aliased to the verified Private DNS name. To enable this functionality, you must provide GitLab a list of PHZ entries on your support ticket. After the PHZ is created in the tenant environment, DNS resolution of any of the provided records correctly resolves to the PrivateLink endpoint. -1. GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any outbound calls made from the tenant GitLab instance are directed through the PrivateLink into your VPC. +1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service + will be available to GitLab Dedicated. Provide the associated `Service Endpoint Name` on a new + [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). +1. In your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650), GitLab will provide you with the ARN of an + IAM role that will be initiating the connection to your endpoint service. You must ensure this ARN is included, or otherwise covered by other + entries, in the list of "Allowed Principals" on the Endpoint Service, as described by the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). + Though it's optional, you should you add it explicitly, allowing you to set `Acceptance required` to No so that Dedicated can connect in a single operation. + If you leave `Acceptance required` as Yes, then you must manually accept the connection after Dedicated has initiated it. +1. To connect to services using the Endpoint, the Dedicated services require a DNS name. Private Link automatically creates an internal name, but + it is machine-generated and not generally directly useful. There are two options available: + - In your Endpoint Service, enable [Private DNS name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html), perform the + required validation, and let GitLab know in the support ticket that you are using this option. If `Acceptance Required` is set to Yes on your + Endpoint Service, also note this on the support ticket because Dedicated will need to initiate the connection without Private DNS, wait for you + to confirm it has been accepted, and then update the connection to enable the use of Private DNS. + - Dedicated can manage a Private Hosted Zone (PHZ) within the Dedicated AWS Account and alias any arbitrary DNS names to the Endpoint, directing + requests for those names to your Endpoint Service. This may be useful if you have multiple DNS names/aliases that will be accessed using a + single Endpoint (for example, if you are running a reverse proxy to connect to more than one service in your environment), or if the domain you + want to use is not public and cannot be validated for use by Private DNS. Let GitLab know on the support ticket if you are using this option and + provide a list of DNS names that should resolve to the Private Link Endpoint. This list can be updated as needed in future. + +GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any matching outbound +connections made from the tenant GitLab instance are directed through the PrivateLink into your VPC. #### Custom certificates @@ -258,9 +298,9 @@ To activate SAML for your GitLab Dedicated instance: "admin_groups": [ // optional ], - "idp_cert_fingerprint": "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8", + "idp_cert_fingerprint": "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8", "idp_sso_target_url": "https://login.example.com/idp", - "label": "IDP Name", + "label": "IDP Name", "name_identifier_format": "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent", "security": { // optional @@ -305,7 +345,8 @@ To enable group sync: ### Access to application logs -GitLab [application logs](../../administration/logs/index.md) are delivered to an S3 bucket in the GitLab tenant account, which can be shared with you. +GitLab [application logs](../../administration/logs/index.md) are delivered to an S3 bucket in the GitLab tenant account, which can be shared with you. Logs stored in the S3 bucket are retained indefinitely, until the 1 year retention policy is fully enforced. GitLab team members can view more information in this confidential issue: +`https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/483`. To gain read only access to this bucket: diff --git a/doc/administration/diff_limits.md b/doc/administration/diff_limits.md index 48b52950ee3..dcf5a2b1b32 100644 --- a/doc/administration/diff_limits.md +++ b/doc/administration/diff_limits.md @@ -33,7 +33,7 @@ set values are presented as **Too large** are cannot be expanded in the UI. To configure these values: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Diff limits**. diff --git a/doc/administration/email_from_gitlab.md b/doc/administration/email_from_gitlab.md index 9272f58d2b9..ec231d25da2 100644 --- a/doc/administration/email_from_gitlab.md +++ b/doc/administration/email_from_gitlab.md @@ -22,7 +22,7 @@ For information about email notifications originating from GitLab, read ## Sending emails to users from GitLab -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select **Send email to users**. diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index b8e04b6fa08..4ffb1495ec8 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -13,7 +13,7 @@ GitLab can read settings for certain features from encrypted settings files. The - [Incoming email `user` and `password`](incoming_email.md#use-encrypted-credentials). - [LDAP `bind_dn` and `password`](auth/ldap/index.md#use-encrypted-credentials). -- [Service Desk email `user` and `password`](../user/project/service_desk/index.md#use-encrypted-credentials). +- [Service Desk email `user` and `password`](../user/project/service_desk/configure.md#use-encrypted-credentials). - [SMTP `user_name` and `password`](raketasks/smtp.md#secrets). To enable the encrypted configuration settings, a new base key must be generated for diff --git a/doc/administration/external_users.md b/doc/administration/external_users.md index 9be49fab95f..56a6debb7b1 100644 --- a/doc/administration/external_users.md +++ b/doc/administration/external_users.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -30,7 +30,7 @@ For example, if an external user is added as Guest, and your project is internal private, they do not have access to the code; you need to grant the external user access at the Reporter level or above if you want them to have access to the code. You should always take into account the -[project's visibility and permissions settings](../user/project/settings/index.md#configure-project-visibility-features-and-permissions) +[project's visibility](../user/public_access.md#change-project-visibility) and [permissions settings](../user/project/settings/index.md#configure-project-features-and-permissions) as well as the permission level of the user. NOTE: @@ -40,7 +40,7 @@ An administrator can flag a user as external by either of the following methods: - [Through the API](../api/users.md#user-modification). - Using the GitLab UI: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. There, you can find the option to flag the user as external. @@ -56,7 +56,7 @@ Additionally, users can be set as external users using: By default, new users are not set as external users. This behavior can be changed by an administrator: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index a67edc815c8..a31261892bb 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -49,7 +49,7 @@ Feature.enable('geo_repository_verification') On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming @@ -60,7 +60,7 @@ On the **primary** site: On the **secondary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming @@ -89,7 +89,7 @@ increase load and vice versa. On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** for the **primary** site to customize the minimum @@ -138,7 +138,7 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** sites have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: 1. On the **primary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 151c0f7d9fb..0c160e85570 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -67,7 +67,7 @@ must disable the **primary** site. - Physically disconnect a machine. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. + you may wish to maintain a low TTL to ensure fast propagation of DNS changes. ### Step 3. Promoting a **secondary** site @@ -79,11 +79,7 @@ This issue has been fixed in GitLab 13.4 and later. Note the following when promoting a secondary: -- If replication was paused on the secondary site (for example as a part of - upgrading, while you were running a version of GitLab earlier than 13.4), you - _must_ [enable the site by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. If the secondary site - [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion +- If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. - A new **secondary** should not be added at this time. If you want to add a new @@ -92,9 +88,6 @@ Note the following when promoting a secondary: - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error message during this process, for more information, see this [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-site). -- If you run into errors when using `--force` or `--skip-preflight-checks` before 13.5 during this process, - for more information, see this - [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). #### Promoting a **secondary** site running on a single node running GitLab 14.5 and later @@ -225,8 +218,6 @@ do this manually. sudo gitlab-ctl promote-db ``` - In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). - 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to reflect its new status as **primary** by removing any of the following lines that might be present: @@ -708,6 +699,13 @@ If you are running GitLab 14.5 and later: kubectl --namespace gitlab exec -ti gitlab-geo-toolbox-XXX -- gitlab-rake geo:set_secondary_as_primary ``` + Environment variables can be provided to modify the behavior of the task. The + available variables are: + + | Name | Default value | Description | + | ---- | ------------- | ------- | + | `ENABLE_SILENT_MODE` | `false` | If `true`, enables [Silent Mode](../../silent_mode/index.md) before promotion (GitLab 16.4 and later) | + If you are running GitLab 14.4 and earlier: 1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 6ac67c3d21e..cdc52b55c3e 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -90,6 +90,11 @@ gitlab-ctl promotion-preflight-checks Each step is described in more detail below. +### DNS TTL + +If you plan to [update the primary domain DNS record](index.md#step-4-optional-updating-the-primary-domain-dns-record), +you may wish to maintain a low TTL to ensure fast propagation of DNS changes. + ### Object storage If you have a large GitLab installation or cannot tolerate downtime, consider @@ -149,7 +154,7 @@ ensure these processes are close to 100% as possible during active use. On the **secondary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. Replicated objects (shown in green) should be close to 100%, @@ -178,7 +183,7 @@ This [content was moved to another location](background_verification.md). On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Messages**. 1. Add a message notifying users on the maintenance window. @@ -192,7 +197,7 @@ To ensure that all data is replicated to a secondary site, updates (write reques be disabled on the **primary** site: 1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** site. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. @@ -203,13 +208,10 @@ be disabled on the **primary** site: ## Finish replicating and verifying all data -NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appears to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). - 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -225,7 +227,7 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index a96963cfea1..787e7c4669e 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -63,12 +63,9 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, because there isn't provided an automated way to promote a Geo replica and perform a failover. -NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). - On the **secondary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites** to see its status. Replicated objects (shown in green) should be close to 100%, @@ -108,7 +105,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -124,7 +121,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -257,8 +254,6 @@ Data that was created on the primary while the secondary was paused is lost. sudo gitlab-ctl promote-db ``` - In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). - 1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 7f94d6e4c1a..e29b19ca040 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -51,9 +51,6 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. -NOTE: -GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses appears to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). - On the **secondary** site, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of @@ -118,7 +115,7 @@ follow these steps to avoid unnecessary data loss: connection. 1. On the **primary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. @@ -137,7 +134,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -153,7 +150,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index 2e9a637eb5c..d0f94ba39f5 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -14,8 +14,8 @@ these definitions yet. These are the defined terms to describe all aspects of Geo. Using a set of clearly defined terms helps us to communicate efficiently and avoids confusion. The language - on this page aims to be [ubiquitous](https://about.gitlab.com/handbook/communication/#ubiquitous-language) - and [as simple as possible](https://about.gitlab.com/handbook/communication/#simple-language). + on this page aims to be [ubiquitous](https://handbook.gitlab.com/handbook/communication/#ubiquitous-language) + and [as simple as possible](https://handbook.gitlab.com/handbook/communication/#simple-language). We provide example diagrams and statements to demonstrate correct usage of terms. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index f300549223a..78bd685e06f 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -161,7 +161,7 @@ public URL of the primary site is used. To update the internal URL of the primary Geo site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** on the primary site. @@ -193,9 +193,12 @@ This new architecture allows GitLab to be resilient to connectivity issues betwe WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. -- Pushing directly to a **secondary** site redirects (for HTTP) or proxies (for SSH) the request to the **primary** site instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded in the URI. For example, `https://user:password@secondary.tld`. +- Pushing directly to a **secondary** site redirects (for HTTP) or proxies (for SSH) the request to the **primary** site instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381). The limitation is that you cannot use Git over HTTP with credentials embedded in the URI, for example, `https://user:personal-access-token@secondary.tld`. For more information, see how to [use a Geo Site](replication/usage.md). - The **primary** site has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** site to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). -- The installation takes multiple manual steps that together can take about an hour depending on circumstances. Consider using [the GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to deploy and operate production GitLab instances based on our [Reference Architectures](../reference_architectures/index.md), including automation of common daily tasks. We are planning to [improve Geo's installation even further](https://gitlab.com/groups/gitlab-org/-/epics/1465). +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. Consider using the + [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) Terraform and Ansible scripts to deploy and operate production + GitLab instances based on our [Reference Architectures](../reference_architectures/index.md), including automation of common daily tasks. + [Epic 1465](https://gitlab.com/groups/gitlab-org/-/epics/1465) proposes to improve Geo installation even more. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site. - Using Geo secondary sites to accelerate runners is not officially supported. Support for this functionality is planned and can be tracked in [epic 9779](https://gitlab.com/groups/gitlab-org/-/epics/9779). If a replication lag occurs between the primary and secondary site, and the pipeline ref is not available on the secondary site when the job is executed, the job will fail. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). @@ -243,9 +246,9 @@ Linux package-managed database. External databases are not supported. In some circumstances, like during [upgrades](replication/upgrading_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. -Pausing and resuming replication is done via a command line tool from the node in the secondary site where the `postgresql` service is enabled. +Pausing and resuming replication is done through a command-line tool from the node in the secondary site where the `postgresql` service is enabled. -If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_name_name` on the application node. +If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_node_name` on the application node. **To Pause: (from secondary)** diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index f63f8edbc72..8722ab574c3 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -95,7 +95,14 @@ This causes all SSH requests to the newly promoted **primary** site to fail due to SSH host key mismatch. To prevent this, the primary SSH host keys must be manually replicated to the **secondary** site. -1. SSH into **each node on your secondary** site and login as the `root` user: +The SSH host key path depends on the used software: + +- If you use OpenSSH, the path is `/etc/ssh`. +- If you use [`gitlab-sshd`](../../operations/gitlab_sshd.md), the path is `/var/opt/gitlab/gitlab-sshd`. + +In the following steps, replace `<ssh_host_key_path>` with the one you're using: + +1. SSH into **each Rails node on your secondary** site and log in as the `root` user: ```shell sudo -i @@ -104,40 +111,40 @@ keys must be manually replicated to the **secondary** site. 1. Make a backup of any existing SSH host keys: ```shell - find /etc/ssh -iname 'ssh_host_*' -exec cp {} {}.backup.`date +%F` \; + find <ssh_host_key_path> -iname 'ssh_host_*' -exec cp {} {}.backup.`date +%F` \; ``` -1. Copy OpenSSH host keys from the **primary** site: +1. Copy the SSH host keys from the **primary** site: If you can access one of the **nodes on your primary** site serving SSH traffic (usually, the main GitLab Rails application nodes) using the **root** user: ```shell # Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server - scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + scp root@<primary_node_fqdn>:<ssh_host_key_path>/ssh_host_*_key* <ssh_host_key_path> ``` If you only have access through a user with `sudo` privileges: ```shell # Run this from the node on your primary site: - sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz <ssh_host_key_path>/ssh_host_*_key* # Run this on each node on your secondary site: scp <user_with_sudo>@<primary_site_fqdn>:geo-host-key.tar.gz . - tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + tar zxvf ~/geo-host-key.tar.gz -C <ssh_host_key_path> ``` -1. On **each node on your secondary** site, ensure the file permissions are correct: +1. On **each Rails node on your secondary** site, ensure the file permissions are correct: ```shell - chown root:root /etc/ssh/ssh_host_*_key* - chmod 0600 /etc/ssh/ssh_host_*_key + chown root:root <ssh_host_key_path>/ssh_host_*_key* + chmod 0600 <ssh_host_key_path>/ssh_host_*_key ``` 1. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site: ```shell - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + for file in <ssh_host_key_path>/ssh_host_*_key; do ssh-keygen -lf $file; done ``` You should get an output similar to this one and they should be identical on both nodes: @@ -153,24 +160,32 @@ keys must be manually replicated to the **secondary** site. ```shell # This will print the fingerprint for private keys: - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + for file in <ssh_host_key_path>/ssh_host_*_key; do ssh-keygen -lf $file; done # This will print the fingerprint for public keys: - for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + for file in <ssh_host_key_path>/ssh_host_*_key.pub; do ssh-keygen -lf $file; done ``` NOTE: The output for private keys and public keys command should generate the same fingerprint. -1. Restart `sshd` on **each node on your secondary** site: +1. Restart either `sshd` for OpenSSH or the `gitlab-sshd` service on **each Rails node on your secondary** site: - ```shell - # Debian or Ubuntu installations - sudo service ssh reload + - For OpenSSH: - # CentOS installations - sudo service sshd reload - ``` + ```shell + # Debian or Ubuntu installations + sudo service ssh reload + + # CentOS installations + sudo service sshd reload + ``` + + - For `gitlab-sshd`: + + ```shell + sudo gitlab-ctl restart gitlab-sshd + ``` 1. Verify SSH is still functional. @@ -202,7 +217,7 @@ keys must be manually replicated to the **secondary** site. ``` 1. Navigate to the Primary Node GitLab Instance: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**. @@ -312,7 +327,7 @@ method to be enabled. This is enabled by default, but if converting an existing On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. @@ -326,7 +341,7 @@ On the **primary** site: You can sign in to the **secondary** site with the same credentials you used with the **primary** site. After you sign in: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Verify that it's correctly identified as a **secondary** Geo site, and that diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index 6dde611a20d..65ecba0d04a 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -166,7 +166,7 @@ For each application and Sidekiq node on the **secondary** site: To verify Container Registry replication is working, on the **secondary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. The initial replication, or "backfill", is probably still in progress. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index cab3ef581cb..896501128f7 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -36,39 +36,40 @@ verification methods: | Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Group wiki repository | Geo with Gitaly | Gitaly Checksum | | Blobs | User uploads _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | LFS objects _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | +| Blobs | CI job artifacts _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | -| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _SHA256 checksum_ | -| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _SHA256 checksum_ | +| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | SHA256 checksum (*3*) | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Terraform Module Registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Terraform Module Registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Terraform Module Registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Pages _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pages _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | CI Secure Files _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | CI Secure Files _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI Secure Files _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Incident Metric Images _(file system)_ | Geo with API/Managed | SHA256 checksum | -| Blobs | Incident Metric Images _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Incident Metric Images _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Alert Metric Images _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Alert Metric Images _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Alert Metric Images _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum (*3*) | | Blobs | Dependency Proxy Images_(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Dependency Proxy Images _(object storage)_ | Geo with API/managed (*2*) | _Not implemented_ | +| Blobs | Dependency Proxy Images _(object storage)_ | Geo with API/managed (*2*) | SHA256 checksum (*3*) | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance native replication feature. +- (*3*): Object Storage verification is behind a [feature flag](../../feature_flags.md), `geo_object_storage_verification`, [introduced in 16.4](https://gitlab.com/groups/gitlab-org/-/epics/8056) and enabled by default. It uses a checksum of the file size to verify the files. ### Git repositories @@ -188,31 +189,31 @@ successfully, you must replicate their data using some other means. |Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | GitLab-managed object storage replication (added in GitLab version) | GitLab-managed object storage verification (added in GitLab version) | Notes | |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| -|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).<br /><br /> All projects, including [archived projects](../../../user/project/settings/index.md#archive-a-project), are replicated. | -|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_wiki_repository_replication`, enabled by default in (15.11). | -|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [**Yes** (16.3)](https://gitlab.com/gitlab-org/gitlab/-/issues/323897) | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | -|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | -|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | -|[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | -|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | -|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | +|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | Not applicable | Not applicable | | +|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | Not applicable | Not applicable | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).<br /><br /> All projects, including [archived projects](../../../user/project/settings/index.md#archive-a-project), are replicated. | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | Not applicable | Not applicable | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_wiki_repository_replication`, enabled by default in (15.11). | +|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [**Yes** (16.3)](https://gitlab.com/gitlab-org/gitlab/-/issues/323897) | Not applicable | Not applicable | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | +|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | Not applicable | Not applicable | | +|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | Not applicable | Not applicable | | +|[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | +|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Persists additional artifacts after a pipeline completes. | +|[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (16.4)](object_storage.md) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | -|[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | N/A | N/A | Designs also require replication of LFS objects and Uploads. Replication is behind the feature flag `geo_design_management_repository_replication`, enabled by default.| -|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | -|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | -|[Project-level Secure files](../../../ci/secure_files/index.md) | **Yes** (15.3) | **Yes** (15.3) | **Yes** (15.3) | [No](object_storage.md#verification-of-files-in-object-storage) | | -| [Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | **Yes** (15.5) | **Yes**(15.5) | **Yes** (15.5) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication/Verification is handled via the Uploads data type. | | -|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | **Yes** (15.5) | **Yes** (15.5) | **Yes** (15.5) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication/Verification is handled via the Uploads data type. | -|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | +|[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | Not applicable | Not applicable | Designs also require replication of LFS objects and Uploads. | +|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | Not applicable | Not applicable | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | +|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | +|[Project-level Secure files](../../../ci/secure_files/index.md) | **Yes** (15.3) | **Yes** (15.3) | **Yes** (15.3) | [**Yes** (16.4)](object_storage.md) | | +| [Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | **Yes** (15.5) | **Yes**(15.5) | **Yes** (15.5) | [**Yes** (16.4)](object_storage.md) | Replication/Verification is handled via the Uploads data type. | | +|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | **Yes** (15.5) | **Yes** (15.5) | **Yes** (15.5) | [**Yes** (16.4)](object_storage.md) | Replication/Verification is handled via the Uploads data type. | +|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | Not applicable | Not applicable | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/advanced_search/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | -|[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [No](object_storage.md#verification-of-files-in-object-storage) | | +|[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (16.4)](object_storage.md) | | |[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | <sup>1</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index bedcca7e311..42c37a79ec2 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -36,7 +36,7 @@ to do that. To remove the **primary** site: 1. [Remove all secondary Geo sites](remove_geo_site.md) -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Remove** for the **primary** node. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 86db8186a13..3b6b214d9e3 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -7,6 +7,8 @@ type: howto # Geo with Object storage **(PREMIUM SELF)** +> Verification of files stored in object storage was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/8056) in GitLab 16.4 [with a flag](../../feature_flags.md) named `geo_object_storage_verification`. Enabled by default. + Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). **Secondary** sites can use one of the following: @@ -41,7 +43,7 @@ whether they are stored on the local file system or in object storage. To enable GitLab replication: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. Select **Edit** on the **secondary** site. @@ -86,7 +88,3 @@ For manual synchronization, or scheduled by `cron`, see: - [`s3cmd sync`](https://s3tools.org/s3cmd-sync) - [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) - -## Verification of files in object storage - -[Files stored in object storage are not verified.](https://gitlab.com/groups/gitlab-org/-/epics/8056) diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index de0ad3fe2fb..a55c05050a0 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -9,7 +9,7 @@ type: howto **Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Nodes**. 1. For the **secondary** site you want to remove, select **Remove**. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index ed11307f57b..dd2693f4ba7 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -20,14 +20,14 @@ Here is a list of steps you should take to attempt to fix problem: Before attempting more advanced troubleshooting: -- Check [the health of the **secondary** site](#check-the-health-of-the-secondary-site). +- Check [the health of the Geo sites](#check-the-health-of-the-geo-sites). - Check [if PostgreSQL replication is working](#check-if-postgresql-replication-is-working). -### Check the health of the **secondary** site +### Check the health of the Geo sites On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. @@ -116,6 +116,38 @@ The following environment variables are supported. |`NTP_PORT` | The NTP port the host listens on. |`ntp`| |`NTP_TIMEOUT`| The NTP timeout in seconds. | The value defined in the `net-ntp` Ruby library ([60 seconds](https://github.com/zencoder/net-ntp/blob/3d0990214f439a5127782e0f50faeaf2c8ca7023/lib/net/ntp/ntp.rb#L6)). | +If the Rake task skips the `OpenSSH configured to use AuthorizedKeysCommand` check, the +following output displays: + +```plaintext +OpenSSH configured to use AuthorizedKeysCommand ... skipped + Reason: + Cannot access OpenSSH configuration file + Try fixing it: + This is expected if you are using SELinux. You may want to check configuration manually + For more information see: + doc/administration/operations/fast_ssh_key_lookup.md +``` + +This issue may occur if: + +- You [use SELinux](../../operations/fast_ssh_key_lookup.md#selinux-support-and-limitations). +- You don't use SELinux, and the `git` user cannot access the OpenSSH configuration file due to restricted file permissions. + +In the latter case, the following output shows that only the `root` user can read this file: + +```plaintext +sudo stat -c '%G:%U %A %a %n' /etc/ssh/sshd_config + +root:root -rw------- 600 /etc/ssh/sshd_config +``` + +To allow the `git` user to read the OpenSSH configuration file, without changing the file owner or permissions, use `acl`: + +```plaintext +sudo setfacl -m u:git:r /etc/ssh/sshd_config +``` + #### Sync status Rake task Current sync information can be found manually by running this Rake task on any @@ -142,11 +174,10 @@ http://secondary.example.com/ Health Status: Healthy Repositories: succeeded 12345 / total 12345 (100%) Verified Repositories: succeeded 12345 / total 12345 (100%) - Wikis: succeeded 6789 / total 6789 (100%) - Verified Wikis: succeeded 6789 / total 6789 (100%) + Project Wiki Repositories: succeeded 6789 / total 6789 (100%) Attachments: succeeded 4 / total 4 (100%) CI job artifacts: succeeded 0 / total 0 (0%) - Design repositories: succeeded 1 / total 1 (100%) + Design management repositories: succeeded 1 / total 1 (100%) LFS Objects: failed 1 / succeeded 2 / total 3 (67%) Merge Request Diffs: succeeded 0 / total 0 (0%) Package Files: failed 1 / succeeded 2 / total 3 (67%) @@ -160,6 +191,7 @@ http://secondary.example.com/ Terraform State Versions Verified: succeeded 0 / total 10 (0%) Snippet Repositories Verified: succeeded 99 / total 100 (99%) Pipeline Artifacts Verified: succeeded 0 / total 10 (0%) + Project Wiki Repositories Verified: succeeded 6789 / total 6789 (100%) Sync Settings: Full Database replication lag: 0 seconds Last event ID seen from primary: 12345 (about 2 minutes ago) @@ -188,119 +220,7 @@ If you notice replication or verification failures, you can try to [resolve them If there are Repository check failures, you can try to [resolve them](#find-repository-check-failures-in-a-geo-secondary-site). -### Check if PostgreSQL replication is working - -To check if PostgreSQL replication is working, check if: - -- [Sites are pointing to the correct database node](#are-sites-pointing-to-the-correct-database-node). -- [Geo can detect the current site correctly](#can-geo-detect-the-current-site-correctly). - -#### Are sites pointing to the correct database node? - -You should make sure your **primary** Geo [site](../glossary.md) points to -the database node that has write permissions. - -Any **secondary** sites should point only to read-only database nodes. - -#### Can Geo detect the current site correctly? - -Geo finds the current Puma or Sidekiq node's Geo [site](../glossary.md) name in -`/etc/gitlab/gitlab.rb` with the following logic: - -1. Get the "Geo node name" (there is - [an issue to rename the settings to "Geo site name"](https://gitlab.com/gitlab-org/gitlab/-/issues/335944)): - - Linux package: get the `gitlab_rails['geo_node_name']` setting. - - GitLab Helm charts: get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)). -1. If that is not defined, then get the `external_url` setting. - -This name is used to look up the Geo site with the same **Name** in the **Geo Sites** -dashboard. - -To check if the current machine has a site name that matches a site in the -database, run the check task: - -```shell -sudo gitlab-rake gitlab:geo:check -``` - -It displays the current machine's site name and whether the matching database -record is a **primary** or **secondary** site. - -```plaintext -This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" -``` - -```plaintext -This machine's Geo node name matches a database record ... no - Try fixing it: - You could add or update a Geo node database record, setting the name to "https://example.com/". - Or you could set this machine's Geo node name to match the name of an existing database record: "London", "Shanghai" - For more information see: - doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly -``` - -For more information about recommended site names in the description of the Name field, see -[Geo Admin Area Common Settings](../../../administration/geo_sites.md#common-settings). - -### Reverify all uploads (or any SSF data type which is verified) - -1. SSH into a GitLab Rails node in the primary Geo site. -1. Open [Rails console](../../operations/rails_console.md). -1. Mark all uploads as "pending verification": - -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. - - ```ruby - Upload.verification_state_table_class.each_batch do |relation| - relation.update_all(verification_state: 0) - end - ``` - -1. This causes the primary to start checksumming all Uploads. -1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values. - -A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../../development/geo/framework.md) which have implemented verification: - -- `LfsObject` -- `MergeRequestDiff` -- `Packages::PackageFile` -- `Terraform::StateVersion` -- `SnippetRepository` -- `Ci::PipelineArtifact` -- `PagesDeployment` -- `Upload` -- `Ci::JobArtifact` -- `Ci::SecureFile` - -NOTE: -`GroupWikiRepository` is not in the previous list since verification is not implemented. -There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). - -### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing - -If a replication slot is inactive, -the `pg_wal` logs corresponding to the slot are reserved forever -(or until the slot is active again). This causes continuous disk usage growth -and the following messages appear repeatedly in the -[PostgreSQL logs](../../logs/index.md#postgresql-logs): - -```plaintext -WARNING: oldest xmin is far in the past -HINT: Close open transactions soon to avoid wraparound problems. -You might also need to commit or roll back old prepared transactions, or drop stale replication slots. -``` - -To fix this: - -1. [Connect to the primary database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). - -1. Run `SELECT * FROM pg_replication_slots;`. - Note the `slot_name` that reports `active` as `f` (false). - -1. Follow [the steps to remove that Geo site](remove_geo_site.md). - -## Fixing errors found when running the Geo check Rake task +##### Fixing errors found when running the Geo check Rake task When running this Rake task, you may see error messages if the nodes are not properly configured: @@ -367,7 +287,7 @@ sudo gitlab-rake gitlab:geo:check Checking Geo ... Finished ``` - Ensure you have added the secondary site in the **Main menu > Admin > Geo > Sites** on the web interface for the **primary** site. + Ensure you have added the secondary site in the Admin Area under **Geo > Sites** on the web interface for the **primary** site. Also ensure you entered the `gitlab_rails['geo_node_name']` when adding the secondary site in the Admin Area of the **primary** site. In GitLab 12.3 and earlier, edit the secondary site in the Admin Area of the **primary** @@ -423,7 +343,7 @@ sudo gitlab-rake gitlab:geo:check - If you are running the secondary site's tracking database in an external database, then follow [Geo with external PostgreSQL instances](../setup/external_database.md#configure-the-tracking-database) - If the Geo check task was run on a node which is not running a service which runs the GitLab Rails app (Puma, Sidekiq, or Geo Log Cursor), then this error can be ignored. The node does not need Rails to be configured. -### Message: Machine clock is synchronized ... Exception +##### Message: Machine clock is synchronized ... Exception The Rake task attempts to verify that the server clock is synchronized with NTP. Synchronized clocks are required for Geo to function correctly. As an example, for security, when the server time on the @@ -457,7 +377,7 @@ generate an error because containers in Kubernetes do not have access to the hos Machine clock is synchronized ... Exception: getaddrinfo: Servname not supported for ai_socktype ``` -### Message: `ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction` +##### Message: `ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction` When this error is encountered on a secondary site, it likely affects all usages of GitLab Rails such as `gitlab-rails` or `gitlab-rake` commands, as well the Puma, Sidekiq, and Geo Log Cursor services. @@ -490,8 +410,122 @@ This situation can occur during initial configuration when a secondary site is n To resolve the error, follow [Step 3. Add the secondary site](configuration.md#step-3-add-the-secondary-site). +### Check if PostgreSQL replication is working + +To check if PostgreSQL replication is working, check if: + +- [Sites are pointing to the correct database node](#are-sites-pointing-to-the-correct-database-node). +- [Geo can detect the current site correctly](#can-geo-detect-the-current-site-correctly). + +#### Are sites pointing to the correct database node? + +You should make sure your **primary** Geo [site](../glossary.md) points to +the database node that has write permissions. + +Any **secondary** sites should point only to read-only database nodes. + +#### Can Geo detect the current site correctly? + +Geo finds the current Puma or Sidekiq node's Geo [site](../glossary.md) name in +`/etc/gitlab/gitlab.rb` with the following logic: + +1. Get the "Geo node name" (there is + [an issue to rename the settings to "Geo site name"](https://gitlab.com/gitlab-org/gitlab/-/issues/335944)): + - Linux package: get the `gitlab_rails['geo_node_name']` setting. + - GitLab Helm charts: get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)). +1. If that is not defined, then get the `external_url` setting. + +This name is used to look up the Geo site with the same **Name** in the **Geo Sites** +dashboard. + +To check if the current machine has a site name that matches a site in the +database, run the check task: + +```shell +sudo gitlab-rake gitlab:geo:check +``` + +It displays the current machine's site name and whether the matching database +record is a **primary** or **secondary** site. + +```plaintext +This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" +``` + +```plaintext +This machine's Geo node name matches a database record ... no + Try fixing it: + You could add or update a Geo node database record, setting the name to "https://example.com/". + Or you could set this machine's Geo node name to match the name of an existing database record: "London", "Shanghai" + For more information see: + doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly +``` + +For more information about recommended site names in the description of the Name field, see +[Geo Admin Area Common Settings](../../../administration/geo_sites.md#common-settings). + +### Check OS locale data compatibility + +If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. + +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, it can cause erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). + +For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. +See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). + +On all hosts running PostgreSQL, across all Geo sites, run the following shell command: + +```shell +( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort +``` + +The output looks like either: + +```plaintext +1-1 +11 +``` + +or the reverse order: + +```plaintext +11 +1-1 +``` + +If the output is identical on all hosts, then they running compatible versions of locale data. + +If the output differs on some hosts, PostgreSQL replication does not work properly: indexes are corrupted on the database replicas. You should select operating system versions that are compatible. + +A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. + +This check is also required when using a mixture of GitLab deployments. The locale might be different between an Linux package install, a GitLab Docker container, a Helm chart deployment, or external database services. + ## Fixing PostgreSQL database replication errors +### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing + +If a replication slot is inactive, +the `pg_wal` logs corresponding to the slot are reserved forever +(or until the slot is active again). This causes continuous disk usage growth +and the following messages appear repeatedly in the +[PostgreSQL logs](../../logs/index.md#postgresql-logs): + +```plaintext +WARNING: oldest xmin is far in the past +HINT: Close open transactions soon to avoid wraparound problems. +You might also need to commit or roll back old prepared transactions, or drop stale replication slots. +``` + +To fix this: + +1. [Connect to the primary database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). + +1. Run `SELECT * FROM pg_replication_slots;`. + Note the `slot_name` that reports `active` as `f` (false). + +1. Follow [the steps to remove that Geo site](remove_geo_site.md). + The following sections outline troubleshooting steps for fixing replication error messages (indicated by `Database replication working? ... no` in the [`geo:check` output](#health-check-rake-task). @@ -645,6 +679,43 @@ In GitLab 13.4, a seed project is added when GitLab is first installed. This mak on a new Geo secondary site. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) when checking the database. +## Synchronization errors + +### Reverify all uploads (or any SSF data type which is verified) + +1. SSH into a GitLab Rails node in the primary Geo site. +1. Open [Rails console](../../operations/rails_console.md). +1. Mark all uploads as "pending verification": + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + + ```ruby + Upload.verification_state_table_class.each_batch do |relation| + relation.update_all(verification_state: 0) + end + ``` + +1. This causes the primary to start checksumming all Uploads. +1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values. + +You can perform a similar operation with other the Models handled by the [Geo Self-Service Framework](../../../development/geo/framework.md) which have implemented verification: + +- `LfsObject` +- `MergeRequestDiff` +- `Packages::PackageFile` +- `Terraform::StateVersion` +- `SnippetRepository` +- `Ci::PipelineArtifact` +- `PagesDeployment` +- `Upload` +- `Ci::JobArtifact` +- `Ci::SecureFile` + +NOTE: +`GroupWikiRepository` is not in the previous list since verification is not implemented. +There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). + ### Message: `Synchronization failed - Error syncing repository` WARNING: @@ -686,7 +757,7 @@ To solve this: 1. Sign in on the web interface for the secondary Geo site. -1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). +1. Back up [the `.git` folder](../../repository_storage_paths.md#translate-hashed-storage-paths). 1. Optional. [Spot-check](../../logs/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond @@ -717,212 +788,11 @@ To solve this: end ``` -### Very large repositories never successfully synchronize on the **secondary** site - -#### GitLab 10.1 and earlier - -GitLab places a timeout on all repository clones, including project imports -and Geo synchronization operations. If a fresh `git clone` of a repository -on the **primary** takes more than the default three hours, you may be affected by this. - -To increase the timeout: - -1. On the **Sidekiq nodes on your secondary** site, -add the following line to `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['gitlab_shell_git_timeout'] = 14400 - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -This increases the timeout to four hours (14400 seconds). Choose a time -long enough to accommodate a full clone of your largest repositories. - -#### GitLab 10.2 and later - -Geo [replicates Git repositories over HTTPS](../index.md#how-it-works). GitLab does not place a timeout on these requests. If a Git repository is failing to replicate, with a consistent job execution time, then you should check for timeouts applied by external components such as load balancers. - -### New LFS objects are never replicated - -If new LFS objects are never replicated to secondary Geo sites, check the version of -GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by -[a bug that results in new LFS objects not being replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). - -To resolve the issue, upgrade to GitLab 12.1 or later. - ### Failures during backfill During a [backfill](../index.md#backfill), failures are scheduled to be retried at the end of the backfill queue, therefore these failures only clear up **after** the backfill completes. -### Resetting Geo **secondary** site replication - -If you get a **secondary** site in a broken state and want to reset the replication state, -to start again from scratch, there are a few steps that can help you: - -1. Stop Sidekiq and the Geo LogCursor. - - It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and - wait until the current jobs to finish processing. - - You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** - when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. - - ```shell - gitlab-ctl status sidekiq - # run: sidekiq: (pid 10180) <- this is the PID you will use - kill -TSTP 10180 # change to the correct PID - - gitlab-ctl stop sidekiq - gitlab-ctl stop geo-logcursor - ``` - - You can watch the [Sidekiq logs](../../logs/index.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: - - ```shell - gitlab-ctl tail sidekiq - ``` - -1. Rename repository storage folders and create new ones. If you are not concerned about possible orphaned directories and files, you can skip this step. - - ```shell - mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old - mkdir -p /var/opt/gitlab/git-data/repositories - chown git:git /var/opt/gitlab/git-data/repositories - ``` - - NOTE: - You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future - as soon as you confirmed that you don't need it anymore, to save disk space. - -1. Optional. Rename other data folders and create new ones. - - WARNING: - You may still have files on the **secondary** site that have been removed from the **primary** site, but this - removal has not been reflected. If you skip this step, these files are not removed from the Geo **secondary** site. - - Any uploaded content (like file attachments, avatars, or LFS objects) is stored in a - subfolder in one of these paths: - - - `/var/opt/gitlab/gitlab-rails/shared` - - `/var/opt/gitlab/gitlab-rails/uploads` - - To rename all of them: - - ```shell - gitlab-ctl stop - - mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old - mkdir -p /var/opt/gitlab/gitlab-rails/shared - - mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old - mkdir -p /var/opt/gitlab/gitlab-rails/uploads - - gitlab-ctl start postgresql - gitlab-ctl start geo-postgresql - ``` - - Reconfigure to recreate the folders and make sure permissions and ownership - are correct: - - ```shell - gitlab-ctl reconfigure - ``` - -1. Reset the Tracking Database. - - WARNING: - If you skipped the optional step 3, be sure both `geo-postgresql` and `postgresql` services are running. - - ```shell - gitlab-rake db:drop:geo DISABLE_DATABASE_ENVIRONMENT_CHECK=1 # on a secondary app node - gitlab-ctl reconfigure # on the tracking database node - gitlab-rake db:migrate:geo # on a secondary app node - ``` - -1. Restart previously stopped services. - - ```shell - gitlab-ctl start - ``` - -### Design repository failures on mirrored projects and project imports - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. - -If the Design repositories progress bar shows -`Synced` and `Failed` greater than 100%, and negative `Queued`, the instance -is likely affected by -[a bug in GitLab 13.2 and 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/241668). -It was [fixed in GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40643). - -To determine the actual replication status of design repositories in -a [Rails console](../../operations/rails_console.md): - -```ruby -secondary = Gitlab::Geo.current_node -counts = {} -secondary.designs.select("projects.id").find_each do |p| - registry = Geo::DesignRegistry.find_by(project_id: p.id) - state = registry ? "#{registry.state}" : "registry does not exist yet" - # puts "Design ID##{p.id}: #{state}" # uncomment this for granular information - counts[state] ||= 0 - counts[state] += 1 -end -puts "\nCounts:", counts -``` - -Example output: - -```plaintext -Design ID#5: started -Design ID#6: synced -Design ID#7: failed -Design ID#8: pending -Design ID#9: synced - -Counts: -{"started"=>1, "synced"=>2, "failed"=>1, "pending"=>1} -``` - -Example output if there are actually zero design repository replication failures: - -```plaintext -Design ID#5: synced -Design ID#6: synced -Design ID#7: synced - -Counts: -{"synced"=>3} -``` - -#### If you are promoting a Geo secondary site running on a single node - -`gitlab-ctl promotion-preflight-checks` fails due to the existence of -`failed` rows in the `geo_design_registry` table. Use the -[previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to -determine the actual replication status of Design repositories. - -`gitlab-ctl promote-to-primary-node` fails since it runs preflight checks. -If the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) -shows that all designs are synced, you can use the -`--skip-preflight-checks` option or the `--force` option to move forward with -promotion. - -#### If you are promoting a Geo secondary site running on multiple servers - -`gitlab-ctl promotion-preflight-checks` fails due to the existence of -`failed` rows in the `geo_design_registry` table. Use the -[previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to -determine the actual replication status of Design repositories. - ### Sync failure message: "Verification failed with: Error during verification: File is not checksummable" #### Missing files on the Geo primary site @@ -1020,272 +890,280 @@ We recommend transferring each failing repository individually and checking for after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) to transfer each affected repository from the primary to the secondary site. -## Fixing errors during a failover or when promoting a secondary to a primary site - -The following are possible error messages that might be encountered during failover or -when promoting a secondary to a primary site with strategies to resolve them. +### Project or project wiki repositories -### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` +#### Find repository verification failures -When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), -you might encounter the following error message: +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +**on the secondary Geo site** to gather more information. -```plaintext -Running gitlab-rake geo:set_secondary_as_primary... +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -rake aborted! -ActiveRecord::RecordInvalid: Validation failed: Name has already been taken -/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:236:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Tasks: TOP => geo:set_secondary_as_primary -(See full trace by running task with --trace) +##### Get the number of verification failed repositories -You successfully promoted this node! +```ruby +Geo::ProjectRegistry.verification_failed('repository').count ``` -If you encounter this message when running `gitlab-rake geo:set_secondary_as_primary` -or `gitlab-ctl promote-to-primary-node`, either: +##### Find the verification failed repositories -- Enter a Rails console and run: +```ruby +Geo::ProjectRegistry.verification_failed('repository') +``` - ```ruby - Rails.application.load_tasks; nil - Gitlab::Geo.expire_cache! - Rake::Task['geo:set_secondary_as_primary'].invoke - ``` +##### Find repositories that failed to sync -- Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, - if the failover was just a test. A - [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. +```ruby +Geo::ProjectRegistry.sync_failed('repository') +``` -### Message: `ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled` +#### Resync project and project wiki repositories -If you disabled a secondary site, either with the [replication pause task](../index.md#pausing-and-resuming-replication) -(GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first -re-enable the site before you can continue. This is fixed in GitLab 13.4. +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +**on the secondary Geo site** to perform the following changes. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -This can be fixed in the database. +##### Queue up all repositories for resync -1. Start a database console: +When you run this, the sync is handled in the background by Sidekiq. - In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/341210): +```ruby +Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) +``` - ```shell - sudo gitlab-rails dbconsole --database main - ``` +##### Sync individual repository now - In GitLab 14.1 and earlier: +```ruby +project = Project.find_by_full_path('<group/project>') - ```shell - sudo gitlab-rails dbconsole - ``` +Geo::RepositorySyncService.new(project).execute +``` -1. Run the following command, replacing `https://<secondary url>/` with the URL - for your secondary node. You can use either `http` or `https`, but ensure that you - end the URL with a slash (`/`): +##### Sync all failed repositories now - ```sql - UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" - ``` +The following script: - This should update one row. +- Loops over all currently failed repositories. +- Displays the project details and the reasons for the last failure. +- Attempts to resync the repository. +- Reports back if a failure occurs, and why. -### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` +```ruby +Geo::ProjectRegistry.sync_failed('repository').find_each do |p| + begin + project = p.project + puts "#{project.full_path} | id: #{p.project_id} | last error: '#{p.last_repository_sync_failure}'" + Geo::RepositorySyncService.new(project).execute + rescue => e + puts "ID: #{p.project_id} failed: '#{e}'", e.backtrace.join("\n") + end +end ; nil +``` -When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), -you might encounter the following error message: +### Find repository check failures in a Geo secondary site -```plaintext -sudo gitlab-rake geo:set_secondary_as_primary +When [enabled for all projects](../../repository_checks.md#enable-repository-checks-for-all-projects), [Repository checks](../../repository_checks.md) are also performed on Geo secondary sites. The metadata is stored in the Geo tracking database. -rake aborted! -NoMethodError: undefined method `secondary?' for nil:NilClass -/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:232:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Tasks: TOP => geo:set_secondary_as_primary -(See full trace by running task with --trace) -``` +Repository check failures on a Geo secondary site do not necessarily imply a replication problem. Here is a general approach to resolve these failures. -This command is intended to be executed on a secondary site only, and this error message -is displayed if you attempt to run this command on a primary site. +1. Find affected repositories as mentioned below, as well as their [logged errors](../../repository_checks.md#what-to-do-if-a-check-failed). +1. Try to diagnose specific `git fsck` errors. The range of possible errors is wide, try putting them into search engines. +1. Test typical functions of the affected repositories. Pull from the secondary, view the files. +1. Check if the primary site's copy of the repository has an identical `git fsck` error. If you are planning a failover, then consider prioritizing that the secondary site has the same information that the primary site has. Ensure you have a backup of the primary, and follow [planned failover guidelines](../disaster_recovery/planned_failover.md). +1. Push to the primary and check if the change gets replicated to the secondary site. +1. If replication is not automatically working, try to manually sync the repository. -### Message: `sudo: gitlab-pg-ctl: command not found` +[Start a Rails console session](../../operations/rails_console.md#starting-a-rails-console-session) +to enact the following, basic troubleshooting steps. -When -[promoting a **secondary** site with multiple nodes](../disaster_recovery/index.md#promoting-a-secondary-site-with-multiple-nodes-running-gitlab-144-and-earlier), -you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL -read-replica database. +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -In GitLab 12.8 and earlier, this command fails with the message: +#### Get the number of repositories that failed the repository check -```plaintext -sudo: gitlab-pg-ctl: command not found +```ruby +Geo::ProjectRegistry.where(last_repository_check_failed: true).count ``` -In this case, the workaround is to use the full path to the binary, for example: +#### Find the repositories that failed the repository check -```shell -sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote +```ruby +Geo::ProjectRegistry.where(last_repository_check_failed: true) ``` -GitLab 12.9 and later are [unaffected by this error message](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). +#### Recheck repositories that failed the repository check -### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promotion-preflight-checks` +When you run this, `fsck` is executed against each failed repository. -In GitLab 13.7 and earlier, if you have a data type with zero items to sync, -this command reports `ERROR - Replication is not up-to-date` even if -replication is actually up-to-date. This bug was fixed in GitLab 13.8 and -later. +The [`fsck` Rake command](../../raketasks/check.md#check-project-code-repositories) can be used on the secondary site to understand why the repository check might be failing. -### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promote-to-primary-node` +```ruby +Geo::ProjectRegistry.where(last_repository_check_failed: true).each do |pr| + RepositoryCheck::SingleRepositoryWorker.new.perform(pr.project_id) +end +``` -In GitLab 13.7 and earlier, if you have a data type with zero items to sync, -this command reports `ERROR - Replication is not up-to-date` even if -replication is actually up-to-date. If replication and verification output -shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. +## Fixing non-PostgreSQL replication failures -### Errors when using `--skip-preflight-checks` or `--force` +If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: -In GitLab 13.4 and earlier, you could receive one of the following error messages when using -`--skip-preflight-checks` or `--force`: +1. Geo automatically retries failures. If the failures are new and few in number, or if you suspect the root cause is already resolved, then you can wait to see if the failures go away. +1. If failures were present for a long time, then many retries have already occurred, and the interval between automatic retries has increased to up to 4 hours depending on the type of failure. If you suspect the root cause is already resolved, you can [manually retry replication or verification](#manually-retry-replication-or-verification). +1. If the failures persist, use the following sections to try to resolve them. -```plaintext -get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::InvalidOption) +### Manually retry replication or verification -get_ctl_options': invalid option: --force (OptionParser::InvalidOption) -``` +Project Git repositories and Project Wiki Git repositories have the ability in `Admin > Geo > Replication` to `Resync all`, `Reverify all`, or for a single resource, `Resync` or `Reverify`. -This can happen with XFS or file systems that list files in lexical order, because the -load order of the Linux package command files can be different than expected, and a global function would get redefined. -More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). +Adding this ability to other data types is proposed in issue [364725](https://gitlab.com/gitlab-org/gitlab/-/issues/364725). -The workaround is to manually run the preflight checks and promote the database, by running -the following commands on the Geo secondary site: +The following sections describe how to use internal application commands in the [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) to cause replication or verification immediately. -```shell -sudo gitlab-ctl promotion-preflight-checks -sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote -sudo gitlab-ctl reconfigure -sudo gitlab-rake geo:set_secondary_as_primary -``` +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -## Expired artifacts +### Blob types -If you notice for some reason there are more artifacts on the Geo -**secondary** site than on the Geo **primary** site, you can use the Rake task -to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). +- `Ci::JobArtifact` +- `Ci::PipelineArtifact` +- `Ci::SecureFile` +- `LfsObject` +- `MergeRequestDiff` +- `Packages::PackageFile` +- `PagesDeployment` +- `Terraform::StateVersion` +- `Upload` -On a Geo **secondary** site, this command also cleans up all Geo -registry record related to the orphan files on disk. +`Packages::PackageFile` is used in the following +[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +examples, but things generally work the same for the other types. -## Fixing sign in errors +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -### Message: The redirect URI included is not valid +### Repository types, except for project or project wiki repositories -If you are able to sign in to the web interface for the **primary** site, but you receive this error message -when attempting to sign in to a **secondary** web interface, you should verify the Geo -site's URL matches its external URL. +- `SnippetRepository` +- `GroupWikiRepository` -On the **primary** site: +`SnippetRepository` is used in the examples below, but things generally work the same for the other Repository types. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. -1. Find the affected **secondary** site and select **Edit**. -1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` - in `external_url "https://gitlab.example.com"` on the **Rails nodes of the secondary** site. +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +to enact the following, basic troubleshooting steps. -### Authenticating with SAML on the secondary site always lands on the primary site +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -This [problem is usually encountered when upgrading to GitLab 15.1](version_specific_upgrades.md#upgrading-to-151). To fix this problem, see [configuring instance-wide SAML in Geo with Single Sign-On](single_sign_on.md#configuring-instance-wide-saml). +#### The Replicator -## Fixing common errors +The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it): -This section documents common error messages reported in the Admin Area on the web interface, and how to fix them. +```ruby +model_record = Packages::PackageFile.last +model_record.replicator.registry.replicator.model_record # just showing that these methods exist +``` -### Geo database configuration file is missing +#### Replicate a package file, synchronously, given an ID -GitLab cannot find or doesn't have permission to access the `database_geo.yml` configuration file. +```ruby +model_record = Packages::PackageFile.find(id) +model_record.replicator.send(:download) +``` -In a Linux package installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. -If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. +#### Replicate a package file, synchronously, given a registry ID -If this path is mounted on a remote volume, ensure your volume configuration -has the correct permissions. +```ruby +registry = Geo::PackageFileRegistry.find(registry_id) +registry.replicator.send(:download) +``` -### An existing tracking database cannot be reused +#### Find registry records of blobs that failed to sync -Geo cannot reuse an existing tracking database. +```ruby +Geo::PackageFileRegistry.failed +``` -It is safest to use a fresh secondary, or reset the whole secondary by following -[Resetting Geo secondary site replication](#resetting-geo-secondary-site-replication). +#### Find registry records of blobs that are missing on the primary site -### Geo site has a database that is writable which is an indication it is not configured for replication with the primary site +```ruby +Geo::PackageFileRegistry.where(last_sync_failure: 'The file is missing on the Geo primary site') +``` -This error message refers to a problem with the database replica on a **secondary** site, -which Geo expects to have access to. It usually means, either: +#### Verify package files on the secondary manually -- An unsupported replication method was used (for example, logical replication). -- The instructions to set up a [Geo database replication](../setup/database.md) were not followed correctly. -- Your database connection details are incorrect, that is you have specified the wrong - user in your `/etc/gitlab/gitlab.rb` file. +This iterates over all package files on the secondary, looking at the +`verification_checksum` stored in the database (which came from the primary) +and then calculate this value on the secondary to check if they match. This +does not change anything in the UI. -Geo **secondary** sites require two separate PostgreSQL instances: +For GitLab 14.4 and later: -- A read-only replica of the **primary** site. -- A regular, writable instance that holds replication metadata. That is, the Geo tracking database. +```ruby +# Run on secondary +status = {} -This error message indicates that the replica database in the **secondary** site is misconfigured and replication has stopped. +Packages::PackageFile.find_each do |package_file| + primary_checksum = package_file.verification_checksum + secondary_checksum = Packages::PackageFile.sha256_hexdigest(package_file.file.path) + verification_status = (primary_checksum == secondary_checksum) -To restore the database and resume replication, you can do one of the following: + status[verification_status.to_s] ||= [] + status[verification_status.to_s] << package_file.id +end -- [Reset the Geo secondary site replication](#resetting-geo-secondary-site-replication). -- [Set up a new Geo secondary using the Linux package](../setup/index.md#using-linux-package-installations). +# Count how many of each value we get +status.keys.each {|key| puts "#{key} count: #{status[key].count}"} -If you set up a new secondary from scratch, you must also [remove the old site from the Geo cluster](remove_geo_site.md#removing-secondary-geo-sites). +# See the output in its entirety +status +``` -### Geo site does not appear to be replicating the database from the primary site +For GitLab 14.3 and earlier: -The most common problems that prevent the database from replicating correctly are: +```ruby +# Run on secondary +status = {} -- **Secondary** sites cannot reach the **primary** site. Check credentials and - [firewall rules](../index.md#firewall-rules). -- SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** site. -- Database storage disk is full. -- Database replication slot is misconfigured. -- Database is not using a replication slot or another alternative and cannot catch-up because WAL files were purged. +Packages::PackageFile.find_each do |package_file| + primary_checksum = package_file.verification_checksum + secondary_checksum = Packages::PackageFile.hexdigest(package_file.file.path) + verification_status = (primary_checksum == secondary_checksum) -Make sure you follow the [Geo database replication](../setup/database.md) instructions for supported configuration. + status[verification_status.to_s] ||= [] + status[verification_status.to_s] << package_file.id +end -### Geo database version (...) does not match latest migration (...) +# Count how many of each value we get +status.keys.each {|key| puts "#{key} count: #{status[key].count}"} -If you are using the Linux package installation, something might have failed during upgrade. You can: +# See the output in its entirety +status +``` -- Run `sudo gitlab-ctl reconfigure`. -- Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** site. +#### Reverify all uploads (or any SSF data type which is verified) -### GitLab indicates that more than 100% of repositories were synced +1. SSH into a GitLab Rails node in the primary Geo site. +1. Open [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Mark all uploads as "pending verification": -This can be caused by orphaned records in the project registry. You can clear them -[using the Rake task to remove orphaned project registries](../../../administration/raketasks/geo.md#remove-orphaned-project-registries). + ```ruby + Upload.verification_state_table_class.each_batch do |relation| + relation.update_all(verification_state: 0) + end + ``` -### Geo Admin Area returns 404 error for a secondary site +1. This causes the primary to start checksumming all Uploads. +1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values. -Sometimes `sudo gitlab-rake gitlab:geo:check` indicates that **Rails nodes of the secondary** sites are -healthy, but a 404 Not Found error message for the **secondary** site is returned in the Geo Admin Area on the web interface for -the **primary** site. +For other SSF data types replace `Upload` in the command above with the desired model class. -To resolve this issue: +NOTE: +There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). -- Try restarting **each Rails, Sidekiq and Gitaly nodes on your secondary site** using `sudo gitlab-ctl restart`. -- Check `/var/log/gitlab/gitlab-rails/geo.log` on Sidekiq nodes to see if the **secondary** site is - using IPv6 to send its status to the **primary** site. If it is, add an entry to - the **primary** site using IPv4 in the `/etc/hosts` file. Alternatively, you should - [enable IPv6 on the **primary** site](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +## HTTP response code errors ### Secondary site returns 502 errors with Geo proxying @@ -1311,14 +1189,6 @@ and changing the `8k` size, for example by doubling it to `16k`. If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the `/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. -### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode - -In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites stops getting updated. After 10 minutes, the status changes to `Unhealthy`. - -Geo secondary sites continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. - -This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). - ### Primary site returns 500 error when accessing `/admin/geo/replication/projects` Navigating to **Admin > Geo > Replication** (or `/admin/geo/replication/projects`) on a primary Geo site, shows a 500 error, while that same link on the secondary works fine. The primary's `production.log` has a similar entry to the following: @@ -1379,340 +1249,219 @@ The bug causes all wildcard domains (`.example.com`) to be ignored except for th You can have only one wildcard domain in the `no_proxy` list. -### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site - -If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for the primary site or changed the protocol from `http` to `https`, you may see that secondary sites are shown as `Unhealthy`. You may also find the following error in `geo.log`: - -```plaintext -"class": "Geo::NodeStatusRequestService", -... -"message": "Failed to Net::HTTP::Post to primary url: http://primary-site.gitlab.tld/api/v4/geo/status", - "error": "Failed to open TCP connection to <PRIMARY_IP_ADDRESS>:80 (Connection refused - connect(2) for \"<PRIMARY_ID_ADDRESS>\" port 80)" -``` - -In this case, make sure to update the changed URL on all your sites: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. -1. Change the URL and save the change. - -### Message: `ERROR: canceling statement due to conflict with recovery` during backup - -Running a backup on a Geo **secondary** [is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/211668). - -When running a backup on a **secondary** you might encounter the following error message: - -```plaintext -Dumping PostgreSQL database gitlabhq_production ... -pg_dump: error: Dumping the contents of table "notes" failed: PQgetResult() failed. -pg_dump: error: Error message from server: ERROR: canceling statement due to conflict with recovery -DETAIL: User query might have needed to see row versions that must be removed. -pg_dump: error: The command was: COPY public.notes (id, note, [...], last_edited_at) TO stdout; -``` - -To prevent a database backup being made automatically during GitLab upgrades on your Geo **secondaries**, -create the following empty file: - -```shell -sudo touch /etc/gitlab/skip-auto-backup -``` - -## Fixing non-PostgreSQL replication failures - -If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: - -1. Geo automatically retries failures. If the failures are new and few in number, or if you suspect the root cause is already resolved, then you can wait to see if the failures go away. -1. If failures were present for a long time, then many retries have already occurred, and the interval between automatic retries has increased to up to 4 hours depending on the type of failure. If you suspect the root cause is already resolved, you can [manually retry replication or verification](#manually-retry-replication-or-verification). -1. If the failures persist, use the following sections to try to resolve them. - -### Manually retry replication or verification +### Geo Admin Area returns 404 error for a secondary site -Project Git repositories and Project Wiki Git repositories have the ability in `Admin > Geo > Replication` to `Resync all`, `Reverify all`, or for a single resource, `Resync` or `Reverify`. +Sometimes `sudo gitlab-rake gitlab:geo:check` indicates that **Rails nodes of the secondary** sites are +healthy, but a 404 Not Found error message for the **secondary** site is returned in the Geo Admin Area on the web interface for +the **primary** site. -Adding this ability to other data types is proposed in issue [364725](https://gitlab.com/gitlab-org/gitlab/-/issues/364725). +To resolve this issue: -The following sections describe how to use internal application commands in the [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) to cause replication or verification immediately. +- Try restarting **each Rails, Sidekiq and Gitaly nodes on your secondary site** using `sudo gitlab-ctl restart`. +- Check `/var/log/gitlab/gitlab-rails/geo.log` on Sidekiq nodes to see if the **secondary** site is + using IPv6 to send its status to the **primary** site. If it is, add an entry to + the **primary** site using IPv4 in the `/etc/hosts` file. Alternatively, you should + [enable IPv6 on the **primary** site](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +## Fixing common errors -### Blob types +This section documents common error messages reported in the Admin Area on the web interface, and how to fix them. -- `Ci::JobArtifact` -- `Ci::PipelineArtifact` -- `Ci::SecureFile` -- `LfsObject` -- `MergeRequestDiff` -- `Packages::PackageFile` -- `PagesDeployment` -- `Terraform::StateVersion` -- `Upload` +### Geo database configuration file is missing -`Packages::PackageFile` is used in the following -[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -examples, but things generally work the same for the other types. +GitLab cannot find or doesn't have permission to access the `database_geo.yml` configuration file. -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +In a Linux package installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. +If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. -#### The Replicator +If this path is mounted on a remote volume, ensure your volume configuration +has the correct permissions. -The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it): +### An existing tracking database cannot be reused -```ruby -model_record = Packages::PackageFile.last -model_record.replicator.registry.replicator.model_record # just showing that these methods exist -``` +Geo cannot reuse an existing tracking database. -#### Replicate a package file, synchronously, given an ID +It is safest to use a fresh secondary, or reset the whole secondary by following +[Resetting Geo secondary site replication](#resetting-geo-secondary-site-replication). -```ruby -model_record = Packages::PackageFile.find(id) -model_record.replicator.send(:download) -``` +### Geo site has a database that is writable which is an indication it is not configured for replication with the primary site -#### Replicate a package file, synchronously, given a registry ID +This error message refers to a problem with the database replica on a **secondary** site, +which Geo expects to have access to. It usually means, either: -```ruby -registry = Geo::PackageFileRegistry.find(registry_id) -registry.replicator.send(:download) -``` +- An unsupported replication method was used (for example, logical replication). +- The instructions to set up a [Geo database replication](../setup/database.md) were not followed correctly. +- Your database connection details are incorrect, that is you have specified the wrong + user in your `/etc/gitlab/gitlab.rb` file. -#### Find registry records of blobs that failed to sync +Geo **secondary** sites require two separate PostgreSQL instances: -```ruby -Geo::PackageFileRegistry.failed -``` +- A read-only replica of the **primary** site. +- A regular, writable instance that holds replication metadata. That is, the Geo tracking database. -#### Find registry records of blobs that are missing on the primary site +This error message indicates that the replica database in the **secondary** site is misconfigured and replication has stopped. -```ruby -Geo::PackageFileRegistry.where(last_sync_failure: 'The file is missing on the Geo primary site') -``` +To restore the database and resume replication, you can do one of the following: -#### Verify package files on the secondary manually +- [Reset the Geo secondary site replication](#resetting-geo-secondary-site-replication). +- [Set up a new Geo secondary using the Linux package](../setup/index.md#using-linux-package-installations). -This iterates over all package files on the secondary, looking at the -`verification_checksum` stored in the database (which came from the primary) -and then calculate this value on the secondary to check if they match. This -does not change anything in the UI. +If you set up a new secondary from scratch, you must also [remove the old site from the Geo cluster](remove_geo_site.md#removing-secondary-geo-sites). -For GitLab 14.4 and later: +### Geo site does not appear to be replicating the database from the primary site -```ruby -# Run on secondary -status = {} +The most common problems that prevent the database from replicating correctly are: -Packages::PackageFile.find_each do |package_file| - primary_checksum = package_file.verification_checksum - secondary_checksum = Packages::PackageFile.sha256_hexdigest(package_file.file.path) - verification_status = (primary_checksum == secondary_checksum) +- **Secondary** sites cannot reach the **primary** site. Check credentials and + [firewall rules](../index.md#firewall-rules). +- SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** site. +- Database storage disk is full. +- Database replication slot is misconfigured. +- Database is not using a replication slot or another alternative and cannot catch-up because WAL files were purged. - status[verification_status.to_s] ||= [] - status[verification_status.to_s] << package_file.id -end +Make sure you follow the [Geo database replication](../setup/database.md) instructions for supported configuration. -# Count how many of each value we get -status.keys.each {|key| puts "#{key} count: #{status[key].count}"} +### Geo database version (...) does not match latest migration (...) -# See the output in its entirety -status -``` +If you are using the Linux package installation, something might have failed during upgrade. You can: -For GitLab 14.3 and earlier: +- Run `sudo gitlab-ctl reconfigure`. +- Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** site. -```ruby -# Run on secondary -status = {} +### GitLab indicates that more than 100% of repositories were synced -Packages::PackageFile.find_each do |package_file| - primary_checksum = package_file.verification_checksum - secondary_checksum = Packages::PackageFile.hexdigest(package_file.file.path) - verification_status = (primary_checksum == secondary_checksum) +This can be caused by orphaned records in the project registry. You can clear them +[using the Rake task to remove orphaned project registries](../../../administration/raketasks/geo.md#remove-orphaned-project-registries). - status[verification_status.to_s] ||= [] - status[verification_status.to_s] << package_file.id -end +### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site -# Count how many of each value we get -status.keys.each {|key| puts "#{key} count: #{status[key].count}"} +If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for the primary site or changed the protocol from `http` to `https`, you may see that secondary sites are shown as `Unhealthy`. You may also find the following error in `geo.log`: -# See the output in its entirety -status +```plaintext +"class": "Geo::NodeStatusRequestService", +... +"message": "Failed to Net::HTTP::Post to primary url: http://primary-site.gitlab.tld/api/v4/geo/status", + "error": "Failed to open TCP connection to <PRIMARY_IP_ADDRESS>:80 (Connection refused - connect(2) for \"<PRIMARY_ID_ADDRESS>\" port 80)" ``` -#### Reverify all uploads (or any SSF data type which is verified) - -1. SSH into a GitLab Rails node in the primary Geo site. -1. Open [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session). -1. Mark all uploads as "pending verification": - - ```ruby - Upload.verification_state_table_class.each_batch do |relation| - relation.update_all(verification_state: 0) - end - ``` - -1. This causes the primary to start checksumming all Uploads. -1. When a primary successfully checksums a record, then all secondaries recalculate the checksum as well, and they compare the values. - -For other SSF data types replace `Upload` in the command above with the desired model class. - -NOTE: -There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). - -### Repository types, except for project or project wiki repositories - -- `SnippetRepository` -- `GroupWikiRepository` - -`SnippetRepository` is used in the examples below, but things generally work the same for the other Repository types. - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to enact the following, basic troubleshooting steps. - -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +In this case, make sure to update the changed URL on all your sites: -#### The Replicator +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Geo > Sites**. +1. Change the URL and save the change. -The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it). +### Message: `ERROR: canceling statement due to conflict with recovery` during backup -```ruby -model_record = SnippetRepository.last -model_record.replicator.registry.replicator.model_record # just showing that these methods exist -``` +Running a backup on a Geo **secondary** [is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/211668). -#### Replicate a snippet repository, synchronously, given an ID +When running a backup on a **secondary** you might encounter the following error message: -```ruby -model_record = SnippetRepository.find(id) -model_record.replicator.send(:sync_repository) +```plaintext +Dumping PostgreSQL database gitlabhq_production ... +pg_dump: error: Dumping the contents of table "notes" failed: PQgetResult() failed. +pg_dump: error: Error message from server: ERROR: canceling statement due to conflict with recovery +DETAIL: User query might have needed to see row versions that must be removed. +pg_dump: error: The command was: COPY public.notes (id, note, [...], last_edited_at) TO stdout; ``` -#### Replicate a snippet repository, synchronously, given a registry ID +To prevent a database backup being made automatically during GitLab upgrades on your Geo **secondaries**, +create the following empty file: -```ruby -registry = Geo::SnippetRepositoryRegistry.find(registry_id) -registry.replicator.send(:sync_repository) +```shell +sudo touch /etc/gitlab/skip-auto-backup ``` -### Project or project wiki repositories - -#### Find repository verification failures - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -**on the secondary Geo site** to gather more information. - -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +## Fixing errors during a failover or when promoting a secondary to a primary site -##### Get the number of verification failed repositories +The following are possible error messages that might be encountered during failover or +when promoting a secondary to a primary site with strategies to resolve them. -```ruby -Geo::ProjectRegistry.verification_failed('repository').count -``` +### Message: `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` -##### Find the verification failed repositories +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), +you might encounter the following error message: -```ruby -Geo::ProjectRegistry.verification_failed('repository') -``` +```plaintext +Running gitlab-rake geo:set_secondary_as_primary... -##### Find repositories that failed to sync +rake aborted! +ActiveRecord::RecordInvalid: Validation failed: Name has already been taken +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:236:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => geo:set_secondary_as_primary +(See full trace by running task with --trace) -```ruby -Geo::ProjectRegistry.sync_failed('repository') +You successfully promoted this node! ``` -#### Resync project and project wiki repositories - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -**on the secondary Geo site** to perform the following changes. - -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. - -##### Queue up all repositories for resync - -When you run this, the sync is handled in the background by Sidekiq. - -```ruby -Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) -``` +If you encounter this message when running `gitlab-rake geo:set_secondary_as_primary` +or `gitlab-ctl promote-to-primary-node`, either: -##### Sync individual repository now +- Enter a Rails console and run: -```ruby -project = Project.find_by_full_path('<group/project>') + ```ruby + Rails.application.load_tasks; nil + Gitlab::Geo.expire_cache! + Rake::Task['geo:set_secondary_as_primary'].invoke + ``` -Geo::RepositorySyncService.new(project).execute -``` +- Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, + if the failover was just a test. A + [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. -##### Sync all failed repositories now +### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` -The following script: +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), +you might encounter the following error message: -- Loops over all currently failed repositories. -- Displays the project details and the reasons for the last failure. -- Attempts to resync the repository. -- Reports back if a failure occurs, and why. +```plaintext +sudo gitlab-rake geo:set_secondary_as_primary -```ruby -Geo::ProjectRegistry.sync_failed('repository').find_each do |p| - begin - project = p.project - puts "#{project.full_path} | id: #{p.project_id} | last error: '#{p.last_repository_sync_failure}'" - Geo::RepositorySyncService.new(project).execute - rescue => e - puts "ID: #{p.project_id} failed: '#{e}'", e.backtrace.join("\n") - end -end ; nil +rake aborted! +NoMethodError: undefined method `secondary?' for nil:NilClass +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:232:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => geo:set_secondary_as_primary +(See full trace by running task with --trace) ``` -#### Find repository check failures in a Geo secondary site - -When [enabled for all projects](../../repository_checks.md#enable-repository-checks-for-all-projects), [Repository checks](../../repository_checks.md) are also performed on Geo secondary sites. The metadata is stored in the Geo tracking database. - -Repository check failures on a Geo secondary site do not necessarily imply a replication problem. Here is a general approach to resolve these failures. - -1. Find affected repositories as mentioned below, as well as their [logged errors](../../repository_checks.md#what-to-do-if-a-check-failed). -1. Try to diagnose specific `git fsck` errors. The range of possible errors is wide, try putting them into search engines. -1. Test typical functions of the affected repositories. Pull from the secondary, view the files. -1. Check if the primary site's copy of the repository has an identical `git fsck` error. If you are planning a failover, then consider prioritizing that the secondary site has the same information that the primary site has. Ensure you have a backup of the primary, and follow [planned failover guidelines](../disaster_recovery/planned_failover.md). -1. Push to the primary and check if the change gets replicated to the secondary site. -1. If replication is not automatically working, try to manually sync the repository. +This command is intended to be executed on a secondary site only, and this error message +is displayed if you attempt to run this command on a primary site. -[Start a Rails console session](../../operations/rails_console.md#starting-a-rails-console-session) -to enact the following, basic troubleshooting steps. +### Expired artifacts -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +If you notice for some reason there are more artifacts on the Geo +**secondary** site than on the Geo **primary** site, you can use the Rake task +to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). -##### Get the number of repositories that failed the repository check +On a Geo **secondary** site, this command also cleans up all Geo +registry record related to the orphan files on disk. -```ruby -Geo::ProjectRegistry.where(last_repository_check_failed: true).count -``` +### Fixing sign in errors -##### Find the repositories that failed the repository check +#### Message: The redirect URI included is not valid -```ruby -Geo::ProjectRegistry.where(last_repository_check_failed: true) -``` +If you are able to sign in to the web interface for the **primary** site, but you receive this error message +when attempting to sign in to a **secondary** web interface, you should verify the Geo +site's URL matches its external URL. -##### Recheck repositories that failed the repository check +On the **primary** site: -When you run this, `fsck` is executed against each failed repository. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Geo > Sites**. +1. Find the affected **secondary** site and select **Edit**. +1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` + in `external_url "https://gitlab.example.com"` on the **Rails nodes of the secondary** site. -The [`fsck` Rake command](../../raketasks/check.md#check-project-code-repositories) can be used on the secondary site to understand why the repository check might be failing. +#### Authenticating with SAML on the secondary site always lands on the primary site -```ruby -Geo::ProjectRegistry.where(last_repository_check_failed: true).each do |pr| - RepositoryCheck::SingleRepositoryWorker.new.perform(pr.project_id) -end -``` +This [problem is usually encountered when upgrading to GitLab 15.1](../../../update/versions/gitlab_15_changes.md#1510). To fix this problem, see [configuring instance-wide SAML in Geo with Single Sign-On](single_sign_on.md#configuring-instance-wide-saml). ## Fixing client errors @@ -1784,43 +1533,6 @@ If the above steps are **not successful**, proceed through the next steps: 1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -## Check OS locale data compatibility - -If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. - -Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, it can cause erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). - -For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. -See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). - -On all hosts running PostgreSQL, across all Geo sites, run the following shell command: - -```shell -( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort -``` - -The output looks like either: - -```plaintext -1-1 -11 -``` - -or the reverse order: - -```plaintext -11 -1-1 -``` - -If the output is identical on all hosts, then they running compatible versions of locale data. - -If the output differs on some hosts, PostgreSQL replication does not work properly: indexes are corrupted on the database replicas. You should select operating system versions that are compatible. - -A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. - -This check is also required when using a mixture of GitLab deployments. The locale might be different between an Linux package install, a GitLab Docker container, a Helm chart deployment, or external database services. - ## Investigate causes of database replication lag If the output of `sudo gitlab-rake geo:status` shows that `Database replication lag` remains significantly high over time, the primary node in database replication can be checked to determine the status of lag for @@ -1847,3 +1559,94 @@ If one or more of these values is significantly high, this could indicate a prob - The difference in time between `write_lag` and `flush_lag` indicates that WAL bytes have been sent to the underlying storage system, but it has not reported that they were flushed. This data is most likely not fully written to a persistent storage, and likely held in some kind of volatile write cache. - The difference between `flush_lag` and `replay_lag` indicates WAL bytes that have been successfully persisted to storage, but could not be replayed by the database system. + +## Resetting Geo **secondary** site replication + +If you get a **secondary** site in a broken state and want to reset the replication state, +to start again from scratch, there are a few steps that can help you: + +1. Stop Sidekiq and the Geo LogCursor. + + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. + + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + + ```shell + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID + + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` + + You can watch the [Sidekiq logs](../../logs/index.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: + + ```shell + gitlab-ctl tail sidekiq + ``` + +1. Rename repository storage folders and create new ones. If you are not concerned about possible orphaned directories and files, you can skip this step. + + ```shell + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` + + NOTE: + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. + +1. Optional. Rename other data folders and create new ones. + + WARNING: + You may still have files on the **secondary** site that have been removed from the **primary** site, but this + removal has not been reflected. If you skip this step, these files are not removed from the Geo **secondary** site. + + Any uploaded content (like file attachments, avatars, or LFS objects) is stored in a + subfolder in one of these paths: + + - `/var/opt/gitlab/gitlab-rails/shared` + - `/var/opt/gitlab/gitlab-rails/uploads` + + To rename all of them: + + ```shell + gitlab-ctl stop + + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared + + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + + gitlab-ctl start postgresql + gitlab-ctl start geo-postgresql + ``` + + Reconfigure to recreate the folders and make sure permissions and ownership + are correct: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Reset the Tracking Database. + + WARNING: + If you skipped the optional step 3, be sure both `geo-postgresql` and `postgresql` services are running. + + ```shell + gitlab-rake db:drop:geo DISABLE_DATABASE_ENVIRONMENT_CHECK=1 # on a secondary app node + gitlab-ctl reconfigure # on the tracking database node + gitlab-rake db:migrate:geo # on a secondary app node + ``` + +1. Restart previously stopped services. + + ```shell + gitlab-ctl start + ``` diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 9e1638643cb..3a4366cfd63 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -14,7 +14,7 @@ in the background. On the **primary** site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Edit** of the secondary site you want to tune. diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md index 7e3160822b9..13d00ced3cd 100644 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ b/doc/administration/geo/replication/version_specific_upgrades.md @@ -1,106 +1,11 @@ --- -stage: Systems -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../../update/index.md#version-specific-upgrading-instructions' +remove_date: '2023-12-01' --- -# Version-specific upgrade instructions **(PREMIUM SELF)** +This document was moved to [another location](../../../update/index.md#version-specific-upgrading-instructions). -NOTE: -We're in the process of merging all the version-specific upgrade information -into a single page. For more information, -see [epic 9581](https://gitlab.com/groups/gitlab-org/-/epics/9581). -For the latest Geo version-specific upgrade instructions, -see the [general upgrade page](../../../update/index.md). - -Review this page for upgrade instructions for your version. These steps -accompany the [general steps](upgrading_the_geo_sites.md#general-upgrade-steps) -for upgrading Geo sites. - -## Upgrading to 15.1 - -[Geo proxying](../secondary_proxy/index.md) was [enabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in 15.1. This may be a breaking change. If needed, you may [disable Geo proxying](../secondary_proxy/index.md#disable-geo-proxying). - -If you are using SAML with different URLs, you must modify your SAML configuration and your Identity Provider configuration. For more information, see the [Geo with Single Sign-On (SSO) documentation](single_sign_on.md). - -## Upgrading to 14.9 - -**Do not** upgrade to GitLab 14.9.0. Instead, use 14.9.1 or later. - -We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue was fixed in [the GitLab 14.9.1 patch release](https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/). - -If you have already upgraded to GitLab 14.9.0, you can disable the feature causing the issue by [disabling the `geo_job_artifact_replication` feature flag](../../feature_flags.md#how-to-enable-and-disable-features-behind-flags). - -## Upgrading to 14.2 through 14.7 - -There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) -that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. - -Since GitLab 14.2, verification failures result in synchronization failures and cause -a resynchronization of these objects. - -As verification is not yet implemented for files stored in object storage (see -[issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this -results in a loop that consistently fails for all objects stored in object storage. - -For information on how to fix this, see -[Troubleshooting - Failed syncs with GitLab-managed object storage replication](troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). - -## Upgrading to 14.4 - -There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#1440) that can affect Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later. - -## Upgrading to 14.1, 14.2, 14.3 - -### Multi-arch images - -We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. - -You can check if you are affected by running: - -```shell -docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' -``` - -Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site. -If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` -(there can be a `mediaType` entry at several levels, we only care about the top level entry), -then you don't need to do anything. - -Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../operations/rails_console.md), and run the following: - - ```ruby - list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' - - Geo::ContainerRepositoryRegistry.synced.each do |gcr| - cr = gcr.container_repository - primary = Geo::ContainerRepositorySync.new(cr) - cr.tags.each do |tag| - primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name)) - next unless primary_manifest['mediaType'].eql?(list_type) - - cr.delete_tag_by_name(tag.name) - end - primary.execute - end - ``` - -If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](upgrading_the_geo_sites.md) to at least GitLab 14.1. - -### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode - -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). - -## Upgrading to GitLab 14.0/14.1 - -### Primary sites cannot be removed from the UI - -We found an issue where [Primary sites cannot be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). - -This bug only exists in the UI and does not block the removal of Primary sites using any other method. - -If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Sites API](../../../api/geo_nodes.md#delete-a-geo-node). - -### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode - -GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) causes Geo secondary site statuses to appear to stop upgrading and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 11e5cb1b7b8..f6572ded4a9 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -71,7 +71,7 @@ a single URL used by all Geo sites, including the primary. is using the secondary proxying and set the `URL` field to the single URL. Make sure the primary site is also using this URL. -In Kubernetes, you can use the same domain under `global.hosts.domain` as for the primary site. +In Kubernetes, you can [use the same domain under `global.hosts.domain` as for the primary site](https://docs.gitlab.com/charts/advanced/geo/index.html). ## Geo proxying with Separate URLs diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 061ae2d4eb8..6083d7331b6 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -7,8 +7,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Geo with external PostgreSQL instances **(PREMIUM SELF)** This document is relevant if you are using a PostgreSQL instance that is not -managed by the Linux package. This includes cloud-managed instances like Amazon RDS, or -manually installed and configured PostgreSQL instances. +managed by the Linux package. This includes +[cloud-managed instances](../../reference_architectures/index.md#recommendation-notes-for-the-database-services), +or manually installed and configured PostgreSQL instances. Ensure that you are using one of the PostgreSQL versions that the [Linux package ships with](../../package_information/postgresql_versions.md) diff --git a/doc/administration/geo/setup/two_single_node_sites.md b/doc/administration/geo/setup/two_single_node_sites.md index 00002d501b2..da7ec455c6a 100644 --- a/doc/administration/geo/setup/two_single_node_sites.md +++ b/doc/administration/geo/setup/two_single_node_sites.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Set up Geo for two single-node sites **(PREMIUM SELF)** -The following guide provides concise instructions on how to deploy GitLab Geo for a two single-node site installation using two Linux package instances. +The following guide provides concise instructions on how to deploy GitLab Geo for a two single-node site installation using two Linux package instances with no external services set up. Prerequisites: @@ -552,8 +552,9 @@ You must manually replicate the secret file across all of your secondary sites, ``` 1. Navigate to the primary node GitLab instance: - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Geo > Sites**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **Admin Area**. + 1. Select **Geo > Sites**. 1. Select **Add site**.  @@ -607,8 +608,9 @@ If you convert an existing site to Geo, you should check that the clone method i On the primary site: -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > General**. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. 1. Expand **Visibility and access controls**. 1. If you use Git over SSH: 1. Ensure **Enabled Git access protocols** is set to **Both SSH and HTTP(S)**. @@ -622,8 +624,9 @@ the primary site. After you sign in: -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Geo > Sites**. 1. Verify that the site is correctly identified as a secondary Geo site, and that Geo is enabled. diff --git a/doc/administration/geo_sites.md b/doc/administration/geo_sites.md index 81cc3a87941..f0c3e24f643 100644 --- a/doc/administration/geo_sites.md +++ b/doc/administration/geo_sites.md @@ -11,7 +11,7 @@ You can configure various settings for GitLab Geo sites. For more information, s On either the primary or secondary site: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Geo > Sites**. @@ -72,7 +72,7 @@ the primary uses the secondary's internal URL to contact it directly. The internal URL defaults to external URL. To change it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Geo > Sites**. 1. Select **Edit** on the site you want to customize. diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index b26ae84eef0..9f3afae0e9f 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -276,11 +276,11 @@ Configure Gitaly server in one of two ways: storage: [ { name: 'default', - path: '/var/opt/gitlab/git-data', + path: '/var/opt/gitlab/git-data/repositories', }, { name: 'storage1', - path: '/mnt/gitlab/git-data', + path: '/mnt/gitlab/git-data/repositories', }, ], } @@ -294,7 +294,7 @@ Configure Gitaly server in one of two ways: storage: [ { name: 'storage2', - path: '/srv/gitlab/git-data', + path: '/srv/gitlab/git-data/repositories', }, ], } @@ -517,7 +517,7 @@ gitaly['configuration'] = { storage: [ { name: 'storage1', - path: '/mnt/gitlab/git-data', + path: '/mnt/gitlab/git-data/repositories', }, ], } @@ -786,13 +786,13 @@ gitaly['configuration'] = { { rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel', max_per_repo: 20, - max_queue_time: '1s', + max_queue_wait: '1s', max_queue_size: 10, }, { rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel', max_per_repo: 20, - max_queue_time: '1s', + max_queue_wait: '1s', max_queue_size: 10, }, ], @@ -801,7 +801,7 @@ gitaly['configuration'] = { - `rpc` is the name of the RPC to set a concurrency limit for per repository. - `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository. -- `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to +- `max_queue_wait` is the maximum amount of time a request can wait in the concurrency queue to be picked up by Gitaly. - `max_queue_size` is the maximum size the concurrency queue (per RPC method) can grow to before requests are rejected by Gitaly. @@ -822,6 +822,50 @@ When these limits are reached, users are disconnected. You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more information, see the [relevant documentation](monitoring.md#monitor-gitaly-concurrency-limiting). +## Limit pack-objects concurrency + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7891) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `gitaly_pack_objects_limiting_remote_ip`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5772) in GitLab 16.0. Feature flag `gitaly_pack_objects_limiting_remote_ip` removed. + +Gitaly triggers `git-pack-objects` processes when handling both SSH and HTTPS traffic to clone or pull repositories. These processes generate a `pack-file` and can +consume a significant amount of resources, especially in situations such as unexpectedly high traffic or concurrent pulls from a large repository. On GitLab.com, we also +observe problems with clients that have slow internet connections. + +You can limit these processes from overwhelming your Gitaly server by setting pack-objects concurrency limits in the Gitaly configuration file. This setting limits the +number of in-flight pack-object processes per remote IP address. + +WARNING: +Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits +disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and +[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. + +Example configuration: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['pack_objects_limiting'] = { + 'max_concurrency' => 15, + 'max_queue_length' => 200, + 'max_queue_wait' => '60s', +} +``` + +- `max_concurrency` is the maximum number of in-flight pack-object processes per key. +- `max_queue_length` is the maximum size the concurrency queue (per key) can grow to before requests are rejected by Gitaly. +- `max_queue_wait` is the maximum amount of time a request can wait in the concurrency queue to be picked up by Gitaly. + +In the example above: + +- Each remote IP can have at most 15 simultaneous pack-object processes in flight on a Gitaly node. +- If another request comes in from an IP that has used up its 15 slots, that request gets queued. +- If a request waits in the queue for more than 1 minute, it is rejected with an error. +- If the queue grows beyond 200, subsequent requests are rejected with an error. + +When the pack-object cache is enabled, pack-objects limiting kicks in only if the cache is missed. For more, see [Pack-objects cache](#pack-objects-cache). + +You can observe the behavior of this queue using Gitaly logs and Prometheus. For more information, see +[Monitor Gitaly pack-objects concurrency limiting](monitoring.md#monitor-gitaly-pack-objects-concurrency-limiting). + ## Control groups WARNING: @@ -831,10 +875,6 @@ When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. -FLAG: -On self-managed GitLab, by default repository cgroups are not available. To make it available, an administrator can -[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`. - When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as processes may switch to using that instead of being terminated. This situation could lead to notably compromised performance. @@ -1288,47 +1328,34 @@ the cache hit rate. ### Observe the cache -The cache can be observed [using metrics](monitoring.md#pack-objects-cache) and in the following logged -information: +> Logs for pack-objects caching was [changed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5719) in GitLab 16.0. -|Message|Fields|Description| -|:---|:---|:---| -|`generated bytes`|`bytes`, `cache_key`|Logged when an entry was added to the cache| -|`served bytes`|`bytes`, `cache_key`|Logged when an entry was read from the cache| +You can observe the cache [using metrics](monitoring.md#pack-objects-cache) and in the following logged information. These logs are part of the gRPC logs and can +be discovered when a call is executed. + +| Field | Description | +|:---|:---| +| `pack_objects_cache.hit` | Indicates whether the current pack-objects cache was hit (`true` or `false`) | +| `pack_objects_cache.key` | Cache key used for the pack-objects cache | +| `pack_objects_cache.generated_bytes` | Size (in bytes) of the new cache being written | +| `pack_objects_cache.served_bytes` | Size (in bytes) of the cache being served | +| `pack_objects.compression_statistics` | Statistics regarding pack-objects generation | +| `pack_objects.enumerate_objects_ms` | Total time (in ms) spent enumerating objects sent by clients | +| `pack_objects.prepare_pack_ms` | Total time (in ms) spent preparing the packfile before sending it back to the client | +| `pack_objects.write_pack_file_ms` | Total time (in ms) spent sending back the packfile to the client. Highly dependent on the client's internet connection | +| `pack_objects.written_object_count` | Total number of objects Gitaly sends back to the client | In the case of a: -- Cache miss, Gitaly logs both a `generated bytes` and a `served bytes` message. -- Cache hit, Gitaly logs only a `served bytes` message. +- Cache miss, Gitaly logs both a `pack_objects_cache.generated_bytes` and `pack_objects_cache.served_bytes` message. Gitaly also logs some more detailed statistics of + pack-object generation. +- Cache hit, Gitaly logs only a `pack_objects_cache.served_bytes` message. Example: ```json { "bytes":26186490, - "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", - "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", - "grpc.meta.deadline_type":"none", - "grpc.method":"PackObjectsHook", - "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook", - "grpc.request.glProjectPath":"root/gitlab-workhorse", - "grpc.request.glRepository":"project-2", - "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git", - "grpc.request.repoStorage":"default", - "grpc.request.topLevelGroup":"@hashed", - "grpc.service":"gitaly.HookService", - "grpc.start_time":"2021-03-25T14:57:52.747Z", - "level":"info", - "msg":"generated bytes", - "peer.address":"@", - "pid":20961, - "span.kind":"server", - "system":"grpc", - "time":"2021-03-25T14:57:53.543Z" -} -{ - "bytes":26186490, - "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", "grpc.meta.deadline_type":"none", "grpc.method":"PackObjectsHook", @@ -1341,12 +1368,23 @@ Example: "grpc.service":"gitaly.HookService", "grpc.start_time":"2021-03-25T14:57:52.747Z", "level":"info", - "msg":"served bytes", + "msg":"finished unary call with code OK", "peer.address":"@", "pid":20961, "span.kind":"server", "system":"grpc", - "time":"2021-03-25T14:57:53.543Z" + "time":"2021-03-25T14:57:53.543Z", + "pack_objects.compression_statistics": "Total 145991 (delta 68), reused 6 (delta 2), pack-reused 145911", + "pack_objects.enumerate_objects_ms": 170, + "pack_objects.prepare_pack_ms": 7, + "pack_objects.write_pack_file_ms": 786, + "pack_objects.written_object_count": 145991, + "pack_objects_cache.generated_bytes": 49533030, + "pack_objects_cache.hit": "false", + "pack_objects_cache.key": "123456789", + "pack_objects_cache.served_bytes": 49533030, + "peer.address": "127.0.0.1", + "pid": 8813, } ``` @@ -1496,7 +1534,13 @@ value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" ## Configure commit signing for GitLab UI commits -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. +> - Displaying **Verified** badge for signed GitLab UI commits [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124218) in GitLab 16.3 [with a flag](../feature_flags.md) named `gitaly_gpg_signing`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flag](../feature_flags.md) named `gitaly_gpg_signing`. +On GitLab.com, this feature is not available. By default, Gitaly doesn't sign commits made using GitLab UI. For example, commits made using: @@ -1513,13 +1557,21 @@ Configure Gitaly to sign commits made with the GitLab UI in one of two ways: :::TabTitle Linux package (Omnibus) -1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key) - and export it. For optimal performance, consider using an EdDSA key. +1. [Create a GPG key](../../user/project/repository/signed_commits/gpg.md#create-a-gpg-key) + and export it, or [create an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). For optimal performance, use an EdDSA key. + + Export GPG key: ```shell gpg --export-secret-keys <ID> > signing_key.gpg ``` + Or create an SSH key (with no passphrase): + + ```shell + ssh-keygen -t ed25519 -f signing_key.ssh + ``` + 1. On the Gitaly nodes, copy the key into `/etc/gitlab/gitaly/`. 1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`: @@ -1537,13 +1589,21 @@ Configure Gitaly to sign commits made with the GitLab UI in one of two ways: :::TabTitle Self-compiled (source) -1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key) - and export it. For optimal performance, consider using an EdDSA key. +1. [Create a GPG key](../../user/project/repository/signed_commits/gpg.md#create-a-gpg-key) + and export it, or [create an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). For optimal performance, use an EdDSA key. + + Export GPG key: ```shell gpg --export-secret-keys <ID> > signing_key.gpg ``` + Or create an SSH key (with no passphrase): + + ```shell + ssh-keygen -t ed25519 -f signing_key.ssh + ``` + 1. On the Gitaly nodes, copy the key into `/etc/gitlab`. 1. Edit `/home/git/gitaly/config.toml` and configure `signing_key`: @@ -1610,3 +1670,172 @@ Gitaly fails to start up if either: - The configuration command fails. - The output produced by the command cannot be parsed as valid JSON. + +## Configure server-side backups + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. + +Repository backups can be configured so that the Gitaly node that hosts each +repository is responsible for creating the backup and streaming it to +object storage. This helps reduce the network resources required to create and +restore a backup. + +Each Gitaly node must be configured to connect to object storage for backups. + +After configuring server-side backups, you can +[create a server-side repository backup](../backup_restore/backup_gitlab.md#create-server-side-repository-backups). + +### Configure Azure Blob storage + +How you configure Azure Blob storage for backups depends on the type of installation you have. For self-compiled installations, you must set +the `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_KEY` environment variables outside of GitLab. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`: + +```ruby +gitaly['env'] = { + 'AZURE_STORAGE_ACCOUNT' => 'azure_storage_account', + 'AZURE_STORAGE_KEY' => 'azure_storage_key' # or 'AZURE_STORAGE_SAS_TOKEN' +} +gitaly['configuration'] = { + backup: { + go_cloud_url: 'azblob://gitaly-backups' + } +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`: + +```toml +[backup] +go_cloud_url = "azblob://gitaly-backups" +``` + +::EndTabs + +### Configure Google Cloud storage + +Google Cloud storage (GCP) authenticates using Application Default Credentials. Set up Application Default Credentials on each Gitaly server using either: + +- The [`gcloud auth application-default login`](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command. +- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For self-compiled installations, set the environment + variable outside of GitLab. + +For more information, see [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc). + +The destination bucket is configured using the `go_cloud_url` option. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`: + +```ruby +gitaly['env'] = { + 'GOOGLE_APPLICATION_CREDENTIALS' => '/path/to/service.json' +} +gitaly['configuration'] = { + backup: { + go_cloud_url: 'gs://gitaly-backups' + } +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`: + +```toml +[backup] +go_cloud_url = "gs://gitaly-backups" +``` + +::EndTabs + +### Configure S3 storage + +To configure S3 storage authentication: + +- If you authenticate with the AWS CLI, you can use the default AWS session. +- Otherwise, you can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. For self-compiled installations, set the environment + variables outside of GitLab. + +For more information, see [AWS Session documentation](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/). + +The destination bucket and region are configured using the `go_cloud_url` option. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`: + +```ruby +gitaly['env'] = { + 'AWS_ACCESS_KEY_ID' => 'aws_access_key_id', + 'AWS_SECRET_ACCESS_KEY' => 'aws_secret_access_key' +} +gitaly['configuration'] = { + backup: { + go_cloud_url: 's3://gitaly-backups?region=us-west-1' + } +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`: + +```toml +[backup] +go_cloud_url = "s3://gitaly-backups?region=us-west-1" +``` + +::EndTabs + +#### Configure S3-compatible servers + +S3-compatible servers such as MinIO are configured similarly to S3 with the addition of the `endpoint` parameter. + +The following parameters are supported: + +- `region`: The AWS region. +- `endpoint`: The endpoint URL. +- `disabledSSL`: A value of `true` disables SSL. +- `s3ForcePathStyle`: A value of `true` forces path-style addressing. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`: + +```ruby +gitaly['env'] = { + 'AWS_ACCESS_KEY_ID' => 'minio_access_key_id', + 'AWS_SECRET_ACCESS_KEY' => 'minio_secret_access_key' +} +gitaly['configuration'] = { + backup: { + go_cloud_url: 's3://gitaly-backups?region=minio&endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true' + } +} +``` + +:::TabTitle Self-compiled (source) + +Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`: + +```toml +[backup] +go_cloud_url = "s3://gitaly-backups?region=minio&endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true" +``` + +::EndTabs diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index db11ac8c769..46f6a5829c8 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -25,6 +25,7 @@ Gitaly implements a client-server architecture: - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) - [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) + - [GitLab Zoekt Indexer](https://gitlab.com/gitlab-org/gitlab-zoekt-indexer) Gitaly manages only Git repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. @@ -57,7 +58,7 @@ If you have not yet migrated to Gitaly Cluster, you have two options: - A sharded Gitaly instance. - Gitaly Cluster. -Contact your [Customer Success Manager](https://about.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions. +Contact your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions. ### Known issues @@ -326,7 +327,7 @@ conflicts that could occur due to partially applied operations. Repositories are stored in the storages at the relative path determined by the [Gitaly client](#gitaly-architecture). These paths can be identified by them not beginning with the `@cluster` prefix. The relative paths -follow the [hashed storage](../repository_storage_types.md#hashed-storage) schema. +follow the [hashed storage](../repository_storage_paths.md#hashed-storage) schema. #### Praefect-generated replica paths (GitLab 15.0 and later) @@ -377,7 +378,7 @@ Use the [`praefect metadata`](troubleshooting.md#view-repository-metadata) subco The repository on disk also contains the project path in the Git configuration file. The configuration file can be used to determine the project path even if the repository's metadata has been deleted. -Follow the [instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). +Follow the [instructions in hashed storage's documentation](../repository_storage_paths.md#from-hashed-path-to-project-name). #### Atomicity of operations diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 00d0499faa2..cbf5722f2c5 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -31,18 +31,64 @@ of requests dropped due to request limiting. The `reason` label indicates why a ## Monitor Gitaly concurrency limiting -You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using -the Gitaly logs and Prometheus: - -- In the [Gitaly logs](../logs/index.md#gitaly-logs), look for the string (or structured log field) - `acquire_ms`. Messages that have this field are reporting about the concurrency limiter. -- In Prometheus, look for the following metrics: - - `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are - being processed. - - `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given - repository are waiting due to the concurrency limit being reached. - - `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to - wait due to concurrency limits before being processed. +You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using Gitaly logs and Prometheus. + +In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as: + +| Log Field | Description | +| --- | --- | +| `limit.concurrency_queue_length` | Indicates the current length of the queue specific to the RPC type of the ongoing call. It provides insight into the number of requests waiting to be processed due to concurrency limits. | +| `limit.concurrency_queue_ms` | Represents the duration, in milliseconds, that a request has spent waiting in the queue due to the limit on concurrent RPCs. This field helps understand the impact of concurrency limits on request processing times. | +| `limit.concurrency_dropped` | If the request is dropped due to limits being reached, this field specifies the reason: either `max_time` (request waited in the queue longer than the maximum allowed time) or `max_size` (the queue reached its maximum size). | +| `limit.limiting_key` | Identifies the key used for limiting. | +| `limit.limiting_type` | Specifies the type of process being limited. In this context, it's `per-rpc`, indicating that the concurrency limiting is applied on a per-RPC basis. | + +For example: + +```json +{ + "limit .concurrency_queue_length": 1, + "limit .concurrency_queue_ms": 0, + "limit.limiting_key": "@hashed/79/02/7902699be42c8a8e46fbbb450172651786b22c56a189f7625a6da49081b2451.git", + "limit.limiting_type": "per-rpc" +} +``` + +In Prometheus, look for the following metrics: + +- `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are being processed. +- `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given repository are waiting due to the concurrency limit being reached. +- `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to wait due to concurrency limits before being processed. + +## Monitor Gitaly pack-objects concurrency limiting + +You can observe specific behavior of [pack-objects limiting](configure_gitaly.md#limit-pack-objects-concurrency) using Gitaly logs and Prometheus. + +In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as: + +| Log Field | Description | +|:---|:---| +| `limit.concurrency_queue_length` | Current length of the queue for the pack-objects processes. Indicates the number of requests that are waiting to be processed because the limit on concurrent processes has been reached. | +| `limit.concurrency_queue_ms` | Time a request has spent waiting in the queue, in milliseconds. Indicates how long a request has had to wait because of the limits on concurrency. | +| `limit.limiting_key` | Remote IP of the sender. | +| `limit.limiting_type` | Type of process being limited. In this case, `pack-objects`. | + +Example configuration: + +```json +{ + "limit .concurrency_queue_length": 1, + "limit .concurrency_queue_ms": 0, + "limit.limiting_key": "1.2.3.4", + "limit.limiting_type": "pack-objects" +} +``` + +In Prometheus, look for the following metrics: + +- `gitaly_pack_objects_in_progress` indicates how many pack-objects processes are being processed concurrently. +- `gitaly_pack_objects_queued` indicates how many requests for pack-objects processes are waiting due to the concurrency limit being reached. +- `gitaly_pack_objects_acquiring_seconds` indicates how long a request for a pack-object process has to wait due to concurrency limits before being processed. ## Monitor Gitaly cgroups diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 6eefd0a7bb5..0297f295e6f 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1133,15 +1133,15 @@ For more information on Gitaly server configuration, see our storage: [ { name: 'gitaly-1', - path: '/var/opt/gitlab/git-data', + path: '/var/opt/gitlab/git-data/repositories', }, { name: 'gitaly-2', - path: '/var/opt/gitlab/git-data', + path: '/var/opt/gitlab/git-data/repositories', }, { name: 'gitaly-3', - path: '/var/opt/gitlab/git-data', + path: '/var/opt/gitlab/git-data/repositories', }, ], } @@ -1337,7 +1337,7 @@ Particular attention should be shown to: 1. Check that the Praefect storage is configured to store new repositories: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand the **Repository storage** section. @@ -1498,9 +1498,12 @@ For a replication factor: > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) in GitLab 15.0. Praefect stores metadata about the repositories in a database. If the repositories are modified on disk -without going through Praefect, the metadata can become inaccurate. Because the metadata is used for replication -and routing decisions, any inaccuracies may cause problems. Praefect contains a background worker that -periodically verifies the metadata against the actual state on the disks. The worker: +without going through Praefect, the metadata can become inaccurate. For example if a Gitaly node is +rebuilt, rather than being replaced with a new node, repository verification ensures this is detected. + +The metadata is used for replication and routing decisions, so any inaccuracies may cause problems. +Praefect contains a background worker that periodically verifies the metadata against the actual state on the disks. +The worker: 1. Picks up a batch of replicas to verify on healthy storages. The replicas are either unverified or have exceeded the configured verification interval. Replicas that have never been verified are prioritized, followed by @@ -1512,8 +1515,8 @@ periodically verifies the metadata against the actual state on the disks. The wo The worker acquires an exclusive verification lease on each of the replicas it is about to verify. This avoids multiple workers from verifying the same replica concurrently. The worker releases the leases when it has completed its check. -Praefect contains a background goroutine that releases stale leases every 10 seconds when workers are terminated for -some reason without releasing the lease. +If workers are terminated for some reason without releasing the lease, Praefect contains a background goroutine +that releases stale leases every 10 seconds. The worker logs each of the metadata removals prior to executing them. The `perform_deletions` key indicates whether the invalid metadata records are actually deleted or not. For example: diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index aa487917ef0..45bde083a1a 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -39,7 +39,7 @@ To use a different name for the replacement node for a Gitaly Cluster that has [ For example: ```shell - $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 + $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 current assignments: gitaly-1, gitaly-2 ``` @@ -224,7 +224,7 @@ repository so care must be taken. ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss --virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> +-virtual-storage <virtual-storage> -relative-path <relative-path> -authoritative-storage <storage-name> ``` ### Enable writes or accept data loss @@ -237,7 +237,7 @@ Praefect provides the following subcommands to re-enable writes or accept data l of the up-to-date nodes back online, you might have to accept data loss: ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -relative-path <relative-path> -authoritative-storage <storage-name> ``` When accepting data loss, Praefect: @@ -338,7 +338,7 @@ The `remove-repository` Praefect sub-command removes a repository from a Gitaly In GitLab 14.6 and later, by default, the command operates in dry-run mode. In earlier versions, the command didn't support dry-run mode. For example: ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -relative-path <repository> ``` - Replace `<virtual-storage>` with the name of the virtual storage containing the repository. @@ -349,7 +349,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example: ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply + sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -relative-path <repository> -apply ``` - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration']['virtual_storage]` and looks like the following: @@ -372,7 +372,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t In this example, the virtual storage to specify is `default` or `storage-1`. -- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). +- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_paths.md#hashed-storage). For example: ```plaintext @@ -440,7 +440,7 @@ inaccessible. The `track-repository` Praefect sub-command adds repositories on disk to the Praefect tracking database to be tracked. ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -relative-path <repository> -replica-path <disk_path> -replicate-immediately ``` - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration'][:virtual_storage]` and looks like the following: @@ -463,13 +463,14 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t In this example, the virtual storage to specify is `default` or `storage-1`. -- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage). +- `-relative-path` is the relative path in the virtual storage. Usually [beginning with `@hashed`](../repository_storage_paths.md#hashed-storage). For example: ```plaintext @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git ``` +- `-replica-path` is the relative path on physical storage. Can start with [`@cluster` or match `relative_path`](../repository_storage_paths.md#gitaly-cluster-storage). - `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if [per-repository replication](praefect.md#configure-replication-factor) is set as the replication strategy. - `-replicate-immediately`, available in GitLab 14.6 and later, causes the command to replicate the repository to its secondaries immediately. @@ -523,8 +524,8 @@ If any entry fails these checks, the command aborts prior to attempting to track For example: ```json - {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","authoritative_storage":"gitaly-1","virtual_storage":"default"} - {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","authoritative_storage":"gitaly-2","virtual_storage":"default"} + {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","replica_path":"@cluster/fe/d3/1","authoritative_storage":"gitaly-1","virtual_storage":"default"} + {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","replica_path":"@cluster/7b/28/2","authoritative_storage":"gitaly-2","virtual_storage":"default"} ``` - `-replicate-immediately`, causes the command to replicate the repository to its secondaries immediately. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index aa04c9a92c4..5b26c93cea8 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -155,25 +155,43 @@ Prometheus query to see the hit rate: sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) ``` -### GitLab Shell +#### Custom Hooks -For historical reasons -[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains -the Git hooks that allow GitLab to validate and react to Git pushes. -Because Gitaly "owns" Git pushes, GitLab Shell must therefore be -installed alongside Gitaly. +> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. + +Gitaly supports [custom Git hooks](../server_hooks.md) that are +used to perform tasks based on changes performed in any repository. The custom +hooks directory is configured in the `[hooks]` section: | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | -| `dir` | string | yes | The directory where GitLab Shell is installed.| +| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. | Example: ```toml -[gitlab-shell] -dir = "/home/git/gitlab-shell" +[hooks] +custom_hooks_dir = "/home/git/custom-hooks" ``` +If left unset, no custom hooks are used. + +### GitLab + +> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. + +Gitaly must connect to the GitLab application to perform access +checks when a user performs a change. The parameters to connect to GitLab must +be configured in the `[gitlab]` section. + +| Name | Type | Required | Description | +| ---- | ---- | -------- | ----------- | +| `url` | string | yes | The URL of the GitLab server. +| `secret` | string | no | The secret token used to authenticate with GitLab. | +| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. | + +Only one of `secret` or `secret_file` can be configured. + ### Prometheus You can optionally configure Gitaly to record histogram latencies on GRPC method diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 3d8110e1dab..556bc29b76f 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -20,7 +20,7 @@ and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#par When using standalone Gitaly servers, you must make sure they are the same version as GitLab to ensure full compatibility: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Gitaly Servers**. 1. Confirm all Gitaly servers indicate that they are up to date. @@ -269,7 +269,7 @@ git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* Any other namespaces that the administrator wants to push can be included there as well via additional patterns. -### Command line tools cannot connect to Gitaly +### Command-line tools cannot connect to Gitaly gRPC cannot reach your Gitaly server if: @@ -556,6 +556,9 @@ You can retrieve a repository's metadata by its Praefect-assigned repository ID: sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id> ``` +When the physical path on the physical storage starts with `@cluster`, you can +[find the repository ID in the physical path](index.md#praefect-generated-replica-paths-gitlab-150-and-later). + You can also retrieve a repository's metadata by its virtual storage and relative path: ```shell diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index eed14fe1bf1..4b07b27d702 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -76,7 +76,7 @@ frequently. You can change how often Gitaly is asked to optimize a repository. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. @@ -109,7 +109,7 @@ housekeeping tasks. The manual trigger can be useful when either: To trigger housekeeping tasks manually: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Advanced**. 1. Select **Run housekeeping**. @@ -136,7 +136,7 @@ reduce the likelihood of such race conditions. To trigger a manual prune of unreachable objects: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Advanced**. 1. Select **Run housekeeping**. diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md index 278d585f2d9..b7f71505e70 100644 --- a/doc/administration/inactive_project_deletion.md +++ b/doc/administration/inactive_project_deletion.md @@ -23,7 +23,7 @@ For the default setting on GitLab.com, see the [GitLab.com settings page](../use To configure deletion of inactive projects: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 34f6da1d01d..112b296b96b 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -11,6 +11,31 @@ GitLab, like most large applications, enforces limits in certain features to mai minimum quality of performance. Allowing some features to be limitless could affect security, performance, data, or could even exhaust the allocated resources for the application. +## Instance configuration + +In the instance configuration page, you can find information about some of the +settings that are used in your current GitLab instance. + +Depending on which limits you have configured, you can see: + +- SSH host keys information +- CI/CD limits +- GitLab Pages limits +- Package Registry limits +- Rate limits +- Size limits + +Because this page is visible to everybody, unauthenticated users only see the +the information that is relevant to them. + +To visit the instance configuration page: + +1. On the left sidebar, select **Help** (**{question-o}**) > **Help**. +1. On the Help page, select **Check the current instance configuration**. + +The direct URL is `<gitlab_url>/help/instance_configuration`. For GitLab.com, +you can visit <https://gitlab.com/help/instance_configuration>. + ## Rate limits Rate limits can be used to improve the security and durability of GitLab. @@ -600,6 +625,8 @@ Plan.default.actual_limits.update!(project_ci_variables: 10000) ### Maximum file size per type of artifact +> `ci_max_artifact_size_annotations` limit [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38337) in GitLab 16.3. + Job artifacts defined with [`artifacts:reports`](../ci/yaml/index.md#artifactsreports) that are uploaded by the runner are rejected if the file size exceeds the maximum file size limit. The limit is determined by comparing the project's @@ -615,6 +642,7 @@ setting is used: | Artifact limit name | Default value | |---------------------------------------------|---------------| | `ci_max_artifact_size_accessibility` | 0 | +| `ci_max_artifact_size_annotations` | 0 | | `ci_max_artifact_size_api_fuzzing` | 0 | | `ci_max_artifact_size_archive` | 0 | | `ci_max_artifact_size_browser_performance` | 0 | @@ -814,6 +842,37 @@ To set this limit to 5 KB on a self-managed installation, run the following in t Plan.default.actual_limits.update!(dotenv_size: 5.kilobytes) ``` +### Limit CI/CD job annotations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38337) in GitLab 16.3. + +You can set a limit on the maximum number of [annotations](../ci/yaml/artifacts_reports.md#artifactsreportsannotations) +per CI/CD job. + +Set the limit to `0` to disable it. Defaults to `20` on self-managed instances. + +To set this limit to `100` on a self-managed instance, run the following command in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(ci_job_annotations_num: 100) +``` + +### Limit CI/CD job annotations file size + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38337) in GitLab 16.3. + +You can set a limit on the maximum size of a CI/CD job [annotation](../ci/yaml/artifacts_reports.md#artifactsreportsannotations). + +Set the limit to `0` to disable it. Defaults to 80 KB. + +To set this limit to 100 KB on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(ci_job_annotations_size: 100.kilobytes) +``` + ## Instance monitoring and metrics ### Limit inbound incident management alerts @@ -1045,7 +1104,7 @@ Issues and merge requests enforce these maximums: ## CDN-based limits on GitLab.com -In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://about.gitlab.com/handbook/communication/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations. +In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://handbook.gitlab.com/handbook/communication/confidentiality-levels/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations. ## Container Repository tag deletion limit @@ -1074,6 +1133,20 @@ The [changelog API](../api/repositories.md#add-changelog-data-to-a-changelog-fil - Each namespace (such as a group or a project) can have a maximum of 50 value streams. - Each value stream can have a maximum of 15 stages. +## Audit events streaming destination limits + +### Custom HTTP Endpoint + +- Each top-level group can have a maximum of 5 custom HTTP streaming destinations. + +### Google Cloud Logging + +- Each top-level group can have a maximum of 5 Google Cloud Logging streaming destinations. + +### Amazon S3 + +- Each top-level group can have a maximum of 5 Amazon S3 streaming destinations. + ## List all instance limits To list all instance limit values, run the following from the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -1168,7 +1241,8 @@ web_hook_calls: 0, project_access_token_limit: 0, google_cloud_logging_configurations: 5, ml_model_max_file_size: 10737418240, -limits_history: {} +limits_history: {}, +audit_events_amazon_s3_configurations: 5 ``` Some limit values display as `[FILTERED]` in the list due to diff --git a/doc/administration/integration/diagrams_net.md b/doc/administration/integration/diagrams_net.md index 7f8c01ad7bd..13be572d524 100644 --- a/doc/administration/integration/diagrams_net.md +++ b/doc/administration/integration/diagrams_net.md @@ -7,6 +7,9 @@ type: reference, howto # Diagrams.net **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86206) in GitLab 15.10. +> - Offline environment support [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116281) in GitLab 16.1. + With the [diagrams.net](https://www.diagrams.net/) integration, you can create and embed SVG diagrams in wikis. The diagram editor is available in both the plain text editor and the rich text editor. @@ -43,7 +46,7 @@ For more information, see [Run your own diagrams.net server with Docker](https:/ ## Enable Diagrams.net integration 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Diagrams.net**. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 0356212d6dd..926833cfa1a 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -36,9 +36,9 @@ Supported libraries include: <!-- vale gitlab.Spelling = NO --> -- [Bytefield](https://bytefield-svg.deepsymmetry.org/) +- [Bytefield](https://bytefield-svg.deepsymmetry.org/bytefield-svg/intro.html) - [D2](https://d2lang.com/tour/intro/) -- [DBML](https://www.dbml.org/home/) +- [DBML](https://dbml.dbdiagram.io/home/) - [Ditaa](https://ditaa.sourceforge.net) - [Erd](https://github.com/BurntSushi/erd) - [GraphViz](https://www.graphviz.org/) @@ -61,7 +61,7 @@ read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images You need to enable Kroki integration from Settings under Admin Area. To do that, sign in with an administrator account and follow these steps: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Go to **Settings > General**. 1. Expand the **Kroki** section. diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index 87428d27c66..4fe846e99a6 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -19,7 +19,7 @@ After completing the integration, Mailgun `temporary_failure` and `permanent_fai ## Configure your Mailgun domain > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the `/-/members/mailgun/permanent_failures` URL in GitLab 15.0. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the URL to handle both temporary and permanent failures in GitLab 15.0. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/359113) the URL to handle both temporary and permanent failures in GitLab 15.0. Before you can enable Mailgun in GitLab, set up your own Mailgun endpoints to receive the webhooks. @@ -43,7 +43,7 @@ After configuring your Mailgun domain for the webhook endpoints, you're ready to enable the Mailgun integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General** and expand the **Mailgun** section. 1. Select the **Enable Mailgun** checkbox. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 5e499e302db..0155f0300d4 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -73,8 +73,6 @@ Inside the block you can add any of the diagrams PlantUML supports, such as: You can add parameters to block definitions: -- `format`: Can be either `png` (default) or `svg`. Use `svg` with care, as it's - not supported by all browsers, and isn't supported by Markdown. - `id`: A CSS ID added to the diagram HTML tag. - `width`: Width attribute added to the image tag. - `height`: Height attribute added to the image tag. @@ -319,7 +317,7 @@ stop; After configuring your local PlantUML server, you're ready to enable the PlantUML integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** checkbox. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 2939e227a04..6e39ab8015c 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -112,7 +112,7 @@ they receive a `Connection failed` message. By default, terminal sessions do not expire. To limit the terminal session lifetime in your GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Web terminal**. 1. Set a `max session time`. diff --git a/doc/administration/job_artifacts_troubleshooting.md b/doc/administration/job_artifacts_troubleshooting.md index ba60cbd7aba..c2be485a9ad 100644 --- a/doc/administration/job_artifacts_troubleshooting.md +++ b/doc/administration/job_artifacts_troubleshooting.md @@ -270,7 +270,7 @@ To change the number of job artifacts listed, change the number in `limit(50)`. ### Delete job artifacts from jobs completed before a specific date WARNING: -These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. +These commands remove data permanently from database and storage. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. If you need to manually remove job artifacts associated with multiple jobs while **retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): @@ -453,7 +453,16 @@ value for the total storage space used by artifacts. To recalculate the artifact usage statistics for all projects in the instance, you can run this background script: ```shell -bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]' +gitlab-rake gitlab:refresh_project_statistics_build_artifacts_size[https://example.com/path/file.csv] +``` + +The `https://example.com/path/file.csv` file must list the project IDs for +all projects for which you want to recalculate artifact storage usage. Use this format for the file: + +```csv +PROJECT_ID +1 +2 ``` The artifact usage value can fluctuate to `0` while the script is running. After diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 4659917cf8b..26034ce1d01 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -453,7 +453,7 @@ settings to allow the GitLab domain. See the following documentation for more details: 1. [AWS S3](https://repost.aws/knowledge-center/s3-configure-cors) -1. [Google Cloud Storage](https://cloud.google.com/storage/docs/configuring-cors) +1. [Google Cloud Storage](https://cloud.google.com/storage/docs/using-cors) 1. [Azure Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services). ## Known limitations diff --git a/doc/administration/license.md b/doc/administration/license.md index 732c2840217..05b60955ed9 100644 --- a/doc/administration/license.md +++ b/doc/administration/license.md @@ -28,7 +28,7 @@ To activate your instance with an activation code: - Your subscription confirmation email. - The [Customers Portal](https://customers.gitlab.com/customers/sign_in), on the **Manage Purchases** page. 1. Sign in to your GitLab self-managed instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. 1. Paste the activation code in **Activation code**. @@ -65,19 +65,23 @@ This error occurs when you use an activation code to activate your instance, but You may have connectivity issues due to the following reasons: -- **You have an offline environment**: - - Configure your setup to allow connection to GitLab servers. If connection to GitLab servers is not possible, contact your Sales Representative to request a license key. You can also contact [GitLab support](https://about.gitlab.com/support/#contact-support) if you need help finding your Sales Representative. -- **Customers Portal is not operational**: - - To check for performance or service disruptions, check the Customers Portal [status](https://status.gitlab.com/). - **Firewall settings**: - - Check if your GitLab instance has an encrypted connection to `customers.gitlab.com` (with IP addresses 172.64.146.11 and 104.18.41.245) on port 443: + - Confirm that GitLab instance can establish an encrypted connection to `https://customers.gitlab.com` on port 443. + Note: IP addresses for `https://customers.gitlab.com` are 172.64.146.11 and 104.18.41.245) ```shell curl --verbose "https://customers.gitlab.com/" - ``` + ``` - - If the curl command returns a failure, either: + - If the curl command returns an error, either: - [Configure a proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html) in `gitlab.rb` to point to your server. - - Contact your network administrator to make changes to the proxy. - - If an SSL inspection appliance is used, you must add the appliance's root CA certificate to `/etc/gitlab/trusted-certs` on the server, then run `gitlab-ctl reconfigure`. -
\ No newline at end of file + - Contact your network administrator to make changes to an existing proxy or firewall. + - If an SSL inspection appliance is used, you must add the appliance's root CA certificate to `/etc/gitlab/trusted-certs` on your instance, then run `gitlab-ctl reconfigure`. + +- **Customers Portal is not operational**: + - Check for any active disruptions to the Customers Portal on [status](https://status.gitlab.com/). + +- **You have an offline environment**: + - If you are unable to configure your setup to allow connection to GitLab servers, contact your Sales Representative to request an [Offline license](https://about.gitlab.com/pricing/licensing-faq/cloud-licensing/#what-is-an-offline-cloud-license). + + For assistance finding your sales representative you can contact [GitLab support](https://about.gitlab.com/support/#contact-support). diff --git a/doc/administration/license_file.md b/doc/administration/license_file.md index 5f82536698b..2086e4888a7 100644 --- a/doc/administration/license_file.md +++ b/doc/administration/license_file.md @@ -18,7 +18,7 @@ link to the **Add license** page should be displayed. Otherwise, to add your license: 1. Sign in to GitLab as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. In the **Add License** area, add a license by either uploading the file or entering the key. @@ -96,7 +96,7 @@ To go back to Free features, [delete all expired licenses](#remove-a-license). To remove a license from a self-managed instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. 1. Select **Remove license**. @@ -107,7 +107,7 @@ Repeat these steps to remove all licenses, including those applied in the past. To view your license details: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index a862fd46a3f..1662febc8cc 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -113,7 +113,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. ## Readiness check -It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../administration/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma doesn't accept requests. +It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../administration/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when using Puma, because there is a brief period during a restart where Puma doesn't accept requests. WARNING: Using the `all=1` parameter with the readiness check in GitLab versions 15.4 to 15.8 may cause [increased Praefect memory usage](https://gitlab.com/gitlab-org/gitaly/-/issues/4751) and lead to memory errors. diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index bfab17d37e8..e093c03a13e 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -505,22 +505,16 @@ and as follows. ### `sidekiq.log` +> The default log format for Helm chart installations [changed from `text` to `json`](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/3169) in GitLab 16.0 and later. + This file is located at: - `/var/log/gitlab/sidekiq/current` on Linux package installations. - `/home/git/gitlab/log/sidekiq.log` on self-compiled installations. GitLab uses background jobs for processing tasks which can take a long -time. All information about processing these jobs are written down to -this file. For example: - -```plaintext -2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read' -2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} -``` - -Instead of the previous format, you can opt to generate JSON logs for -Sidekiq. For example: +time. All information about processing these jobs are written to this +file. For example: ```json { @@ -547,10 +541,25 @@ Sidekiq. For example: } ``` +Instead of JSON logs, you can opt to generate text logs for Sidekiq. For example: + +```plaintext +2023-05-16T16:08:55.272Z pid=82525 tid=23rl INFO: Initializing websocket +2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Booted Rails 6.1.7.2 application in production environment +2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Running in ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [arm64-darwin22] +2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: See LICENSE and the LGPL-3.0 for licensing details. +2023-05-16T16:08:55.279Z pid=82525 tid=23rl INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org +2023-05-16T16:08:55.286Z pid=82525 tid=7p4t INFO: Cleaning working queues +2023-05-16T16:09:06.043Z pid=82525 tid=7p7d class=ScheduleMergeRequestCleanupRefsWorker jid=efcc73f169c09a514b06da3f INFO: start +2023-05-16T16:09:06.050Z pid=82525 tid=7p7d class=ScheduleMergeRequestCleanupRefsWorker jid=efcc73f169c09a514b06da3f INFO: arguments: [] +2023-05-16T16:09:06.065Z pid=82525 tid=7p81 class=UserStatusCleanup::BatchWorker jid=e279aa6409ac33031a314822 INFO: start +2023-05-16T16:09:06.066Z pid=82525 tid=7p81 class=UserStatusCleanup::BatchWorker jid=e279aa6409ac33031a314822 INFO: arguments: [] +``` + For Linux package installations, add the configuration option: ```ruby -sidekiq['log_format'] = 'json' +sidekiq['log_format'] = 'text' ``` For self-compiled installations, edit the `gitlab.yml` and set the Sidekiq @@ -559,7 +568,7 @@ For self-compiled installations, edit the `gitlab.yml` and set the Sidekiq ```yaml ## Sidekiq sidekiq: - log_format: json + log_format: text ``` ### `sidekiq_client.log` diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md index 45c0ce37102..79c8dcf4d81 100644 --- a/doc/administration/logs/tracing_correlation_id.md +++ b/doc/administration/logs/tracing_correlation_id.md @@ -25,7 +25,7 @@ You can use your browser's developer tools to monitor and inspect network activity with the site that you're visiting. See the links below for network monitoring documentation for some popular browsers. -- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) +- [Network Monitor - Firefox Developer Tools](https://firefox-source-docs.mozilla.org/devtools-user/network_monitor/index.html) - [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/) - [Safari Web Development Tools](https://developer.apple.com/safari/tools/) - [Microsoft Edge Network panel](https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index c5674527291..36c702bbda1 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -21,7 +21,7 @@ Maintenance Mode allows most external actions that do not change internal state. Enable Maintenance Mode as an administrator in one of these ways: - **Web UI**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. @@ -46,7 +46,7 @@ Enable Maintenance Mode as an administrator in one of these ways: Disable Maintenance Mode in one of three ways: - **Web UI**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Maintenance Mode**, and toggle **Enable Maintenance Mode**. @@ -181,7 +181,7 @@ you should disable all cron jobs except for those related to Geo. To monitor queues and disable jobs: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. In the Sidekiq dashboard, select **Cron** and disable jobs individually or all at once by selecting **Disable All**. diff --git a/doc/administration/merge_requests_approvals.md b/doc/administration/merge_requests_approvals.md index 6cd0edf22eb..8c554b00048 100644 --- a/doc/administration/merge_requests_approvals.md +++ b/doc/administration/merge_requests_approvals.md @@ -19,7 +19,7 @@ and can no longer be changed: To enable merge request approval settings for an instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Push Rules**. 1. Expand **Merge request approvals**. diff --git a/doc/administration/moderate_users.md b/doc/administration/moderate_users.md index 42f1f26586f..3095f696978 100644 --- a/doc/administration/moderate_users.md +++ b/doc/administration/moderate_users.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto @@ -42,7 +42,7 @@ sign in. To view user sign ups pending approval: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Pending approval** tab. @@ -53,7 +53,7 @@ A user sign up pending approval can be approved or rejected from the Admin Area. To approve or reject a user sign up: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Pending approval** tab. @@ -79,7 +79,7 @@ administrators can choose to block the user. Users can be blocked [via an abuse report](../administration/review_abuse_reports.md#blocking-users), by removing them in LDAP, or directly from the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Optional. Select a user. @@ -103,7 +103,7 @@ Users can also be blocked using the [GitLab API](../api/users.md#block-user). A blocked user can be unblocked from the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Blocked** tab. @@ -120,7 +120,7 @@ Users can also be unblocked using the [GitLab API](../api/users.md#unblock-user) The unblock option may be unavailable for LDAP users. To enable the unblock option, the LDAP identity first needs to be deleted: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Blocked** tab. @@ -160,7 +160,7 @@ Users are notified about account deactivation if A user can be deactivated from the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Optional. Select a user. @@ -189,7 +189,7 @@ Administrators can enable automatic deactivation of users who either: To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -240,7 +240,7 @@ A deactivated user can be activated from the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Deactivated** tab. @@ -271,7 +271,7 @@ To block a user and hide their contributions, administrators can ban the user. Users can be banned using the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Optional. Select a user. @@ -284,7 +284,7 @@ The banned user does not consume a [seat](../subscriptions/self_managed/index.md A banned user can be unbanned using the Admin Area. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Banned** tab. @@ -299,7 +299,7 @@ The user's state is set to active and they consume a Use the Admin Area to delete users. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Banned** tab. @@ -314,7 +314,7 @@ You can only delete a user if there are inherited or direct owners of a group. Y You can also delete a user and their contributions, such as merge requests, issues, and groups of which they are the only group owner. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Banned** tab. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md deleted file mode 100644 index dbdcdf22007..00000000000 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Self-monitoring project (removed) **(FREE SELF)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) -in GitLab 14.9 and [removed](https://gitlab.com/groups/gitlab-org/-/epics/10030) in 16.0. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index a1def4764f6..a32a38a11e5 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 72240be0b35..ba0fed0fac1 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -35,7 +35,7 @@ see the [GitLab Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashbo After setting up Grafana, you can enable a link to access it from the GitLab sidebar: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Metrics - Grafana**. @@ -43,14 +43,14 @@ GitLab sidebar: 1. Configure the **Grafana URL**. Enter the full URL of the Grafana instance. 1. Select **Save changes**. -GitLab displays your link in the **Main menu > Admin > Monitoring > Metrics Dashboard**. +GitLab displays your link in the Admin Area under **Monitoring > Metrics Dashboard**. ## Required Scopes > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5822) in GitLab 13.10. -When setting up Grafana through the process above, no scope shows in the screen at -**Main menu > Admin > Applications > GitLab Grafana**. However, the `read_user` scope is +When setting up Grafana through the process above, no scope shows in the screen in +the Admin Area under **Applications > GitLab Grafana**. However, the `read_user` scope is required and is provided to the application automatically. Setting any scope other than `read_user` without also including `read_user` leads to this error when you try to sign in using GitLab as the OAuth provider: diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 8afec54dab2..65aa2208944 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -107,7 +107,7 @@ The performance bar is disabled by default for non-administrators. To enable it for a given group: 1. Sign in as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 6566def823c..d91fd5f8156 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log in to GitLab as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and select **Enable GitLab Prometheus metrics endpoint**. @@ -107,8 +107,8 @@ The following metrics are available: | `gitlab_transaction_db_<role>_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls, grouped by database roles (primary/replica) | `controller`, `action` | | `gitlab_transaction_db_<role>_wal_count_total` | Counter | 14.0 | Counter for total number of WAL (write ahead log location) queries, grouped by database roles (primary/replica) | `controller`, `action` | | `gitlab_transaction_db_<role>_wal_cached_count_total` | Counter | 14.1 | Counter for total number of cached WAL (write ahead log location) queries, grouped by database roles (primary/replica)| `controller`, `action` | -| `http_elasticsearch_requests_duration_seconds` **(PREMIUM)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | -| `http_elasticsearch_requests_total` **(PREMIUM)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_duration_seconds` **(PREMIUM ALL)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_total` **(PREMIUM ALL)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | | `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | @@ -250,18 +250,12 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | | `geo_repositories_checksummed` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | | `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | -| `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | -| `geo_wikis_checksum_failed` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | | `geo_repositories_verified` | Gauge | 10.7 | Number of repositories successfully verified on secondary | `url` | | `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories that failed verification on secondary | `url` | | `geo_repositories_checksum_mismatch` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | -| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis successfully verified on secondary | `url` | -| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis that failed verification on secondary | `url` | -| `geo_wikis_checksum_mismatch` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | | `geo_repositories_checked` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | | `geo_repositories_checked_failed` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | | `geo_repositories_retrying_verification` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | -| `geo_wikis_retrying_verification` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | | `geo_package_files` | Gauge | 13.0 | Number of package files on primary | `url` | | `geo_package_files_checksummed` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | | `geo_package_files_checksum_failed` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary | `url` | @@ -507,6 +501,7 @@ instance. For example, `cache` or `shared_state`. | `gitlab_redis_client_requests_total` | Counter | 13.2 | Number of Redis client requests | | `gitlab_redis_client_requests_duration_seconds` | Histogram | 13.2 | Redis request latency, excluding blocking commands | | `gitlab_redis_client_redirections_total` | Counter | 15.10 | Number of Redis Cluster MOVED/ASK redirections, broken down by redirection type | +| `gitlab_redis_client_requests_pipelined_commands` | Histogram | 16.4 | Number of commands per pipeline sent to a single Redis server | ## Metrics shared directory diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index abdd2f1d0d3..df6dd87c896 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -109,7 +109,7 @@ prometheus['scrape_configs'] = [ ### Standalone Prometheus using the Linux package -The Linux package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md). +The Linux package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md). A standalone Monitoring node is recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md). The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with the Linux package: @@ -169,7 +169,7 @@ ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb` WARNING: Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network. -A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md). +A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. To use an external Prometheus server: diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index aa17452f260..0b12d41eb4e 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -156,12 +156,12 @@ For the storage-specific form, [direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331) because it does not require a shared folder. -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated form, refer to the following guides: +For configuring object storage in GitLab 13.1 and earlier, _or_ for storage types not +For storage types not supported by the consolidated form, refer to the following guides: | Object storage type | Supported by consolidated form? | |---------------------|------------------------------------------| -| [Project-level Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No | +| [Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No | | [Backups](../administration/backup_restore/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | | [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | @@ -237,7 +237,7 @@ To set up an instance profile: } ``` -1. [Attach this role](https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/) +1. [Attach this role](https://repost.aws/knowledge-center/attach-replace-ec2-instance-profile) to the EC2 instance hosting your GitLab instance. 1. Set the `use_iam_profile` GitLab configuration option to `true`. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 270bd778ea3..888ec79d4b6 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -121,7 +121,7 @@ users as long as a large file exists. To disable writes to the `authorized_keys` file: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. @@ -141,7 +141,7 @@ This overview is brief. Refer to the above instructions for more context. 1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file). 1. Enable writes to the `authorized_keys` file. - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. @@ -165,6 +165,8 @@ Additional technical documentation for `gitlab-sshd` may be found in the ## Troubleshooting +### SSH traffic slow or high CPU load + If your SSH traffic is [slow](https://github.com/linux-pam/linux-pam/issues/270) or causing high CPU load, be sure to check the size of `/var/log/btmp`, and ensure it is rotated on a regular basis or after reaching a certain size. If this file is very large, GitLab SSH fast lookup can cause the bottleneck to be hit more frequently, thus decreasing performance even further. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index c27bedd39de..49746c7cc72 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -264,9 +264,16 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ git@newserver:/mnt/gitlab/repositories' ``` +<!--- start_remove The following content will be removed on remove_date: '2024-05-16' --> + ### Thousands of Git repositories: use one `rsync` per repository WARNING: +The Rake task `gitlab:list_repos` was +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384361) in GitLab 16.4 and is planned for +removal in 17.0. Use [backup and restore](#recommended-approach-in-all-cases) instead. + +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -292,6 +299,11 @@ This process: #### Parallel `rsync` for all repositories known to GitLab WARNING: +The Rake task `gitlab:list_repos` was +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384361) in GitLab 16.4 and is planned for +removal in 17.0. Use [backup and restore](#recommended-approach-in-all-cases) instead. + +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -353,6 +365,11 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\ #### Parallel `rsync` only for repositories with recent activity WARNING: +The Rake task `gitlab:list_repos` was +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384361) in GitLab 16.4 and is planned for +removal in 17.0. Use [backup and restore](#recommended-approach-in-all-cases) instead. + +WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). @@ -381,3 +398,5 @@ sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ /home/git/repositories \ /mnt/gitlab/repositories ``` + +<!--- end_remove -->
\ No newline at end of file diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index eea4b8035b7..4bc1861a22b 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -28,18 +28,19 @@ architecture. | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | +| OpenSUSE 15.5 | GitLab CE / GitLab EE 16.4.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | RHEL 9 | GitLab CE / GitLab EE 16.0.0 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2032 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Oct 2027 | <https://www.suse.com/lifecycle/> | | SLES 15 | GitLab EE 14.8.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://www.suse.com/lifecycle/> | -| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | +| Oracle Linux 7 | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Dec 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | | Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 22.04 | GitLab CE / GitLab EE 15.5.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2027 | <https://wiki.ubuntu.com/Releases> | | Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2025 | <https://aws.amazon.com/amazon-linux-2/faqs/> | -| Amazon Linux 2022 | GitLab CE / GitLab EE 15.9.0 | amd64, arm64 | [Amazon Linux 2022 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2022) | October 2027 | <https://aws.amazon.com/linux/amazon-linux-2022/faqs/> | -| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | +| Amazon Linux 2023 | GitLab CE / GitLab EE 16.3.0 | amd64, arm64 | [Amazon Linux 2023 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2023) | 2028 | <https://docs.aws.amazon.com/linux/al2023/ug/release-cadence.html> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | June 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | | Raspberry Pi OS (Bullseye) | GitLab CE 15.5.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2026 | [Raspberry Pi Details](https://www.raspberrypi.com/news/raspberry-pi-os-debian-bullseye/) | NOTE: diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 384df9a6e6a..dcc6b768eed 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -343,7 +343,7 @@ the Container Registry by themselves, follow the steps below. In GitLab, tokens for the Container Registry expire every five minutes. To increase the token duration: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Container Registry**. @@ -500,7 +500,7 @@ To configure the `s3` storage driver for a Linux package installation: `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). You can set a rate limit on connections to S3 to avoid 503 errors from the S3 API. To do this, - set `maxrequestspersecond` to a number within the [S3 request rate threshold](https://repost.aws/knowledge-center/s3-503-within-request-rate-prefix): + set `maxrequestspersecond` to a number within the [S3 request rate threshold](https://repost.aws/knowledge-center/http-5xx-errors-s3): ```ruby registry['storage'] = { @@ -823,7 +823,7 @@ In the examples below we set the Registry's port to `5010`. ## Disable Container Registry per project If Registry is enabled in your GitLab instance, but you don't need it for your -project, you can [disable it from your project's settings](../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). +project, you can [disable it from your project's settings](../../user/project/settings/index.md#configure-project-features-and-permissions). ## Use an external container registry with GitLab as an auth endpoint @@ -1252,6 +1252,16 @@ itself on the system so that the `gitlab-ctl` command can bring the registry ser Also, there's no way to save progress or results during the mark phase of the process. Only once blobs start being deleted is anything permanent done. +### Continuous Zero Downtime Garbage Collection **(BETA)** + +You can run garbage collection in the background without the need to schedule it or require read-only mode, +if you migrate to the [metadata database (beta)](#use-a-postgresql-database-for-metadata). + +NOTE: +If you would like to try this [beta feature](../../policy/experiment-beta-support.md#beta), +you should review the [known limitations](#known-limitations). If you have any feedback, +you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423459). + ## Configure GitLab and Registry to run on separate nodes (Linux package installations) By default, package assumes that both services are running on the same node. @@ -1353,6 +1363,42 @@ including all the supported storage backends. To migrate to the GitLab Container you can follow the instructions on this page, and use the same storage backend as the Distribution Registry. The GitLab Container Registry should accept the same configuration that you are using for the Distribution Registry. +## Use a PostgreSQL database for metadata **(FREE SELF BETA)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/423459) in GitLab 16.4 as a [Beta feature](../../policy/experiment-beta-support.md) for self-managed GitLab instances. + +WARNING: +While the metadata database is already in use on GitLab.com, it is in early beta for self-managed GitLab instances. + +By default, the container registry uses object storage to persist metadata +related to container images. This method to store metadata limits how efficiently +the data can be accessed, especially data spanning multiple images, such as when listing tags. +By using a database to store this data, many new features are possible, including +[online garbage collection](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/db/online-garbage-collection.md) +which removes old data automatically with zero downtime. + +This database works in conjunction with the object storage already used by the registry, but does not replace object storage. +You must continue to maintain an object storage solution even after migrating to a metadata database. + +### Known Limitations + +- No support for online migrations. +- Geo Support is not confirmed. +- Registry database migrations must be ran manually when upgrading versions. + +### Migration Instructions and Feedback + +Instructions on how to migrate to the database may be found in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423459) for the beta period. +This issue also serves as a place to report issues and to get an overview of the beta status. + +### Metadata database feature support + +You can migrate existing registries to the metadata database, and use online garbage collection. + +Some database-enabled features are only enabled for GitLab.com and automatic database provisioning for +the registry database is not available. Review the feature support table in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423459#supported-feature-status) +for the status of features related to the container registry database. + ## Troubleshooting Before diving in to the following sections, here's some basic troubleshooting: diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index c84e5eb8c8c..f64c53e28a2 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -141,8 +141,8 @@ The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: ```ruby - external_url "http://gitlab.example.com" # external_url here is only for reference - pages_external_url "http://pages.example.com" # not a subdomain of external_url + external_url "http://example.com" # external_url here is only for reference + pages_external_url 'http://example.io' # Important: not a subdomain of external_url, so cannot be http://pages.example.com ``` 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). @@ -163,12 +163,12 @@ URL scheme: `https://<namespace>.example.io/<project_slug>` NGINX proxies all requests to the daemon. Pages daemon doesn't listen to the outside world. -1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. +1. Place the wildcard LTS certificate for `*.example.io` and the key inside `/etc/gitlab/ssl`. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - external_url "https://gitlab.example.com" # external_url here is only for reference - pages_external_url "https://pages.example.com" # not a subdomain of external_url + external_url "https://example.com" # external_url here is only for reference + pages_external_url 'https://example.io' # Important: not a subdomain of external_url, so cannot be https://pages.example.com pages_nginx['redirect_http_to_https'] = true ``` @@ -211,8 +211,8 @@ This setup is primarily intended to be used when [installing a GitLab POC on Ama 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - external_url "https://gitlab.example.com" # external_url here is only for reference - pages_external_url "https://pages.example.com" # not a subdomain of external_url + external_url "https://example.com" # external_url here is only for reference + pages_external_url 'https://example.io' # Important: not a subdomain of external_url, so cannot be https://pages.example.com pages_nginx['enable'] = true pages_nginx['listen_port'] = 80 @@ -334,8 +334,8 @@ world. Custom domains are supported, but no TLS. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - external_url "http://gitlab.example.com" # external_url here is only for reference - pages_external_url "http://pages.example.com" # not a subdomain of external_url + external_url "http://example.com" # external_url here is only for reference + pages_external_url 'http://example.io' # Important: not a subdomain of external_url, so cannot be http://pages.example.com nginx['listen_addresses'] = ['192.0.2.1'] # The primary IP of the GitLab instance pages_nginx['enable'] = false gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] # The secondary IPs for the GitLab Pages daemon @@ -361,12 +361,12 @@ In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside world. Custom domains and TLS are supported. -1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. +1. Place the wildcard LTS certificate for `*.example.io` and the key inside `/etc/gitlab/ssl`. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby - external_url "https://gitlab.example.com" # external_url here is only for reference - pages_external_url "https://pages.example.com" # not a subdomain of external_url + external_url "https://example.com" # external_url here is only for reference + pages_external_url 'https://example.io' # Important: not a subdomain of external_url, so cannot be https://pages.example.com nginx['listen_addresses'] = ['192.0.2.1'] # The primary IP of the GitLab instance pages_nginx['enable'] = false gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] # The secondary IPs for the GitLab Pages daemon @@ -406,7 +406,7 @@ domain as a custom domain to their project. If your user base is private or otherwise trusted, you can disable the verification requirement: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. @@ -424,7 +424,7 @@ sites served under a custom domain. To enable it: 1. Choose an email address on which you want to receive notifications about expiring domains. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. @@ -478,7 +478,7 @@ pre-existing applications must modify the GitLab Pages OAuth application. Follow this: 1. Enable [access control](#access-control). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Applications**. 1. Expand **GitLab Pages**. @@ -498,7 +498,7 @@ This can be helpful to restrict information published with Pages websites to the of your instance only. To do that: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. @@ -689,7 +689,7 @@ Prerequisite: To set the global maximum pages size for a project: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. @@ -704,7 +704,7 @@ Prerequisite: To set the maximum size of each GitLab Pages site in a group, overriding the inherited setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Pages**. 1. Enter a value under **Maximum size** in MB. @@ -718,7 +718,7 @@ Prerequisite: To set the maximum size of GitLab Pages site in a project, overriding the inherited setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. In **Maximum size of pages**, enter the size in MB. 1. Select **Save changes**. @@ -731,7 +731,7 @@ Prerequisite: To set the maximum number of GitLab Pages custom domains for a project: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**, and expand **Pages**. 1. Enter a value for **Maximum number of custom domains per project**. Use `0` for unlimited domains. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 97dacfc1902..fcbc04ed0c3 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -479,7 +479,7 @@ The default for the maximum size of unpacked archives per project is 100 MB. To change this value: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. diff --git a/doc/administration/pages/troubleshooting.md b/doc/administration/pages/troubleshooting.md index a103c793763..be5b8ba27ee 100644 --- a/doc/administration/pages/troubleshooting.md +++ b/doc/administration/pages/troubleshooting.md @@ -152,7 +152,7 @@ This can happen to GitLab instances with multiple servers running both the core GitLab application and GitLab Pages. This can also happen when a single container is running both the core GitLab application and GitLab Pages. -AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) +AWS [recommends using an IP target type](https://repost.aws/knowledge-center/target-connection-fails-load-balancer) to resolve this issue. Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) @@ -170,7 +170,7 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com This problem comes from the permissions of the GitLab Pages OAuth application. To fix it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Applications > GitLab Pages**. 1. Edit the application. @@ -214,7 +214,7 @@ You may see this error if `pages_external_url` was updated at some point of time 1. Check the [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application): - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Applications** and then **Add new application**. 1. Ensure the **Callback URL/Redirect URI** is using the protocol (HTTP or HTTPS) that diff --git a/doc/administration/polling.md b/doc/administration/polling.md index 50dbc7467d3..b27613663da 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -26,7 +26,7 @@ The default value (`1`) is recommended for the majority of GitLab installations. To adjust the polling interval multiplier: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Polling interval multiplier**. diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index f8b6be1fb21..d640dbe9341 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -182,9 +182,9 @@ To configure these options with a hosts list, use the following example: ```ruby gitlab_rails['db_load_balancing'] = { - 'hosts' => ['primary.example.com', 'secondary1.example.com', 'secondary2.example.com'] - 'max_replication_difference' => 16777216 # 16 MB - 'max_replication_lag_time' => 30 + 'hosts' => ['primary.example.com', 'secondary1.example.com', 'secondary2.example.com'], + 'max_replication_difference' => 16777216, # 16 MB + 'max_replication_lag_time' => 30, 'replica_check_interval' => 30 } ``` diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index c547e5c3638..a1fc8c49ee3 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1086,7 +1086,7 @@ Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the sam `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. -### Near zero downtime upgrade of PostgreSQL in a Patroni cluster (Experimental) +### Near zero downtime upgrade of PostgreSQL in a Patroni cluster **(EXPERIMENT)** Patroni enables you to run a major PostgreSQL upgrade without shutting down the cluster. However, this requires additional resources to host the new Patroni nodes with the upgraded PostgreSQL. In practice, with this diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 1babafc902e..82f3ffa2193 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -18,6 +18,9 @@ Bear in mind that the syntax is very specific. Remove any spaces in the argument before/after the brackets. Also, some shells (for example, Zsh) can interpret the open/close brackets (`[]`) separately. You may want to either escape the brackets or use double quotes. +You can only import repositories that are in the namespace of the owner of the GitHub personal access token being used to import. For more information, see +[issue 424105](https://gitlab.com/gitlab-org/gitlab/-/issues/424105). + Prerequisite: - At least the Maintainer role on the destination group to import to. diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 4a657b04bdc..e6700288631 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 87cce30723e..cdb70ca715b 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -176,13 +176,15 @@ Running? ... yes Checking Sidekiq ... Finished -Checking GitLab ... +Checking GitLab App... Database config exists? ... yes Database is SQLite ... no All migrations up? ... yes GitLab config exists? ... yes -GitLab config outdated? ... no +GitLab config up to date? ... no +Cable config exists? ... yes +Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Init script exists? ... yes @@ -359,7 +361,7 @@ status in the output of the `sudo gitlab-rake db:migrate:status` command. sudo gitlab-ctl restart sidekiq ``` -## Rebuild database indexes (Experiment) +## Rebuild database indexes **(EXPERIMENT)** WARNING: This feature is experimental, and isn't enabled by default. Use caution when diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md index 8d8585bb171..21b4050a93b 100644 --- a/doc/administration/raketasks/service_desk_email.md +++ b/doc/administration/raketasks/service_desk_email.md @@ -12,7 +12,7 @@ The following are Service Desk email-related Rake tasks. ## Secrets -GitLab can use [Service Desk email](../../user/project/service_desk/index.md#configure-service-desk-alias-email) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. +GitLab can use [Service Desk email](../../user/project/service_desk/configure.md#configure-service-desk-alias-email) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. ### Show secret diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 9e0a89fa7cb..ce931e78c2b 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is a collection of Rake tasks to help you list and migrate existing projects and their attachments to the new -[hashed storage](../repository_storage_types.md) that GitLab +[hashed storage](../repository_storage_paths.md) that GitLab uses to organize the Git data. ## List projects and attachments @@ -75,7 +75,7 @@ To have a summary and then a list of projects and their attachments using hashed ## Migrate to hashed storage WARNING: -In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +In GitLab 13.0, [hashed storage](../repository_storage_paths.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. @@ -109,7 +109,7 @@ sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 To monitor the progress in GitLab: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue @@ -129,7 +129,7 @@ You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your r ## Rollback from hashed storage to legacy storage WARNING: -In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +In GitLab 13.0, [hashed storage](../repository_storage_paths.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index c94c76c6532..92f9205e76d 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -500,7 +500,7 @@ cluster to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: @@ -1277,7 +1277,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2b91d493145..87e2ff157ab 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -517,7 +517,7 @@ cluster to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: @@ -1296,7 +1296,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index bd184c372b3..8f22343e770 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -249,7 +249,7 @@ to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index cf2d9667481..2d40adf8166 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -787,7 +787,7 @@ cluster to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: @@ -1223,7 +1223,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 2187b0ff02c..6f09077cab7 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -509,7 +509,7 @@ cluster to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: @@ -1290,7 +1290,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 2b85426de58..7666e7cc0b5 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -780,7 +780,7 @@ cluster to be used with GitLab. You can optionally use a [third party external service for PostgreSQL](../../administration/postgresql/external.md). -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default from [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. If you use a third party external service: @@ -1217,7 +1217,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/versions/gitlab_14_changes.md#1440). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 98bbcc464b8..2ad9380a00f 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -107,9 +107,19 @@ This is an alternative and more **advanced** setup compared to a standard Refere ### GitLab Geo (Cross Regional Distribution / Disaster Recovery) -With [GitLab Geo](../geo/index.md) you can have both distributed environments in different regions and a full Disaster Recovery (DR) setup in place. With this setup you would have 2 or more separate environments, with one being a primary that gets replicated to the others. In the rare event the primary site went down completely you could fail over to one of the other environments. +With [GitLab Geo](../geo/index.md), you can achieve distributed environments in +different regions with a full Disaster Recovery (DR) setup in place. GitLab Geo +requires at least two separate environments: -This is an **advanced and complex** setup and should only be undertaken if you have DR as a key requirement. Decisions then on how each environment are configured would also need to be taken, such as if each environment itself would be the full size and / or have HA. +- One primary site. +- One or more secondary sites that serve as replicas. + +If the primary site becomes unavailable, you can fail over to one of the secondary sites. + +This **advanced and complex** setup should only be undertaken if DR is +a key requirement for your environment. You must also make additional decisions +on how each site is configured, such as if each secondary site would be the +same architecture as the primary, or if each site is configured for HA. ### Cloud provider services @@ -194,7 +204,7 @@ However, additional workloads can multiply the impact of operations by triggerin You may need to adjust the suggested specifications to compensate if you use, for example: - Security software on the nodes. -- Hundreds of concurrent CI jobs for [large repositories](../../ci/large_repositories/index.md). +- Hundreds of concurrent CI jobs for [large repositories](../../user/project/repository/managing_large_repositories.md). - Custom scripts that [run at high frequency](../logs/log_parsing.md#print-top-api-user-agents). - [Integrations](../../integration/index.md) in many large projects. - [Server hooks](../server_hooks.md). @@ -334,7 +344,7 @@ If you choose to use a third party external service: Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: -- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/versions/gitlab_14_changes.md#1440) for more details. - [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. @@ -413,7 +423,7 @@ For more information, see our [handbook page](https://about.gitlab.com/handbook/ Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: -- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. +- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) Terraform and Ansible scripts for building the environments. - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. Network latency on the test environments between components on all Cloud Providers were measured at <5 ms. This is shared as an observation and not as an implicit recommendation. diff --git a/doc/administration/reporting/git_abuse_rate_limit.md b/doc/administration/reporting/git_abuse_rate_limit.md index 270a7cb4800..7e2c340f20d 100644 --- a/doc/administration/reporting/git_abuse_rate_limit.md +++ b/doc/administration/reporting/git_abuse_rate_limit.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -21,7 +21,7 @@ GitLab team members can view more information in this confidential epic: ## Configure Git abuse rate limiting -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Reporting**. 1. Expand **Git abuse rate limit**. @@ -41,7 +41,7 @@ If automatic banning is enabled, an email notification is sent when a user is ab ## Unban a user -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Banned** tab and search for the account you want to unban. diff --git a/doc/administration/reporting/ip_addr_restrictions.md b/doc/administration/reporting/ip_addr_restrictions.md index 5b749c62c30..2e152b0b176 100644 --- a/doc/administration/reporting/ip_addr_restrictions.md +++ b/doc/administration/reporting/ip_addr_restrictions.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -19,7 +19,7 @@ unique IP addresses. Therefore, the IP addresses per user limit should take into ## Configure IP address restrictions -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Reporting**. 1. Expand **Spam and Anti-bot Protection**. diff --git a/doc/administration/reporting/spamcheck.md b/doc/administration/reporting/spamcheck.md index 8e478729299..c92336a39b9 100644 --- a/doc/administration/reporting/spamcheck.md +++ b/doc/administration/reporting/spamcheck.md @@ -40,7 +40,7 @@ Spamcheck is only available for package-based installations: ## Configure GitLab to use Spamcheck -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Reporting**. 1. Expand **Spam and Anti-bot Protection**. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 1a33b6b5746..241f88793ff 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -20,7 +20,7 @@ committed to a repository. GitLab administrators can: To check a project's repository using GitLab UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Projects**. 1. Select the project to check. @@ -33,7 +33,7 @@ project page in the Admin Area. If the checks fail, see [what to do](#what-to-do Instead of checking repositories manually, GitLab can be configured to run the checks periodically: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. @@ -65,7 +65,7 @@ You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command li by default. - For GitLab Helm chart installations, repositories are stored in the `/home/git/repositories` directory inside the Gitaly pod by default. -1. [Identify the subdirectory that contains the repository](repository_storage_types.md#from-project-name-to-hashed-path) +1. [Identify the subdirectory that contains the repository](repository_storage_paths.md#from-project-name-to-hashed-path) that you need to check. 1. Run the check. For example: @@ -87,7 +87,7 @@ If a repository check fails, locate the error in the [`repocheck.log` file](logs If periodic repository checks cause false alarms, you can clear all repository check states: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). 1. Expand the **Repository maintenance** section. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index c09c88ee020..a3e0158fd24 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -9,144 +9,203 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab stores [repositories](../user/project/repository/index.md) on repository storage. Repository storage is either: -- A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly to the directory where the repositories are stored. GitLab - directly accessing a directory containing repositories - [is deprecated](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). - GitLab should be configured to access GitLab repositories through a `gitaly_address`. - -GitLab allows you to define multiple repository storages to distribute the storage load between -several mount points. For example: - -- When using Gitaly (Linux package installation-style configuration): - - ```ruby - git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, - }) - ``` - -- When using direct repository storage (self-compiled installation-style configuration): +- Physical storage configured with a `gitaly_address` that points to a [Gitaly node](gitaly/index.md). +- [Virtual storage](gitaly/index.md#virtual-storage) that stores repositories on a Gitaly Cluster. - ```plaintext - default: - gitaly_address: tcp://gitaly1.example:8075 - storage2: - gitaly_address: tcp://gitaly2.example:8075 - ``` +WARNING: +Repository storage could be configured as a `path` that points directly to the directory where the repositories are +stored. GitLab directly accessing a directory containing repositories is deprecated. You should configure GitLab to +access repositories through a physical or virtual storage. For more information on: -- Configuring Gitaly, see [Configure Gitaly](gitaly/index.md#configure-gitaly). -- Configuring direct repository access, see the following section below. +- Configuring Gitaly, see [Configure Gitaly](gitaly/configure_gitaly.md). +- Configuring Gitaly Cluster, see [Configure Gitaly Cluster](gitaly/praefect.md). -## Configure repository storage paths +## Hashed storage -WARNING: -The following information is for configuring GitLab to directly access repositories. This -configuration option is deprecated in favor of using [Gitaly](gitaly/index.md). -[Issue 403318](https://gitlab.com/gitlab-org/gitlab/-/issues/403318) proposes to remove this configuration option. +> **Storage name** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly storage name** and **Relative path** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly relative path** in GitLab 16.3. -To configure repository storage paths: +Hashed storage stores projects on disk in a location based on a hash of the project's ID. This makes the folder +structure immutable and eliminates the need to synchronize state from URLs to disk structure. This means that renaming a +group, user, or project: -1. Edit the necessary configuration files: - - `/etc/gitlab/gitlab.rb`, for Linux package installations. - - `gitlab.yml`, for self-compiled installations. -1. Add the required repository storage paths. +- Costs only the database transaction. +- Takes effect immediately. -For repository storage paths: +The hash also helps spread the repositories more evenly on the disk. The top-level directory +contains fewer folders than the total number of top-level namespaces. -- You must have at least one storage path called `default`. -- The paths are defined in key-value pairs. Apart from `default`, the key can be any name you choose - to name the file path. -- The target directories and any of its sub paths must not be a symlink. -- No target directory may be a sub-directory of another. That is, no nesting. For example, the - following configuration is invalid: +The hash format is based on the hexadecimal representation of a SHA256, calculated with +`SHA256(project.id)`. The top-level folder uses the first two characters, followed by another folder +with the next two characters. They are both stored in a special `@hashed` folder so they can +co-exist with existing legacy storage projects. For example: - ```plaintext - default: - path: /mnt/git-storage-1 - storage2: - path: /mnt/git-storage-1/git-storage-2 # <- NOT OK because of nesting - ``` +```ruby +# Project's repository: +"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" -### Configure for backups +# Wiki's repository: +"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" +``` -For [backups](../administration/backup_restore/index.md) to work correctly: +### Translate hashed storage paths -- The repository storage path cannot be a mount point. -- The GitLab user must have correct permissions for the parent directory of the path. +Troubleshooting problems with the Git repositories, adding hooks, and other tasks requires you +translate between the human-readable project name and the hashed storage path. You can translate: -The Linux package takes care of these issues for you but for self-compiled installations, you should be extra -careful. +- From a [project's name to its hashed path](#from-project-name-to-hashed-path). +- From a [hashed path to a project's name](#from-hashed-path-to-project-name). -While restoring a backup, the current contents of `/home/git/repositories` are moved to -`/home/git/repositories.old`. If `/home/git/repositories` is a mount point, then `mv` would be -moving things between mount points, and problems can occur. +#### From project name to hashed path -Ideally, `/home/git` is the mount point, so things remain inside the same mount point. Linux package -installations guarantee this because they don't specify the full repository path but instead -the parent path, but self-compiled installations do not. +Administrators can look up a project's hashed path from its name or ID using: -### Example configuration +- The [Admin Area](../administration/admin_area.md#administering-projects). +- A Rails console. -In the examples below, we add two additional repository storage paths configured to two additional -mount points. +To look up a project's hash path in the Admin Area: -For compatibility reasons `gitlab.yml` has a different structure than Linux package installation configuration: +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Overview > Projects** and select the project. +1. Locate the **Relative path** field. The value is similar to: -- In `gitlab.yml`, you indicate the path for the repositories. For example, `/home/git/repositories`. -- In Linux package installation configuration, you indicate `git_data_dirs`, which could be `/home/git` for - example. The Linux package installation then creates a `repositories` directory under that path to use with - `gitlab.yml`. + ```plaintext + "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" + ``` -For self-compiled installations: +To look up a project's hash path using a Rails console: -1. Edit `gitlab.yml` and add the storage paths: +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Run a command similar to this example (use either the project's ID or its name): - ```yaml - repositories: - # Paths where repositories can be stored. Give the canonicalized absolute pathname. - # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! - storages: # You must have at least a 'default' repository storage path. - default: - path: /home/git/repositories - storage1: - path: /mnt/storage1/repositories - storage2: - path: /mnt/storage2/repositories + ```ruby + Project.find(16).disk_path + Project.find_by_full_path('group/project').disk_path ``` -1. [Restart GitLab](restart_gitlab.md#self-compiled-installations) for the changes to take effect. +#### From hashed path to project name -1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). +Administrators can look up a project's name from its hashed relative path using: -For Linux package installations: +- A Rails console. +- The `config` file in the `*.git` directory. -1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default one: +To look up a project's name using the Rails console: + +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Run a command similar to this example: ```ruby - git_data_dirs({ - "default" => { "path" => "/var/opt/gitlab/git-data" }, - "storage1" => { "path" => "/mnt/storage1/git-data" }, - "storage2" => { "path" => "/mnt/storage2/git-data" } - }) + ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project ``` -1. [Restart GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. +The quoted string in that command is the directory tree you can find on your GitLab server. For +example, on a default Linux package installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` +with `.git` from the end of the directory name removed. -1. [Configure where new repositories are stored](#configure-where-new-repositories-are-stored). +The output includes the project ID and the project name. For example: -NOTE: -Linux package installations store the repositories in a `repositories` subdirectory of the `git-data` directory. +```plaintext +=> #<Project id:16 it/supportteam/ticketsystem> +``` + +To look up a project's name using the `config` file in the `*.git` directory: + +1. Locate the `*.git` directory. This directory is located in `/var/opt/gitlab/git-data/repositories/@hashed/`, where the first four + characters of the hash are the first two directories in the path under `@hashed/`. For example, on a default Linux package installation the + `*.git` directory of the hash `b17eb17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9` would be + `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`. +1. Open the `config` file and locate the `fullpath=` key under `[gitlab]`. + +### Hashed object pools + +Object pools are repositories used to deduplicate forks of public and internal projects and +contain the objects from the source project. Using `objects/info/alternates`, the source project and +forks use the object pool for shared objects. For more information, see +[How Git object deduplication works in GitLab](../development/git_object_deduplication.md). + +Objects are moved from the source project to the object pool when housekeeping is run on the source +project. Object pool repositories are stored similarly to regular repositories in a directory called `@pools` instead of `@hashed` + +```ruby +# object pool paths +"@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" +``` + +WARNING: +Do not run `git prune` or `git gc` in object pool repositories, which are stored in the `@pools` directory. +This can cause data loss in the regular repositories that depend on the object pool. + +### Group wiki storage + +Unlike project wikis that are stored in the `@hashed` directory, group wikis are stored in a directory called `@groups`. +Like project wikis, group wikis follow the hashed storage folder convention, but use a hash of the group ID rather than the project ID. + +For example: + +```ruby +# group wiki paths +"@groups/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" +``` + +### Gitaly Cluster storage + +If Gitaly Cluster is used, Praefect manages storage locations. The internal path used by Praefect for the repository +differs from the hashed path. For more information, see +[Praefect-generated replica paths](gitaly/index.md#praefect-generated-replica-paths-gitlab-150-and-later). + +### Object storage support + +This table shows which storable objects are storable in each storage type: + +| Storable object | Hashed storage | S3 compatible | +|:-----------------|:---------------|:--------------| +| Repository | Yes | - | +| Attachments | Yes | - | +| Avatars | No | - | +| Pages | No | - | +| Docker Registry | No | - | +| CI/CD job logs | No | - | +| CI/CD artifacts | No | Yes | +| CI/CD cache | No | Yes | +| LFS objects | Similar | Yes | +| Repository pools | Yes | - | + +Files stored in an S3-compatible endpoint can have the same advantages as +[hashed storage](#hashed-storage), as long as they are not prefixed with +`#{namespace}/#{project_name}`. This is true for CI/CD cache and LFS objects. + +#### Avatars + +Each file is stored in a directory that matches the `id` assigned to it in the database. The +filename is always `avatar.png` for user avatars. When an avatar is replaced, the `Upload` model is +destroyed and a new one takes place with a different `id`. + +#### CI/CD artifacts + +CI/CD artifacts are S3-compatible. + +#### LFS objects + +[LFS Objects in GitLab](../topics/git/lfs/index.md) implement a similar +storage pattern using two characters and two-level folders, following the Git implementation: + +```ruby +"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" + +# Based on object `oid`: `8909029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c`, path will be: +"shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c" +``` + +LFS objects are also [S3-compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). ## Configure where new repositories are stored -After you [configure](#configure-repository-storage-paths) multiple repository storage paths, you -can choose where new repositories are stored: +After you configure multiple repository storages, you can choose where new repositories are stored: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository** and expand the **Repository storage** section. @@ -157,8 +216,7 @@ Each repository storage path can be assigned a weight from 0-100. When a new pro these weights are used to determine the storage location the repository is created on. The higher the weight of a given repository storage path relative to other repository storages -paths, the more often it is chosen. That is, -`(storage weight) / (sum of all weights) * 100 = chance %`. +paths, the more often it is chosen (`(storage weight) / (sum of all weights) * 100 = chance %`). By default, if repository weights have not been configured earlier: diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 6e47d3099bd..844dc76b23d 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,245 +1,8 @@ --- -stage: Systems -group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'repository_storage_paths.md' +remove_date: '2023-11-29' --- -# Repository storage types **(FREE SELF)** +This document was moved to [another location](repository_storage_paths.md). -GitLab can be configured to use one or multiple repository storages. These storages can be: - -- Accessed via [Gitaly](gitaly/index.md), optionally on - [its own server](gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). -- Mounted to the local disk. This [method](repository_storage_paths.md#configure-repository-storage-paths) - is deprecated and [scheduled to be removed](https://gitlab.com/groups/gitlab-org/-/epics/2320) in - GitLab 14.0. -- Exposed as an NFS shared volume. This method is deprecated and - [scheduled to be removed](https://gitlab.com/groups/gitlab-org/-/epics/3371) in GitLab 14.0. - -In GitLab: - -- Repository storages are configured in: - - `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash for Linux package installations. - - `gitlab.yml` by the `repositories.storages` key for self-compiled installations. -- The `default` repository storage is available in any installations that haven't customized it. By - default, it points to a Gitaly node. - -The repository storage types documented here apply to any repository storage defined in -`git_data_dirs({})` or `repositories.storages`. - -## Hashed storage - -> **Storage name** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly storage name** and **Relative path** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly relative path** in GitLab 16.3. - -Hashed storage stores projects on disk in a location based on a hash of the project's ID. Hashed -storage is different to [legacy storage](#legacy-storage) where a project is stored based on: - -- The project's URL. -- The folder structure where the repository is stored on disk. - -This makes the folder structure immutable and eliminates the need to synchronize state from URLs to -disk structure. This means that renaming a group, user, or project: - -- Costs only the database transaction. -- Takes effect immediately. - -The hash also helps spread the repositories more evenly on the disk. The top-level directory -contains fewer folders than the total number of top-level namespaces. - -The hash format is based on the hexadecimal representation of a SHA256, calculated with -`SHA256(project.id)`. The top-level folder uses the first two characters, followed by another folder -with the next two characters. They are both stored in a special `@hashed` folder so they can -co-exist with existing legacy storage projects. For example: - -```ruby -# Project's repository: -"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" - -# Wiki's repository: -"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" -``` - -### Translate hashed storage paths - -Troubleshooting problems with the Git repositories, adding hooks, and other tasks requires you -translate between the human-readable project name and the hashed storage path. You can translate: - -- From a [project's name to its hashed path](#from-project-name-to-hashed-path). -- From a [hashed path to a project's name](#from-hashed-path-to-project-name). - -#### From project name to hashed path - -Administrators can look up a project's hashed path from its name or ID using: - -- The [Admin Area](../administration/admin_area.md#administering-projects). -- A Rails console. - -To look up a project's hash path in the Admin Area: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Overview > Projects** and select the project. -1. Locate the **Relative path** field. The value is similar to: - - ```plaintext - "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" - ``` - -To look up a project's hash path using a Rails console: - -1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). -1. Run a command similar to this example (use either the project's ID or its name): - - ```ruby - Project.find(16).disk_path - Project.find_by_full_path('group/project').disk_path - ``` - -#### From hashed path to project name - -Administrators can look up a project's name from its hashed relative path using: - -- A Rails console. -- The `config` file in the `*.git` directory. - -To look up a project's name using the Rails console: - -1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). -1. Run a command similar to this example: - - ```ruby - ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project - ``` - -The quoted string in that command is the directory tree you can find on your GitLab server. For -example, on a default Linux package installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` -with `.git` from the end of the directory name removed. - -The output includes the project ID and the project name. For example: - -```plaintext -=> #<Project id:16 it/supportteam/ticketsystem> -``` - -To look up a project's name using the `config` file in the `*.git` directory: - -1. Locate the `*.git` directory. This directory is located in `/var/opt/gitlab/git-data/repositories/@hashed/`, where the first four - characters of the hash are the first two directories in the path under `@hashed/`. For example, on a default Linux package installation the - `*.git` directory of the hash `b17eb17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9` would be - `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`. -1. Open the `config` file and locate the `fullpath=` key under `[gitlab]`. - -### Hashed object pools - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. - -Object pools are repositories used to deduplicate forks of public and internal projects and -contain the objects from the source project. Using `objects/info/alternates`, the source project and -forks use the object pool for shared objects. For more information, see -[How Git object deduplication works in GitLab](../development/git_object_deduplication.md). - -Objects are moved from the source project to the object pool when housekeeping is run on the source -project. Object pool repositories are stored similarly to regular repositories in a directory called `@pools` instead of `@hashed` - -```ruby -# object pool paths -"@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" -``` - -WARNING: -Do not run `git prune` or `git gc` in object pool repositories, which are stored in the `@pools` directory. -This can cause data loss in the regular repositories that depend on the object pool. - -### Group wiki storage - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in GitLab 13.5. - -Unlike project wikis that are stored in the `@hashed` directory, group wikis are stored in a directory called `@groups`. -Like project wikis, group wikis follow the hashed storage folder convention, but use a hash of the group ID rather than the project ID. - -For example: - -```ruby -# group wiki paths -"@groups/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" -``` - -### Gitaly Cluster storage - -If Gitaly Cluster is used, Praefect manages storage locations. For more information, see [Praefect-generated replica paths](gitaly/index.md#praefect-generated-replica-paths-gitlab-150-and-later). - -### Object storage support - -This table shows which storable objects are storable in each storage type: - -| Storable object | Legacy storage | Hashed storage | S3 compatible | GitLab version | -|:-----------------|:---------------|:---------------|:--------------|:---------------| -| Repository | Yes | Yes | - | 10.0 | -| Attachments | Yes | Yes | - | 10.2 | -| Avatars | Yes | No | - | - | -| Pages | Yes | No | - | - | -| Docker Registry | Yes | No | - | - | -| CI/CD job logs | No | No | - | - | -| CI/CD artifacts | No | No | Yes | 9.4 / 10.6 | -| CI/CD cache | No | No | Yes | - | -| LFS objects | Yes | Similar | Yes | 10.0 / 10.7 | -| Repository pools | No | Yes | - | 11.6 | - -Files stored in an S3-compatible endpoint can have the same advantages as -[hashed storage](#hashed-storage), as long as they are not prefixed with -`#{namespace}/#{project_name}`. This is true for CI/CD cache and LFS objects. - -#### Avatars - -Each file is stored in a directory that matches the `id` assigned to it in the database. The -filename is always `avatar.png` for user avatars. When an avatar is replaced, the `Upload` model is -destroyed and a new one takes place with a different `id`. - -#### CI/CD artifacts - -CI/CD artifacts are: - -- S3-compatible since GitLab 9.4, initially available in [GitLab Premium](https://about.gitlab.com/pricing/). -- Available in [GitLab Free](https://about.gitlab.com/pricing/) since GitLab 10.6. - -#### LFS objects - -[LFS Objects in GitLab](../topics/git/lfs/index.md) implement a similar -storage pattern using two characters and two-level folders, following the Git implementation: - -```ruby -"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" - -# Based on object `oid`: `8909029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c`, path will be: -"shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c" -``` - -LFS objects are also [S3-compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). - -## Legacy storage - -WARNING: -In GitLab 13.0, legacy storage is deprecated. If you haven't migrated to hashed storage yet, check -the [migration instructions](raketasks/storage.md#migrate-to-hashed-storage). Support for legacy -storage is [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in GitLab -14.0. In GitLab 13.0 and later, switching new projects to legacy storage is not possible. The -option to choose between hashed and legacy storage in the Admin Area is disabled. - -Legacy storage was the storage behavior prior to version GitLab 10.0. For historical reasons, -GitLab replicated the same mapping structure from the projects URLs: - -- Project's repository: `#{namespace}/#{project_name}.git`. -- Project's wiki: `#{namespace}/#{project_name}.wiki.git`. - -This structure enabled you to migrate from existing solutions to GitLab, and for Administrators to -find where the repository was stored. This approach also had some drawbacks: - -- Storage location concentrated a large number of top-level namespaces. The impact could be - reduced by [multiple repository storage paths](repository_storage_paths.md). -- Because backups were a snapshot of the same URL mapping, if you tried to recover a very old - backup, you needed to verify whether any project had taken the place of an old removed or renamed - project sharing the same URL. This meant that `mygroup/myproject` from your backup may not have - been the same original project that was at that same URL today. -- Any change in the URL needed to be reflected on disk (when groups, users, or projects were - renamed. This could add a lot of load in big installations, especially if using any type of - network-based file system. +<!-- This redirect file can be deleted after <2023-11-29>. --> diff --git a/doc/administration/review_abuse_reports.md b/doc/administration/review_abuse_reports.md index e3891cbe68a..4ff53a4e1b0 100644 --- a/doc/administration/review_abuse_reports.md +++ b/doc/administration/review_abuse_reports.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, howto @@ -16,7 +16,7 @@ reports in the Admin Area. To receive notifications of new abuse reports by email: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Reporting**. 1. Expand the **Abuse reports** section. @@ -34,7 +34,7 @@ To find out more about reporting abuse, see To access abuse reports: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Abuse Reports**. diff --git a/doc/administration/secure_files.md b/doc/administration/secure_files.md index d4a57ed5d51..6c61c390655 100644 --- a/doc/administration/secure_files.md +++ b/doc/administration/secure_files.md @@ -120,7 +120,7 @@ See [the available connection settings for different providers](object_storage.m ```ruby gitlab_rails['ci_secure_files_object_store_enabled'] = true - gitlab_rails['ci_secure_files_object_store_remote_directory'] = "terraform" + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "ci_secure_files" gitlab_rails['ci_secure_files_object_store_connection'] = { 'provider' => 'AWS', 'region' => 'eu-central-1', diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 104eaafd8ad..4d8377b4117 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -39,7 +39,7 @@ Prerequisites: - The [storage name](gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage), path to the Gitaly configuration file (default is `/var/opt/gitlab/gitaly/config.toml` on Linux package instances), and the - [repository relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository. + [repository relative path](repository_storage_paths.md#from-project-name-to-hashed-path) for the repository. To set server hooks for a repository: @@ -71,15 +71,15 @@ If you implemented the server hook code correctly, it should execute when the Gi To create server hooks for a repository: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Go to **Overview > Projects** and select the project you want to add a server hook to. 1. On the page that appears, locate the value of **Relative path**. This path is where server hooks must be located. - - If you are using [hashed storage](repository_storage_types.md#hashed-storage), see - [Translate hashed storage paths](repository_storage_types.md#translate-hashed-storage-paths) for information on + - If you are using [hashed storage](repository_storage_paths.md#hashed-storage), see + [Translate hashed storage paths](repository_storage_paths.md#translate-hashed-storage-paths) for information on interpreting the relative path. - - If you are not using [hashed storage](repository_storage_types.md#hashed-storage): + - If you are not using [hashed storage](repository_storage_paths.md#hashed-storage): - For Linux package installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. - For self-compiled installations, the path is usually `/home/git/repositories/<group>/<project>.git`. 1. On the file system, create a new directory in the correct location called `custom_hooks`. @@ -109,7 +109,7 @@ To accomplish this, follow the same steps for setting custom repository hooks fo The location to copy the scripts to depends on where repositories are stored: -- In GitLab 15.2 and earlier, Gitaly Cluster uses the [hashed storage path](repository_storage_types.md#hashed-storage) +- In GitLab 15.2 and earlier, Gitaly Cluster uses the [hashed storage path](repository_storage_paths.md#hashed-storage) reported by the GitLab application. - In GitLab 15.3 and later, new repositories are created using [Praefect-generated replica paths](gitaly/index.md#praefect-generated-replica-paths-gitlab-150-and-later), @@ -169,7 +169,7 @@ subdirectories. Prerequisites: -- The [storage name and relative path](repository_storage_types.md#from-project-name-to-hashed-path) for the repository. +- The [storage name and relative path](repository_storage_paths.md#from-project-name-to-hashed-path) for the repository. To remove server hooks, pass an empty tarball to `hook set` to indicate that the repository should contain no hooks. For example: diff --git a/doc/administration/settings/account_and_limit_settings.md b/doc/administration/settings/account_and_limit_settings.md index 3d632880113..930448b3bd3 100644 --- a/doc/administration/settings/account_and_limit_settings.md +++ b/doc/administration/settings/account_and_limit_settings.md @@ -16,7 +16,7 @@ the [project limits for existing users](#projects-limit-for-a-user). To configure the maximum number of projects in personal namespaces for new users: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -30,7 +30,7 @@ in their users personal namespace. However, projects can still be created in a g You can edit a specific user, and change the maximum number of projects this user can create in their personal namespace: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview** > **Users**. 1. From the list of users, select a user. @@ -44,7 +44,7 @@ can create in their personal namespace: The maximum file size for attachments in GitLab comments and replies is 100 MB. To change the maximum attachment size: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -60,7 +60,7 @@ For GitLab.com repository size limits, read [accounts and limit settings](../../ You can change the maximum push size for your instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -75,86 +75,6 @@ because the [web server](../../development/architecture.md#components) must receive the file before GitLab can generate the commit. Use [Git LFS](../../topics/git/lfs/index.md) to add large files to a repository. -## Max export size - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86124) in GitLab 15.0. - -To modify the maximum file size for exports in GitLab: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**, then expand **Account and limit**. -1. Increase or decrease by changing the value in **Maximum export size (MB)**. - -## Max import size - -> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. - -To modify the maximum file size for imports in GitLab: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Account and limit**. -1. Increase or decrease by changing the value in **Maximum import size (MB)**. - -This setting applies only to repositories -[imported from a GitLab export file](../../user/project/settings/import_export.md#import-a-project-and-its-data). - -If you choose a size larger than the configured value for the web server, -you may receive errors. See the [troubleshooting section](#troubleshooting) for more -details. - -For GitLab.com repository size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). - -## Maximum remote file size for imports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384976) in GitLab 16.3. - -You can modify the maximum remote file size for imports from external object storages (for example, AWS) in GitLab. - -To modify the maximum import remote file size: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Account and limit**. -1. Increase or decrease by changing the value in **Maximum import remote file size (MB)**. Set to `0` to set no file size limit. - -## Maximum download file size for imports by direct transfer - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384976) in GitLab 16.3. - -You can modify the maximum download file size for imports by direct transfer in GitLab. - -To modify the maximum download file size for imports by direct transfer: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Account and limit**. -1. Increase or decrease by changing the value in **Direct transfer maximum download file size (MB)**. Set to `0` to set no download file size limit. - -## Maximum decompressed file size for imported archives - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128218) in GitLab 16.3. - -When you import a project using [file exports](../../user/project/settings/import_export.md) or [direct transfer](../../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended), you can specify the maximum decompressed file size for imported archives. The default value is 25 GB. - -When you import a compressed file, the decompressed size cannot exceed the maximum decompressed file size limit. If the decompressed size exceeds the configured limit, the following error is returned: - -```plaintext -Decompressed archive size validation failed. -``` - -To modify the maximum decompressed file size for imports in GitLab: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Account and limit**. -1. Set another value for **Maximum decompressed size (MiB)**. - ## Personal access token prefix > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20968) in GitLab 13.7. @@ -172,7 +92,7 @@ The default prefix is `glpat-` but administrators can change it. To change the default global prefix: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -219,7 +139,7 @@ These settings can be found in: 1. Fill in the **Repository size limit (MB)** field in the **Naming, visibility** section. 1. Select **Save changes**. - GitLab global settings: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -242,7 +162,7 @@ For details on manually purging files, see [reducing the repository size using G You can change how long users can remain signed in without activity. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. The set duration is in **Session duration (minutes)**. @@ -257,7 +177,7 @@ For details, see [cookies used for sign-in](../../user/profile/index.md#cookies- Users can select the **Remember me** checkbox on sign-in, and their session will remain active for an indefinite period of time when accessed from that specific browser. You can turn off this setting if you need sessions to expire for security or compliance purposes. Turning off this setting will ensure users' sessions expire after the number of minutes of inactivity set when you [customize your session duration](#customize-the-default-session-duration). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -275,7 +195,7 @@ GitLab administrators can choose to customize the session duration (in minutes) To set a limit on how long these sessions are valid: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -303,7 +223,7 @@ there are no restrictions. To set a lifetime on how long SSH keys are valid: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -340,7 +260,7 @@ there are no restrictions. To set a lifetime on how long access tokens are valid: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Account and limit** section. @@ -363,7 +283,7 @@ To maintain integrity of user details in [Audit Events](../../administration/aud To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -385,7 +305,7 @@ By default, new users can create top-level groups. GitLab administrators can pre - The [application setting API](../../api/settings.md#change-application-settings). - In GitLab 15.4 and earlier, a [configuration file](../../administration/user_settings.md#use-configuration-files-to-prevent-new-users-from-creating-top-level-groups). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -397,7 +317,7 @@ By default, new users can create top-level groups. GitLab administrators can pre By default, newly created users have a public profile. GitLab administrators can set new users to have a private profile by default: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. @@ -410,7 +330,7 @@ By default, newly created users have a public profile. GitLab administrators can By default, users can delete their own accounts. GitLab administrators can prevent users from deleting their own accounts: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Account and limit**. diff --git a/doc/administration/settings/continuous_integration.md b/doc/administration/settings/continuous_integration.md index edf61701a33..f0423021e8b 100644 --- a/doc/administration/settings/continuous_integration.md +++ b/doc/administration/settings/continuous_integration.md @@ -15,7 +15,7 @@ job artifacts. To enable (or disable) [Auto DevOps](../../topics/autodevops/index.md) for all projects: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Check (or uncheck to disable) the box that says **Default to Auto DevOps pipeline for all projects**. @@ -33,7 +33,7 @@ If you want to disable it for a specific project, you can do so in You can set all new projects to have the instance's shared runners available by default. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. @@ -53,7 +53,7 @@ you can assign that runner to other projects. To enable a project runner for more than one project: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. From the left sidebar, select **CI/CD > Runners**. 1. Select the runner you want to edit. @@ -67,7 +67,7 @@ To enable a project runner for more than one project: To display details about the instance's shared runners in all projects' runner settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. @@ -77,7 +77,7 @@ runner settings: To view the rendered details: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -85,19 +85,19 @@ To view the rendered details: ## Maximum artifacts size -The maximum size of the [job artifacts](../../administration/job_artifacts.md) -can be set at: +An administrator can set the maximum size of the +[job artifacts](../../administration/job_artifacts.md) at: -- The instance level. -- [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/21688), the project and group level. +- The instance level +- The project and group level For the setting on GitLab.com, see [Artifacts maximum size](../../user/gitlab_com/index.md#gitlab-cicd). -The value is in MB and the default is 100 MB per job. To change it at the: +The value is in MB, and the default is 100 MB per job. An administrator can change the default value at the: - Instance level: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD > Continuous Integration and Deployment**. 1. Change the value of **Maximum artifacts size (MB)**. @@ -115,9 +115,6 @@ The value is in MB and the default is 100 MB per job. To change it at the: 1. Change the value of **maximum artifacts size** (in MB). 1. Select **Save changes** for the changes to take effect. -NOTE: -The setting at all levels is only available to GitLab administrators. - ## Default artifacts expiration The default expiration time of the [job artifacts](../../administration/job_artifacts.md) @@ -125,7 +122,7 @@ can be set in the Admin Area of your GitLab instance. The syntax of duration is described in [`artifacts:expire_in`](../../ci/yaml/index.md#artifactsexpire_in) and the default value is `30 days`. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Change the value of default expiration time. @@ -157,7 +154,7 @@ If disabled at the instance level, you cannot enable this per-project. To disable the setting: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. @@ -181,7 +178,7 @@ blueprint for more details. To set the duration for which the jobs are considered as old and expired: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Continuous Integration and Deployment** section. @@ -199,7 +196,7 @@ For the value set for GitLab.com, see [Scheduled job archiving](../../user/gitla To set all new [CI/CD variables](../../ci/variables/index.md) as [protected](../../ci/variables/index.md#protect-a-cicd-variable) by default: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Select **Protect CI/CD variables by default**. @@ -211,7 +208,7 @@ To set all new [CI/CD variables](../../ci/variables/index.md) as The maximum number of [includes](../../ci/yaml/includes.md) per pipeline can be set at the instance level. The default is `150`. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Change the value of **Maximum includes**. @@ -224,7 +221,7 @@ The default is `150`. The default CI/CD configuration file and path for new projects can be set in the Admin Area of your GitLab instance (`.gitlab-ci.yml` if not set): -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Input the new file and path in the **Default CI/CD configuration file** field. @@ -239,7 +236,7 @@ It is also possible to specify a [custom CI/CD configuration file for a specific You can configure some [CI/CD limits](../../administration/instance_limits.md#cicd-limits) from the Admin Area: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Continuous Integration and Deployment** section. @@ -263,7 +260,7 @@ walkthrough on how to add one. To enable or disable the banner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Select or clear the **Enable pipeline suggestion banner** checkbox. @@ -300,7 +297,7 @@ in the pipeline editor. To select a CI/CD template for the required pipeline configuration: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Required pipeline configuration** section. @@ -315,7 +312,7 @@ GitLab administrators can disable the forwarding of Maven requests to [Maven Cen To disable forwarding Maven requests: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Package Registry** section. @@ -328,7 +325,7 @@ GitLab administrators can disable the forwarding of npm requests to [npmjs.com]( To disable it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Package Registry** section. @@ -341,7 +338,7 @@ GitLab administrators can disable the forwarding of PyPI requests to [pypi.org]( To disable it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Package Registry** section. @@ -354,7 +351,7 @@ GitLab administrators can adjust the maximum allowed file size for each package To set the maximum file size: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand the **Package Registry** section. @@ -375,7 +372,7 @@ By default, all members of a project and group are able to register runners. To restrict all users in an instance from registering runners: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -398,7 +395,7 @@ GitLab administrators can adjust group permissions to restrict runner registrati To restrict runner registration by members in a specific group: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Groups** and find your group. 1. Select **Edit**. @@ -413,7 +410,7 @@ By default, GitLab instances periodically fetch official runner version data fro To disable your instance fetching this data: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. diff --git a/doc/administration/settings/deprecated_api_rate_limits.md b/doc/administration/settings/deprecated_api_rate_limits.md index f8db0810af5..21385a6779d 100644 --- a/doc/administration/settings/deprecated_api_rate_limits.md +++ b/doc/administration/settings/deprecated_api_rate_limits.md @@ -34,7 +34,7 @@ Prerequisite: To override the general user and IP rate limits for requests to deprecated API endpoints: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Deprecated API Rate Limits**. diff --git a/doc/administration/settings/email.md b/doc/administration/settings/email.md index e4972897aab..c79394ee407 100644 --- a/doc/administration/settings/email.md +++ b/doc/administration/settings/email.md @@ -21,7 +21,7 @@ address in the body of the email instead. To include the author's email address in the email body: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. @@ -34,7 +34,7 @@ GitLab can send email in multipart format (HTML and plain text) or plain text on To enable multipart email: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. @@ -50,7 +50,7 @@ This configuration option sets the email hostname for [private commit emails](.. To change the hostname used in private commit emails: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. @@ -69,7 +69,7 @@ can be used for legal, auditing, or compliance reasons, for example. To add additional text to emails: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. @@ -82,7 +82,7 @@ GitLab sends email notifications to users when their account has been deactivate To disable these notifications: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. @@ -105,7 +105,7 @@ setting. To add additional text to deactivation emails: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Email**. diff --git a/doc/administration/settings/external_authorization.md b/doc/administration/settings/external_authorization.md index 45887fdccb8..2d110dc2ea7 100644 --- a/doc/administration/settings/external_authorization.md +++ b/doc/administration/settings/external_authorization.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -45,7 +45,7 @@ Alternatively, learn where to install custom certificates by using The external authorization service can be enabled by an administrator: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **External authorization**. @@ -66,7 +66,7 @@ Prerequisites: To allow authorization with deploy tokens and keys: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **External authorization**, and: diff --git a/doc/administration/settings/files_api_rate_limits.md b/doc/administration/settings/files_api_rate_limits.md index cb5442c957f..025f8c86993 100644 --- a/doc/administration/settings/files_api_rate_limits.md +++ b/doc/administration/settings/files_api_rate_limits.md @@ -30,7 +30,7 @@ Prerequisite: To override the general user and IP rate limits for requests to the Repository files API: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Files API Rate Limits**. diff --git a/doc/administration/settings/floc.md b/doc/administration/settings/floc.md index 6bd5a6dfed4..f461472bfe7 100644 --- a/doc/administration/settings/floc.md +++ b/doc/administration/settings/floc.md @@ -22,7 +22,7 @@ Permissions-Policy: interest-cohort=() To enable it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Federated Learning of Cohorts (FLoC)**. diff --git a/doc/administration/settings/git_lfs_rate_limits.md b/doc/administration/settings/git_lfs_rate_limits.md index cb2cc80e397..acd3945750f 100644 --- a/doc/administration/settings/git_lfs_rate_limits.md +++ b/doc/administration/settings/git_lfs_rate_limits.md @@ -21,7 +21,7 @@ rate limits. Git LFS rate limits are disabled by default. If enabled and configured, these limits supersede the [general user and IP rate limits](../settings/user_and_ip_rate_limits.md): -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Git LFS Rate Limits**. diff --git a/doc/administration/settings/gitaly_timeouts.md b/doc/administration/settings/gitaly_timeouts.md index 49dc7763cd0..3304db3d148 100644 --- a/doc/administration/settings/gitaly_timeouts.md +++ b/doc/administration/settings/gitaly_timeouts.md @@ -11,7 +11,7 @@ configured to make sure that long-running Gitaly calls don't needlessly take up To access Gitaly timeout settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand the **Gitaly timeouts** section. diff --git a/doc/administration/settings/help_page.md b/doc/administration/settings/help_page.md index 46c2c395102..23ca7d55cde 100644 --- a/doc/administration/settings/help_page.md +++ b/doc/administration/settings/help_page.md @@ -16,7 +16,7 @@ the GitLab sign-in page. You can add a help message, which is shown at the top of the GitLab `/help` page (for example, <https://gitlab.com/help>): -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sign-in and Help page**. @@ -34,7 +34,7 @@ is restricted, `/help` is visible only to authenticated users. You can add a help message, which is shown on the GitLab sign-in page. The message appears on the sign-in page: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sign-in and Help page**. @@ -48,7 +48,7 @@ You can now see the message on the sign-in page. GitLab marketing-related entries are occasionally shown on the Help page. To hide these entries: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sign-in and Help page**. @@ -62,7 +62,7 @@ You can specify a custom URL to which users are directed when they: - Select **Support** from the Help dropdown list. - Select **See our website for help** on the Help page. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sign-in and Help page**. @@ -77,7 +77,7 @@ You can specify a custom URL to which users are directed when they: You can redirect all `/help` links to a destination that meets the [necessary requirements](#destination-requirements). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sign-in and Help page**. diff --git a/doc/administration/settings/import_and_export_settings.md b/doc/administration/settings/import_and_export_settings.md new file mode 100644 index 00000000000..af1729246ec --- /dev/null +++ b/doc/administration/settings/import_and_export_settings.md @@ -0,0 +1,153 @@ +--- +stage: Manage +group: Import and Integrate +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference +--- + +# Import and export settings **(FREE SELF)** + +Settings for import- and export-related features. + +## Configure allowed import sources + +Before you can import projects from other systems, you must enable the +[import source](../../user/gitlab_com/index.md#default-import-sources) for that system. + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Import and export settings** section. +1. Select each of **Import sources** to allow. +1. Select **Save changes**. + +## Enable project export + +To enable the export of +[projects and their data](../../user/project/settings/import_export.md#export-a-project-and-its-data): + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Import and export settings** section. +1. Scroll to **Project export**. +1. Select the **Enabled** checkbox. +1. Select **Save changes**. + +## Enable migration of groups and projects by direct transfer + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. + +You can enable migration of groups by direct transfer using the UI. + +To enable migration of groups by direct transfer: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Import and export settings** section. +1. Scroll to **Allow migrating GitLab groups and projects by direct transfer**. +1. Select the **Enabled** checkbox. +1. Select **Save changes**. + +The same setting +[is available](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) in the API as the +`bulk_import_enabled` attribute. + +## Max export size + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86124) in GitLab 15.0. + +To modify the maximum file size for exports in GitLab: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**, then expand **Import and export settings**. +1. Increase or decrease by changing the value in **Maximum export size (MiB)**. + +## Max import size + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. + +To modify the maximum file size for imports in GitLab: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Import and export settings**. +1. Increase or decrease by changing the value in **Maximum import size (MiB)**. + +This setting applies only to repositories +[imported from a GitLab export file](../../user/project/settings/import_export.md#import-a-project-and-its-data). + +If you choose a size larger than the configured value for the web server, +you may receive errors. See the [troubleshooting section](../../administration/settings/account_and_limit_settings.md#troubleshooting) for more +details. + +For GitLab.com repository size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). + +## Maximum remote file size for imports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384976) in GitLab 16.3. + +You can modify the maximum remote file size for imports from external object storages (for example, AWS) in GitLab. + +To modify the maximum import remote file size: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Import and export settings**. +1. Increase or decrease by changing the value in **Maximum import remote file size (MiB)**. Set to `0` to set no file size limit. + +## Maximum download file size for imports by direct transfer + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384976) in GitLab 16.3. + +You can modify the maximum download file size for imports by direct transfer in GitLab. + +To modify the maximum download file size for imports by direct transfer: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Import and export settings**. +1. Increase or decrease by changing the value in **Direct transfer maximum download file size (MiB)**. Set to `0` to set no download file size limit. + +## Maximum decompressed file size for imported archives + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128218) in GitLab 16.3. +> - **Maximum decompressed file size for archives from imports** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130081) from **Maximum decompressed size** in GitLab 16.4. + +When you import a project using [file exports](../../user/project/settings/import_export.md) or [direct transfer](../../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended), you can specify the maximum decompressed file size for imported archives. The default value is 25 GB. + +When you import a compressed file, the decompressed size cannot exceed the maximum decompressed file size limit. If the decompressed size exceeds the configured limit, the following error is returned: + +```plaintext +Decompressed archive size validation failed. +``` + +To modify the maximum decompressed file size for imports in GitLab: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Import and export settings**. +1. Set another value for **Maximum decompressed file size for archives from imports (MiB)**. + +## Timeout for decompressing archived files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128218) in GitLab 16.4. + +When you [import a project](../../user/project/settings/import_export.md), you can specify the maximum time out for decompressing imported archives. The default value is 210 seconds. + +To modify the maximum decompressed file size for imports in GitLab: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Import and export settings**. +1. Set another value for **Timeout for decompressing archived files (seconds)**. diff --git a/doc/administration/settings/import_export_rate_limits.md b/doc/administration/settings/import_export_rate_limits.md index 99385c77cdf..dad72ffdf44 100644 --- a/doc/administration/settings/import_export_rate_limits.md +++ b/doc/administration/settings/import_export_rate_limits.md @@ -12,7 +12,7 @@ You can configure the rate limits for imports and exports of projects and groups To change a rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Import and export rate limits**. diff --git a/doc/administration/settings/incident_management_rate_limits.md b/doc/administration/settings/incident_management_rate_limits.md index 2a74c843107..5a8b1b96c6d 100644 --- a/doc/administration/settings/incident_management_rate_limits.md +++ b/doc/administration/settings/incident_management_rate_limits.md @@ -30,7 +30,7 @@ Requests that exceed the limit are logged into `auth.log`. To set inbound incident management alert limits: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Incident Management Limits**. diff --git a/doc/administration/settings/index.md b/doc/administration/settings/index.md index a5746ad26e4..77bac42d899 100644 --- a/doc/administration/settings/index.md +++ b/doc/administration/settings/index.md @@ -19,7 +19,7 @@ read [GitLab.com settings](../../user/gitlab_com/index.md). To access the **Admin Area**: 1. Sign in to your GitLab instance as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. ## Change the default first day of the week @@ -27,7 +27,7 @@ To access the **Admin Area**: You can change the [Default first day of the week](../../user/profile/preferences.md) for the entire GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Scroll to the **Localization** section, and select your desired first day of the week. @@ -37,7 +37,7 @@ for the entire GitLab instance: You can change the [Default language](../../user/profile/preferences.md) for the entire GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Scroll to the **Localization** section, and select your desired default language. diff --git a/doc/administration/settings/instance_template_repository.md b/doc/administration/settings/instance_template_repository.md index 8c8c9f44998..510c88e7738 100644 --- a/doc/administration/settings/instance_template_repository.md +++ b/doc/administration/settings/instance_template_repository.md @@ -20,7 +20,7 @@ while the project remains secure. To select a project to serve as the custom template repository: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Templates**. 1. Expand **Templates** @@ -32,6 +32,9 @@ After you add templates, you can use them for the entire instance. They are available in the [Web Editor](../../user/project/repository/web_editor.md) and through the [API settings](../../api/settings.md). +These templates cannot be used as a value of the +[`include:template`](../../ci/yaml/index.md#includetemplate) key in `.gitlab-ci.yml`. + ## Supported file types and locations Templates must be added to a specific subdirectory in the repository, diff --git a/doc/administration/settings/package_registry_rate_limits.md b/doc/administration/settings/package_registry_rate_limits.md index ffba5bbf15a..2ddb6bfcd17 100644 --- a/doc/administration/settings/package_registry_rate_limits.md +++ b/doc/administration/settings/package_registry_rate_limits.md @@ -30,7 +30,7 @@ no difference in functionality compared to the general user and IP rate limits. To enable the unauthenticated request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Package registry rate limits**. @@ -45,7 +45,7 @@ To enable the unauthenticated request rate limit: To enable the authenticated API request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network** 1. Expand **Package registry rate limits**. diff --git a/doc/administration/settings/project_integration_management.md b/doc/administration/settings/project_integration_management.md index 95dddf34182..ad43c70e253 100644 --- a/doc/administration/settings/project_integration_management.md +++ b/doc/administration/settings/project_integration_management.md @@ -18,11 +18,15 @@ for all projects that didn't have it already enabled. Only the entire settings for an integration can be inherited. Per-field inheritance is proposed in [epic 2137](https://gitlab.com/groups/gitlab-org/-/epics/2137). -## Manage instance-level default settings for a project integration **(FREE SELF)** +## Manage instance-level default settings for a project integration + +Prerequisite: + +- You must have administrator access to the instance. To manage instance-level default settings for a project integration: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select an integration. @@ -58,9 +62,13 @@ is proposed in [epic 2137](https://gitlab.com/groups/gitlab-org/-/epics/2137). ### Remove an instance-level default setting +Prerequisite: + +- You must have administrator access to the instance. + To remove an instance-level default setting: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select an integration. @@ -72,19 +80,27 @@ Resetting an instance-level default setting removes the integration from all pro > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218252) in GitLab 14.2. +Prerequisite: + +- You must have administrator access to the instance. + To view projects in your instance that [use custom settings](#use-custom-settings-for-a-project-or-group-integration): -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select an integration. 1. Select the **Projects using custom settings** tab. -## Manage group-level default settings for a project integration +## Manage group-level default settings for a project integration **(FREE ALL)** + +Prerequisite: + +- You must have at least the Maintainer role for the group. To manage group-level default settings for a project integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Integrations**. 1. Select an integration. 1. Complete the fields. @@ -119,20 +135,28 @@ is proposed in [epic 2137](https://gitlab.com/groups/gitlab-org/-/epics/2137). ### Remove a group-level default setting +Prerequisite: + +- You must have at least the Maintainer role for the group. + To remove a group-level default setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Integrations**. 1. Select an integration. 1. Select **Reset** and confirm. Resetting a group-level default setting removes integrations that use default settings and belong to a project or subgroup of the group. -## Use instance-level or group-level default settings for a project integration +## Use instance-level or group-level default settings for a project integration **(FREE ALL)** + +Prerequisite: + +- You must have at least the Maintainer role for the project. To use instance-level or group-level default settings for a project integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select an integration. 1. On the right, from the dropdown list, select **Use default settings**. @@ -140,11 +164,15 @@ To use instance-level or group-level default settings for a project integration: 1. Complete the fields. 1. Select **Save changes**. -## Use custom settings for a project or group integration +## Use custom settings for a project or group integration **(FREE ALL)** + +Prerequisite: + +- You must have at least the Maintainer role for the project or group. To use custom settings for a project or group integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Integrations**. 1. Select an integration. 1. On the right, from the dropdown list, select **Use custom settings**. diff --git a/doc/administration/settings/protected_paths.md b/doc/administration/settings/protected_paths.md index 5deba7dca11..3244c25e906 100644 --- a/doc/administration/settings/protected_paths.md +++ b/doc/administration/settings/protected_paths.md @@ -34,7 +34,7 @@ See also: Throttling of protected paths is enabled by default and can be disabled or customized. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Protected paths**. diff --git a/doc/administration/settings/push_event_activities_limit.md b/doc/administration/settings/push_event_activities_limit.md index ff924e0d208..36aa1eec306 100644 --- a/doc/administration/settings/push_event_activities_limit.md +++ b/doc/administration/settings/push_event_activities_limit.md @@ -26,7 +26,7 @@ the activity feed. To modify this setting: - In the Admin Area: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Performance optimization**. diff --git a/doc/administration/settings/rate_limit_on_issues_creation.md b/doc/administration/settings/rate_limit_on_issues_creation.md index 20f3febbf28..14d484d91b1 100644 --- a/doc/administration/settings/rate_limit_on_issues_creation.md +++ b/doc/administration/settings/rate_limit_on_issues_creation.md @@ -18,7 +18,7 @@ action blocks requests that exceed a rate of 300 per minute. Access to the endpo To limit the number of requests made to the issue and epic creation endpoints: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Issues Rate Limits**. diff --git a/doc/administration/settings/rate_limit_on_notes_creation.md b/doc/administration/settings/rate_limit_on_notes_creation.md index 59548836e78..3bd6eee8303 100644 --- a/doc/administration/settings/rate_limit_on_notes_creation.md +++ b/doc/administration/settings/rate_limit_on_notes_creation.md @@ -13,7 +13,7 @@ You can configure the per-user rate limit for requests to the note creation endp To change the note creation rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Notes rate limit**. diff --git a/doc/administration/settings/rate_limit_on_pipelines_creation.md b/doc/administration/settings/rate_limit_on_pipelines_creation.md index 19e1410ef73..d083acb190e 100644 --- a/doc/administration/settings/rate_limit_on_pipelines_creation.md +++ b/doc/administration/settings/rate_limit_on_pipelines_creation.md @@ -26,7 +26,7 @@ Requests that exceed the limit are logged in the `application_json.log` file. To limit the number of pipeline requests: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Pipelines Rate Limits**. diff --git a/doc/administration/settings/rate_limit_on_projects_api.md b/doc/administration/settings/rate_limit_on_projects_api.md index 2192e4355c0..33304e4f088 100644 --- a/doc/administration/settings/rate_limit_on_projects_api.md +++ b/doc/administration/settings/rate_limit_on_projects_api.md @@ -16,7 +16,7 @@ You can configure the rate limit per IP address for unauthenticated requests to To change the rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Projects API rate limit**. diff --git a/doc/administration/settings/rate_limit_on_users_api.md b/doc/administration/settings/rate_limit_on_users_api.md index 9424e508d86..3669d79a953 100644 --- a/doc/administration/settings/rate_limit_on_users_api.md +++ b/doc/administration/settings/rate_limit_on_users_api.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -13,7 +13,7 @@ You can configure the per user rate limit for requests to [Users API](../../api/ To change the rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Users API rate limit**. diff --git a/doc/administration/settings/rate_limits_on_git_ssh_operations.md b/doc/administration/settings/rate_limits_on_git_ssh_operations.md index 64acb15b8ac..cb0d5e2a136 100644 --- a/doc/administration/settings/rate_limits_on_git_ssh_operations.md +++ b/doc/administration/settings/rate_limits_on_git_ssh_operations.md @@ -24,6 +24,8 @@ Users on self-managed GitLab can disable this rate limit. ## Configure GitLab Shell operation limit +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123761) in GitLab 16.2. + `Git operations using SSH` is enabled by default. Defaults to 600 per user per minute. 1. On the left sidebar, select **Your work > Admin Area**. diff --git a/doc/administration/settings/rate_limits_on_raw_endpoints.md b/doc/administration/settings/rate_limits_on_raw_endpoints.md index 78e65f7ba7b..237b57f4436 100644 --- a/doc/administration/settings/rate_limits_on_raw_endpoints.md +++ b/doc/administration/settings/rate_limits_on_raw_endpoints.md @@ -11,7 +11,7 @@ type: reference This setting defaults to `300` requests per minute, and allows you to rate limit the requests to raw endpoints: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **Performance optimization**. diff --git a/doc/administration/settings/scim_setup.md b/doc/administration/settings/scim_setup.md index 6a02a5b832c..6028abf6eab 100644 --- a/doc/administration/settings/scim_setup.md +++ b/doc/administration/settings/scim_setup.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -13,6 +13,7 @@ You can use the open standard System for Cross-domain Identity Management (SCIM) - Create users. - Block users. +- Re-add users (reactivate SCIM identity). The [internal GitLab SCIM API](../../development/internal_api/index.md#instance-scim-api) implements part of [the RFC7644 protocol](https://www.rfc-editor.org/rfc/rfc7644). @@ -26,7 +27,7 @@ Prerequisites: To configure GitLab SCIM: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **SCIM Token** section and select **Generate a SCIM token**. @@ -41,3 +42,14 @@ the GitLab instance, while the SCIM identity remains linked to the GitLab user. To update the user SCIM identity, use the [internal GitLab SCIM API](../../development/internal_api/index.md#update-a-single-scim-provisioned-user-1). + +### Reactivate access + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/379149) in GitLab 16.0 [with a flag](../feature_flags.md) named `skip_saml_identity_destroy_during_scim_deprovision`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121226) in GitLab 16.4. Feature flag `skip_saml_identity_destroy_during_scim_deprovision` removed. + +After a user is removed or deactivated through SCIM, you can reactivate that user by +adding them to the SCIM identity provider. + +After the identity provider performs a sync based on its configured schedule, +the user's SCIM identity is reactivated and their GitLab instance access is restored. diff --git a/doc/administration/settings/security_and_compliance.md b/doc/administration/settings/security_and_compliance.md index 2237866ad9c..78923b19b04 100644 --- a/doc/administration/settings/security_and_compliance.md +++ b/doc/administration/settings/security_and_compliance.md @@ -11,13 +11,9 @@ The settings for package metadata synchronization are located in the [Admin Area ## Choose package registry metadata to sync -WARNING: -The full package metadata sync can add up to 30 GB to the PostgreSQL database. Ensure you have provisioned enough disk space for the database before enabling this feature. -We are actively working on reducing this data size in [epic 10415](https://gitlab.com/groups/gitlab-org/-/epics/10415). +To choose the packages you want to synchronize with the GitLab Package Metadata Database for [License Compliance](../../user/compliance/license_scanning_of_cyclonedx_files/index.md) and [Continuous Vulnerability Scanning](../../user/application_security/continuous_vulnerability_scanning/index.md): -To choose the packages you want to synchronize with the GitLab License Database for [License Compliance](../../user/compliance/license_scanning_of_cyclonedx_files/index.md): - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Security and Compliance**. 1. Expand **License Compliance**. diff --git a/doc/administration/settings/sidekiq_job_limits.md b/doc/administration/settings/sidekiq_job_limits.md index d5cd24c5237..6c5ff29e96b 100644 --- a/doc/administration/settings/sidekiq_job_limits.md +++ b/doc/administration/settings/sidekiq_job_limits.md @@ -17,7 +17,7 @@ Redis. To avoid excessive memory for Redis, we: To access Sidekiq job size limits: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Sidekiq job size limits**. diff --git a/doc/administration/settings/sign_in_restrictions.md b/doc/administration/settings/sign_in_restrictions.md index 393c7dd6aeb..6d38610192b 100644 --- a/doc/administration/settings/sign_in_restrictions.md +++ b/doc/administration/settings/sign_in_restrictions.md @@ -12,7 +12,7 @@ You can use **Sign-in restrictions** to customize authentication restrictions fo To access sign-in restriction settings: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Sign-in restrictions** section. @@ -76,7 +76,7 @@ Open the [Rails console](../operations/rails_console.md) and run the following: To enable Admin Mode through the UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-in restrictions**. @@ -86,7 +86,7 @@ To enable Admin Mode through the UI: To turn on Admin Mode for your current session and access potentially dangerous resources: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Enter Admin Mode**. 1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). @@ -104,7 +104,7 @@ authentication are supported by Admin Mode. Admin Mode status is stored in the c To turn off Admin Mode for your current session: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Leave Admin Mode**. ### Limitations of Admin Mode @@ -181,7 +181,7 @@ For example, if you include the following information in the noted text box: To access this text box: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Sign-in restrictions** section. diff --git a/doc/administration/settings/sign_up_restrictions.md b/doc/administration/settings/sign_up_restrictions.md index f255e15c1be..da40a2bab05 100644 --- a/doc/administration/settings/sign_up_restrictions.md +++ b/doc/administration/settings/sign_up_restrictions.md @@ -22,7 +22,7 @@ you do not expect public users to sign up for an account. To disable sign ups: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -40,7 +40,7 @@ enabled by default for new GitLab instances. It is only applicable if sign ups a To require administrator approval for new sign ups: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -66,7 +66,7 @@ their email address before they are allowed to sign in. To enforce confirmation of the email address used for new sign ups: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -107,7 +107,7 @@ Prerequisite: To set a user cap: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -127,7 +127,7 @@ Prerequisite: To remove the user cap: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -153,7 +153,7 @@ You can add additional complexity requirements. Changes to password complexity r Existing passwords are unaffected. To change password complexity requirements: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. @@ -183,7 +183,7 @@ reduce the risk of malicious users creating spam accounts with disposable email To create an email domain allowlist or denylist: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Sign-up restrictions**. diff --git a/doc/administration/settings/slack_app.md b/doc/administration/settings/slack_app.md index a8e672bc8fa..ef756dfeff7 100644 --- a/doc/administration/settings/slack_app.md +++ b/doc/administration/settings/slack_app.md @@ -26,7 +26,7 @@ To create a GitLab for Slack app: - **In GitLab**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Slack app**. @@ -46,7 +46,7 @@ You're then redirected to Slack for the next steps. After you've [created a GitLab for Slack app](#create-a-gitlab-for-slack-app), you can configure the settings in GitLab: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Slack app**. @@ -80,7 +80,7 @@ To update your copy of the GitLab for Slack app: - **In GitLab**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Slack app**. diff --git a/doc/administration/settings/terms.md b/doc/administration/settings/terms.md index 4b4972acc8e..12678b6416a 100644 --- a/doc/administration/settings/terms.md +++ b/doc/administration/settings/terms.md @@ -17,7 +17,7 @@ for example `https://gitlab.example.com/-/users/terms`. To enforce acceptance of a Terms of Service and Privacy Policy: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Terms of Service and Privacy Policy** section. diff --git a/doc/administration/settings/terraform_limits.md b/doc/administration/settings/terraform_limits.md index 5ba8bfe63e0..7f744b0a4e5 100644 --- a/doc/administration/settings/terraform_limits.md +++ b/doc/administration/settings/terraform_limits.md @@ -15,7 +15,7 @@ state file version, and is checked whenever a new version is created. To add a storage limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Preferences**. 1. Expand **Terraform limits**. diff --git a/doc/administration/settings/third_party_offers.md b/doc/administration/settings/third_party_offers.md index 39e2275f411..b91f13a6a30 100644 --- a/doc/administration/settings/third_party_offers.md +++ b/doc/administration/settings/third_party_offers.md @@ -18,7 +18,7 @@ questions when creating a group. To toggle the display of customer experience improvement content and third-party offers: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Customer experience improvement and third-party offers**. diff --git a/doc/administration/settings/usage_statistics.md b/doc/administration/settings/usage_statistics.md index f77298dd038..b8b87f42475 100644 --- a/doc/administration/settings/usage_statistics.md +++ b/doc/administration/settings/usage_statistics.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -64,7 +64,7 @@ Registration is not yet required for participation, but may be added in a future ### Enable registration features 1. Sign in as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. @@ -122,7 +122,7 @@ If your GitLab instance is behind a proxy, set the appropriate To enable or disable Service Ping and version check: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Metrics and profiling**. 1. Expand **Usage statistics**. @@ -184,10 +184,9 @@ the Admin Area. You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To view the payload: 1. Sign in as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. -1. Select **Settings > Metrics and profiling**. -1. Expand the **Usage statistics** section. +1. Select **Settings > Service usage data**. 1. Select **Preview payload**. For an example payload, see [Example Service Ping payload](../../development/internal_analytics/service_ping/index.md#example-service-ping-payload). @@ -203,9 +202,9 @@ or if the Service Ping [cron job](../../development/internal_analytics/service_p To upload the payload manually: 1. Sign in as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. -1. Select **Settings > Service** usage data. +1. Select **Settings > Service usage data**. 1. Select **Download payload**. 1. Save the JSON file. 1. Visit [Service usage data center](https://version.gitlab.com/usage_data/new). diff --git a/doc/administration/settings/user_and_ip_rate_limits.md b/doc/administration/settings/user_and_ip_rate_limits.md index 822ba4dd03e..09ddb784191 100644 --- a/doc/administration/settings/user_and_ip_rate_limits.md +++ b/doc/administration/settings/user_and_ip_rate_limits.md @@ -31,7 +31,7 @@ counted as web traffic. To enable the unauthenticated request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **User and IP rate limits**. @@ -46,7 +46,7 @@ To enable the unauthenticated request rate limit: To enable the unauthenticated request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **User and IP rate limits**. @@ -61,7 +61,7 @@ To enable the unauthenticated request rate limit: To enable the authenticated API request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **User and IP rate limits**. @@ -76,7 +76,7 @@ To enable the authenticated API request rate limit: To enable the unauthenticated request rate limit: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **User and IP rate limits**. @@ -96,7 +96,7 @@ plain-text body, which by default is `Retry later`. To use a custom response: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Network**. 1. Expand **User and IP rate limits**. diff --git a/doc/administration/settings/visibility_and_access_controls.md b/doc/administration/settings/visibility_and_access_controls.md index 5e1e35db244..93dbfaaf990 100644 --- a/doc/administration/settings/visibility_and_access_controls.md +++ b/doc/administration/settings/visibility_and_access_controls.md @@ -13,7 +13,7 @@ specific controls on branches, projects, snippets, groups, and more. To access the visibility and access control options: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -25,7 +25,7 @@ Instance-level protections for project creation define which roles can on the instance. To alter which roles have permission to create projects: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -42,7 +42,7 @@ on the instance. To alter which roles have permission to create projects: By default both administrators and anyone with the **Owner** role can delete a project. To restrict project deletion to only administrators: 1. Sign in to GitLab as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -80,7 +80,7 @@ then it gets automatically changed to `1` while also disabling deletion protecti To configure delayed project deletion: 1. Sign in to GitLab as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -121,7 +121,7 @@ Alternatively, projects that are marked for removal can be deleted immediately. To set the default [visibility levels for new projects](../../user/public_access.md): 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -132,12 +132,15 @@ To set the default [visibility levels for new projects](../../user/public_access - **Public** - The project can be accessed without any authentication. 1. Select **Save changes**. +For more details on project visibility, see +[Project visibility](../../user/public_access.md). + ## Configure snippet visibility defaults To set the default visibility levels for new [snippets](../../user/snippets.md): 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -145,14 +148,14 @@ To set the default visibility levels for new [snippets](../../user/snippets.md): 1. Select **Save changes**. For more details on snippet visibility, read -[Project visibility](../../user/public_access.md). +[Snippet visibility](../../user/snippets.md). ## Configure group visibility defaults To set the default visibility levels for new groups: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -167,6 +170,9 @@ For more details on group visibility, see ## Restrict visibility levels +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124649) in GitLab 16.3 to prevent restricting default project and group visibility, [with a flag](../feature_flags.md) named `prevent_visibility_restriction`. Disabled by default. +> - `prevent_visibility_restriction` [enabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131203) by default in GitLab 16.4. + When restricting visibility levels, consider how these restrictions interact with permissions for subgroups and projects that inherit their visibility from the item you're changing. @@ -174,7 +180,7 @@ the item you're changing. To restrict visibility levels for groups, projects, snippets, and selected pages: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -191,56 +197,8 @@ To restrict visibility levels for groups, projects, snippets, and selected pages - Only administrators are able to create private groups, projects, and snippets. 1. Select **Save changes**. -For more details on project visibility, see -[Project visibility](../../user/public_access.md). - -## Configure allowed import sources - -Before you can import projects from other systems, you must enable the -[import source](../../user/gitlab_com/index.md#default-import-sources) for that system. - -1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand the **Visibility and access controls** section. -1. Select each of **Import sources** to allow. -1. Select **Save changes**. - -## Enable project export - -To enable the export of -[projects and their data](../../user/project/settings/import_export.md#export-a-project-and-its-data): - -1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand the **Visibility and access controls** section. -1. Scroll to **Project export**. -1. Select the **Enabled** checkbox. -1. Select **Save changes**. - -## Enable migration of groups and projects by direct transfer - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. - -You can enable migration of groups by direct transfer using the UI. - -To enable migration of groups by direct transfer: - -1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. Select **Settings > General**. -1. Expand the **Visibility and access controls** section. -1. Scroll to **Allow migrating GitLab groups and projects by direct transfer**. -1. Select the **Enabled** checkbox. -1. Select **Save changes**. - -The same setting -[is available](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) in the API as the -`bulk_import_enabled` attribute. +NOTE: +You cannot select the restricted default visibility level for new projects and groups. ## Configure enabled Git access protocols @@ -252,7 +210,7 @@ The GitLab restrictions apply at the application level. To specify the enabled Git access protocols: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -339,7 +297,7 @@ include the `10.0.0.0/24` range. To add a IP address range to the group-level allowlist: 1. Sign in to GitLab as a user with Administrator access level. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Visibility and access controls** section. diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md index d85eae7a7f6..89de4cd6cf3 100644 --- a/doc/administration/sidekiq/extra_sidekiq_processes.md +++ b/doc/administration/sidekiq/extra_sidekiq_processes.md @@ -48,7 +48,7 @@ to all available queues: To view the Sidekiq processes in GitLab: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md index 89da174eb34..10a1faea850 100644 --- a/doc/administration/sidekiq/sidekiq_job_migration.md +++ b/doc/administration/sidekiq/sidekiq_job_migration.md @@ -26,7 +26,7 @@ If the Sidekiq routing rules are changed, administrators need to take care with Step 4 involves rewriting some Sidekiq job data for jobs that are already stored in Redis, but due to run in future. There are two sets of jobs to run in future: scheduled jobs and jobs to be retried. We provide a separate Rake task to migrate each set: - `gitlab:sidekiq:migrate_jobs:retry` for jobs to be retried. -- `gitlab:sidekiq:migrate_jobs:scheduled` for scheduled jobs. +- `gitlab:sidekiq:migrate_jobs:schedule` for scheduled jobs. Queued jobs that are yet to be run can also be migrated with a Rake task ([available in GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101348) and later): diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md index 005add1d2f4..379b00536f3 100644 --- a/doc/administration/silent_mode/index.md +++ b/doc/administration/silent_mode/index.md @@ -4,11 +4,12 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Silent Mode (Experiment) **(FREE SELF)** +# GitLab Silent Mode **(FREE SELF EXPERIMENT)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9826) in GitLab 15.11. This feature is an [Experiment](../../policy/experiment-beta-support.md#experiment). +> - Enabling and disabling Silent Mode through the web UI was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131090) in GitLab 16.4 -Silent Mode allows you to suppress outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are: +Silent Mode allows you to silence outbound communication, such as emails, from GitLab. Silent Mode is not intended to be used on environments which are in-use. Two use-cases are: - Validating Geo site promotion. You have a secondary Geo site as part of your [disaster recovery](../geo/disaster_recovery/index.md) solution. You want to regularly test promoting it to become a primary Geo site, as a best practice to ensure your disaster recovery plan actually works. But you don't want to actually perform an entire failover, since the primary site lives in a region which provides the lowest latency to your users. And you don't want to take downtime during every regular test. So, you let the primary site remain up, while you promote the secondary site. You start smoke testing the promoted site. But, the promoted site starts emailing users, the push mirrors push changes to external Git repositories, etc. This is where Silent Mode comes in. You can enable it as part of site promotion, to avoid this issue. - Validating GitLab backups. You set up a testing instance to test that your backups restore successfully. As part of the restore, you enable Silent Mode, for example to avoid sending invalid emails to users. @@ -19,7 +20,15 @@ Prerequisites: - You must have administrator access. -There are two ways to enable Silent Mode: +There are multiple ways to enable Silent Mode: + +- **Web UI** + + 1. On the left sidebar, select **Search or go to**. + 1. Select **Admin Area**. + 1. On the left sidebar, select **Settings > General**. + 1. Expand **Silent Mode**, and toggle **Enable Silent Mode**. + 1. Changes are saved immediately. - [**API**](../../api/settings.md): @@ -41,7 +50,15 @@ Prerequisites: - You must have administrator access. -There are two ways to disable Silent Mode: +There are multiple ways to disable Silent Mode: + +- **Web UI** + + 1. On the left sidebar, select **Search or go to**. + 1. Select **Admin Area**. + 1. On the left sidebar, select **Settings > General**. + 1. Expand **Silent Mode**, and toggle **Enable Silent Mode**. + 1. Changes are saved immediately. - [**API**](../../api/settings.md): @@ -61,32 +78,32 @@ It may take up to a minute to take effect. [Issue 405433](https://gitlab.com/git This section documents the current behavior of GitLab when Silent Mode is enabled. While Silent Mode is an Experiment, the behavior may change without notice. The work for the first iteration of Silent Mode is tracked by [Epic 9826](https://gitlab.com/groups/gitlab-org/-/epics/9826). -### Service Desk - -Incoming emails still raise issues, but the users who sent the emails to [Service Desk](../../user/project/service_desk/index.md) are not notified of issue creation or comments on their issues. - -### Webhooks - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393639) in GitLab 16.3. - -[Project and group webhooks](../../user/project/integrations/webhooks.md) and [system hooks](../system_hooks.md) are suppressed. - -In GitLab 16.2 and earlier, webhooks were triggered when Silent Mode was enabled, but the [webhook HTTP request was blocked](#outbound-http-requests). - -Triggering webhook tests via the UI results in HTTP status 500 responses. - -### Remote mirrors - -Updates on [remote mirrors](../../user/project/repository/mirror/index.md) (pushing to, and pulling from them) are suppressed. +When Silent Mode is enabled, a banner is displayed at the top of the page for all users stating the setting is enabled and **All outbound communications are blocked.**. -### Integrations +### Outbound communications that are silenced -Executable [integrations](../../user/project/integrations/index.md) are suppressed. +Outbound communications from the following features are silenced by Silent Mode. -### Outbound emails +| Feature | Notes | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Project and group webhooks](../../user/project/integrations/webhooks.md) | Triggering webhook tests via the UI results in HTTP status 500 responses. | +| [System hooks](../system_hooks.md) | | +| [Remote mirrors](../../user/project/repository/mirror/index.md) | Pushes to remote mirrors are skipped. Pulls from remote mirrors is skipped. | +| [Executable integrations](../../user/project/integrations/index.md) | The integrations are not executed. | +| [Service Desk](../../user/project/service_desk/index.md) | Incoming emails still raise issues, but the users who sent the emails to Service Desk are not notified of issue creation or comments on their issues. | +| Outbound emails | | +| Outbound HTTP requests | Many HTTP requests are blocked where features are not blocked or skipped explicitly. These may produce errors. If a particular error is problematic for testing during Silent Mode, please consult [GitLab Support](https://about.gitlab.com/support/). | -Outbound emails are suppressed. +### Outbound communications that are not silenced -### Outbound HTTP requests +Outbound communications from the following features are not silenced by Silent Mode. -Many outbound HTTP requests are suppressed. A list of unsuppressed requests does not exist at this time, since more suppression is planned. +| Feature | Notes | +| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Dependency proxy](../packages/dependency_proxy.md) | Pulling images that are not cached will fetch from the source as usual. Consider pull rate limits. | +| [File hooks](../file_hooks.md) | | +| [Server hooks](../server_hooks.md) | | +| [Advanced search](../../integration/advanced_search/elasticsearch.md) | If two GitLab instances are using the same Advanced Search instance, then they can both modify Search data. This is a split-brain scenario which can occur for example after promoting a secondary Geo site while the primary Geo site is live. | +| [Snowplow](../../user/product_analytics/index.md) | There is [a proposal to silence these requests](https://gitlab.com/gitlab-org/gitlab/-/issues/409661). | +| [Deprecated Kubernetes Connections](../../user/clusters/agent/index.md) | There is [a proposal to silence these requests](https://gitlab.com/gitlab-org/gitlab/-/issues/396470). | +| [Container registry webhooks](../packages/container_registry.md#configure-container-registry-notifications) | There is [a proposal to silence these requests](https://gitlab.com/gitlab-org/gitlab/-/issues/409682). | diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 6b232ddc25f..203f59b60f7 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -16,7 +16,7 @@ storage such as a content delivery network (CDN). To configure external storage for static objects: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand the **External storage for repository static objects** section. diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md index 1c84d4fadb8..61f89bacd66 100644 --- a/doc/administration/system_hooks.md +++ b/doc/administration/system_hooks.md @@ -54,7 +54,7 @@ for Push and Tag events, but we never display commits. To create a system hook: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **System Hooks**. 1. Select **Add new webhook**. diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 53bde005232..e7965d451b6 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -18,14 +18,70 @@ If you're on a [paid tier](https://about.gitlab.com/pricing/) and aren't sure how to use these commands, [contact Support](https://about.gitlab.com/support/) for assistance with any issues you're having. +## Start a database console + +::Tabs + +:::TabTitle Linux package (Omnibus) + +Recommended for: + +- Single-node instances. +- Scaled out or hybrid environments, on the Patroni nodes, usually the leader. +- Scaled out or hybrid environments, on the server running the PostgreSQL service. + +```shell +sudo gitlab-psql +``` + +On a single-node instance, or a web or Sidekiq node you can also use the Rails database console, but +it takes longer to initialize: + +- In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/341210): + + ```shell + sudo gitlab-rails db-console --database main + ``` + +- In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails db-console + ``` + +:::TabTitle Docker + +```shell +docker exec -it <container-id> gitlab-psql +``` + +:::TabTitle Self-compiled (source) + +Use the `psql` command that's part of [your PostgreSQL installation](../../install/installation.md#7-database). + +```shell +sudo -u git -H psql -d gitlabhq_production +``` + +:::TabTitle Helm chart (Kubernetes) + +- If you run a hybrid environment, and PostgreSQL runs on a Linux packaged installation (Omnibus), + the recommended approach is to use the database console locally on those servers. Refer to the details + for Linux package. +- Use the console that's part of your external third-party PostgreSQL service. +- Run `gitlab-rails db-console` in the toolbox pod. + - Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. + +::EndTabs + +To exit the console, type: `quit`. + ## Other GitLab PostgreSQL documentation This section is for links to information elsewhere in the GitLab documentation. ### Procedures -- [Connect to the PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). - - [Database procedures for Linux package installations](https://docs.gitlab.com/omnibus/settings/database.html) including: - SSL: enabling, disabling, and verifying. - Enabling Write Ahead Log (WAL) archiving. diff --git a/doc/administration/user_cohorts.md b/doc/administration/user_cohorts.md index 6f2798f437c..c849d3115d2 100644 --- a/doc/administration/user_cohorts.md +++ b/doc/administration/user_cohorts.md @@ -34,7 +34,7 @@ How do we measure the activity of users? GitLab considers a user active if: To view user cohorts: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select the **Cohorts** tab. diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 43e88bf6eac..d1884bf179b 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -44,7 +44,7 @@ For self-compiled installations: Administrators can: -- Use the Admin Area to [prevent an existing user from creating top-level groups](../administration/admin_area.md#prevent-a-user-from-creating-groups). +- Use the Admin Area to [prevent an existing user from creating top-level groups](../administration/admin_area.md#prevent-a-user-from-creating-top-level-groups). - Use the [modify an existing user API endpoint](../api/users.md#user-modification) to change the `can_create_group` setting. ## Prevent users from changing their usernames diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index 8e1fc412a62..6d186ef0af5 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -31,7 +31,7 @@ To access the **What's new** feature: You can configure **What's new** to display features based on the tier, or you can hide it. To configure it: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **What's new**, and choose one of the following options: diff --git a/doc/api/access_requests.md b/doc/api/access_requests.md index 341cb768154..bd4c4df5660 100644 --- a/doc/api/access_requests.md +++ b/doc/api/access_requests.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 5b918fa50ab..9f2a57dd84f 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -34,7 +34,7 @@ The following API resources are available in the project context: | [Conan distributions](packages/conan.md) | `/projects/:id/packages/conan` (also available standalone) | | [Debian distributions](packages/debian_project_distributions.md) | `/projects/:id/debian_distributions` (also available for groups) | | [Debian packages](packages/debian.md) | `/projects/:id/packages/debian` (also available for groups) | -| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | +| [Dependencies](dependencies.md) **(ULTIMATE ALL)** | `/projects/:id/dependencies` | | [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | | [Deploy tokens](deploy_tokens.md) | `/projects/:id/deploy_tokens` (also available for groups and standalone) | | [Deployments](deployments.md) | `/projects/:id/deployments` | @@ -56,14 +56,14 @@ The following API resources are available in the project context: | [Issue links](issue_links.md) | `/projects/:id/issues/.../links` | | [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | | [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | +| [Iterations](iterations.md) **(PREMIUM ALL)** | `/projects/:id/iterations` (also available for groups) | | [Project CI/CD job token scope](project_job_token_scopes.md) | `/projects/:id/job_token_scope` | | [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | | [Jobs Artifacts](job_artifacts.md) | `/projects/:id/jobs/:job_id/artifacts` | | [Labels](labels.md) | `/projects/:id/labels` | | [Maven repository](packages/maven.md) | `/projects/:id/packages/maven` (also available for groups and standalone) | | [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge request approvals](merge_request_approvals.md) **(PREMIUM ALL)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | | [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | | [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | | [Metadata](metadata.md) | `/metadata` | @@ -82,7 +82,7 @@ The following API resources are available in the project context: | [Project milestones](milestones.md) | `/projects/:id/milestones` | | [Project snippets](project_snippets.md) | `/projects/:id/snippets` | | [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | +| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE ALL)** | `/projects/:id/vulnerabilities` | | [Project wikis](wikis.md) | `/projects/:id/wikis` | | [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | | [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | @@ -103,46 +103,47 @@ The following API resources are available in the project context: | [Tags](tags.md) | `/projects/:id/repository/tags` | | [Terraform modules](packages/terraform-modules.md) | `/projects/:id/packages/terraform/modules` (also available standalone) | | [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | -| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | -| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | -| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | -| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | +| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM ALL)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | +| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE ALL)** | `/vulnerabilities/:id` | +| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE ALL)** | `/projects/:id/vulnerability_exports` | +| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE ALL)** | `/projects/:id/vulnerability_findings` | ## Group resources The following API resources are available in the group context: -| Resource | Available endpoints | -|:-----------------------------------------------------------------|:--------------------| -| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | -| [Access tokens](group_access_tokens.md) | `/groups/:id/access_tokens` (also available for projects) | -| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | -| [Debian distributions](packages/debian_group_distributions.md) | `/groups/:id/-/packages/debian` (also available for projects) | -| [Deploy tokens](deploy_tokens.md) | `/groups/:id/deploy_tokens` (also available for projects and standalone) | -| [Discussions](discussions.md) (comments and threads) | `/groups/:id/epics/.../discussions` (also available for projects) | -| [Epic issues](epic_issues.md) **(PREMIUM)** | `/groups/:id/epics/.../issues` | -| [Epic links](epic_links.md) **(PREMIUM)** | `/groups/:id/epics/.../epics` | -| [Epics](epics.md) **(PREMIUM)** | `/groups/:id/epics` | -| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | -| [Group badges](group_badges.md) | `/groups/:id/badges` | -| [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group iterations](group_iterations.md) **(PREMIUM)** | `/groups/:id/iterations` (also available for projects) | -| [Group labels](group_labels.md) | `/groups/:id/labels` | -| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | -| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | -| [Group releases](group_releases.md) | `/groups/:id/releases`| -| [Group wikis](group_wikis.md) **(PREMIUM)** | `/groups/:id/wikis` | -| [Invitations](invitations.md) | `/groups/:id/invitations` (also available for projects) | -| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | -| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | -| [Linked epics](linked_epics.md) | `/groups/:id/epics/.../related_epics` | -| [Member Roles](member_roles.md) | `/groups/:id/member_roles` | -| [Members](members.md) | `/groups/:id/members` (also available for projects) | -| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | -| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | -| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | -| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | -| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | +| Resource | Available endpoints | +|:----------------------------------------------------------------------|:---------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | +| [Access tokens](group_access_tokens.md) | `/groups/:id/access_tokens` (also available for projects) | +| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | +| [Debian distributions](packages/debian_group_distributions.md) | `/groups/:id/-/packages/debian` (also available for projects) | +| [Deploy tokens](deploy_tokens.md) | `/groups/:id/deploy_tokens` (also available for projects and standalone) | +| [Discussions](discussions.md) (comments and threads) | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **(PREMIUM ALL)** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **(PREMIUM ALL)** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **(PREMIUM ALL)** | `/groups/:id/epics` | +| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | +| [Group badges](group_badges.md) | `/groups/:id/badges` | +| [Group issue boards](group_boards.md) | `/groups/:id/boards` | +| [Group iterations](group_iterations.md) **(PREMIUM ALL)** | `/groups/:id/iterations` (also available for projects) | +| [Group labels](group_labels.md) | `/groups/:id/labels` | +| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | +| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Group releases](group_releases.md) | `/groups/:id/releases` | +| [Group SSH certificates](group_ssh_certificates.md) **(PREMIUM ALL)** | `/groups/:id/ssh_certificates` | +| [Group wikis](group_wikis.md) **(PREMIUM ALL)** | `/groups/:id/wikis` | +| [Invitations](invitations.md) | `/groups/:id/invitations` (also available for projects) | +| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | +| [Linked epics](linked_epics.md) | `/groups/:id/epics/.../related_epics` | +| [Member Roles](member_roles.md) | `/groups/:id/member_roles` | +| [Members](members.md) | `/groups/:id/members` (also available for projects) | +| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | +| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | +| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | +| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | +| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | ## Standalone resources @@ -156,8 +157,9 @@ The following API resources are available outside of project and group contexts | [Avatar](avatar.md) | `/avatar` | | [Broadcast messages](broadcast_messages.md) | `/broadcast_messages` | | [Code snippets](snippets.md) | `/snippets` | -| [Code suggestions](code_suggestions.md) | `/code_suggestions` | +| [Code Suggestions](code_suggestions.md) | `/code_suggestions` | | [Custom attributes](custom_attributes.md) | `/users/:id/custom_attributes` (also available for groups and projects) | +| [Dependency list exports](dependency_list_export.md) **(ULTIMATE ALL)** | `/pipelines/:id/dependency_list_exports`, `/projects/:id/dependency_list_exports`, `/groups/:id/dependency_list_exports`, `/security/dependency_list_exports/:id`, `/security/dependency_list_exports/:id/download` | | [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | | [Deploy tokens](deploy_tokens.md) | `/deploy_tokens` (also available for projects and groups) | | [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | diff --git a/doc/api/appearance.md b/doc/api/appearance.md index 94fb092c739..4018e787acf 100644 --- a/doc/api/appearance.md +++ b/doc/api/appearance.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/applications.md b/doc/api/applications.md index f597e1acc44..9875dfab96d 100644 --- a/doc/api/applications.md +++ b/doc/api/applications.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index 714c79c42c5..0e60fdbcd0a 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -11,8 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Instance Audit Events **(PREMIUM SELF)** -The Audit Events API allows you to retrieve [instance audit events](../administration/audit_events.md#instance-events). -This API cannot retrieve group or project audit events. +Use this API to retrieve instance audit events. To retrieve audit events using the API, you must [authenticate yourself](rest/index.md#authentication) as an Administrator. @@ -165,8 +164,7 @@ Example response: > Support for keyset pagination [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333968) in GitLab 15.2. -The Group Audit Events API allows you to retrieve [group audit events](../administration/audit_events.md#group-events). -This API cannot retrieve project audit events. +Use this API to retrieve group audit events. A user with: @@ -283,7 +281,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219238) in GitLab 13.1. -The Project Audit Events API allows you to retrieve [project audit events](../administration/audit_events.md#project-events). +Use this API to retrieve project audit events. A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. diff --git a/doc/api/avatar.md b/doc/api/avatar.md index 68031a05f43..98e644d9ae9 100644 --- a/doc/api/avatar.md +++ b/doc/api/avatar.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/award_emoji.md b/doc/api/award_emoji.md index 2591c6ea490..1ccc59601a0 100644 --- a/doc/api/award_emoji.md +++ b/doc/api/award_emoji.md @@ -13,7 +13,7 @@ An [emoji reaction](../user/award_emojis.md) tells a thousand words. We call GitLab objects on which you can react with an emoji "awardables". You can react with emoji on the following: -- [Epics](../user/group/epics/index.md) ([API](epics.md)). **(PREMIUM)** +- [Epics](../user/group/epics/index.md) ([API](epics.md)). **(PREMIUM ALL)** - [Issues](../user/project/issues/index.md) ([API](issues.md)). - [Merge requests](../user/project/merge_requests/index.md) ([API](merge_requests.md)). - [Snippets](../user/snippets.md) ([API](snippets.md)). diff --git a/doc/api/boards.md b/doc/api/boards.md index 2438508f2f9..34aa7d6e68a 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -228,10 +228,10 @@ PUT /projects/:id/boards/:board_id | `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `board_id` | integer | yes | The ID of a board | | `name` | string | no | The new name of the board | -| `assignee_id` **(PREMIUM)** | integer | no | The assignee the board should be scoped to | -| `milestone_id` **(PREMIUM)** | integer | no | The milestone the board should be scoped to | -| `labels` **(PREMIUM)** | string | no | Comma-separated list of label names which the board should be scoped to | -| `weight` **(PREMIUM)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | +| `assignee_id` **(PREMIUM ALL)** | integer | no | The assignee the board should be scoped to | +| `milestone_id` **(PREMIUM ALL)** | integer | no | The milestone the board should be scoped to | +| `labels` **(PREMIUM ALL)** | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` **(PREMIUM ALL)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/boards/1?name=new_name&milestone_id=43&assignee_id=1&labels=Doing&weight=4" @@ -421,8 +421,8 @@ POST /projects/:id/boards/:board_id/lists | `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `board_id` | integer | yes | The ID of a board | | `label_id` | integer | no | The ID of a label | -| `assignee_id` **(PREMIUM)** | integer | no | The ID of a user | -| `milestone_id` **(PREMIUM)** | integer | no | The ID of a milestone | +| `assignee_id` **(PREMIUM ALL)** | integer | no | The ID of a user | +| `milestone_id` **(PREMIUM ALL)** | integer | no | The ID of a milestone | NOTE: Label, assignee and milestone arguments are mutually exclusive, diff --git a/doc/api/branches.md b/doc/api/branches.md index b925d3ddadf..cd0266a05f1 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -50,19 +50,22 @@ Example response: "can_push": true, "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/main", "commit": { - "author_email": "john@example.com", - "author_name": "John Smith", - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00", - "committer_email": "john@example.com", - "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", "short_id": "7b5c3cc", - "title": "add projects API", - "message": "add projects API", + "created_at": "2012-06-28T03:44:20-07:00", "parent_ids": [ "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - ] + ], + "title": "add projects API", + "message": "add projects API", + "author_name": "John Smith", + "author_email": "john@example.com", + "authored_date": "2012-06-27T05:51:39-07:00", + "committer_name": "John Smith", + "committer_email": "john@example.com", + "committed_date": "2012-06-28T03:44:20-07:00", + "trailers": {}, + "web_url": "https://gitlab.example.com/my-group/my-project/-/commit/7b5c3cc8be40ee161ae89a06bba6229da1032a0c" } }, ... @@ -97,7 +100,7 @@ Example response: ```json { - "name": "master", + "name": "main", "merged": false, "protected": true, "default": true, @@ -106,19 +109,22 @@ Example response: "can_push": true, "web_url": "https://gitlab.example.com/my-group/my-project/-/tree/main", "commit": { - "author_email": "john@example.com", - "author_name": "John Smith", - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00", - "committer_email": "john@example.com", - "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", "short_id": "7b5c3cc", - "title": "add projects API", - "message": "add projects API", + "created_at": "2012-06-28T03:44:20-07:00", "parent_ids": [ "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - ] + ], + "title": "add projects API", + "message": "add projects API", + "author_name": "John Smith", + "author_email": "john@example.com", + "authored_date": "2012-06-27T05:51:39-07:00", + "committer_name": "John Smith", + "committer_email": "john@example.com", + "committed_date": "2012-06-28T03:44:20-07:00", + "trailers": {}, + "web_url": "https://gitlab.example.com/my-group/my-project/-/commit/7b5c3cc8be40ee161ae89a06bba6229da1032a0c" } } ``` @@ -160,19 +166,22 @@ Example response: ```json { "commit": { - "author_email": "john@example.com", - "author_name": "John Smith", - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00", - "committer_email": "john@example.com", - "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", "short_id": "7b5c3cc", - "title": "add projects API", - "message": "add projects API", + "created_at": "2012-06-28T03:44:20-07:00", "parent_ids": [ "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - ] + ], + "title": "add projects API", + "message": "add projects API", + "author_name": "John Smith", + "author_email": "john@example.com", + "authored_date": "2012-06-27T05:51:39-07:00", + "committer_name": "John Smith", + "committer_email": "john@example.com", + "committed_date": "2012-06-28T03:44:20-07:00", + "trailers": {}, + "web_url": "https://gitlab.example.com/my-group/my-project/-/commit/7b5c3cc8be40ee161ae89a06bba6229da1032a0c" }, "name": "newbranch", "merged": false, diff --git a/doc/api/bulk_imports.md b/doc/api/bulk_imports.md index ec1a7d44253..db508d1edfa 100644 --- a/doc/api/bulk_imports.md +++ b/doc/api/bulk_imports.md @@ -42,19 +42,19 @@ POST /bulk_imports | `configuration[access_token]` | String | yes | Access token to the source GitLab instance. | | `entities` | Array | yes | List of entities to import. | | `entities[source_type]` | String | yes | Source entity type. Valid values are `group_entity` (GitLab 14.2 and later) and `project_entity` (GitLab 15.11 and later). | -| `entities[source_full_path]` | String | yes | Source full path of the entity to import. | -| `entities[destination_slug]` | String | yes | Destination slug for the entity. | +| `entities[source_full_path]` | String | yes | Source full path of the entity to import. For example, `gitlab-org/gitlab`. | +| `entities[destination_slug]` | String | yes | Destination slug for the entity. GitLab uses the slug as the URL path to the entity. The name of the imported entity is copied from the name of the source entity and not the slug. | | `entities[destination_name]` | String | no | Deprecated: Use `destination_slug` instead. Destination slug for the entity. | -| `entities[destination_namespace]` | String | yes | Destination namespace for the entity. | +| `entities[destination_namespace]` | String | yes | Full path of the destination group [namespace](../user/namespace/index.md) for the entity. Must be an existing group in the destination instance. | | `entities[migrate_projects]` | Boolean | no | Also import all nested projects of the group (if `source_type` is `group_entity`). Defaults to `true`. | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/bulk_imports" \ +curl --request POST --header "PRIVATE-TOKEN: <your_access_token_for_destination_gitlab_instance>" "https://destination-gitlab-instance.example.com/api/v4/bulk_imports" \ --header "Content-Type: application/json" \ --data '{ "configuration": { - "url": "http://gitlab.example/", - "access_token": "access_token" + "url": "https://source-gitlab-instance.example.com", + "access_token": "<your_access_token_for_source_gitlab_instance>" }, "entities": [ { @@ -250,7 +250,7 @@ curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab ```json { - "id": 1, + "id": 2, "status": "finished", "source_type": "gitlab", "created_at": "2021-06-18T09:45:55.358Z", diff --git a/doc/api/code_suggestions.md b/doc/api/code_suggestions.md index 528f7db067b..d9fc4de3c8e 100644 --- a/doc/api/code_suggestions.md +++ b/doc/api/code_suggestions.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: AI assisted +stage: Create +group: Code Creation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -19,7 +19,7 @@ POST /code_suggestions/tokens ``` ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/code_suggestions/tokens" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/code_suggestions/tokens" ``` Example response: @@ -32,7 +32,7 @@ Example response: } ``` -## Generate code completions (Experiment) +## Generate code completions **(EXPERIMENT)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415581) in GitLab 16.2 [with a flag](../administration/feature_flags.md) named `code_suggestions_completion_api`. Disabled by default. This feature is an Experiment. > - Requirement to generate a JWT before calling this endpoint was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127863) in GitLab 16.3. @@ -51,7 +51,7 @@ POST /code_suggestions/completions Requests to this endpoint are proxied directly to the [model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#completions). The documentation for the endpoint is currently the SSoT for named parameters. ```shell -curl --header "Authorization: Bearer <YOUR_ACCESS_TOKEN>" --data "<JSON_BODY>" https://gitlab.example.com/api/v4/code_suggestions/completions +curl --request POST --header "Authorization: Bearer <YOUR_ACCESS_TOKEN>" --data "<JSON_BODY>" https://gitlab.example.com/api/v4/code_suggestions/completions ``` Example body: diff --git a/doc/api/commits.md b/doc/api/commits.md index cd955717e39..68c453aa317 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -120,7 +120,7 @@ POST /projects/:id/repository/commits ```shell PAYLOAD=$(cat << 'JSON' { - "branch": "master", + "branch": "main", "commit_message": "some commit message", "actions": [ { @@ -188,9 +188,9 @@ GitLab supports [form encoding](rest/index.md#encoding-api-parameters-of-array-a ```shell curl --request POST \ - --form "branch=master" \ + --form "branch=main" \ --form "commit_message=some commit message" \ - --form "start_branch=master" \ + --form "start_branch=main" \ --form "actions[][action]=create" \ --form "actions[][file_path]=foo/bar" \ --form "actions[][content]=</path/to/local.file" \ @@ -227,7 +227,7 @@ Parameters: | `stats` | boolean | no | Include commit stats. Default is true | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/main" ``` Example response: @@ -250,7 +250,7 @@ Example response: ], "last_pipeline" : { "id": 8, - "ref": "master", + "ref": "main", "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0", "status": "created" }, @@ -317,7 +317,7 @@ Parameters: ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --form "branch=master" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master/cherry_pick" + --form "branch=main" "https://gitlab.example.com/api/v4/projects/5/repository/commits/main/cherry_pick" ``` Example response: @@ -388,7 +388,7 @@ Parameters: | `dry_run` | boolean | no | Does not commit any changes. Default is false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231032) in GitLab 13.3 | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "branch=master" \ +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "branch=main" \ "https://gitlab.example.com/api/v4/projects/5/repository/commits/a738f717824ff53aebad8b090c1b79a14f2bd9e8/revert" ``` @@ -454,7 +454,7 @@ Parameters: | `sha` | string | yes | The commit hash or name of a repository branch or tag | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master/diff" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/main/diff" ``` Example response: @@ -462,7 +462,7 @@ Example response: ```json [ { - "diff": "--- a/doc/update/5.4-to-6.0.md\n+++ b/doc/update/5.4-to-6.0.md\n@@ -71,6 +71,8 @@\n sudo -u git -H bundle exec rake migrate_keys RAILS_ENV=production\n sudo -u git -H bundle exec rake migrate_inline_notes RAILS_ENV=production\n \n+sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production\n+\n ```\n \n ### 6. Update config files", + "diff": "@@ -71,6 +71,8 @@\n sudo -u git -H bundle exec rake migrate_keys RAILS_ENV=production\n sudo -u git -H bundle exec rake migrate_inline_notes RAILS_ENV=production\n \n+sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production\n+\n ```\n \n ### 6. Update config files", "new_path": "doc/update/5.4-to-6.0.md", "old_path": "doc/update/5.4-to-6.0.md", "a_mode": null, @@ -490,7 +490,7 @@ Parameters: | `sha` | string | yes | The commit hash or name of a repository branch or tag | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master/comments" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits/main/comments" ``` Example response: @@ -679,7 +679,7 @@ Example response: "target_url" : "https://gitlab.example.com/janedoe/gitlab-foss/builds/91", "finished_at" : null, "id" : 91, - "ref" : "master" + "ref" : "main" }, { "started_at" : null, @@ -690,7 +690,7 @@ Example response: "target_url" : "https://gitlab.example.com/janedoe/gitlab-foss/builds/90", "id" : 90, "finished_at" : null, - "ref" : "master", + "ref" : "main", "sha" : "18f3e63d05582537db6d183d9d557be09e1f90c8", "author" : { "id" : 28, @@ -789,7 +789,7 @@ Example response: "state":"opened", "created_at":"2018-03-26T17:26:30.916Z", "updated_at":"2018-03-26T17:26:30.916Z", - "target_branch":"master", + "target_branch":"main", "source_branch":"test-branch", "upvotes":0, "downvotes":0, @@ -830,7 +830,7 @@ Example response: ## Get GPG signature of a commit -Get the [GPG signature from a commit](../user/project/repository/gpg_signed_commits/index.md), +Get the [GPG signature from a commit](../user/project/repository/signed_commits/gpg.md), if it is signed. For unsigned commits, it results in a 404 response. ```plaintext diff --git a/doc/api/dependency_list_export.md b/doc/api/dependency_list_export.md new file mode 100644 index 00000000000..083f7a640fc --- /dev/null +++ b/doc/api/dependency_list_export.md @@ -0,0 +1,158 @@ +--- +stage: Govern +group: Threat Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Dependency list export API **(ULTIMATE ALL)** + +Every call to this endpoint requires authentication. + +## Create a pipeline-level dependency list export **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333463) in GitLab 16.4 [with a flag](../administration/feature_flags.md) named `merge_sbom_api`. Enabled by default. This feature is an [Experiment](../policy/experiment-beta-support.md#experiment). + +FLAG: +On self-managed GitLab, by default this feature is available. +To hide the feature, an administrator can [disable the feature flag](../administration/feature_flags.md) named `merge_sbom_api`. +On GitLab.com, this feature is available. + +WARNING: +This feature is an [Experiment](../policy/experiment-beta-support.md#experiment) +and subject to change without notice. + +Create a new CycloneDX JSON export for all the project dependencies detected in a pipeline. + +If an authenticated user doesn't have permission to +[read_dependency](../user/permissions.md#custom-role-requirements), +this request returns a `403 Forbidden` status code. + +SBOM exports can be only accessed by the export's author. + +```plaintext +POST /pipelines/:id/dependency_list_exports +``` + +| Attribute | Type | Required | Description | +| ------------------- | ----------------- | ---------- | -----------------------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID of the pipeline which the authenticated user has access to. | +| `export_type` | string | yes | This must be set to `sbom`. | + +```shell +curl --request POST --header "PRIVATE-TOKEN: <private_token>" "https://gitlab.example.com/api/v4/pipelines/1/dependency_list_exports" --data "export_type=sbom" +``` + +The created dependency list export is automatically deleted after 1 hour. + +Example response: + +```json +{ + "id": 2, + "has_finished": false, + "self": "http://gitlab.example.com/api/v4/dependency_list_exports/2", + "download": "http://gitlab.example.com/api/v4/dependency_list_exports/2/download" +} +``` + +## Get single dependency list export + +Get a single dependency list export. + +```plaintext +GET /security/dependency_list_exports/:id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of the dependency list export. | + +```shell +curl --header "PRIVATE-TOKEN: <private_token>" "https://gitlab.example.com/api/v4/security/dependency_list_exports/2" +``` + +The status code is `202 Accepted` when the dependency list export is being generated, and `200 OK` when it's ready. + +Example response: + +```json +{ + "id": 4, + "has_finished": true, + "self": "http://gitlab.example.com/api/v4/dependency_list_exports/4", + "download": "http://gitlab.example.com/api/v4/dependency_list_exports/4/download" +} +``` + +## Download dependency list export + +Download a single dependency list export. + +```plaintext +GET /security/dependency_list_exports/:id/download +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of the dependency list export. | + +```shell +curl --header "PRIVATE-TOKEN: <private_token>" "https://gitlab.example.com/api/v4/security/dependency_list_exports/2/download" +``` + +The response is `404 Not Found` if the dependency list export is not finished yet or was not found. + +Example response: + +```json +{ + "bomFormat": "CycloneDX", + "specVersion": "1.4", + "serialNumber": "urn:uuid:aec33827-20ae-40d0-ae83-18ee846364d2", + "version": 1, + "metadata": { + "tools": [ + { + "vendor": "Gitlab", + "name": "Gemnasium", + "version": "2.34.0" + } + ], + "authors": [ + { + "name": "Gitlab", + "email": "support@gitlab.com" + } + ], + "properties": [ + { + "name": "gitlab:dependency_scanning:input_file", + "value": "package-lock.json" + } + ] + }, + "components": [ + { + "name": "com.fasterxml.jackson.core/jackson-core", + "purl": "pkg:maven/com.fasterxml.jackson.core/jackson-core@2.9.2", + "version": "2.9.2", + "type": "library", + "licenses": [ + { + "license": { + "id": "MIT", + "url": "https://spdx.org/licenses/MIT.html" + } + }, + { + "license": { + "id": "BSD-3-Clause", + "url": "https://spdx.org/licenses/BSD-3-Clause.html" + } + } + ] + } + ] +} + +``` diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index 3d5fa092416..ee568b41832 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -26,7 +26,7 @@ Supported attributes: | Attribute | Type | Required | Description | |:------------|:---------|:---------|:----------------------| -| `public` | boolean | **{dotted-circle}** No | Only return deploy keys that are public. Defaults to `false`. | +| `public` | boolean | No | Only return deploy keys that are public. Defaults to `false`. | Example request: diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index bc0cee1d627..c0e6a57a9d7 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -20,7 +20,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|----------|------------------------|-------------| -| `active` | boolean | **{dotted-circle}** No | Limit by active status. | +| `active` | boolean | No | Limit by active status. | Example request: @@ -66,8 +66,8 @@ Parameters: | Attribute | Type | Required | Description | |:---------------|:---------------|:-----------------------|:------------| -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `active` | boolean | **{dotted-circle}** No | Limit by active status. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `active` | boolean | No | Limit by active status. | Example request: @@ -108,8 +108,8 @@ Parameters: | Attribute | Type | Required | Description | | ---------- | -------------- | ---------------------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `token_id` | integer | **{check-circle}** Yes | ID of the deploy token | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `token_id` | integer | Yes | ID of the deploy token | Example request: @@ -148,11 +148,11 @@ Parameters: | Attribute | Type | Required | Description | | ------------ | ---------------- | ---------------------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | **{check-circle}** Yes | New deploy token's name | -| `scopes` | array of strings | **{check-circle}** Yes | Indicates the deploy token scopes. Must be at least one of `read_repository`, `read_registry`, `write_registry`, `read_package_registry`, or `write_package_registry`. | -| `expires_at` | datetime | **{dotted-circle}** No | Expiration date for the deploy token. Does not expire if no value is provided. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `username` | string | **{dotted-circle}** No | Username for deploy token. Default is `gitlab+deploy-token-{n}` | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | Yes | New deploy token's name | +| `scopes` | array of strings | Yes | Indicates the deploy token scopes. Must be at least one of `read_repository`, `read_registry`, `write_registry`, `read_package_registry`, or `write_package_registry`. | +| `expires_at` | datetime | No | Expiration date for the deploy token. Does not expire if no value is provided. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | +| `username` | string | No | Username for deploy token. Default is `gitlab+deploy-token-{n}` | Example request: @@ -193,8 +193,8 @@ Parameters: | Attribute | Type | Required | Description | | ---------- | -------------- | ---------------------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `token_id` | integer | **{check-circle}** Yes | ID of the deploy token | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `token_id` | integer | Yes | ID of the deploy token | Example request: @@ -222,8 +222,8 @@ Parameters: | Attribute | Type | Required | Description | |:---------------|:---------------|:-----------------------|:------------| -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | -| `active` | boolean | **{dotted-circle}** No | Limit by active status. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | +| `active` | boolean | No | Limit by active status. | Example request: @@ -264,8 +264,8 @@ Parameters: | Attribute | Type | Required | Description | | ----------- | -------------- | ---------------------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `token_id` | integer | **{check-circle}** Yes | ID of the deploy token | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `token_id` | integer | Yes | ID of the deploy token | Example request: @@ -304,11 +304,11 @@ Parameters: | Attribute | Type | Required | Description | | ------------ | ---- | --------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | **{check-circle}** Yes | New deploy token's name | -| `scopes` | array of strings | **{check-circle}** Yes | Indicates the deploy token scopes. Must be at least one of `read_repository`, `read_registry`, `write_registry`, `read_package_registry`, or `write_package_registry`. | -| `expires_at` | datetime | **{dotted-circle}** No | Expiration date for the deploy token. Does not expire if no value is provided. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `username` | string | **{dotted-circle}** No | Username for deploy token. Default is `gitlab+deploy-token-{n}` | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | Yes | New deploy token's name | +| `scopes` | array of strings | Yes | Indicates the deploy token scopes. Must be at least one of `read_repository`, `read_registry`, `write_registry`, `read_package_registry`, or `write_package_registry`. | +| `expires_at` | datetime | No | Expiration date for the deploy token. Does not expire if no value is provided. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | +| `username` | string | No | Username for deploy token. Default is `gitlab+deploy-token-{n}` | Example request: @@ -349,8 +349,8 @@ Parameters: | Attribute | Type | Required | Description | | ----------- | -------------- | ---------------------- | ----------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `token_id` | integer | **{check-circle}** Yes | ID of the deploy token | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `token_id` | integer | Yes | ID of the deploy token | Example request: diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 6b475473790..dac2c886166 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -990,7 +990,7 @@ For example, if a commit (`<COMMIT_ID>`) deletes line 463 in the README, you can on the deletion by referencing line 463 in the *old* file: ```shell -curl --request POST --header "PRIVATE-TOKEN: [ACCESS_TOKEN]" \ +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ --form "note=Very clever to remove this unnecessary line!" \ --form "path=README" --form "line=463" --form "line_type=old" \ "https://gitlab.com/api/v4/projects/47/repository/commits/<COMMIT_ID>/comments" @@ -1000,7 +1000,7 @@ If a commit (`<COMMIT_ID>`) adds line 157 to `hello.rb`, you can comment on the addition by referencing line 157 in the *new* file: ```shell -curl --request POST --header "PRIVATE-TOKEN: [ACCESS_TOKEN]" \ +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ --form "note=This is brilliant!" --form "path=hello.rb" \ --form "line=157" --form "line_type=new" \ "https://gitlab.com/api/v4/projects/47/repository/commits/<COMMIT_ID>/comments" diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index 0858e105a1f..fb22e96c20b 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Observability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/events.md b/doc/api/events.md index 9889e8d7701..8564b9b0942 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -185,7 +185,7 @@ Example response: "ref_type": "branch", "commit_from": "50d4420237a9de7be1304607147aec22e4a14af7", "commit_to": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", - "ref": "master", + "ref": "main", "commit_title": "Add simple search to projects in public area" }, "target_title": null diff --git a/doc/api/feature_flags.md b/doc/api/feature_flags.md index d7bf4fe826e..824c892c17b 100644 --- a/doc/api/feature_flags.md +++ b/doc/api/feature_flags.md @@ -59,7 +59,8 @@ Example response: "id": 1, "environment_scope": "production" } - ] + ], + "user_list": null } ] }, @@ -81,7 +82,36 @@ Example response: "id": 2, "environment_scope": "staging" } - ] + ], + "user_list": null + } + ] + }, + { + "name":"user_list", + "description":"This feature is about user list", + "active": true, + "version": "new_version_flag", + "created_at":"2019-11-04T08:13:10.507Z", + "updated_at":"2019-11-04T08:13:10.507Z", + "scopes":[], + "strategies": [ + { + "id": 2, + "name": "gitlabUserList", + "parameters": {}, + "scopes": [ + { + "id": 2, + "environment_scope": "staging" + } + ], + "user_list": { + "id": 1, + "iid": 1, + "name": "My user list", + "user_xids": "user1,user2,user3" + } } ] } @@ -126,7 +156,8 @@ Example response: "id": 37, "environment_scope": "production" } - ] + ], + "user_list": null } ] } @@ -147,11 +178,12 @@ POST /projects/:id/feature_flags | `version` | string | yes | The version of the feature flag. Must be `new_version_flag`. Omit to create a Legacy feature flag. | | `description` | string | no | The description of the feature flag. | | `active` | boolean | no | The active state of the flag. Defaults to true. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | -| `strategies` | JSON | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | +| `strategies` | array of strategy JSON objects | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | | `strategies:name` | JSON | no | The strategy name. Can be `default`, `gradualRolloutUserId`, `userWithId`, or `gitlabUserList`. In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/36380) and later, can be [`flexibleRollout`](https://docs.getunleash.io/user_guide/activation_strategy/#gradual-rollout). | | `strategies:parameters` | JSON | no | The strategy parameters. | | `strategies:scopes` | JSON | no | The scopes for the strategy. | | `strategies:scopes:environment_scope` | string | no | The environment scope of the scope. | +| `strategies:user_list_id` | integer/string | no | The ID of the feature flag user list. If strategy is `gitlabUserList`. | ```shell curl "https://gitlab.example.com/api/v4/projects/1/feature_flags" \ @@ -208,7 +240,7 @@ PUT /projects/:id/feature_flags/:feature_flag_name | `description` | string | no | The description of the feature flag. | | `active` | boolean | no | The active state of the flag. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | | `name` | string | no | The new name of the feature flag. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | -| `strategies` | JSON | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | +| `strategies` | array of strategy JSON objects | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | | `strategies:id` | JSON | no | The feature flag strategy ID. | | `strategies:name` | JSON | no | The strategy name. | | `strategies:_destroy` | boolean | no | Delete the strategy when true. | @@ -217,6 +249,7 @@ PUT /projects/:id/feature_flags/:feature_flag_name | `strategies:scopes:id` | JSON | no | The environment scope ID. | | `strategies:scopes:environment_scope` | string | no | The environment scope of the scope. | | `strategies:scopes:_destroy` | boolean | no | Delete the scope when true. | +| `strategies:user_list_id` | integer/string | no | The ID of the feature flag user list. If strategy is `gitlabUserList`. | ```shell curl "https://gitlab.example.com/api/v4/projects/1/feature_flags/awesome_feature" \ diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index 4ec57274bd7..e709a810255 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -334,19 +334,11 @@ Example response: "job_artifacts_failed_count": null, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "0.00%", - "design_repositories_count": 3, - "design_repositories_synced_count": null, - "design_repositories_failed_count": null, - "design_repositories_synced_in_percentage": "0.00%", "projects_count": 41, "repositories_count": 41, "repositories_failed_count": null, "repositories_synced_count": null, "repositories_synced_in_percentage": "0.00%", - "wikis_count": 41, - "wikis_failed_count": null, - "wikis_synced_count": null, - "wikis_synced_in_percentage": "0.00%", "replication_slots_count": 1, "replication_slots_used_count": 1, "replication_slots_used_in_percentage": "100.00%", @@ -357,19 +349,11 @@ Example response: "repositories_checksummed_count": 20, "repositories_checksum_failed_count": 5, "repositories_checksummed_in_percentage": "48.78%", - "wikis_checksummed_count": 10, - "wikis_checksum_failed_count": 3, - "wikis_checksummed_in_percentage": "24.39%", "repositories_verified_count": 20, "repositories_verification_failed_count": 5, "repositories_verified_in_percentage": "48.78%", "repositories_checksum_mismatch_count": 3, - "wikis_verified_count": 10, - "wikis_verification_failed_count": 3, - "wikis_verified_in_percentage": "24.39%", - "wikis_checksum_mismatch_count": 1, "repositories_retrying_verification_count": 1, - "wikis_retrying_verification_count": 3, "last_event_id": 23, "last_event_timestamp": 1509681166, "cursor_last_event_id": null, @@ -598,10 +582,6 @@ Example response: "job_artifacts_failed_count": 1, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "50.00%", - "design_repositories_count": 3, - "design_repositories_synced_count": null, - "design_repositories_failed_count": null, - "design_repositories_synced_in_percentage": "0.00%", "design_management_repositories_count": 5, "design_management_repositories_synced_count": 5, "design_management_repositories_failed_count": 5, @@ -619,10 +599,6 @@ Example response: "repositories_failed_count": 1, "repositories_synced_count": 40, "repositories_synced_in_percentage": "97.56%", - "wikis_count": 41, - "wikis_failed_count": 0, - "wikis_synced_count": 41, - "wikis_synced_in_percentage": "100.00%", "replication_slots_count": null, "replication_slots_used_count": null, "replication_slots_used_in_percentage": "0.00%", @@ -630,19 +606,11 @@ Example response: "repositories_checksummed_count": 20, "repositories_checksum_failed_count": 5, "repositories_checksummed_in_percentage": "48.78%", - "wikis_checksummed_count": 10, - "wikis_checksum_failed_count": 3, - "wikis_checksummed_in_percentage": "24.39%", "repositories_verified_count": 20, "repositories_verification_failed_count": 5, "repositories_verified_in_percentage": "48.78%", "repositories_checksum_mismatch_count": 3, - "wikis_verified_count": 10, - "wikis_verification_failed_count": 3, - "wikis_verified_in_percentage": "24.39%", - "wikis_checksum_mismatch_count": 1, "repositories_retrying_verification_count": 4, - "wikis_retrying_verification_count": 2, "repositories_checked_count": 5, "repositories_checked_failed_count": 1, "repositories_checked_in_percentage": "12.20%", @@ -841,19 +809,11 @@ Example response: "job_artifacts_failed_count": 1, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "50.00%", - "design_repositories_count": 3, - "design_repositories_synced_count": null, - "design_repositories_failed_count": null, - "design_repositories_synced_in_percentage": "0.00%", "projects_count": 41, "repositories_count": 41, "repositories_failed_count": 1, "repositories_synced_count": 40, "repositories_synced_in_percentage": "97.56%", - "wikis_count": 41, - "wikis_failed_count": 0, - "wikis_synced_count": 41, - "wikis_synced_in_percentage": "100.00%", "replication_slots_count": null, "replication_slots_used_count": null, "replication_slots_used_in_percentage": "0.00%", diff --git a/doc/api/geo_sites.md b/doc/api/geo_sites.md index ff06f668514..2c0ceab62e2 100644 --- a/doc/api/geo_sites.md +++ b/doc/api/geo_sites.md @@ -296,31 +296,17 @@ Example response: "repositories_replication_enabled": null, "repositories_synced_count": null, "repositories_failed_count": null, - "wikis_synced_count": null, - "wikis_failed_count": null, "repositories_verified_count": null, "repositories_verification_failed_count": null, "repositories_verification_total_count": null, - "wikis_verified_count": null, - "wikis_verification_failed_count": null, - "wikis_verification_total_count": null, "job_artifacts_synced_missing_on_primary_count": null, "repositories_checksummed_count": 19, "repositories_checksum_failed_count": 0, "repositories_checksum_mismatch_count": null, "repositories_checksum_total_count": 19, - "wikis_checksummed_count": 0, - "wikis_checksum_failed_count": 0, - "wikis_checksum_mismatch_count": null, - "wikis_checksum_total_count": 19, "repositories_retrying_verification_count": null, - "wikis_retrying_verification_count": null, "projects_count": 19, "container_repositories_replication_enabled": null, - "design_repositories_replication_enabled": null, - "design_repositories_count": null, - "design_repositories_synced_count": null, - "design_repositories_failed_count": null, "lfs_objects_count": 0, "lfs_objects_checksum_total_count": 0, "lfs_objects_checksummed_count": 0, @@ -479,11 +465,7 @@ Example response: "repositories_checksummed_in_percentage": "100.00%", "repositories_verified_in_percentage": "0.00%", "repositories_checked_in_percentage": "0.00%", - "wikis_synced_in_percentage": "0.00%", - "wikis_checksummed_in_percentage": "0.00%", - "wikis_verified_in_percentage": "0.00%", "replication_slots_used_in_percentage": "100.00%", - "design_repositories_synced_in_percentage": "0.00%", "lfs_objects_synced_in_percentage": "0.00%", "lfs_objects_verified_in_percentage": "0.00%", "merge_request_diffs_synced_in_percentage": "0.00%", @@ -515,7 +497,6 @@ Example response: "project_wiki_repositories_synced_in_percentage": "0.00%", "project_wiki_repositories_verified_in_percentage": "0.00%", "repositories_count": 19, - "wikis_count": 19, "replication_slots_count": 1, "replication_slots_used_count": 1, "healthy": true, @@ -548,31 +529,17 @@ Example response: "repositories_replication_enabled": true, "repositories_synced_count": 18, "repositories_failed_count": 0, - "wikis_synced_count": 18, - "wikis_failed_count": 0, "repositories_verified_count": 0, "repositories_verification_failed_count": 0, "repositories_verification_total_count": 19, - "wikis_verified_count": 0, - "wikis_verification_failed_count": 0, - "wikis_verification_total_count": 19, "job_artifacts_synced_missing_on_primary_count": null, "repositories_checksummed_count": null, "repositories_checksum_failed_count": null, "repositories_checksum_mismatch_count": 0, "repositories_checksum_total_count": null, - "wikis_checksummed_count": null, - "wikis_checksum_failed_count": null, - "wikis_checksum_mismatch_count": 0, - "wikis_checksum_total_count": null, "repositories_retrying_verification_count": 0, - "wikis_retrying_verification_count": 0, "projects_count": 19, "container_repositories_replication_enabled": null, - "design_repositories_replication_enabled": true, - "design_repositories_count": 0, - "design_repositories_synced_count": 0, - "design_repositories_failed_count": 0, "lfs_objects_count": 0, "lfs_objects_checksum_total_count": null, "lfs_objects_checksummed_count": null, @@ -731,11 +698,7 @@ Example response: "repositories_checksummed_in_percentage": "0.00%", "repositories_verified_in_percentage": "0.00%", "repositories_checked_in_percentage": "0.00%", - "wikis_synced_in_percentage": "94.74%", - "wikis_checksummed_in_percentage": "0.00%", - "wikis_verified_in_percentage": "0.00%", "replication_slots_used_in_percentage": "0.00%", - "design_repositories_synced_in_percentage": "0.00%", "lfs_objects_synced_in_percentage": "0.00%", "lfs_objects_verified_in_percentage": "0.00%", "merge_request_diffs_synced_in_percentage": "0.00%", @@ -767,7 +730,6 @@ Example response: "project_wiki_repositories_synced_in_percentage": "100.00%", "project_wiki_repositories_verified_in_percentage": "100.00%", "repositories_count": 19, - "wikis_count": 19, "replication_slots_count": null, "replication_slots_used_count": null, "healthy": false, @@ -816,31 +778,17 @@ Example response: "repositories_replication_enabled": true, "repositories_synced_count": 18, "repositories_failed_count": 0, - "wikis_synced_count": 18, - "wikis_failed_count": 0, "repositories_verified_count": 0, "repositories_verification_failed_count": 0, "repositories_verification_total_count": 19, - "wikis_verified_count": 0, - "wikis_verification_failed_count": 0, - "wikis_verification_total_count": 19, "job_artifacts_synced_missing_on_primary_count": null, "repositories_checksummed_count": null, "repositories_checksum_failed_count": null, "repositories_checksum_mismatch_count": 0, "repositories_checksum_total_count": null, - "wikis_checksummed_count": null, - "wikis_checksum_failed_count": null, - "wikis_checksum_mismatch_count": 0, - "wikis_checksum_total_count": null, "repositories_retrying_verification_count": 0, - "wikis_retrying_verification_count": 0, "projects_count": 19, "container_repositories_replication_enabled": null, - "design_repositories_replication_enabled": true, - "design_repositories_count": 0, - "design_repositories_synced_count": 0, - "design_repositories_failed_count": 0, "lfs_objects_count": 0, "lfs_objects_checksum_total_count": null, "lfs_objects_checksummed_count": null, @@ -999,11 +947,7 @@ Example response: "repositories_checksummed_in_percentage": "0.00%", "repositories_verified_in_percentage": "0.00%", "repositories_checked_in_percentage": "0.00%", - "wikis_synced_in_percentage": "94.74%", - "wikis_checksummed_in_percentage": "0.00%", - "wikis_verified_in_percentage": "0.00%", "replication_slots_used_in_percentage": "0.00%", - "design_repositories_synced_in_percentage": "0.00%", "lfs_objects_synced_in_percentage": "0.00%", "lfs_objects_verified_in_percentage": "0.00%", "merge_request_diffs_synced_in_percentage": "0.00%", @@ -1035,7 +979,6 @@ Example response: "project_wiki_repositories_synced_in_percentage": "100.00%", "project_wiki_repositories_verified_in_percentage": "100.00%", "repositories_count": 19, - "wikis_count": 19, "replication_slots_count": null, "replication_slots_used_count": null, "healthy": false, diff --git a/doc/api/graphql/custom_emoji.md b/doc/api/graphql/custom_emoji.md index a0a329ca4b5..7efadb7e9dd 100644 --- a/doc/api/graphql/custom_emoji.md +++ b/doc/api/graphql/custom_emoji.md @@ -20,9 +20,9 @@ Parameters: | Attribute | Type | Required | Description | | :----------- | :------------- | :--------------------- | :------------------------------------------------------------------------ | -| `group_path` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the top-level group](../rest/index.md#namespaced-path-encoding) | -| `name` | string | **{check-circle}** Yes | Name of the custom emoji. | -| `file` | string | **{check-circle}** Yes | URL of the custom emoji image. | +| `group_path` | integer/string | Yes | ID or [URL-encoded path of the top-level group](../rest/index.md#namespaced-path-encoding) | +| `name` | string | Yes | Name of the custom emoji. | +| `file` | string | Yes | URL of the custom emoji image. | ## Create a custom emoji diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index c9fc446303f..65736f10ba9 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -74,13 +74,13 @@ four standard [pagination arguments](#connection-pagination-arguments): ### `Query.aiMessages` -Find AI messages. +Find GitLab Duo Chat messages. WARNING: **Introduced** in 16.1. This feature is an Experiment. It can be changed or removed at any time. -Returns [`AiCachedMessageTypeConnection!`](#aicachedmessagetypeconnection). +Returns [`AiChatMessageConnection!`](#aichatmessageconnection). This field returns a [connection](#connections). It accepts the four standard [pagination arguments](#connection-pagination-arguments): @@ -91,7 +91,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="queryaimessagesrequestids"></a>`requestIds` | [`[ID!]`](#id) | Array of request IDs to fetch. | -| <a id="queryaimessagesroles"></a>`roles` | [`[AiCachedMessageRole!]`](#aicachedmessagerole) | Array of roles to fetch. | +| <a id="queryaimessagesroles"></a>`roles` | [`[AiChatMessageRole!]`](#aichatmessagerole) | Array of roles to fetch. | ### `Query.auditEventDefinitions` @@ -204,6 +204,24 @@ Returns [`CiStage`](#cistage). | ---- | ---- | ----------- | | <a id="querycipipelinestageid"></a>`id` | [`CiStageID!`](#cistageid) | Global ID of the CI stage. | +### `Query.ciQueueingHistory` + +Time it took for ci job to be picked up by runner in percentiles. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`QueueingDelayHistory`](#queueingdelayhistory). + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="queryciqueueinghistoryfromtime"></a>`fromTime` | [`Time`](#time) | Start of the requested time frame. Defaults to 3 hours ago. | +| <a id="queryciqueueinghistoryrunnertype"></a>`runnerType` | [`CiRunnerType`](#cirunnertype) | Filter jobs by the type of runner that executed them. | +| <a id="queryciqueueinghistorytotime"></a>`toTime` | [`Time`](#time) | End of the requested time frame. Defaults to current time. | + ### `Query.ciVariables` List of the instance's CI/CD variables. @@ -361,6 +379,16 @@ This field returns a [connection](#connections). It accepts the four standard [pagination arguments](#connection-pagination-arguments): `before: String`, `after: String`, `first: Int`, `last: Int`. +### `Query.instanceGoogleCloudLoggingConfigurations` + +Instance level google cloud logging configurations. + +Returns [`InstanceGoogleCloudLoggingConfigurationTypeConnection`](#instancegooglecloudloggingconfigurationtypeconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + ### `Query.instanceSecurityDashboard` Fields related to Instance Security Dashboard. @@ -462,6 +490,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="queryjobsfailurereason"></a>`failureReason` **{warning-solid}** | [`CiJobFailureReason`](#cijobfailurereason) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Filter jobs by failure reason. Currently only `RUNNER_SYSTEM_FAILURE` together with `runnerTypes: INSTANCE_TYPE` is supported. | +| <a id="queryjobsrunnertypes"></a>`runnerTypes` **{warning-solid}** | [`[CiRunnerType!]`](#cirunnertype) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Filter jobs by runner type if feature flag `:admin_jobs_filter_runner_type` is enabled. | | <a id="queryjobsstatuses"></a>`statuses` | [`[CiJobStatus!]`](#cijobstatus) | Filter jobs by status. | ### `Query.licenseHistoryEntries` @@ -474,6 +504,16 @@ This field returns a [connection](#connections). It accepts the four standard [pagination arguments](#connection-pagination-arguments): `before: String`, `after: String`, `first: Int`, `last: Int`. +### `Query.memberRolePermissions` + +List of all customizable permissions. + +Returns [`CustomizablePermissionConnection`](#customizablepermissionconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + ### `Query.mergeRequest` Find a merge request. @@ -532,6 +572,22 @@ Returns [`Note`](#note). | ---- | ---- | ----------- | | <a id="querynoteid"></a>`id` | [`NoteID!`](#noteid) | Global ID of the note. | +### `Query.organization` + +Find an organization. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`Organization`](#organization). + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="queryorganizationid"></a>`id` | [`OrganizationsOrganizationID!`](#organizationsorganizationid) | ID of the organization. | + ### `Query.package` Find a package. This field can only be resolved for one query in any single request. Returns `null` if a package has no `default` status. @@ -632,7 +688,7 @@ Returns [`RunnerSetup`](#runnersetup). ### `Query.runners` -Find runners visible to the current user. +Get all runners in the GitLab instance (project and shared). Access is restricted to users with administrator access. Returns [`CiRunnerConnection`](#cirunnerconnection). @@ -821,6 +877,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="queryvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="queryvulnerabilitiesdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. Only dismissed Vulnerabilities will be included with the filter. | | <a id="queryvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | +| <a id="queryvulnerabilitieshasmergerequest"></a>`hasMergeRequest` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked merge requests. | | <a id="queryvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="queryvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="queryvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | @@ -933,6 +990,30 @@ mutation($id: NoteableID!, $body: String!) { } ``` +### `Mutation.abuseReportLabelCreate` + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `AbuseReportLabelCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationabusereportlabelcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationabusereportlabelcreatecolor"></a>`color` | [`String`](#string) | The color of the label given in 6-digit hex notation with leading '#' sign (for example, `#FFAABB`) or one of the CSS color names. | +| <a id="mutationabusereportlabelcreatetitle"></a>`title` | [`String!`](#string) | Title of the label. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationabusereportlabelcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationabusereportlabelcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationabusereportlabelcreatelabel"></a>`label` | [`Label`](#label) | Label after mutation. | + ### `Mutation.achievementsAward` WARNING: @@ -1128,13 +1209,13 @@ Input type: `AiActionInput` | <a id="mutationaiactionanalyzecijobfailure"></a>`analyzeCiJobFailure` | [`AnalyzeCiJobFailureInput`](#analyzecijobfailureinput) | Input for analyze_ci_job_failure AI action. | | <a id="mutationaiactionchat"></a>`chat` | [`AiChatInput`](#aichatinput) | Input for chat AI action. | | <a id="mutationaiactionclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationaiactionclientsubscriptionid"></a>`clientSubscriptionId` | [`String`](#string) | Client generated ID that can be subscribed to, to receive a response for the mutation. | | <a id="mutationaiactionexplaincode"></a>`explainCode` | [`AiExplainCodeInput`](#aiexplaincodeinput) | Input for explain_code AI action. | | <a id="mutationaiactionexplainvulnerability"></a>`explainVulnerability` | [`AiExplainVulnerabilityInput`](#aiexplainvulnerabilityinput) | Input for explain_vulnerability AI action. | | <a id="mutationaiactionfillinmergerequesttemplate"></a>`fillInMergeRequestTemplate` | [`AiFillInMergeRequestTemplateInput`](#aifillinmergerequesttemplateinput) | Input for fill_in_merge_request_template AI action. | | <a id="mutationaiactiongeneratecommitmessage"></a>`generateCommitMessage` | [`AiGenerateCommitMessageInput`](#aigeneratecommitmessageinput) | Input for generate_commit_message AI action. | | <a id="mutationaiactiongeneratedescription"></a>`generateDescription` | [`AiGenerateDescriptionInput`](#aigeneratedescriptioninput) | Input for generate_description AI action. | | <a id="mutationaiactiongeneratetestfile"></a>`generateTestFile` | [`GenerateTestFileInput`](#generatetestfileinput) | Input for generate_test_file AI action. | -| <a id="mutationaiactionmarkupformat"></a>`markupFormat` | [`MarkupFormat`](#markupformat) | Indicates the response format. | | <a id="mutationaiactionsummarizecomments"></a>`summarizeComments` | [`AiSummarizeCommentsInput`](#aisummarizecommentsinput) | Input for summarize_comments AI action. | | <a id="mutationaiactionsummarizereview"></a>`summarizeReview` | [`AiSummarizeReviewInput`](#aisummarizereviewinput) | Input for summarize_review AI action. | | <a id="mutationaiactiontanukibot"></a>`tanukiBot` | [`AiTanukiBotInput`](#aitanukibotinput) | Input for tanuki_bot AI action. | @@ -1701,7 +1782,7 @@ Input type: `CiAiGenerateConfigInput` | ---- | ---- | ----------- | | <a id="mutationciaigenerateconfigclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationciaigenerateconfigerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationciaigenerateconfigusermessage"></a>`userMessage` | [`AiMessageType`](#aimessagetype) | User chat message. | +| <a id="mutationciaigenerateconfigusermessage"></a>`userMessage` | [`AiMessage`](#aimessage) | User chat message. | ### `Mutation.ciJobTokenScopeAddProject` @@ -2365,6 +2446,7 @@ Input type: `CreateTestCaseInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationcreatetestcaseclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcreatetestcaseconfidential"></a>`confidential` | [`Boolean`](#boolean) | Sets the test case confidentiality. | | <a id="mutationcreatetestcasedescription"></a>`description` | [`String`](#string) | Test case description. | | <a id="mutationcreatetestcaselabelids"></a>`labelIds` | [`[ID!]`](#id) | IDs of labels to be added to the test case. | | <a id="mutationcreatetestcaseprojectpath"></a>`projectPath` | [`ID!`](#id) | Project full path to create the test case in. | @@ -2817,7 +2899,7 @@ Input type: `DeleteAnnotationInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdeleteannotationclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdeleteannotationid"></a>`id` | [`MetricsDashboardAnnotationID!`](#metricsdashboardannotationid) | Global ID of the annotation to delete. | +| <a id="mutationdeleteannotationid"></a>`id` | [`String!`](#string) | Global ID of the annotation to delete. | #### Fields @@ -3690,6 +3772,32 @@ Input type: `ExternalAuditEventDestinationUpdateInput` | <a id="mutationexternalauditeventdestinationupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationexternalauditeventdestinationupdateexternalauditeventdestination"></a>`externalAuditEventDestination` | [`ExternalAuditEventDestination`](#externalauditeventdestination) | Updated destination. | +### `Mutation.geoRegistriesBulkUpdate` + +Mutates multiple Geo registries for a given registry class. Does not mutate the registries if `geo_registries_update_mutation` feature flag is disabled. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `GeoRegistriesBulkUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationgeoregistriesbulkupdateaction"></a>`action` | [`GeoRegistriesBulkAction!`](#georegistriesbulkaction) | Action to be executed on Geo registries. | +| <a id="mutationgeoregistriesbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationgeoregistriesbulkupdateregistryclass"></a>`registryClass` | [`GeoRegistryClass!`](#georegistryclass) | Class of the Geo registries to be updated. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationgeoregistriesbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationgeoregistriesbulkupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationgeoregistriesbulkupdateregistryclass"></a>`registryClass` | [`GeoRegistryClass`](#georegistryclass) | Updated Geo registry class. | + ### `Mutation.geoRegistriesUpdate` Mutates a Geo registry. Does not mutate the registry entry if `geo_registries_update_mutation` feature flag is disabled. @@ -3706,7 +3814,7 @@ Input type: `GeoRegistriesUpdateInput` | ---- | ---- | ----------- | | <a id="mutationgeoregistriesupdateaction"></a>`action` | [`GeoRegistryAction!`](#georegistryaction) | Action to be executed on a Geo registry. | | <a id="mutationgeoregistriesupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationgeoregistriesupdateregistryclass"></a>`registryClass` | [`GeoRegistryClass!`](#georegistryclass) | Class of the Geo registry to be updated. | +| <a id="mutationgeoregistriesupdateregistryclass"></a>`registryClass` | [`GeoRegistryClass`](#georegistryclass) | Class of the Geo registry to be updated. | | <a id="mutationgeoregistriesupdateregistryid"></a>`registryId` | [`GeoBaseRegistryID!`](#geobaseregistryid) | ID of the Geo registry entry to be updated. | #### Fields @@ -3750,6 +3858,7 @@ Input type: `GoogleCloudLoggingConfigurationCreateInput` | <a id="mutationgooglecloudloggingconfigurationcreategoogleprojectidname"></a>`googleProjectIdName` | [`String!`](#string) | Unique identifier of the Google Cloud project to which the logging configuration belongs. | | <a id="mutationgooglecloudloggingconfigurationcreategrouppath"></a>`groupPath` | [`ID!`](#id) | Group path. | | <a id="mutationgooglecloudloggingconfigurationcreatelogidname"></a>`logIdName` | [`String`](#string) | Unique identifier used to distinguish and manage different logs within the same Google Cloud project.(defaults to `audit_events`). | +| <a id="mutationgooglecloudloggingconfigurationcreatename"></a>`name` | [`String`](#string) | Destination name. | | <a id="mutationgooglecloudloggingconfigurationcreateprivatekey"></a>`privateKey` | [`String!`](#string) | Private Key associated with the service account. This key is used to authenticate the service account and authorize it to interact with the Google Cloud Logging service. | #### Fields @@ -3791,6 +3900,7 @@ Input type: `GoogleCloudLoggingConfigurationUpdateInput` | <a id="mutationgooglecloudloggingconfigurationupdategoogleprojectidname"></a>`googleProjectIdName` | [`String`](#string) | Unique identifier of the Google Cloud project to which the logging configuration belongs. | | <a id="mutationgooglecloudloggingconfigurationupdateid"></a>`id` | [`AuditEventsGoogleCloudLoggingConfigurationID!`](#auditeventsgooglecloudloggingconfigurationid) | ID of the google Cloud configuration to update. | | <a id="mutationgooglecloudloggingconfigurationupdatelogidname"></a>`logIdName` | [`String`](#string) | Unique identifier used to distinguish and manage different logs within the same Google Cloud project. | +| <a id="mutationgooglecloudloggingconfigurationupdatename"></a>`name` | [`String`](#string) | Destination name. | | <a id="mutationgooglecloudloggingconfigurationupdateprivatekey"></a>`privateKey` | [`String`](#string) | Private Key associated with the service account. This key is used to authenticate the service account and authorize it to interact with the Google Cloud Logging service. | #### Fields @@ -3986,6 +4096,47 @@ Input type: `InstanceExternalAuditEventDestinationUpdateInput` | <a id="mutationinstanceexternalauditeventdestinationupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationinstanceexternalauditeventdestinationupdateinstanceexternalauditeventdestination"></a>`instanceExternalAuditEventDestination` | [`InstanceExternalAuditEventDestination`](#instanceexternalauditeventdestination) | Updated destination. | +### `Mutation.instanceGoogleCloudLoggingConfigurationCreate` + +Input type: `InstanceGoogleCloudLoggingConfigurationCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateclientemail"></a>`clientEmail` | [`String!`](#string) | Email address associated with the service account that will be used to authenticate and interact with the Google Cloud Logging service. This is part of the IAM credentials. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreategoogleprojectidname"></a>`googleProjectIdName` | [`String!`](#string) | Unique identifier of the Google Cloud project to which the logging configuration belongs. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreatelogidname"></a>`logIdName` | [`String`](#string) | Unique identifier used to distinguish and manage different logs within the same Google Cloud project.(defaults to `audit_events`). | +| <a id="mutationinstancegooglecloudloggingconfigurationcreatename"></a>`name` | [`String`](#string) | Destination name. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateprivatekey"></a>`privateKey` | [`String!`](#string) | Private Key associated with the service account. This key is used to authenticate the service account and authorize it to interact with the Google Cloud Logging service. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationinstancegooglecloudloggingconfigurationcreateinstancegooglecloudloggingconfiguration"></a>`instanceGoogleCloudLoggingConfiguration` | [`InstanceGoogleCloudLoggingConfigurationType`](#instancegooglecloudloggingconfigurationtype) | configuration created. | + +### `Mutation.instanceGoogleCloudLoggingConfigurationDestroy` + +Input type: `InstanceGoogleCloudLoggingConfigurationDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationinstancegooglecloudloggingconfigurationdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationinstancegooglecloudloggingconfigurationdestroyid"></a>`id` | [`AuditEventsInstanceGoogleCloudLoggingConfigurationID!`](#auditeventsinstancegooglecloudloggingconfigurationid) | ID of the Google Cloud logging configuration to destroy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationinstancegooglecloudloggingconfigurationdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationinstancegooglecloudloggingconfigurationdestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.issuableResourceLinkCreate` Input type: `IssuableResourceLinkCreateInput` @@ -4393,7 +4544,7 @@ Input type: `IssuesBulkUpdateInput` | <a id="mutationissuesbulkupdateids"></a>`ids` | [`[IssueID!]!`](#issueid) | Global ID array of the issues that will be updated. IDs that the user can't update will be ignored. A max of 100 can be provided. | | <a id="mutationissuesbulkupdateiterationid"></a>`iterationId` | [`IterationID`](#iterationid) | Global ID of the iteration that will be assigned to the issues. | | <a id="mutationissuesbulkupdatemilestoneid"></a>`milestoneId` | [`MilestoneID`](#milestoneid) | Global ID of the milestone that will be assigned to the issues. | -| <a id="mutationissuesbulkupdateparentid"></a>`parentId` | [`IssueParentID!`](#issueparentid) | Global ID of the parent to which the bulk update will be scoped. The parent can be a project **(FREE)** or a group **(PREMIUM)**. Example `IssueParentID` are `"gid://gitlab/Project/1"` and `"gid://gitlab/Group/1"`. | +| <a id="mutationissuesbulkupdateparentid"></a>`parentId` | [`IssueParentID!`](#issueparentid) | Global ID of the parent to which the bulk update will be scoped. The parent can be a project **(FREE ALL)** or a group **(PREMIUM ALL)**. Example `IssueParentID` are `"gid://gitlab/Project/1"` and `"gid://gitlab/Group/1"`. | | <a id="mutationissuesbulkupdateremovelabelids"></a>`removeLabelIds` | [`[LabelID!]`](#labelid) | Global ID array of the labels that will be removed from the issues. | | <a id="mutationissuesbulkupdatestateevent"></a>`stateEvent` | [`IssueStateEvent`](#issuestateevent) | Close or reopen an issue. | | <a id="mutationissuesbulkupdatesubscriptionevent"></a>`subscriptionEvent` | [`IssuableSubscriptionEvent`](#issuablesubscriptionevent) | Subscribe to or unsubscribe from issue notifications. | @@ -4946,7 +5097,7 @@ Input type: `MergeRequestUpdateInput` | <a id="mutationmergerequestupdateprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the merge request to mutate is in. | | <a id="mutationmergerequestupdatestate"></a>`state` | [`MergeRequestNewState`](#mergerequestnewstate) | Action to perform to change the state. | | <a id="mutationmergerequestupdatetargetbranch"></a>`targetBranch` | [`String`](#string) | Target branch of the merge request. | -| <a id="mutationmergerequestupdatetimeestimate"></a>`timeEstimate` | [`String`](#string) | Estimated time to complete the merge request, or `0` to remove the current estimate. | +| <a id="mutationmergerequestupdatetimeestimate"></a>`timeEstimate` | [`String`](#string) | Estimated time to complete the merge request. Use `null` or `0` to remove the current estimate. | | <a id="mutationmergerequestupdatetitle"></a>`title` | [`String`](#string) | Title of the merge request. | #### Fields @@ -5443,6 +5594,7 @@ Input type: `ProjectCiCdSettingsUpdateInput` | <a id="mutationprojectcicdsettingsupdatekeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Indicates if the latest artifact should be kept for the project. | | <a id="mutationprojectcicdsettingsupdatemergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Indicates if merge pipelines are enabled for the project. | | <a id="mutationprojectcicdsettingsupdatemergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Indicates if merge trains are enabled for the project. | +| <a id="mutationprojectcicdsettingsupdatemergetrainsskiptrainallowed"></a>`mergeTrainsSkipTrainAllowed` | [`Boolean`](#boolean) | Indicates whether an option is allowed to merge without refreshing the merge train. Ignored unless the `merge_trains_skip_train` feature flag is also enabled. | #### Fields @@ -5517,6 +5669,28 @@ Input type: `ProjectSetComplianceFrameworkInput` | <a id="mutationprojectsetcomplianceframeworkerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationprojectsetcomplianceframeworkproject"></a>`project` | [`Project`](#project) | Project after mutation. | +### `Mutation.projectSetContinuousVulnerabilityScanning` + +Enable/disable Continuous Vulnerability Scanning for the given project. + +Input type: `ProjectSetContinuousVulnerabilityScanningInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningenable"></a>`enable` | [`Boolean!`](#boolean) | Desired status for Continuous Vulnerability Scanning feature. | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningcontinuousvulnerabilityscanningenabled"></a>`continuousVulnerabilityScanningEnabled` | [`Boolean!`](#boolean) | Whether feature is enabled. | +| <a id="mutationprojectsetcontinuousvulnerabilityscanningerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.projectSetLocked` Input type: `ProjectSetLockedInput` @@ -6763,7 +6937,7 @@ Input type: `UpdateIssueInput` | <a id="mutationupdateissueprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the issue to mutate is in. | | <a id="mutationupdateissueremovelabelids"></a>`removeLabelIds` | [`[ID!]`](#id) | IDs of labels to be removed from the issue. | | <a id="mutationupdateissuestateevent"></a>`stateEvent` | [`IssueStateEvent`](#issuestateevent) | Close or reopen an issue. | -| <a id="mutationupdateissuetimeestimate"></a>`timeEstimate` | [`String`](#string) | Estimated time to complete the issue, or `0` to remove the current estimate. | +| <a id="mutationupdateissuetimeestimate"></a>`timeEstimate` | [`String`](#string) | Estimated time to complete the issue. Use `null` or `0` to remove the current estimate. | | <a id="mutationupdateissuetitle"></a>`title` | [`String`](#string) | Title of the issue. | | <a id="mutationupdateissuetype"></a>`type` | [`IssueType`](#issuetype) | Type of the issue. | | <a id="mutationupdateissueweight"></a>`weight` | [`Int`](#int) | Weight of the issue. | @@ -6988,7 +7162,7 @@ Input type: `UserAddOnAssignmentCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationuseraddonassignmentcreateaddonpurchaseid"></a>`addOnPurchaseId` | [`GitlabSubscriptionsAddOnPurchaseID!`](#gitlabsubscriptionsaddonpurchaseid) | Global ID of AddOnPurchase to be assinged to. | +| <a id="mutationuseraddonassignmentcreateaddonpurchaseid"></a>`addOnPurchaseId` | [`GitlabSubscriptionsAddOnPurchaseID!`](#gitlabsubscriptionsaddonpurchaseid) | Global ID of AddOnPurchase to be assigned to. | | <a id="mutationuseraddonassignmentcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationuseraddonassignmentcreateuserid"></a>`userId` | [`UserID!`](#userid) | Global ID of user to be assigned. | @@ -6999,6 +7173,7 @@ Input type: `UserAddOnAssignmentCreateInput` | <a id="mutationuseraddonassignmentcreateaddonpurchase"></a>`addOnPurchase` | [`AddOnPurchase`](#addonpurchase) | AddOnPurchase state after mutation. | | <a id="mutationuseraddonassignmentcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationuseraddonassignmentcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationuseraddonassignmentcreateuser"></a>`user` | [`AddOnUser`](#addonuser) | User who the add-on purchase was assigned to. | ### `Mutation.userAddOnAssignmentRemove` @@ -7023,6 +7198,7 @@ Input type: `UserAddOnAssignmentRemoveInput` | <a id="mutationuseraddonassignmentremoveaddonpurchase"></a>`addOnPurchase` | [`AddOnPurchase`](#addonpurchase) | AddOnPurchase state after mutation. | | <a id="mutationuseraddonassignmentremoveclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationuseraddonassignmentremoveerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationuseraddonassignmentremoveuser"></a>`user` | [`AddOnUser`](#addonuser) | User that the add-on was removed from. | ### `Mutation.userCalloutCreate` @@ -7094,7 +7270,7 @@ Input type: `VulnerabilitiesDismissInput` | <a id="mutationvulnerabilitiesdismissclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationvulnerabilitiesdismisscomment"></a>`comment` | [`String`](#string) | Comment why vulnerability was dismissed (maximum 50,000 characters). | | <a id="mutationvulnerabilitiesdismissdismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason why vulnerability should be dismissed. | -| <a id="mutationvulnerabilitiesdismissvulnerabilityids"></a>`vulnerabilityIds` | [`[VulnerabilityID!]!`](#vulnerabilityid) | IDs of the vulnerabilities to be dismissed. | +| <a id="mutationvulnerabilitiesdismissvulnerabilityids"></a>`vulnerabilityIds` | [`[VulnerabilityID!]!`](#vulnerabilityid) | IDs of the vulnerabilities to be dismissed (maximum 100 entries). | #### Fields @@ -7472,6 +7648,33 @@ Input type: `WorkItemExportInput` | <a id="mutationworkitemexporterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationworkitemexportmessage"></a>`message` | [`String`](#string) | Export request result message. | +### `Mutation.workItemRemoveLinkedItems` + +Remove items linked to the work item. + +WARNING: +**Introduced** in 16.3. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `WorkItemRemoveLinkedItemsInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemremovelinkeditemsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemremovelinkeditemsid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | +| <a id="mutationworkitemremovelinkeditemsworkitemsids"></a>`workItemsIds` | [`[WorkItemID!]!`](#workitemid) | Global IDs of the items to unlink. Maximum number of IDs you can provide: 3. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemremovelinkeditemsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemremovelinkeditemserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationworkitemremovelinkeditemsmessage"></a>`message` | [`String`](#string) | Linked items update result message. | +| <a id="mutationworkitemremovelinkeditemsworkitem"></a>`workItem` | [`WorkItem`](#workitem) | Updated work item. | + ### `Mutation.workItemSubscribe` WARNING: @@ -7704,51 +7907,51 @@ The edge type for [`AgentConfiguration`](#agentconfiguration). | <a id="agentconfigurationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="agentconfigurationedgenode"></a>`node` | [`AgentConfiguration`](#agentconfiguration) | The item at the end of the edge. | -#### `AiCachedMessageTypeConnection` +#### `AiChatMessageConnection` -The connection type for [`AiCachedMessageType`](#aicachedmessagetype). +The connection type for [`AiChatMessage`](#aichatmessage). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aicachedmessagetypeconnectionedges"></a>`edges` | [`[AiCachedMessageTypeEdge]`](#aicachedmessagetypeedge) | A list of edges. | -| <a id="aicachedmessagetypeconnectionnodes"></a>`nodes` | [`[AiCachedMessageType]`](#aicachedmessagetype) | A list of nodes. | -| <a id="aicachedmessagetypeconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | +| <a id="aichatmessageconnectionedges"></a>`edges` | [`[AiChatMessageEdge]`](#aichatmessageedge) | A list of edges. | +| <a id="aichatmessageconnectionnodes"></a>`nodes` | [`[AiChatMessage]`](#aichatmessage) | A list of nodes. | +| <a id="aichatmessageconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | -#### `AiCachedMessageTypeEdge` +#### `AiChatMessageEdge` -The edge type for [`AiCachedMessageType`](#aicachedmessagetype). +The edge type for [`AiChatMessage`](#aichatmessage). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aicachedmessagetypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="aicachedmessagetypeedgenode"></a>`node` | [`AiCachedMessageType`](#aicachedmessagetype) | The item at the end of the edge. | +| <a id="aichatmessageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="aichatmessageedgenode"></a>`node` | [`AiChatMessage`](#aichatmessage) | The item at the end of the edge. | -#### `AiMessageTypeConnection` +#### `AiMessageConnection` -The connection type for [`AiMessageType`](#aimessagetype). +The connection type for [`AiMessage`](#aimessage). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aimessagetypeconnectionedges"></a>`edges` | [`[AiMessageTypeEdge]`](#aimessagetypeedge) | A list of edges. | -| <a id="aimessagetypeconnectionnodes"></a>`nodes` | [`[AiMessageType]`](#aimessagetype) | A list of nodes. | -| <a id="aimessagetypeconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | +| <a id="aimessageconnectionedges"></a>`edges` | [`[AiMessageEdge]`](#aimessageedge) | A list of edges. | +| <a id="aimessageconnectionnodes"></a>`nodes` | [`[AiMessage]`](#aimessage) | A list of nodes. | +| <a id="aimessageconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | -#### `AiMessageTypeEdge` +#### `AiMessageEdge` -The edge type for [`AiMessageType`](#aimessagetype). +The edge type for [`AiMessage`](#aimessage). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aimessagetypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="aimessagetypeedgenode"></a>`node` | [`AiMessageType`](#aimessagetype) | The item at the end of the edge. | +| <a id="aimessageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="aimessageedgenode"></a>`node` | [`AiMessage`](#aimessage) | The item at the end of the edge. | #### `AlertManagementAlertConnection` @@ -8997,6 +9200,98 @@ The edge type for [`CustomerRelationsOrganization`](#customerrelationsorganizati | <a id="customerrelationsorganizationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="customerrelationsorganizationedgenode"></a>`node` | [`CustomerRelationsOrganization`](#customerrelationsorganization) | The item at the end of the edge. | +#### `CustomizableDashboardConnection` + +The connection type for [`CustomizableDashboard`](#customizabledashboard). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardconnectionedges"></a>`edges` | [`[CustomizableDashboardEdge]`](#customizabledashboardedge) | A list of edges. | +| <a id="customizabledashboardconnectionnodes"></a>`nodes` | [`[CustomizableDashboard]`](#customizabledashboard) | A list of nodes. | +| <a id="customizabledashboardconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CustomizableDashboardEdge` + +The edge type for [`CustomizableDashboard`](#customizabledashboard). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="customizabledashboardedgenode"></a>`node` | [`CustomizableDashboard`](#customizabledashboard) | The item at the end of the edge. | + +#### `CustomizableDashboardPanelConnection` + +The connection type for [`CustomizableDashboardPanel`](#customizabledashboardpanel). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardpanelconnectionedges"></a>`edges` | [`[CustomizableDashboardPanelEdge]`](#customizabledashboardpaneledge) | A list of edges. | +| <a id="customizabledashboardpanelconnectionnodes"></a>`nodes` | [`[CustomizableDashboardPanel]`](#customizabledashboardpanel) | A list of nodes. | +| <a id="customizabledashboardpanelconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CustomizableDashboardPanelEdge` + +The edge type for [`CustomizableDashboardPanel`](#customizabledashboardpanel). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardpaneledgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="customizabledashboardpaneledgenode"></a>`node` | [`CustomizableDashboardPanel`](#customizabledashboardpanel) | The item at the end of the edge. | + +#### `CustomizableDashboardVisualizationConnection` + +The connection type for [`CustomizableDashboardVisualization`](#customizabledashboardvisualization). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardvisualizationconnectionedges"></a>`edges` | [`[CustomizableDashboardVisualizationEdge]`](#customizabledashboardvisualizationedge) | A list of edges. | +| <a id="customizabledashboardvisualizationconnectionnodes"></a>`nodes` | [`[CustomizableDashboardVisualization]`](#customizabledashboardvisualization) | A list of nodes. | +| <a id="customizabledashboardvisualizationconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CustomizableDashboardVisualizationEdge` + +The edge type for [`CustomizableDashboardVisualization`](#customizabledashboardvisualization). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardvisualizationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="customizabledashboardvisualizationedgenode"></a>`node` | [`CustomizableDashboardVisualization`](#customizabledashboardvisualization) | The item at the end of the edge. | + +#### `CustomizablePermissionConnection` + +The connection type for [`CustomizablePermission`](#customizablepermission). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizablepermissionconnectionedges"></a>`edges` | [`[CustomizablePermissionEdge]`](#customizablepermissionedge) | A list of edges. | +| <a id="customizablepermissionconnectionnodes"></a>`nodes` | [`[CustomizablePermission]`](#customizablepermission) | A list of nodes. | +| <a id="customizablepermissionconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CustomizablePermissionEdge` + +The edge type for [`CustomizablePermission`](#customizablepermission). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizablepermissionedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="customizablepermissionedgenode"></a>`node` | [`CustomizablePermission`](#customizablepermission) | The item at the end of the edge. | + #### `DastProfileConnection` The connection type for [`DastProfile`](#dastprofile). @@ -9898,6 +10193,29 @@ The edge type for [`InstanceExternalAuditEventDestination`](#instanceexternalaud | <a id="instanceexternalauditeventdestinationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="instanceexternalauditeventdestinationedgenode"></a>`node` | [`InstanceExternalAuditEventDestination`](#instanceexternalauditeventdestination) | The item at the end of the edge. | +#### `InstanceGoogleCloudLoggingConfigurationTypeConnection` + +The connection type for [`InstanceGoogleCloudLoggingConfigurationType`](#instancegooglecloudloggingconfigurationtype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancegooglecloudloggingconfigurationtypeconnectionedges"></a>`edges` | [`[InstanceGoogleCloudLoggingConfigurationTypeEdge]`](#instancegooglecloudloggingconfigurationtypeedge) | A list of edges. | +| <a id="instancegooglecloudloggingconfigurationtypeconnectionnodes"></a>`nodes` | [`[InstanceGoogleCloudLoggingConfigurationType]`](#instancegooglecloudloggingconfigurationtype) | A list of nodes. | +| <a id="instancegooglecloudloggingconfigurationtypeconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `InstanceGoogleCloudLoggingConfigurationTypeEdge` + +The edge type for [`InstanceGoogleCloudLoggingConfigurationType`](#instancegooglecloudloggingconfigurationtype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancegooglecloudloggingconfigurationtypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="instancegooglecloudloggingconfigurationtypeedgenode"></a>`node` | [`InstanceGoogleCloudLoggingConfigurationType`](#instancegooglecloudloggingconfigurationtype) | The item at the end of the edge. | + #### `IssuableResourceLinkConnection` The connection type for [`IssuableResourceLink`](#issuableresourcelink). @@ -10570,6 +10888,29 @@ The edge type for [`OncallParticipantType`](#oncallparticipanttype). | <a id="oncallparticipanttypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="oncallparticipanttypeedgenode"></a>`node` | [`OncallParticipantType`](#oncallparticipanttype) | The item at the end of the edge. | +#### `OrganizationUserConnection` + +The connection type for [`OrganizationUser`](#organizationuser). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="organizationuserconnectionedges"></a>`edges` | [`[OrganizationUserEdge]`](#organizationuseredge) | A list of edges. | +| <a id="organizationuserconnectionnodes"></a>`nodes` | [`[OrganizationUser]`](#organizationuser) | A list of nodes. | +| <a id="organizationuserconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `OrganizationUserEdge` + +The edge type for [`OrganizationUser`](#organizationuser). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="organizationuseredgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="organizationuseredgenode"></a>`node` | [`OrganizationUser`](#organizationuser) | The item at the end of the edge. | + #### `PackageBaseConnection` The connection type for [`PackageBase`](#packagebase). @@ -10897,75 +11238,6 @@ The edge type for [`PipelineTrigger`](#pipelinetrigger). | <a id="pipelinetriggeredgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="pipelinetriggeredgenode"></a>`node` | [`PipelineTrigger`](#pipelinetrigger) | The item at the end of the edge. | -#### `ProductAnalyticsDashboardConnection` - -The connection type for [`ProductAnalyticsDashboard`](#productanalyticsdashboard). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardconnectionedges"></a>`edges` | [`[ProductAnalyticsDashboardEdge]`](#productanalyticsdashboardedge) | A list of edges. | -| <a id="productanalyticsdashboardconnectionnodes"></a>`nodes` | [`[ProductAnalyticsDashboard]`](#productanalyticsdashboard) | A list of nodes. | -| <a id="productanalyticsdashboardconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | - -#### `ProductAnalyticsDashboardEdge` - -The edge type for [`ProductAnalyticsDashboard`](#productanalyticsdashboard). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="productanalyticsdashboardedgenode"></a>`node` | [`ProductAnalyticsDashboard`](#productanalyticsdashboard) | The item at the end of the edge. | - -#### `ProductAnalyticsDashboardPanelConnection` - -The connection type for [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardpanelconnectionedges"></a>`edges` | [`[ProductAnalyticsDashboardPanelEdge]`](#productanalyticsdashboardpaneledge) | A list of edges. | -| <a id="productanalyticsdashboardpanelconnectionnodes"></a>`nodes` | [`[ProductAnalyticsDashboardPanel]`](#productanalyticsdashboardpanel) | A list of nodes. | -| <a id="productanalyticsdashboardpanelconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | - -#### `ProductAnalyticsDashboardPanelEdge` - -The edge type for [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardpaneledgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="productanalyticsdashboardpaneledgenode"></a>`node` | [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel) | The item at the end of the edge. | - -#### `ProductAnalyticsDashboardVisualizationConnection` - -The connection type for [`ProductAnalyticsDashboardVisualization`](#productanalyticsdashboardvisualization). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardvisualizationconnectionedges"></a>`edges` | [`[ProductAnalyticsDashboardVisualizationEdge]`](#productanalyticsdashboardvisualizationedge) | A list of edges. | -| <a id="productanalyticsdashboardvisualizationconnectionnodes"></a>`nodes` | [`[ProductAnalyticsDashboardVisualization]`](#productanalyticsdashboardvisualization) | A list of nodes. | -| <a id="productanalyticsdashboardvisualizationconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | - -#### `ProductAnalyticsDashboardVisualizationEdge` - -The edge type for [`ProductAnalyticsDashboardVisualization`](#productanalyticsdashboardvisualization). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardvisualizationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="productanalyticsdashboardvisualizationedgenode"></a>`node` | [`ProductAnalyticsDashboardVisualization`](#productanalyticsdashboardvisualization) | The item at the end of the edge. | - #### `ProjectConnection` The connection type for [`Project`](#project). @@ -12080,6 +12352,29 @@ The edge type for [`UserAchievement`](#userachievement). | <a id="userachievementedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="userachievementedgenode"></a>`node` | [`UserAchievement`](#userachievement) | The item at the end of the edge. | +#### `UserAddOnAssignmentConnection` + +The connection type for [`UserAddOnAssignment`](#useraddonassignment). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="useraddonassignmentconnectionedges"></a>`edges` | [`[UserAddOnAssignmentEdge]`](#useraddonassignmentedge) | A list of edges. | +| <a id="useraddonassignmentconnectionnodes"></a>`nodes` | [`[UserAddOnAssignment]`](#useraddonassignment) | A list of nodes. | +| <a id="useraddonassignmentconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `UserAddOnAssignmentEdge` + +The edge type for [`UserAddOnAssignment`](#useraddonassignment). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="useraddonassignmentedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="useraddonassignmentedgenode"></a>`node` | [`UserAddOnAssignment`](#useraddonassignment) | The item at the end of the edge. | + #### `UserCalloutConnection` The connection type for [`UserCallout`](#usercallout). @@ -12461,6 +12756,305 @@ Represents AddOn purchase for Namespace. | <a id="addonpurchasename"></a>`name` | [`String!`](#string) | Name of AddOn. | | <a id="addonpurchasepurchasedquantity"></a>`purchasedQuantity` | [`Int!`](#int) | Number of seats purchased. | +### `AddOnUser` + +A user with add-on data. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuseravatarurl"></a>`avatarUrl` | [`String`](#string) | URL of the user's avatar. | +| <a id="addonuserbio"></a>`bio` | [`String`](#string) | Bio of the user. | +| <a id="addonuserbot"></a>`bot` | [`Boolean!`](#boolean) | Indicates if the user is a bot. | +| <a id="addonusercallouts"></a>`callouts` | [`UserCalloutConnection`](#usercalloutconnection) | User callouts that belong to the user. (see [Connections](#connections)) | +| <a id="addonusercommitemail"></a>`commitEmail` | [`String`](#string) | User's default commit email. | +| <a id="addonusercreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of when the user was created. | +| <a id="addonuserdiscord"></a>`discord` | [`String`](#string) | Discord ID of the user. | +| <a id="addonuseremail"></a>`email` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.7. This was renamed. Use: [`User.publicEmail`](#userpublicemail). | +| <a id="addonuseremails"></a>`emails` | [`EmailConnection`](#emailconnection) | User's email addresses. (see [Connections](#connections)) | +| <a id="addonusergitpodenabled"></a>`gitpodEnabled` | [`Boolean`](#boolean) | Whether Gitpod is enabled at the user level. | +| <a id="addonusergroupcount"></a>`groupCount` | [`Int`](#int) | Group count for the user. | +| <a id="addonusergroupmemberships"></a>`groupMemberships` | [`GroupMemberConnection`](#groupmemberconnection) | Group memberships of the user. (see [Connections](#connections)) | +| <a id="addonuserid"></a>`id` | [`ID!`](#id) | ID of the user. | +| <a id="addonuseride"></a>`ide` | [`Ide`](#ide) | IDE settings. | +| <a id="addonuserjobtitle"></a>`jobTitle` | [`String`](#string) | Job title of the user. | +| <a id="addonuserlinkedin"></a>`linkedin` | [`String`](#string) | LinkedIn profile name of the user. | +| <a id="addonuserlocation"></a>`location` | [`String`](#string) | Location of the user. | +| <a id="addonusername"></a>`name` | [`String!`](#string) | Human-readable name of the user. Returns `****` if the user is a project bot and the requester does not have permission to view the project. | +| <a id="addonusernamespace"></a>`namespace` | [`Namespace`](#namespace) | Personal namespace of the user. | +| <a id="addonusernamespacecommitemails"></a>`namespaceCommitEmails` | [`NamespaceCommitEmailConnection`](#namespacecommitemailconnection) | User's custom namespace commit emails. (see [Connections](#connections)) | +| <a id="addonuserorganization"></a>`organization` | [`String`](#string) | Who the user represents or works for. | +| <a id="addonuserpreferencesgitpodpath"></a>`preferencesGitpodPath` | [`String`](#string) | Web path to the Gitpod section within user preferences. | +| <a id="addonuserprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | +| <a id="addonuserprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | +| <a id="addonuserpronouns"></a>`pronouns` | [`String`](#string) | Pronouns of the user. | +| <a id="addonuserpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | +| <a id="addonusersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | +| <a id="addonuserstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | +| <a id="addonuserstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="addonusertwitter"></a>`twitter` | [`String`](#string) | Twitter username of the user. | +| <a id="addonuseruserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is an Experiment. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | +| <a id="addonuseruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | +| <a id="addonuserusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | +| <a id="addonuserwebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | +| <a id="addonuserweburl"></a>`webUrl` | [`String!`](#string) | Web URL of the user. | + +#### Fields with arguments + +##### `AddOnUser.addOnAssignments` + +Add-on purchase assignments for the user. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`UserAddOnAssignmentConnection`](#useraddonassignmentconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuseraddonassignmentsaddonpurchaseids"></a>`addOnPurchaseIds` | [`[GitlabSubscriptionsAddOnPurchaseID!]!`](#gitlabsubscriptionsaddonpurchaseid) | Global IDs of the add on purchases to find assignments for. | + +##### `AddOnUser.assignedMergeRequests` + +Merge requests assigned to the user. + +Returns [`MergeRequestConnection`](#mergerequestconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuserassignedmergerequestsapproved"></a>`approved` | [`Boolean`](#boolean) | Limit results to approved merge requests. Available only when the feature flag `mr_approved_filter` is enabled. | +| <a id="addonuserassignedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="addonuserassignedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="addonuserassignedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | +| <a id="addonuserassignedmergerequestsdraft"></a>`draft` | [`Boolean`](#boolean) | Limit result to draft merge requests. | +| <a id="addonuserassignedmergerequestsgroupid"></a>`groupId` | [`GroupID`](#groupid) | The global ID of the group the authored merge requests should be in. Merge requests in subgroups are included. | +| <a id="addonuserassignedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | +| <a id="addonuserassignedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | +| <a id="addonuserassignedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | +| <a id="addonuserassignedmergerequestsmergedbefore"></a>`mergedBefore` | [`Time`](#time) | Merge requests merged before this date. | +| <a id="addonuserassignedmergerequestsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Title of the milestone. | +| <a id="addonuserassignedmergerequestsnot"></a>`not` | [`MergeRequestsResolverNegatedParams`](#mergerequestsresolvernegatedparams) | List of negated arguments. Warning: this argument is experimental and a subject to change in future. | +| <a id="addonuserassignedmergerequestsprojectid"></a>`projectId` | [`ProjectID`](#projectid) | The global ID of the project the authored merge requests should be in. Incompatible with projectPath. | +| <a id="addonuserassignedmergerequestsprojectpath"></a>`projectPath` | [`String`](#string) | The full-path of the project the authored merge requests should be in. Incompatible with projectId. | +| <a id="addonuserassignedmergerequestsreviewerusername"></a>`reviewerUsername` | [`String`](#string) | Username of the reviewer. | +| <a id="addonuserassignedmergerequestssort"></a>`sort` | [`MergeRequestSort`](#mergerequestsort) | Sort merge requests by this criteria. | +| <a id="addonuserassignedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | +| <a id="addonuserassignedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | +| <a id="addonuserassignedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="addonuserassignedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="addonuserassignedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | + +##### `AddOnUser.authoredMergeRequests` + +Merge requests authored by the user. + +Returns [`MergeRequestConnection`](#mergerequestconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuserauthoredmergerequestsapproved"></a>`approved` | [`Boolean`](#boolean) | Limit results to approved merge requests. Available only when the feature flag `mr_approved_filter` is enabled. | +| <a id="addonuserauthoredmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="addonuserauthoredmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="addonuserauthoredmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | +| <a id="addonuserauthoredmergerequestsdraft"></a>`draft` | [`Boolean`](#boolean) | Limit result to draft merge requests. | +| <a id="addonuserauthoredmergerequestsgroupid"></a>`groupId` | [`GroupID`](#groupid) | The global ID of the group the authored merge requests should be in. Merge requests in subgroups are included. | +| <a id="addonuserauthoredmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | +| <a id="addonuserauthoredmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | +| <a id="addonuserauthoredmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | +| <a id="addonuserauthoredmergerequestsmergedbefore"></a>`mergedBefore` | [`Time`](#time) | Merge requests merged before this date. | +| <a id="addonuserauthoredmergerequestsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Title of the milestone. | +| <a id="addonuserauthoredmergerequestsnot"></a>`not` | [`MergeRequestsResolverNegatedParams`](#mergerequestsresolvernegatedparams) | List of negated arguments. Warning: this argument is experimental and a subject to change in future. | +| <a id="addonuserauthoredmergerequestsprojectid"></a>`projectId` | [`ProjectID`](#projectid) | The global ID of the project the authored merge requests should be in. Incompatible with projectPath. | +| <a id="addonuserauthoredmergerequestsprojectpath"></a>`projectPath` | [`String`](#string) | The full-path of the project the authored merge requests should be in. Incompatible with projectId. | +| <a id="addonuserauthoredmergerequestsreviewerusername"></a>`reviewerUsername` | [`String`](#string) | Username of the reviewer. | +| <a id="addonuserauthoredmergerequestssort"></a>`sort` | [`MergeRequestSort`](#mergerequestsort) | Sort merge requests by this criteria. | +| <a id="addonuserauthoredmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | +| <a id="addonuserauthoredmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | +| <a id="addonuserauthoredmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="addonuserauthoredmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="addonuserauthoredmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | + +##### `AddOnUser.groups` + +Groups where the user has access. + +Returns [`GroupConnection`](#groupconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonusergroupspermissionscope"></a>`permissionScope` | [`GroupPermission`](#grouppermission) | Filter by permissions the user has on groups. | +| <a id="addonusergroupssearch"></a>`search` | [`String`](#string) | Search by group name or path. | + +##### `AddOnUser.reviewRequestedMergeRequests` + +Merge requests assigned to the user for review. + +Returns [`MergeRequestConnection`](#mergerequestconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuserreviewrequestedmergerequestsapproved"></a>`approved` | [`Boolean`](#boolean) | Limit results to approved merge requests. Available only when the feature flag `mr_approved_filter` is enabled. | +| <a id="addonuserreviewrequestedmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="addonuserreviewrequestedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="addonuserreviewrequestedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="addonuserreviewrequestedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | +| <a id="addonuserreviewrequestedmergerequestsdraft"></a>`draft` | [`Boolean`](#boolean) | Limit result to draft merge requests. | +| <a id="addonuserreviewrequestedmergerequestsgroupid"></a>`groupId` | [`GroupID`](#groupid) | The global ID of the group the authored merge requests should be in. Merge requests in subgroups are included. | +| <a id="addonuserreviewrequestedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | +| <a id="addonuserreviewrequestedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | +| <a id="addonuserreviewrequestedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | +| <a id="addonuserreviewrequestedmergerequestsmergedbefore"></a>`mergedBefore` | [`Time`](#time) | Merge requests merged before this date. | +| <a id="addonuserreviewrequestedmergerequestsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Title of the milestone. | +| <a id="addonuserreviewrequestedmergerequestsnot"></a>`not` | [`MergeRequestsResolverNegatedParams`](#mergerequestsresolvernegatedparams) | List of negated arguments. Warning: this argument is experimental and a subject to change in future. | +| <a id="addonuserreviewrequestedmergerequestsprojectid"></a>`projectId` | [`ProjectID`](#projectid) | The global ID of the project the authored merge requests should be in. Incompatible with projectPath. | +| <a id="addonuserreviewrequestedmergerequestsprojectpath"></a>`projectPath` | [`String`](#string) | The full-path of the project the authored merge requests should be in. Incompatible with projectId. | +| <a id="addonuserreviewrequestedmergerequestssort"></a>`sort` | [`MergeRequestSort`](#mergerequestsort) | Sort merge requests by this criteria. | +| <a id="addonuserreviewrequestedmergerequestssourcebranches"></a>`sourceBranches` | [`[String!]`](#string) | Array of source branch names. All resolved merge requests will have one of these branches as their source. | +| <a id="addonuserreviewrequestedmergerequestsstate"></a>`state` | [`MergeRequestState`](#mergerequeststate) | Merge request state. If provided, all resolved merge requests will have this state. | +| <a id="addonuserreviewrequestedmergerequeststargetbranches"></a>`targetBranches` | [`[String!]`](#string) | Array of target branch names. All resolved merge requests will have one of these branches as their target. | +| <a id="addonuserreviewrequestedmergerequestsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Merge requests updated after this timestamp. | +| <a id="addonuserreviewrequestedmergerequestsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Merge requests updated before this timestamp. | + +##### `AddOnUser.savedReply` + +Saved reply authored by the user. Will not return saved reply if `saved_replies` feature flag is disabled. + +Returns [`SavedReply`](#savedreply). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonusersavedreplyid"></a>`id` | [`UsersSavedReplyID!`](#userssavedreplyid) | ID of a saved reply. | + +##### `AddOnUser.snippets` + +Snippets authored by the user. + +Returns [`SnippetConnection`](#snippetconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonusersnippetsids"></a>`ids` | [`[SnippetID!]`](#snippetid) | Array of global snippet IDs. For example, `gid://gitlab/ProjectSnippet/1`. | +| <a id="addonusersnippetstype"></a>`type` | [`TypeEnum`](#typeenum) | Type of snippet. | +| <a id="addonusersnippetsvisibility"></a>`visibility` | [`VisibilityScopesEnum`](#visibilityscopesenum) | Visibility of the snippet. | + +##### `AddOnUser.starredProjects` + +Projects starred by the user. + +Returns [`ProjectConnection`](#projectconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuserstarredprojectssearch"></a>`search` | [`String`](#string) | Search query. | + +##### `AddOnUser.timelogs` + +Time logged by the user. + +Returns [`TimelogConnection`](#timelogconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonusertimelogsenddate"></a>`endDate` | [`Time`](#time) | List timelogs within a date range where the logged date is equal to or before endDate. | +| <a id="addonusertimelogsendtime"></a>`endTime` | [`Time`](#time) | List timelogs within a time range where the logged time is equal to or before endTime. | +| <a id="addonusertimelogsgroupid"></a>`groupId` | [`GroupID`](#groupid) | List timelogs for a group. | +| <a id="addonusertimelogsprojectid"></a>`projectId` | [`ProjectID`](#projectid) | List timelogs for a project. | +| <a id="addonusertimelogssort"></a>`sort` | [`TimelogSort`](#timelogsort) | List timelogs in a particular order. | +| <a id="addonusertimelogsstartdate"></a>`startDate` | [`Time`](#time) | List timelogs within a date range where the logged date is equal to or after startDate. | +| <a id="addonusertimelogsstarttime"></a>`startTime` | [`Time`](#time) | List timelogs within a time range where the logged time is equal to or after startTime. | +| <a id="addonusertimelogsusername"></a>`username` | [`String`](#string) | List timelogs for a user. | + +##### `AddOnUser.todos` + +To-do items of the user. + +Returns [`TodoConnection`](#todoconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonusertodosaction"></a>`action` | [`[TodoActionEnum!]`](#todoactionenum) | Action to be filtered. | +| <a id="addonusertodosauthorid"></a>`authorId` | [`[ID!]`](#id) | ID of an author. | +| <a id="addonusertodosgroupid"></a>`groupId` | [`[ID!]`](#id) | ID of a group. | +| <a id="addonusertodosprojectid"></a>`projectId` | [`[ID!]`](#id) | ID of a project. | +| <a id="addonusertodosstate"></a>`state` | [`[TodoStateEnum!]`](#todostateenum) | State of the todo. | +| <a id="addonusertodostype"></a>`type` | [`[TodoTargetEnum!]`](#todotargetenum) | Type of the todo. | + +##### `AddOnUser.workspaces` + +Workspaces owned by the current user. + +Returns [`WorkspaceConnection`](#workspaceconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="addonuserworkspacesids"></a>`ids` | [`[RemoteDevelopmentWorkspaceID!]`](#remotedevelopmentworkspaceid) | Array of global workspace IDs. For example, `["gid://gitlab/RemoteDevelopment::Workspace/1"]`. | +| <a id="addonuserworkspacesincludeactualstates"></a>`includeActualStates` | [`[String!]`](#string) | Includes all workspaces that match any of the actual states. | +| <a id="addonuserworkspacesprojectids"></a>`projectIds` | [`[ProjectID!]`](#projectid) | Filter workspaces by project id. | + ### `AgentConfiguration` Configuration details for an Agent. @@ -12484,30 +13078,44 @@ Information about a connected Agent. | <a id="agentmetadatapodnamespace"></a>`podNamespace` | [`String`](#string) | Namespace of the pod running the Agent. | | <a id="agentmetadataversion"></a>`version` | [`String`](#string) | Agent version tag. | -### `AiCachedMessageType` +### `AiChatMessage` + +GitLab Duo Chat message. #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aicachedmessagetypecontent"></a>`content` | [`String`](#string) | Content of the message. Can be null for failed responses. | -| <a id="aicachedmessagetypeerrors"></a>`errors` | [`[String!]!`](#string) | Errors that occurred while asynchronously fetching an AI (assistant) response. | -| <a id="aicachedmessagetypeid"></a>`id` | [`ID`](#id) | UUID of the message. | -| <a id="aicachedmessagetyperequestid"></a>`requestId` | [`ID`](#id) | UUID of the original request message. | -| <a id="aicachedmessagetyperole"></a>`role` | [`AiCachedMessageRole!`](#aicachedmessagerole) | Message role. | -| <a id="aicachedmessagetypetimestamp"></a>`timestamp` | [`Time!`](#time) | Message timestamp. | +| <a id="aichatmessagecontent"></a>`content` | [`String`](#string) | Content of the message. Can be null for failed responses. | +| <a id="aichatmessagecontenthtml"></a>`contentHtml` | [`String`](#string) | Content of the message in HTML format. Can be null for failed responses. | +| <a id="aichatmessageerrors"></a>`errors` | [`[String!]!`](#string) | Errors that occurred while asynchronously fetching an AI (assistant) response. | +| <a id="aichatmessageextras"></a>`extras` | [`AiMessageExtras`](#aimessageextras) | Extra message metadata. | +| <a id="aichatmessageid"></a>`id` | [`ID`](#id) | UUID of the message. | +| <a id="aichatmessagerequestid"></a>`requestId` | [`ID`](#id) | UUID of the original request message. Shared between chat prompt and response. | +| <a id="aichatmessagerole"></a>`role` | [`AiChatMessageRole!`](#aichatmessagerole) | Message role. | +| <a id="aichatmessagetimestamp"></a>`timestamp` | [`Time!`](#time) | Message timestamp. | -### `AiMessageType` +### `AiMessage` #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="aimessagetypecontent"></a>`content` | [`String`](#string) | Content of the message or null if loading. | -| <a id="aimessagetypeerrors"></a>`errors` | [`[String!]!`](#string) | Errors that occurred while asynchronously fetching an AI(assistant) response. | -| <a id="aimessagetypeid"></a>`id` | [`ID`](#id) | Global ID of the message. | -| <a id="aimessagetypeisfetching"></a>`isFetching` | [`Boolean`](#boolean) | Whether the content is still being fetched, for a message with the assistant role. | -| <a id="aimessagetyperole"></a>`role` | [`String!`](#string) | Role of the message (system, user, assistant). | +| <a id="aimessagecontent"></a>`content` | [`String`](#string) | Content of the message or null if loading. | +| <a id="aimessageerrors"></a>`errors` | [`[String!]!`](#string) | Errors that occurred while asynchronously fetching an AI(assistant) response. | +| <a id="aimessageid"></a>`id` | [`ID`](#id) | Global ID of the message. | +| <a id="aimessageisfetching"></a>`isFetching` | [`Boolean`](#boolean) | Whether the content is still being fetched, for a message with the assistant role. | +| <a id="aimessagerole"></a>`role` | [`String!`](#string) | Role of the message (system, user, assistant). | + +### `AiMessageExtras` + +Extra metadata for AI message. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="aimessageextrassources"></a>`sources` | [`[JSON!]`](#json) | Sources used to form the message. | ### `AiResponse` @@ -12515,11 +13123,18 @@ Information about a connected Agent. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="airesponsechunkid"></a>`chunkId` | [`Int`](#int) | Incremental ID for a chunk from a streamed response. Null when it is not a streamed response. | +| <a id="airesponsecontent"></a>`content` | [`String`](#string) | Raw response content. | +| <a id="airesponsecontenthtml"></a>`contentHtml` | [`String`](#string) | Response content as HTML. | | <a id="airesponseerrors"></a>`errors` | [`[String!]`](#string) | Errors return by AI API as response. | +| <a id="airesponseextras"></a>`extras` | [`AiMessageExtras`](#aimessageextras) | Extra message metadata. | +| <a id="airesponseid"></a>`id` | [`ID`](#id) | UUID of the message. | | <a id="airesponserequestid"></a>`requestId` | [`String`](#string) | ID of the original request. | -| <a id="airesponseresponsebody"></a>`responseBody` | [`String`](#string) | Response body from AI API. | -| <a id="airesponserole"></a>`role` | [`AiCachedMessageRole!`](#aicachedmessagerole) | Message role. | +| <a id="airesponseresponsebody"></a>`responseBody` **{warning-solid}** | [`String`](#string) | **Deprecated** in 16.4. Moved to content attribute. | +| <a id="airesponseresponsebodyhtml"></a>`responseBodyHtml` **{warning-solid}** | [`String`](#string) | **Deprecated** in 16.4. Moved to contentHtml attribute. | +| <a id="airesponserole"></a>`role` | [`AiChatMessageRole!`](#aichatmessagerole) | Message role. | | <a id="airesponsetimestamp"></a>`timestamp` | [`Time!`](#time) | Message timestamp. | +| <a id="airesponsetype"></a>`type` | [`AiMessageType`](#aimessagetype) | Message type. | ### `AlertManagementAlert` @@ -12754,6 +13369,7 @@ Represents a HTTP header key/value that belongs to an audit streaming destinatio | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="auditeventstreamingheaderactive"></a>`active` | [`Boolean!`](#boolean) | Header is active or not. | | <a id="auditeventstreamingheaderid"></a>`id` | [`ID!`](#id) | ID of the header. | | <a id="auditeventstreamingheaderkey"></a>`key` | [`String!`](#string) | Key of the header. | | <a id="auditeventstreamingheadervalue"></a>`value` | [`String!`](#string) | Value of the header. | @@ -12766,13 +13382,14 @@ Represents a HTTP header key/value that belongs to an instance level audit strea | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="auditeventsstreaminginstanceheaderactive"></a>`active` | [`Boolean!`](#boolean) | Header is active or not. | | <a id="auditeventsstreaminginstanceheaderid"></a>`id` | [`ID!`](#id) | ID of the header. | | <a id="auditeventsstreaminginstanceheaderkey"></a>`key` | [`String!`](#string) | Key of the header. | | <a id="auditeventsstreaminginstanceheadervalue"></a>`value` | [`String!`](#string) | Value of the header. | ### `AutocompletedUser` -Core represention of a GitLab user. +Core representation of a GitLab user. #### Fields @@ -13086,6 +13703,15 @@ An emoji awarded by a user. | <a id="baseserviceservicetype"></a>`serviceType` | [`ServiceType`](#servicetype) | Type of the service. | | <a id="baseservicetype"></a>`type` | [`String`](#string) | Class name of the service. | +### `Blame` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="blamefirstline"></a>`firstLine` | [`String`](#string) | First line of Git Blame for given range. | +| <a id="blamegroups"></a>`groups` | [`[Groups!]`](#groups) | Git Blame grouped by contiguous lines for commit. | + ### `Blob` #### Fields @@ -13710,7 +14336,8 @@ CI/CD variables for a GitLab instance. | <a id="cijobpipeline"></a>`pipeline` | [`Pipeline`](#pipeline) | Pipeline the job belongs to. | | <a id="cijobplaypath"></a>`playPath` | [`String`](#string) | Play path of the job. | | <a id="cijobplayable"></a>`playable` | [`Boolean!`](#boolean) | Indicates the job can be played. | -| <a id="cijobpreviousstagejobsorneeds"></a>`previousStageJobsOrNeeds` | [`JobNeedUnionConnection`](#jobneedunionconnection) | Jobs that must complete before the job runs. Returns `BuildNeed`, which is the needed jobs if the job uses the `needs` keyword, or the previous stage jobs otherwise. (see [Connections](#connections)) | +| <a id="cijobpreviousstagejobs"></a>`previousStageJobs` | [`CiJobConnection`](#cijobconnection) | Jobs from the previous stage. (see [Connections](#connections)) | +| <a id="cijobpreviousstagejobsorneeds"></a>`previousStageJobsOrNeeds` **{warning-solid}** | [`JobNeedUnionConnection`](#jobneedunionconnection) | **Deprecated** in 16.4. Replaced by previousStageJobs and needs fields. | | <a id="cijobproject"></a>`project` | [`Project`](#project) | Project that the job belongs to. | | <a id="cijobqueuedat"></a>`queuedAt` | [`Time`](#time) | When the job was enqueued and marked as pending. | | <a id="cijobqueuedduration"></a>`queuedDuration` | [`Duration`](#duration) | How long the job was enqueued before starting. | @@ -13759,11 +14386,23 @@ CI/CD variables for a GitLab instance. ### `CiJobTrace` -#### Fields +#### Fields with arguments + +##### `CiJobTrace.htmlSummary` + +HTML summary that contains the tail lines of the trace. Returns at most 16KB of raw bytes from the trace. The returned string might start with an unexpected invalid UTF-8 code point due to truncation. + +WARNING: +**Introduced** in 15.11. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`String!`](#string). + +###### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="cijobtracehtmlsummary"></a>`htmlSummary` **{warning-solid}** | [`String!`](#string) | **Introduced** in 15.11. This feature is an Experiment. It can be changed or removed at any time. HTML summary containing the last 10 lines of the trace. | +| <a id="cijobtracehtmlsummarylastlines"></a>`lastLines` | [`Int`](#int) | Number of tail lines to return, up to a maximum of 100 lines. | ### `CiJobsDurationStatistics` @@ -14141,6 +14780,58 @@ Code Quality report for a pipeline. | <a id="codequalityreportsummaryminor"></a>`minor` | [`Int`](#int) | Total number of minor status. | | <a id="codequalityreportsummaryunknown"></a>`unknown` | [`Int`](#int) | Total number of unknown status. | +### `CodequalityReportsComparer` + +Represents reports comparison for code quality. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="codequalityreportscomparerreport"></a>`report` | [`CodequalityReportsComparerReport`](#codequalityreportscomparerreport) | Compared codequality report. | + +### `CodequalityReportsComparerReport` + +Represents compared code quality report. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="codequalityreportscomparerreportexistingerrors"></a>`existingErrors` | [`[CodequalityReportsComparerReportDegradation!]`](#codequalityreportscomparerreportdegradation) | All code quality degradations. | +| <a id="codequalityreportscomparerreportnewerrors"></a>`newErrors` | [`[CodequalityReportsComparerReportDegradation!]!`](#codequalityreportscomparerreportdegradation) | New code quality degradations. | +| <a id="codequalityreportscomparerreportresolvederrors"></a>`resolvedErrors` | [`[CodequalityReportsComparerReportDegradation!]`](#codequalityreportscomparerreportdegradation) | Resolved code quality degradations. | +| <a id="codequalityreportscomparerreportstatus"></a>`status` | [`CodequalityReportsComparerReportStatus!`](#codequalityreportscomparerreportstatus) | Status of report. | +| <a id="codequalityreportscomparerreportsummary"></a>`summary` | [`CodequalityReportsComparerReportSummary!`](#codequalityreportscomparerreportsummary) | Codequality report summary. | + +### `CodequalityReportsComparerReportDegradation` + +Represents a degradation on the compared codequality report. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="codequalityreportscomparerreportdegradationdescription"></a>`description` | [`String!`](#string) | Description of the code quality degradation. | +| <a id="codequalityreportscomparerreportdegradationenginename"></a>`engineName` | [`String!`](#string) | Code quality plugin that reported the degradation. | +| <a id="codequalityreportscomparerreportdegradationfilepath"></a>`filePath` | [`String!`](#string) | Relative path to the file containing the code quality degradation. | +| <a id="codequalityreportscomparerreportdegradationfingerprint"></a>`fingerprint` | [`String!`](#string) | Unique fingerprint to identify the code quality degradation. For example, an MD5 hash. | +| <a id="codequalityreportscomparerreportdegradationline"></a>`line` | [`Int!`](#int) | Line on which the code quality degradation occurred. | +| <a id="codequalityreportscomparerreportdegradationseverity"></a>`severity` | [`CodeQualityDegradationSeverity!`](#codequalitydegradationseverity) | Severity of the code quality degradation (BLOCKER, CRITICAL, MAJOR, MINOR, INFO, UNKNOWN). | +| <a id="codequalityreportscomparerreportdegradationweburl"></a>`webUrl` | [`String`](#string) | URL to the file along with line number. | + +### `CodequalityReportsComparerReportSummary` + +Represents a summary of the compared codequality report. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="codequalityreportscomparerreportsummaryerrored"></a>`errored` | [`Int`](#int) | Count of code quality errors. | +| <a id="codequalityreportscomparerreportsummaryresolved"></a>`resolved` | [`Int`](#int) | Count of resolved code quality degradations. | +| <a id="codequalityreportscomparerreportsummarytotal"></a>`total` | [`Int`](#int) | Total count of code quality degradations. | + ### `Commit` #### Fields @@ -14196,6 +14887,19 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="commitpipelinesupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Pipelines updated before this date. | | <a id="commitpipelinesusername"></a>`username` | [`String`](#string) | Filter pipelines by the user that triggered the pipeline. | +### `CommitData` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="commitdataagemapclass"></a>`ageMapClass` | [`String!`](#string) | CSS class for age of commit. | +| <a id="commitdataauthoravatar"></a>`authorAvatar` | [`String!`](#string) | Link to author avatar. | +| <a id="commitdatacommitauthorlink"></a>`commitAuthorLink` | [`String!`](#string) | Link to the commit author. | +| <a id="commitdatacommitlink"></a>`commitLink` | [`String!`](#string) | Link to the commit. | +| <a id="commitdataprojectblamelink"></a>`projectBlameLink` | [`String`](#string) | Link to blame prior to the change. | +| <a id="commitdatatimeagotooltip"></a>`timeAgoTooltip` | [`String!`](#string) | Time of commit. | + ### `CommitParentNames` #### Fields @@ -14302,7 +15006,7 @@ Represents a ComplianceFramework associated with a Project. | <a id="complianceframeworkdescription"></a>`description` | [`String!`](#string) | Description of the compliance framework. | | <a id="complianceframeworkid"></a>`id` | [`ID!`](#id) | Compliance framework ID. | | <a id="complianceframeworkname"></a>`name` | [`String!`](#string) | Name of the compliance framework. | -| <a id="complianceframeworkpipelineconfigurationfullpath"></a>`pipelineConfigurationFullPath` | [`String`](#string) | Full path of the compliance pipeline configuration stored in a project repository, such as `.gitlab/.compliance-gitlab-ci.yml@compliance/hipaa` **(ULTIMATE)**. | +| <a id="complianceframeworkpipelineconfigurationfullpath"></a>`pipelineConfigurationFullPath` | [`String`](#string) | Full path of the compliance pipeline configuration stored in a project repository, such as `.gitlab/.compliance-gitlab-ci.yml@compliance/hipaa` **(ULTIMATE ALL)**. | ### `ComplianceStandardsAdherence` @@ -14636,6 +15340,61 @@ A custom emoji uploaded by user. | <a id="customerrelationsorganizationname"></a>`name` | [`String!`](#string) | Name of the organization. | | <a id="customerrelationsorganizationupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp the organization was last updated. | +### `CustomizableDashboard` + +Represents a product analytics dashboard. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardcategory"></a>`category` | [`CustomizableDashboardCategory!`](#customizabledashboardcategory) | Category of dashboard. | +| <a id="customizabledashboardconfigurationproject"></a>`configurationProject` | [`Project`](#project) | Project which contains the dashboard definition. | +| <a id="customizabledashboarddescription"></a>`description` | [`String`](#string) | Description of the dashboard. | +| <a id="customizabledashboardpanels"></a>`panels` | [`CustomizableDashboardPanelConnection!`](#customizabledashboardpanelconnection) | Panels shown on the dashboard. (see [Connections](#connections)) | +| <a id="customizabledashboardslug"></a>`slug` | [`String!`](#string) | Slug of the dashboard. | +| <a id="customizabledashboardtitle"></a>`title` | [`String!`](#string) | Title of the dashboard. | +| <a id="customizabledashboarduserdefined"></a>`userDefined` | [`Boolean!`](#boolean) | Indicates whether the dashboard is user-defined or provided by GitLab. | + +### `CustomizableDashboardPanel` + +Represents a product analytics dashboard panel. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardpanelgridattributes"></a>`gridAttributes` | [`JSON`](#json) | Description of the position and size of the panel. | +| <a id="customizabledashboardpanelqueryoverrides"></a>`queryOverrides` | [`JSON`](#json) | Overrides for the visualization query object. | +| <a id="customizabledashboardpaneltitle"></a>`title` | [`String!`](#string) | Title of the panel. | +| <a id="customizabledashboardpanelvisualization"></a>`visualization` | [`CustomizableDashboardVisualization!`](#customizabledashboardvisualization) | Visualization of the panel. | + +### `CustomizableDashboardVisualization` + +Represents a product analytics dashboard visualization. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizabledashboardvisualizationdata"></a>`data` | [`JSON!`](#json) | Data of the visualization. | +| <a id="customizabledashboardvisualizationerrors"></a>`errors` | [`[String!]`](#string) | Validation errors in the visualization. | +| <a id="customizabledashboardvisualizationoptions"></a>`options` | [`JSON!`](#json) | Options of the visualization. | +| <a id="customizabledashboardvisualizationslug"></a>`slug` | [`String!`](#string) | Slug of the visualization. | +| <a id="customizabledashboardvisualizationtype"></a>`type` | [`String!`](#string) | Type of the visualization. | + +### `CustomizablePermission` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="customizablepermissionavailablefor"></a>`availableFor` | [`[String!]!`](#string) | Objects the permission is available for. | +| <a id="customizablepermissiondescription"></a>`description` | [`String`](#string) | Description of the permission. | +| <a id="customizablepermissionname"></a>`name` | [`String!`](#string) | Localized name of the permission. | +| <a id="customizablepermissionrequirement"></a>`requirement` | [`String`](#string) | Requirement of the permission. | +| <a id="customizablepermissionvalue"></a>`value` | [`String!`](#string) | Value of the permission. | + ### `DastPreScanVerification` Represents a DAST Pre Scan Verification. @@ -14763,7 +15522,7 @@ Represents a DAST Site Profile. ### `DastSiteProfileAuth` -Input type for DastSiteProfile authentication. +DastSiteProfile authentication. #### Fields @@ -14831,6 +15590,7 @@ A software dependency used by a project. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="dependencyid"></a>`id` | [`GlobalID!`](#globalid) | ID of the dependency. | +| <a id="dependencylicenses"></a>`licenses` | [`[License!]`](#license) | Licenses associated to the dependency. | | <a id="dependencylocation"></a>`location` | [`Location`](#location) | Information about where the dependency is located. | | <a id="dependencyname"></a>`name` | [`String!`](#string) | Name of the dependency. | | <a id="dependencypackager"></a>`packager` | [`PackageManager`](#packagemanager) | Description of the tool used to manage the dependency. | @@ -15881,6 +16641,7 @@ Relationship between an epic and an issue. | <a id="epicissueepicissueid"></a>`epicIssueId` | [`ID!`](#id) | ID of the epic-issue relation. | | <a id="epicissueescalationpolicy"></a>`escalationPolicy` | [`EscalationPolicyType`](#escalationpolicytype) | Escalation policy associated with the issue. Available for issues which support escalation. | | <a id="epicissueescalationstatus"></a>`escalationStatus` | [`IssueEscalationStatus`](#issueescalationstatus) | Escalation status of the issue. | +| <a id="epicissueexternalauthor"></a>`externalAuthor` | [`String`](#string) | Email address of non-GitLab user reporting the issue. For guests, the email address is obfuscated. | | <a id="epicissuehasepic"></a>`hasEpic` | [`Boolean!`](#boolean) | Indicates if the issue belongs to an epic. Can return true and not show an associated epic when the user has no access to the epic. | | <a id="epicissuehealthstatus"></a>`healthStatus` | [`HealthStatus`](#healthstatus) | Current health status. | | <a id="epicissuehidden"></a>`hidden` | [`Boolean`](#boolean) | Indicates the issue is hidden because the author has been banned. | @@ -16321,7 +17082,7 @@ four standard [pagination arguments](#connection-pagination-arguments): ##### `GeoNode.designManagementRepositoryRegistries` -Find Design Repository registries on this Geo node. Ignored if `geo_design_management_repository_replication` feature flag is disabled. +Find Design Management Repository registries on this Geo node. WARNING: **Introduced** in 16.1. @@ -16583,7 +17344,7 @@ Stores Google Cloud Logging configurations associated with IAM service accounts, | <a id="googlecloudloggingconfigurationtypegroup"></a>`group` | [`Group!`](#group) | Group the configuration belongs to. | | <a id="googlecloudloggingconfigurationtypeid"></a>`id` | [`ID!`](#id) | ID of the configuration. | | <a id="googlecloudloggingconfigurationtypelogidname"></a>`logIdName` | [`String!`](#string) | Log ID. | -| <a id="googlecloudloggingconfigurationtypeprivatekey"></a>`privateKey` | [`String!`](#string) | Private key. | +| <a id="googlecloudloggingconfigurationtypename"></a>`name` | [`String!`](#string) | Name of the external destination to send audit events to. | ### `GpgSignature` @@ -16680,7 +17441,6 @@ GPG signature for a signed commit. | <a id="groupvisibility"></a>`visibility` | [`String`](#string) | Visibility of the namespace. | | <a id="groupvulnerabilityscanners"></a>`vulnerabilityScanners` | [`VulnerabilityScannerConnection`](#vulnerabilityscannerconnection) | Vulnerability scanners reported on the project vulnerabilities of the group and its subgroups. (see [Connections](#connections)) | | <a id="groupweburl"></a>`webUrl` | [`String!`](#string) | Web URL of the group. | -| <a id="groupworkitems"></a>`workItems` **{warning-solid}** | [`WorkItemConnection`](#workitemconnection) | **Introduced** in 16.3. This feature is an Experiment. It can be changed or removed at any time. Work items that belong to the namespace. | #### Fields with arguments @@ -16899,6 +17659,47 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupcontributionsfrom"></a>`from` | [`ISO8601Date!`](#iso8601date) | Start date of the reporting time range. | | <a id="groupcontributionsto"></a>`to` | [`ISO8601Date!`](#iso8601date) | End date of the reporting time range. The end date must be within 93 days after the start date. | +##### `Group.customizableDashboardVisualizations` + +Visualizations of the group or associated configuration project. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`CustomizableDashboardVisualizationConnection`](#customizabledashboardvisualizationconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupcustomizabledashboardvisualizationsslug"></a>`slug` | [`String`](#string) | Slug of the visualization to return. | + +##### `Group.customizableDashboards` + +Customizable dashboards for the group. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`CustomizableDashboardConnection`](#customizabledashboardconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupcustomizabledashboardscategory"></a>`category` | [`CustomizableDashboardCategory`](#customizabledashboardcategory) | Find by dashboard type. | +| <a id="groupcustomizabledashboardsslug"></a>`slug` | [`String`](#string) | Find by dashboard slug. | + ##### `Group.dataTransfer` Data transfer data point for a specific period. This is mocked data under a development feature flag. @@ -17461,6 +18262,23 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="grouptimelogsstarttime"></a>`startTime` | [`Time`](#time) | List timelogs within a time range where the logged time is equal to or after startTime. | | <a id="grouptimelogsusername"></a>`username` | [`String`](#string) | List timelogs for a user. | +##### `Group.valueStreamDashboardUsageOverview` + +Aggregated usage counts within the group. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`ValueStreamDashboardCount`](#valuestreamdashboardcount). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupvaluestreamdashboardusageoverviewidentifier"></a>`identifier` | [`ValueStreamDashboardMetric!`](#valuestreamdashboardmetric) | Type of counts to retrieve. | +| <a id="groupvaluestreamdashboardusageoverviewtimeframe"></a>`timeframe` | [`Timeframe!`](#timeframe) | Counts recorded during this time frame, usually from beginning of the month until the end of the month (the system runs monthly aggregations). | + ##### `Group.vulnerabilities` Vulnerabilities reported on the projects in the group and its subgroups. @@ -17479,6 +18297,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="groupvulnerabilitiesdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. Only dismissed Vulnerabilities will be included with the filter. | | <a id="groupvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | +| <a id="groupvulnerabilitieshasmergerequest"></a>`hasMergeRequest` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked merge requests. | | <a id="groupvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="groupvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="groupvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | @@ -17530,6 +18349,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="groupvulnerabilityseveritiescountclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | +| <a id="groupvulnerabilityseveritiescountdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. | | <a id="groupvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | | <a id="groupvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="groupvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -17540,6 +18360,22 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | <a id="groupvulnerabilityseveritiescountseverity"></a>`severity` | [`[VulnerabilitySeverity!]`](#vulnerabilityseverity) | Filter vulnerabilities by severity. | | <a id="groupvulnerabilityseveritiescountstate"></a>`state` | [`[VulnerabilityState!]`](#vulnerabilitystate) | Filter vulnerabilities by state. | +##### `Group.workItem` + +Find a work item by IID directly associated with the group. Returns `null` if the `namespace_level_work_items` feature flag is disabled. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`WorkItem`](#workitem). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupworkitemiid"></a>`iid` | [`String!`](#string) | IID of the work item. | + ##### `Group.workItemTypes` Work item types available to the group. @@ -17556,6 +18392,35 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="groupworkitemtypestaskable"></a>`taskable` | [`Boolean`](#boolean) | If `true`, only taskable work item types will be returned. Argument is experimental and can be removed in the future without notice. | +##### `Group.workItems` + +Work items that belong to the namespace. + +WARNING: +**Introduced** in 16.3. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`WorkItemConnection`](#workitemconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupworkitemsauthorusername"></a>`authorUsername` **{warning-solid}** | [`String`](#string) | **Introduced** in 15.9. This feature is an Experiment. It can be changed or removed at any time. Filter work items by author username. | +| <a id="groupworkitemsiid"></a>`iid` | [`String`](#string) | IID of the work item. For example, "1". | +| <a id="groupworkitemsiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of work items. For example, `["1", "2"]`. | +| <a id="groupworkitemsin"></a>`in` | [`[IssuableSearchableField!]`](#issuablesearchablefield) | Specify the fields to perform the search in. Defaults to `[TITLE, DESCRIPTION]`. Requires the `search` argument.'. | +| <a id="groupworkitemsrequirementlegacywidget"></a>`requirementLegacyWidget` **{warning-solid}** | [`RequirementLegacyFilterInput`](#requirementlegacyfilterinput) | **Deprecated** in 15.9. Use work item IID filter instead. | +| <a id="groupworkitemssearch"></a>`search` | [`String`](#string) | Search query for title or description. | +| <a id="groupworkitemssort"></a>`sort` | [`WorkItemSort`](#workitemsort) | Sort work items by criteria. | +| <a id="groupworkitemsstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of the work item. | +| <a id="groupworkitemsstatuswidget"></a>`statusWidget` | [`StatusFilterInput`](#statusfilterinput) | Input for status widget filter. Ignored if `work_items_mvc_2` is disabled. | +| <a id="groupworkitemstypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter work items by the given work item types. | + ### `GroupDataTransfer` #### Fields @@ -17755,6 +18620,18 @@ Represents the Geo sync and verification state of a group wiki repository. | <a id="groupwikirepositoryregistryverificationstate"></a>`verificationState` | [`VerificationStateEnum`](#verificationstateenum) | Verification state of the GroupWikiRepositoryRegistry. | | <a id="groupwikirepositoryregistryverifiedat"></a>`verifiedAt` | [`Time`](#time) | Timestamp of the most recent successful verification of the GroupWikiRepositoryRegistry. | +### `Groups` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupscommit"></a>`commit` | [`Commit!`](#commit) | Commit responsible for specified group. | +| <a id="groupscommitdata"></a>`commitData` | [`CommitData`](#commitdata) | HTML data derived from commit needed to present blame. | +| <a id="groupslineno"></a>`lineno` | [`Int!`](#int) | Starting line number for the commit group. | +| <a id="groupslines"></a>`lines` | [`[String!]!`](#string) | Array of lines added for the commit group. | +| <a id="groupsspan"></a>`span` | [`Int!`](#int) | Number of contiguous lines which the blame spans for the commit group. | + ### `HelmFileMetadata` Helm file metadata. @@ -17888,6 +18765,20 @@ Represents an external resource to send instance audit events to. | <a id="instanceexternalauditeventdestinationname"></a>`name` | [`String!`](#string) | Name of the external destination to send audit events to. | | <a id="instanceexternalauditeventdestinationverificationtoken"></a>`verificationToken` | [`String!`](#string) | Verification token to validate source of event. | +### `InstanceGoogleCloudLoggingConfigurationType` + +Stores instance level Google Cloud Logging configurations associated with IAM service accounts,used for generating access tokens. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancegooglecloudloggingconfigurationtypeclientemail"></a>`clientEmail` | [`String!`](#string) | Client email. | +| <a id="instancegooglecloudloggingconfigurationtypegoogleprojectidname"></a>`googleProjectIdName` | [`String!`](#string) | Google project ID. | +| <a id="instancegooglecloudloggingconfigurationtypeid"></a>`id` | [`ID!`](#id) | ID of the configuration. | +| <a id="instancegooglecloudloggingconfigurationtypelogidname"></a>`logIdName` | [`String!`](#string) | Log ID. | +| <a id="instancegooglecloudloggingconfigurationtypename"></a>`name` | [`String!`](#string) | Name of the external destination to send audit events to. | + ### `InstanceSecurityDashboard` #### Fields @@ -17954,6 +18845,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="instancesecuritydashboardvulnerabilityseveritiescountclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | +| <a id="instancesecuritydashboardvulnerabilityseveritiescountdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. | | <a id="instancesecuritydashboardvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | | <a id="instancesecuritydashboardvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -18010,6 +18902,7 @@ Describes an issuable resource link for incident issues. | <a id="issueepic"></a>`epic` | [`Epic`](#epic) | Epic to which this issue belongs. | | <a id="issueescalationpolicy"></a>`escalationPolicy` | [`EscalationPolicyType`](#escalationpolicytype) | Escalation policy associated with the issue. Available for issues which support escalation. | | <a id="issueescalationstatus"></a>`escalationStatus` | [`IssueEscalationStatus`](#issueescalationstatus) | Escalation status of the issue. | +| <a id="issueexternalauthor"></a>`externalAuthor` | [`String`](#string) | Email address of non-GitLab user reporting the issue. For guests, the email address is obfuscated. | | <a id="issuehasepic"></a>`hasEpic` | [`Boolean!`](#boolean) | Indicates if the issue belongs to an epic. Can return true and not show an associated epic when the user has no access to the epic. | | <a id="issuehealthstatus"></a>`healthStatus` | [`HealthStatus`](#healthstatus) | Current health status. | | <a id="issuehidden"></a>`hidden` | [`Boolean`](#boolean) | Indicates the issue is hidden because the author has been banned. | @@ -18340,6 +19233,7 @@ Represents an SSH key. | <a id="labeldescription"></a>`description` | [`String`](#string) | Description of the label (Markdown rendered as HTML for caching). | | <a id="labeldescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="labelid"></a>`id` | [`ID!`](#id) | Label ID. | +| <a id="labellockonmerge"></a>`lockOnMerge` | [`Boolean!`](#boolean) | Indicates this label is locked for merge requests that have been merged. | | <a id="labeltextcolor"></a>`textColor` | [`String!`](#string) | Text color of the label. | | <a id="labeltitle"></a>`title` | [`String!`](#string) | Content of the label. | | <a id="labelupdatedat"></a>`updatedAt` | [`Time!`](#time) | When this label was last updated. | @@ -18368,6 +19262,15 @@ Represents the Geo sync and verification state of an LFS object. | <a id="lfsobjectregistryverificationstate"></a>`verificationState` | [`VerificationStateEnum`](#verificationstateenum) | Verification state of the LfsObjectRegistry. | | <a id="lfsobjectregistryverifiedat"></a>`verifiedAt` | [`Time`](#time) | Timestamp of the most recent successful verification of the LfsObjectRegistry. | +### `License` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="licensename"></a>`name` | [`String!`](#string) | Name of the license. | +| <a id="licenseurl"></a>`url` | [`String!`](#string) | License URL in relation to SPDX. | + ### `LicenseHistoryEntry` Represents an entry from the Cloud License history. @@ -18457,6 +19360,7 @@ Defines which user roles, users, or groups can merge into a protected branch. | <a id="mergerequestautomergestrategy"></a>`autoMergeStrategy` | [`String`](#string) | Selected auto merge strategy. | | <a id="mergerequestavailableautomergestrategies"></a>`availableAutoMergeStrategies` | [`[String!]`](#string) | Array of available auto merge strategies. | | <a id="mergerequestawardemoji"></a>`awardEmoji` | [`AwardEmojiConnection`](#awardemojiconnection) | List of emoji reactions associated with the merge request. (see [Connections](#connections)) | +| <a id="mergerequestcodequalityreportscomparer"></a>`codequalityReportsComparer` **{warning-solid}** | [`CodequalityReportsComparer`](#codequalityreportscomparer) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Code quality reports comparison reported on the merge request. Returns `null` if `sast_reports_in_inline_diff` feature flag is disabled. | | <a id="mergerequestcommenters"></a>`commenters` | [`UserCoreConnection!`](#usercoreconnection) | All commenters on this noteable. (see [Connections](#connections)) | | <a id="mergerequestcommitcount"></a>`commitCount` | [`Int`](#int) | Number of commits in the merge request. | | <a id="mergerequestcommits"></a>`commits` | [`CommitConnection`](#commitconnection) | Merge request commits. (see [Connections](#connections)) | @@ -18523,6 +19427,7 @@ Defines which user roles, users, or groups can merge into a protected branch. | <a id="mergerequeststate"></a>`state` | [`MergeRequestState!`](#mergerequeststate) | State of the merge request. | | <a id="mergerequestsubscribed"></a>`subscribed` | [`Boolean!`](#boolean) | Indicates if the currently logged in user is subscribed to this merge request. | | <a id="mergerequestsuggestedreviewers"></a>`suggestedReviewers` **{warning-solid}** | [`SuggestedReviewersType`](#suggestedreviewerstype) | **Introduced** in 15.4. This feature is an Experiment. It can be changed or removed at any time. Suggested reviewers for merge request. Returns `null` if `suggested_reviewers` feature flag is disabled. This flag is disabled by default and only available on GitLab.com because the feature is experimental and is subject to change without notice. | +| <a id="mergerequestsupportslockonmerge"></a>`supportsLockOnMerge` | [`Boolean!`](#boolean) | Indicates if the merge request supports locked labels. | | <a id="mergerequesttargetbranch"></a>`targetBranch` | [`String!`](#string) | Target branch of the merge request. | | <a id="mergerequesttargetbranchexists"></a>`targetBranchExists` | [`Boolean!`](#boolean) | Indicates if the target branch of the merge request exists. | | <a id="mergerequesttargetproject"></a>`targetProject` | [`Project!`](#project) | Target project of the merge request. | @@ -19554,6 +20459,7 @@ A review summary generated by AI. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestreviewllmsummarycontent"></a>`content` | [`String!`](#string) | Content of the review summary. | +| <a id="mergerequestreviewllmsummarycontenthtml"></a>`contentHtml` | [`String!`](#string) | HTML content of the review summary, converted from Markdown. | | <a id="mergerequestreviewllmsummarycreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp of when the review summary was created. | | <a id="mergerequestreviewllmsummarymergerequestdiffid"></a>`mergeRequestDiffId` | [`ID!`](#id) | ID of the Merge Request diff associated with the review summary. | | <a id="mergerequestreviewllmsummaryprovider"></a>`provider` | [`String!`](#string) | AI provider that generated the summary. | @@ -20222,6 +21128,40 @@ Active period time range for on-call rotation. | <a id="oncallrotationactiveperiodtypeendtime"></a>`endTime` | [`String`](#string) | End of the rotation active period. | | <a id="oncallrotationactiveperiodtypestarttime"></a>`startTime` | [`String`](#string) | Start of the rotation active period. | +### `Organization` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="organizationid"></a>`id` **{warning-solid}** | [`ID!`](#id) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. ID of the organization. | +| <a id="organizationname"></a>`name` **{warning-solid}** | [`String!`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Name of the organization. | +| <a id="organizationorganizationusers"></a>`organizationUsers` **{warning-solid}** | [`OrganizationUserConnection!`](#organizationuserconnection) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Users with access to the organization. | +| <a id="organizationpath"></a>`path` **{warning-solid}** | [`String!`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Path of the organization. | + +#### Fields with arguments + +##### `Organization.groups` + +Groups within this organization that the user has access to. + +WARNING: +**Introduced** in 16.4. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`GroupConnection!`](#groupconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="organizationgroupssearch"></a>`search` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Search query for group name or full path. | +| <a id="organizationgroupssort"></a>`sort` **{warning-solid}** | [`OrganizationGroupSort`](#organizationgroupsort) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Criteria to sort organization groups by. | + ### `OrganizationStateCounts` Represents the total number of organizations for the represented states. @@ -20234,6 +21174,18 @@ Represents the total number of organizations for the represented states. | <a id="organizationstatecountsall"></a>`all` | [`Int`](#int) | Number of organizations with state `ALL`. | | <a id="organizationstatecountsinactive"></a>`inactive` | [`Int`](#int) | Number of organizations with state `INACTIVE`. | +### `OrganizationUser` + +A user with access to the organization. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="organizationuserbadges"></a>`badges` **{warning-solid}** | [`[String!]`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Badges describing the user within the organization. | +| <a id="organizationuserid"></a>`id` **{warning-solid}** | [`ID!`](#id) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. ID of the organization user. | +| <a id="organizationuseruser"></a>`user` **{warning-solid}** | [`UserCore!`](#usercore) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. User that is associated with the organization. | + ### `Package` Represents a package with pipelines in the Package Registry. @@ -20887,47 +21839,6 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="previewbillableuserchangeseatsinsubscription"></a>`seatsInSubscription` | [`Int`](#int) | Number of seats in subscription. | | <a id="previewbillableuserchangewillincreaseoverage"></a>`willIncreaseOverage` | [`Boolean`](#boolean) | If the group will have an increased overage after change. | -### `ProductAnalyticsDashboard` - -Represents a product analytics dashboard. - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboarddescription"></a>`description` | [`String`](#string) | Description of the dashboard. | -| <a id="productanalyticsdashboardpanels"></a>`panels` | [`ProductAnalyticsDashboardPanelConnection!`](#productanalyticsdashboardpanelconnection) | Panels shown on the dashboard. (see [Connections](#connections)) | -| <a id="productanalyticsdashboardslug"></a>`slug` | [`String!`](#string) | Slug of the dashboard. | -| <a id="productanalyticsdashboardtitle"></a>`title` | [`String!`](#string) | Title of the dashboard. | -| <a id="productanalyticsdashboarduserdefined"></a>`userDefined` | [`Boolean!`](#boolean) | Indicates whether the dashboard is user-defined or provided by GitLab. | - -### `ProductAnalyticsDashboardPanel` - -Represents a product analytics dashboard panel. - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardpanelgridattributes"></a>`gridAttributes` | [`JSON`](#json) | Description of the position and size of the panel. | -| <a id="productanalyticsdashboardpanelqueryoverrides"></a>`queryOverrides` | [`JSON`](#json) | Overrides for the visualization query object. | -| <a id="productanalyticsdashboardpaneltitle"></a>`title` | [`String!`](#string) | Title of the panel. | -| <a id="productanalyticsdashboardpanelvisualization"></a>`visualization` | [`ProductAnalyticsDashboardVisualization!`](#productanalyticsdashboardvisualization) | Visualization of the panel. | - -### `ProductAnalyticsDashboardVisualization` - -Represents a product analytics dashboard visualization. - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="productanalyticsdashboardvisualizationdata"></a>`data` | [`JSON!`](#json) | Data of the visualization. | -| <a id="productanalyticsdashboardvisualizationerrors"></a>`errors` | [`[String!]`](#string) | Validation errors in the visualization. | -| <a id="productanalyticsdashboardvisualizationoptions"></a>`options` | [`JSON!`](#json) | Options of the visualization. | -| <a id="productanalyticsdashboardvisualizationslug"></a>`slug` | [`String!`](#string) | Slug of the visualization. | -| <a id="productanalyticsdashboardvisualizationtype"></a>`type` | [`String!`](#string) | Type of the visualization. | - ### `Project` #### Fields @@ -21277,6 +22188,47 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectcontainerrepositoriesname"></a>`name` | [`String`](#string) | Filter the container repositories by their name. | | <a id="projectcontainerrepositoriessort"></a>`sort` | [`ContainerRepositorySort`](#containerrepositorysort) | Sort container repositories by this criteria. | +##### `Project.customizableDashboardVisualizations` + +Visualizations of the project or associated configuration project. + +WARNING: +**Introduced** in 16.1. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`CustomizableDashboardVisualizationConnection`](#customizabledashboardvisualizationconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectcustomizabledashboardvisualizationsslug"></a>`slug` | [`String`](#string) | Slug of the visualization to return. | + +##### `Project.customizableDashboards` + +Customizable dashboards for the project. + +WARNING: +**Introduced** in 15.6. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`CustomizableDashboardConnection`](#customizabledashboardconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectcustomizabledashboardscategory"></a>`category` | [`CustomizableDashboardCategory`](#customizabledashboardcategory) | Find by dashboard type. | +| <a id="projectcustomizabledashboardsslug"></a>`slug` | [`String`](#string) | Find by dashboard slug. | + ##### `Project.dastProfile` DAST Profile associated with the project. @@ -21905,7 +22857,7 @@ Network Policies of the project. WARNING: **Deprecated** in 14.8. -Network policies are deprecated and will be removed in GitLab 16.0. Since GitLab 15.0 this field returns no data. +Network policies are deprecated and will be removed in GitLab 17.0. This field returns no data in GitLab 15.0 and later. Returns [`NetworkPolicyConnection`](#networkpolicyconnection). @@ -22006,46 +22958,6 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectpipelinesupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Pipelines updated before this date. | | <a id="projectpipelinesusername"></a>`username` | [`String`](#string) | Filter pipelines by the user that triggered the pipeline. | -##### `Project.productAnalyticsDashboards` - -Product Analytics dashboards of the project. - -WARNING: -**Introduced** in 15.6. -This feature is an Experiment. It can be changed or removed at any time. - -Returns [`ProductAnalyticsDashboardConnection`](#productanalyticsdashboardconnection). - -This field returns a [connection](#connections). It accepts the -four standard [pagination arguments](#connection-pagination-arguments): -`before: String`, `after: String`, `first: Int`, `last: Int`. - -###### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="projectproductanalyticsdashboardsslug"></a>`slug` | [`String`](#string) | Find by dashboard slug. | - -##### `Project.productAnalyticsVisualizations` - -Visualizations of the project or associated configuration project. - -WARNING: -**Introduced** in 16.1. -This feature is an Experiment. It can be changed or removed at any time. - -Returns [`ProductAnalyticsDashboardVisualizationConnection`](#productanalyticsdashboardvisualizationconnection). - -This field returns a [connection](#connections). It accepts the -four standard [pagination arguments](#connection-pagination-arguments): -`before: String`, `after: String`, `first: Int`, `last: Int`. - -###### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="projectproductanalyticsvisualizationsslug"></a>`slug` | [`String`](#string) | Slug of the visualization to return. | - ##### `Project.projectMembers` Members of the project. @@ -22340,6 +23252,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="projectvulnerabilitiesdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. Only dismissed Vulnerabilities will be included with the filter. | | <a id="projectvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | +| <a id="projectvulnerabilitieshasmergerequest"></a>`hasMergeRequest` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked merge requests. | | <a id="projectvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="projectvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="projectvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | @@ -22378,6 +23291,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectvulnerabilityseveritiescountclusteragentid"></a>`clusterAgentId` | [`[ClustersAgentID!]`](#clustersagentid) | Filter vulnerabilities by `cluster_agent_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | +| <a id="projectvulnerabilityseveritiescountdismissalreason"></a>`dismissalReason` | [`[VulnerabilityDismissalReason!]`](#vulnerabilitydismissalreason) | Filter by dismissal reason. | | <a id="projectvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | | <a id="projectvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="projectvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -22444,6 +23358,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectcicdsettingkeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Whether to keep the latest builds artifacts. | | <a id="projectcicdsettingmergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Whether merge pipelines are enabled. | | <a id="projectcicdsettingmergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Whether merge trains are enabled. | +| <a id="projectcicdsettingmergetrainsskiptrainallowed"></a>`mergeTrainsSkipTrainAllowed` | [`Boolean!`](#boolean) | Whether merge immediately is allowed for merge trains. | | <a id="projectcicdsettingproject"></a>`project` | [`Project`](#project) | Project the CI/CD settings belong to. | ### `ProjectConversations` @@ -22452,7 +23367,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="projectconversationsciconfigmessages"></a>`ciConfigMessages` **{warning-solid}** | [`AiMessageTypeConnection`](#aimessagetypeconnection) | **Introduced** in 16.0. This feature is an Experiment. It can be changed or removed at any time. Messages generated by open ai and the user. | +| <a id="projectconversationsciconfigmessages"></a>`ciConfigMessages` **{warning-solid}** | [`AiMessageConnection`](#aimessageconnection) | **Introduced** in 16.0. This feature is an Experiment. It can be changed or removed at any time. Messages generated by open ai and the user. | ### `ProjectDataTransfer` @@ -22849,6 +23764,31 @@ Pypi metadata. | <a id="querycomplexitylimit"></a>`limit` | [`Int`](#int) | GraphQL query complexity limit. See [GitLab documentation on this limit](https://docs.gitlab.com/ee/api/graphql/index.html#max-query-complexity). | | <a id="querycomplexityscore"></a>`score` | [`Int`](#int) | GraphQL query complexity score. | +### `QueueingDelayHistory` + +Aggregated statistics about queueing times for CI jobs. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="queueingdelayhistorytimeseries"></a>`timeSeries` | [`[QueueingHistoryTimeSeries!]`](#queueinghistorytimeseries) | Time series. | + +### `QueueingHistoryTimeSeries` + +The amount of time for a job to be picked up by a runner, in percentiles. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="queueinghistorytimeseriesp50"></a>`p50` **{warning-solid}** | [`Duration`](#duration) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. 50th percentile. 50% of the durations are lower than this value. | +| <a id="queueinghistorytimeseriesp75"></a>`p75` **{warning-solid}** | [`Duration`](#duration) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. 75th percentile. 75% of the durations are lower than this value. | +| <a id="queueinghistorytimeseriesp90"></a>`p90` **{warning-solid}** | [`Duration`](#duration) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. 90th percentile. 90% of the durations are lower than this value. | +| <a id="queueinghistorytimeseriesp95"></a>`p95` **{warning-solid}** | [`Duration`](#duration) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. 95th percentile. 95% of the durations are lower than this value. | +| <a id="queueinghistorytimeseriesp99"></a>`p99` **{warning-solid}** | [`Duration`](#duration) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. 99th percentile. 99% of the durations are lower than this value. | +| <a id="queueinghistorytimeseriestime"></a>`time` | [`Time!`](#time) | Start of the time interval. | + ### `RecentFailures` Recent failure history of a test case. @@ -23101,6 +24041,25 @@ Returns [`RepositoryCodeownerValidation`](#repositorycodeownervalidation). | <a id="repositoryblobstoredexternally"></a>`storedExternally` | [`Boolean`](#boolean) | Whether the blob's content is stored externally (for instance, in LFS). | | <a id="repositoryblobwebpath"></a>`webPath` | [`String`](#string) | Web path of the blob. | +#### Fields with arguments + +##### `RepositoryBlob.blame` + +Blob blame. Available only when feature flag `graphql_git_blame` is enabled. + +WARNING: +**Introduced** in 16.3. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`Blame`](#blame). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="repositoryblobblamefromline"></a>`fromLine` | [`Int`](#int) | Range starting from the line. Cannot be less than 1 or greater than `to_line`. | +| <a id="repositoryblobblametoline"></a>`toLine` | [`Int`](#int) | Range ending on the line. Cannot be less than 1 or less than `to_line`. | + ### `RepositoryCodeownerError` #### Fields @@ -23342,6 +24301,7 @@ Represents the scan execution policy. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="scanexecutionpolicydescription"></a>`description` | [`String!`](#string) | Description of the policy. | +| <a id="scanexecutionpolicyeditpath"></a>`editPath` | [`String!`](#string) | URL of policy edit page. | | <a id="scanexecutionpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | | <a id="scanexecutionpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | | <a id="scanexecutionpolicysource"></a>`source` | [`SecurityPolicySource!`](#securitypolicysource) | Source of the policy. Its fields depend on the source type. | @@ -23358,6 +24318,7 @@ Represents the scan result policy. | ---- | ---- | ----------- | | <a id="scanresultpolicyallgroupapprovers"></a>`allGroupApprovers` | [`[PolicyApprovalGroup!]`](#policyapprovalgroup) | All potential approvers of the group type, including groups inaccessible to the user. | | <a id="scanresultpolicydescription"></a>`description` | [`String!`](#string) | Description of the policy. | +| <a id="scanresultpolicyeditpath"></a>`editPath` | [`String!`](#string) | URL of policy edit page. | | <a id="scanresultpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | | <a id="scanresultpolicygroupapprovers"></a>`groupApprovers` | [`[Group!]`](#group) | Approvers of the group type. | | <a id="scanresultpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | @@ -24228,6 +25189,16 @@ Represents a recorded measurement (object count) for the Admins. | <a id="userachievementupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp the achievement was last updated. | | <a id="userachievementuser"></a>`user` | [`UserCore!`](#usercore) | Achievement recipient. | +### `UserAddOnAssignment` + +An assignment of an AddOnPurchase to a User. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="useraddonassignmentaddonpurchase"></a>`addOnPurchase` | [`AddOnPurchase!`](#addonpurchase) | Add-on purchase the user is assigned to. | + ### `UserCallout` #### Fields @@ -24239,7 +25210,7 @@ Represents a recorded measurement (object count) for the Admins. ### `UserCore` -Core represention of a GitLab user. +Core representation of a GitLab user. #### Fields @@ -24574,6 +25545,18 @@ fields relate to interactions between the two entities. | <a id="valuestreamanalyticsmetricunit"></a>`unit` | [`String`](#string) | Unit of measurement. | | <a id="valuestreamanalyticsmetricvalue"></a>`value` | [`Float`](#float) | Value for the metric. | +### `ValueStreamDashboardCount` + +Represents a recorded measurement (object count) for the requested group. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="valuestreamdashboardcountcount"></a>`count` | [`Int`](#int) | Object count. | +| <a id="valuestreamdashboardcountidentifier"></a>`identifier` | [`ValueStreamDashboardMetric!`](#valuestreamdashboardmetric) | Type of object being measured. | +| <a id="valuestreamdashboardcountrecordedat"></a>`recordedAt` | [`Time`](#time) | Time the measurement was taken. | + ### `ValueStreamMetricLinkType` #### Fields @@ -24618,7 +25601,7 @@ Represents a vulnerability. | <a id="vulnerabilitydetails"></a>`details` | [`[VulnerabilityDetail!]!`](#vulnerabilitydetail) | Details of the vulnerability. | | <a id="vulnerabilitydetectedat"></a>`detectedAt` | [`Time!`](#time) | Timestamp of when the vulnerability was first detected. | | <a id="vulnerabilitydiscussions"></a>`discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. (see [Connections](#connections)) | -| <a id="vulnerabilitydismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason for dismissal. Returns `null` for states other than `dismissed`. Returns `null` if `expose_dismissal_reason` feature flag is disabled. | +| <a id="vulnerabilitydismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason for dismissal. Returns `null` for states other than `dismissed`. | | <a id="vulnerabilitydismissedat"></a>`dismissedAt` | [`Time`](#time) | Timestamp of when the vulnerability state was changed to dismissed. | | <a id="vulnerabilitydismissedby"></a>`dismissedBy` | [`UserCore`](#usercore) | User that dismissed the vulnerability. | | <a id="vulnerabilityexternalissuelinks"></a>`externalIssueLinks` | [`VulnerabilityExternalIssueLinkConnection!`](#vulnerabilityexternalissuelinkconnection) | List of external issue links related to the vulnerability. (see [Connections](#connections)) | @@ -24640,6 +25623,7 @@ Represents a vulnerability. | <a id="vulnerabilityresolvedondefaultbranch"></a>`resolvedOnDefaultBranch` | [`Boolean!`](#boolean) | Indicates whether the vulnerability is fixed on the default branch or not. | | <a id="vulnerabilityscanner"></a>`scanner` | [`VulnerabilityScanner`](#vulnerabilityscanner) | Scanner metadata for the vulnerability. | | <a id="vulnerabilityseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (INFO, UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL). | +| <a id="vulnerabilitysolution"></a>`solution` | [`String`](#string) | Recommended solution for the vulnerability. | | <a id="vulnerabilitystate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (DETECTED, CONFIRMED, RESOLVED, DISMISSED). | | <a id="vulnerabilitystatecomment"></a>`stateComment` | [`String`](#string) | Comment given for the vulnerability state change. | | <a id="vulnerabilitystatetransitions"></a>`stateTransitions` | [`VulnerabilityStateTransitionTypeConnection`](#vulnerabilitystatetransitiontypeconnection) | List of state transitions related to the vulnerability. (see [Connections](#connections)) | @@ -24647,6 +25631,7 @@ Represents a vulnerability. | <a id="vulnerabilityupdatedat"></a>`updatedAt` | [`Time`](#time) | Timestamp of when the vulnerability was last updated. | | <a id="vulnerabilityusernotescount"></a>`userNotesCount` | [`Int!`](#int) | Number of user notes attached to the vulnerability. | | <a id="vulnerabilityuserpermissions"></a>`userPermissions` | [`VulnerabilityPermissions!`](#vulnerabilitypermissions) | Permissions for the current user on the resource. | +| <a id="vulnerabilityuuid"></a>`uuid` | [`String!`](#string) | UUID of the vulnerability finding. Can be used to look up the associated security report finding. | | <a id="vulnerabilityvulnerabilitypath"></a>`vulnerabilityPath` | [`String`](#string) | Path to the vulnerability's details page. | | <a id="vulnerabilityweburl"></a>`webUrl` | [`String`](#string) | URL to the vulnerability's details page. | @@ -25296,6 +26281,7 @@ Check permissions for the current user on a work item. | ---- | ---- | ----------- | | <a id="workitempermissionsadminparentlink"></a>`adminParentLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_parent_link` on this resource. | | <a id="workitempermissionsadminworkitem"></a>`adminWorkItem` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_work_item` on this resource. | +| <a id="workitempermissionsadminworkitemlink"></a>`adminWorkItemLink` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_work_item_link` on this resource. | | <a id="workitempermissionscreatenote"></a>`createNote` | [`Boolean!`](#boolean) | Indicates the user can perform `create_note` on this resource. | | <a id="workitempermissionsdeleteworkitem"></a>`deleteWorkItem` | [`Boolean!`](#boolean) | Indicates the user can perform `delete_work_item` on this resource. | | <a id="workitempermissionsreadworkitem"></a>`readWorkItem` | [`Boolean!`](#boolean) | Indicates the user can perform `read_work_item` on this resource. | @@ -25439,9 +26425,30 @@ Represents the linked items widget. | <a id="workitemwidgetlinkeditemsblocked"></a>`blocked` | [`Boolean`](#boolean) | Indicates the work item is blocked. Returns `null`if `linked_work_items` feature flag is disabled. | | <a id="workitemwidgetlinkeditemsblockedbycount"></a>`blockedByCount` | [`Int`](#int) | Count of items blocking the work item. Returns `null`if `linked_work_items` feature flag is disabled. | | <a id="workitemwidgetlinkeditemsblockingcount"></a>`blockingCount` | [`Int`](#int) | Count of items the work item is blocking. Returns `null`if `linked_work_items` feature flag is disabled. | -| <a id="workitemwidgetlinkeditemslinkeditems"></a>`linkedItems` **{warning-solid}** | [`LinkedWorkItemTypeConnection`](#linkedworkitemtypeconnection) | **Introduced** in 16.3. This feature is an Experiment. It can be changed or removed at any time. Linked items for the work item. Returns `null`if `linked_work_items` feature flag is disabled. | | <a id="workitemwidgetlinkeditemstype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | +#### Fields with arguments + +##### `WorkItemWidgetLinkedItems.linkedItems` + +Linked items for the work item. Returns `null` if `linked_work_items` feature flag is disabled. + +WARNING: +**Introduced** in 16.3. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`LinkedWorkItemTypeConnection`](#linkedworkitemtypeconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetlinkeditemslinkeditemsfilter"></a>`filter` | [`WorkItemRelatedLinkType`](#workitemrelatedlinktype) | Filter by link type. Supported values: RELATED, BLOCKED_BY, and BLOCKS. Returns all types if omitted. | + ### `WorkItemWidgetMilestone` Represents a milestone widget. @@ -25672,14 +26679,23 @@ Agent token statuses. | <a id="agenttokenstatusactive"></a>`ACTIVE` | Active agent token. | | <a id="agenttokenstatusrevoked"></a>`REVOKED` | Revoked agent token. | -### `AiCachedMessageRole` +### `AiChatMessageRole` Roles to filter in chat message. | Value | Description | | ----- | ----------- | -| <a id="aicachedmessageroleassistant"></a>`ASSISTANT` | Filter only assistant messages. | -| <a id="aicachedmessageroleuser"></a>`USER` | Filter only user messages. | +| <a id="aichatmessageroleassistant"></a>`ASSISTANT` | Filter only assistant messages. | +| <a id="aichatmessagerolesystem"></a>`SYSTEM` | Filter only system messages. | +| <a id="aichatmessageroleuser"></a>`USER` | Filter only user messages. | + +### `AiMessageType` + +Types of messages returned from AI features. + +| Value | Description | +| ----- | ----------- | +| <a id="aimessagetypetool"></a>`TOOL` | Tool selection message. | ### `AlertManagementAlertSort` @@ -25900,6 +26916,47 @@ Values for sorting inherited variables. | <a id="cigroupvariablessortkey_asc"></a>`KEY_ASC` | Key by ascending order. | | <a id="cigroupvariablessortkey_desc"></a>`KEY_DESC` | Key by descending order. | +### `CiJobFailureReason` + +| Value | Description | +| ----- | ----------- | +| <a id="cijobfailurereasonapi_failure"></a>`API_FAILURE` | A job that failed due to api failure. | +| <a id="cijobfailurereasonarchived_failure"></a>`ARCHIVED_FAILURE` | A job that failed due to archived failure. | +| <a id="cijobfailurereasonbridge_pipeline_is_child_pipeline"></a>`BRIDGE_PIPELINE_IS_CHILD_PIPELINE` | A job that failed due to bridge pipeline is child pipeline. | +| <a id="cijobfailurereasonbuilds_disabled"></a>`BUILDS_DISABLED` | A job that failed due to builds disabled. | +| <a id="cijobfailurereasonci_quota_exceeded"></a>`CI_QUOTA_EXCEEDED` | A job that failed due to ci quota exceeded. | +| <a id="cijobfailurereasondata_integrity_failure"></a>`DATA_INTEGRITY_FAILURE` | A job that failed due to data integrity failure. | +| <a id="cijobfailurereasondeployment_rejected"></a>`DEPLOYMENT_REJECTED` | A job that failed due to deployment rejected. | +| <a id="cijobfailurereasondownstream_bridge_project_not_found"></a>`DOWNSTREAM_BRIDGE_PROJECT_NOT_FOUND` | A job that failed due to downstream bridge project not found. | +| <a id="cijobfailurereasondownstream_pipeline_creation_failed"></a>`DOWNSTREAM_PIPELINE_CREATION_FAILED` | A job that failed due to downstream pipeline creation failed. | +| <a id="cijobfailurereasonenvironment_creation_failure"></a>`ENVIRONMENT_CREATION_FAILURE` | A job that failed due to environment creation failure. | +| <a id="cijobfailurereasonfailed_outdated_deployment_job"></a>`FAILED_OUTDATED_DEPLOYMENT_JOB` | A job that failed due to failed outdated deployment job. | +| <a id="cijobfailurereasonforward_deployment_failure"></a>`FORWARD_DEPLOYMENT_FAILURE` | A job that failed due to forward deployment failure. | +| <a id="cijobfailurereasoninsufficient_bridge_permissions"></a>`INSUFFICIENT_BRIDGE_PERMISSIONS` | A job that failed due to insufficient bridge permissions. | +| <a id="cijobfailurereasoninsufficient_upstream_permissions"></a>`INSUFFICIENT_UPSTREAM_PERMISSIONS` | A job that failed due to insufficient upstream permissions. | +| <a id="cijobfailurereasoninvalid_bridge_trigger"></a>`INVALID_BRIDGE_TRIGGER` | A job that failed due to invalid bridge trigger. | +| <a id="cijobfailurereasonip_restriction_failure"></a>`IP_RESTRICTION_FAILURE` | A job that failed due to ip restriction failure. | +| <a id="cijobfailurereasonjob_execution_timeout"></a>`JOB_EXECUTION_TIMEOUT` | A job that failed due to job execution timeout. | +| <a id="cijobfailurereasonmissing_dependency_failure"></a>`MISSING_DEPENDENCY_FAILURE` | A job that failed due to missing dependency failure. | +| <a id="cijobfailurereasonno_matching_runner"></a>`NO_MATCHING_RUNNER` | A job that failed due to no matching runner. | +| <a id="cijobfailurereasonpipeline_loop_detected"></a>`PIPELINE_LOOP_DETECTED` | A job that failed due to pipeline loop detected. | +| <a id="cijobfailurereasonproject_deleted"></a>`PROJECT_DELETED` | A job that failed due to project deleted. | +| <a id="cijobfailurereasonprotected_environment_failure"></a>`PROTECTED_ENVIRONMENT_FAILURE` | A job that failed due to protected environment failure. | +| <a id="cijobfailurereasonreached_max_descendant_pipelines_depth"></a>`REACHED_MAX_DESCENDANT_PIPELINES_DEPTH` | A job that failed due to reached max descendant pipelines depth. | +| <a id="cijobfailurereasonreached_max_pipeline_hierarchy_size"></a>`REACHED_MAX_PIPELINE_HIERARCHY_SIZE` | A job that failed due to reached max pipeline hierarchy size. | +| <a id="cijobfailurereasonrunner_system_failure"></a>`RUNNER_SYSTEM_FAILURE` | A job that failed due to runner system failure. | +| <a id="cijobfailurereasonrunner_unsupported"></a>`RUNNER_UNSUPPORTED` | A job that failed due to runner unsupported. | +| <a id="cijobfailurereasonscheduler_failure"></a>`SCHEDULER_FAILURE` | A job that failed due to scheduler failure. | +| <a id="cijobfailurereasonscript_failure"></a>`SCRIPT_FAILURE` | A job that failed due to script failure. | +| <a id="cijobfailurereasonsecrets_provider_not_found"></a>`SECRETS_PROVIDER_NOT_FOUND` | A job that failed due to secrets provider not found. | +| <a id="cijobfailurereasonstale_schedule"></a>`STALE_SCHEDULE` | A job that failed due to stale schedule. | +| <a id="cijobfailurereasonstuck_or_timeout_failure"></a>`STUCK_OR_TIMEOUT_FAILURE` | A job that failed due to stuck or timeout failure. | +| <a id="cijobfailurereasontrace_size_exceeded"></a>`TRACE_SIZE_EXCEEDED` | A job that failed due to trace size exceeded. | +| <a id="cijobfailurereasonunknown_failure"></a>`UNKNOWN_FAILURE` | A job that failed due to unknown failure. | +| <a id="cijobfailurereasonunmet_prerequisites"></a>`UNMET_PREREQUISITES` | A job that failed due to unmet prerequisites. | +| <a id="cijobfailurereasonupstream_bridge_project_not_found"></a>`UPSTREAM_BRIDGE_PROJECT_NOT_FOUND` | A job that failed due to upstream bridge project not found. | +| <a id="cijobfailurereasonuser_blocked"></a>`USER_BLOCKED` | A job that failed due to user blocked. | + ### `CiJobKind` | Value | Description | @@ -26025,6 +27082,16 @@ Values for sorting variables. | <a id="codequalitydegradationseverityminor"></a>`MINOR` | Code Quality degradation has a status of minor. | | <a id="codequalitydegradationseverityunknown"></a>`UNKNOWN` | Code Quality degradation has a status of unknown. | +### `CodequalityReportsComparerReportStatus` + +Report comparison status. + +| Value | Description | +| ----- | ----------- | +| <a id="codequalityreportscomparerreportstatusfailed"></a>`FAILED` | Report failed to generate. | +| <a id="codequalityreportscomparerreportstatusnot_found"></a>`NOT_FOUND` | Head report or base report not found. | +| <a id="codequalityreportscomparerreportstatussuccess"></a>`SUCCESS` | Report successfully generated. | + ### `CommitActionMode` Mode of a commit action. @@ -26261,6 +27328,14 @@ Values for sorting tags. | <a id="customerrelationsorganizationstateall"></a>`all` | All available organizations. | | <a id="customerrelationsorganizationstateinactive"></a>`inactive` | Inactive organizations. | +### `CustomizableDashboardCategory` + +Categories for customizable dashboards. + +| Value | Description | +| ----- | ----------- | +| <a id="customizabledashboardcategoryanalytics"></a>`ANALYTICS` | Analytics category for customizable dashboards. | + ### `DastPreScanVerificationCheckType` Check type of the pre scan verification step. @@ -26620,9 +27695,18 @@ List of statuses for forecasting model. | <a id="forecaststatusready"></a>`READY` | Forecast is ready. | | <a id="forecaststatusunavailable"></a>`UNAVAILABLE` | Forecast is unavailable. | +### `GeoRegistriesBulkAction` + +Action to trigger on multiple Geo registries. + +| Value | Description | +| ----- | ----------- | +| <a id="georegistriesbulkactionresync_all"></a>`RESYNC_ALL` | Resync multiple registries. | +| <a id="georegistriesbulkactionreverify_all"></a>`REVERIFY_ALL` | Reverify multiple registries. | + ### `GeoRegistryAction` -Action to trigger on one or more Geo registries. +Action to trigger on an individual Geo registry. | Value | Description | | ----- | ----------- | @@ -26951,16 +28035,6 @@ List limit metric setting. | <a id="listlimitmetricissue_count"></a>`issue_count` | Limit list by number of issues. | | <a id="listlimitmetricissue_weights"></a>`issue_weights` | Limit list by total weight of issues. | -### `MarkupFormat` - -List markup formats. - -| Value | Description | -| ----- | ----------- | -| <a id="markupformathtml"></a>`HTML` | HTML format. | -| <a id="markupformatmarkdown"></a>`MARKDOWN` | Markdown format. | -| <a id="markupformatraw"></a>`RAW` | Raw format. | - ### `MeasurementIdentifier` Possible identifier types for a measurement. @@ -27216,6 +28290,23 @@ Rotation length unit of an on-call rotation. | <a id="oncallrotationunitenumhours"></a>`HOURS` | Hours. | | <a id="oncallrotationunitenumweeks"></a>`WEEKS` | Weeks. | +### `OrganizationGroupSort` + +Values for sorting organization groups. + +| Value | Description | +| ----- | ----------- | +| <a id="organizationgroupsortcreated_at_asc"></a>`CREATED_AT_ASC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Created at in ascending order. | +| <a id="organizationgroupsortcreated_at_desc"></a>`CREATED_AT_DESC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Created at in descending order. | +| <a id="organizationgroupsortid_asc"></a>`ID_ASC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. ID in ascending order. | +| <a id="organizationgroupsortid_desc"></a>`ID_DESC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. ID in descending order. | +| <a id="organizationgroupsortname_asc"></a>`NAME_ASC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Name in ascending order. | +| <a id="organizationgroupsortname_desc"></a>`NAME_DESC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Name in descending order. | +| <a id="organizationgroupsortpath_asc"></a>`PATH_ASC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Path in ascending order. | +| <a id="organizationgroupsortpath_desc"></a>`PATH_DESC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Path in descending order. | +| <a id="organizationgroupsortupdated_at_asc"></a>`UPDATED_AT_ASC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Updated at in ascending order. | +| <a id="organizationgroupsortupdated_at_desc"></a>`UPDATED_AT_DESC` **{warning-solid}** | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Updated at in descending order. | + ### `OrganizationSort` Values for sorting organizations. @@ -27805,14 +28896,12 @@ Name of the feature that the callout is for. | Value | Description | | ----- | ----------- | | <a id="usercalloutfeaturenameenumactive_user_count_threshold"></a>`ACTIVE_USER_COUNT_THRESHOLD` | Callout feature name for active_user_count_threshold. | -| <a id="usercalloutfeaturenameenumartifacts_management_page_feedback_banner"></a>`ARTIFACTS_MANAGEMENT_PAGE_FEEDBACK_BANNER` | Callout feature name for artifacts_management_page_feedback_banner. | | <a id="usercalloutfeaturenameenumbranch_rules_info_callout"></a>`BRANCH_RULES_INFO_CALLOUT` | Callout feature name for branch_rules_info_callout. | | <a id="usercalloutfeaturenameenumbuy_pipeline_minutes_notification_dot"></a>`BUY_PIPELINE_MINUTES_NOTIFICATION_DOT` | Callout feature name for buy_pipeline_minutes_notification_dot. | | <a id="usercalloutfeaturenameenumcanary_deployment"></a>`CANARY_DEPLOYMENT` | Callout feature name for canary_deployment. | | <a id="usercalloutfeaturenameenumci_deprecation_warning_for_types_keyword"></a>`CI_DEPRECATION_WARNING_FOR_TYPES_KEYWORD` | Callout feature name for ci_deprecation_warning_for_types_keyword. | | <a id="usercalloutfeaturenameenumcloud_licensing_subscription_activation_banner"></a>`CLOUD_LICENSING_SUBSCRIPTION_ACTIVATION_BANNER` | Callout feature name for cloud_licensing_subscription_activation_banner. | | <a id="usercalloutfeaturenameenumcluster_security_warning"></a>`CLUSTER_SECURITY_WARNING` | Callout feature name for cluster_security_warning. | -| <a id="usercalloutfeaturenameenumcode_suggestions_third_party_callout"></a>`CODE_SUGGESTIONS_THIRD_PARTY_CALLOUT` | Callout feature name for code_suggestions_third_party_callout. | | <a id="usercalloutfeaturenameenumcreate_runner_workflow_banner"></a>`CREATE_RUNNER_WORKFLOW_BANNER` | Callout feature name for create_runner_workflow_banner. | | <a id="usercalloutfeaturenameenumeoa_bronze_plan_banner"></a>`EOA_BRONZE_PLAN_BANNER` | Callout feature name for eoa_bronze_plan_banner. | | <a id="usercalloutfeaturenameenumfeature_flags_new_version"></a>`FEATURE_FLAGS_NEW_VERSION` | Callout feature name for feature_flags_new_version. | @@ -27846,6 +28935,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumsecurity_configuration_devops_alert"></a>`SECURITY_CONFIGURATION_DEVOPS_ALERT` | Callout feature name for security_configuration_devops_alert. | | <a id="usercalloutfeaturenameenumsecurity_configuration_upgrade_banner"></a>`SECURITY_CONFIGURATION_UPGRADE_BANNER` | Callout feature name for security_configuration_upgrade_banner. | | <a id="usercalloutfeaturenameenumsecurity_newsletter_callout"></a>`SECURITY_NEWSLETTER_CALLOUT` | Callout feature name for security_newsletter_callout. | +| <a id="usercalloutfeaturenameenumsecurity_policy_protected_branch_modification"></a>`SECURITY_POLICY_PROTECTED_BRANCH_MODIFICATION` | Callout feature name for security_policy_protected_branch_modification. | | <a id="usercalloutfeaturenameenumsecurity_training_feature_promotion"></a>`SECURITY_TRAINING_FEATURE_PROMOTION` | Callout feature name for security_training_feature_promotion. | | <a id="usercalloutfeaturenameenumsubmit_license_usage_data_banner"></a>`SUBMIT_LICENSE_USAGE_DATA_BANNER` | Callout feature name for submit_license_usage_data_banner. | | <a id="usercalloutfeaturenameenumsuggest_pipeline"></a>`SUGGEST_PIPELINE` | Callout feature name for suggest_pipeline. | @@ -27860,6 +28950,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumunfinished_tag_cleanup_callout"></a>`UNFINISHED_TAG_CLEANUP_CALLOUT` | Callout feature name for unfinished_tag_cleanup_callout. | | <a id="usercalloutfeaturenameenumuser_reached_limit_free_plan_alert"></a>`USER_REACHED_LIMIT_FREE_PLAN_ALERT` | Callout feature name for user_reached_limit_free_plan_alert. | | <a id="usercalloutfeaturenameenumverification_reminder"></a>`VERIFICATION_REMINDER` | Callout feature name for verification_reminder. | +| <a id="usercalloutfeaturenameenumvsd_feedback_banner"></a>`VSD_FEEDBACK_BANNER` | Callout feature name for vsd_feedback_banner. | | <a id="usercalloutfeaturenameenumweb_ide_alert_dismissed"></a>`WEB_IDE_ALERT_DISMISSED` | Callout feature name for web_ide_alert_dismissed. | | <a id="usercalloutfeaturenameenumweb_ide_ci_environments_guidance"></a>`WEB_IDE_CI_ENVIRONMENTS_GUIDANCE` | Callout feature name for web_ide_ci_environments_guidance. | @@ -27873,6 +28964,18 @@ Possible states of a user. | <a id="userstateblocked"></a>`blocked` | User has been blocked and is prevented from using the system. | | <a id="userstatedeactivated"></a>`deactivated` | User is no longer active and is unable to use the system. | +### `ValueStreamDashboardMetric` + +Possible identifier types for a measurement. + +| Value | Description | +| ----- | ----------- | +| <a id="valuestreamdashboardmetricgroups"></a>`GROUPS` | Group count. | +| <a id="valuestreamdashboardmetricissues"></a>`ISSUES` | Issue count. | +| <a id="valuestreamdashboardmetricmerge_requests"></a>`MERGE_REQUESTS` | Merge request count. | +| <a id="valuestreamdashboardmetricpipelines"></a>`PIPELINES` | Pipeline count. | +| <a id="valuestreamdashboardmetricprojects"></a>`PROJECTS` | Project count. | + ### `VerificationStateEnum` | Value | Description | @@ -28055,6 +29158,7 @@ Values for work item award emoji update enum. | ----- | ----------- | | <a id="workitemawardemojiupdateactionadd"></a>`ADD` | Adds the emoji. | | <a id="workitemawardemojiupdateactionremove"></a>`REMOVE` | Removes the emoji. | +| <a id="workitemawardemojiupdateactiontoggle"></a>`TOGGLE` | Toggles the status of the emoji. | ### `WorkItemRelatedLinkType` @@ -28212,6 +29316,12 @@ A `AuditEventsInstanceExternalAuditEventDestinationID` is a global ID. It is enc An example `AuditEventsInstanceExternalAuditEventDestinationID` is: `"gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/1"`. +### `AuditEventsInstanceGoogleCloudLoggingConfigurationID` + +A `AuditEventsInstanceGoogleCloudLoggingConfigurationID` is a global ID. It is encoded as a string. + +An example `AuditEventsInstanceGoogleCloudLoggingConfigurationID` is: `"gid://gitlab/AuditEvents::Instance::GoogleCloudLoggingConfiguration/1"`. + ### `AuditEventsStreamingHeaderID` A `AuditEventsStreamingHeaderID` is a global ID. It is encoded as a string. @@ -28653,12 +29763,6 @@ A `MergeRequestID` is a global ID. It is encoded as a string. An example `MergeRequestID` is: `"gid://gitlab/MergeRequest/1"`. -### `MetricsDashboardAnnotationID` - -A `MetricsDashboardAnnotationID` is a global ID. It is encoded as a string. - -An example `MetricsDashboardAnnotationID` is: `"gid://gitlab/Metrics::Dashboard::Annotation/1"`. - ### `MilestoneID` A `MilestoneID` is a global ID. It is encoded as a string. @@ -28689,6 +29793,12 @@ A `NoteableID` is a global ID. It is encoded as a string. An example `NoteableID` is: `"gid://gitlab/Noteable/1"`. +### `OrganizationsOrganizationID` + +A `OrganizationsOrganizationID` is a global ID. It is encoded as a string. + +An example `OrganizationsOrganizationID` is: `"gid://gitlab/Organizations::Organization/1"`. + ### `PackagesConanFileMetadatumID` A `PackagesConanFileMetadatumID` is a global ID. It is encoded as a string. @@ -29063,6 +30173,7 @@ Implementations: | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="baseheaderinterfaceactive"></a>`active` | [`Boolean!`](#boolean) | Header is active or not. | | <a id="baseheaderinterfaceid"></a>`id` | [`ID!`](#id) | ID of the header. | | <a id="baseheaderinterfacekey"></a>`key` | [`String!`](#string) | Key of the header. | | <a id="baseheaderinterfacevalue"></a>`value` | [`String!`](#string) | Value of the header. | @@ -29206,6 +30317,23 @@ Implementations: | <a id="externalauditeventdestinationinterfacename"></a>`name` | [`String!`](#string) | Name of the external destination to send audit events to. | | <a id="externalauditeventdestinationinterfaceverificationtoken"></a>`verificationToken` | [`String!`](#string) | Verification token to validate source of event. | +#### `GoogleCloudLoggingConfigurationInterface` + +Implementations: + +- [`GoogleCloudLoggingConfigurationType`](#googlecloudloggingconfigurationtype) +- [`InstanceGoogleCloudLoggingConfigurationType`](#instancegooglecloudloggingconfigurationtype) + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="googlecloudloggingconfigurationinterfaceclientemail"></a>`clientEmail` | [`String!`](#string) | Client email. | +| <a id="googlecloudloggingconfigurationinterfacegoogleprojectidname"></a>`googleProjectIdName` | [`String!`](#string) | Google project ID. | +| <a id="googlecloudloggingconfigurationinterfaceid"></a>`id` | [`ID!`](#id) | ID of the configuration. | +| <a id="googlecloudloggingconfigurationinterfacelogidname"></a>`logIdName` | [`String!`](#string) | Log ID. | +| <a id="googlecloudloggingconfigurationinterfacename"></a>`name` | [`String!`](#string) | Name of the external destination to send audit events to. | + #### `MemberInterface` Implementations: @@ -29273,6 +30401,7 @@ Implementations: | Name | Type | Description | | ---- | ---- | ----------- | | <a id="orchestrationpolicydescription"></a>`description` | [`String!`](#string) | Description of the policy. | +| <a id="orchestrationpolicyeditpath"></a>`editPath` | [`String!`](#string) | URL of policy edit page. | | <a id="orchestrationpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | | <a id="orchestrationpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | | <a id="orchestrationpolicyupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp of when the policy YAML was last updated. | @@ -29372,6 +30501,7 @@ Representation of a GitLab user. Implementations: +- [`AddOnUser`](#addonuser) - [`AutocompletedUser`](#autocompleteduser) - [`MergeRequestAssignee`](#mergerequestassignee) - [`MergeRequestAuthor`](#mergerequestauthor) @@ -29880,7 +31010,7 @@ Attributes for defining a CI/CD variable. | <a id="complianceframeworkinputdefault"></a>`default` | [`Boolean`](#boolean) | Set this compliance framework as the default framework for the group. | | <a id="complianceframeworkinputdescription"></a>`description` | [`String`](#string) | New description for the compliance framework. | | <a id="complianceframeworkinputname"></a>`name` | [`String`](#string) | New name for the compliance framework. | -| <a id="complianceframeworkinputpipelineconfigurationfullpath"></a>`pipelineConfigurationFullPath` | [`String`](#string) | Full path of the compliance pipeline configuration stored in a project repository, such as `.gitlab/.compliance-gitlab-ci.yml@compliance/hipaa` **(ULTIMATE)**. | +| <a id="complianceframeworkinputpipelineconfigurationfullpath"></a>`pipelineConfigurationFullPath` | [`String`](#string) | Full path of the compliance pipeline configuration stored in a project repository, such as `.gitlab/.compliance-gitlab-ci.yml@compliance/hipaa` **(ULTIMATE ALL)**. | ### `ComplianceStandardsAdherenceInput` diff --git a/doc/api/graphql/users_example.md b/doc/api/graphql/users_example.md index 2234528c52a..0fd61689b8e 100644 --- a/doc/api/graphql/users_example.md +++ b/doc/api/graphql/users_example.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/group_access_tokens.md b/doc/api/group_access_tokens.md index d8b221a8f94..b99c91d2e5c 100644 --- a/doc/api/group_access_tokens.md +++ b/doc/api/group_access_tokens.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -143,6 +143,24 @@ POST /groups/:id/access_tokens/:token_id/rotate curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/<group_id>/access_tokens/<token_id>/rotate" ``` +Example response: + +```json +{ + "id": 42, + "name": "Rotated Token", + "revoked": false, + "created_at": "2023-08-01T15:00:00.000Z", + "scopes": ["api"], + "user_id": 1337, + "last_used_at": null, + "active": true, + "expires_at": "2023-08-15", + "access_level": 30, + "token": "s3cr3t" +} +``` + ### Responses - `200: OK` if existing token is successfully revoked and the new token is created. diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index fc7ec51af4d..61a2ef4a89b 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -52,7 +52,7 @@ Example response: "id": 1, "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge", "kind": "group" } @@ -84,7 +84,7 @@ Example response: "id": 1, "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge", "kind": "group" } @@ -201,7 +201,7 @@ Example response: { "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge" } ``` diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 267b9feb750..5a7b67b562c 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -296,10 +296,10 @@ PUT /groups/:id/boards/:board_id | `name` | string | no | The new name of the board | | `hide_backlog_list` | boolean | no | Hide the Open list | | `hide_closed_list` | boolean | no | Hide the Closed list | -| `assignee_id` **(PREMIUM)** | integer | no | The assignee the board should be scoped to | -| `milestone_id` **(PREMIUM)** | integer | no | The milestone the board should be scoped to | -| `labels` **(PREMIUM)** | string | no | Comma-separated list of label names which the board should be scoped to | -| `weight` **(PREMIUM)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | +| `assignee_id` **(PREMIUM ALL)** | integer | no | The assignee the board should be scoped to | +| `milestone_id` **(PREMIUM ALL)** | integer | no | The milestone the board should be scoped to | +| `labels` **(PREMIUM ALL)** | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` **(PREMIUM ALL)** | integer | no | The weight range from 0 to 9, to which the board should be scoped to | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/boards/1?name=new_name&milestone_id=44&assignee_id=1&labels=GroupLabel&weight=4" diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index 912d5e197c6..c32a14ee21e 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -179,7 +179,7 @@ Parameters: | `platform_kubernetes_attributes[token]` | string | yes | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | string | no | TLS certificate. Required if API is using a self-signed TLS certificate. | | `platform_kubernetes_attributes[authorization_type]` | string | no | The cluster authorization type: `rbac`, `abac` or `unknown_authorization`. Defaults to `rbac`. | -| `environment_scope` | string | no | The associated environment to the cluster. Defaults to `*` **(PREMIUM)** | +| `environment_scope` | string | no | The associated environment to the cluster. Defaults to `*` **(PREMIUM ALL)** | Example request: @@ -250,7 +250,7 @@ Parameters: | `platform_kubernetes_attributes[api_url]` | string | no | The URL to access the Kubernetes API | | `platform_kubernetes_attributes[token]` | string | no | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | string | no | TLS certificate. Required if API is using a self-signed TLS certificate. | -| `environment_scope` | string | no | The associated environment to the cluster **(PREMIUM)** | +| `environment_scope` | string | no | The associated environment to the cluster **(PREMIUM ALL)** | NOTE: `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index e95f4b307c8..e431d3c47d8 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -93,7 +93,7 @@ returns either: The maximum import file size can be set by the Administrator on self-managed instances (default is `0` (unlimited)). As an administrator, you can modify the maximum import file size either: -- In the [Admin Area](../administration/settings/account_and_limit_settings.md). +- In the [Admin Area](../administration/settings/import_and_export_settings.md). - By using the `max_import_size` option in the [Application settings API](settings.md#change-application-settings). For information on the maximum import file size on GitLab.com, see diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index 8aebbff1814..ef373e5b7fa 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -94,7 +94,7 @@ POST /groups/:id/variables | `protected` | boolean | No | Whether the variable is protected | | `masked` | boolean | No | Whether the variable is masked | | `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | -| `environment_scope` **(PREMIUM)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | +| `environment_scope` **(PREMIUM ALL)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | | `description` | string | No | The `description` of the variable. Default: `null` | ```shell @@ -132,7 +132,7 @@ PUT /groups/:id/variables/:key | `protected` | boolean | No | Whether the variable is protected | | `masked` | boolean | No | Whether the variable is masked | | `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | -| `environment_scope` **(PREMIUM)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | +| `environment_scope` **(PREMIUM ALL)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | | `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | ```shell diff --git a/doc/api/group_protected_branches.md b/doc/api/group_protected_branches.md index c017d0741ce..10b7ff92e9c 100644 --- a/doc/api/group_protected_branches.md +++ b/doc/api/group_protected_branches.md @@ -49,7 +49,7 @@ Example response: [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -114,7 +114,7 @@ GET /groups/:id/protected_branches/:name ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/groups/5/protected_branches/master" + "https://gitlab.example.com/api/v4/groups/5/protected_branches/main" ``` Example response: @@ -122,7 +122,7 @@ Example response: ```json { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -270,7 +270,7 @@ curl --request POST \ --header "PRIVATE-TOKEN: <your_access_token>" \ --header "Content-Type: application/json" \ --data '{ - "name": "master", + "name": "main", "allowed_to_push": [{"access_level": 30}], "allowed_to_merge": [{ "access_level": 30 @@ -286,7 +286,7 @@ Example response: ```json { "id": 5, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -348,7 +348,7 @@ Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [ { "id": 12, @@ -406,14 +406,14 @@ To delete: curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{access_level: 40}]}' \ --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" + "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [ { "id": 12, @@ -431,14 +431,14 @@ Example response: ```shell curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{"id": 12, "access_level": 0}]' \ - --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [ { "id": 12, @@ -456,14 +456,14 @@ Example response: ```shell curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{"id": 12, "_destroy": true}]}' \ - --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [] } ``` diff --git a/doc/api/group_ssh_certificates.md b/doc/api/group_ssh_certificates.md new file mode 100644 index 00000000000..d6dc17a5c15 --- /dev/null +++ b/doc/api/group_ssh_certificates.md @@ -0,0 +1,115 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Group SSH certificates API **(PREMIUM ALL)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/421915) in GitLab 16.4 [with a flag](../user/feature_flags.md) named `ssh_certificates_rest_endpoints`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `ssh_certificates_rest_endpoints`. +On GitLab.com, this feature is not available. + +Use this API to create, read and delete SSH certificates for a group. +Only top-level groups can store SSH certificates. +To use this API you must [authenticate yourself](rest/index.md#authentication) as an Administrator. + +## Get all SSH certificates for a particular group + +```plaintext +GET groups/:id/ssh_certificates +``` + +Parameters: + +| Attribute | Type | Required | Description | +| ---------- | ------ | -------- |----------------------| +| `id` | integer | Yes | The ID of the group. | + +By default, `GET` requests return 20 results at a time because the API results are paginated. +Read more on [pagination](rest/index.md#pagination). + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://primary.example.com/api/v4/groups/90/ssh_certificates" +``` + +Example response: + +```json +[ + { + "id": 12345, + "title": "SSH Title 1", + "key": "ssh-rsa AAAAB3NzaC1ea2dAAAADAQABAAAAgQDGbLkF44ScxRQi2FfA7VsHgGqptguSbmW26jkJhEiRZpGS4/+UzaaSqc8Psw2OhSsKc5QwfrB/ANpO4LhOjDzhf2FuD8ACkv3R7XtaJ+rN6PlyzoBfLAiSyzxhEoMFDBprTgaiZKgg2yQ9dRH55w3f6XMZ4hnaUae53nQgfQLxFw== example@gitlab.com", + "created_at": "2023-09-08T12:39:00.172Z" + }, + { + "id":12346, + "title":"SSH Title 2", + "key": "ssh-rsa AAAAB3NzaC1ac2EAAAADAQABAAAAgQDTl/hHfu1F/KlR+QfgM2wUmyxcN5YeiaWluEGIrfXUeJuI+bK6xjpE3+2afHDYtE9VQkeL32KRjefX2d72Jeoa68ewt87Vn8CcGkUTOTpHNzeL8pHMKFs3m7ArSBxNg5vTdgAsq5dbDGNtat7b2WCHTNvtWoON1Jetne30uW2EwQ== example@gitlab.com", + "created_at": "2023-09-08T12:39:00.244Z" + } +] +``` + +## Create SSH Certificate + +Create a new SSH certificate in the group. + +```plaintext +POST /groups/:id/ssh_certificates +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-----------|------------| -------- |---------------------------------------| +| `id` | integer | Yes | The ID of the group. | +| `key` | string | Yes | The public key of the SSH certificate.| +| `title` | string | Yes | The title of the SSH certificate. | + +Example request: + +```shell +curl --request POST \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/groups/5/ssh_certificates?title=newtitle&key=ssh-rsa+REDACTED+example%40gitlab.com" +``` + +Example response: + +```json +{ + "id": 54321, + "title": "newtitle", + "key": "ssh-rsa ssh-rsa AAAAB3NzaC1ea2dAAAADAQABAAAAgQDGbLkF44ScxRQi2FfA7VsHgGqptguSbmW26jkJhEiRZpGS4/+UzaaSqc8Psw2OhSsKc5QwfrB/ANpO4LhOjDzhf2FuD8ACkv3R7XtaJ+rN6PlyzoBfLAiSyzxhEoMFDBprTgaiZKgg2yQ9dRH55w3f6XMZ4hnaUae53nQgfQLxFw== example@gitlab.com", + "created_at": "2023-09-08T12:39:00.172Z" +} +``` + +## Delete group SSH certificate + +Delete a SSH certificate from a group. + +```plaintext +DELETE /groups/:id/ssh_certificate/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-----------|---------| -------- |-------------------------------| +| `id` | integer | Yes | The ID of the group | +| `id` | integer | Yes | The ID of the SSH certificate | + +Example request: + +```shell +curl --request DELETE \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/groups/5/ssh_certificates/12345" +``` diff --git a/doc/api/group_wikis.md b/doc/api/group_wikis.md index 7396758ac40..2d758779f79 100644 --- a/doc/api/group_wikis.md +++ b/doc/api/group_wikis.md @@ -207,7 +207,7 @@ Example response: { "file_name" : "dk.png", "file_path" : "uploads/6a061c4cf9f1c28cb22c384b4b8d4e3c/dk.png", - "branch" : "master", + "branch" : "main", "link" : { "url" : "uploads/6a061c4cf9f1c28cb22c384b4b8d4e3c/dk.png", "markdown" : "" diff --git a/doc/api/groups.md b/doc/api/groups.md index 930a682c157..9ea37f4bb7c 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -38,7 +38,7 @@ Parameters: | `owned` | boolean | no | Limit to groups explicitly owned by the current user | | `min_access_level` | integer | no | Limit to groups where current user has at least this [role (`access_level`)](members.md#roles) | | `top_level_only` | boolean | no | Limit to top level groups, excluding all subgroups | -| `repository_storage` **(PREMIUM)** | string | no | Filter by repository storage used by the group _(administrators only)_. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419643) in GitLab 16.3 | +| `repository_storage` **(PREMIUM ALL)** | string | no | Filter by repository storage used by the group _(administrators only)_. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419643) in GitLab 16.3 | ```plaintext GET /groups @@ -314,7 +314,7 @@ Parameters: | `include_subgroups` | boolean | no | Include projects in subgroups of this group. Default is `false` | | `min_access_level` | integer | no | Limit to projects where current user has at least this [role (`access_level`)](members.md#roles) | | `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (administrators only) | -| `with_security_reports` **(ULTIMATE)** | boolean | no | Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | +| `with_security_reports` **(ULTIMATE ALL)** | boolean | no | Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | 1. Order by similarity: Orders the results by a similarity score calculated from the provided `search` URL parameter. When using `order_by=similarity`, the `sort` parameter is ignored. When the `search` @@ -327,7 +327,7 @@ Example response: { "id": 9, "description": "foo", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "archived": false, @@ -407,13 +407,13 @@ Example response: "path":"html5-boilerplate", "path_with_namespace":"h5bp/html5-boilerplate", "created_at":"2020-04-27T06:13:22.642Z", - "default_branch":"master", + "default_branch":"main", "tag_list":[], //deprecated, use `topics` instead "topics":[], "ssh_url_to_repo":"ssh://git@gitlab.com/h5bp/html5-boilerplate.git", "http_url_to_repo":"https://gitlab.com/h5bp/html5-boilerplate.git", "web_url":"https://gitlab.com/h5bp/html5-boilerplate", - "readme_url":"https://gitlab.com/h5bp/html5-boilerplate/-/blob/master/README.md", + "readme_url":"https://gitlab.com/h5bp/html5-boilerplate/-/blob/main/README.md", "avatar_url":null, "star_count":0, "forks_count":4, @@ -573,7 +573,7 @@ Example response: { "id": 7, "description": "Voluptas veniam qui et beatae voluptas doloremque explicabo facilis.", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "archived": false, @@ -612,7 +612,7 @@ Example response: { "id": 6, "description": "Aspernatur omnis repudiandae qui voluptatibus eaque.", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "archived": false, @@ -653,7 +653,7 @@ Example response: { "id": 8, "description": "Velit eveniet provident fugiat saepe eligendi autem.", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "archived": false, @@ -829,10 +829,10 @@ Parameters: | `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | | `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | | `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `membership_lock` **(PREMIUM)** | boolean | no | Users cannot be added to projects in this group. | +| `membership_lock` **(PREMIUM ALL)** | boolean | no | Users cannot be added to projects in this group. | | `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Additional compute minutes for this group. | | `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Maximum number of monthly compute minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | -| `wiki_access_level` **(PREMIUM)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | +| `wiki_access_level` **(PREMIUM ALL)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | ### Options for `default_branch_protection` @@ -893,8 +893,8 @@ GET /groups/:id/transfer_locations | Attribute | Type | Required | Description | |-------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the group to be transferred](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | The group names to search for. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the group to be transferred](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | The group names to search for. | Example request: @@ -988,17 +988,17 @@ PUT /groups/:id | `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | | `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | | `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Additional compute minutes for this group. | -| `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | -| `membership_lock` **(PREMIUM)** | boolean | no | Users cannot be added to projects in this group. | -| `prevent_forking_outside_group` **(PREMIUM)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces. | +| `file_template_project_id` **(PREMIUM ALL)** | integer | no | The ID of a project to load custom file templates from. | +| `membership_lock` **(PREMIUM ALL)** | boolean | no | Users cannot be added to projects in this group. | +| `prevent_forking_outside_group` **(PREMIUM ALL)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces. | | `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Maximum number of monthly compute minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | -| `unique_project_download_limit` **(ULTIMATE)** | integer | no | Maximum number of unique projects a user can download in the specified time period before they are banned. Available only on top-level groups. Default: 0, Maximum: 10,000. | -| `unique_project_download_limit_interval_in_seconds` **(ULTIMATE)** | integer | no | Time period during which a user can download a maximum amount of projects before they are banned. Available only on top-level groups. Default: 0, Maximum: 864,000 seconds (10 days). | -| `unique_project_download_limit_allowlist` **(ULTIMATE)** | array of strings | no | List of usernames excluded from the unique project download limit. Available only on top-level groups. Default: `[]`, Maximum: 100 usernames. | -| `unique_project_download_limit_alertlist` **(ULTIMATE)** | array of integers | no | List of user IDs that are emailed when the unique project download limit is exceeded. Available only on top-level groups. Default: `[]`, Maximum: 100 user IDs. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110201) in GitLab 15.9. | -| `auto_ban_user_on_excessive_projects_download` **(ULTIMATE)** | boolean | no | When enabled, users are automatically banned from the group when they download more than the maximum number of unique projects specified by `unique_project_download_limit` and `unique_project_download_limit_interval_in_seconds`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94159) in GitLab 15.4. | -| `ip_restriction_ranges` **(PREMIUM)** | string | no | Comma-separated list of IP addresses or subnet masks to restrict group access. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351493) in GitLab 15.4. | -| `wiki_access_level` **(PREMIUM)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | +| `unique_project_download_limit` **(ULTIMATE ALL)** | integer | no | Maximum number of unique projects a user can download in the specified time period before they are banned. Available only on top-level groups. Default: 0, Maximum: 10,000. | +| `unique_project_download_limit_interval_in_seconds` **(ULTIMATE ALL)** | integer | no | Time period during which a user can download a maximum amount of projects before they are banned. Available only on top-level groups. Default: 0, Maximum: 864,000 seconds (10 days). | +| `unique_project_download_limit_allowlist` **(ULTIMATE ALL)** | array of strings | no | List of usernames excluded from the unique project download limit. Available only on top-level groups. Default: `[]`, Maximum: 100 usernames. | +| `unique_project_download_limit_alertlist` **(ULTIMATE ALL)** | array of integers | no | List of user IDs that are emailed when the unique project download limit is exceeded. Available only on top-level groups. Default: `[]`, Maximum: 100 user IDs. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110201) in GitLab 15.9. | +| `auto_ban_user_on_excessive_projects_download` **(ULTIMATE ALL)** | boolean | no | When enabled, users are automatically banned from the group when they download more than the maximum number of unique projects specified by `unique_project_download_limit` and `unique_project_download_limit_interval_in_seconds`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94159) in GitLab 15.4. | +| `ip_restriction_ranges` **(PREMIUM ALL)** | string | no | Comma-separated list of IP addresses or subnet masks to restrict group access. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351493) in GitLab 15.4. | +| `wiki_access_level` **(PREMIUM ALL)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | NOTE: The `projects` and `shared_projects` attributes in the response are deprecated and [scheduled for removal in API v5](https://gitlab.com/gitlab-org/gitlab/-/issues/213797). @@ -1039,7 +1039,7 @@ Example response: { "id": 9, "description": "foo", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "public": false, @@ -1161,8 +1161,8 @@ Parameters: | Attribute | Type | Required | Description | |----------------------|------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | -| `permanently_remove` **(PREMIUM)** | boolean/string | no | Immediately deletes a subgroup if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368276) in GitLab 15.4 | -| `full_path` **(PREMIUM)** | string | no | Full path of subgroup to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368276) in GitLab 15.4. To find the subgroup path, see the [group details](groups.md#details-of-a-group) | +| `permanently_remove` **(PREMIUM ALL)** | boolean/string | no | Immediately deletes a subgroup if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368276) in GitLab 15.4 | +| `full_path` **(PREMIUM ALL)** | string | no | Full path of subgroup to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368276) in GitLab 15.4. To find the subgroup path, see the [group details](groups.md#details-of-a-group) | The response is `202 Accepted` if the user has authorization. diff --git a/doc/api/index.md b/doc/api/index.md index 7cb25c4ce17..8837c8a6016 100644 --- a/doc/api/index.md +++ b/doc/api/index.md @@ -6,13 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Develop with GitLab **(FREE ALL)** -Automate and interact with GitLab, and integrate with external applications. +Automate with GitLab and integrate with external applications. - [Integrations](../integration/index.md) - [Webhooks](../user/project/integrations/webhooks.md) - [REST API](rest/index.md) - [GraphQL API](graphql/index.md) - [OAuth 2.0 identity provider API](oauth2.md) -- [GitLab CLI (glab)](../integration/glab/index.md) -- [Visual Studio Code extension](../user/project/repository/vscode.md) -- [Code Suggestions](../user/project/repository/code_suggestions.md) +- [Editor and IDE extensions](../editor_extensions/index.md) +- [Code Suggestions](../user/project/repository/code_suggestions/index.md) diff --git a/doc/api/integrations.md b/doc/api/integrations.md index 530506c9bed..f8bfa1279d4 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -17,9 +17,9 @@ This API requires an access token with the Maintainer or Owner role. ## List all active integrations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21330) in GitLab 12.7. +> `vulnerability_events` field [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131831) in GitLab 16.5. -Get a list of all active project integrations. +Get a list of all active project integrations. The `vulnerability_events` field is only available for GitLab Enterprise Edition. ```plaintext GET /projects/:id/integrations @@ -49,7 +49,8 @@ Example response: "pipeline_events": true, "wiki_page_events": true, "job_events": true, - "comment_on_event_enabled": true + "comment_on_event_enabled": true, + "vulnerability_events": true }, { "id": 76, @@ -71,7 +72,8 @@ Example response: "pipeline_events": true, "wiki_page_events": true, "job_events": true, - "comment_on_event_enabled": true + "comment_on_event_enabled": true, + "vulnerability_events": true } ] ``` @@ -434,6 +436,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `token` | string | true | The Telegram bot token. For example, `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11`. | | `room` | string | true | Unique identifier for the target chat or the username of the target channel (in the format `@channelusername`) | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | | `push_events` | boolean | true | Enable notifications for push events | | `issues_events` | boolean | true | Enable notifications for issue events | | `confidential_issues_events` | boolean | true | Enable notifications for confidential issue events | diff --git a/doc/api/issue_links.md b/doc/api/issue_links.md index a7cacd64cbb..653f21e2f0b 100644 --- a/doc/api/issue_links.md +++ b/doc/api/issue_links.md @@ -77,9 +77,9 @@ Supported attributes: | Attribute | Type | Required | Description | |-----------------|----------------|------------------------|-----------------------------------------------------------------------------| -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `issue_iid` | integer | **{check-circle}** Yes | Internal ID of a project's issue. | -| `issue_link_id` | integer/string | **{check-circle}** Yes | ID of an issue relationship. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `issue_iid` | integer | Yes | Internal ID of a project's issue. | +| `issue_link_id` | integer/string | Yes | ID of an issue relationship. | Response body attributes: diff --git a/doc/api/issues.md b/doc/api/issues.md index f318515e0a6..07678f3ca42 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -64,13 +64,13 @@ GET /issues?state=opened | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | -| `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ -| `health_status` **(ULTIMATE)** | string | no | Return issues with the specified `health_status`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370721) in GitLab 15.4)._ In [GitLab 15.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/370721), `None` returns issues with no health status assigned, and `Any` returns issues with a health status assigned. +| `epic_id` **(PREMIUM ALL)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ +| `health_status` **(ULTIMATE ALL)** | string | no | Return issues with the specified `health_status`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370721) in GitLab 15.4)._ In [GitLab 15.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/370721), `None` returns issues with no health status assigned, and `Any` returns issues with a health status assigned. | `iids[]` | integer array | no | Return only the issues having the given `iid` | | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | | `issue_type` | string | no | Filter to a given type of issue. One of `issue`, `incident`, or `test_case`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260375) in GitLab 13.12)_ | -| `iteration_id` **(PREMIUM)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | -| `iteration_title` **(PREMIUM)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_id` **(PREMIUM ALL)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_title` **(PREMIUM ALL)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. Using `None` or `Any` will be [deprecated in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/336044). Please use `milestone_id` attribute instead. `milestone` and `milestone_id` are mutually exclusive. | | `milestone_id` | string | no | Returns issues assigned to milestones with a given timebox value (`None`, `Any`, `Upcoming`, and `Started`). `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. `Upcoming` lists all issues assigned to milestones due in the future. `Started` lists all issues assigned to open, started milestones. `milestone` and `milestone_id` are mutually exclusive. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335939) in GitLab 14.3)_ | @@ -84,7 +84,7 @@ GET /issues?state=opened | `state` | string | no | Return `all` issues or just those that are `opened` or `closed` | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `weight` **(PREMIUM)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | +| `weight` **(PREMIUM ALL)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `with_labels_details` | boolean | no | If `true`, the response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. The `description_html` attribute was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) in GitLab 12.7| ```shell @@ -294,12 +294,12 @@ GET /groups/:id/issues?state=opened | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | -| `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ +| `epic_id` **(PREMIUM ALL)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `id` | integer/string | yes | The global ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | integer array | no | Return only the issues having the given `iid` | | `issue_type` | string | no | Filter to a given type of issue. One of `issue`, `incident`, or `test_case`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260375) in GitLab 13.12)_ | -| `iteration_id` **(PREMIUM)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | -| `iteration_title` **(PREMIUM)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_id` **(PREMIUM ALL)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_title` **(PREMIUM ALL)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | @@ -312,7 +312,7 @@ GET /groups/:id/issues?state=opened | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `weight` **(PREMIUM)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | +| `weight` **(PREMIUM ALL)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `with_labels_details` | boolean | no | If `true`, the response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. The `description_html` attribute was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) in GitLab 12.7 | ```shell @@ -498,12 +498,12 @@ GET /projects/:id/issues?state=opened | `created_after` | datetime | no | Return issues created on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `created_before` | datetime | no | Return issues created on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `due_date` | string | no | Return issues that have no due date, are overdue, or whose due date is this week, this month, or between two weeks ago and next month. Accepts: `0` (no due date), `any`, `today`, `tomorrow`, `overdue`, `week`, `month`, `next_month_and_previous_two_weeks`. | -| `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ +| `epic_id` **(PREMIUM ALL)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `id` | integer/string | yes | The global ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | integer array | no | Return only the issues having the given `iid` | | `issue_type` | string | no | Filter to a given type of issue. One of `issue`, `incident`, or `test_case`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260375) in GitLab 13.12)_ | -| `iteration_id` **(PREMIUM)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | -| `iteration_title` **(PREMIUM)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_id` **(PREMIUM ALL)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | +| `iteration_title` **(PREMIUM ALL)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | @@ -515,7 +515,7 @@ GET /projects/:id/issues?state=opened | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `updated_after` | datetime | no | Return issues updated on or after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | | `updated_before` | datetime | no | Return issues updated on or before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `weight` **(PREMIUM)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | +| `weight` **(PREMIUM ALL)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `with_labels_details` | boolean | no | If `true`, the response returns more details for each label in labels field: `:name`, `:color`, `:description`, `:description_html`, `:text_color`. Default is `false`. `description_html` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21413) in GitLab 12.7 | ```shell @@ -1004,14 +1004,14 @@ POST /projects/:id/issues | Attribute | Type | Required | Description | |-------------------------------------------|----------------|----------|--------------| | `assignee_id` | integer | no | The ID of the user to assign the issue to. Only appears on GitLab Free. | -| `assignee_ids` **(PREMIUM)** | integer array | no | The IDs of the users to assign the issue to. | +| `assignee_ids` **(PREMIUM ALL)** | integer array | no | The IDs of the users to assign the issue to. | | `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | | `created_at` | string | no | When the issue was created. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | | `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | | `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This fills out the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | | `due_date` | string | no | The due date. Date time string in the format `YYYY-MM-DD`, for example `2016-03-11` | -| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | -| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/35157) in API version 5) | +| `epic_id` **(PREMIUM ALL)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | +| `epic_iid` **(PREMIUM ALL)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/35157) in API version 5) | | `id` | integer/string | yes | The global ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `iid` | integer/string | no | The internal ID of the project's issue (requires administrator or project owner rights) | | `issue_type` | string | no | The type of issue. One of `issue`, `incident`, or `test_case`. Default is `issue`. | @@ -1019,7 +1019,7 @@ POST /projects/:id/issues | `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This fills out the issue with a default description and mark all discussions as resolved. When passing a description or title, these values take precedence over the default values.| | `milestone_id` | integer | no | The global ID of a milestone to assign issue. To find the `milestone_id` associated with a milestone, view an issue with the milestone assigned and [use the API](#single-project-issue) to retrieve the issue's details. | | `title` | string | yes | The title of an issue | -| `weight` **(PREMIUM)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | +| `weight` **(PREMIUM ALL)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug" @@ -1178,8 +1178,8 @@ PUT /projects/:id/issues/:issue_iid | `description` | string | no | The description of an issue. Limited to 1,048,576 characters. | | `discussion_locked` | boolean | no | Flag indicating if the issue's discussion is locked. If the discussion is locked only project members can add or edit comments. | | `due_date` | string | no | The due date. Date time string in the format `YYYY-MM-DD`, for example `2016-03-11` | -| `epic_id` **(PREMIUM)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | -| `epic_iid` **(PREMIUM)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/35157) in API version 5) | +| `epic_id` **(PREMIUM ALL)** | integer | no | ID of the epic to add the issue to. Valid values are greater than or equal to 0. | +| `epic_iid` **(PREMIUM ALL)** | integer | no | IID of the epic to add the issue to. Valid values are greater than or equal to 0. (deprecated, [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/35157) in API version 5) | | `id` | integer/string | yes | The global ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `issue_iid` | integer | yes | The internal ID of a project's issue | | `issue_type` | string | no | Updates the type of issue. One of `issue`, `incident`, or `test_case`. | @@ -1189,7 +1189,7 @@ PUT /projects/:id/issues/:issue_iid | `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it | | `title` | string | no | The title of an issue | | `updated_at` | string | no | When the issue was updated. Date time string, ISO 8601 formatted, for example `2016-03-11T03:45:40Z` (requires administrator or project owner rights). Empty string or null values are not accepted.| -| `weight` **(PREMIUM)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. 0 | +| `weight` **(PREMIUM ALL)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. 0 | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues/85?state_event=close" @@ -1515,10 +1515,10 @@ POST /projects/:id/issues/:issue_iid/clone | Attribute | Type | Required | Description | | --------------- | -------------- | ---------------------- | --------------------------------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `issue_iid` | integer | **{check-circle}** Yes | Internal ID of a project's issue. | -| `to_project_id` | integer | **{check-circle}** Yes | ID of the new project. | -| `with_notes` | boolean | **{dotted-circle}** No | Clone the issue with [notes](notes.md). Default is `false`. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `issue_iid` | integer | Yes | Internal ID of a project's issue. | +| `to_project_id` | integer | Yes | ID of the new project. | +| `with_notes` | boolean | No | Clone the issue with [notes](notes.md). Default is `false`. | ```shell curl --request POST \ @@ -2337,7 +2337,7 @@ Example response: "state": "opened", "created_at": "2017-04-06T18:33:34.168Z", "updated_at": "2017-04-09T20:10:24.983Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "feature.custom-highlighting", "upvotes": 0, "downvotes": 0, diff --git a/doc/api/issues_statistics.md b/doc/api/issues_statistics.md index d2148b001cb..2c3383aa328 100644 --- a/doc/api/issues_statistics.md +++ b/doc/api/issues_statistics.md @@ -41,7 +41,7 @@ GET /issues_statistics?confidential=true | `author_username` | string | no | Return issues created by the given `username`. Similar to `author_id` and mutually exclusive with `author_id`. | | `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. | | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In GitLab CE `assignee_username` array should only contain a single value or an invalid parameter error is returned otherwise. | -| `epic_id` **(PREMIUM)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ +| `epic_id` **(PREMIUM ALL)** | integer | no | Return issues associated with the given epic ID. `None` returns issues that are not associated with an epic. `Any` returns issues that are associated with an epic. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46887) in GitLab 13.6)_ | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | | `iids[]` | integer array | no | Return only the issues having the given `iid` | | `search` | string | no | Search issues against their `title` and `description` | diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index 5a3861f888e..a65de457581 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -12,6 +12,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w Get the job's artifacts zipped archive of a project. +If you use cURL to download artifacts from GitLab.com, use the `--location` parameter +as the request might redirect through a CND. + ```plaintext GET /projects/:id/jobs/:job_id/artifacts ``` @@ -20,16 +23,16 @@ GET /projects/:id/jobs/:job_id/artifacts |---------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `job_id` | integer | Yes | ID of a job. | -| `job_token` **(PREMIUM)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request using the `PRIVATE-TOKEN` header: ```shell -curl --location --output artifacts.zip --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts" +curl --location --output artifacts.zip --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts" ``` To use this in a [`script` definition](../ci/yaml/index.md#script) inside -`.gitlab-ci.yml` **(PREMIUM)**, you can use either: +`.gitlab-ci.yml` **(PREMIUM ALL)**, you can use either: - The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. For example, the following job downloads the artifacts of the job with ID @@ -69,6 +72,9 @@ the given reference name and job, provided the job finished successfully. This is the same as [getting the job's artifacts](#get-job-artifacts), but by defining the job's name instead of its ID. +If you use cURL to download artifacts from GitLab.com, use the `--location` parameter +as the request might redirect through a CND. + NOTE: If a pipeline is [parent of other child pipelines](../ci/pipelines/downstream_pipelines.md#parent-child-pipelines), artifacts are searched in hierarchical order from parent to child. For example, if both parent and @@ -85,7 +91,7 @@ Parameters | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `ref_name` | string | Yes | Branch or tag name in repository. HEAD or SHA references are not supported. | | `job` | string | Yes | The name of the job. | -| `job_token` **(PREMIUM)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request using the `PRIVATE-TOKEN` header: @@ -94,7 +100,7 @@ curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.ex ``` To use this in a [`script` definition](../ci/yaml/index.md#script) inside -`.gitlab-ci.yml` **(PREMIUM)**, you can use either: +`.gitlab-ci.yml` **(PREMIUM ALL)**, you can use either: - The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. For example, the following job downloads the artifacts of the `test` job @@ -132,6 +138,9 @@ Download a single artifact file from a job with a specified ID from inside the job's artifacts zipped archive. The file is extracted from the archive and streamed to the client. +If you use cURL to download artifacts from GitLab.com, use the `--location` parameter +as the request might redirect through a CND. + ```plaintext GET /projects/:id/jobs/:job_id/artifacts/*artifact_path ``` @@ -143,7 +152,7 @@ Parameters | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `job_id` | integer | Yes | The unique job identifier. | | `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | -| `job_token` **(PREMIUM)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request: @@ -172,6 +181,9 @@ Artifacts for [parent and child pipelines](../ci/pipelines/downstream_pipelines. are searched in hierarchical order from parent to child. For example, if both parent and child pipelines have a job with the same name, the artifact from the parent pipeline is returned. +If you use cURL to download artifacts from GitLab.com, use the `--location` parameter +as the request might redirect through a CND. + ```plaintext GET /projects/:id/jobs/artifacts/:ref_name/raw/*artifact_path?job=name ``` @@ -184,7 +196,7 @@ Parameters: | `ref_name` | string | Yes | Branch or tag name in repository. `HEAD` or `SHA` references are not supported. | | `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | | `job` | string | Yes | The name of the job. | -| `job_token` **(PREMIUM)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request: diff --git a/doc/api/license.md b/doc/api/license.md index 39da6af30d4..d393229fcf2 100644 --- a/doc/api/license.md +++ b/doc/api/license.md @@ -265,3 +265,40 @@ Returns: | Attribute | Type | Description | |:-----------------------------|:--------------|:------------------------------------------| | `success` | boolean | Whether the request succeeded or not. | + +## Retrieve usage information about the current license + +Gets usage information about the current license and exports it in CSV format. + +```plaintext +GET /license/usage_export.csv +``` + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license/usage_export.csv" +``` + +Example response: + +```csv +License Key,"eyJkYXRhIjoib1EwRWZXU3RobDY2Yl= +" +Email,user@example.com +License Start Date,2023-02-22 +License End Date,2024-02-22 +Company,Example Corp. +Generated At,2023-09-05 06:56:23 +"","" +Date,Billable User Count +2023-07-11 12:00:05,21 +2023-07-13 12:00:06,21 +2023-08-16 12:00:02,21 +2023-09-04 12:00:12,21 + +``` + +Returns: + +- `202 Accepted` if the request to refresh billable users is successfully initiated. +- `403 Forbidden` if the current user in not permitted to refresh billable users for the license. +- `404 Not Found` if the license could not be found. diff --git a/doc/api/linked_epics.md b/doc/api/linked_epics.md index 1cb70fa38f0..4d0f1dbe6a0 100644 --- a/doc/api/linked_epics.md +++ b/doc/api/linked_epics.md @@ -17,14 +17,14 @@ Get a list of a given group's related epic links within group and sub-groups, fi The user needs to have access to the `source_epic` and `target_epic` to access the related epic link. ```plaintext -GET /groups/:id/epics/related_epic_links +GET /groups/:id/related_epic_links ``` Supported attributes: | Attribute | Type | Required | Description | | ---------- | -------------- | ---------------------- | ------------------------------------------------------------------------- | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | | `created_after` | string | no | Return related epic links created on or after the given time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | | `created_before` | string | no | Return related epic links created on or before the given time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | | `updated_after` | string | no | Return related epic links updated on or after the given time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | @@ -145,8 +145,8 @@ Supported attributes: | Attribute | Type | Required | Description | | ---------- | -------------- | ---------------------- | ------------------------------------------------------------------------- | -| `epic_iid` | integer | **{check-circle}** Yes | Internal ID of a group's epic | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | +| `epic_iid` | integer | Yes | Internal ID of a group's epic | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding). | Example request: @@ -224,11 +224,11 @@ Supported attributes: | Attribute | Type | Required | Description | |---------------------|----------------|-----------------------------|---------------------------------------| -| `epic_iid` | integer | **{check-circle}** Yes | Internal ID of a group's epic. | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `target_epic_iid` | integer/string | **{check-circle}** Yes | Internal ID of a target group's epic. | -| `target_group_id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the target group](rest/index.md#namespaced-path-encoding). | -| `link_type` | string | **{dotted-circle}** No | Type of the relation (`relates_to`, `blocks`, `is_blocked_by`), defaults to `relates_to`. | +| `epic_iid` | integer | Yes | Internal ID of a group's epic. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `target_epic_iid` | integer/string | Yes | Internal ID of a target group's epic. | +| `target_group_id` | integer/string | Yes | ID or [URL-encoded path of the target group](rest/index.md#namespaced-path-encoding). | +| `link_type` | string | No | Type of the relation (`relates_to`, `blocks`, `is_blocked_by`), defaults to `relates_to`. | Example request: @@ -346,9 +346,9 @@ Supported attributes: | Attribute | Type | Required | Description | |--------------------------|----------------|-----------------------------|---------------------------------------| -| `epic_iid` | integer | **{check-circle}** Yes | Internal ID of a group's epic. | -| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `related_epic_link_id` | integer/string | **{check-circle}** Yes | Internal ID of a related epic link. | +| `epic_iid` | integer | Yes | Internal ID of a group's epic. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `related_epic_link_id` | integer/string | Yes | Internal ID of a related epic link. | Example request: diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md deleted file mode 100644 index d9f74ddddb2..00000000000 --- a/doc/api/managed_licenses.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -stage: Fulfillment -group: Utilization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: 'index.md' ---- - -# Managed Licenses API (removed) **(ULTIMATE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390417) in GitLab 15.9 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/397067) in 16.0. - -<!-- This redirect file can be deleted after <2023-08-22>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/api/member_roles.md b/doc/api/member_roles.md index 9d3e51efabd..76ae681bfb4 100644 --- a/doc/api/member_roles.md +++ b/doc/api/member_roles.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -12,6 +12,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Admin vulnerability added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) in GitLab 16.1. > - [Read dependency added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126247) in GitLab 16.3. > - [Name and description fields added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126423) in GitLab 16.3. +> - [Admin merge request introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128302) in GitLab 16.4 [with a flag](../administration/feature_flags.md) named `admin_merge_request`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `admin_merge_request`. +On GitLab.com, this feature is not available. ## List all member roles of a group @@ -34,6 +39,7 @@ If successful, returns [`200`](rest/index.md#status-codes) and the following res | `[].description` | string | The description of the member role. | | `[].group_id` | integer | The ID of the group that the member role belongs to. | | `[].base_access_level` | integer | Base access level for member role. Valid values are 10 (Guest), 20 (Reporter), 30 (Developer), 40 (Maintainer), or 50 (Owner).| +| `[].admin_merge_request` | boolean | Permission to admin project merge requests and enables the ability to `download_code`. | | `[].admin_vulnerability` | boolean | Permission to admin project vulnerabilities. | | `[].read_code` | boolean | Permission to read project code. | | `[].read_dependency` | boolean | Permission to read project dependencies. | @@ -42,7 +48,7 @@ If successful, returns [`200`](rest/index.md#status-codes) and the following res Example request: ```shell -curl --header "Authorization: Bearer <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/member_roles" +curl --header "Authorization: Bearer <your_access_token>" "https://gitlab.example.com/api/v4/groups/84/member_roles" ``` Example response: @@ -55,6 +61,7 @@ Example response: "description: "Custom guest that can read code", "group_id": 84, "base_access_level": 10, + "admin_merge_request": false, "admin_vulnerability": false, "read_code": true, "read_dependency": false, @@ -66,6 +73,7 @@ Example response: "description: "Custom guest that read and admin security entities", "group_id": 84, "base_access_level": 10, + "admin_merge_request": false, "admin_vulnerability": true, "read_code": false, "read_dependency": true, @@ -92,6 +100,7 @@ To add a member role to a group, the group must be at root-level (have no parent | `name` | string | yes | The name of the member role. | | `description` | string | no | The description of the member role. | | `base_access_level` | integer | yes | Base access level for configured role. Valid values are 10 (Guest), 20 (Reporter), 30 (Developer), 40 (Maintainer), or 50 (Owner).| +| `admin_merge_request` | boolean | no | Permission to admin project merge requests. | | `admin_vulnerability` | boolean | no | Permission to admin project vulnerabilities. | | `read_code` | boolean | no | Permission to read project code. | | `read_dependency` | boolean | no | Permission to read project dependencies. | @@ -106,6 +115,7 @@ If successful, returns [`201`](rest/index.md#status-codes) and the following att | `description` | string | The description of the member role. | | `group_id` | integer | The ID of the group that the member role belongs to. | | `base_access_level` | integer | Base access level for member role. | +| `admin_merge_request` | boolean | Permission to admin project merge requests. | | `admin_vulnerability` | boolean | Permission to admin project vulnerabilities. | | `read_code` | boolean | Permission to read project code. | | `read_dependency` | boolean | Permission to read project dependencies. | @@ -114,7 +124,7 @@ If successful, returns [`201`](rest/index.md#status-codes) and the following att Example request: ```shell - curl --request POST --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"name" : "Custom guest", "base_access_level" : 10, "read_code" : true}' "https://example.gitlab.com/api/v4/groups/:id/member_roles" + curl --request POST --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" --data '{"name" : "Custom guest", "base_access_level" : 10, "read_code" : true}' "https://gitlab.example.com/api/v4/groups/84/member_roles" ``` Example response: @@ -126,6 +136,7 @@ Example response: "description": null, "group_id": 84, "base_access_level": 10, + "admin_merge_requests": false, "admin_vulnerability": false, "read_code": true, "read_dependency": false, @@ -157,5 +168,5 @@ If successful, returns [`204`](rest/index.md#status-codes) and an empty response Example request: ```shell -curl --request DELETE --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" "https://example.gitlab.com/api/v4/groups/:group_id/member_roles/:member_role_id" +curl --request DELETE --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" "https://gitlab.example.com/api/v4/groups/84/member_roles/1" ``` diff --git a/doc/api/members.md b/doc/api/members.md index f7e3d6898ec..7e085a101bb 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -135,7 +135,7 @@ GET /projects/:id/members/all | `query` | string | no | A query string to search for members | | `user_ids` | array of integers | no | Filter the results on the given user IDs | | `show_seat_info` | boolean | no | Show seat information for users | -| `state` | string | no | Filter results by member state, one of `awaiting` or `active` **(PREMIUM)** | +| `state` | string | no | Filter results by member state, one of `awaiting` or `active` **(PREMIUM ALL)** | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/all" @@ -519,7 +519,7 @@ POST /projects/:id/members | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project or group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `user_id` | integer/string | yes | The user ID of the new member or multiple IDs separated by commas | -| `access_level` | integer | yes | A valid access level | +| `access_level` | integer | yes | [A valid access level](access_requests.md#valid-access-levels) | | `expires_at` | string | no | A date string in the format `YEAR-MONTH-DAY` | | `invite_source` | string | no | The source of the invitation that starts the member creation process. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/327120>`. | | `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | @@ -571,9 +571,9 @@ PUT /projects/:id/members/:user_id | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project or group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `user_id` | integer | yes | The user ID of the member | -| `access_level` | integer | yes | A valid access level | +| `access_level` | integer | yes | A [valid access level](access_requests.md#valid-access-levels) | | `expires_at` | string | no | A date string in the format `YEAR-MONTH-DAY` | -| `member_role_id` | integer | no | The ID of a member role **(ULTIMATE)** | +| `member_role_id` | integer | no | The ID of a member role **(ULTIMATE ALL)** | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/:user_id?access_level=40" diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index d64c71367a9..d00252da207 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -162,7 +162,7 @@ Supported attributes: "protected_branches": [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "access_level": 30, @@ -264,7 +264,7 @@ Supported attributes: "protected_branches": [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "access_level": 30, @@ -374,7 +374,7 @@ Supported attributes: "protected_branches": [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "access_level": 30, @@ -508,7 +508,7 @@ Supported attributes: "protected_branches": [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "access_level": 30, diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 5f637dd28a3..53a605c56f0 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -51,8 +51,8 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------------------------- | -------------- | -------- | ----------- | -| `approved_by_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`. Maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | -| `approver_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have specified all the users with the given `id` as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approved_by_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`. Maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | +| `approver_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have specified all the users with the given `id` as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `approved` | string | No | Filters merge requests by their `approved` status. `yes` returns only approved merge requests. `no` returns only non-approved merge requests. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3159) in GitLab 15.11. Available only when the feature flag `mr_approved_filter` is enabled. | | `assignee_id` | integer | No | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | | `author_id` | integer | No | Returns merge requests created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. | @@ -114,7 +114,7 @@ Supported attributes: "closed_at": null, "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -242,8 +242,8 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------------------------- | -------------- | -------- | ----------- | | `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `approved_by_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | -| `approver_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have specified all the users with the given `id` as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approved_by_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | +| `approver_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have specified all the users with the given `id` as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `approved` | string | No | Filters merge requests by their `approved` status. `yes` returns only approved merge requests. `no` returns only non-approved merge requests. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3159) in GitLab 15.11. Available only when the feature flag `mr_approved_filter` is enabled. | | `assignee_id` | integer | No | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | | `author_id` | integer | No | Returns merge requests created by the given user `id`. Mutually exclusive with `author_username`. | @@ -303,7 +303,7 @@ Supported attributes: "closed_at": null, "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -419,9 +419,9 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------------------------- | -------------- | -------- | ----------- | | `id` | integer or string | Yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `approved_by_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | -| `approved_by_usernames` **(PREMIUM)** | string array | No | Returns merge requests which have been approved by all the users with the given `username`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | -| `approver_ids` **(PREMIUM)** | integer array | No | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approved_by_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have been approved by all the users with the given `id`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | +| `approved_by_usernames` **(PREMIUM ALL)** | string array | No | Returns merge requests which have been approved by all the users with the given `username`, with a maximum of 5. `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | +| `approver_ids` **(PREMIUM ALL)** | integer array | No | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `approved` | string | No | Filters merge requests by their `approved` status. `yes` returns only approved merge requests. `no` returns only non-approved merge requests. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3159) in GitLab 15.11. Available only when the feature flag `mr_approved_filter` is enabled. | | `assignee_id` | integer | No | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | | `author_id` | integer | No | Returns merge requests created by the given user `id`. Mutually exclusive with `author_username`. | @@ -479,7 +479,7 @@ Supported attributes: "closed_at": null, "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -598,7 +598,7 @@ Supported attributes: | Attribute | Type | Description | |----------------------------------|------|-------------| -| `approvals_before_merge`| integer | **(PREMIUM)** Number of approvals required before this merge request can merge. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | +| `approvals_before_merge`| integer | **(PREMIUM ALL)** Number of approvals required before this merge request can merge. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | | `assignee` | object | First assignee of the merge request. | | `assignees` | array | Assignees of the merge request. | | `author` | object | User who created this merge request. | @@ -672,7 +672,7 @@ Supported attributes: "prepared_at": "2018-09-04T11:16:17.520Z", "closed_by": null, "closed_at": null, - "target_branch": "master", + "target_branch": "main", "source_branch": "manual-job-rules", "user_notes_count": 0, "upvotes": 0, @@ -1084,7 +1084,7 @@ Supported attributes: "new_path": "VERSION", "a_mode": "100644", "b_mode": "100644", - "diff": "--- a/VERSION\ +++ b/VERSION\ @@ -1 +1 @@\ -1.9.7\ +1.9.8", + "diff": "@@ -1 +1 @@\ -1.9.7\ +1.9.8", "new_file": false, "renamed_file": false, "deleted_file": false @@ -1140,7 +1140,7 @@ Example response: "new_path": "README", "a_mode": "100644", "b_mode": "100644", - "diff": "--- a/README\ +++ b/README\ @@ -1 +1 @@\ -Title\ +README", + "diff": "@@ -1 +1 @@\ -Title\ +README", "new_file": false, "renamed_file": false, "deleted_file": false @@ -1150,7 +1150,7 @@ Example response: "new_path": "VERSION", "a_mode": "100644", "b_mode": "100644", - "diff": "--- a/VERSION\ +++ b/VERSION\ @@ -1 +1 @@\ -1.9.7\ +1.9.8", + "diff": "@@\ -1.9.7\ +1.9.8", "new_file": false, "renamed_file": false, "deleted_file": false @@ -1179,7 +1179,7 @@ Supported attributes: { "id": 77, "sha": "959e04d7c7a30600c894bd3c0cd0e1ce7f42c11d", - "ref": "master", + "ref": "main", "status": "success" } ] @@ -1262,7 +1262,7 @@ POST /projects/:id/merge_requests | `target_branch` | string | Yes | The target branch. | | `title` | string | Yes | Title of MR. | | `allow_collaboration` | boolean | No | Allow commits from members who can merge to the target branch. | -| `approvals_before_merge` **(PREMIUM)** | integer | No | Number of approvals required before this can be merged (see below). To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | Number of approvals required before this can be merged (see below). To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | | `allow_maintainer_to_push` | boolean | No | Alias of `allow_collaboration`. | | `assignee_id` | integer | No | Assignee user ID. | | `assignee_ids` | integer array | No | The ID of the users to assign the merge request to. Set to `0` or provide an empty value to unassign all assignees. | @@ -1284,7 +1284,7 @@ POST /projects/:id/merge_requests "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -1439,7 +1439,7 @@ Must include at least one non-required attribute from above. "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -1617,7 +1617,7 @@ Supported attributes: "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -1819,7 +1819,7 @@ Supported attributes: "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -2121,7 +2121,7 @@ Example response: "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -2281,7 +2281,7 @@ Example response: "state": "merged", "created_at": "2017-04-29T08:46:00Z", "updated_at": "2017-04-29T08:46:00Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "test1", "upvotes": 0, "downvotes": 0, @@ -2568,7 +2568,8 @@ Example response: "created_at": "2016-07-26T14:44:48.926Z", "merge_request_id": 105, "state": "collected", - "real_size": "1" + "real_size": "1", + "patch_id_sha": "d504412d5b6e6739647e752aff8e468dde093f2f" }, { "id": 108, "head_commit_sha": "3eed087b29835c48015768f839d76e5ea8f07a24", @@ -2577,7 +2578,8 @@ Example response: "created_at": "2016-07-25T14:21:33.028Z", "merge_request_id": 105, "state": "collected", - "real_size": "1" + "real_size": "1", + "patch_id_sha": "72c30d1f0115fc1d2bb0b29b24dc2982cbcdfd32" }] ``` @@ -2620,6 +2622,7 @@ Example response: "merge_request_id": 105, "state": "collected", "real_size": "1", + "patch_id_sha": "d504412d5b6e6739647e752aff8e468dde093f2f", "commits": [{ "id": "33e2ee8579fda5bc36accc9c6fbd0b4fefda9e30", "short_id": "33e2ee85", @@ -2650,7 +2653,7 @@ Example response: "new_path": "LICENSE", "a_mode": "0", "b_mode": "100644", - "diff": "--- /dev/null\n+++ b/LICENSE\n@@ -0,0 +1,21 @@\n+The MIT License (MIT)\n+\n+Copyright (c) 2018 Administrator\n+\n+Permission is hereby granted, free of charge, to any person obtaining a copy\n+of this software and associated documentation files (the \"Software\"), to deal\n+in the Software without restriction, including without limitation the rights\n+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n+copies of the Software, and to permit persons to whom the Software is\n+furnished to do so, subject to the following conditions:\n+\n+The above copyright notice and this permission notice shall be included in all\n+copies or substantial portions of the Software.\n+\n+THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n+SOFTWARE.\n", + "diff": "@@ -0,0 +1,21 @@\n+The MIT License (MIT)\n+\n+Copyright (c) 2018 Administrator\n+\n+Permission is hereby granted, free of charge, to any person obtaining a copy\n+of this software and associated documentation files (the \"Software\"), to deal\n+in the Software without restriction, including without limitation the rights\n+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n+copies of the Software, and to permit persons to whom the Software is\n+furnished to do so, subject to the following conditions:\n+\n+The above copyright notice and this permission notice shall be included in all\n+copies or substantial portions of the Software.\n+\n+THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n+SOFTWARE.\n", "new_file": true, "renamed_file": false, "deleted_file": false diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 1ede530578b..569fcc3d644 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index c4b2d90f2c7..e498c3c91fb 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -38,7 +38,7 @@ If the `custom` level is used, specific email events can be controlled. Availabl - `success_pipeline` - `moved_project` - `merge_when_pipeline_succeeds` -- `new_epic` **(ULTIMATE)** +- `new_epic` **(ULTIMATE ALL)** ## Global notification settings @@ -94,7 +94,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab | `success_pipeline` | boolean | no | Enable/disable this notification | | `moved_project` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30371) in GitLab 13.3) | | `merge_when_pipeline_succeeds` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/244840) in GitLab 13.9) | -| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5863) in GitLab 11.3) **(ULTIMATE)** | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5863) in GitLab 11.3) **(ULTIMATE ALL)** | Example response: @@ -166,7 +166,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab | `success_pipeline` | boolean | no | Enable/disable this notification | | `moved_project` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30371) in GitLab 13.3) | | `merge_when_pipeline_succeeds` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/244840) in GitLab 13.9) | -| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5863) in GitLab 11.3) **(ULTIMATE)** | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5863) in GitLab 11.3) **(ULTIMATE ALL)** | Example responses: diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 75b50829ddd..00e8a4c86c6 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/packages.md b/doc/api/packages.md index ac692956f22..a378be26a24 100644 --- a/doc/api/packages.md +++ b/doc/api/packages.md @@ -380,7 +380,7 @@ Example response: "iid": 2, "project_id": 9, "sha": "e564015ac6cb3d8617647802c875b27d392f72a6", - "ref": "master", + "ref": "main", "status": "canceled", "source": "push", "created_at": "2023-02-01T12:23:23.694Z", diff --git a/doc/api/packages/debian_group_distributions.md b/doc/api/packages/debian_group_distributions.md index 0c7f4cdfeb8..68763a197aa 100644 --- a/doc/api/packages/debian_group_distributions.md +++ b/doc/api/packages/debian_group_distributions.md @@ -83,7 +83,7 @@ GET /groups/:id/-/debian_distributions/:codename | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The `codename` of a distribution. | +| `codename` | string | yes | The `codename` of a distribution. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/-/debian_distributions/unstable" @@ -122,7 +122,7 @@ GET /groups/:id/-/debian_distributions/:codename/key.asc | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The `codename` of a distribution. | +| `codename` | string | yes | The `codename` of a distribution. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/-/debian_distributions/unstable/key.asc" @@ -166,8 +166,8 @@ POST /groups/:id/-/debian_distributions | `version` | string | no | The version of the new Debian distribution. | | `description` | string | no | The description of the new Debian distribution. | | `valid_time_duration_seconds` | integer | no | The valid time duration (in seconds) of the new Debian distribution. | -| `components` | architectures | no | The new Debian distribution's list of components. | -| `architectures` | architectures | no | The new Debian distribution's list of architectures. | +| `components` | string array | no | The new Debian distribution's list of components. | +| `architectures` | string array | no | The new Debian distribution's list of architectures. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/-/debian_distributions?codename=sid" @@ -213,8 +213,8 @@ PUT /groups/:id/-/debian_distributions/:codename | `version` | string | no | The Debian distribution's new version. | | `description` | string | no | The Debian distribution's new description. | | `valid_time_duration_seconds` | integer | no | The Debian distribution's new valid time duration (in seconds). | -| `components` | architectures | no | The Debian distribution's new list of components. | -| `architectures` | architectures | no | The Debian distribution's new list of architectures. | +| `components` | string array | no | The Debian distribution's new list of components. | +| `architectures` | string array | no | The Debian distribution's new list of architectures. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/-/debian_distributions/unstable?suite=new-suite&valid_time_duration_seconds=604800" @@ -253,7 +253,7 @@ DELETE /groups/:id/-/debian_distributions/:codename | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The codename of the Debian distribution. | +| `codename` | string | yes | The codename of the Debian distribution. | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/-/debian_distributions/unstable" diff --git a/doc/api/packages/debian_project_distributions.md b/doc/api/packages/debian_project_distributions.md index 0a43546e2e1..952b75ceb0a 100644 --- a/doc/api/packages/debian_project_distributions.md +++ b/doc/api/packages/debian_project_distributions.md @@ -82,7 +82,7 @@ GET /projects/:id/debian_distributions/:codename | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The `codename` of a distribution. | +| `codename` | string | yes | The `codename` of a distribution. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions/unstable" @@ -121,7 +121,7 @@ GET /projects/:id/debian_distributions/:codename/key.asc | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The `codename` of a distribution. | +| `codename` | string | yes | The `codename` of a distribution. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions/unstable/key.asc" @@ -165,8 +165,8 @@ POST /projects/:id/debian_distributions | `version` | string | no | The new Debian distribution's version. | | `description` | string | no | The new Debian distribution's description. | | `valid_time_duration_seconds` | integer | no | The new Debian distribution's valid time duration (in seconds). | -| `components` | architectures | no | The new Debian distribution's list of components. | -| `architectures` | architectures | no | The new Debian distribution's list of architectures. | +| `components` | string array | no | The new Debian distribution's list of components. | +| `architectures` | string array | no | The new Debian distribution's list of architectures. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions?codename=sid" @@ -212,8 +212,8 @@ PUT /projects/:id/debian_distributions/:codename | `version` | string | no | The Debian distribution's new version. | | `description` | string | no | The Debian distribution's new description. | | `valid_time_duration_seconds` | integer | no | The Debian distribution's new valid time duration (in seconds). | -| `components` | architectures | no | The Debian distribution's new list of components. | -| `architectures` | architectures | no | The Debian distribution's new list of architectures. | +| `components` | string array | no | The Debian distribution's new list of components. | +| `architectures` | string array | no | The Debian distribution's new list of architectures. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions/unstable?suite=new-suite&valid_time_duration_seconds=604800" @@ -252,7 +252,7 @@ DELETE /projects/:id/debian_distributions/:codename | Attribute | Type | Required | Description | | ---------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `codename` | integer | yes | The Debian distribution's codename. | +| `codename` | string | yes | The Debian distribution's codename. | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions/unstable" diff --git a/doc/api/packages/nuget.md b/doc/api/packages/nuget.md index a549d6af086..ee304ab28df 100644 --- a/doc/api/packages/nuget.md +++ b/doc/api/packages/nuget.md @@ -425,10 +425,12 @@ Example response: } ``` -## V2 Feed Metadata Endpoint +## V2 Feed Metadata Endpoints > Introduced in GitLab 16.3. +### $metadata endpoint + Authentication is not required. Returns metadata for a V2 feed available endpoints: ```plaintext @@ -436,7 +438,7 @@ GET <route-prefix>/v2/$metadata ``` ```shell - curl "https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2/$metadata" +curl "https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2/$metadata" ``` Example response: @@ -475,3 +477,36 @@ Example response: </edmx:DataServices> </edmx:Edmx> ``` + +### OData package entry endpoints + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127667) in GitLab 16.4. + +| Endpoint | Description | +| -------- | ----------- | +| `GET projects/:id/packages/nuget/v2/Packages()?$filter=(tolower(Id) eq '<package_name>')` | Returns an OData XML document containing information about the package with the given name. | +| `GET projects/:id/packages/nuget/v2/FindPackagesById()?id='<package_name>'` | Returns an OData XML document containing information about the package with the given name. | +| `GET projects/:id/packages/nuget/v2/Packages(Id='<package_name>',Version='<package_version>')` | Returns an OData XML document containing information about the package with the given name and version. | + +NOTE: +GitLab doesn't receive an authentication token for the `Packages()` and `FindPackagesByID()` endpoints. +To not reveal the package version to unauthenticated users, the actual latest package version is not returned. Instead, a placeholder version is returned. +The latest version is obtained in the subsequent download request where the authentication token is sent. + +```shell +curl "https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2/Packages()?$filter=(tolower(Id) eq 'mynugetpkg')" +``` + +Example response: + +```xml +<entry xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:georss="http://www.georss.org/georss" xmlns:gml="http://www.opengis.net/gml" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xml:base="https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2"> + <id>https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2/Packages(Id='mynugetpkg',Version='0.0.0-latest-version')</id> + <category term="V2FeedPackage" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme"/> + <title type="text">mynugetpkg</title> + <content type="application/zip" src="https://gitlab.example.com/api/v4/projects/1/packages/nuget/v2/download/mynugetpkg/latest"/> + <m:properties> + <d:Version>0.0.0-latest-version</d:Version> + </m:properties> + </entry> +``` diff --git a/doc/api/personal_access_tokens.md b/doc/api/personal_access_tokens.md index 901f99caee7..2131a29eb5b 100644 --- a/doc/api/personal_access_tokens.md +++ b/doc/api/personal_access_tokens.md @@ -45,14 +45,14 @@ Supported attributes: | Attribute | Type | Required | Description | |---------------------|----------------|----------|---------------------| -| `created_after` | datetime (ISO 8601) | **{dotted-circle}** No | Limit results to PATs created after specified time. | -| `created_before` | datetime (ISO 8601) | **{dotted-circle}** No | Limit results to PATs created before specified time. | -| `last_used_after` | datetime (ISO 8601) | **{dotted-circle}** No | Limit results to PATs last used after specified time. | -| `last_used_before` | datetime (ISO 8601) | **{dotted-circle}** No | Limit results to PATs last used before specified time. | -| `revoked` | boolean | **{dotted-circle}** No | Limit results to PATs with specified revoked state. Valid values are `true` and `false`. | -| `search` | string | **{dotted-circle}** No | Limit results to PATs with name containing search string. | -| `state` | string | **{dotted-circle}** No | Limit results to PATs with specified state. Valid values are `active` and `inactive`. | -| `user_id` | integer or string | **{dotted-circle}** No | Limit results to PATs owned by specified user. | +| `created_after` | datetime (ISO 8601) | No | Limit results to PATs created after specified time. | +| `created_before` | datetime (ISO 8601) | No | Limit results to PATs created before specified time. | +| `last_used_after` | datetime (ISO 8601) | No | Limit results to PATs last used after specified time. | +| `last_used_before` | datetime (ISO 8601) | No | Limit results to PATs last used before specified time. | +| `revoked` | boolean | No | Limit results to PATs with specified revoked state. Valid values are `true` and `false`. | +| `search` | string | No | Limit results to PATs with name containing search string. | +| `state` | string | No | Limit results to PATs with specified state. Valid values are `active` and `inactive`. | +| `user_id` | integer or string | No | Limit results to PATs owned by specified user. | Example request: @@ -226,6 +226,23 @@ Non-administrators can rotate their own tokens. Administrators can rotate tokens curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/personal_access_tokens/<personal_access_token_id>/rotate" ``` +Example response: + +```json +{ + "id": 42, + "name": "Rotated Token", + "revoked": false, + "created_at": "2023-08-01T15:00:00.000Z", + "scopes": ["api"], + "user_id": 1337, + "last_used_at": null, + "active": true, + "expires_at": "2023-08-15", + "token": "s3cr3t" +} +``` + ### Responses - `200: OK` if the existing token is successfully revoked and the new token successfully created. @@ -243,12 +260,13 @@ For each rotated token, the previous and now revoked token is referenced. This chain of references defines a token family. In a token family, only the latest token is active, and all other tokens in that family are revoked. -When a revoked token from a token family is used in an authentication attempt, -that attempt fails and the active token from the token family gets revoked. +When a revoked token from a token family is used in an authentication attempt +for the token rotation endpoint, that attempt fails and the active token from +the token family gets revoked. This mechanism helps to prevent compromise when a personal access token is leaked. -Automatic reuse detection is enabled for API requests. +Automatic reuse detection is enabled for token rotation API requests. ## Revoke a personal access token diff --git a/doc/api/plan_limits.md b/doc/api/plan_limits.md index 53dedca3312..789acf46205 100644 --- a/doc/api/plan_limits.md +++ b/doc/api/plan_limits.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/project_access_tokens.md b/doc/api/project_access_tokens.md index 41bfcbd209b..793eb49c767 100644 --- a/doc/api/project_access_tokens.md +++ b/doc/api/project_access_tokens.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -152,6 +152,24 @@ POST /projects/:id/access_tokens/:token_id/rotate curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens/<token_id>/rotate" ``` +Example response: + +```json +{ + "id": 42, + "name": "Rotated Token", + "revoked": false, + "created_at": "2023-08-01T15:00:00.000Z", + "scopes": ["api"], + "user_id": 1337, + "last_used_at": null, + "active": true, + "expires_at": "2023-08-15", + "access_level": 30, + "token": "s3cr3t" +} +``` + ### Responses - `200: OK` if the existing token is successfully revoked and the new token is successfully created. diff --git a/doc/api/project_aliases.md b/doc/api/project_aliases.md index 6bad94f48b3..b8ca5c2674c 100644 --- a/doc/api/project_aliases.md +++ b/doc/api/project_aliases.md @@ -48,7 +48,7 @@ GET /project_aliases/:name | Attribute | Type | Required | Description | |-----------|--------|----------|-----------------------| -| `name` | string | **{check-circle}** Yes | The name of the alias. | +| `name` | string | Yes | The name of the alias. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab" @@ -75,8 +75,8 @@ POST /project_aliases | Attribute | Type | Required | Description | |--------------|----------------|----------|----------------------------------------| -| `name` | string | **{check-circle}** Yes | The name of the alias. Must be unique. | -| `project_id` | integer or string | **{check-circle}** Yes | The ID or path of the project. | +| `name` | string | Yes | The name of the alias. Must be unique. | +| `project_id` | integer or string | Yes | The ID or path of the project. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -111,7 +111,7 @@ DELETE /project_aliases/:name | Attribute | Type | Required | Description | |-----------|--------|----------|-----------------------| -| `name` | string | **{check-circle}** Yes | The name of the alias. | +| `name` | string | Yes | The name of the alias. | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab" diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md index 5c621bf6426..d615df112cf 100644 --- a/doc/api/project_badges.md +++ b/doc/api/project_badges.md @@ -47,7 +47,7 @@ Example response: "id": 1, "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge", "kind": "project" }, @@ -56,7 +56,7 @@ Example response: "id": 2, "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge", "kind": "group" } @@ -88,7 +88,7 @@ Example response: "id": 1, "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge", "kind": "project" } @@ -111,7 +111,7 @@ POST /projects/:id/badges ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/master&image_url=https://shields.io/my/badge1&name=mybadge" \ + --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/main&image_url=https://shields.io/my/badge1&name=mybadge" \ "https://gitlab.example.com/api/v4/projects/:id/badges" ``` @@ -121,9 +121,9 @@ Example response: { "id": 1, "name": "mybadge", - "link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/master", + "link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/main", "image_url": "https://shields.io/my/badge1", - "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/master", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/main", "rendered_image_url": "https://shields.io/my/badge1", "kind": "project" } @@ -155,9 +155,9 @@ Example response: { "id": 1, "name": "mybadge", - "link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/master", + "link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/main", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/master", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-foss/commits/main", "rendered_image_url": "https://shields.io/my/badge", "kind": "project" } @@ -204,7 +204,7 @@ Example response: { "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", - "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=main", "rendered_image_url": "https://shields.io/my/badge" } ``` diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 85aa28804b0..6e61c0d56b3 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -201,7 +201,7 @@ Parameters: | `platform_kubernetes_attributes[ca_cert]` | string | no | TLS certificate. Required if API is using a self-signed TLS certificate. | | `platform_kubernetes_attributes[namespace]` | string | no | The unique namespace related to the project | | `platform_kubernetes_attributes[authorization_type]` | string | no | The cluster authorization type: `rbac`, `abac` or `unknown_authorization`. Defaults to `rbac`. | -| `environment_scope` | string | no | The associated environment to the cluster. Defaults to `*` **(PREMIUM)** | +| `environment_scope` | string | no | The associated environment to the cluster. Defaults to `*` **(PREMIUM ALL)** | Example request: diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index 113504aba8a..1a6b0ccc2c4 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -32,7 +32,7 @@ project to a web server or to any S3-compatible platform. For exports, GitLab: time and is available throughout the export process. - Administrators can modify the maximum export file size. By default, the maximum is unlimited (`0`). To change this, edit `max_export_size` using either: - - [GitLab UI](../administration/settings/account_and_limit_settings.md). + - [GitLab UI](../administration/settings/import_and_export_settings.md). - [Application settings API](settings.md#change-application-settings) - Has a fixed limit for the maximum import file size on GitLab.com. For more information, see [Account and limit settings](../user/gitlab_com/index.md#account-and-limit-settings). @@ -205,7 +205,7 @@ NOTE: The maximum import file size can be set by the Administrator. It defaults to `0` (unlimited). As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](settings.md#change-application-settings) or the [Admin Area](../administration/settings/account_and_limit_settings.md). Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. -## Import a file from a remote object storage (Beta) +## Import a file from a remote object storage **(BETA)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282503) in GitLab 13.12 in [Beta](../policy/experiment-beta-support.md#beta) [with a flag](../administration/feature_flags.md) named `import_project_from_remote_file`. Enabled by default. diff --git a/doc/api/project_relations_export.md b/doc/api/project_relations_export.md index bce3c6271af..628bb467a02 100644 --- a/doc/api/project_relations_export.md +++ b/doc/api/project_relations_export.md @@ -63,10 +63,6 @@ curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" \ The status can be one of the following: -- `0`: `started` -- `1`: `finished` -- `-1`: `failed` - - `0` - `started` - `1` - `finished` - `-1` - `failed` diff --git a/doc/api/project_templates.md b/doc/api/project_templates.md index dad09b0e479..f4fb17d8c57 100644 --- a/doc/api/project_templates.md +++ b/doc/api/project_templates.md @@ -20,7 +20,7 @@ It deprecates these endpoints, which are scheduled for removal in API version 5. In addition to templates common to the entire instance, project-specific templates are also available from this API endpoint. -Support is also available for [group-level file templates](../user/group/manage.md#group-file-templates). **(PREMIUM)** +Support is also available for [group-level file templates](../user/group/manage.md#group-file-templates). **(PREMIUM ALL)** ## Get all templates of a particular type @@ -30,8 +30,8 @@ GET /projects/:id/templates/:type | Attribute | Type | Required | Description | | ---------- | ------ | -------- | ----------- | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `type` | string | **{check-circle}** Yes | The type of the template. Accepted values are: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, or `merge_requests`. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `type` | string | Yes | The type of the template. Accepted values are: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, or `merge_requests`. | Example response (licenses): @@ -96,12 +96,12 @@ GET /projects/:id/templates/:type/:name | Attribute | Type | Required | Description | | ---------- | ------ | -------- | ----------- | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `name` | string | **{check-circle}** Yes | The key of the template, as obtained from the collection endpoint. | -| `type` | string | **{check-circle}** Yes | The type of the template. One of: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, or `merge_requests`. | -| `fullname` | string | **{dotted-circle}** No | The full name of the copyright holder to use when expanding placeholders in the template. Affects only licenses. | -| `project` | string | **{dotted-circle}** No | The project name to use when expanding placeholders in the template. Affects only licenses. | -| `source_template_project_id` | integer | **{dotted-circle}** No | The project ID where a given template is being stored. Helpful when multiple templates from different projects have the same name. If multiple templates have the same name, the match from `closest ancestor` is returned if `source_template_project_id` is not specified, | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `name` | string | Yes | The key of the template, as obtained from the collection endpoint. | +| `type` | string | Yes | The type of the template. One of: `dockerfiles`, `gitignores`, `gitlab_ci_ymls`, `licenses`, `issues`, or `merge_requests`. | +| `fullname` | string | No | The full name of the copyright holder to use when expanding placeholders in the template. Affects only licenses. | +| `project` | string | No | The project name to use when expanding placeholders in the template. Affects only licenses. | +| `source_template_project_id` | integer | No | The project ID where a given template is being stored. Helpful when multiple templates from different projects have the same name. If multiple templates have the same name, the match from `closest ancestor` is returned if `source_template_project_id` is not specified, | Example response (Dockerfile): diff --git a/doc/api/project_vulnerabilities.md b/doc/api/project_vulnerabilities.md index c38ee31ddfc..2b4d3ec50df 100644 --- a/doc/api/project_vulnerabilities.md +++ b/doc/api/project_vulnerabilities.md @@ -90,7 +90,7 @@ Example response: "path": "security-reports", "path_with_namespace": "gitlab-org/security-reports" }, - "project_default_branch": "master", + "project_default_branch": "main", "report_type": "dependency_scanning", "resolved_at": null, "resolved_by_id": null, @@ -178,7 +178,7 @@ Example response: "path": "security-reports", "path_with_namespace": "gitlab-org/security-reports" }, - "project_default_branch": "master", + "project_default_branch": "main", "report_type": "dependency_scanning", "resolved_at": null, "resolved_by_id": null, diff --git a/doc/api/projects.md b/doc/api/projects.md index 5bd2ec07647..2c00e1cbdc8 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -53,34 +53,34 @@ GET /projects | Attribute | Type | Required | Description | |--------------------------------------------|----------|------------------------|-------------| -| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | -| `id_after` | integer | **{dotted-circle}** No | Limit results to projects with IDs greater than the specified ID. | -| `id_before` | integer | **{dotted-circle}** No | Limit results to projects with IDs less than the specified ID. | -| `imported` | boolean | **{dotted-circle}** No | Limit results to projects which were imported from external systems by current user. | -| `last_activity_after` | datetime | **{dotted-circle}** No | Limit results to projects with last activity after specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | -| `last_activity_before` | datetime | **{dotted-circle}** No | Limit results to projects with last activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | -| `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | -| `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | -| `repository_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the repository checksum calculation has failed. | -| `repository_storage` | string | **{dotted-circle}** No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | -| `search_namespaces` | boolean | **{dotted-circle}** No | Include ancestor namespaces when matching search criteria. Default is `false`. | -| `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. This operation is a no-op without authentication where only simple fields are returned. | -| `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | -| `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Available only to users with at least the Reporter role. | -| `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | -| `topic_id` | integer | **{dotted-circle}** No | Limit results to projects with the assigned topic given by the topic ID. | -| `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | -| `with_programming_language` | string | **{dotted-circle}** No | Limit by projects which use the given programming language. | -| `updated_before` | datetime | **{dotted-circle}** No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | -| `updated_after` | datetime | **{dotted-circle}** No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | +| `archived` | boolean | No | Limit by archived status. | +| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | +| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | +| `imported` | boolean | No | Limit results to projects which were imported from external systems by current user. | +| `last_activity_after` | datetime | No | Limit results to projects with last activity after specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | +| `last_activity_before` | datetime | No | Limit results to projects with last activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `repository_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the repository checksum calculation has failed. | +| `repository_storage` | string | No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | +| `search_namespaces` | boolean | No | Include ancestor namespaces when matching search criteria. Default is `false`. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. This operation is a no-op without authentication where only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `topic` | string | No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | +| `topic_id` | integer | No | Limit results to projects with the assigned topic given by the topic ID. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `wiki_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the wiki checksum calculation has failed. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `with_programming_language` | string | No | Limit by projects which use the given programming language. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | This endpoint supports [keyset pagination](rest/index.md#keyset-based-pagination) for selected `order_by` options. @@ -162,7 +162,7 @@ When the user is authenticated and `simple` is not set this returns something li "ssh_url_to_repo": "git@gitlab.example.com:diaspora/diaspora-client.git", "http_url_to_repo": "https://gitlab.example.com/diaspora/diaspora-client.git", "web_url": "https://gitlab.example.com/diaspora/diaspora-client", - "readme_url": "https://gitlab.example.com/diaspora/diaspora-client/blob/master/README.md", + "readme_url": "https://gitlab.example.com/diaspora/diaspora-client/blob/main/README.md", "avatar_url": "https://gitlab.example.com/uploads/project/avatar/4/uploads/avatar.png", "forks_count": 0, "star_count": 0, @@ -322,26 +322,26 @@ GET /users/:user_id/projects | Attribute | Type | Required | Description | |-------------------------------|---------|------------------------|-------------| -| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | -| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | -| `id_after` | integer | **{dotted-circle}** No | Limit results to projects with IDs greater than the specified ID. | -| `id_before` | integer | **{dotted-circle}** No | Limit results to projects with IDs less than the specified ID. | -| `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | -| `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | -| `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | -| `with_programming_language` | string | **{dotted-circle}** No | Limit by projects which use the given programming language. | -| `updated_before` | datetime | **{dotted-circle}** No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | **{dotted-circle}** No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `user_id` | string | Yes | The ID or username of the user. | +| `archived` | boolean | No | Limit by archived status. | +| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | +| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `with_programming_language` | string | No | Limit by projects which use the given programming language. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | ```json [ @@ -349,12 +349,12 @@ GET /users/:user_id/projects "id": 4, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", - "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-client/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora client" @@ -457,12 +457,12 @@ GET /users/:user_id/projects "id": 6, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", - "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", + "readme_url": "http://example.com/brightbox/puppet/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "puppet" @@ -596,10 +596,10 @@ GET /users/:user_id/contributed_projects | Attribute | Type | Required | Description | |-------------------------------|---------|------------------------|-------------| -| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `user_id` | string | Yes | The ID or username of the user. | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/5/contributed_projects" @@ -613,12 +613,12 @@ Example response: "id": 4, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", - "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-client/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora client" @@ -709,12 +709,12 @@ Example response: "id": 6, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", - "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", + "readme_url": "http://example.com/brightbox/puppet/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "puppet" @@ -841,23 +841,23 @@ GET /users/:user_id/starred_projects | Attribute | Type | Required | Description | |-------------------------------|---------|------------------------|-------------| -| `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | -| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | -| `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | -| `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | -| `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | -| `updated_before` | datetime | **{dotted-circle}** No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | **{dotted-circle}** No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `user_id` | string | Yes | The ID or username of the user. | +| `archived` | boolean | No | Limit by archived status. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/5/starred_projects" @@ -871,12 +871,12 @@ Example response: "id": 4, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", - "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-client/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora client" @@ -967,12 +967,12 @@ Example response: "id": 6, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", - "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", + "readme_url": "http://example.com/brightbox/puppet/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "puppet" @@ -1099,22 +1099,22 @@ GET /projects/:id | Attribute | Type | Required | Description | |--------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `license` | boolean | **{dotted-circle}** No | Include project license data. | -| `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Available only to users with at least the Reporter role. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `license` | boolean | No | Include project license data. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | ```json { "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -1182,7 +1182,7 @@ GET /projects/:id }, "archived": false, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", - "license_url": "http://example.com/diaspora/diaspora-client/blob/master/LICENSE", + "license_url": "http://example.com/diaspora/diaspora-client/blob/main/LICENSE", "license": { "key": "lgpl-3.0", "name": "GNU Lesser General Public License v3.0", @@ -1305,14 +1305,14 @@ target the upstream project by default. "path":"gitlab-foss", "path_with_namespace":"gitlab-org/gitlab-foss", "created_at":"2013-09-26T06:02:36.000Z", - "default_branch":"master", + "default_branch":"main", "tag_list":[], //deprecated, use `topics` instead "topics":[], "ssh_url_to_repo":"git@gitlab.com:gitlab-org/gitlab-foss.git", "http_url_to_repo":"https://gitlab.com/gitlab-org/gitlab-foss.git", "web_url":"https://gitlab.com/gitlab-org/gitlab-foss", "avatar_url":"https://gitlab.com/uploads/-/system/project/avatar/13083/logo-extra-whitespace.png", - "license_url": "https://gitlab.com/gitlab-org/gitlab/-/blob/master/LICENSE", + "license_url": "https://gitlab.com/gitlab-org/gitlab/-/blob/main/LICENSE", "license": { "key": "mit", "name": "MIT License", @@ -1365,9 +1365,9 @@ GET /projects/:id/users | Attribute | Type | Required | Description | |--------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | Search for specific users. | -| `skip_users` | integer array | **{dotted-circle}** No | Filter out users with the specified IDs. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific users. | +| `skip_users` | integer array | No | Filter out users with the specified IDs. | ```json [ @@ -1400,12 +1400,12 @@ GET /projects/:id/groups | Attribute | Type | Required | Description | |-----------------------------|-------------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | Search for specific groups. | -| `shared_min_access_level` | integer | **{dotted-circle}** No | Limit to shared groups with at least this [role (`access_level`)](members.md#roles). | -| `shared_visible_only` | boolean | **{dotted-circle}** No | Limit to shared groups user has access to. | -| `skip_groups` | array of integers | **{dotted-circle}** No | Skip the group IDs passed. | -| `with_shared` | boolean | **{dotted-circle}** No | Include projects shared with this group. Default is `false`. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific groups. | +| `shared_min_access_level` | integer | No | Limit to shared groups with at least this [role (`access_level`)](members.md#roles). | +| `shared_visible_only` | boolean | No | Limit to shared groups user has access to. | +| `skip_groups` | array of integers | No | Skip the group IDs passed. | +| `with_shared` | boolean | No | Include projects shared with this group. Default is `false`. | ```json [ @@ -1438,8 +1438,8 @@ GET /projects/:id/share_locations | Attribute | Type | Required | Description | |-----------------------------|-------------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | Search for specific groups. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific groups. | ```json [ @@ -1492,77 +1492,77 @@ curl --request POST --header "PRIVATE-TOKEN: <your-token>" \ | Attribute | Type | Required | Description | |-------------------------------------------------------------|---------|------------------------|-------------| -| `name` | string | **{check-circle}** Yes (if `path` isn't provided) | The name of the new project. Equals path if not provided. | -| `path` | string | **{check-circle}** Yes (if `name` isn't provided) | Repository name for new project. Generated based on name if not provided (generated as lowercase with dashes). Starting with GitLab 14.9, path must not start or end with a special character and must not contain consecutive special characters. | -| `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE)** | boolean | **{dotted-circle}** No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | -| `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | -| `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | -| `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | **{dotted-circle}** No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | **{dotted-circle}** No | Image file for avatar of the project. | -| `build_git_strategy` | string | **{dotted-circle}** No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). See the [Container Registry](../user/packages/container_registry/reduce_container_registry_storage.md#use-the-cleanup-policy-api) documentation for more information on `cadence`, `keep_n` and `older_than` values. | -| `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | -| `description` | string | **{dotted-circle}** No | Short project description. | -| `emails_disabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | **{dotted-circle}** No | Enable email notifications. | -| `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | -| `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | -| `group_with_project_templates_id` **(PREMIUM)** | integer | **{dotted-circle}** No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | -| `import_url` | string | **{dotted-circle}** No | URL to import repository from. When the URL value isn't empty, you must not set `initialize_with_readme` to `true`. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | -| `initialize_with_readme` | boolean | **{dotted-circle}** No | Whether to create a Git repository with just a `README.md` file. Default is `false`. When this boolean is true, you must not pass `import_url` or other attributes of this endpoint which specify alternative contents for the repository. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | -| `issues_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | -| `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | -| `merge_pipelines_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge pipelines. | -| `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `merge_trains_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge trains. | -| `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | -| `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | -| `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) in the project settings. | -| `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | -| `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | -| `releases_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrator only)_ | -| `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | -| `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | -| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | -| `template_project_id` **(PREMIUM)** | integer | **{dotted-circle}** No | When used with `use_custom_template`, project ID of a custom project template. Using a project ID is preferable to using `template_name` since `template_name` may be ambiguous. | -| `topics` | array | **{dotted-circle}** No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | -| `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| `name` | string | Yes (if `path` isn't provided) | The name of the new project. Equals path if not provided. | +| `path` | string | Yes (if `name` isn't provided) | Repository name for new project. Generated based on name if not provided (generated as lowercase with dashes). Starting with GitLab 14.9, path must not start or end with a special character and must not contain consecutive special characters. | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). See the [Container Registry](../user/packages/container_registry/reduce_container_registry_storage.md#use-the-cleanup-policy-api) documentation for more information on `cadence`, `keep_n` and `older_than` values. | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| +| `emails_enabled` | boolean | No | Enable email notifications. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | +| `import_url` | string | No | URL to import repository from. When the URL value isn't empty, you must not set `initialize_with_readme` to `true`. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | +| `initialize_with_readme` | boolean | No | Whether to create a Git repository with just a `README.md` file. Default is `false`. When this boolean is true, you must not pass `import_url` or other attributes of this endpoint which specify alternative contents for the repository. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | No | Enable or disable merge pipelines. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) in the project settings. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrator only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default award emojis. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | +| `template_project_id` **(PREMIUM ALL)** | integer | No | When used with `use_custom_template`, project ID of a custom project template. Using a project ID is preferable to using `template_name` since `template_name` may be ambiguous. | +| `topics` | array | No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Create project for user @@ -1580,79 +1580,79 @@ POST /projects/user/:user_id | Attribute | Type | Required | Description | |-------------------------------------------------------------|---------|------------------------|-------------| -| `user_id` | integer | **{check-circle}** Yes | The user ID of the project owner. | -| `name` | string | **{check-circle}** Yes | The name of the new project. | -| `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE)** | boolean | **{dotted-circle}** No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | -| `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | -| `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | -| `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | **{dotted-circle}** No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | **{dotted-circle}** No | Image file for avatar of the project. | -| `build_git_strategy` | string | **{dotted-circle}** No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | -| `description` | string | **{dotted-circle}** No | Short project description. | -| `emails_disabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | **{dotted-circle}** No | Enable email notifications. | -| `enforce_auth_checks_on_uploads` | boolean | **{dotted-circle}** No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | -| `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | -| `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | -| `group_with_project_templates_id` **(PREMIUM)** | integer | **{dotted-circle}** No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | -| `import_url` | string | **{dotted-circle}** No | URL to import repository from. | -| `initialize_with_readme` | boolean | **{dotted-circle}** No | `false` by default. | -| `issues_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | -| `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | -| `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | -| `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | -| `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | -| `namespace_id` | integer | **{dotted-circle}** No | Namespace for the new project (defaults to the current user's namespace). | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful jobs. | -| `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | -| `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `path` | string | **{dotted-circle}** No | Custom repository name for new project. By default generated based on name. | -| `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project-members. | -| `releases_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | -| `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | -| `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | -| `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `issue_branch_template` | string | **{dotted-circle}** No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | -| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | -| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | -| `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | -| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | -| `topics` | array | **{dotted-circle}** No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | -| `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| `user_id` | integer | Yes | The user ID of the project owner. | +| `name` | string | Yes | The name of the new project. | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| +| `emails_enabled` | boolean | No | Enable email notifications. | +| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | +| `import_url` | string | No | URL to import repository from. | +| `initialize_with_readme` | boolean | No | `false` by default. | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `path` | string | No | Custom repository name for new project. By default generated based on name. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project-members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default award emojis. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | +| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `suggestion_commit_message` | string | No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | +| `topics` | array | No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Edit project @@ -1681,92 +1681,92 @@ Supported attributes: | Attribute | Type | Required | Description | |-------------------------------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | -| `allow_pipeline_trigger_approve_deployment` **(PREMIUM)** | boolean | **{dotted-circle}** No | Set whether or not a pipeline triggerer is allowed to approve deployments. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE)** | boolean | **{dotted-circle}** No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false.<br/><br/>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. The feature flag was enabled by default in GitLab 15.9. | -| `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | -| `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual`, or `timed_incremental`). | -| `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | **{dotted-circle}** No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | **{dotted-circle}** No | Image file for avatar of the project. | -| `build_git_strategy` | string | **{dotted-circle}** No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `ci_default_git_depth` | integer | **{dotted-circle}** No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | -| `ci_forward_deployment_enabled` | boolean | **{dotted-circle}** No | Enable or disable [prevent outdated deployment jobs](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | -| `ci_forward_deployment_rollback_allowed` | boolean | **{dotted-circle}** No | Enable or disable [allow job retries for rollback deployments](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | -| `ci_allow_fork_pipelines_to_run_in_parent_project` | boolean | **{dotted-circle}** No | Enable or disable [running pipelines in the parent project for merge requests from forks](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325189) in GitLab 15.3.)_ | -| `ci_separated_caches` | boolean | **{dotted-circle}** No | Set whether or not caches should be [separated](../ci/caching/index.md#cache-key-names) by branch protection status. | -| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | -| `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. | -| `description` | string | **{dotted-circle}** No | Short project description. | -| `emails_disabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | **{dotted-circle}** No | Enable email notifications. | -| `enforce_auth_checks_on_uploads` | boolean | **{dotted-circle}** No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | -| `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | -| `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | -| `import_url` | string | **{dotted-circle}** No | URL the repository was imported from. | -| `issues_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `issues_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `keep_latest_artifact` | boolean | **{dotted-circle}** No | Disable or enable the ability to keep the latest artifact for this project. | -| `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | -| `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | -| `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | -| `merge_pipelines_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge pipelines. | -| `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `merge_requests_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for merge requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `merge_trains_enabled` | boolean | **{dotted-circle}** No | Enable or disable merge trains. | -| `mirror_overwrites_diverged_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirror overwrites diverged branches. | -| `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | -| `mirror_user_id` **(PREMIUM)** | integer | **{dotted-circle}** No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | -| `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | -| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | -| `name` | string | **{dotted-circle}** No | The name of the project. | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | **{dotted-circle}** No | Set whether merge requests can only be merged with successful jobs. | -| `only_mirror_protected_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Only mirror protected branches. | -| `packages_enabled` | boolean | **{dotted-circle}** No | Enable or disable packages repository feature. | -| `pages_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled`, or `public`. | -| `path` | string | **{dotted-circle}** No | Custom repository name for the project. By default generated based on name. | -| `printing_merge_request_link_enabled` | boolean | **{dotted-circle}** No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | -| `releases_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | -| `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | -| `requirements_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, `enabled` or `public` | -| `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `restrict_user_defined_variables` | boolean | **{dotted-circle}** No | Allow only users with the Maintainer role to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | -| `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `service_desk_enabled` | boolean | **{dotted-circle}** No | Enable or disable Service Desk feature. | -| `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | -| `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `issue_branch_template` | string | **{dotted-circle}** No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | -| `squash_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | -| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | -| `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request suggestions. | -| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `topics` | array | **{dotted-circle}** No | The list of topics for the project. This replaces any existing topics that are already added to the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `allow_pipeline_trigger_approve_deployment` **(PREMIUM ALL)** | boolean | No | Set whether or not a pipeline triggerer is allowed to approve deployments. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false.<br/><br/>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. The feature flag was enabled by default in GitLab 15.9. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual`, or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `ci_default_git_depth` | integer | No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | +| `ci_forward_deployment_enabled` | boolean | No | Enable or disable [prevent outdated deployment jobs](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | +| `ci_forward_deployment_rollback_allowed` | boolean | No | Enable or disable [allow job retries for rollback deployments](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | +| `ci_allow_fork_pipelines_to_run_in_parent_project` | boolean | No | Enable or disable [running pipelines in the parent project for merge requests from forks](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325189) in GitLab 15.3.)_ | +| `ci_separated_caches` | boolean | No | Set whether or not caches should be [separated](../ci/caching/index.md#cache-key-names) by branch protection status. | +| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| +| `emails_enabled` | boolean | No | Enable email notifications. | +| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `import_url` | string | No | URL the repository was imported from. | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `issues_template` **(PREMIUM ALL)** | string | No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `keep_latest_artifact` | boolean | No | Disable or enable the ability to keep the latest artifact for this project. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | No | Enable or disable merge pipelines. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_requests_template` **(PREMIUM ALL)** | string | No | Default description for merge requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | +| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | +| `mirror_overwrites_diverged_branches` **(PREMIUM ALL)** | boolean | No | Pull mirror overwrites diverged branches. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror_user_id` **(PREMIUM ALL)** | integer | No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | +| `name` | string | No | The name of the project. | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | +| `only_mirror_protected_branches` **(PREMIUM ALL)** | boolean | No | Only mirror protected branches. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `path` | string | No | Custom repository name for the project. By default generated based on name. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `restrict_user_defined_variables` | boolean | No | Allow only users with the Maintainer role to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `service_desk_enabled` | boolean | No | Enable or disable Service Desk feature. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default award emojis. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | +| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `suggestion_commit_message` | string | No | The commit message used to apply merge request suggestions. | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `topics` | array | No | The list of topics for the project. This replaces any existing topics that are already added to the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Fork project @@ -1782,15 +1782,15 @@ POST /projects/:id/fork | Attribute | Type | Required | Description | |------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `description` | string | **{dotted-circle}** No | The description assigned to the resultant project after forking. | -| `mr_default_target_self` | boolean | **{dotted-circle}** No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | -| `name` | string | **{dotted-circle}** No | The name assigned to the resultant project after forking. | -| `namespace_id` | integer | **{dotted-circle}** No | The ID of the namespace that the project is forked to. | -| `namespace_path` | string | **{dotted-circle}** No | The path of the namespace that the project is forked to. | -| `namespace` | integer or string | **{dotted-circle}** No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | -| `path` | string | **{dotted-circle}** No | The path assigned to the resultant project after forking. | -| `visibility` | string | **{dotted-circle}** No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `description` | string | No | The description assigned to the resultant project after forking. | +| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | +| `name` | string | No | The name assigned to the resultant project after forking. | +| `namespace_id` | integer | No | The ID of the namespace that the project is forked to. | +| `namespace_path` | string | No | The path of the namespace that the project is forked to. | +| `namespace` | integer or string | No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | +| `path` | string | No | The path assigned to the resultant project after forking. | +| `visibility` | string | No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | ## List forks of a project @@ -1805,23 +1805,23 @@ GET /projects/:id/forks | Attribute | Type | Required | Description | |-------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | -| `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | -| `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | -| `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | -| `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | -| `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | -| `updated_before` | datetime | **{dotted-circle}** No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | **{dotted-circle}** No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `archived` | boolean | No | Limit by archived status. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/forks" @@ -1835,12 +1835,12 @@ Example responses: "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "internal", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -1924,7 +1924,7 @@ POST /projects/:id/star | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/star" @@ -1937,12 +1937,12 @@ Example response: "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "internal", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -1980,7 +1980,7 @@ Example response: "import_status": "none", "archived": true, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", - "license_url": "http://example.com/diaspora/diaspora-client/blob/master/LICENSE", + "license_url": "http://example.com/diaspora/diaspora-client/blob/main/LICENSE", "license": { "key": "lgpl-3.0", "name": "GNU Lesser General Public License v3.0", @@ -2032,7 +2032,7 @@ POST /projects/:id/unstar | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/unstar" @@ -2045,12 +2045,12 @@ Example response: "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "internal", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -2088,7 +2088,7 @@ Example response: "import_status": "none", "archived": true, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", - "license_url": "http://example.com/diaspora/diaspora-client/blob/master/LICENSE", + "license_url": "http://example.com/diaspora/diaspora-client/blob/main/LICENSE", "license": { "key": "lgpl-3.0", "name": "GNU Lesser General Public License v3.0", @@ -2138,8 +2138,8 @@ GET /projects/:id/starrers | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | Search for specific users. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific users. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/starrers" @@ -2184,7 +2184,7 @@ GET /projects/:id/languages | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/languages" @@ -2215,7 +2215,7 @@ POST /projects/:id/archive | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/archive" @@ -2228,12 +2228,12 @@ Example response: "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -2287,7 +2287,7 @@ Example response: }, "archived": true, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", - "license_url": "http://example.com/diaspora/diaspora-client/blob/master/LICENSE", + "license_url": "http://example.com/diaspora/diaspora-client/blob/main/LICENSE", "license": { "key": "lgpl-3.0", "name": "GNU Lesser General Public License v3.0", @@ -2347,7 +2347,7 @@ POST /projects/:id/unarchive | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/unarchive" @@ -2360,12 +2360,12 @@ Example response: "id": 3, "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit.", "description_html": "<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>", - "default_branch": "master", + "default_branch": "main", "visibility": "private", "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", - "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", + "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/main/README.md", "tag_list": [ //deprecated, use `topics` instead "example", "disapora project" @@ -2419,7 +2419,7 @@ Example response: }, "archived": false, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", - "license_url": "http://example.com/diaspora/diaspora-client/blob/master/LICENSE", + "license_url": "http://example.com/diaspora/diaspora-client/blob/main/LICENSE", "license": { "key": "lgpl-3.0", "name": "GNU Lesser General Public License v3.0", @@ -2494,9 +2494,9 @@ DELETE /projects/:id | Attribute | Type | Required | Description | |------------------------------------|-------------------|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `permanently_remove` **(PREMIUM)** | boolean/string | no | Immediately deletes a project if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11 | -| `full_path` **(PREMIUM)** | string | no | Full path of project to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11. To find the project path, use `path_with_namespace` from [get single project](projects.md#get-single-project) | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `permanently_remove` **(PREMIUM ALL)** | boolean/string | no | Immediately deletes a project if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11 | +| `full_path` **(PREMIUM ALL)** | string | no | Full path of project to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11. To find the project path, use `path_with_namespace` from [get single project](projects.md#get-single-project) | ## Restore project marked for deletion **(PREMIUM ALL)** @@ -2510,7 +2510,7 @@ POST /projects/:id/restore | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Upload a file @@ -2527,8 +2527,8 @@ POST /projects/:id/uploads | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `file` | string | **{check-circle}** Yes | The file to be uploaded. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `file` | string | Yes | The file to be uploaded. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | To upload a file from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -2565,8 +2565,8 @@ PUT /projects/:id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `avatar` | string | **{check-circle}** Yes | The file to be uploaded. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `avatar` | string | Yes | The file to be uploaded. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | To upload an avatar from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -2611,10 +2611,10 @@ POST /projects/:id/share | Attribute | Type | Required | Description | |----------------|----------------|------------------------|-------------| -| `group_access` | integer | **{check-circle}** Yes | The [role (`access_level`)](members.md#roles) to grant the group. | -| `group_id` | integer | **{check-circle}** Yes | The ID of the group to share with. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `expires_at` | string | **{dotted-circle}** No | Share expiration date in ISO 8601 format: 2016-09-26 | +| `group_access` | integer | Yes | The [role (`access_level`)](members.md#roles) to grant the group. | +| `group_id` | integer | Yes | The ID of the group to share with. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `expires_at` | string | No | Share expiration date in ISO 8601 format: 2016-09-26 | ## Delete a shared project link within a group @@ -2626,8 +2626,8 @@ DELETE /projects/:id/share/:group_id | Attribute | Type | Required | Description | |------------|----------------|------------------------|-------------| -| `group_id` | integer | **{check-circle}** Yes | The ID of the group. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `group_id` | integer | Yes | The ID of the group. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/share/17" @@ -2648,8 +2648,8 @@ POST /projects/:id/import_project_members/:project_id | Attribute | Type | Required | Description | |--------------|-------------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the target project to receive the members. | -| `project_id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the source project to import the members from. | +| `id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the target project to receive the members. | +| `project_id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the source project to import the members from. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/import_project_members/32" @@ -2703,7 +2703,7 @@ GET /projects/:id/hooks | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ### Get project hook @@ -2715,8 +2715,8 @@ GET /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|---------------------------| -| `hook_id` | integer | **{check-circle}** Yes | The ID of a project hook. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `hook_id` | integer | Yes | The ID of a project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```json { @@ -2755,23 +2755,23 @@ POST /projects/:id/hooks | Attribute | Type | Required | Description | |------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `url` | string | **{check-circle}** Yes | The hook URL. | -| `confidential_issues_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential issues events. | -| `confidential_note_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential note events. | -| `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | -| `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | -| `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | -| `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | -| `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | -| `note_events` | boolean | **{dotted-circle}** No | Trigger hook on note events. | -| `pipeline_events` | boolean | **{dotted-circle}** No | Trigger hook on pipeline events. | -| `push_events_branch_filter` | string | **{dotted-circle}** No | Trigger hook on push events for matching branches only. | -| `push_events` | boolean | **{dotted-circle}** No | Trigger hook on push events. | -| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | -| `tag_push_events` | boolean | **{dotted-circle}** No | Trigger hook on tag push events. | -| `token` | string | **{dotted-circle}** No | Secret token to validate received payloads; the token isn't returned in the response. | -| `wiki_page_events` | boolean | **{dotted-circle}** No | Trigger hook on wiki events. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `url` | string | Yes | The hook URL. | +| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | +| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | +| `deployment_events` | boolean | No | Trigger hook on deployment events. | +| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | +| `issues_events` | boolean | No | Trigger hook on issues events. | +| `job_events` | boolean | No | Trigger hook on job events. | +| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | +| `note_events` | boolean | No | Trigger hook on note events. | +| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | +| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | +| `push_events` | boolean | No | Trigger hook on push events. | +| `releases_events` | boolean | No | Trigger hook on release events. | +| `tag_push_events` | boolean | No | Trigger hook on tag push events. | +| `token` | string | No | Secret token to validate received payloads; the token isn't returned in the response. | +| `wiki_page_events` | boolean | No | Trigger hook on wiki events. | ### Edit project hook @@ -2783,24 +2783,24 @@ PUT /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |------------------------------|----------------|------------------------|-------------| -| `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `url` | string | **{check-circle}** Yes | The hook URL. | -| `confidential_issues_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential issues events. | -| `confidential_note_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential note events. | -| `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | -| `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | -| `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | -| `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | -| `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | -| `note_events` | boolean | **{dotted-circle}** No | Trigger hook on note events. | -| `pipeline_events` | boolean | **{dotted-circle}** No | Trigger hook on pipeline events. | -| `push_events_branch_filter` | string | **{dotted-circle}** No | Trigger hook on push events for matching branches only. | -| `push_events` | boolean | **{dotted-circle}** No | Trigger hook on push events. | -| `releases_events` | boolean | **{dotted-circle}** No | Trigger hook on release events. | -| `tag_push_events` | boolean | **{dotted-circle}** No | Trigger hook on tag push events. | -| `token` | string | **{dotted-circle}** No | Secret token to validate received payloads. Not returned in the response. When you change the webhook URL, the secret token is reset and not retained. | -| `wiki_page_events` | boolean | **{dotted-circle}** No | Trigger hook on wiki page events. | +| `hook_id` | integer | Yes | The ID of the project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `url` | string | Yes | The hook URL. | +| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | +| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | +| `deployment_events` | boolean | No | Trigger hook on deployment events. | +| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | +| `issues_events` | boolean | No | Trigger hook on issues events. | +| `job_events` | boolean | No | Trigger hook on job events. | +| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | +| `note_events` | boolean | No | Trigger hook on note events. | +| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | +| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | +| `push_events` | boolean | No | Trigger hook on push events. | +| `releases_events` | boolean | No | Trigger hook on release events. | +| `tag_push_events` | boolean | No | Trigger hook on tag push events. | +| `token` | string | No | Secret token to validate received payloads. Not returned in the response. When you change the webhook URL, the secret token is reset and not retained. | +| `wiki_page_events` | boolean | No | Trigger hook on wiki page events. | ### Delete project hook @@ -2813,8 +2813,8 @@ DELETE /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `hook_id` | integer | Yes | The ID of the project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Note the JSON response differs if the hook is available or not. If the project hook is available before it's returned in the JSON response or an empty response @@ -2833,8 +2833,8 @@ POST /projects/:id/fork/:forked_from_id | Attribute | Type | Required | Description | |------------------|----------------|------------------------|-------------| -| `forked_from_id` | ID | **{check-circle}** Yes | The ID of the project that was forked from. | -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `forked_from_id` | ID | Yes | The ID of the project that was forked from. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ### Delete an existing forked from relationship @@ -2844,7 +2844,7 @@ DELETE /projects/:id/fork | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Search for projects by name @@ -2858,9 +2858,9 @@ GET /projects | Attribute | Type | Required | Description | |------------|--------|------------------------|-------------| -| `search` | string | **{check-circle}** Yes | A string contained in the project name. | -| `order_by` | string | **{dotted-circle}** No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | -| `sort` | string | **{dotted-circle}** No | Return requests sorted in `asc` or `desc` order. | +| `search` | string | Yes | A string contained in the project name. | +| `order_by` | string | No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | +| `sort` | string | No | Return requests sorted in `asc` or `desc` order. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?search=test" @@ -2874,8 +2874,8 @@ POST /projects/:id/housekeeping | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `task` | string | **{dotted-circle}** No | `prune` to trigger manual prune of unreachable objects or `eager` to trigger eager housekeeping. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `task` | string | No | `prune` to trigger manual prune of unreachable objects or `eager` to trigger eager housekeeping. | ## Push rules **(PREMIUM ALL)** @@ -2890,7 +2890,7 @@ GET /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | ```json { @@ -2921,18 +2921,18 @@ POST /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | -| `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository if the committer email is one of their own verified emails. | -| `commit_message_negative_regex` | string | **{dotted-circle}** No | No commit message is allowed to match this, for example `ssh\:\/\/`. | -| `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | -| `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | -| `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | -| `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | -| `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | -| `reject_unsigned_commits` | boolean | **{dotted-circle}** No | Reject commit when it's not signed through GPG. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | +| `branch_name_regex` | string | No | All branch names must match this, for example `(feature|hotfix)\/*`. | +| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | +| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | +| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | +| `deny_delete_tag` | boolean | No | Deny deleting a tag. | +| `file_name_regex` | string | No | All committed filenames must **not** match this, for example `(jar|exe)$`. | +| `max_file_size` | integer | No | Maximum file size (MB). | +| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | +| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | +| `reject_unsigned_commits` | boolean | No | Reject commit when it's not signed through GPG. | ### Edit project push rule @@ -2944,18 +2944,18 @@ PUT /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | -| `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository if the committer email is one of their own verified emails. | -| `commit_message_negative_regex` | string | **{dotted-circle}** No | No commit message is allowed to match this, for example `ssh\:\/\/`. | -| `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | -| `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | -| `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | -| `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | -| `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | -| `reject_unsigned_commits` | boolean | **{dotted-circle}** No | Reject commits when they are not GPG signed. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | +| `branch_name_regex` | string | No | All branch names must match this, for example `(feature|hotfix)\/*`. | +| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | +| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | +| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | +| `deny_delete_tag` | boolean | No | Deny deleting a tag. | +| `file_name_regex` | string | No | All committed filenames must **not** match this, for example `(jar|exe)$`. | +| `max_file_size` | integer | No | Maximum file size (MB). | +| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | +| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | +| `reject_unsigned_commits` | boolean | No | Reject commits when they are not GPG signed. | ### Delete project push rule @@ -2970,7 +2970,7 @@ DELETE /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Get groups to which a user can transfer a project @@ -2984,8 +2984,8 @@ GET /projects/:id/transfer_locations | Attribute | Type | Required | Description | |-------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | **{dotted-circle}** No | The group names to search for. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | The group names to search for. | Example request: @@ -3029,8 +3029,8 @@ PUT /projects/:id/transfer | Attribute | Type | Required | Description | |-------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `namespace` | integer or string | **{check-circle}** Yes | The ID or path of the namespace to transfer to project to. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `namespace` | integer or string | Yes | The ID or path of the namespace to transfer to project to. | Example request: @@ -3051,13 +3051,13 @@ Example response: "path_with_namespace": "cute-cats/hello-world", "created_at": "2020-10-15T16:25:22.415Z", "updated_at": "2020-10-15T16:25:22.415Z", - "default_branch": "master", + "default_branch": "main", "tag_list": [], //deprecated, use `topics` instead "topics": [], "ssh_url_to_repo": "git@gitlab.example.com:cute-cats/hello-world.git", "http_url_to_repo": "https://gitlab.example.com/cute-cats/hello-world.git", "web_url": "https://gitlab.example.com/cute-cats/hello-world", - "readme_url": "https://gitlab.example.com/cute-cats/hello-world/-/blob/master/README.md", + "readme_url": "https://gitlab.example.com/cute-cats/hello-world/-/blob/main/README.md", "avatar_url": null, "forks_count": 0, "star_count": 0, @@ -3180,7 +3180,7 @@ Supported attributes: | Attribute | Type | Required | Description | |:----------|:------|:------------|:------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: @@ -3220,11 +3220,41 @@ with the API scope enabled. | Attribute | Type | Required | Description | |--------------|---------|------------------------|-------------| -| `import_url` | string | **{check-circle}** Yes | URL of remote repository being mirrored (with `user:token` if needed). | -| `mirror` | boolean | **{check-circle}** Yes | Enables pull mirroring on project when set to `true`. | -| `mirror_trigger_builds`| boolean | **{dotted-circle}** No | Trigger pipelines for mirror updates when set to `true`. | -| `only_mirror_protected_branches`| boolean | **{dotted-circle}** No | Limits mirroring to only protected branches when set to `true`. | -| `mirror_branch_regex` | String | **{dotted-circle}** No | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_mirror_protected_branches` to be disabled. | +| `import_url` | string | Yes | URL of remote repository being mirrored (with `user:token` if needed). | +| `mirror` | boolean | Yes | Enables pull mirroring on project when set to `true`. | +| `mirror_trigger_builds`| boolean | No | Trigger pipelines for mirror updates when set to `true`. | +| `only_mirror_protected_branches`| boolean | No | Limits mirroring to only protected branches when set to `true`. | +| `mirror_branch_regex` | String | No | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_mirror_protected_branches` to be disabled. | + +Example creating a project with pull mirroring: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{ + "name": "new_project", + "namespace_id": "1", + "mirror": true, + "import_url": "https://username:token@gitlab.example.com/group/project.git" + }' \ + --url "https://gitlab.example.com/api/v4/projects/" +``` + +Example adding pull mirroring: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/:id" \ + --data "mirror=true&import_url=https://username:token@gitlab.example.com/group/project.git" +``` + +Example removing pull mirroring: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/:id" \ + --data "mirror=false" +``` ## Start the pull mirroring process for a Project **(PREMIUM ALL)** @@ -3236,7 +3266,7 @@ POST /projects/:id/mirror/pull | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/mirror/pull" @@ -3263,8 +3293,8 @@ GET /projects/:id/snapshot | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `wiki` | boolean | **{dotted-circle}** No | Whether to download the wiki, rather than project, repository. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `wiki` | boolean | No | Whether to download the wiki, rather than project, repository. | ## Get the path to repository storage @@ -3281,7 +3311,7 @@ GET /projects/:id/storage | Attribute | Type | Required | Description | |--------------|----------------|------------------------|-------------| -| `id` | integer or string | **{check-circle}** Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `id` | integer or string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```json [ diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index 6bccc777e04..93a1280f7df 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -44,7 +44,7 @@ Example response: [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -102,7 +102,7 @@ Example response: [ { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -151,7 +151,7 @@ GET /projects/:id/protected_branches/:name | `name` | string | yes | The name of the branch or wildcard | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches/master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches/main" ``` Example response: @@ -159,7 +159,7 @@ Example response: ```json { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -187,7 +187,7 @@ Example response: ```json { "id": 1, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -229,10 +229,10 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla | `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | `name` | string | yes | The name of the branch or wildcard. | `allow_force_push` | boolean | no | When enabled, members who can push to this branch can also force push. (default: `false`) -| `allowed_to_merge` **(PREMIUM)** | array | no | Array of merge access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. -| `allowed_to_push` **(PREMIUM)** | array | no | Array of push access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. -| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of unprotect access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. The access level `No access` is not available for this field. | -| `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/codeowners/index.md). (defaults: false) +| `allowed_to_merge` **(PREMIUM ALL)** | array | no | Array of merge access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_push` **(PREMIUM ALL)** | array | no | Array of push access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_unprotect` **(PREMIUM ALL)** | array | no | Array of unprotect access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. The access level `No access` is not available for this field. | +| `code_owner_approval_required` **(PREMIUM ALL)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/codeowners/index.md). (defaults: false) | `merge_access_level` | integer | no | Access levels allowed to merge. (defaults: `40`, Maintainer role) | `push_access_level` | integer | no | Access levels allowed to push. (defaults: `40`, Maintainer role) | `unprotect_access_level` | integer | no | Access levels allowed to unprotect. (defaults: `40`, Maintainer role) @@ -368,7 +368,7 @@ curl --request POST \ --header "PRIVATE-TOKEN: <your_access_token>" \ --header "Content-Type: application/json" \ --data '{ - "name": "master", + "name": "main", "allowed_to_push": [{"access_level": 30}], "allowed_to_merge": [{ "access_level": 30 @@ -384,7 +384,7 @@ Example response: ```json { "id": 5, - "name": "master", + "name": "main", "push_access_levels": [ { "id": 1, @@ -460,10 +460,10 @@ curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitl | `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | `name` | string | yes | The name of the branch or wildcard. | `allow_force_push` | boolean | no | When enabled, members who can push to this branch can also force push. -| `allowed_to_merge` **(PREMIUM)** | array | no | Array of merge access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. -| `allowed_to_push` **(PREMIUM)** | array | no | Array of push access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. -| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of unprotect access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, `{access_level: integer}`, or `{id: integer, _destroy: true}` to destroy an existing access level. The access level `No access` is not available for this field. | -| `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/codeowners/index.md). | +| `allowed_to_merge` **(PREMIUM ALL)** | array | no | Array of merge access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_push` **(PREMIUM ALL)** | array | no | Array of push access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_unprotect` **(PREMIUM ALL)** | array | no | Array of unprotect access levels, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, `{access_level: integer}`, or `{id: integer, _destroy: true}` to destroy an existing access level. The access level `No access` is not available for this field. | +| `code_owner_approval_required` **(PREMIUM ALL)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/codeowners/index.md). | Elements in the `allowed_to_push`, `allowed_to_merge` and `allowed_to_unprotect` arrays should be one of `user_id`, `group_id` or `access_level`, and take the form `{user_id: integer}`, `{group_id: integer}` or @@ -486,14 +486,14 @@ To delete: curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{"access_level": 40}]}' \ --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/master" + "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [ { "id": 12, @@ -511,14 +511,14 @@ Example response: ```shell curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{"id": 12, "access_level": 0}]}' \ - --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/master" + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [ { "id": 12, @@ -536,14 +536,14 @@ Example response: ```shell curl --header 'Content-Type: application/json' --request PATCH \ --data '{"allowed_to_push": [{"id": 12, "_destroy": true}]}' \ - --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/master" + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/22034114/protected_branches/main" ``` Example response: ```json { - "name": "master", + "name": "main", "push_access_levels": [] } ``` diff --git a/doc/api/remote_mirrors.md b/doc/api/remote_mirrors.md index 6b7bd3a485f..4808c317dc4 100644 --- a/doc/api/remote_mirrors.md +++ b/doc/api/remote_mirrors.md @@ -107,7 +107,7 @@ POST /projects/:id/remote_mirrors | `enabled` | Boolean | no | Determines if the mirror is enabled. | | `keep_divergent_refs` | Boolean | no | Determines if divergent refs are skipped. | | `only_protected_branches` | Boolean | no | Determines if only protected branches are mirrored. | -| `mirror_branch_regex` **(PREMIUM)** | String | no | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_protected_branches` to be disabled. | +| `mirror_branch_regex` **(PREMIUM ALL)** | String | no | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_protected_branches` to be disabled. | Example request: @@ -148,7 +148,7 @@ PUT /projects/:id/remote_mirrors/:mirror_id | `enabled` | Boolean | no | Determines if the mirror is enabled. | | `keep_divergent_refs` | Boolean | no | Determines if divergent refs are skipped. | | `only_protected_branches` | Boolean | no | Determines if only protected branches are mirrored. | -| `mirror_branch_regex`**(PREMIUM)** | String | no | Determines if only the branch whose name matches the regex is mirrored. It does not work with `only_protected_branches` enabled. | +| `mirror_branch_regex`**(PREMIUM ALL)** | String | no | Determines if only the branch whose name matches the regex is mirrored. It does not work with `only_protected_branches` enabled. | Example request: diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 49415f42789..aae475a0356 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -189,7 +189,7 @@ Supported attributes: | `straight` | boolean | no | Comparison method: `true` for direct comparison between `from` and `to` (`from`..`to`), `false` to compare using merge base (`from`...`to`)'. Default is `false`. | ```plaintext -GET /projects/:id/repository/compare?from=master&to=feature +GET /projects/:id/repository/compare?from=main&to=feature ``` Example response: @@ -217,7 +217,7 @@ Example response: "new_path": "files/js/application.js", "a_mode": null, "b_mode": "100644", - "diff": "--- a/files/js/application.js\n+++ b/files/js/application.js\n@@ -24,8 +24,10 @@\n //= require g.raphael-min\n //= require g.bar-min\n //= require branch-graph\n-//= require highlightjs.min\n-//= require ace/ace\n //= require_tree .\n //= require d3\n //= require underscore\n+\n+function fix() { \n+ alert(\"Fixed\")\n+}", + "diff": "@@ -24,8 +24,10 @@\n //= require g.raphael-min\n //= require g.bar-min\n //= require branch-graph\n-//= require highlightjs.min\n-//= require ace/ace\n //= require_tree .\n //= require d3\n //= require underscore\n+\n+function fix() { \n+ alert(\"Fixed\")\n+}", "new_file": false, "renamed_file": false, "deleted_file": false diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 969470dcc54..cc0b4ea1972 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -35,7 +35,7 @@ GET /projects/:id/repository/files/:file_path ``` ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=main" ``` | Attribute | Type | Required | Description | @@ -54,7 +54,7 @@ Example response: "encoding": "base64", "content": "IyA9PSBTY2hlbWEgSW5mb3...", "content_sha256": "4c294617b60715c1d218e61164a3abd4808a4284cbc30e6728a01ad9aada4481", - "ref": "master", + "ref": "main", "blob_id": "79f7bbd25901e8334750839545a9bd021f0e4c83", "commit_id": "d5a3ff139356ce33e37e73add446f16869741b50", "last_commit_id": "570e7b2abdd848b95f2f578043fc23bd6f6fd24d", @@ -73,7 +73,7 @@ HEAD /projects/:id/repository/files/:file_path ``` ```shell -curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=master" +curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=main" ``` Example response: @@ -88,7 +88,7 @@ X-Gitlab-Encoding: base64 X-Gitlab-File-Name: key.rb X-Gitlab-File-Path: app/models/key.rb X-Gitlab-Last-Commit-Id: 570e7b2abdd848b95f2f578043fc23bd6f6fd24d -X-Gitlab-Ref: master +X-Gitlab-Ref: main X-Gitlab-Size: 1476 X-Gitlab-Execute-Filemode: false ... @@ -112,7 +112,7 @@ GET /projects/:id/repository/files/:file_path/blame | `range` | hash | no | Blame range. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=main" ``` Example response: @@ -147,7 +147,7 @@ NOTE: `HEAD` method returns just file metadata, as in [Get file from repository](repository_files.md#get-file-from-repository). ```shell -curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master" +curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=main" ``` Example response: @@ -162,7 +162,7 @@ X-Gitlab-Encoding: base64 X-Gitlab-File-Name: file.rb X-Gitlab-File-Path: path/to/file.rb X-Gitlab-Last-Commit-Id: 570e7b2abdd848b95f2f578043fc23bd6f6fd24d -X-Gitlab-Ref: master +X-Gitlab-Ref: main X-Gitlab-Size: 1476 X-Gitlab-Execute-Filemode: false ... @@ -174,7 +174,7 @@ To request a blame range, specify `range[start]` and `range[end]` parameters wit the starting and ending line numbers of the file. ```shell -curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master&range[start]=1&range[end]=2" +curl --head --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=main&range[start]=1&range[end]=2" ``` Example response: @@ -218,7 +218,7 @@ GET /projects/:id/repository/files/:file_path/raw | `lfs` | boolean | no | Determines if the response should be Git LFS file contents, rather than the pointer. If the file is not tracked by Git LFS, ignored. Defaults to `false`. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=master" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=main" ``` NOTE: @@ -251,7 +251,7 @@ POST /projects/:id/repository/files/:file_path ```shell curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", + --data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname", "content": "some content", "commit_message": "create a new file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` @@ -261,7 +261,7 @@ Example response: ```json { "file_path": "app/project.rb", - "branch": "master" + "branch": "main" } ``` @@ -293,7 +293,7 @@ PUT /projects/:id/repository/files/:file_path ```shell curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", + --data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname", "content": "some content", "commit_message": "update file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` @@ -303,7 +303,7 @@ Example response: ```json { "file_path": "app/project.rb", - "branch": "master" + "branch": "main" } ``` @@ -339,7 +339,7 @@ DELETE /projects/:id/repository/files/:file_path ```shell curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", + --data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname", "commit_message": "delete file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` diff --git a/doc/api/repository_submodules.md b/doc/api/repository_submodules.md index 650dc7783b8..7ed8e0bd33a 100644 --- a/doc/api/repository_submodules.md +++ b/doc/api/repository_submodules.md @@ -27,7 +27,7 @@ PUT /projects/:id/repository/submodules/:submodule ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/submodules/lib%2Fmodules%2Fexample" \ ---data "branch=master&commit_sha=3ddec28ea23acc5caa5d8331a6ecb2a65fc03e88&commit_message=Update submodule reference" +--data "branch=main&commit_sha=3ddec28ea23acc5caa5d8331a6ecb2a65fc03e88&commit_message=Update submodule reference" ``` Example response: diff --git a/doc/api/rest/deprecations.md b/doc/api/rest/deprecations.md index 80480e3f88f..154945977b2 100644 --- a/doc/api/rest/deprecations.md +++ b/doc/api/rest/deprecations.md @@ -61,10 +61,9 @@ The `changes from a single merge request` endpoint will be removed in v5 of the Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/397067). -The endpoint to get -[all managed licenses for a given project](../managed_licenses.md) -has been deprecated in favor the +The endpoint to get all managed licenses for a given project has been deprecated in favor the [License Approval policy](../../user/compliance/license_approval_policies.md) feature. + Users who wish to continue to enforce approvals based on detected licenses are encouraged to create a new [License Approval policy](../../user/compliance/license_approval_policies.md) instead. The `managed licenses` endpoint will be removed in v5 of the GitLab REST API. diff --git a/doc/api/rest/index.md b/doc/api/rest/index.md index ba705a771c1..17da691b720 100644 --- a/doc/api/rest/index.md +++ b/doc/api/rest/index.md @@ -22,7 +22,7 @@ endpoint. New features can be added to the API in the same version number. New features and bug fixes are released in tandem with GitLab. Apart -from incidental patch and security releases, GitLab is released on the 22nd of each +from incidental patch and security releases, new minor versions of GitLab are released every month. Major API version changes, and removal of entire API versions, are done in tandem with major GitLab releases. @@ -136,13 +136,10 @@ curl --header "Authorization: Bearer OAUTH-TOKEN" "https://gitlab.example.com/ap Read more about [GitLab as an OAuth 2.0 provider](../oauth2.md). NOTE: -You should give OAuth access tokens an expiration. You can use the `refresh_token` parameter -to refresh tokens. Integrations may need to be updated to use refresh tokens prior to -expiration, which is based on the [`expires_in`](https://datatracker.ietf.org/doc/html/rfc6749#appendix-A.14) -property in the token endpoint response. See [OAuth 2.0 token](../oauth2.md) documentation -for examples requesting a new access token using a refresh token. - -A default refresh setting of two hours is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336598). +All OAuth access tokens are valid for two hours after they are created. You can +use the `refresh_token` parameter to refresh tokens. See +[OAuth 2.0 token](../oauth2.md) documentation for how to request a new access +token using a refresh token. ### Personal/project/group access tokens @@ -167,6 +164,24 @@ You can also use personal, project, or group access tokens with OAuth-compliant curl --header "Authorization: Bearer <your_access_token>" "https://gitlab.example.com/api/v4/projects" ``` +### Job tokens + +You can use job tokens to authenticate with [specific API endpoints](../../ci/jobs/ci_job_token.md) +by passing the token in the `job_token` parameter or the `JOB-TOKEN` header. +To pass the token in GitLab CI/CD jobs, use the `CI_JOB_TOKEN` variable. + +Example of using the job token in a parameter: + +```shell +curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts?job_token=$CI_JOB_TOKEN" +``` + +Example of using the job token in a header: + +```shell +curl --header "JOB-TOKEN:$CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/releases" +``` + ### Session cookie Signing in to the main GitLab application sets a `_gitlab_session` cookie. The @@ -525,6 +540,7 @@ options: | [Project jobs](../jobs.md#list-project-jobs) | `order_by=id`, `sort=desc` only | Authenticated users only. | | [Project audit events](../audit_events.md#retrieve-all-project-audit-events) | `order_by=id`, `sort=desc` only | Authenticated users only. | | [Projects](../projects.md) | `order_by=id` only | Authenticated and unauthenticated users. | +| [Users](../users.md) | `order_by=id`, `order_by=name`, `order_by=username` | Authenticated and unauthenticated users. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419556) in GitLab 15.4 [with a flag](../../user/feature_flags.md)) named `api_keyset_pagination_multi_order`. Disabled by default. | ### Pagination response headers diff --git a/doc/api/runners.md b/doc/api/runners.md index 7c2bceb70eb..dba37edcb01 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -395,7 +395,7 @@ Example response: "status": "running", "stage": "test", "name": "test", - "ref": "master", + "ref": "main", "tag": false, "coverage": null, "created_at": "2017-11-16T08:50:29.000Z", @@ -439,7 +439,7 @@ Example response: "pipeline": { "id": 2, "sha": "97de212e80737a608d939f648d959671fb0a0142", - "ref": "master", + "ref": "main", "status": "running" }, "project": { diff --git a/doc/api/saml.md b/doc/api/saml.md index a44e1537f1e..60fd549f84f 100644 --- a/doc/api/saml.md +++ b/doc/api/saml.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/scim.md b/doc/api/scim.md index 94662250313..5fec030c110 100644 --- a/doc/api/scim.md +++ b/doc/api/scim.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/search.md b/doc/api/search.md index 06e62d28534..4a271d9ebdc 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -53,13 +53,13 @@ Example response: "path": "flight", "path_with_namespace": "twitter/flight", "created_at": "2017-09-05T07:58:01.621Z", - "default_branch": "master", + "default_branch": "main", "tag_list":[], //deprecated, use `topics` instead "topics":[], "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", "http_url_to_repo": "http://localhost:3000/twitter/flight.git", "web_url": "http://localhost:3000/twitter/flight", - "readme_url": "http://localhost:3000/twitter/flight/-/blob/master/README.md", + "readme_url": "http://localhost:3000/twitter/flight/-/blob/main/README.md", "avatar_url": null, "star_count": 0, "forks_count": 0, @@ -153,7 +153,7 @@ Example response: "state": "opened", "created_at": "2018-01-22T14:21:50.830Z", "updated_at": "2018-02-06T12:40:33.295Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "jaja-test", "upvotes": 0, "downvotes": 0, @@ -290,7 +290,7 @@ Example response: "path": "home.md", "filename": "home.md", "id": null, - "ref": "master", + "ref": "main", "startline": 5, "project_id": 6, "group_id": null @@ -367,7 +367,7 @@ Example response: "path": "README.md", "filename": "README.md", "id": null, - "ref": "master", + "ref": "main", "startline": 46, "project_id": 6 } @@ -475,13 +475,13 @@ Example response: "path": "flight", "path_with_namespace": "twitter/flight", "created_at": "2017-09-05T07:58:01.621Z", - "default_branch": "master", + "default_branch": "main", "tag_list":[], //deprecated, use `topics` instead "topics":[], "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", "http_url_to_repo": "http://localhost:3000/twitter/flight.git", "web_url": "http://localhost:3000/twitter/flight", - "readme_url": "http://localhost:3000/twitter/flight/-/blob/master/README.md", + "readme_url": "http://localhost:3000/twitter/flight/-/blob/main/README.md", "avatar_url": null, "star_count": 0, "forks_count": 0, @@ -575,7 +575,7 @@ Example response: "state": "opened", "created_at": "2018-01-22T14:21:50.830Z", "updated_at": "2018-02-06T12:40:33.295Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "jaja-test", "upvotes": 0, "downvotes": 0, @@ -681,7 +681,7 @@ Example response: "path": "home.md", "filename": "home.md", "id": null, - "ref": "master", + "ref": "main", "startline": 5, "project_id": 6, "group_id": 1 @@ -758,7 +758,7 @@ Example response: "path": "README.md", "filename": "README.md", "id": null, - "ref": "master", + "ref": "main", "startline": 46, "project_id": 6 } @@ -934,7 +934,7 @@ Example response: "state": "opened", "created_at": "2018-01-22T14:21:50.830Z", "updated_at": "2018-02-06T12:40:33.295Z", - "target_branch": "master", + "target_branch": "main", "source_branch": "jaja-test", "upvotes": 0, "downvotes": 0, @@ -1094,7 +1094,7 @@ Example response: "path": "home.md", "filename": "home.md", "id": null, - "ref": "master", + "ref": "main", "startline": 5, "project_id": 6, "group_id": 1 @@ -1177,7 +1177,7 @@ Example response: "path": "README.md", "filename": "README.md", "id": null, - "ref": "master", + "ref": "main", "startline": 46, "project_id": 6 } diff --git a/doc/api/settings.md b/doc/api/settings.md index aeb95f54431..d1921739d5c 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -17,6 +17,9 @@ For information on how to control the application settings cache for an instance ## Get current application settings +> - `always_perform_delayed_deletion` feature flag [enabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332) in GitLab 15.11. +> - `delayed_project_deletion` and `delayed_group_deletion` attributes removed in GitLab 16.0. + List the current [application settings](#list-of-settings-that-can-be-accessed-via-api-calls) of the GitLab instance. @@ -65,6 +68,7 @@ Example response: "container_registry_expiration_policies_caching": true, "container_registry_expiration_policies_worker_capacity": 4, "container_registry_token_expire_delay": 5, + "decompress_archive_file_timeout": 210, "repository_storages_weighted": {"default": 100}, "plantuml_enabled": false, "plantuml_url": null, @@ -126,8 +130,6 @@ these parameters: - `file_template_project_id` - `geo_node_allowed_ips` - `geo_status_timeout` -- `delayed_project_deletion` -- `delayed_group_deletion` - `default_project_deletion_protection` - `deletion_adjourned_period` - `disable_personal_access_tokens` @@ -135,9 +137,6 @@ these parameters: - `delete_unconfirmed_users` - `unconfirmed_users_delete_after_days` -From [GitLab 15.11](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332), with the `always_perform_delayed_deletion` feature flag enabled, -the `delayed_project_deletion` and `delayed_group_deletion` attributes will not be exposed. These attributes will be removed in GitLab 16.0. - ```json { "id": 1, @@ -145,8 +144,6 @@ the `delayed_project_deletion` and `delayed_group_deletion` attributes will not "group_owners_can_manage_default_branch_protection": true, "file_template_project_id": 1, "geo_node_allowed_ips": "0.0.0.0/0, ::/0", - "delayed_project_deletion": false, - "delayed_group_deletion": false, "default_project_deletion_protection": false, "deletion_adjourned_period": 7, "disable_personal_access_tokens": false, @@ -156,6 +153,9 @@ the `delayed_project_deletion` and `delayed_group_deletion` attributes will not ## Change application settings +> - `always_perform_delayed_deletion` feature flag [enabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332) in GitLab 15.11. +> - `delayed_project_deletion` and `delayed_group_deletion` attributes removed in GitLab 16.0. + Use an API call to modify GitLab instance [application settings](#list-of-settings-that-can-be-accessed-via-api-calls). @@ -208,6 +208,7 @@ Example response: "container_registry_expiration_policies_caching": true, "container_registry_expiration_policies_worker_capacity": 4, "container_registry_token_expire_delay": 5, + "decompress_archive_file_timeout": 210, "package_registry_cleanup_policies_worker_capacity": 2, "repository_storages": ["default"], "plantuml_enabled": false, @@ -271,8 +272,6 @@ these parameters: - `file_template_project_id` - `geo_node_allowed_ips` - `geo_status_timeout` -- `delayed_project_deletion` -- `delayed_group_deletion` - `default_project_deletion_protection` - `deletion_adjourned_period` - `disable_personal_access_tokens` @@ -280,9 +279,6 @@ these parameters: - `delete_unconfirmed_users` - `unconfirmed_users_delete_after_days` -From [GitLab 15.11](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332), with the `always_perform_delayed_deletion` feature flag enabled, -the `delayed_project_deletion` and `delayed_group_deletion` attributes will not be exposed. These attributes will be removed in GitLab 16.0. - Example responses: **(PREMIUM SELF)** ```json @@ -312,8 +308,8 @@ listed in the descriptions of the relevant settings. | `after_sign_up_text` | string | no | Text shown to the user after signing up. | | `akismet_api_key` | string | required by: `akismet_enabled` | API key for Akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable Akismet spam protection. | -| `allow_account_deletion` **(PREMIUM)** | boolean | no | Set to `true` to allow users to delete their accounts. | -| `allow_group_owners_to_manage_ldap` **(PREMIUM)** | boolean | no | Set to `true` to allow group owners to manage LDAP. | +| `allow_account_deletion` **(PREMIUM ALL)** | boolean | no | Set to `true` to allow users to delete their accounts. | +| `allow_group_owners_to_manage_ldap` **(PREMIUM ALL)** | boolean | no | Set to `true` to allow group owners to manage LDAP. | | `allow_local_requests_from_hooks_and_services` | boolean | no | (Deprecated: Use `allow_local_requests_from_web_hooks_and_services` instead) Allow requests to the local network from webhooks and integrations. | | `allow_local_requests_from_system_hooks` | boolean | no | Allow requests to the local network from system hooks. | | `allow_local_requests_from_web_hooks_and_services` | boolean | no | Allow requests to the local network from webhooks and integrations. | @@ -328,10 +324,10 @@ listed in the descriptions of the relevant settings. | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It automatically builds, tests, and deploys applications based on a predefined CI/CD configuration. | | `automatic_purchased_storage_allocation` | boolean | no | Enabling this permits automatic allocation of purchased storage in a namespace. Relevant only to EE distributions. | -| `bulk_import_enabled` | boolean | no | Enable migrating GitLab groups by direct transfer. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. Setting also [available](../administration/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer) in the Admin Area. | +| `bulk_import_enabled` | boolean | no | Enable migrating GitLab groups by direct transfer. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. Setting also [available](../administration/settings/import_and_export_settings.md#enable-migration-of-groups-and-projects-by-direct-transfer) in the Admin Area. | | `bulk_import_max_download_file_size` | integer | no | Maximum download file size when importing from source GitLab instances by direct transfer. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384976) in GitLab 16.3. | | `can_create_group` | boolean | no | Indicates whether users can create top-level groups. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367754) in GitLab 15.5. Defaults to `true`. | -| `check_namespace_plan` **(PREMIUM)** | boolean | no | Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | +| `check_namespace_plan` **(PREMIUM ALL)** | boolean | no | Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `ci_max_total_yaml_size_bytes` | integer | no | The maximum amount of memory, in bytes, that can be allocated for the pipeline configuration, with all included YAML configuration files. | | `ci_max_includes` | integer | no | The [maximum number of includes](../administration/settings/continuous_integration.md#maximum-includes) per pipeline. Default is `150`. | | `commit_email_hostname` | string | no | Custom hostname (for private commit emails). | @@ -346,22 +342,21 @@ listed in the descriptions of the relevant settings. | `allow_account_deletion` | boolean | no | Enable [users to delete their accounts](../administration/settings/account_and_limit_settings.md#prevent-users-from-deleting-their-accounts). | | `deactivate_dormant_users` | boolean | no | Enable [automatic deactivation of dormant users](../administration/moderate_users.md#automatically-deactivate-dormant-users). | | `deactivate_dormant_users_period` | integer | no | Length of time (in days) after which a user is considered dormant. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336747) in GitLab 15.3. | +| `decompress_archive_file_timeout` | integer | no | Default timeout for decompressing archived files, in seconds. Set to 0 to disable timeouts. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129161) in GitLab 16.4. | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_name` | string | no | [Instance-level custom initial branch name](../user/project/repository/branches/default.md#instance-level-custom-initial-branch-name). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225258) in GitLab 13.2. | | `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both users with the Developer role or Maintainer role can push new commits and force push)_, `1` _(partially protected, users with the Developer role or Maintainer role can push new commits, but cannot force push)_ or `2` _(fully protected, users with the Developer or Maintainer role cannot push new commits, but users with the Developer or Maintainer role can; no one can force push)_ as a parameter. Default is `2`. | | `default_ci_config_path` | string | no | Default CI/CD configuration file and path for new projects (`.gitlab-ci.yml` if not set). | -| `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | +| `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131203) in GitLab 16.4: cannot be set to any levels in `restricted_visibility_levels`.| | `default_preferred_language` | string | no | Default preferred language for users who are not logged in. | | `default_project_creation` | integer | no | Default project creation protection. Can take: `0` _(No one)_, `1` _(Maintainers)_ or `2` _(Developers + Maintainers)_| -| `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | +| `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131203) in GitLab 16.4: cannot be set to any levels in `restricted_visibility_levels`.| | `default_projects_limit` | integer | no | Project limit per user. Default is `100000`. | | `default_snippet_visibility` | string | no | What visibility level new snippets receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | -| `default_syntax_highlighting_theme` | integer | no | Default syntax highlighting theme for new users and users who are not signed in. See [IDs of available themes](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/themes.rb#L16). -| `delayed_project_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed project deletion by default in new groups. Default is `false`. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), can only be enabled when `delayed_group_deletion` is true. From [GitLab 15.11](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332), with the `always_perform_delayed_deletion` feature flag enabled, this attribute has been removed. This attribute will be completely removed in GitLab 16.0. | -| `delayed_group_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed group deletion. Default is `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352959) in GitLab 15.0. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), disables and locks the group-level setting for delayed protect deletion when set to `false`. From [GitLab 15.11](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113332), with the `always_perform_delayed_deletion` feature flag enabled, this attribute has been removed. This attribute will be completely removed in GitLab 16.0. | +| `default_syntax_highlighting_theme` | integer | no | Default syntax highlighting theme for users who are new or not signed in. See [IDs of available themes](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/themes.rb#L16). | | `default_project_deletion_protection` **(PREMIUM SELF)** | boolean | no | Enable default project deletion protection so only administrators can delete projects. Default is `false`. | | `delete_unconfirmed_users` **(PREMIUM SELF)** | boolean | no | Specifies whether users who have not confirmed their email should be deleted. Default is `false`. When set to `true`, unconfirmed users are deleted after `unconfirmed_users_delete_after_days` days. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352514) in GitLab 16.1. | -| `deletion_adjourned_period` **(PREMIUM SELF)** | integer | no | The number of days to wait before deleting a project or group that is marked for deletion. Value must be between `1` and `90`. Defaults to `7`. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), a hook on `deletion_adjourned_period` sets the period to `1` on every update, and sets both `delayed_project_deletion` and `delayed_group_deletion` to `false` if the period is `0`. | +| `deletion_adjourned_period` **(PREMIUM SELF)** | integer | no | Number of days to wait before deleting a project or group that is marked for deletion. Value must be between `1` and `90`. Defaults to `7`. | | `diagramsnet_enabled` | boolean | no | (If enabled, requires `diagramsnet_url`) Enable [Diagrams.net integration](../administration/integration/diagrams_net.md). Default is `true`. | | `diagramsnet_url` | string | required by: `diagramsnet_enabled` | The Diagrams.net instance URL for integration. | | `diff_max_patch_bytes` | integer | no | Maximum [diff patch size](../administration/diff_limits.md), in bytes. | @@ -384,25 +379,25 @@ listed in the descriptions of the relevant settings. | `eks_account_id` | string | no | Amazon account ID. | | `eks_integration_enabled` | boolean | no | Enable integration with Amazon EKS. | | `eks_secret_access_key` | string | no | AWS IAM secret access key. | -| `elasticsearch_aws_access_key` **(PREMIUM)** | string | no | AWS IAM access key. | -| `elasticsearch_aws_region` **(PREMIUM)** | string | no | The AWS region the Elasticsearch domain is configured. | -| `elasticsearch_aws_secret_access_key` **(PREMIUM)** | string | no | AWS IAM secret access key. | -| `elasticsearch_aws` **(PREMIUM)** | boolean | no | Enable the use of AWS hosted Elasticsearch. | -| `elasticsearch_indexed_field_length_limit` **(PREMIUM)** | integer | no | Maximum size of text fields to index by Elasticsearch. 0 value means no limit. This does not apply to repository and wiki indexing. | -| `elasticsearch_indexed_file_size_limit_kb` **(PREMIUM)** | integer | no | Maximum size of repository and wiki files that are indexed by Elasticsearch. | -| `elasticsearch_indexing` **(PREMIUM)** | boolean | no | Enable Elasticsearch indexing. | -| `elasticsearch_requeue_workers` **(PREMIUM)** | boolean | no | Enable automatic requeuing of indexing workers. This improves non-code indexing throughput by enqueuing Sidekiq jobs until all documents are processed. | -| `elasticsearch_limit_indexing` **(PREMIUM)** | boolean | no | Limit Elasticsearch to index certain namespaces and projects. | -| `elasticsearch_max_bulk_concurrency` **(PREMIUM)** | integer | no | Maximum concurrency of Elasticsearch bulk requests per indexing operation. This only applies to repository indexing operations. | -| `elasticsearch_worker_number_of_shards` **(PREMIUM)** | integer | no | Number of indexing worker shards. This improves non-code indexing throughput by enqueuing more parallel Sidekiq jobs. Default is `2`. | -| `elasticsearch_max_bulk_size_mb` **(PREMIUM)** | integer | no | Maximum size of Elasticsearch bulk indexing requests in MB. This only applies to repository indexing operations. | -| `elasticsearch_namespace_ids` **(PREMIUM)** | array of integers | no | The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | -| `elasticsearch_project_ids` **(PREMIUM)** | array of integers | no | The projects to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | -| `elasticsearch_search` **(PREMIUM)** | boolean | no | Enable Elasticsearch search. | -| `elasticsearch_url` **(PREMIUM)** | string | no | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (for example, `http://localhost:9200, http://localhost:9201"`). | -| `elasticsearch_username` **(PREMIUM)** | string | no | The `username` of your Elasticsearch instance. | -| `elasticsearch_password` **(PREMIUM)** | string | no | The password of your Elasticsearch instance. | -| `email_additional_text` **(PREMIUM)** | string | no | Additional text added to the bottom of every email for legal/auditing/compliance reasons. | +| `elasticsearch_aws_access_key` **(PREMIUM ALL)** | string | no | AWS IAM access key. | +| `elasticsearch_aws_region` **(PREMIUM ALL)** | string | no | The AWS region the Elasticsearch domain is configured. | +| `elasticsearch_aws_secret_access_key` **(PREMIUM ALL)** | string | no | AWS IAM secret access key. | +| `elasticsearch_aws` **(PREMIUM ALL)** | boolean | no | Enable the use of AWS hosted Elasticsearch. | +| `elasticsearch_indexed_field_length_limit` **(PREMIUM ALL)** | integer | no | Maximum size of text fields to index by Elasticsearch. 0 value means no limit. This does not apply to repository and wiki indexing. | +| `elasticsearch_indexed_file_size_limit_kb` **(PREMIUM ALL)** | integer | no | Maximum size of repository and wiki files that are indexed by Elasticsearch. | +| `elasticsearch_indexing` **(PREMIUM ALL)** | boolean | no | Enable Elasticsearch indexing. | +| `elasticsearch_requeue_workers` **(PREMIUM ALL)** | boolean | no | Enable automatic requeuing of indexing workers. This improves non-code indexing throughput by enqueuing Sidekiq jobs until all documents are processed. | +| `elasticsearch_limit_indexing` **(PREMIUM ALL)** | boolean | no | Limit Elasticsearch to index certain namespaces and projects. | +| `elasticsearch_max_bulk_concurrency` **(PREMIUM ALL)** | integer | no | Maximum concurrency of Elasticsearch bulk requests per indexing operation. This only applies to repository indexing operations. | +| `elasticsearch_worker_number_of_shards` **(PREMIUM ALL)** | integer | no | Number of indexing worker shards. This improves non-code indexing throughput by enqueuing more parallel Sidekiq jobs. Default is `2`. | +| `elasticsearch_max_bulk_size_mb` **(PREMIUM ALL)** | integer | no | Maximum size of Elasticsearch bulk indexing requests in MB. This only applies to repository indexing operations. | +| `elasticsearch_namespace_ids` **(PREMIUM ALL)** | array of integers | no | The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `elasticsearch_project_ids` **(PREMIUM ALL)** | array of integers | no | The projects to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `elasticsearch_search` **(PREMIUM ALL)** | boolean | no | Enable Elasticsearch search. | +| `elasticsearch_url` **(PREMIUM ALL)** | string | no | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (for example, `http://localhost:9200, http://localhost:9201"`). | +| `elasticsearch_username` **(PREMIUM ALL)** | string | no | The `username` of your Elasticsearch instance. | +| `elasticsearch_password` **(PREMIUM ALL)** | string | no | The password of your Elasticsearch instance. | +| `email_additional_text` **(PREMIUM ALL)** | string | no | Additional text added to the bottom of every email for legal/auditing/compliance reasons. | | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `email_confirmation_setting` | string | no | Specifies whether users must confirm their email before sign in. Possible values are `off`, `soft`, and `hard`. | | `custom_http_clone_url_root` | string | no | Set a custom Git clone URL for HTTP(S). | @@ -421,12 +416,12 @@ listed in the descriptions of the relevant settings. | `external_pipeline_validation_service_timeout` | integer | no | How long to wait for a response from the pipeline validation service. Assumes `OK` if it times out. | | `static_objects_external_storage_url` | string | no | URL to an external storage for repository static objects. | | `static_objects_external_storage_auth_token` | string | required by: `static_objects_external_storage_url` | Authentication token for the external storage linked in `static_objects_external_storage_url`. | -| `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | +| `file_template_project_id` **(PREMIUM ALL)** | integer | no | The ID of a project to load custom file templates from. | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `globally_allowed_ips` | string | no | Comma-separated list of IP addresses and CIDRs always allowed for inbound traffic. For example, `1.1.1.1, 2.2.2.0/24`. | -| `geo_node_allowed_ips` **(PREMIUM)** | string | yes | Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | -| `geo_status_timeout` **(PREMIUM)** | integer | no | The amount of seconds after which a request to get a secondary node status times out. | -| `git_two_factor_session_expiry` **(PREMIUM)** | integer | no | Maximum duration (in minutes) of a session for Git operations when 2FA is enabled. | +| `geo_node_allowed_ips` **(PREMIUM ALL)** | string | yes | Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | +| `geo_status_timeout` **(PREMIUM ALL)** | integer | no | The amount of seconds after which a request to get a secondary node status times out. | +| `git_two_factor_session_expiry` **(PREMIUM ALL)** | integer | no | Maximum duration (in minutes) of a session for Git operations when 2FA is enabled. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | @@ -439,7 +434,7 @@ listed in the descriptions of the relevant settings. | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | | `help_page_support_url` | string | no | Alternate support URL for help page and help dropdown list. | | `help_page_text` | string | no | Custom text displayed on the help page. | -| `help_text` **(PREMIUM)** | string | no | Deprecated: Use `description` parameter in the [Appearance API](../api/appearance.md). Custom text in sign-in page. | +| `help_text` **(PREMIUM ALL)** | string | no | Deprecated: Use `description` parameter in the [Appearance API](../api/appearance.md). Custom text in sign-in page. | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties in GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | no | Deprecated. Git pack file bitmap creation is always enabled and cannot be changed via API and UI. Always returns `true`. | @@ -457,8 +452,8 @@ listed in the descriptions of the relevant settings. | `local_markdown_version` | integer | no | Increase this value when any cached Markdown should be invalidated. | | `mailgun_signing_key` | string | no | The Mailgun HTTP webhook signing key for receiving events from webhook. | | `mailgun_events_enabled` | boolean | no | Enable Mailgun event receiver. | -| `maintenance_mode_message` **(PREMIUM)** | string | no | Message displayed when instance is in maintenance mode. | -| `maintenance_mode` **(PREMIUM)** | boolean | no | When instance is in maintenance mode, non-administrative users can sign in with read-only access and make read-only API requests. | +| `maintenance_mode_message` **(PREMIUM ALL)** | string | no | Message displayed when instance is in maintenance mode. | +| `maintenance_mode` **(PREMIUM ALL)** | boolean | no | When instance is in maintenance mode, non-administrative users can sign in with read-only access and make read-only API requests. | | `max_artifacts_size` | integer | no | Maximum artifacts size in MB. | | `max_attachment_size` | integer | no | Limit attachment size in MB. | | `max_decompressed_archive_size` | integer | no | Maximum decompressed file size for imported archives in MB. Set to `0` for unlimited. Default is `25600`. | @@ -478,23 +473,23 @@ listed in the descriptions of the relevant settings. | `git_rate_limit_users_alertlist` **(ULTIMATE SELF)** | array of integers | no | List of user IDs that are emailed when the Git abuse rate limit is exceeded. Default: `[]`, Maximum: 100 user IDs. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110201) in GitLab 15.9. | | `auto_ban_user_on_excessive_projects_download` **(ULTIMATE SELF)** | boolean | no | When enabled, users will get automatically banned from the application when they download more than the maximum number of unique projects in the time period specified by `max_number_of_repository_downloads` and `max_number_of_repository_downloads_within_time_period` respectively. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94153) in GitLab 15.4 | | `mirror_available` | boolean | no | Allow repository mirroring to configured by project Maintainers. If disabled, only Administrators can configure repository mirroring. | -| `mirror_capacity_threshold` **(PREMIUM)** | integer | no | Minimum capacity to be available before scheduling more mirrors preemptively. | -| `mirror_max_capacity` **(PREMIUM)** | integer | no | Maximum number of mirrors that can be synchronizing at the same time. | -| `mirror_max_delay` **(PREMIUM)** | integer | no | Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | -| `maven_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use repo.maven.apache.org as a default remote repository when the package is not found in the GitLab Package Registry for Maven. | -| `npm_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use npmjs.org as a default remote repository when the package is not found in the GitLab Package Registry for npm. | -| `pypi_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use pypi.org as a default remote repository when the package is not found in the GitLab Package Registry for PyPI. | +| `mirror_capacity_threshold` **(PREMIUM ALL)** | integer | no | Minimum capacity to be available before scheduling more mirrors preemptively. | +| `mirror_max_capacity` **(PREMIUM ALL)** | integer | no | Maximum number of mirrors that can be synchronizing at the same time. | +| `mirror_max_delay` **(PREMIUM ALL)** | integer | no | Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | +| `maven_package_requests_forwarding` **(PREMIUM ALL)** | boolean | no | Use repo.maven.apache.org as a default remote repository when the package is not found in the GitLab Package Registry for Maven. | +| `npm_package_requests_forwarding` **(PREMIUM ALL)** | boolean | no | Use npmjs.org as a default remote repository when the package is not found in the GitLab Package Registry for npm. | +| `pypi_package_requests_forwarding` **(PREMIUM ALL)** | boolean | no | Use pypi.org as a default remote repository when the package is not found in the GitLab Package Registry for PyPI. | | `outbound_local_requests_whitelist` | array of strings | no | Define a list of trusted domains or IP addresses to which local requests are allowed when local requests for webhooks and integrations are disabled. | `package_registry_allow_anyone_to_pull_option` | boolean | no | Enable to [allow anyone to pull from Package Registry](../user/packages/package_registry/index.md#allow-anyone-to-pull-from-package-registry) visible and changeable. | `package_metadata_purl_types` **(ULTIMATE SELF)** | array of integers | no | List of [package registry metadata to sync](../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync). See [the list](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/enums/package_metadata.rb#L5) of the available values. | `pages_domain_verification_enabled` | boolean | no | Require users to prove ownership of custom domains. Domain verification is an essential security measure for public GitLab sites. Users are required to demonstrate they control a domain before it is enabled. | | `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | | `password_authentication_enabled_for_web` | boolean | no | Enable authentication for the web interface via a GitLab account password. Default is `true`. | -| `minimum_password_length` **(PREMIUM)** | integer | no | Indicates whether passwords require a minimum length. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | -| `password_number_required` **(PREMIUM)** | boolean | no | Indicates whether passwords require at least one number. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | -| `password_symbol_required` **(PREMIUM)** | boolean | no | Indicates whether passwords require at least one symbol character. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | -| `password_uppercase_required` **(PREMIUM)** | boolean | no | Indicates whether passwords require at least one uppercase letter. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | -| `password_lowercase_required` **(PREMIUM)** | boolean | no | Indicates whether passwords require at least one lowercase letter. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | +| `minimum_password_length` **(PREMIUM ALL)** | integer | no | Indicates whether passwords require a minimum length. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | +| `password_number_required` **(PREMIUM ALL)** | boolean | no | Indicates whether passwords require at least one number. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | +| `password_symbol_required` **(PREMIUM ALL)** | boolean | no | Indicates whether passwords require at least one symbol character. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | +| `password_uppercase_required` **(PREMIUM ALL)** | boolean | no | Indicates whether passwords require at least one uppercase letter. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | +| `password_lowercase_required` **(PREMIUM ALL)** | boolean | no | Indicates whether passwords require at least one lowercase letter. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85763) in GitLab 15.1. | | `performance_bar_allowed_group_id` | string | no | (Deprecated: Use `performance_bar_allowed_group_path` instead) Path of the group that is allowed to toggle the performance bar. | | `performance_bar_allowed_group_path` | string | no | Path of the group that is allowed to toggle the performance bar. | | `performance_bar_enabled` | boolean | no | (Deprecated: Pass `performance_bar_allowed_group_path: nil` instead) Allow enabling the performance bar. | @@ -528,21 +523,21 @@ listed in the descriptions of the relevant settings. | `recaptcha_site_key` | string | required by: `recaptcha_enabled` | Site key for reCAPTCHA. | | `receive_max_input_size` | integer | no | Maximum push size (MB). | | `repository_checks_enabled` | boolean | no | GitLab periodically runs `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | -| `repository_size_limit` **(PREMIUM)** | integer | no | Size limit per repository (MB) | +| `repository_size_limit` **(PREMIUM ALL)** | integer | no | Size limit per repository (MB) | | `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#configure-where-new-repositories-are-stored). New projects are created in one of these stores, chosen by a weighted random selection. | | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../administration/moderate_users.md) by an administrator. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | -| `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-Administrator users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | +| `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-Administrator users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction.[Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131203) in GitLab 16.4: cannot select levels that are set as `default_project_visibility` and `default_group_visibility`. | | `rsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded RSA key. Default is `0` (no restriction). `-1` disables RSA keys. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes. | | `security_policy_global_group_approvers_enabled` | boolean | no | Whether to look up scan result policy approval groups globally or within project hierarchies. | | `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text` and `shared_runners_minutes`) Enable shared runners for new projects. | -| `shared_runners_minutes` **(PREMIUM)** | integer | required by: `shared_runners_enabled` | Set the maximum number of compute minutes that a group can use on shared runners per month. | +| `shared_runners_minutes` **(PREMIUM ALL)** | integer | required by: `shared_runners_enabled` | Set the maximum number of compute minutes that a group can use on shared runners per month. | | `shared_runners_text` | string | required by: `shared_runners_enabled` | Shared runners text. | -| `runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered instance runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-authentication-tokens). | -| `group_runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered group runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-authentication-tokens). | -| `project_runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered project runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-authentication-tokens). | +| `runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered instance runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-runner-authentication-tokens). | +| `group_runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered group runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-runner-authentication-tokens). | +| `project_runner_token_expiration_interval` | integer | no | Set the expiration time (in seconds) of authentication tokens of newly registered project runners. Minimum value is 7200 seconds. For more information, see [Automatically rotate authentication tokens](../ci/runners/configure_runners.md#automatically-rotate-runner-authentication-tokens). | | `sidekiq_job_limiter_mode` | string | no | `track` or `compress`. Sets the behavior for [Sidekiq job size limits](../administration/settings/sidekiq_job_limits.md). Default: 'compress'. | | `sidekiq_job_limiter_compression_threshold_bytes` | integer | no | The threshold in bytes at which Sidekiq jobs are compressed before being stored in Redis. Default: 100,000 bytes (100 KB). | | `sidekiq_job_limiter_limit_bytes` | integer | no | The threshold in bytes at which Sidekiq jobs are rejected. Default: 0 bytes (doesn't reject any job). | @@ -558,6 +553,7 @@ listed in the descriptions of the relevant settings. | `snippet_size_limit` | integer | no | Max snippet content size in **bytes**. Default: 52428800 Bytes (50 MB).| | `snowplow_app_id` | string | no | The Snowplow site name / application ID. (for example, `gitlab`) | | `snowplow_collector_hostname` | string | required by: `snowplow_enabled` | The Snowplow collector hostname. (for example, `snowplow.trx.gitlab.net`) | +| `snowplow_database_collector_hostname` | string | no | The Snowplow collector for database events hostname. (for example, `db-snowplow.trx.gitlab.net`) | | `snowplow_cookie_domain` | string | no | The Snowplow cookie domain. (for example, `.gitlab.com`) | | `snowplow_enabled` | boolean | no | Enable snowplow tracking. | | `sourcegraph_enabled` | boolean | no | Enables Sourcegraph integration. Default is `false`. **If enabled, requires** `sourcegraph_url`. | @@ -602,7 +598,7 @@ listed in the descriptions of the relevant settings. | `user_default_external` | boolean | no | Newly registered users are external by default. | | `user_default_internal_regex` | string | no | Specify an email address regex pattern to identify default internal users. | | `user_defaults_to_private_profile` | boolean | no | Newly created users have private profile by default. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231301) in GitLab 15.8. Defaults to `false`. | -| `user_oauth_applications` | boolean | no | Allow users to register any application to use GitLab as an OAuth provider. | +| `user_oauth_applications` | boolean | no | Allow users to register any application to use GitLab as an OAuth provider. This setting does not affect group-level OAuth applications. | | `user_show_add_ssh_key_message` | boolean | no | When set to `false` disable the `You won't be able to pull or push project code via SSH` warning shown to users with no uploaded SSH key. | | `version_check_enabled` | boolean | no | Let GitLab inform you when an update is available. | | `valid_runner_registrars` | array of strings | no | List of types which are allowed to register a GitLab Runner. Can be `[]`, `['group']`, `['project']` or `['group', 'project']`. | diff --git a/doc/api/snippets.md b/doc/api/snippets.md index 6fa6be3a43b..84b1baab6c4 100644 --- a/doc/api/snippets.md +++ b/doc/api/snippets.md @@ -184,7 +184,7 @@ Parameters: Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/snippets/1/files/master/snippet%2Erb/raw" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/snippets/1/files/main/snippet%2Erb/raw" ``` Example response: @@ -270,7 +270,7 @@ Example response: "files": [ { "path": "text.txt", - "raw_url": "https://gitlab.example.com/-/snippets/1/raw/master/renamed.md" + "raw_url": "https://gitlab.example.com/-/snippets/1/raw/main/renamed.md" } ] } @@ -355,7 +355,7 @@ Example response: "files": [ { "path": "renamed.md", - "raw_url": "https://gitlab.example.com/-/snippets/1/raw/master/renamed.md" + "raw_url": "https://gitlab.example.com/-/snippets/1/raw/main/renamed.md" } ] } diff --git a/doc/api/statistics.md b/doc/api/statistics.md index 691b03505a7..b60b012e339 100644 --- a/doc/api/statistics.md +++ b/doc/api/statistics.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/status_checks.md b/doc/api/status_checks.md index c988f8d63fc..3ed3755732f 100644 --- a/doc/api/status_checks.md +++ b/doc/api/status_checks.md @@ -35,7 +35,7 @@ GET /projects/:id/external_status_checks { "id": 14, "project_id": 6, - "name": "master", + "name": "main", "created_at": "2020-10-12T14:04:50.787Z", "updated_at": "2020-10-12T14:04:50.787Z", "code_owner_approval_required": false @@ -216,7 +216,7 @@ In case status check is already passed status code is 422 "namespace": "Flightjs", "visibility_level": 20, "path_with_namespace": "flightjs/Flight", - "default_branch": "master", + "default_branch": "main", "ci_config_path": null, "homepage": "http://example.com/flightjs/Flight", "url": "ssh://example.com/flightjs/Flight.git", @@ -242,10 +242,10 @@ In case status check is already passed status code is 422 "merge_user_id": null, "merge_when_pipeline_succeeds": false, "milestone_id": null, - "source_branch": "root-master-patch-30152", + "source_branch": "root-main-patch-30152", "source_project_id": 6, "state_id": 1, - "target_branch": "master", + "target_branch": "main", "target_project_id": 6, "time_estimate": 0, "title": "Update README.md", @@ -263,7 +263,7 @@ In case status check is already passed status code is 422 "namespace": "Flightjs", "visibility_level": 20, "path_with_namespace": "flightjs/Flight", - "default_branch": "master", + "default_branch": "main", "ci_config_path": null, "homepage": "http://example.com/flightjs/Flight", "url": "ssh://example.com/flightjs/Flight.git", @@ -281,7 +281,7 @@ In case status check is already passed status code is 422 "namespace": "Flightjs", "visibility_level": 20, "path_with_namespace": "flightjs/Flight", - "default_branch": "master", + "default_branch": "main", "ci_config_path": null, "homepage": "http://example.com/flightjs/Flight", "url": "ssh://example.com/flightjs/Flight.git", diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index 14a078ff68d..446c91d7971 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -10,7 +10,7 @@ All methods require administrator authorization. You can configure the URL endpoint of the system hooks from the GitLab user interface: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **System Hooks** (`/admin/hooks`). diff --git a/doc/api/tags.md b/doc/api/tags.md index 4b85c70f901..cc4ff7be577 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -10,8 +10,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > `version` value for the `order_by` attribute [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95150) in GitLab 15.4. -Get a list of repository tags from a project, sorted by update date and time in descending order. This endpoint can be accessed without authentication if the -repository is publicly accessible. +Get a list of repository tags from a project, sorted by update date and time in +descending order. + +NOTE: +If the repository is publicly accessible, authentication +(`--header "PRIVATE-TOKEN: <your_access_token>"`) is not required. ```plaintext GET /projects/:id/repository/tags @@ -19,12 +23,19 @@ GET /projects/:id/repository/tags Parameters: -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer or string| yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `order_by` | string | no | Return tags ordered by `name`, `updated`, or `version`. Default is `updated`. | -| `sort` | string | no | Return tags sorted in `asc` or `desc` order. Default is `desc`. | -| `search` | string | no | Return list of tags matching the search criteria. You can use `^term` and `term$` to find tags that begin and end with `term` respectively. No other regular expressions are supported. | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `order_by` | string | no | Return tags ordered by `name`, `updated`, or `version`. Default is `updated`. | +| `sort` | string | no | Return tags sorted in `asc` or `desc` order. Default is `desc`. | +| `search` | string | no | Return a list of tags matching the search criteria. You can use `^term` and `term$` to find tags that begin and end with `term` respectively. No other regular expressions are supported. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/5/repository/tags" +``` + +Example Response: ```json [ @@ -68,13 +79,14 @@ GET /projects/:id/repository/tags/:tag_name Parameters: -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `tag_name` | string | yes | The name of the tag | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `tag_name` | string | yes | The name of a tag. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/tags/v1.0.0" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/5/repository/tags/v1.0.0" ``` Example Response: @@ -115,15 +127,17 @@ POST /projects/:id/repository/tags Parameters: -| Attribute | Type | Required | Description | -| --------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------- | -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `tag_name` | string | yes | The name of a tag | -| `ref` | string | yes | Create tag using commit SHA, another tag name, or branch name | -| `message` | string | no | Creates annotated tag | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `tag_name` | string | yes | The name of a tag. | +| `ref` | string | yes | Create a tag from a commit SHA, another tag name, or branch name. | +| `message` | string | no | Create an annotated tag. | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/tags?tag_name=test&ref=master" +curl --request POST \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/5/repository/tags?tag_name=test&ref=main" ``` Example response: @@ -154,13 +168,12 @@ Example response: } ``` -The message is `null` when creating a lightweight tag. Otherwise, it contains the annotation. +The type of tag created determines the contents of `target` and `message`: -The target contains the tag objects ID when creating annotated tags, -otherwise it contains the commit ID when creating lightweight tags. +- For annotated tags, `message` contains the annotation, and `target` contains the tag object's ID. +- For lightweight tags, `message` is null, and `target` contains the commit ID. -In case of an error, -status code `405` with an explaining error message is returned. +Errors return status code `405` with an explanatory error message. ## Delete a tag @@ -172,16 +185,16 @@ DELETE /projects/:id/repository/tags/:tag_name Parameters: -| Attribute | Type | Required | Description | -| ---------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------- | -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `tag_name` | string | yes | The name of a tag | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `tag_name` | string | yes | The name of a tag. | ## Get X.509 signature of a tag > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106578) in GitLab 15.7. -Get the [X.509 signature from a tag](../user/project/repository/x509_signed_commits/index.md), +Get the [X.509 signature from a tag](../user/project/repository/signed_commits/x509.md), if it is signed. Unsigned tags return a `404 Not Found` response. ```plaintext @@ -190,13 +203,14 @@ GET /projects/:id/repository/tags/:tag_name/signature Parameters: -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | -| `tag_name` | string | yes | The name of a tag. | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `tag_name` | string | yes | The name of a tag. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/repository/tags/v1.1.1/signature" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + --url "https://gitlab.example.com/api/v4/projects/1/repository/tags/v1.1.1/signature" ``` Example response if tag is X.509 signed: diff --git a/doc/api/topics.md b/doc/api/topics.md index 5d413eec7c0..0c729a95fe5 100644 --- a/doc/api/topics.md +++ b/doc/api/topics.md @@ -22,10 +22,10 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------------ | ------- | ---------------------- | ----------- | -| `page` | integer | **{dotted-circle}** No | Page to retrieve. Defaults to `1`. | -| `per_page` | integer | **{dotted-circle}** No | Number of records to return per page. Defaults to `20`. | -| `search` | string | **{dotted-circle}** No | Search topics against their `name`. | -| `without_projects` | boolean | **{dotted-circle}** No | Limit results to topics without assigned projects. | +| `page` | integer | No | Page to retrieve. Defaults to `1`. | +| `per_page` | integer | No | Number of records to return per page. Defaults to `20`. | +| `search` | string | No | Search topics against their `name`. | +| `without_projects` | boolean | No | Limit results to topics without assigned projects. | Example request: @@ -76,7 +76,7 @@ Supported attributes: | Attribute | Type | Required | Description | | --------- | ------- | ---------------------- | ------------------- | -| `id` | integer | **{check-circle}** Yes | ID of project topic | +| `id` | integer | Yes | ID of project topic | Example request: @@ -117,10 +117,10 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------- | ------- | ---------------------- | ----------- | -| `name` | string | **{check-circle}** Yes | Slug (name) | -| `title` | string | **{check-circle}** Yes | Title | -| `avatar` | file | **{dotted-circle}** No | Avatar | -| `description` | string | **{dotted-circle}** No | Description | +| `name` | string | Yes | Slug (name) | +| `title` | string | Yes | Title | +| `avatar` | file | No | Avatar | +| `description` | string | No | Description | Example request: @@ -156,11 +156,11 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------- | ------- | ---------------------- | ------------------- | -| `id` | integer | **{check-circle}** Yes | ID of project topic | -| `avatar` | file | **{dotted-circle}** No | Avatar | -| `description` | string | **{dotted-circle}** No | Description | -| `name` | string | **{dotted-circle}** No | Slug (name) | -| `title` | string | **{dotted-circle}** No | Title | +| `id` | integer | Yes | ID of project topic | +| `avatar` | file | No | Avatar | +| `description` | string | No | Description | +| `name` | string | No | Slug (name) | +| `title` | string | No | Title | Example request: @@ -228,7 +228,7 @@ Supported attributes: | Attribute | Type | Required | Description | | ------------- | ------- | ---------------------- | ------------------- | -| `id` | integer | **{check-circle}** Yes | ID of project topic | +| `id` | integer | Yes | ID of project topic | Example request: @@ -253,8 +253,8 @@ Supported attributes: | Attribute | Type | Required | Description | | ----------------- | ------- | ---------------------- | -------------------------- | -| `source_topic_id` | integer | **{check-circle}** Yes | ID of source project topic | -| `target_topic_id` | integer | **{check-circle}** Yes | ID of target project topic | +| `source_topic_id` | integer | Yes | ID of source project topic | +| `target_topic_id` | integer | Yes | ID of target project topic | Example request: diff --git a/doc/api/usage_data.md b/doc/api/usage_data.md index 1a2c3e95002..a8e6c6ac205 100644 --- a/doc/api/usage_data.md +++ b/doc/api/usage_data.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, api diff --git a/doc/api/users.md b/doc/api/users.md index 9b00dc5c7e0..0652742c557 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -58,6 +58,8 @@ GET /users ] ``` +This endpoint supports [keyset pagination](rest/index.md#keyset-based-pagination). Keyset pagination [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419556) in GitLab 16.5 [with a flag](../user/feature_flags.md) named `api_keyset_pagination_multi_order`. Disabled by default. + You can also use `?search=` to search for users by name, username, or public email. For example, `/users?search=John`. When you search for a: - Public email, you must use the full email address to get an exact match. A search might return a partial match. For example, if you search for the email `on@example.com`, the search can return both `on@example.com` and `jon@example.com`. @@ -97,7 +99,7 @@ GET /users?external=true ``` GitLab supports bot users such as the [alert bot](../operations/incident_management/integrations.md) -or the [support bot](../user/project/service_desk/index.md#support-bot-user). +or the [support bot](../user/project/service_desk/configure.md#support-bot-user). You can exclude the following types of [internal users](../development/internal_users.md#internal-users) from the users' list with the `exclude_internal=true` parameter ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241144) in GitLab 13.4): @@ -146,9 +148,9 @@ You can use all [parameters available for everyone](#for-non-administrator-users | `two_factor` | string | no | Filter users by Two-factor authentication. Filter values are `enabled` or `disabled`. By default it returns all users | | `without_projects` | boolean | no | Filter users without projects. Default is `false`, which means that all users are returned, with and without projects. | | `admins` | boolean | no | Return only administrators. Default is `false` | -| `auditors` **(PREMIUM)** | boolean | no | Return only auditor users. Default is `false`. If not included, it returns all users. | -| `saml_provider_id` **(PREMIUM)** | number | no | Return only users created by the specified SAML provider ID. If not included, it returns all users. | -| `skip_ldap` **(PREMIUM)** | boolean | no | Skip LDAP users. | +| `auditors` **(PREMIUM ALL)** | boolean | no | Return only auditor users. Default is `false`. If not included, it returns all users. | +| `saml_provider_id` **(PREMIUM ALL)** | number | no | Return only users created by the specified SAML provider ID. If not included, it returns all users. | +| `skip_ldap` **(PREMIUM ALL)** | boolean | no | Skip LDAP users. | ```json [ @@ -326,6 +328,10 @@ Get a single user. ### For user +Prerequisites: + +- You must be signed in to use this endpoint. + ```plaintext GET /users/:id ``` @@ -532,7 +538,7 @@ Parameters: | Attribute | Required | Description | | :----------------------------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ | | `admin` | No | User is an administrator. Valid values are `true` or `false`. Defaults to false. -| `auditor` **(PREMIUM)** | No | User is an auditor. Valid values are `true` or `false`. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366404) in GitLab 15.3. | +| `auditor` **(PREMIUM ALL)** | No | User is an auditor. Valid values are `true` or `false`. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366404) in GitLab 15.3. | | `avatar` | No | Image file for user's avatar | | `bio` | No | User's biography | | `can_create_group` | No | User can create top-level groups - true or false | @@ -540,7 +546,7 @@ Parameters: | `email` | Yes | Email | | `extern_uid` | No | External UID | | `external` | No | Flags the user as external - true or false (default) | -| `extra_shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Additional compute minutes for this user. | +| `extra_shared_runners_minutes_limit` **(PREMIUM ALL)** | No | Can be set by administrators only. Additional compute minutes for this user. | | `force_random_password` | No | Set user password to a random value - true or false (default) | | `group_id_for_saml` | No | ID of group where SAML has been configured | | `linkedin` | No | LinkedIn | @@ -553,7 +559,7 @@ Parameters: | `projects_limit` | No | Number of projects user can create | | `provider` | No | External provider name | | `reset_password` | No | Send user password reset link - true or false(default) | -| `shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Maximum number of monthly compute minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | +| `shared_runners_minutes_limit` **(PREMIUM ALL)** | No | Can be set by administrators only. Maximum number of monthly compute minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | | `skip_confirmation` | No | Skip confirmation - true or false (default) | | `skype` | No | Skype ID | | `theme_id` | No | GitLab theme for the user (for more information, see the [user preference documentation](../user/profile/preferences.md#change-the-color-theme) for more information) | @@ -581,7 +587,7 @@ Parameters: | Attribute | Required | Description | | :----------------------------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ | | `admin` | No |User is an administrator. Valid values are `true` or `false`. Defaults to false. -| `auditor` **(PREMIUM)** | No | User is an auditor. Valid values are `true` or `false`. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366404) in GitLab 15.3.(default) | +| `auditor` **(PREMIUM ALL)** | No | User is an auditor. Valid values are `true` or `false`. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366404) in GitLab 15.3.(default) | | `avatar` | No | Image file for user's avatar | | `bio` | No | User's biography | | `can_create_group` | No | User can create groups - true or false | @@ -590,7 +596,7 @@ Parameters: | `email` | No | Email | | `extern_uid` | No | External UID | | `external` | No | Flags the user as external - true or false (default) | -| `extra_shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Additional compute minutes for this user. | +| `extra_shared_runners_minutes_limit` **(PREMIUM ALL)** | No | Can be set by administrators only. Additional compute minutes for this user. | | `group_id_for_saml` | No | ID of group where SAML has been configured | | `id` | Yes | ID of the user | | `linkedin` | No | LinkedIn | @@ -604,7 +610,7 @@ Parameters: | `pronouns` | No | Pronouns | | `provider` | No | External provider name | | `public_email` | No | Public email of the user (must be already verified) | -| `shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Maximum number of monthly compute minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0`. | +| `shared_runners_minutes_limit` **(PREMIUM ALL)** | No | Can be set by administrators only. Maximum number of monthly compute minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0`. | | `skip_reconfirmation` | No | Skip reconfirmation - true or false (default) | | `skype` | No | Skype ID | | `theme_id` | No | GitLab theme for the user (for more information, see the [user preference documentation](../user/profile/preferences.md#change-the-color-theme) for more information) | @@ -1265,7 +1271,7 @@ error occurs a `400 Bad Request` is returned with a message explaining the error ``` NOTE: -This also adds an audit event, as described in [audit instance events](../administration/audit_events.md#instance-events). **(PREMIUM)** +This also adds an audit event. **(PREMIUM ALL)** ## Delete SSH key for current user @@ -1871,7 +1877,8 @@ Example response: "id" : 2, "created_at" : "2017-03-17T17:18:09.283Z", "impersonation" : true, - "expires_at" : "2017-04-04" + "expires_at" : "2017-04-04", + "last_used_at": "2017-03-24T09:44:21.722Z" }, { "active" : false, @@ -1884,7 +1891,8 @@ Example response: "created_at" : "2017-03-17T17:19:28.697Z", "id" : 3, "impersonation" : true, - "expires_at" : "2017-04-14" + "expires_at" : "2017-04-14", + "last_used_at": "2017-03-24T09:44:21.722Z" } ] ``` @@ -2026,7 +2034,7 @@ POST /users/:user_id/impersonation_tokens | ------------ | ------- | -------- | --------------------------------------------------------------------------- | | `user_id` | integer | yes | ID of the user | | `name` | string | yes | Name of the impersonation token | -| `expires_at` | date | no | Expiration date of the impersonation token in ISO format (`YYYY-MM-DD`) | +| `expires_at` | date | yes | Expiration date of the impersonation token in ISO format (`YYYY-MM-DD`) | | `scopes` | array | yes | Array of scopes of the impersonation token (`api`, `read_user`) | ```shell diff --git a/doc/api/vulnerability_exports.md b/doc/api/vulnerability_exports.md index ab25ca9e511..0886db257dd 100644 --- a/doc/api/vulnerability_exports.md +++ b/doc/api/vulnerability_exports.md @@ -187,11 +187,11 @@ The response is `404 Not Found` if the vulnerability export is not finished yet Example response: ```csv -Group Name,Project Name,Tool,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers,Detected At,Location,Activity,Comments, -Gitlab.org,Defend,container_scanning,Trivy,resolved,CVE-2019-14697 in musl-utils-1.1.20-r4,"musl libc through 1.1.23 has an x87 floating-point stack adjustment imbalance, related to the math/i386/ directory. In some cases, use of this library could introduce out-of-bounds writes that are not present in an application's source code.",CVE-2019-14697 in musl-utils-1.1.20-r4,critical,CVE-2019-14697,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl-utils""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"2022-10-07 13:41:08 UTC|root|resolved|changed vulnerability status to resolved", -Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2019-19242 in sqlite-libs-3.26.0-r3,"SQLite 3.30.1 mishandles pExpr->y.pTab, as demonstrated by the TK_COLUMN case in sqlite3ExprCodeTarget in expr.c.",CVE-2019-19242 in sqlite-libs-3.26.0-r3,medium,CVE-2019-19242,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""sqlite-libs""}, ""version""=>""3.26.0-r3""}, ""operating_system""=>""alpine 3.9.2""}",true,"", -Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2020-28928 in musl-1.1.20-r4,"In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow).",CVE-2020-28928 in musl-1.1.20-r4,medium,CVE-2020-28928,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"", -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,Carefully crafted requests can cause shell escape sequences to be written to the terminal via Rack's Lint middleware and CommonLogger middleware. These escape sequences can be leveraged to possibly execute commands in the victim's terminal.,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,unknown,Gemfile.lock:rack:gemnasium:60b5a27f-4e4d-4ab4-8ae7-74b4b212e177,,Gemnasium-60b5a27f-4e4d-4ab4-8ae7-74b4b212e177; GHSA-wq4h-7r42-5hrr,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"", -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Denial of Service Vulnerability in Rack Multipart Parsing in rack,"Carefully crafted multipart POST requests can cause Rack's multipart parser to take much longer than expected, leading to a possible denial of service vulnerability. Impacted code will use Rack's multipart parser to parse multipart posts.",Denial of Service Vulnerability in Rack Multipart Parsing in rack,unknown,Gemfile.lock:rack:gemnasium:20daa17a-47b5-4f79-80c2-cd8f2db9805c,,Gemnasium-20daa17a-47b5-4f79-80c2-cd8f2db9805c; GHSA-hxqx-xwvh-44m2,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"", -Gitlab.org,Defend,sast,Brakeman,detected,Possible SQL injection,,Possible SQL injection,medium,e52f23a259cd489168b4313317ac94a3f13bffde57b9635171c1a44a9f329e9a,,"""Brakeman Warning Code 0""",2022-10-13 15:16:36 UTC,"{""file""=>""main.rb"", ""class""=>""User"", ""method""=>""index"", ""start_line""=>3}",false,"" +Group Name,Project Name,Tool,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers,Detected At,Location,Activity,Comments,Full Path +Gitlab.org,Defend,container_scanning,Trivy,resolved,CVE-2019-14697 in musl-utils-1.1.20-r4,"musl libc through 1.1.23 has an x87 floating-point stack adjustment imbalance, related to the math/i386/ directory. In some cases, use of this library could introduce out-of-bounds writes that are not present in an application's source code.",CVE-2019-14697 in musl-utils-1.1.20-r4,critical,CVE-2019-14697,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl-utils""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"2022-10-07 13:41:08 UTC|root|resolved|changed vulnerability status to resolved",group/project/1 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2019-19242 in sqlite-libs-3.26.0-r3,"SQLite 3.30.1 mishandles pExpr->y.pTab, as demonstrated by the TK_COLUMN case in sqlite3ExprCodeTarget in expr.c.",CVE-2019-19242 in sqlite-libs-3.26.0-r3,medium,CVE-2019-19242,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""sqlite-libs""}, ""version""=>""3.26.0-r3""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/2 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2020-28928 in musl-1.1.20-r4,"In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow).",CVE-2020-28928 in musl-1.1.20-r4,medium,CVE-2020-28928,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/3 +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,Carefully crafted requests can cause shell escape sequences to be written to the terminal via Rack's Lint middleware and CommonLogger middleware. These escape sequences can be leveraged to possibly execute commands in the victim's terminal.,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,unknown,Gemfile.lock:rack:gemnasium:60b5a27f-4e4d-4ab4-8ae7-74b4b212e177,,Gemnasium-60b5a27f-4e4d-4ab4-8ae7-74b4b212e177; GHSA-wq4h-7r42-5hrr,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"",group/project/4 +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Denial of Service Vulnerability in Rack Multipart Parsing in rack,"Carefully crafted multipart POST requests can cause Rack's multipart parser to take much longer than expected, leading to a possible denial of service vulnerability. Impacted code will use Rack's multipart parser to parse multipart posts.",Denial of Service Vulnerability in Rack Multipart Parsing in rack,unknown,Gemfile.lock:rack:gemnasium:20daa17a-47b5-4f79-80c2-cd8f2db9805c,,Gemnasium-20daa17a-47b5-4f79-80c2-cd8f2db9805c; GHSA-hxqx-xwvh-44m2,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"",group/project/5 +Gitlab.org,Defend,sast,Brakeman,detected,Possible SQL injection,,Possible SQL injection,medium,e52f23a259cd489168b4313317ac94a3f13bffde57b9635171c1a44a9f329e9a,,"""Brakeman Warning Code 0""",2022-10-13 15:16:36 UTC,"{""file""=>""main.rb"", ""class""=>""User"", ""method""=>""index"", ""start_line""=>3}",false,"",group/project/6 ``` diff --git a/doc/api/vulnerability_findings.md b/doc/api/vulnerability_findings.md index a031e07fddf..05ae42d9100 100644 --- a/doc/api/vulnerability_findings.md +++ b/doc/api/vulnerability_findings.md @@ -142,63 +142,81 @@ To prepare for the [upcoming deprecation](https://gitlab.com/groups/gitlab-org/- the Vulnerability Findings REST API endpoint, use the examples below to perform the equivalent operations with the GraphQL API. -### GraphQL - Project vulnerabilities +### GraphQL - Project vulnerability findings -Use [`Project.vulnerabilities`](graphql/reference/index.md#projectvulnerabilities). +Use [`Pipeline.securityReportFindings`](graphql/reference/index.md#pipelinesecurityreportfindings). ```graphql -{ - project(fullPath: "root/security-reports") { - vulnerabilities { - nodes{ - id - reportType - title - severity - scanner { - externalId - name - vendor - } - identifiers { - externalType - externalId - name - url - } - falsePositive - project { - id - name - fullPath - } - description - links { - name - url - } - location { - ... on - VulnerabilityLocationSast { - file - startLine - endLine - vulnerableClass - vulnerableMethod - blobPath - } - } - details { - ... on - VulnerabilityDetailCode { +query VulnerabilityFindings { + project(fullPath: "gitlab-examples/security/security-reports") { + pipelines(first:1) { + nodes { + securityReportFindings(first:1) { + nodes { + title + severity + state + scanner { + externalId + name + vendor + } + identifiers { + externalType + externalId + name + url + } + uuid + falsePositive description - fieldName - lang - name - value + location { + ... on VulnerabilityLocationSast { + file + startLine + endLine + vulnerableClass + vulnerableMethod + blobPath + } + + ... on VulnerabilityLocationContainerScanning { + dependency { + package { + name + } + version + } + image + operatingSystem + } + + ... on VulnerabilityLocationDependencyScanning { + file + blobPath + dependency { + version + } + } + } + remediations { + diff + summary + } + solution + evidence { + request { + body + headers { + name + value + } + method + url + } + } } } - state } } } @@ -211,50 +229,56 @@ Example response: { "data": { "project": { - "vulnerabilities": { + "pipelines": { "nodes": [ { - "id": "gid://gitlab/Vulnerability/236", - "reportType": "SAST", - "title": "Generic Object Injection Sink", - "severity": "CRITICAL", - "scanner": { - "externalId": "eslint", - "name": "ESLint", - "vendor": "GitLab" - }, - "identifiers": [ - { - "externalType": "eslint_rule_id", - "externalId": "security/detect-object-injection", - "name": "ESLint rule ID security/detect-object-injection", - "url": "https://github.com/nodesecurity/eslint-plugin-security#detect-object-injection" - }, - { - "externalType": "cwe", - "externalId": "94", - "name": "CWE-94", - "url": "https://cwe.mitre.org/data/definitions/94.html" - } - ], - "falsePositive": false, - "project": { - "id": "gid://gitlab/Project/20", - "name": "Security Reports", - "fullPath": "root/security-reports" - }, - "description": "Bracket object notation with user input is present, this might allow an attacker to access all properties of the object and even it's prototype, leading to possible code execution.", - "links": [], - "location": { - "file": "src/js/main.js", - "startLine": "28", - "endLine": "28", - "vulnerableClass": null, - "vulnerableMethod": null, - "blobPath": "/root/security-reports/-/blob/91031428a5b5dbb81e8d889738b1875c1bfea787/src/js/main.js" - }, - "details": [], - "state": "DETECTED" + "securityReportFindings": { + "nodes": [ + { + "title": "Deserialization of Untrusted Data", + "severity": "CRITICAL", + "state": "CONFIRMED", + "scanner": { + "externalId": "gemnasium", + "name": "Gemnasium", + "vendor": "GitLab" + }, + "identifiers": [ + { + "externalType": "gemnasium", + "externalId": "b60c2d6b-9083-4a97-a1b2-f7dc79bff74c", + "name": "Gemnasium-b60c2d6b-9083-4a97-a1b2-f7dc79bff74c", + "url": "https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/gem/activerecord/CVE-2022-32224.yml" + }, + { + "externalType": "cve", + "externalId": "CVE-2022-32224", + "name": "CVE-2022-32224", + "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-32224" + }, + { + "externalType": "ghsa", + "externalId": "GHSA-3hhc-qp5v-9p2j", + "name": "GHSA-3hhc-qp5v-9p2j", + "url": "https://github.com/advisories/GHSA-3hhc-qp5v-9p2j" + } + ], + "uuid": "c9e40395-72cd-54f5-962f-e1d52c0dffab", + "falsePositive": false, + "description": "A possible escalation to RCE vulnerability exists when using YAML serialized columns in Active Record < 7.0.3.1, <6.1.6.1, <6.0.5.1 and <5.2.8.1 which could allow an attacker, that can manipulate data in the database (via means like SQL injection), the ability to escalate to an RCE.", + "location": { + "file": "dependency-scanning-files/Gemfile.lock", + "blobPath": null, + "dependency": { + "version": "5.0.0" + } + }, + "remediations": [], + "solution": "Upgrade to versions 5.2.8.1, 6.0.5.1, 6.1.6.1, 7.0.3.1 or above.", + "evidence": null + } + ] + } } ] } diff --git a/doc/api/wikis.md b/doc/api/wikis.md index 3a5b8b075f5..c33d66a14e4 100644 --- a/doc/api/wikis.md +++ b/doc/api/wikis.md @@ -203,7 +203,7 @@ Example response: { "file_name" : "dk.png", "file_path" : "uploads/6a061c4cf9f1c28cb22c384b4b8d4e3c/dk.png", - "branch" : "master", + "branch" : "main", "link" : { "url" : "uploads/6a061c4cf9f1c28cb22c384b4b8d4e3c/dk.png", "markdown" : "" diff --git a/doc/architecture/blueprints/ai_gateway/index.md b/doc/architecture/blueprints/ai_gateway/index.md index c9947723739..08cd8b691d4 100644 --- a/doc/architecture/blueprints/ai_gateway/index.md +++ b/doc/architecture/blueprints/ai_gateway/index.md @@ -46,7 +46,7 @@ connect to 3rd party providers. ## Language: Python The AI-Gateway was originally started as the "model-gateway" that -handled requests from IDEs to provide code suggestions. It was written +handled requests from IDEs to provide Code Suggestions. It was written in Python. Python is an object oriented language that is familiar enough for @@ -96,7 +96,7 @@ GitLab instances, they differ on these items: | + Strict protocol definition that is easier to evolve versionless | - No strict schema, so the implementation needs to take good care of supporting multiple versions | | + A new Ruby-gRPC server for vscode: likely faster because we can limit dependencies to load ([modular monolith](https://gitlab.com/gitlab-org/gitlab/-/issues/365293)) | - Existing Grape API for vscode: meaning slow boot time and unneeded resources loaded | | + Bi-directional streaming | - Straight forward way to stream requests and responses (could still be added) | -| - A new Python-gRPC server: we don't have experience running gRPC-Python servers | + Existing Python fastapi server, already running for code suggestions to extend | +| - A new Python-gRPC server: we don't have experience running gRPC-Python servers | + Existing Python fastapi server, already running for Code Suggestions to extend | | - Hard to pass on unknown messages from vscode through GitLab to ai-gateway | + Easier support for newer vscode + newer ai-gatway, through old GitLab instance | | - Unknown support for gRPC in other clients (vscode, jetbrains, other editors) | + Support in all external clients | | - Possible protocol mismatch (VSCode --REST--> Rails --gRPC--> AI gateway) | + Same protocol across the stack | @@ -116,7 +116,7 @@ with a stable API over the [provider API we expose](#exposing-ai-providers) as a direct proxy. Some features will have specific endpoints, while others can share -endpoints. For example, code suggestions or chat could have their own +endpoints. For example, Code Suggestions or chat could have their own endpoint, while several features that summarize issues or merge requests could use the same endpoint but make the distinction on what information is provided in the payload. @@ -215,9 +215,9 @@ what is in the payload. To document and validate the content of `payload` we can specify their format using [JSON-schema](https://json-schema.org/). -#### Example feature: code suggestions +#### Example feature: Code Suggestions -For example, a rough code suggestions service could look like this: +For example, a rough Code Suggestions service could look like this: ```plaintext POST /internal/code-suggestions/completions @@ -281,7 +281,7 @@ The `metadata` field contains information that could be used in a telemetry endpoint on the AI-gateway where we could count suggestion-acceptance rates among other things. -The way we will receive telemetry for code suggestions is being +The way we will receive telemetry for Code Suggestions is being discussed in [#415745](https://gitlab.com/gitlab-org/gitlab/-/issues/415745). We will try to come up with an architecture for all AI-related features. @@ -336,7 +336,7 @@ purpose API endpoint before we make the feature [generally available](../../../p for self-managed installations. This makes it easier for us to support features long-term even if the landscape of AI providers change. -The [Experimental REST API](../../../development/ai_features.md#experimental-rest-api) +The [Experimental REST API](../../../development/ai_features/index.md#experimental-rest-api) available to GitLab team members should also use this proxy in the short term. In the longer term, we should provide developers access to a separate proxy that allows them to use GitLab owned authentication diff --git a/doc/architecture/blueprints/cells/cells-feature-admin-area.md b/doc/architecture/blueprints/cells/cells-feature-admin-area.md index a9cd170b2a7..3f23e56c3af 100644 --- a/doc/architecture/blueprints/cells/cells-feature-admin-area.md +++ b/doc/architecture/blueprints/cells/cells-feature-admin-area.md @@ -1,81 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Admin Area' +redirect_to: 'impacted_features/admin-area.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Admin Area - -In our Cells architecture proposal we plan to share all admin related tables in GitLab. -This allows for simpler management of all Cells in one interface and reduces the risk of settings diverging in different Cells. -This introduces challenges with Admin Area pages that allow you to manage data that will be spread across all Cells. - -## 1. Definition - -There are consequences for Admin Area pages that contain data that span "the whole instance" as the Admin Area pages may be served by any Cell or possibly just one Cell. -There are already many parts of the Admin Area that will have data that span many Cells. -For example lists of all Groups, Projects, Topics, Jobs, Analytics, Applications and more. -There are also administrative monitoring capabilities in the Admin Area that will span many Cells such as the "Background Jobs" and "Background Migrations" pages. - -## 2. Data flow - -## 3. Proposal - -We will need to decide how to handle these exceptions with a few possible -options: - -1. Move all these pages out into a dedicated per-Cell admin section. Probably - the URL will need to be routable to a single Cell like `/cells/<cell_id>/admin`, - then we can display these data per Cell. These pages will be distinct from - other Admin Area pages which control settings that are shared across all Cells. We - will also need to consider how this impacts self-managed customers and - whether, or not, this should be visible for single-Cell instances of GitLab. -1. Build some aggregation interfaces for this data so that it can be fetched - from all Cells and presented in a single UI. This may be beneficial to an - administrator that needs to see and filter all data at a glance, especially - when they don't know which Cell the data is on. The downside, however, is - that building this kind of aggregation is very tricky when all Cells are - designed to be totally independent, and it does also enforce stricter - requirements on compatibility between Cells. - -The following overview describes at what level each feature contained in the current Admin Area will be managed: - -| Feature | Cluster | Cell | Organization | -| --- | --- | --- | --- | -| Abuse reports | | | | -| Analytics | | | | -| Applications | | | | -| Deploy keys | | | | -| Labels | | | | -| Messages | ✓ | | | -| Monitoring | | ✓ | | -| Subscription | | | | -| System hooks | | | | -| Overview | | | | -| Settings - General | ✓ | | | -| Settings - Integrations | ✓ | | | -| Settings - Repository | ✓ | | | -| Settings - CI/CD (1) | ✓ | ✓ | | -| Settings - Reporting | ✓ | | | -| Settings - Metrics | ✓ | | | -| Settings - Service usage data | | ✓ | | -| Settings - Network | ✓ | | | -| Settings - Appearance | ✓ | | | -| Settings - Preferences | ✓ | | | - -(1) Depending on the specific setting, some will be managed at the cluster-level, and some at the Cell-level. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/admin-area.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md index 37347cf836d..050b3a922b1 100644 --- a/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md +++ b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Agent for Kubernetes' +redirect_to: 'impacted_features/agent-for-kubernetes.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Agent for Kubernetes - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/agent-for-kubernetes.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-backups.md b/doc/architecture/blueprints/cells/cells-feature-backups.md index 3d20d6e2caa..a0c38171ce6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-backups.md +++ b/doc/architecture/blueprints/cells/cells-feature-backups.md @@ -1,53 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Backups' +redirect_to: 'impacted_features/backups.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Backups - -Each Cell will take its own backups, and consequently have its own isolated backup/restore procedure. - -## 1. Definition - -GitLab backup takes a backup of the PostgreSQL database used by the application, and also Git repository data. - -## 2. Data flow - -Each Cell has a number of application databases to back up (for example, `main`, and `ci`). -Additionally, there may be cluster-wide metadata tables (for example, `users` table) which is directly accessible via PostgreSQL. - -## 3. Proposal - -### 3.1. Cluster-wide metadata - -It is currently unknown how cluster-wide metadata tables will be accessible. -We may choose to have cluster-wide metadata tables backed up separately, or have each Cell back up its copy of cluster-wide metadata tables. - -### 3.2 Consistency - -#### 3.2.1 Take backups independently - -As each Cell will communicate with each other via API, and there will be no joins to the `users` table, it should be acceptable for each Cell to take a backup independently of each other. - -#### 3.2.2 Enforce snapshots - -We can require that each Cell take a snapshot for the PostgreSQL databases at around the same time to allow for a consistent enough backup. - -## 4. Evaluation - -As the number of Cells increases, it will likely not be feasible to take a snapshot at the same time for all Cells. -Hence taking backups independently is the better option. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/backups.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-ci-runners.md b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md index 4e7cea5bfd5..a14f2a47237 100644 --- a/doc/architecture/blueprints/cells/cells-feature-ci-runners.md +++ b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md @@ -1,144 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: CI Runners' +redirect_to: 'impacted_features/ci-runners.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: CI Runners - -GitLab executes CI jobs via [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), very often managed by customers in their infrastructure. -All CI jobs created as part of the CI pipeline are run in the context of a Project. -This poses a challenge how to manage GitLab Runners. - -## 1. Definition - -There are 3 different types of runners: - -- Instance-wide: Runners that are registered globally with specific tags (selection criteria) -- Group runners: Runners that execute jobs from a given top-level Group or Projects in that Group -- Project runners: Runners that execute jobs from one Projects or many Projects: some runners might - have Projects assigned from Projects in different top-level Groups. - -This, alongside with the existing data structure where `ci_runners` is a table describing all types of runners, poses a challenge as to how the `ci_runners` should be managed in a Cells environment. - -## 2. Data flow - -GitLab runners use a set of globally scoped endpoints to: - -- Register a new runner via registration token `https://gitlab.com/api/v4/runners` - ([subject for removal](../runner_tokens/index.md)) (`registration token`) -- Create a new runner in the context of a user `https://gitlab.com/api/v4/user/runners` (`runner token`) -- Request jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) -- Upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) -- Upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) -- Download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) - -Currently three types of authentication tokens are used: - -- Runner registration token ([subject for removal](../runner_tokens/index.md)) -- Runner token representing a registered runner in a system with specific configuration (`tags`, `locked`, etc.) -- Build token representing an ephemeral token giving limited access to updating a specific job, uploading artifacts, downloading dependent artifacts, downloading and uploading container registry images - -Each of those endpoints receive an authentication token via header (`JOB-TOKEN` for `/trace`) or body parameter (`token` all other endpoints). - -Since the CI pipeline would be created in the context of a specific Cell, it would be required that pick of a build would have to be processed by that particular Cell. -This requires that build picking depending on a solution would have to be either: - -- Routed to the correct Cell for the first time -- Be two-phased: Request build from global pool, claim build on a specific Cell using a Cell specific URL - -## 3. Proposal - -### 3.1. Authentication tokens - -Even though the paths for CI runners are not routable, they can be made routable with these two possible solutions: - -- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with - a ticketing mechanism (based on `X-GitLab-Last-Update` header). When the runner first - starts, it sends a request to GitLab to which GitLab responds with either a build to pick - by runner. This value is completely controlled by GitLab. This allows GitLab - to use JWT or any other means to encode a `cell` identifier that could be easily - decodable by Router. -- The majority of communication (in terms of volume) is using `build token`, making it - the easiest target to change since GitLab is the sole owner of the token that the runner later - uses for a specific job. There were prior discussions about not storing the `build token` - but rather using a `JWT` token with defined scopes. Such a token could encode the `cell` - to which the Router could route all requests. - -### 3.2. Request body - -- The most used endpoints pass the authentication token in the request body. It might be desired - to use HTTP headers as an easier way to access this information by Router without - a need to proxy requests. - -### 3.3. Instance-wide are Cell-local - -We can pick a design where all runners are always registered and local to a given Cell: - -- Each Cell has its own set of instance-wide runners that are updated at its own pace -- The Project runners can only be linked to Projects from the same Organization, creating strong isolation. -- In this model the `ci_runners` table is local to the Cell. -- In this model we would require the above endpoints to be scoped to a Cell in some way, or be made routable. It might be via prefixing them, adding additional Cell parameters, or providing much more robust ways to decode runner tokens and match it to a Cell. -- If a routable token is used, we could move away from cryptographic random stored in database to rather prefer to use JWT tokens. -- The Admin Area showing registered runners would have to be scoped to a Cell. - -This model might be desired because it provides strong isolation guarantees. -This model does significantly increase maintenance overhead because each Cell is managed separately. -This model may require adjustments to the runner tags feature so that Projects have a consistent runner experience across Cells. - -### 3.4. Instance-wide are cluster-wide - -Contrary to the proposal where all runners are Cell-local, we can consider that runners -are global, or just instance-wide runners are global. - -However, this requires significant overhaul of the system and we would have to change the following aspects: - -- The `ci_runners` table would likely have to be decomposed into `ci_instance_runners`, ... -- All interfaces would have to be adopted to use the correct table. -- Build queuing would have to be reworked to be two-phased where each Cell would know of all pending and running builds, but the actual claim of a build would happen against a Cell containing data. -- It is likely that `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables, increasing the likelihood of creating hotspots in a system related to CI queueing. - -This model is complex to implement from an engineering perspective. -Some data are shared between Cells. -It creates hotspots/scalability issues in a system that might impact the experience of Organizations on other Cells, for instance during abuse. - -### 3.5. GitLab CI Daemon - -Another potential solution to explore is to have a dedicated service responsible for builds queueing, owning its database and working in a model of either sharded or Cell-ed service. -There were prior discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). - -If the service is sharded: - -- Depending on the model, if runners are cluster-wide or Cell-local, this service would have to fetch data from all Cells. -- If the sharded service is used we could adapt a model of sharing a database containing `ci_pending_builds/ci_running_builds` with the service. -- If the sharded service is used we could consider a push model where each Cell pushes to CI/CD Daemon builds that should be picked by runner. -- The sharded service would be aware which Cell is responsible for processing the given build and could route processing requests to the designated Cell. - -If the service is Cell-ed: - -- All expectations of routable endpoints are still valid. - -In general usage of CI Daemon does not help significantly with the stated problem. -However, this offers a few upsides related to more efficient processing and decoupling model: push model and it opens a way to offer stateful communication with GitLab runners (ex. gRPC or Websockets). - -## 4. Evaluation - -Considering all options it appears that the most promising solution is to: - -- Use [Instance-wide are Cell-local](#33-instance-wide-are-cell-local) -- Refine endpoints to have routable identities (either via specific paths, or better tokens) - -Another potential upside is to get rid of `ci_builds.token` and rather use a `JWT token` that can much better and easier encode a wider set of scopes allowed by CI runner. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/ci-runners.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-container-registry.md b/doc/architecture/blueprints/cells/cells-feature-container-registry.md index 25af65a8700..d9ff6da7f62 100644 --- a/doc/architecture/blueprints/cells/cells-feature-container-registry.md +++ b/doc/architecture/blueprints/cells/cells-feature-container-registry.md @@ -1,114 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Container Registry' +redirect_to: 'impacted_features/container-registry.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Container Registry - -GitLab [Container Registry](../../../user/packages/container_registry/index.md) is a feature allowing to store Docker container images in GitLab. - -## 1. Definition - -GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis and Object Storage dependencies. -Right now there's undergoing work to introduce [Container Registry Metadata](../container_registry_metadata_database/index.md) to optimize data storage and image retention policies of Container Registry. - -GitLab Container Registry is serving as a container for stored data, but on its own does not authenticate `docker login`. -The `docker login` is executed with user credentials (can be `personal access token`) or CI build credentials (ephemeral `ci_builds.token`). - -Container Registry uses data deduplication. -It means that the same blob (image layer) that is shared between many Projects is stored only once. -Each layer is hashed by `sha256`. - -The `docker login` does request a JWT time-limited authentication token that is signed by GitLab, but validated by Container Registry service. -The JWT token does store all authorized scopes (`container repository images`) and operation types (`push` or `pull`). -A single JWT authentication token can have many authorized scopes. -This allows Container Registry and client to mount existing blobs from other scopes. -GitLab responds only with authorized scopes. -Then it is up to GitLab Container Registry to validate if the given operation can be performed. - -The GitLab.com pages are always scoped to a Project. -Each Project can have many container registry images attached. - -Currently, on GitLab.com the actual registry service is served via `https://registry.gitlab.com`. - -The main identifiable problems are: - -- The authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com. -- The `https://registry.gitlab.com` that is run by an external service and uses its own data store. -- Data deduplication. The Cells architecture with registry run in a Cell would reduce efficiency of data storage. - -## 2. Data flow - -### 2.1. Authorization request that is send by `docker login` - -```shell -curl \ - --user "username:password" \ - "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" -``` - -Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. - -```json -{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} -``` - -### 2.2. Docker client fetching tags - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list - -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 -``` - -### 2.3. Docker client fetching blobs and manifests - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 -``` - -## 3. Proposal - -### 3.1. Shard Container Registry separately to Cells architecture - -Due to its extensive and in general highly scalable horizontal architecture it should be evaluated if the GitLab Container Registry should be run not in Cell, but in a Cluster and be scaled independently. -This might be easier, but would definitely not offer the same amount of data isolation. - -### 3.2. Run Container Registry within a Cell - -It appears that except `/jwt/auth` which would likely have to be processed by Router (to decode `scope`) the Container Registry could be run as a local service of a Cell. -The actual data at least in case of GitLab.com is not forwarded via registry, but rather served directly from Object Storage / CDN. - -Its design encodes container repository image in a URL that is easily routable. -It appears that we could re-use the same stateless Router service in front of Container Registry to serve manifests and blobs redirect. - -The only downside is increased complexity of managing standalone registry for each Cell, but this might be desired approach. - -## 4. Evaluation - -There do not seem to be any theoretical problems with running GitLab Container Registry in a Cell. -It seems that the service can be easily made routable to work well. -The practical complexities are around managing a complex service from an infrastructure side. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/container-registry.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md index 8e144386908..a87e4ba3391 100644 --- a/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md +++ b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md @@ -1,106 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Contributions: Forks' +redirect_to: 'impacted_features/contributions-forks.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Contributions: Forks - -The [Forking workflow](../../../user/project/repository/forking_workflow.md) allows users to copy existing Project sources into their own namespace of choice (Personal or Group). - -## 1. Definition - -The [Forking workflow](../../../user/project/repository/forking_workflow.md) is a common workflow with various usage patterns: - -- It allows users to contribute back to upstream Project. -- It persists repositories into their Personal Namespace. -- Users can copy to make changes and release as modified Project. - -Forks allow users not having write access to a parent Project to make changes. -The forking workflow is especially important for the open source community to contribute back to public Projects. -However, it is equally important in some companies that prefer a strong split of responsibilities and tighter access control. -The access to a Project is restricted to a designated list of developers. - -Forks enable: - -- Tighter control of who can modify the upstream Project. -- Split of responsibilities: Parent Project might use CI configuration connecting to production systems. -- To run CI pipelines in the context of a fork in a much more restrictive environment. -- To consider all forks to be unvetted which reduces risks of leaking secrets, or any other information tied to the Project. - -The forking model is problematic in a Cells architecture for the following reasons: - -- Forks are clones of existing repositories. Forks could be created across different Organizations, Cells and Gitaly shards. -- Users can create merge requests and contribute back to an upstream Project. This upstream Project might in a different Organization and Cell. -- The merge request CI pipeline is executed in the context of the source Project, but presented in the context of the target Project. - -## 2. Data flow - -## 3. Proposals - -### 3.1. Intra-Cluster forks - -This proposal implements forks as intra-Cluster forks where communication is done via API between all trusted Cells of a cluster: - -- Forks are created always in the context of a user's choice of Group. -- Forks are isolated from the Organization. -- Organization or Group owner could disable forking across Organizations, or forking in general. -- A merge request is created in the context of the target Project, referencing the external Project on another Cell. -- To target Project the merge reference is transferred that is used for presenting information in context of the target Project. -- CI pipeline is fetched in the context of the source Project as it is today, the result is fetched into the merge request of the target Project. -- The Cell holding the target Project internally uses GraphQL to fetch the status of the source Project and includes in context of the information for merge request. - -Upsides: - -- All existing forks continue to work as they are, as they are treated as intra-Cluster forks. - -Downsides: - -- The purpose of Organizations is to provide strong isolation between Organizations. Allowing to fork across does break security boundaries. -- However, this is no different to the ability of users today to clone a repository to a local computer and push it to any repository of choice. -- Access control of source Project can be lower than those of target Project. Today, the system requires that in order to contribute back, the access level needs to be the same for fork and upstream. - -### 3.2. Forks are created in a Personal Namespace of the current Organization - -Instead of creating Projects across Organizations, forks are created in a user's Personal Namespace tied to the Organization. Example: - -- Each user that is part of an Organization receives their Personal Namespace. For example for `GitLab Inc.` it could be `gitlab.com/organization/gitlab-inc/@ayufan`. -- The user has to fork into their own Personal Namespace of the Organization. -- The user has as many Personal Namespaces as Organizations they belongs to. -- The Personal Namespace behaves similar to the currently offered Personal Namespace. -- The user can manage and create Projects within a Personal Namespace. -- The Organization can prevent or disable usage of Personal Namespaces, disallowing forks. -- All current forks are migrated into the Personal Namespace of user in an Organization. -- All forks are part of the Organization. -- Forks are not federated features. -- The Personal Namespace and forked Project do not share configuration with the parent Project. - -### 3.3. Forks are created as internal Projects under current Projects - -Instead of creating Projects across Organizations, forks are attachments to existing Projects. -Each user forking a Project receives their unique Project. Example: - -- For Project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. -- Forks are created in the context of the current Organization, they do not cross Organization boundaries and are managed by the Organization. -- Tied to the user (or any other user-provided name of the fork). -- Forks are not federated features. - -Downsides: - -- Does not answer how to handle and migrate all existing forks. -- Might share current Group/Project settings, which could be breaking some security boundaries. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/contributions-forks.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-data-migration.md b/doc/architecture/blueprints/cells/cells-feature-data-migration.md index 9ff661ddf68..5638bb29dc5 100644 --- a/doc/architecture/blueprints/cells/cells-feature-data-migration.md +++ b/doc/architecture/blueprints/cells/cells-feature-data-migration.md @@ -1,99 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Data migration' +redirect_to: 'impacted_features/data-migration.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Data migration - -It is essential for a Cells architecture to provide a way to migrate data out of big Cells into smaller ones. -This document describes various approaches to provide this type of split. - -We also need to handle cases where data is already violating the expected isolation constraints of Cells, for example references cannot span multiple Organizations. -We know that existing features like linked issues allowed users to link issues across any Projects regardless of their hierarchy. -There are many similar features. -All of this data will need to be migrated in some way before it can be split across different Cells. -This may mean some data needs to be deleted, or the feature needs to be changed and modelled slightly differently before we can properly split or migrate Organizations between Cells. - -Having schema deviations across different Cells, which is a necessary consequence of different databases, will also impact our ability to migrate data between Cells. -Different schemas impact our ability to reliably replicate data across Cells and especially impact our ability to validate that the data is correctly replicated. -It might force us to only be able to move data between Cells when the schemas are all in sync (slowing down deployments and the rebalancing process) or possibly only migrate from newer to older schemas which would be complex. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -### 3.1. Split large Cells - -A single Cell can only be divided into many Cells. -This is based on the principle that it is easier to create an exact clone of an existing Cell in many replicas out of which some will be made authoritative once migrated. -Keeping those replicas up-to-date with Cell 0 is also much easier due to pre-existing replication solutions that can replicate the whole systems: Geo, PostgreSQL physical replication, etc. - -1. All data of an Organization needs to not be divided across many Cells. -1. Split should be doable online. -1. New Cells cannot contain pre-existing data. -1. N Cells contain exact replica of Cell 0. -1. The data of Cell 0 is live replicated to as many Cells it needs to be split. -1. Once consensus is achieved between Cell 0 and N-Cells, the Organizations to be migrated away are marked as read-only cluster-wide. -1. The `routes` is updated on for all Organizations to be split to indicate an authoritative Cell holding the most recent data, like `gitlab-org` on `cell-100`. -1. The data for `gitlab-org` on Cell 0, and on other non-authoritative N-Cells are dormant and will be removed in the future. -1. All accesses to `gitlab-org` on a given Cell are validated about `cell_id` of `routes` to ensure that given Cell is authoritative to handle the data. - -#### More challenges of this proposal - -1. There is no streaming replication capability for Elasticsearch, but you could - snapshot the whole Elasticsearch index and recreate, but this takes hours. - It could be handled by pausing Elasticsearch indexing on the initial Cell during - the migration as indexing downtime is not a big issue, but this still needs - to be coordinated with the migration process. -1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other - new data stores snapshots in an online system would likely lead to gaps - without a long downtime. You need to choose a sync point and at the sync - point you need to stop writes to perform the migration. The more data stores - there are to migrate at the same time the longer the write downtime for the - failover. We would also need to find a reliable place in the application to - actually block updates to all these systems with a high degree of - confidence. In the past we've only been confident by shutting down all Rails - services because any Rails process could write directly to any of these at - any time due to async workloads or other surprising code paths. -1. How to efficiently delete all the orphaned data. Locating all `ci_builds` - associated with half the Organizations would be very expensive if we have to - do joins. We haven't yet determined if we'd want to store an `organization_id` - column on every table, but this is the kind of thing it would be helpful for. - -### 3.2. Migrate Organization from an existing Cell - -This is different to split, as we intend to perform logical and selective replication of data belonging to a single Organization. -Today this type of selective replication is only implemented by Gitaly where we can migrate Git repository from a single Gitaly node to another with minimal downtime. - -In this model we would require identifying all resources belonging to a given Organization: database rows, object storage files, Git repositories, etc. and selectively copy them over to another (likely) existing Cell importing data into it. -Ideally ensuring that we can perform logical replication live of all changed data, but change similarly to split which Cell is authoritative for this Organization. - -1. It is hard to identify all resources belonging to an Organization. -1. It requires either downtime for the Organization or a robust system to identify live changes made. -1. It likely will require a full database structure analysis (more robust than Project import/export) to perform selective PostgreSQL logical replication. - -#### More challenges of this proposal - -1. Logical replication is still not performant enough to keep up with our - scale. Even if we could use logical replication we still don't have an - efficient way to filter data related to a single Organization without - joining all the way to the `organizations` table which will slow down - logical replication dramatically. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/data-migration.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-database-sequences.md b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md index 2aeaaed7d64..9b426ed80a4 100644 --- a/doc/architecture/blueprints/cells/cells-feature-database-sequences.md +++ b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md @@ -1,74 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Database Sequences' +redirect_to: 'impacted_features/database-sequences.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Database Sequences - -GitLab today ensures that every database row create has a unique ID, allowing to access a merge request, CI Job or Project by a known global ID. -Cells will use many distinct and not connected databases, each of them having a separate ID for most entities. -At a minimum, any ID referenced between a Cell and the shared schema will need to be unique across the cluster to avoid ambiguous references. -Further to required global IDs, it might also be desirable to retain globally unique IDs for all database rows to allow migrating resources between Cells in the future. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -These are some preliminary ideas how we can retain unique IDs across the system. - -### 3.1. UUID - -Instead of using incremental sequences, use UUID (128 bit) that is stored in the database. - -- This might break existing IDs and requires adding a UUID column for all existing tables. -- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. - -### 3.2. Use Cell index encoded in ID - -Because a significant number of tables already use 64 bit ID numbers we could use MSB to encode the Cell ID: - -- This might limit the amount of Cells that can be enabled in a system, as we might decide to only allocate 1024 possible Cell numbers. -- This would make it possible to migrate IDs between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a lookup table. -- This requires updating all IDs to 32 bits. - -### 3.3. Allocate sequence ranges from central place - -Each Cell might receive its own range of sequences as they are consumed from a centrally managed place. -Once a Cell consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. -Ranges would be tracked to provide a faster lookup table if a random access pattern is required. - -- This might make IDs migratable between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a much more robust lookup table as we could be breaking previously assigned sequence ranges. -- This does not require updating all IDs to 64 bits. -- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server. -- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. - -### 3.4. Define only some tables to require unique IDs - -Maybe it is acceptable only for some tables to have a globally unique IDs. It could be Projects, Groups and other top-level entities. -All other tables like `merge_requests` would only offer a Cell-local ID, but when referenced outside it would rather use an IID (an ID that is monotonic in context of a given resource, like a Project). - -- This makes the ID 10000 for `merge_requests` be present on all Cells, which might be sometimes confusing regarding the uniqueness of the resource. -- This might make random access by ID (if ever needed) impossible without using a composite key, like: `project_id+merge_request_id`. -- This would require us to implement a transformation/generation of new ID if we need to migrate records to another Cell. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. -- If IDs need to change when moving between Cells this means that any links to records by ID would no longer work even if those links included the `project_id`. -- If we plan to allow these IDs to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/database-sequences.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-explore.md b/doc/architecture/blueprints/cells/cells-feature-explore.md index 4eab99d63e7..95924e3d1e8 100644 --- a/doc/architecture/blueprints/cells/cells-feature-explore.md +++ b/doc/architecture/blueprints/cells/cells-feature-explore.md @@ -1,71 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Explore' +redirect_to: 'impacted_features/explore.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Explore - -Explore may not play a critical role in GitLab as it functions today, but GitLab today is not isolated. It is the isolation that makes Explore or some viable replacement necessary. - -The existing Group and Project Explore will initially be scoped to an Organization. However, there is a need for a global Explore that spans across Organizations to support the discoverability of public Groups and Projects, in particular in the context of discovering open source Projects. See user feedback [here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582#note_1458298192) and [here](https://gitlab.com/gitlab-org/gitlab/-/issues/418228#note_1470045468). - -## 1. Definition - -The Explore functionality helps users in discovering Groups and Projects. Unauthenticated Users are only able to explore public Groups and Projects, authenticated Users can see all the Groups and Projects that they have access to, including private and internal Groups and Projects. - -## 2. Data flow - -## 3. Proposal - -The Explore feature problem falls under the broader umbrella of solving inter-Cell communication. [This topic warrants deeper research](index.md#can-different-cells-communicate-with-each-other). - -Below are possible directions for further investigation. - -### 3.1. Read only table mirror - -- Create a `shared_projects` table in the shared cluster-wide database. -- The model for this table is read-only. No inserts/updates/deletes are allowed. -- The table is filled with data (or a subset of data) from the Projects Cell-local table. - - The write model Project (which is Cell-local) writes to the local database. We will primarily use this model for anything Cell-local. - - This data is synchronized with `shared_projects` via a background job any time something changes. - - The data in `shared_projects` is stored normalized, so that all the information necessary to display the Project Explore is there. -- The Project Explore (as of today) is part of an instance-wide functionality, since it's not namespaced to any organizations/groups. - - This section will read data using the read model for `shared_projects`. -- Once the user clicks on a Project, they are redirected to the Cell containing the Organization. - -Downsides: - -- Need to have an explicit pattern to access instance-wide data. This however may be useful for admin functionalities too. -- The Project Explore may not be as rich in features as it is today (various filtering options, role you have on that Project, etc.). -- Extra complexity in managing CQRS. - -### 3.2 Explore scoped to an Organization - -The Project Explore and Group Explore are scoped to an Organization. - -Downsides: - -- No global discoverability of Groups and Projects. - -## 4. Evaluation - -The existing Group and Project Explore will initially be scoped to an Organization. Considering the [current usage of the Explore feature](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521), we deem this acceptable. Since all existing Users, Groups and Projects will initially be part of the default Organization, Groups and Projects will remain explorable and accessible as they are today. Only once existing Groups and Projects are moved out of the default Organization into different Organizations will this become a noticeable problem. Solutions to mitigate this are discussed in [issue #418228](https://gitlab.com/gitlab-org/gitlab/-/issues/418228). Ultimately, Explore could be replaced with a better search experience altogether. - -## 4.1. Pros - -- Initially the lack of discoverability will not be a problem. -- Only around [1.5% of all exisiting Users are using the Explore functionality on a monthly basis](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521). - -## 4.2. Cons - -- The GitLab owned top-level Groups would be some of the first to be moved into their own Organization and thus be detached from the explorability of the default Organization. +This document was moved to [another location](impacted_features/explore.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-git-access.md b/doc/architecture/blueprints/cells/cells-feature-git-access.md index 611b4db5f43..18fc2b61b1f 100644 --- a/doc/architecture/blueprints/cells/cells-feature-git-access.md +++ b/doc/architecture/blueprints/cells/cells-feature-git-access.md @@ -1,156 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Git Access' +redirect_to: 'impacted_features/git-access.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Git Access - -This document describes impact of Cells architecture on all Git access (over HTTPS and SSH) patterns providing explanation of how potentially those features should be changed to work well with Cells. - -## 1. Definition - -Git access is done throughout the application. -It can be an operation performed by the system (read Git repository) or by a user (create a new file via Web IDE, `git clone` or `git push` via command line). -The Cells architecture defines that all Git repositories will be local to the Cell, so no repository could be shared with another Cell. - -The Cells architecture will require that any Git operation can only be handled by a Cell holding the data. -It means that any operation either via Web interface, API, or GraphQL needs to be routed to the correct Cell. -It means that any `git clone` or `git push` operation can only be performed in the context of a Cell. - -## 2. Data flow - -The are various operations performed today by GitLab on a Git repository. -This describes the data flow how they behave today to better represent the impact. - -It appears that Git access does require changes only to a few endpoints that are scoped to a Project. -There appear to be different types of repositories: - -- Project: assigned to Group -- Wiki: additional repository assigned to Project -- Design: similar to Wiki, additional repository assigned to Project -- Snippet: creates a virtual Project to hold repository, likely tied to the User - -### 2.1. Git clone over HTTPS - -Execution of: `git clone` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC InfoRefsUploadPack - Gitaly ->> User: Response - User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack - Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.2. Git clone over SSH - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.3. Git push over HTTPS - -Execution of: `git push` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC PostReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.4. Git push over SSHD - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC ReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.5. Create commit via Web - -Execution of `Add CHANGELOG` to repository: - -```mermaid -sequenceDiagram - Web ->> Puma: POST /gitlab-org/gitlab/-/create/main - Puma ->> Gitaly: RPC TreeEntry - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> Puma: Response - Puma ->> Web: See CHANGELOG -``` - -## 3. Proposal - -The Cells stateless router proposal requires that any ambiguous path (that is not routable) will be made routable. -It means that at least the following paths will have to be updated to introduce a routable entity (Project, Group, or Organization). - -Change: - -- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` -- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` -- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` -- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` - -Where: - -- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) -- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) - -## 4. Evaluation - -Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. -The one major complication is supporting snippets, but this likely falls in the same category as for the approach to support a user's Personal Namespace. - -## 4.1. Pros - -1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. - -## 4.2. Cons - -1. The sharing of repositories objects is limited to the given Cell and Gitaly node. -1. Cross-Cells forks are likely impossible to be supported (discover: How this works today across different Gitaly node). +This document was moved to [another location](impacted_features/git-access.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md index 7e4ab785095..964423334c1 100644 --- a/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md +++ b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: GitLab Pages' +redirect_to: 'impacted_features/gitlab-pages.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: GitLab Pages - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/gitlab-pages.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-global-search.md b/doc/architecture/blueprints/cells/cells-feature-global-search.md index 475db381ff5..0a2a89b2d45 100644 --- a/doc/architecture/blueprints/cells/cells-feature-global-search.md +++ b/doc/architecture/blueprints/cells/cells-feature-global-search.md @@ -1,35 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Global search' +redirect_to: 'impacted_features/global-search.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Global search - -When we introduce multiple Cells we intend to isolate all services related to those Cells. -This will include Elasticsearch which means our current global search functionality will not work. -It may be possible to implement aggregated search across all Cells, but it is unlikely to be performant to do fan-out searches across all Cells especially once you start to do pagination which requires setting the correct offset and page number for each search. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -Likely the first versions of Cells will not support global searches. -Later, we may consider if building global searches to support popular use cases is worthwhile. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/global-search.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-graphql.md b/doc/architecture/blueprints/cells/cells-feature-graphql.md index e8850dfbee3..69ce2128484 100644 --- a/doc/architecture/blueprints/cells/cells-feature-graphql.md +++ b/doc/architecture/blueprints/cells/cells-feature-graphql.md @@ -1,81 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: GraphQL' +redirect_to: 'impacted_features/graphql.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: GraphQL - -GitLab extensively uses GraphQL to perform efficient data query operations. -GraphQL due to it's nature is not directly routable. -The way GitLab uses it calls the `/api/graphql` endpoint, and only the query or mutation of the body request might define where the data can be accessed. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -There are at least two main ways to implement GraphQL in a Cells architecture. - -### 3.1. GraphQL routable by endpoint - -Change `/api/graphql` to `/api/organization/<organization>/graphql`. - -- This breaks all existing usages of `/api/graphql` endpoint because the API URI is changed. - -### 3.2. GraphQL routable by body - -As part of router parse GraphQL body to find a routable entity, like `project`. - -- This still makes the GraphQL query be executed only in context of a given Cell and not allowing the data to be merged. - -```json -# Good example -{ - project(fullPath:"gitlab-org/gitlab") { - id - description - } -} - -# Bad example, since Merge Request is not routable -{ - mergeRequest(id: 1111) { - iid - description - } -} -``` - -### 3.3. Merging GraphQL Proxy - -Implement as part of router GraphQL Proxy which can parse body and merge results from many Cells. - -- This might make pagination hard to achieve, or we might assume that we execute many queries of which results are merged across all Cells. - -```json -{ - project(fullPath:"gitlab-org/gitlab"){ - id, description - } - group(fullPath:"gitlab-com") { - id, description - } -} -``` - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/graphql.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-organizations.md b/doc/architecture/blueprints/cells/cells-feature-organizations.md index f1527b40ef4..6b589307404 100644 --- a/doc/architecture/blueprints/cells/cells-feature-organizations.md +++ b/doc/architecture/blueprints/cells/cells-feature-organizations.md @@ -1,36 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Organizations' +redirect_to: 'impacted_features/organizations.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Organizations - -One of the major designs of a Cells architecture is strong isolation between Groups. -Organizations as described by the [Organization blueprint](../organization/index.md) provides a way to have plausible UX for joining together many Groups that are isolated from the rest of the system. - -## 1. Definition - -Cells do require that all Groups and Projects of a single Organization can only be stored on a single Cell because a Cell can only access data that it holds locally and has very limited capabilities to read information from other Cells. - -Cells with Organizations do require strong isolation between Organizations. - -It will have significant implications on various user-facing features, like Todos, dropdowns allowing to select Projects, references to other issues or Projects, or any other social functions present at GitLab. -Today those functions were able to reference anything in the whole system. -With the introduction of Organizations this will be forbidden. - -This problem definition aims to answer effort and implications required to add strong isolation between Organizations to the system, including features affected and their data processing flow. -The purpose is to ensure that our solution when implemented consistently avoids data leakage between Organizations residing on a single Cell. - -## 2. Proposal - -See the [Organization blueprint](../organization/index.md). +This document was moved to [another location](impacted_features/organizations.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md b/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md index 3aca9f1e116..115af11e3a6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md +++ b/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md @@ -1,31 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Personal Access Tokens' +redirect_to: 'impacted_features/personal-access-tokens.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Personal Access Tokens - -## 1. Definition - -Personal Access Tokens associated with a User are a way for Users to interact with the API of GitLab to perform operations. -Personal Access Tokens today are scoped to the User, and can access all Groups that a User has access to. - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/personal-access-tokens.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md index e8f5c250a8e..6d5ec0c9dd6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md +++ b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Personal Namespaces' +redirect_to: 'impacted_features/personal-namespaces.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Personal Namespaces - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/personal-namespaces.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md index d403d6ff963..0143ac6ffd9 100644 --- a/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md +++ b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md @@ -1,34 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Router Endpoints Classification' +redirect_to: 'impacted_features/router-endpoints-classification.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Router Endpoints Classification - -Classification of all endpoints is essential to properly route requests hitting the load balancer of a GitLab installation to a Cell that can serve it. -Each Cell should be able to decode each request and classify which Cell it belongs to. - -GitLab currently implements hundreds of endpoints. -This document tries to describe various techniques that can be implemented to allow the Rails to provide this information efficiently. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/router-endpoints-classification.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-schema-changes.md b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md index dd0f6c0705c..bf78a4eae41 100644 --- a/doc/architecture/blueprints/cells/cells-feature-schema-changes.md +++ b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md @@ -1,38 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Schema changes' +redirect_to: 'impacted_features/schema-changes.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Schema changes - -When we introduce multiple Cells that own their own databases this will complicate the process of making schema changes to Postgres and Elasticsearch. -Today we already need to be careful to make changes comply with our zero downtime deployments. -For example, [when removing a column we need to make changes over 3 separate deployments](../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). -We have tooling like `post_migrate` that helps with these kinds of changes to reduce the number of merge requests needed, but these will be complicated when we are dealing with deploying multiple Rails applications that will be at different versions at any one time. -This problem will be particularly tricky to solve for shared databases like our plan to share the `users` related tables among all Cells. - -A key benefit of Cells may be that it allows us to run different customers on different versions of GitLab. -We may choose to update our own Cell before all our customers giving us even more flexibility than our current canary architecture. -But doing this means that schema changes need to have even more versions of backward compatibility support which could slow down development as we need extra steps to make schema changes. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/schema-changes.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-secrets.md b/doc/architecture/blueprints/cells/cells-feature-secrets.md index 681c229711d..1c4c79d96fc 100644 --- a/doc/architecture/blueprints/cells/cells-feature-secrets.md +++ b/doc/architecture/blueprints/cells/cells-feature-secrets.md @@ -1,43 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Secrets' +redirect_to: 'impacted_features/secrets.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Secrets - -Where possible, each Cell should have its own distinct set of secrets. -However, there will be some secrets that will be required to be the same for all Cells in the cluster. - -## 1. Definition - -GitLab has a lot of [secrets](https://docs.gitlab.com/charts/installation/secrets.html) that need to be configured. -Some secrets are for inter-component communication, for example, `GitLab Shell secret`, and used only within a Cell. -Some secrets are used for features, for example, `ci_jwt_signing_key`. - -## 2. Data flow - -## 3. Proposal - -1. Secrets used for features will need to be consistent across all Cells, so that the UX is consistent. - 1. This is especially true for the `db_key_base` secret which is used for - encrypting data at rest in the database - so that Projects that are - transferred to another Cell will continue to work. We do not want to have - to re-encrypt such rows when we move Projects/Groups between Cells. -1. Secrets which are used for intra-Cell communication only should be uniquely generated - per Cell. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/secrets.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-snippets.md b/doc/architecture/blueprints/cells/cells-feature-snippets.md index bde0b098609..2963bbdec2c 100644 --- a/doc/architecture/blueprints/cells/cells-feature-snippets.md +++ b/doc/architecture/blueprints/cells/cells-feature-snippets.md @@ -1,56 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Snippets' +redirect_to: 'impacted_features/snippets.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Snippets - -Snippets will be scoped to an Organization. Initially it will not be possible to aggregate snippet collections across Organizations. See also [issue #416954](https://gitlab.com/gitlab-org/gitlab/-/issues/416954). - -## 1. Definition - -Two different types of snippets exist: - -- [Project snippets](../../../api/project_snippets.md). These snippets have URLs - like `/<group>/<project>/-/snippets/123` -- [Personal snippets](../../../user/snippets.md). These snippets have URLs like - `/-/snippets/123` - -Snippets are backed by a Git repository. - -## 2. Data flow - -## 3. Proposal - -### 3.1. Scoped to an organization - -Both project and personal snippets will be scoped to an Organization. - -- Project snippets URLs will remain unchanged, as the URLs are routable. -- Personal snippets URLs will need to change to be `/-/organizations/<organization>/snippets/123`, - so that the URL is routeable - -Creation of snippets will also be scoped to a User's current Organization. Because of that, we recommend renaming `personal snippets` to `organization snippets` once the Organization is rolled out. A User can create many independent snippet collections across multiple Organizations. - -## 4. Evaluation - -Snippets are scoped to an Organization because Gitaly is confined to a Cell. - -## 4.1. Pros - -- No need to have clusterwide Gitaly. - -## 4.2. Cons - -- We will break [snippet discovery](/ee/user/snippets.md#discover-snippets). -- Snippet access may become subordinate to the visibility of the Organization. +This document was moved to [another location](impacted_features/snippets.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-template.md b/doc/architecture/blueprints/cells/cells-feature-template.md index 3cece3dc99e..c75cc88f46c 100644 --- a/doc/architecture/blueprints/cells/cells-feature-template.md +++ b/doc/architecture/blueprints/cells/cells-feature-template.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Problem A' +redirect_to: 'impacted_features/template.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: A - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/template.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-uploads.md b/doc/architecture/blueprints/cells/cells-feature-uploads.md index fdac3a9977c..eab7a8a4fcd 100644 --- a/doc/architecture/blueprints/cells/cells-feature-uploads.md +++ b/doc/architecture/blueprints/cells/cells-feature-uploads.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Uploads' +redirect_to: 'impacted_features/uploads.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Uploads - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/uploads.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-user-profile.md b/doc/architecture/blueprints/cells/cells-feature-user-profile.md index fc02548f371..73f312f3762 100644 --- a/doc/architecture/blueprints/cells/cells-feature-user-profile.md +++ b/doc/architecture/blueprints/cells/cells-feature-user-profile.md @@ -1,52 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: User Profile' +redirect_to: 'impacted_features/user-profile.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: User Profile - -The existing User Profiles will initially be scoped to an Organization. Long-term, we should consider aggregating parts of the User activity across Organizations to enable Users a global view of their contributions. - -## 1. Definition - -Each GitLab account has a [User Profile](../../../user/profile/index.md), which contains information about the User and their GitLab activity. - -## 2. Data flow - -## 3. Proposal - -User Profiles will be scoped to an Organization. - -- Users can set a Home Organization as their main Organization. -- Users who do not exist in the database at all display a 404 not found error when trying to access their User Profile. -- User who haven't contributed to an Organization display their User Profile with an empty state. -- When displaying a User Profile empty state, if the profile has a Home Organization set to another Organization, we display a call-to-action allowing navigation to the main Organization. -- User Profile URLs will not reference the Organization and remain as: `/<username>`. We follow the same pattern as is used for `Your Work`, meaning that profiles are always seen in the context of an Organization. -- Breadcrumbs on the User Profile will present as `[Organization Name] / [Username]`. - -See [issue #411931](https://gitlab.com/gitlab-org/gitlab/-/issues/411931) for design proposals. - -## 4. Evaluation - -We expect the [majority of Users to perform most of their activity in one single Organization](../organization/index.md#data-exploration). -This is why we deem it acceptable to scope the User Profile to an Organization at first. -More discovery is necessary to understand which aspects of the current User Profile are relevant to showcase contributions in a global context. - -## 4.1. Pros - -- Viewing a User Profile scoped to an Organization allows you to focus on contributions that are most relevant to your Organization, filtering out the User's other activities. -- Existing User Profile URLs do not break. - -## 4.2. Cons - -- Users will lose the ability to display their entire activity, which may lessen the effectiveness of using their User Profile as a resume of achievements when working across multiple Organizations. +This document was moved to [another location](impacted_features/user-profile.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-your-work.md b/doc/architecture/blueprints/cells/cells-feature-your-work.md index 08bb0bed709..344037f2a76 100644 --- a/doc/architecture/blueprints/cells/cells-feature-your-work.md +++ b/doc/architecture/blueprints/cells/cells-feature-your-work.md @@ -1,58 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Your Work' +redirect_to: 'impacted_features/your-work.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Your Work - -Your Work will be scoped to an Organization. -Counts presented in the individual dashboards will relate to the selected Organization. - -## 1. Definition - -When accessing `gitlab.com/dashboard/`, users can find a [focused view of items that they have access to](../../../tutorials/left_sidebar/index.md#use-a-more-focused-view). -This overview contains dashboards relating to: - -- Projects -- Groups -- Issues -- Merge requests -- To-Do list -- Milestones -- Snippets -- Activity -- Workspaces -- Environments -- Operations -- Security - -## 2. Data flow - -## 3. Proposal - -Your Work will be scoped to an Organization, giving the user an overview of all the items they can access in the Organization they are currently viewing. - -- Issue, Merge request and To-Do list counts will refer to the selected Organization. - -## 4. Evaluation - -Scoping Your Work to an Organization makes sense in the context of the [proposed Organization navigation](https://gitlab.com/gitlab-org/gitlab/-/issues/417778). -Considering that [we expect most users to work in a single Organization](../organization/index.md#data-exploration), we deem this impact acceptable. - -## 4.1. Pros - -- Viewing Your Work scoped to an Organization allows Users to focus on content that is most relevant to their currently selected Organization. - -## 4.2. Cons - -- Users working across multiple Organizations will have to navigate to each Organization to access all of their work items. +This document was moved to [another location](impacted_features/your-work.md). diff --git a/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png b/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png Binary files differdeleted file mode 100644 index c5fff9dbca5..00000000000 --- a/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png +++ /dev/null diff --git a/doc/architecture/blueprints/cells/diagrams/index.md b/doc/architecture/blueprints/cells/diagrams/index.md index 77d12612819..14db888382e 100644 --- a/doc/architecture/blueprints/cells/diagrams/index.md +++ b/doc/architecture/blueprints/cells/diagrams/index.md @@ -22,7 +22,7 @@ Load the `.drawio.png` or `.drawio.svg` file directly into **draw.io**, which yo To create a diagram from a file: 1. Copy existing file and rename it. Ensure that the extension is `.drawio.png` or `.drawio.svg`. -1. Edit the diagram. +1. Edit the diagram. 1. Save the file. To create a diagram from scratch using [draw.io desktop](https://github.com/jgraph/drawio-desktop/releases): diff --git a/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png Binary files differindex 84a6d6d1745..639026c801f 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png Binary files differindex a6fd790ba5e..c5e3a0f7c71 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png Binary files differindex f1cb7cd92fe..9bfdba43309 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png Binary files differindex f5535409945..c8f6393f9fc 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png diff --git a/doc/architecture/blueprints/cells/glossary.md b/doc/architecture/blueprints/cells/glossary.md index 11a1fc5acc9..69824663867 100644 --- a/doc/architecture/blueprints/cells/glossary.md +++ b/doc/architecture/blueprints/cells/glossary.md @@ -1,106 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Glossary' +redirect_to: 'goals.md#glossary' +remove_date: '2023-11-24' --- -# Cells: Glossary - -We use the following terms to describe components and properties of the Cells architecture. - -## Cell - -> Pod was renamed to Cell in <https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/121163> - -A Cell is a set of infrastructure components that contains multiple top-level groups that belong to different organizations. The components include both datastores (PostgreSQL, Redis etc.) and stateless services (web etc.). The infrastructure components provided within a Cell are shared among organizations and their top-level groups but not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other. - -<img src="diagrams/term-cell.drawio.png" height="200"> - -### Cell properties - -- Each cell is independent from the others -- Infrastructure components are shared by organizations and their top-level groups within a Cell -- More Cells can be provisioned to provide horizontal scalability -- A failing Cell does not lead to failure of other Cells -- Noisy neighbor effects are limited to within a Cell -- Cells are not visible to organizations; it is an implementation detail -- Cells may be located in different geographical regions (for example, EU, US, JP, UK) - -Discouraged synonyms: GitLab instance, cluster, shard - -## Cluster - -A cluster is a collection of Cells. - -<img src="diagrams/term-cluster.drawio.png" height="300"> - -### Cluster properties - -- A cluster holds cluster-wide metadata, for example Users, Routes, Settings. - -Discouraged synonyms: whale - -## Organizations - -GitLab references [Organizations in the initial set up](../../../topics/set_up_organization.md) and users can add a (free text) organization to their profile. There is no Organization entity established in the GitLab codebase. - -As part of delivering Cells, we propose the introduction of an `organization` entity. Organizations would represent billable entities or customers. - -Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). - -Organizations work under the following assumptions: - -1. Users care about what happens within their organizations. -1. Features need to work within an organization. -1. Only few features need to work across organizations. -1. Users understand that the majority of pages they view are only scoped to a single organization at a time. -1. Organizations are located on a single cell. - - - -### Organization properties - -- Top-level groups belong to organizations -- Organizations are isolated from each other by default meaning that cross-group features will only work for group that exist within a single organization -- User namespaces must not belong to an organization - -Discouraged synonyms: Billable entities, customers - -## Top-Level group - -Top-level group is the name given to the top most group of all other groups. Groups and projects are nested underneath the top-level group. - -Example: - -`https://gitlab.com/gitlab-org/gitlab/`: - -- `gitlab-org` is a `top-level group`; the root for all groups and projects of an organization -- `gitlab` is a `project`; a project of the organization. - -The top-level group has served as the defacto Organization entity. With the creation of Organization, top-level groups will be [nested underneath Organizations](https://gitlab.com/gitlab-org/gitlab/-/issues/394796). - -Over time there won't be a distinction between a top-level group and a group. All features that make Top-level groups different from groups will move to Organization. - -Discouraged synonyms: Root-level namespace - - - -### Top-level group properties - -- Top-level groups belonging to an organization are located on the same Cell -- Top-level groups can interact with other top-level groups that belong to the same organization - -## Users - -Users are available globally and not restricted to a single Cell. Users belong to a single organization, but can participate in many organizations through group and project membership with varying permissions. Inside organizations, users can create multiple top-level groups. User activity is not limited to a single organization but their contributions (for example TODOs) are only aggregated within an organization. This avoids the need for aggregating across cells. - -### User properties - -- Users are shared globally across all Cells -- Users can create multiple top-level groups -- Users can be a member of multiple top-level groups -- Users belong to one organization. See [!395736](https://gitlab.com/gitlab-org/gitlab/-/issues/395736) -- Users can be members of groups and projects in different organizations -- Users can administer organizations -- User activity is aggregated in an organization -- Every user has one personal namespace +This document was moved to [another location](goals.md#glossary). diff --git a/doc/architecture/blueprints/cells/goals.md b/doc/architecture/blueprints/cells/goals.md index 3f3923aa255..f1a255aa879 100644 --- a/doc/architecture/blueprints/cells/goals.md +++ b/doc/architecture/blueprints/cells/goals.md @@ -6,7 +6,9 @@ description: 'Cells: Goals' # Cells: Goals -## Scalability +## Goals + +### Scalability The main goal of this new shared-infrastructure architecture is to provide additional scalability for our SaaS Platform. GitLab.com is largely monolithic and we have estimated (internally) that the current architecture has scalability limitations, @@ -16,34 +18,41 @@ even when database partitioning and decomposition are taken into account. Cells provide a horizontally scalable solution because additional Cells can be created based on demand. Cells can be provisioned and tuned as needed for optimal scalability. -## Increased availability +### Increased availability -A major challenge for shared-infrastructure architectures is a lack of isolation between top-level groups. This can lead to noisy neighbor effects. A organization's behavior inside a top-level group can impact all other organizations. This is highly undesirable. Cells provide isolation at the cell level. A group of organizations is fully isolated from other organizations located on a different Cell. This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of shared infrastructure. +A major challenge for shared-infrastructure architectures is a lack of isolation between top-level Groups. +This can lead to noisy neighbor effects. +An organization's behavior inside a top-level Group can impact all other organizations. +This is highly undesirable. +Cells provide isolation at the Cell level. +A group of Organizations is fully isolated from other Organizations located on a different Cell. +This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of a shared infrastructure. -Additionally, Cells provide a way to implement disaster recovery capabilities. Entire Cells may be replicated to read-only standbys with automatic failover capabilities. +Additionally, Cells provide a way to implement disaster recovery capabilities. +Entire Cells may be replicated to read-only standbys with automatic failover capabilities. -## A consistent experience +### A consistent experience Organizations should have the same user experience on our SaaS platform as they do on a self-managed GitLab instance. -## Regions - -GitLab.com is only hosted within the United States of America. Organizations located in other regions have voiced demand for local SaaS offerings. Cells provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Cells may be deployed within different geographies. Depending on which of the organization's data is located outside a Cell, this may solve data residency and compliance problems. +### Regions -## Market segment +GitLab.com is only hosted within the United States of America. +Organizations located in other regions have voiced demand for local SaaS offerings. +Cells provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Cells may be deployed within different geographies. +Depending on which of an organization's data is located outside a Cell, this may solve data residency and compliance problems. -Cells would provide a solution for organizations in the small to medium business (up to 100 users) and the mid-market segment (up to 2000 users). -(See [segmentation definitions](https://about.gitlab.com/handbook/sales/field-operations/gtm-resources/#segmentation).) -Larger organizations may benefit substantially from [GitLab Dedicated](../../../subscriptions/gitlab_dedicated/index.md). +### Market segment -At this moment, GitLab.com has "social-network"-like capabilities that may not fit well into a more isolated organization model. Removing those features, however, possesses some challenges: +At this moment, GitLab.com has "social network"-like capabilities that may not fit well into a more isolated Organization model. +Removing those features, however, poses some challenges: -1. How will existing `gitlab-org` contributors contribute to the namespace?? -1. How do we move existing top-level groups into the new model (effectively breaking their social features)? +1. How will existing `gitlab-org` contributors contribute to the namespace? +1. How do we move existing top-level Groups into the new model (effectively breaking their social features)? -We should evaluate if the SMB and mid market segment is interested in these features, or if not having them is acceptable in most cases. +We should evaluate if the small to medium business and mid-market segment is interested in these features, or if not having them is acceptable in most cases. -## Self-managed +### Self-managed For reasons of consistency, it is expected that self-managed instances will adopt the cells architecture as well. To expand, self-managed instances can @@ -51,13 +60,341 @@ continue with just a single Cell while supporting the option of adding additiona Cells. Organizations, and possible User decomposition will also be adopted for self-managed instances. -## High-level architecture problems to solve +## Requirements + +| Type | Requirement | Severity | +| ----------- | ------------------------------------------------- | -------- | +| Product | Aggregation of cluster-wide data | Medium | +| Product | All Cells are under a single GitLab.com domain | High | +| Product | User can interact with many Cells | Medium | +| Product | Minimal impact on contributor workflows | High | +| Product | On-premise like experience | Medium | +| Product | Minimize breaking changes | High | +| Product | Enables support for Regions | Low | +| Product | Limit impact on self-managed customers | Low | +| Operational | Provides 10x headroom | High | +| Operational | Provides 100x headroom | Medium | +| Operational | Improve Service Availability | High | +| Operational | Cost per user similar or lower to GitLab.com | Medium | +| Operational | Unified way of deploying Cells | Medium | +| Operational | Cells running in mixed deployments | High | +| Operational | High resilience to a single Cell failure | High | +| Operational | Small Cells | Medium | +| Migration | Robust rollback and disaster recovery scenarios | High | +| Migration | Scale existing GitLab.com database | High | +| Migration | Re-balancing of Cells | Low | +| Migration | Customer can migrate from GitLab.com to Dedicated | Low | +| Development | Easy usage in development environment | Medium | + +### Aggregation of cluster-wide data + +The architecture should provide a way to present the cluster of Cells in a single view. +This might mean to: + +- Aggregate user To-Dos from different Organizations +- Perform cluster-wide searches of public Projects, or published source code + +### All Cells are under a single GitLab.com domain + +General users should be unaware of the presence of Cells. +Cells would only be visible to instance administrators. +Users using Organizations on another Cell or us migrating Organizations +between Cells should be an operational detail that does not require +any user intervention or is not visible to the user. + +### User can interact with many Cells + +Usage of many accounts to interact with different Organizations +that might be on different Cells is strongly discouraged. +The need to use many accounts will reduce adoption and put a barrier +on users wanting to contribute to open-source Projects. + +In the future, users could have Organization specific settings, +depending on the specific customer expectations. +For instance, users might be shown with different display names in different Organizations. + +Users should not be required to use different SSH keys to access different +Organizations due to the complexity of managing many SSH keys by the user. + +### Minimal impact on contributor workflows + +GitLab.com has a number of open-source and open-core projects (including [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab)). +Cells should not make it harder to contribute to public Projects. +Learning new ways to contribute is likely to hinder the adoption of Cells. +The introduced architecture should focus on changing the existing workflows in the most minimal way. + +### On-premise like experience + +Currently on-premise has numerous advantages compared to our SaaS offering. +On-premise allows to control all aspects of a GitLab installation, including but not limited to: managing users, access controls, or instance-wide settings. + +The difference between SaaS and on-premise is problematic for our customers, +and us as it results in a different user experience. + +### Minimize breaking changes + +The introduction of Cells will imply changes in the supported GitLab.com workflows. +Cells might affect how people use GitLab, as long as it provides a compelling user experience story. +It is desired that each change introduced provides either operational value or user value, ideally both. + +Any Cells architecture should focus on not introducing breaking changes, +or if introducing them provide a long window to support the change. +The introduction of breaking changes will reduce the adoption rate +and effectiveness of Cells and make rollout of the architecture more risky. + +It is important to distinguish between different severity of breaking changes: + +- Required: migration to a new architecture by design introduces a breaking change. + In such a case a customer has to use a new workflow, new API, or new endpoints right away + and is required to adapt to all changes at once. The customer has limited options to roll back. + +- Optional: migration to a new architecture does not enforce introducing breaking changes. + In such a case a customer can slowly adapt to introduced changes: a new workflow, new API, + or new endpoints, but retaining major aspects of the system to work as before. + The customer has a way to roll back to a previous known working state. + +### Enables support for Regions + +Support for Regions should enable us to run GitLab in different availability zones, +or completely different data centers. + +This includes, but is not limited to allowing users to create Organizations in Europe, the US West or US East, +and GitLab Inc. to use different cloud infrastructure providers (Google Cloud, AWS) to serve customers. + +Cells enable existing customers running Organizations to move their Organization between different deployment regions/data centers that are supported by GitLab on GitLab.com. + +### Limit impact on self-managed customers (Omnibus/CNG) + +The introduction of Cells should not impact small installations. +Cells should not require running additional components by self-managed customers +unless they are interested in Cells that will increase resource requirements. + +### Provides 10x headroom + +The Cells architecture must provide at least 10x headroom. As such our architecture must be suitable to run 10 Cells. We do not need to initially solve for the complexities that might come with running more than 10 Cells. + +### Provides 100x headroom + +The Cells architecture should be able to run more than 10 Cells. + +### Improve Service Availability + +The Cells architecture should allow us to provide better SLA for some customers +by putting them on specific Cells of the cluster: + +- Cells have to reduce the impact of noisy neighbors, such as legitimate usage caused by a spiky workload. +- Cells have to reduce the impact caused by abuse, for instance CI being used for cryptocurrency mining. + +### Cost per user similar or lower to GitLab.com + +The current GitLab.com architecture is rather cost-effective. +The introduction of Cells will result in more infrastructure components +due to data distribution: + +- The proposed Cells architecture should evaluate the cost of running additional Cells with respect + to the achieved infrastructure component isolation. In some cases it might be beneficial to reuse + infrastructure components to reduce the running cost of Cells. +- The proposed Cells architecture should ensure a high degree of multi-tenancy on Cells. +- The proposed Cells architecture could enable a way to balance the Cells load long term. + +### Unified way of deploying Cells + +The proposed Cells architecture anticipates a need to run 100 Cells and more in a cluster. +This becomes operational overhead. It is strongly desired for the Cells architecture +to reuse existing infrastructure tooling as much as possible for: deployment, monitoring, +logging, disaster recovery, and customer support. + +### Cells running in mixed deployments + +It is strongly required for the Cells architecture to run different versions of applications across the cluster. The purpose is to provide staged deployments allowing to test changes on part of the cluster, and to update part of the cluster less frequently. + +This also indicates that Cells should be able to work with different versions of the underlying +infrastructure components: PostgreSQL version, Redis version, Gitaly version, Elasticsearch version. + +Usage of different versions within a Cell has to have no effect on other Cells. + +### High resilience to a single Cell failure + +Unavailability of a single Cell should not result in other Cells being unable to function properly: + +- Intra-cluster communication between Cells should be resilient to single Cell failure. +- Intra-cluster communication should limit its dependence on data stored in other Cells. +- Cluster-shared data should provide an anti-corruption layer to prevent a single Cell from causing a cluster-wide outage. +- Cells should be monitored and their availability status should be accessible to all other Cells. +- Cluster-wide data aggregation should take the availability status of each Cell into account. +- The loss of cell-local data (groups and projects) does only affect ability to access data located on this cell. + +### Small Cells + +The Cells architecture should provide a cost effective way to run small Cells. +Running small Cells allows to achieve superior quality in terms of latency or performance +due to processing a significantly smaller data set and allowing significant single-node vertical scaling. + +The ability to run small Cells could result in reduced complexity of individual Cell deployment: + +- Prefer single-node vertical scaling as a first technique for increasing capacity (increase CPU count, available memory). +- Reduce amount of data stored within a Cell to reduce time-to-recovery in case of outages. +- Less data results in faster database migrations, improved latency, and superior user-facing performance. + +### Robust rollback and disaster recovery scenarios + +Rollout of the Cells architecture is a fundamental change to how GitLab.com will operate. +This will involve introducing a number of new components, and distributing data horizontally across +different Cells. + +We anticipate that something will go wrong at some point. Each deployment phase of the Cells architecture needs to provide a way to roll back to a prior working state. This includes any user-facing changes, as well as any infrastructure deployed components. + +### Scale existing GitLab.com database + +The existing GitLab.com database will continue to grow while we deploy Cells. +The Cells architecture should provide a way to increase Cell 1 (existing GitLab.com) +capacity as soon as possible. + +### Re-balancing of Cells + +It is expected that data distribution on Cells will become uneven at some point. + +The problem of re-balancing Cells is rather complex, but at some scale it might +provide a positive ROI compared to the running cost of imbalanced Cells. + +This might involve understanding how to selectively migrate customer data between +Cells onto already existing Cells. + +### Customer can migrate from GitLab.com to Dedicated + +Cells will enforce an architecture with more isolation. However, it is expected +that some customers would like to move from SaaS GitLab.com to GitLab Dedicated. + +This implies being able to selectively migrate data from the GitLab.com cluster to Dedicated. +This could be achieved by: + +- Migrating a customer to a Dedicated Cell on the GitLab.com cluster. +- Disconnecting a Cell from the GitLab.com cluster into its own standalone Dedicated Cell. + +### Easy usage in development environment + +The Cells architecture should be easy to run locally by developers as needed to be able to: + +- Implement cluster-wide features. +- Model Cells not being available. + +## Non-Goals + +The following objectives are not part of this document. At some point they +were considered, but dismissed as creating distraction from the goals and requirements documented above. + +### Federation between distinct GitLab instances + +The Cells architecture is meant to provide trusted intra-cluster communication +with some set of data being shared. Federation is meant to solve the problem +of data flow between two completely distinct and external instances. + +### Usage of cloud managed services + +Cells should not interfere with other efforts to move towards using more managed services in the cloud. + +### Cells should replace canary deployments + +When we have Cells we won't need to run canary deployments anymore as we can roll out changes to single Cells at a time. +Our main benefit of canary deployments today is that we can roll out code changes to internal users first. +With Cells, we will have internal users on a specific Cell, which can be deployed first. + +Additionally Cells may solve some issues with canaries today, for example not having a way to upgrade Sidekiq code for a subset of users. + +## Glossary + +| Term | Description | Discouraged Terms | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | +| [Cell](#cell) | A Cell is a set of infrastructure components that contains multiple top-level Groups that belong to different Organizations. | GitLab instance, cluster, shard, Pod | +| [Cluster](#cluster) | A cluster is a collection of Cells. | Whale, GitLab Dedicated instance, instance | +| [Organizations](#organizations) | An Organization is the umbrella for one or multiple top-level Groups. | Billable entities, customers | +| [Top-Level Group](#top-level-group) | Top-level Group is the name given to the top-most Group of all other Groups. Groups and Projects are nested underneath the top-level Group. | Root-level namespace | +| [Users](#users) | An account containing its own personal namespace associated with an email. | Customer | + +### Cell + +> Pod was renamed to Cell in <https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/121163> + +A Cell is a set of infrastructure components that contains multiple top-level Groups that belong to different Organizations. The components include both data stores (PostgreSQL, Redis, etc.) and stateless services (web, etc.). The infrastructure components provided within a Cell are shared among Organizations and their top-level Groups, but are not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other. + +<img src="diagrams/term-cell.drawio.png" height="200"> + +- Each Cell is independent from other Cells +- Infrastructure components are shared by Organizations and their top-level Groups within a Cell +- More Cells can be provisioned to provide horizontal scalability +- A failing Cell does not lead to failure of other Cells +- Noisy neighbor effects are limited to within a Cell +- Cells are not visible to Organizations - they are an implementation detail +- Cells may be located in different geographical regions (for example: EU, US, JP, UK) +- A GitLab Dedicated instance can join the cluster, and become a Cell + +Discouraged synonyms: GitLab instance, cluster, shard, Pod + +### Cluster + +A cluster is a collection of Cells. + +<img src="diagrams/term-cluster.drawio.png" height="300"> + +- A cluster holds cluster-wide metadata, for example: Users, Routes, Settings. + +Discouraged synonyms: Whale, GitLab Dedicated instance, instance + +### Organizations + +An Organization is the umbrella for one or multiple top-level Groups. Organizations are isolated from each other by default meaning that cross-namespace features will only work for namespaces that exist in a single Organization. + + + +See the [Organization blueprint](../../blueprints/organization/index.md). + +Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). + +Organizations work under the following assumptions: + +1. Users care about what happens within their Organization. +1. Features need to work within an Organization. +1. Only few features need to work across Organizations. +1. Users understand that the majority of pages they view are only scoped to a single Organization at a time. +1. Organizations are located on a single Cell. + +Properties: + +- Top-level Groups belong to Organizations. +- Organizations are isolated from each other by default, meaning that cross-Group features will only work for Groups that exist within a single Organization. +- Personal namespaces must not belong to an Organization. + +Discouraged synonyms: Billable entities, customers + +### Top-level Group + +Example: + +`https://gitlab.com/gitlab-org/gitlab/`: + +- `gitlab-org` is a `top-level group`; the root for all Groups and Projects of an Organization +- `gitlab` is a `project`; a Project of the Organization. + +The top-level Group has served as the de facto Organization entity. With the creation of Organizations, top-level Groups will be [nested underneath Organizations](https://gitlab.com/gitlab-org/gitlab/-/issues/394796). + +Over time, there won't be a distinction between a top-level Group and a Group. All features that make top-level Groups different from Groups will move to the Organization. + + + +- Top-level Groups belonging to the same Organization are located on the same Cell. +- Top-level Groups can interact with other top-level Groups that belong to the same Organization. + +Discouraged synonyms: Root-level namespace + +### Users -A number of technical issues need to be resolved to implement Cells (in no particular order). This section will be expanded. +Users are available globally and not restricted to a single Cell. Users belong to a single Organization, but can participate in many Organizations through Group and Project membership with varying permissions. Inside Organizations, users can create multiple top-level Groups. User activity is not limited to a single Organization but their contributions (for example To-Dos) are only aggregated within an Organization. This avoids the need for aggregating across Cells. -1. How are Cells provisioned? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641) -1. What is a Cells topology? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641) -1. How are users of an organization routed to the correct Cell? - -1. How do users authenticate with Cells and Organizations? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/395736) -1. How are Cells rebalanced? -1. How can Cells implement disaster recovery capabilities? +- Users are shared globally across all Cells. +- Users can create multiple top-level Groups. +- Users can be members of multiple top-level Groups. +- Users belong to one Organization. See [!395736](https://gitlab.com/gitlab-org/gitlab/-/issues/395736). +- Users can be members of Groups and Projects in different Organizations. +- Users can administer Organizations. +- User activity is aggregated in an Organization. +- Every user has one personal namespace. diff --git a/doc/architecture/blueprints/cells/impact.md b/doc/architecture/blueprints/cells/impact.md index 30c70dca0cc..1f77b9056be 100644 --- a/doc/architecture/blueprints/cells/impact.md +++ b/doc/architecture/blueprints/cells/impact.md @@ -1,58 +1,7 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Cross-section impact' +redirect_to: 'index.md' +remove_date: '2023-11-22' --- -# Cells: Cross-section impact - -Cells is a fundamental architecture change that impacts other sections and stages. This section summarizes and links to other groups that may be impacted and highlights potential conflicts that need to be resolved. The Tenant Scale group is not responsible for achieving the goals of other groups but we want to ensure that dependencies are resolved. - -## Summary - -Based on discussions with other groups the net impact of introducing Cells and a new entity called organizations is mostly neutral. It may slow down development in some areas. We did not discover major blockers for other teams. - -1. We need to resolve naming conflicts (proposal is TBD) -1. Cells requires introducing Organizations. Organizations are a new entity **above** top-level groups. Because this is a new entity, it may impact the ability to consolidate settings for Group::Organization and influence their decision on [how to approach introducing a an organization](https://gitlab.com/gitlab-org/gitlab/-/issues/376285#approach-2-organization-is-built-on-top-of-top-level-groups) -1. Organizations may make it slightly easier for Fulfillment to realize their billing plans. - -## Impact on Group::Organization - -We synced with the Organization PM and Designer ([recording](https://youtu.be/b5Opn9cFWFk)) and discussed the similarities and differences between the Cells and Organization proposal ([presentation](https://docs.google.com/presentation/d/1FsUi22Up15b_tu6p2m-yLML3hCZ3rgrZrmzJAxUsNmU/edit?usp=sharing)). - -### Goals of Group::Organization - -As defined in the [organization documentation](../../../user/organization/index.md): - -1. Create an entity to manage everything you do as a GitLab administrator, including: - 1. Defining and applying settings to all of your groups, subgroups, and projects. - 1. Aggregating data from all your groups, subgroups, and projects. -1. Reach feature parity between SaaS and self-managed installations, with all Admin Area settings moving to groups (?). Hardware controls remain on the instance level. - -The [organization roadmap outlines](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals) the current goals in detail. - -### Potential conflicts with Cells - -- Organization defines a new entity as the primary organizational object for groups and projects. -- We will only introduce one entity -- Group::Organization highlighted the need to further validate the key assumption that users only care about what happens within their organization. - -## Impact on Fulfillment - -We synced with Fulfillment ([recording](https://youtu.be/FkQF3uF7vTY)) to discuss how Cells would impact them. Fulfillment is supportive of an entity above top-level groups. Their perspective is outline in [!5639](https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/5639/diffs). - -### Goals of Fulfillment - -- Fulfillment has a longstanding plan to move billing from the top-level group to a level above. This would mean that a license applies for an organization and all its top-level groups. -- Fulfillment uses Zuora for billing and would like to have a 1-to-1 relationship between an organization and their Zuora entity called BillingAccount. They want to move away from tying a license to a single user. -- If a customer needs multiple organizations, the corresponding BillingAccounts can be rolled up into a consolidated billing account (similar to [AWS consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html)) -- Ideally, a self-managed instance has a single Organization by default, which should be enough for most customers. -- Fulfillment prefers only one additional entity. - -A rough representation of this is: - - - -### Potential conflicts with Cells - -- There are no known conflicts between Fulfillment's plans and Cells +This document was removed due to being outdated. +Go to [index page](index.md) for the most recent content. diff --git a/doc/architecture/blueprints/cells/impacted_features/admin-area.md b/doc/architecture/blueprints/cells/impacted_features/admin-area.md new file mode 100644 index 00000000000..a9cd170b2a7 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/admin-area.md @@ -0,0 +1,81 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Admin Area' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Admin Area + +In our Cells architecture proposal we plan to share all admin related tables in GitLab. +This allows for simpler management of all Cells in one interface and reduces the risk of settings diverging in different Cells. +This introduces challenges with Admin Area pages that allow you to manage data that will be spread across all Cells. + +## 1. Definition + +There are consequences for Admin Area pages that contain data that span "the whole instance" as the Admin Area pages may be served by any Cell or possibly just one Cell. +There are already many parts of the Admin Area that will have data that span many Cells. +For example lists of all Groups, Projects, Topics, Jobs, Analytics, Applications and more. +There are also administrative monitoring capabilities in the Admin Area that will span many Cells such as the "Background Jobs" and "Background Migrations" pages. + +## 2. Data flow + +## 3. Proposal + +We will need to decide how to handle these exceptions with a few possible +options: + +1. Move all these pages out into a dedicated per-Cell admin section. Probably + the URL will need to be routable to a single Cell like `/cells/<cell_id>/admin`, + then we can display these data per Cell. These pages will be distinct from + other Admin Area pages which control settings that are shared across all Cells. We + will also need to consider how this impacts self-managed customers and + whether, or not, this should be visible for single-Cell instances of GitLab. +1. Build some aggregation interfaces for this data so that it can be fetched + from all Cells and presented in a single UI. This may be beneficial to an + administrator that needs to see and filter all data at a glance, especially + when they don't know which Cell the data is on. The downside, however, is + that building this kind of aggregation is very tricky when all Cells are + designed to be totally independent, and it does also enforce stricter + requirements on compatibility between Cells. + +The following overview describes at what level each feature contained in the current Admin Area will be managed: + +| Feature | Cluster | Cell | Organization | +| --- | --- | --- | --- | +| Abuse reports | | | | +| Analytics | | | | +| Applications | | | | +| Deploy keys | | | | +| Labels | | | | +| Messages | ✓ | | | +| Monitoring | | ✓ | | +| Subscription | | | | +| System hooks | | | | +| Overview | | | | +| Settings - General | ✓ | | | +| Settings - Integrations | ✓ | | | +| Settings - Repository | ✓ | | | +| Settings - CI/CD (1) | ✓ | ✓ | | +| Settings - Reporting | ✓ | | | +| Settings - Metrics | ✓ | | | +| Settings - Service usage data | | ✓ | | +| Settings - Network | ✓ | | | +| Settings - Appearance | ✓ | | | +| Settings - Preferences | ✓ | | | + +(1) Depending on the specific setting, some will be managed at the cluster-level, and some at the Cell-level. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md b/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md new file mode 100644 index 00000000000..37347cf836d --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Agent for Kubernetes' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Agent for Kubernetes + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/backups.md b/doc/architecture/blueprints/cells/impacted_features/backups.md new file mode 100644 index 00000000000..3d20d6e2caa --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/backups.md @@ -0,0 +1,53 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Backups' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Backups + +Each Cell will take its own backups, and consequently have its own isolated backup/restore procedure. + +## 1. Definition + +GitLab backup takes a backup of the PostgreSQL database used by the application, and also Git repository data. + +## 2. Data flow + +Each Cell has a number of application databases to back up (for example, `main`, and `ci`). +Additionally, there may be cluster-wide metadata tables (for example, `users` table) which is directly accessible via PostgreSQL. + +## 3. Proposal + +### 3.1. Cluster-wide metadata + +It is currently unknown how cluster-wide metadata tables will be accessible. +We may choose to have cluster-wide metadata tables backed up separately, or have each Cell back up its copy of cluster-wide metadata tables. + +### 3.2 Consistency + +#### 3.2.1 Take backups independently + +As each Cell will communicate with each other via API, and there will be no joins to the `users` table, it should be acceptable for each Cell to take a backup independently of each other. + +#### 3.2.2 Enforce snapshots + +We can require that each Cell take a snapshot for the PostgreSQL databases at around the same time to allow for a consistent enough backup. + +## 4. Evaluation + +As the number of Cells increases, it will likely not be feasible to take a snapshot at the same time for all Cells. +Hence taking backups independently is the better option. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md b/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md new file mode 100644 index 00000000000..3ca2ff042dc --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md @@ -0,0 +1,54 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: CI/CD Catalog' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: CI/CD Catalog + +The [CI/CD pipeline components catalog](../../ci_pipeline_components/index.md) is a currently experimental feature that aims at helping users reuse pipeline configurations. +Potentially, there are several aspects of the CI/CD catalog that might be affected by Cells: + +1. Namespace catalog, exists today as an experimental feature. With the introduction of Cells we would likely remove the namespace catalog and create a single Organization catalog, where all users would see all available published components in a single place (based on their permissions). This would replace today's offering, where users can have multiple catalogs which are bound to a namespace and completely isolated from each other. +1. The community catalog is supposed to allow users to search across different Organizations. + +## 1. Definition + +The [CI/CD pipeline components catalog](../../ci_pipeline_components/index.md) makes reusing pipeline configurations easier and more efficient. +It provides a way to discover and reuse pipeline constructs, allowing for a more streamlined experience. + +There are several flavors of the CI/CD catalog: + +1. [Namespace catalog (experimental)](../../../../ci/components/index.md): Bound to the top-level namespace (group or personal namespace). The namespace catalog aggregates all published components from Projects it contains. The number of top-level namespaces available in an Organization could potentially be the number of available catalogs. +1. Instance-wide component catalog (planned): Surfacing all the components that are scattered across an instance. All published components in a public or internal Project will be available in the instance-wide catalog. Only a single instance-wide catalog is planned per instance. +1. Community catalog (planned): Allow users to search all published components in different repositories across multiple namespaces. The original plan was to introduce a community catalog within self-managed customer that would act as an aggregator of all published components hosted in that instance. + +## 2. Data flow + +## 3. Proposal + +Moving to Organizations is a great opportunity to improve the user experience and to reach parity for both self-managed and GitLab.com users. + +- We introduce an Organization catalog which aggregates and surfaces all the published components that are hosted in a single Organization. The Organization catalog would make the namespace catalog obsolete. +- Once Organizations exist, GitLab.com users would need a community catalog to surface components across multiple Organizations. We need additional research to understand if such a solution is needed for self-managed customers as well. + +## 4. Evaluation + +Moving to a single Organization will improve the experience for users of the CI/CD component catalog. +Today we can have multiple catalogs based on the number of namespaces, making it difficult for users to surface information across an Organization. + +### 4.1. Pros + +- An Organization catalog will be one unified catalog serving as the single source of truth for an Organization. +- An Organization catalog would serve both self-managed and GitLab.com users, whereas the current plan was to introduce two types of catalogs: an instance-wide component catalog for self-managed and a community catalog for GitLab.com. + +### 4.2. Cons + +- A separate catalog that surfaces components across Organizations would need to be implemented to serve the wider community (community catalog). This catalog will be required for GitLab.com only, later on we can evaluate if a similar catalog is needed for self-managed customers. diff --git a/doc/architecture/blueprints/cells/impacted_features/ci-runners.md b/doc/architecture/blueprints/cells/impacted_features/ci-runners.md new file mode 100644 index 00000000000..e9fe4905573 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/ci-runners.md @@ -0,0 +1,144 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: CI Runners' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: CI Runners + +GitLab executes CI jobs via [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), very often managed by customers in their infrastructure. +All CI jobs created as part of the CI pipeline are run in the context of a Project. +This poses a challenge how to manage GitLab Runners. + +## 1. Definition + +There are 3 different types of runners: + +- Instance-wide: Runners that are registered globally with specific tags (selection criteria) +- Group runners: Runners that execute jobs from a given top-level Group or Projects in that Group +- Project runners: Runners that execute jobs from one Projects or many Projects: some runners might + have Projects assigned from Projects in different top-level Groups. + +This, alongside with the existing data structure where `ci_runners` is a table describing all types of runners, poses a challenge as to how the `ci_runners` should be managed in a Cells environment. + +## 2. Data flow + +GitLab runners use a set of globally scoped endpoints to: + +- Register a new runner via registration token `https://gitlab.com/api/v4/runners` + ([subject for removal](../../runner_tokens/index.md)) (`registration token`) +- Create a new runner in the context of a user `https://gitlab.com/api/v4/user/runners` (`runner token`) +- Request jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) +- Upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) +- Upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) +- Download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) + +Currently three types of authentication tokens are used: + +- Runner registration token ([subject for removal](../../runner_tokens/index.md)) +- Runner token representing a registered runner in a system with specific configuration (`tags`, `locked`, etc.) +- Build token representing an ephemeral token giving limited access to updating a specific job, uploading artifacts, downloading dependent artifacts, downloading and uploading container registry images + +Each of those endpoints receive an authentication token via header (`JOB-TOKEN` for `/trace`) or body parameter (`token` all other endpoints). + +Since the CI pipeline would be created in the context of a specific Cell, it would be required that pick of a build would have to be processed by that particular Cell. +This requires that build picking depending on a solution would have to be either: + +- Routed to the correct Cell for the first time +- Be two-phased: Request build from global pool, claim build on a specific Cell using a Cell specific URL + +## 3. Proposal + +### 3.1. Authentication tokens + +Even though the paths for CI runners are not routable, they can be made routable with these two possible solutions: + +- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with + a ticketing mechanism (based on `X-GitLab-Last-Update` header). When the runner first + starts, it sends a request to GitLab to which GitLab responds with either a build to pick + by runner. This value is completely controlled by GitLab. This allows GitLab + to use JWT or any other means to encode a `cell` identifier that could be easily + decodable by Router. +- The majority of communication (in terms of volume) is using `build token`, making it + the easiest target to change since GitLab is the sole owner of the token that the runner later + uses for a specific job. There were prior discussions about not storing the `build token` + but rather using a `JWT` token with defined scopes. Such a token could encode the `cell` + to which the Router could route all requests. + +### 3.2. Request body + +- The most used endpoints pass the authentication token in the request body. It might be desired + to use HTTP headers as an easier way to access this information by Router without + a need to proxy requests. + +### 3.3. Instance-wide are Cell-local + +We can pick a design where all runners are always registered and local to a given Cell: + +- Each Cell has its own set of instance-wide runners that are updated at its own pace +- The Project runners can only be linked to Projects from the same Organization, creating strong isolation. +- In this model the `ci_runners` table is local to the Cell. +- In this model we would require the above endpoints to be scoped to a Cell in some way, or be made routable. It might be via prefixing them, adding additional Cell parameters, or providing much more robust ways to decode runner tokens and match it to a Cell. +- If a routable token is used, we could move away from cryptographic random stored in database to rather prefer to use JWT tokens. +- The Admin Area showing registered runners would have to be scoped to a Cell. + +This model might be desired because it provides strong isolation guarantees. +This model does significantly increase maintenance overhead because each Cell is managed separately. +This model may require adjustments to the runner tags feature so that Projects have a consistent runner experience across Cells. + +### 3.4. Instance-wide are cluster-wide + +Contrary to the proposal where all runners are Cell-local, we can consider that runners +are global, or just instance-wide runners are global. + +However, this requires significant overhaul of the system and we would have to change the following aspects: + +- The `ci_runners` table would likely have to be decomposed into `ci_instance_runners`, ... +- All interfaces would have to be adopted to use the correct table. +- Build queuing would have to be reworked to be two-phased where each Cell would know of all pending and running builds, but the actual claim of a build would happen against a Cell containing data. +- It is likely that `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables, increasing the likelihood of creating hotspots in a system related to CI queueing. + +This model is complex to implement from an engineering perspective. +Some data are shared between Cells. +It creates hotspots/scalability issues in a system that might impact the experience of Organizations on other Cells, for instance during abuse. + +### 3.5. GitLab CI Daemon + +Another potential solution to explore is to have a dedicated service responsible for builds queueing, owning its database and working in a model of either sharded or Cell-ed service. +There were prior discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). + +If the service is sharded: + +- Depending on the model, if runners are cluster-wide or Cell-local, this service would have to fetch data from all Cells. +- If the sharded service is used we could adapt a model of sharing a database containing `ci_pending_builds/ci_running_builds` with the service. +- If the sharded service is used we could consider a push model where each Cell pushes to CI/CD Daemon builds that should be picked by runner. +- The sharded service would be aware which Cell is responsible for processing the given build and could route processing requests to the designated Cell. + +If the service is Cell-ed: + +- All expectations of routable endpoints are still valid. + +In general usage of CI Daemon does not help significantly with the stated problem. +However, this offers a few upsides related to more efficient processing and decoupling model: push model and it opens a way to offer stateful communication with GitLab runners (ex. gRPC or Websockets). + +## 4. Evaluation + +Considering all options it appears that the most promising solution is to: + +- Use [Instance-wide are Cell-local](#33-instance-wide-are-cell-local) +- Refine endpoints to have routable identities (either via specific paths, or better tokens) + +Another potential upside is to get rid of `ci_builds.token` and rather use a `JWT token` that can much better and easier encode a wider set of scopes allowed by CI runner. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/container-registry.md b/doc/architecture/blueprints/cells/impacted_features/container-registry.md new file mode 100644 index 00000000000..ea15dd52d94 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/container-registry.md @@ -0,0 +1,114 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Container Registry' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Container Registry + +GitLab [Container Registry](../../../../user/packages/container_registry/index.md) is a feature allowing to store Docker container images in GitLab. + +## 1. Definition + +GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis and Object Storage dependencies. +Right now there's undergoing work to introduce [Container Registry Metadata](../../container_registry_metadata_database/index.md) to optimize data storage and image retention policies of Container Registry. + +GitLab Container Registry is serving as a container for stored data, but on its own does not authenticate `docker login`. +The `docker login` is executed with user credentials (can be `personal access token`) or CI build credentials (ephemeral `ci_builds.token`). + +Container Registry uses data deduplication. +It means that the same blob (image layer) that is shared between many Projects is stored only once. +Each layer is hashed by `sha256`. + +The `docker login` does request a JWT time-limited authentication token that is signed by GitLab, but validated by Container Registry service. +The JWT token does store all authorized scopes (`container repository images`) and operation types (`push` or `pull`). +A single JWT authentication token can have many authorized scopes. +This allows Container Registry and client to mount existing blobs from other scopes. +GitLab responds only with authorized scopes. +Then it is up to GitLab Container Registry to validate if the given operation can be performed. + +The GitLab.com pages are always scoped to a Project. +Each Project can have many container registry images attached. + +Currently, on GitLab.com the actual registry service is served via `https://registry.gitlab.com`. + +The main identifiable problems are: + +- The authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com. +- The `https://registry.gitlab.com` that is run by an external service and uses its own data store. +- Data deduplication. The Cells architecture with registry run in a Cell would reduce efficiency of data storage. + +## 2. Data flow + +### 2.1. Authorization request that is send by `docker login` + +```shell +curl \ + --user "username:password" \ + "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" +``` + +Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. + +```json +{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} +``` + +### 2.2. Docker client fetching tags + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list + +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 +``` + +### 2.3. Docker client fetching blobs and manifests + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 +``` + +## 3. Proposal + +### 3.1. Shard Container Registry separately to Cells architecture + +Due to its extensive and in general highly scalable horizontal architecture it should be evaluated if the GitLab Container Registry should be run not in Cell, but in a Cluster and be scaled independently. +This might be easier, but would definitely not offer the same amount of data isolation. + +### 3.2. Run Container Registry within a Cell + +It appears that except `/jwt/auth` which would likely have to be processed by Router (to decode `scope`) the Container Registry could be run as a local service of a Cell. +The actual data at least in case of GitLab.com is not forwarded via registry, but rather served directly from Object Storage / CDN. + +Its design encodes container repository image in a URL that is easily routable. +It appears that we could re-use the same stateless Router service in front of Container Registry to serve manifests and blobs redirect. + +The only downside is increased complexity of managing standalone registry for each Cell, but this might be desired approach. + +## 4. Evaluation + +There do not seem to be any theoretical problems with running GitLab Container Registry in a Cell. +It seems that the service can be easily made routable to work well. +The practical complexities are around managing a complex service from an infrastructure side. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md b/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md new file mode 100644 index 00000000000..2053b87b125 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md @@ -0,0 +1,163 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Contributions: Forks' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: Contributions: Forks + +The [forking workflow](../../../../user/project/repository/forking_workflow.md) allows users to copy existing Project sources into their own namespace of choice (personal or Group). + +## 1. Definition + +The [forking workflow](../../../../user/project/repository/forking_workflow.md) is a common workflow with various usage patterns: + +- It allows users to contribute back to an upstream Project. +- It persists repositories into their personal namespace. +- Users can copy a Project to make changes and release it as a modified Project. + +Forks allow users not having write access to a parent Project to make changes. +The forking workflow is especially important for the open-source community to contribute back to public Projects. +However, it is equally important in some companies that prefer a strong split of responsibilities and tighter access control. +The access to a Project is restricted to a designated list of developers. + +Forks enable: + +- Tighter control of who can modify the upstream Project. +- Split of responsibilities: Parent Project might use CI configuration connecting to production systems. +- To run CI pipelines in the context of a fork in a much more restrictive environment. +- To consider all forks to be unvetted which reduces risks of leaking secrets, or any other information tied to the Project. + +The forking model is problematic in a Cells architecture for the following reasons: + +- Forks are clones of existing repositories. Forks could be created across different Organizations, Cells and Gitaly shards. +- Users can create merge requests and contribute back to an upstream Project. This upstream Project might be in a different Organization and Cell. +- The merge request CI pipeline is executed in the context of the source Project, but presented in the context of the target Project. + +## 2. Data exploration + +From a [data exploration](https://gitlab.com/gitlab-data/product-analytics/-/issues/1380), we retrieved the following information about existing forks: + +- Roughly 1.8m forks exist on GitLab.com at the moment. +- The majority of forks are under a personal namespace (82%). +- We were expecting a minimal use of forks within the same top-level Group and/or organization. Forking is only necessary for users who don't have permissions to access a Project. Inside companies we wouldn't expect teams to use forking workflows much unless they for some reason have different permissions across different team members. The data showed that only 9% of fork relationships have matching ultimate parent namespace identifiers (top-level Groups and personal namespaces). The other 91% of fork relationships are forked across different top-level namespaces. When trying to match top-level Groups to an identifiable company, we saw that: + - 3% of forked Projects are forked from an upstream Project in the same organization. + - 83% of forked Projects do not have an identifiable organization related to either up or downstream Project. + - The remaining 14% are forked from a source Project within a different company. +- 9% of top-level Groups (95k) with activity in the last 12 months have a project with a fork relationship, compared to 5% of top-level Groups (91k) with no activity in the last 12 months. We expect these top-level Groups to be impacted by Cells. + +## 3. Proposal - Forks are created in a dedicated contribution space of the current Organization + +Instead of creating Projects across Organizations, forks are created in a contribution space tied to the Organization. +A contribution space is similar to a personal namespace but rather than existing in the default Organization, it exists within the Organization someone is trying to contribute to. +Example: + +- Any User that can view an Organization (all Users for public Organizations) can create a contribution space in the Organization. This is a dedicated namespace where they can create forks of Projects in that Organization. For example for `Produce Inc.` it could be `gitlab.com/organization/produce-inc/@ayufan`. +- To create a contribution space we do not require membership of an Organization as this would prevent open source workflows where contributors are able to fork and create a merge request without ever being invited to a Group or Project. We strictly respect visibility, so Users would not be able to create a fork in a private Organization without first being invited. +- When creating a fork for a Project Users will only be presented with the option to create forks in Groups that are part of the Organization. We will also give Users the option to create a contribution space and put the fork there. Today there is also a "Create a group" option when creating a fork. This functionality would also be limited to creating a new group in the organization to store the new fork. +- In order to support Users that want to fork without contributing back we might consider an option to create [an unlinked fork](../../../../user/project/repository/forking_workflow.md#unlink-a-fork) in any namespace they have permission to write to. +- The User has as many contribution spaces as Organizations they contribute to. +- The User cannot create additional personal Projects within contribution spaces. Personal Projects can continue to be created in their personal namespace. +- The Organization can prevent or disable usage of contribution spaces. This would disable forking by anyone that does not belong to a Group within the Organization. +- All current forks are migrated into the contribution space of the User in an Organization. Because this may result in data loss when the fork also has links to data outside of the upstream Project we will also keep the personal Project around as archived and remove the fork relationship. +- All forks are part of the Organization. +- Forks are not federated features. +- The contribution space and forked Project do not share configuration with the parent Project. +- If the Organization is deleted, the Projects containing forks will be moved either to the default Organization or we'll create a new Organization to house them, which is essentially a ghost Organization of the former Organization. +- Data in contribution spaces do not contribute to customer usage from a billing perspective. +- Today we do not have organization-scoped runners but if we do implement that they will likely need special settings for how or if they can be used by contribution space projects. + +## 4. Alternative proposals considered + +### 4.1. Intra-cluster forks + +This proposal implements forks as intra-Cluster forks where communication is done via API between all trusted Cells of a cluster: + +- Forks are created always in the context of a user's choice of Group. +- Forks are isolated from the Organization. +- Organization or Group owner could disable forking across Organizations, or forking in general. +- A merge request is created in the context of the target Project, referencing the external Project on another Cell. +- To target Project the merge reference is transferred that is used for presenting information in context of the target Project. +- CI pipeline is fetched in the context of the source Project as it is today, the result is fetched into the merge request of the target Project. +- The Cell holding the target Project internally uses GraphQL to fetch the status of the source Project and includes in context of the information for merge request. + +Pros: + +- All existing forks continue to work as they are, as they are treated as intra-Cluster forks. + +Cons: + +- The purpose of Organizations is to provide strong isolation between Organizations. Allowing to fork across does break security boundaries. +- However, this is no different to the ability of users today to clone a repository to a local computer and push it to any repository of choice. +- Access control of the source Project can be lower than that of the target Project. Today, the system requires that in order to contribute back, the access level needs to be the same for fork and upstream Project. + +### 4.2. Forks are created as internal Projects under current Projects + +Instead of creating Projects across Organizations, forks are attachments to existing Projects. +Each user forking a Project receives their unique Project. +Example: + +- For Project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. +- Forks are created in the context of the current Organization, they do not cross Organization boundaries and are managed by the Organization. +- Tied to the user (or any other user-provided name of the fork). +- Forks are not federated features. + +Cons: + +- Does not answer how to handle and migrate all existing forks. +- Might share current Group/Project settings, which could be breaking some security boundaries. + +## 5. Evaluation + +### 5.1. Pros + +### 5.2. Cons + +## 6. Example + +As an example, we will demonstrate the impact of this proposal for the case that we move `gitlab-org/gitlab` to a different Organization. +`gitlab-org/gitlab` has [over 8K forks](https://gitlab.com/gitlab-org/gitlab/-/forks). + +### Does this direction impact the canonical URLs of those forks? + +Yes canonical URLs will change for forks. +Existing users that have forks in personal namespaces and want to continue contributing merge requests, will be required to migrate their fork to a new fork in a contribution space. +For example, a personal namespace fork at `https://gitlab.com/DylanGriffith/gitlab` will +need to be migrated to `https://gitlab.com/-/contributions/gitlab-inc/@DylanGriffith/gitlab`. +We may offer automated ways to move this, but manually the process would involve: + +1. Create the contribution space fork +1. Push your local branch from your original fork to the new fork +1. Recreate any merge request that was still open and you wanted to merge + +### Does it impact the Git URL of the repositories themselves? + +Yes. +In the above the example the Git URL would change from +`gitlab.com:DylanGriffith/gitlab.git` to `gitlab.com:/-/contributions/gitlab-inc/@DylanGriffith/gitlab.git`. + +### Would there be any user action required to accept their fork being moved within an Organization or towards a contribution space? + +If we offer an automated process we'd present this as an option for the user as they will become the new owner of the contribution space. + +### Can we make promises that we will not break the existing forks of public Projects hosted on GitLab.com? + +Existing fork projects will not be deleted but their fork relationship will be +removed when the source project is moved to another Organization. +The owner of the open source project will be made aware that they will disconnect their +forks when they move the project which will require them to close all existing +merge requests from those forks. +There will need to be some process for keeping the history from these merge requests while effectively losing the ability to +collaborate on them or merge them. + +In the case of `gitlab-org/gitlab` we will attempt to give as much notice of this process and make this process as transparent as possible. +When we make the decision to move this project to an Organization we will seek additional +feedback about what would be the minimum amount of automated migrations necessary to be acceptable here. +But the workflow for contributors will change after the move so this will be a punctuated event regardless. diff --git a/doc/architecture/blueprints/cells/impacted_features/data-migration.md b/doc/architecture/blueprints/cells/impacted_features/data-migration.md new file mode 100644 index 00000000000..9ff661ddf68 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/data-migration.md @@ -0,0 +1,99 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Data migration' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Data migration + +It is essential for a Cells architecture to provide a way to migrate data out of big Cells into smaller ones. +This document describes various approaches to provide this type of split. + +We also need to handle cases where data is already violating the expected isolation constraints of Cells, for example references cannot span multiple Organizations. +We know that existing features like linked issues allowed users to link issues across any Projects regardless of their hierarchy. +There are many similar features. +All of this data will need to be migrated in some way before it can be split across different Cells. +This may mean some data needs to be deleted, or the feature needs to be changed and modelled slightly differently before we can properly split or migrate Organizations between Cells. + +Having schema deviations across different Cells, which is a necessary consequence of different databases, will also impact our ability to migrate data between Cells. +Different schemas impact our ability to reliably replicate data across Cells and especially impact our ability to validate that the data is correctly replicated. +It might force us to only be able to move data between Cells when the schemas are all in sync (slowing down deployments and the rebalancing process) or possibly only migrate from newer to older schemas which would be complex. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +### 3.1. Split large Cells + +A single Cell can only be divided into many Cells. +This is based on the principle that it is easier to create an exact clone of an existing Cell in many replicas out of which some will be made authoritative once migrated. +Keeping those replicas up-to-date with Cell 0 is also much easier due to pre-existing replication solutions that can replicate the whole systems: Geo, PostgreSQL physical replication, etc. + +1. All data of an Organization needs to not be divided across many Cells. +1. Split should be doable online. +1. New Cells cannot contain pre-existing data. +1. N Cells contain exact replica of Cell 0. +1. The data of Cell 0 is live replicated to as many Cells it needs to be split. +1. Once consensus is achieved between Cell 0 and N-Cells, the Organizations to be migrated away are marked as read-only cluster-wide. +1. The `routes` is updated on for all Organizations to be split to indicate an authoritative Cell holding the most recent data, like `gitlab-org` on `cell-100`. +1. The data for `gitlab-org` on Cell 0, and on other non-authoritative N-Cells are dormant and will be removed in the future. +1. All accesses to `gitlab-org` on a given Cell are validated about `cell_id` of `routes` to ensure that given Cell is authoritative to handle the data. + +#### More challenges of this proposal + +1. There is no streaming replication capability for Elasticsearch, but you could + snapshot the whole Elasticsearch index and recreate, but this takes hours. + It could be handled by pausing Elasticsearch indexing on the initial Cell during + the migration as indexing downtime is not a big issue, but this still needs + to be coordinated with the migration process. +1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other + new data stores snapshots in an online system would likely lead to gaps + without a long downtime. You need to choose a sync point and at the sync + point you need to stop writes to perform the migration. The more data stores + there are to migrate at the same time the longer the write downtime for the + failover. We would also need to find a reliable place in the application to + actually block updates to all these systems with a high degree of + confidence. In the past we've only been confident by shutting down all Rails + services because any Rails process could write directly to any of these at + any time due to async workloads or other surprising code paths. +1. How to efficiently delete all the orphaned data. Locating all `ci_builds` + associated with half the Organizations would be very expensive if we have to + do joins. We haven't yet determined if we'd want to store an `organization_id` + column on every table, but this is the kind of thing it would be helpful for. + +### 3.2. Migrate Organization from an existing Cell + +This is different to split, as we intend to perform logical and selective replication of data belonging to a single Organization. +Today this type of selective replication is only implemented by Gitaly where we can migrate Git repository from a single Gitaly node to another with minimal downtime. + +In this model we would require identifying all resources belonging to a given Organization: database rows, object storage files, Git repositories, etc. and selectively copy them over to another (likely) existing Cell importing data into it. +Ideally ensuring that we can perform logical replication live of all changed data, but change similarly to split which Cell is authoritative for this Organization. + +1. It is hard to identify all resources belonging to an Organization. +1. It requires either downtime for the Organization or a robust system to identify live changes made. +1. It likely will require a full database structure analysis (more robust than Project import/export) to perform selective PostgreSQL logical replication. + +#### More challenges of this proposal + +1. Logical replication is still not performant enough to keep up with our + scale. Even if we could use logical replication we still don't have an + efficient way to filter data related to a single Organization without + joining all the way to the `organizations` table which will slow down + logical replication dramatically. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md b/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md new file mode 100644 index 00000000000..92de3df7c27 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md @@ -0,0 +1,39 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Data Pipeline Ingestion' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Data pipeline ingestion + +The Cells architecture will have a significant impact on the current [data pipeline](https://about.gitlab.com/handbook/business-technology/data-team/platform/pipelines/SAAS-Gitlab-com/) which exports data from Postgres to Snowflake for the use of data analytics. This data pipeline fulfils many use cases (i.e. SAAS Service ping, Gainsight metrics and Reporting and Analytics of the SAAS Platform). + +## 1. Definition + +## 2. Data flow + +The current data pipeline is limited by not having the possibility to get data via a CDC mechanism (which leads to data quality issues) and works by polling the Postgres database and looking for new and updated records or fully extracting data for certain tables which causes a lot of overhead. +At the moment the data pipeline runs against two instances that get created from a snapshot of both the `main` and `ci` databases. +This is done to avoid workload on the production databases. +In the Cells architecture there will be more Postgres instances because of which the current pipeline couldn't scale to pull data from all the Postgres instances. Requirements around the data pipeline moving forward are as follows: + +- We need a process that allows capturing all the CDC (insert, update and delete) from all Cells, scaling automatically with N number of Cells. +- We need to have (direct or indirect) access to database instances which allows it to do data catch up in case of major failure or root cause analysis for data anomalies. +- We need monitoring in place to alert any incident that can delay the data ingestion. + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/database-sequences.md b/doc/architecture/blueprints/cells/impacted_features/database-sequences.md new file mode 100644 index 00000000000..2aeaaed7d64 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/database-sequences.md @@ -0,0 +1,74 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Database Sequences' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Database Sequences + +GitLab today ensures that every database row create has a unique ID, allowing to access a merge request, CI Job or Project by a known global ID. +Cells will use many distinct and not connected databases, each of them having a separate ID for most entities. +At a minimum, any ID referenced between a Cell and the shared schema will need to be unique across the cluster to avoid ambiguous references. +Further to required global IDs, it might also be desirable to retain globally unique IDs for all database rows to allow migrating resources between Cells in the future. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +These are some preliminary ideas how we can retain unique IDs across the system. + +### 3.1. UUID + +Instead of using incremental sequences, use UUID (128 bit) that is stored in the database. + +- This might break existing IDs and requires adding a UUID column for all existing tables. +- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. + +### 3.2. Use Cell index encoded in ID + +Because a significant number of tables already use 64 bit ID numbers we could use MSB to encode the Cell ID: + +- This might limit the amount of Cells that can be enabled in a system, as we might decide to only allocate 1024 possible Cell numbers. +- This would make it possible to migrate IDs between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a lookup table. +- This requires updating all IDs to 32 bits. + +### 3.3. Allocate sequence ranges from central place + +Each Cell might receive its own range of sequences as they are consumed from a centrally managed place. +Once a Cell consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. +Ranges would be tracked to provide a faster lookup table if a random access pattern is required. + +- This might make IDs migratable between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a much more robust lookup table as we could be breaking previously assigned sequence ranges. +- This does not require updating all IDs to 64 bits. +- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server. +- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. + +### 3.4. Define only some tables to require unique IDs + +Maybe it is acceptable only for some tables to have a globally unique IDs. It could be Projects, Groups and other top-level entities. +All other tables like `merge_requests` would only offer a Cell-local ID, but when referenced outside it would rather use an IID (an ID that is monotonic in context of a given resource, like a Project). + +- This makes the ID 10000 for `merge_requests` be present on all Cells, which might be sometimes confusing regarding the uniqueness of the resource. +- This might make random access by ID (if ever needed) impossible without using a composite key, like: `project_id+merge_request_id`. +- This would require us to implement a transformation/generation of new ID if we need to migrate records to another Cell. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. +- If IDs need to change when moving between Cells this means that any links to records by ID would no longer work even if those links included the `project_id`. +- If we plan to allow these IDs to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/explore.md b/doc/architecture/blueprints/cells/impacted_features/explore.md new file mode 100644 index 00000000000..f839ce4be34 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/explore.md @@ -0,0 +1,71 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Explore' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Explore + +Explore may not play a critical role in GitLab as it functions today, but GitLab today is not isolated. It is the isolation that makes Explore or some viable replacement necessary. + +The existing Group and Project Explore will initially be scoped to an Organization. However, there is a need for a global Explore that spans across Organizations to support the discoverability of public Groups and Projects, in particular in the context of discovering open source Projects. See user feedback [here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582#note_1458298192) and [here](https://gitlab.com/gitlab-org/gitlab/-/issues/418228#note_1470045468). + +## 1. Definition + +The Explore functionality helps users in discovering Groups and Projects. Unauthenticated Users are only able to explore public Groups and Projects, authenticated Users can see all the Groups and Projects that they have access to, including private and internal Groups and Projects. + +## 2. Data flow + +## 3. Proposal + +The Explore feature problem falls under the broader umbrella of solving inter-Cell communication. [This topic warrants deeper research](../index.md#can-different-cells-communicate-with-each-other). + +Below are possible directions for further investigation. + +### 3.1. Read only table mirror + +- Create a `shared_projects` table in the shared cluster-wide database. +- The model for this table is read-only. No inserts/updates/deletes are allowed. +- The table is filled with data (or a subset of data) from the Projects Cell-local table. + - The write model Project (which is Cell-local) writes to the local database. We will primarily use this model for anything Cell-local. + - This data is synchronized with `shared_projects` via a background job any time something changes. + - The data in `shared_projects` is stored normalized, so that all the information necessary to display the Project Explore is there. +- The Project Explore (as of today) is part of an instance-wide functionality, since it's not namespaced to any organizations/groups. + - This section will read data using the read model for `shared_projects`. +- Once the user clicks on a Project, they are redirected to the Cell containing the Organization. + +Downsides: + +- Need to have an explicit pattern to access instance-wide data. This however may be useful for admin functionalities too. +- The Project Explore may not be as rich in features as it is today (various filtering options, role you have on that Project, etc.). +- Extra complexity in managing CQRS. + +### 3.2 Explore scoped to an Organization + +The Project Explore and Group Explore are scoped to an Organization. + +Downsides: + +- No global discoverability of Groups and Projects. + +## 4. Evaluation + +The existing Group and Project Explore will initially be scoped to an Organization. Considering the [current usage of the Explore feature](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521), we deem this acceptable. Since all existing Users, Groups and Projects will initially be part of the default Organization, Groups and Projects will remain explorable and accessible as they are today. Only once existing Groups and Projects are moved out of the default Organization into different Organizations will this become a noticeable problem. Solutions to mitigate this are discussed in [issue #418228](https://gitlab.com/gitlab-org/gitlab/-/issues/418228). Ultimately, Explore could be replaced with a better search experience altogether. + +## 4.1. Pros + +- Initially the lack of discoverability will not be a problem. +- Only around [1.5% of all exisiting Users are using the Explore functionality on a monthly basis](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521). + +## 4.2. Cons + +- The GitLab owned top-level Groups would be some of the first to be moved into their own Organization and thus be detached from the explorability of the default Organization. diff --git a/doc/architecture/blueprints/cells/impacted_features/git-access.md b/doc/architecture/blueprints/cells/impacted_features/git-access.md new file mode 100644 index 00000000000..611b4db5f43 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/git-access.md @@ -0,0 +1,156 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Git Access' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Git Access + +This document describes impact of Cells architecture on all Git access (over HTTPS and SSH) patterns providing explanation of how potentially those features should be changed to work well with Cells. + +## 1. Definition + +Git access is done throughout the application. +It can be an operation performed by the system (read Git repository) or by a user (create a new file via Web IDE, `git clone` or `git push` via command line). +The Cells architecture defines that all Git repositories will be local to the Cell, so no repository could be shared with another Cell. + +The Cells architecture will require that any Git operation can only be handled by a Cell holding the data. +It means that any operation either via Web interface, API, or GraphQL needs to be routed to the correct Cell. +It means that any `git clone` or `git push` operation can only be performed in the context of a Cell. + +## 2. Data flow + +The are various operations performed today by GitLab on a Git repository. +This describes the data flow how they behave today to better represent the impact. + +It appears that Git access does require changes only to a few endpoints that are scoped to a Project. +There appear to be different types of repositories: + +- Project: assigned to Group +- Wiki: additional repository assigned to Project +- Design: similar to Wiki, additional repository assigned to Project +- Snippet: creates a virtual Project to hold repository, likely tied to the User + +### 2.1. Git clone over HTTPS + +Execution of: `git clone` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC InfoRefsUploadPack + Gitaly ->> User: Response + User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack + Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.2. Git clone over SSH + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.3. Git push over HTTPS + +Execution of: `git push` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC PostReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.4. Git push over SSHD + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC ReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.5. Create commit via Web + +Execution of `Add CHANGELOG` to repository: + +```mermaid +sequenceDiagram + Web ->> Puma: POST /gitlab-org/gitlab/-/create/main + Puma ->> Gitaly: RPC TreeEntry + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> Puma: Response + Puma ->> Web: See CHANGELOG +``` + +## 3. Proposal + +The Cells stateless router proposal requires that any ambiguous path (that is not routable) will be made routable. +It means that at least the following paths will have to be updated to introduce a routable entity (Project, Group, or Organization). + +Change: + +- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` +- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` +- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` +- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` + +Where: + +- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) +- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) + +## 4. Evaluation + +Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. +The one major complication is supporting snippets, but this likely falls in the same category as for the approach to support a user's Personal Namespace. + +## 4.1. Pros + +1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. + +## 4.2. Cons + +1. The sharing of repositories objects is limited to the given Cell and Gitaly node. +1. Cross-Cells forks are likely impossible to be supported (discover: How this works today across different Gitaly node). diff --git a/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md b/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md new file mode 100644 index 00000000000..7e4ab785095 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: GitLab Pages' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GitLab Pages + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/global-search.md b/doc/architecture/blueprints/cells/impacted_features/global-search.md new file mode 100644 index 00000000000..475db381ff5 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/global-search.md @@ -0,0 +1,35 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Global search' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Global search + +When we introduce multiple Cells we intend to isolate all services related to those Cells. +This will include Elasticsearch which means our current global search functionality will not work. +It may be possible to implement aggregated search across all Cells, but it is unlikely to be performant to do fan-out searches across all Cells especially once you start to do pagination which requires setting the correct offset and page number for each search. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +Likely the first versions of Cells will not support global searches. +Later, we may consider if building global searches to support popular use cases is worthwhile. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/graphql.md b/doc/architecture/blueprints/cells/impacted_features/graphql.md new file mode 100644 index 00000000000..e8850dfbee3 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/graphql.md @@ -0,0 +1,81 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: GraphQL' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GraphQL + +GitLab extensively uses GraphQL to perform efficient data query operations. +GraphQL due to it's nature is not directly routable. +The way GitLab uses it calls the `/api/graphql` endpoint, and only the query or mutation of the body request might define where the data can be accessed. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +There are at least two main ways to implement GraphQL in a Cells architecture. + +### 3.1. GraphQL routable by endpoint + +Change `/api/graphql` to `/api/organization/<organization>/graphql`. + +- This breaks all existing usages of `/api/graphql` endpoint because the API URI is changed. + +### 3.2. GraphQL routable by body + +As part of router parse GraphQL body to find a routable entity, like `project`. + +- This still makes the GraphQL query be executed only in context of a given Cell and not allowing the data to be merged. + +```json +# Good example +{ + project(fullPath:"gitlab-org/gitlab") { + id + description + } +} + +# Bad example, since Merge Request is not routable +{ + mergeRequest(id: 1111) { + iid + description + } +} +``` + +### 3.3. Merging GraphQL Proxy + +Implement as part of router GraphQL Proxy which can parse body and merge results from many Cells. + +- This might make pagination hard to achieve, or we might assume that we execute many queries of which results are merged across all Cells. + +```json +{ + project(fullPath:"gitlab-org/gitlab"){ + id, description + } + group(fullPath:"gitlab-com") { + id, description + } +} +``` + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/organizations.md b/doc/architecture/blueprints/cells/impacted_features/organizations.md new file mode 100644 index 00000000000..9fc8241e852 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/organizations.md @@ -0,0 +1,36 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Organizations' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Organizations + +One of the major designs of a Cells architecture is strong isolation between Groups. +Organizations as described by the [Organization blueprint](../../organization/index.md) provides a way to have plausible UX for joining together many Groups that are isolated from the rest of the system. + +## 1. Definition + +Cells do require that all Groups and Projects of a single Organization can only be stored on a single Cell because a Cell can only access data that it holds locally and has very limited capabilities to read information from other Cells. + +Cells with Organizations do require strong isolation between Organizations. + +It will have significant implications on various user-facing features, like Todos, dropdowns allowing to select Projects, references to other issues or Projects, or any other social functions present at GitLab. +Today those functions were able to reference anything in the whole system. +With the introduction of Organizations this will be forbidden. + +This problem definition aims to answer effort and implications required to add strong isolation between Organizations to the system, including features affected and their data processing flow. +The purpose is to ensure that our solution when implemented consistently avoids data leakage between Organizations residing on a single Cell. + +## 2. Proposal + +See the [Organization blueprint](../../organization/index.md). diff --git a/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md b/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md new file mode 100644 index 00000000000..3aca9f1e116 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md @@ -0,0 +1,31 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Personal Access Tokens' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Personal Access Tokens + +## 1. Definition + +Personal Access Tokens associated with a User are a way for Users to interact with the API of GitLab to perform operations. +Personal Access Tokens today are scoped to the User, and can access all Groups that a User has access to. + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md new file mode 100644 index 00000000000..55d974bb351 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md @@ -0,0 +1,71 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Personal Namespaces' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: Personal Namespaces + +Personal Namespaces do not easily fit with our overall architecture in Cells because the Cells architecture depends on all data belonging to a single Organization. +When Users are allowed to work across multiple Organizations there is no natural fit for picking a single Organization to store personal Namespaces and their Projects. + +One important engineering constraint in Cells will be that data belonging to some Organization should not be linked to data belonging to another Organization. +And specifically that functionality in GitLab can be scoped to a single Organization at a time. +This presents a challenge for personal Namespaces as forking is one of the important workloads for personal Namespaces. +Functionality related to forking and the UI that presents forked MRs to users will often require data from both the downstream and upstream Projects at the same time. +Implementing such functionality would be very difficult if that data belonged in different Organizations stored on different +Cells. +This is especially the case with the merge request, as it is one of the most complicated and performance critical features in GitLab. + +Today personal Namespaces serve two purposes that are mostly non-overlapping: + +1. They provide a place for users to create personal Projects + that aren't expected to receive contributions from other people. This use case saves them from having to create a Group just for themselves. +1. They provide a default place for a user to put any forks they + create when contributing to Projects where they don't have permission to push a branch. This again saves them from needing to create a Group just to store these forks. But the primary user need here is because they can't push branches to the upstream Project so they create a fork and contribute merge requests from the fork. + +## 1. Definition + +A [personal Namespace](../../../../user/namespace/index.md#types-of-namespaces) is based on a username and provided when a user creates an account. +Users can create [personal Projects](../../../../user/project/working_with_projects.md#view-personal-projects) under their personal Namespace. + +## 2. Data flow + +## 3. Proposal + +As described above, personal Namespaces serve two purposes today: + +1. A place for users to store their Projects to save them from creating a Group. +1. A place for users to store forks when they want to contribute to a Project where they don't have permission to push a branch. + +In this proposal we will only focus on (1) and assume that (2) will be replaced by suitable workflows described in [Cells: Contributions: Forks](../impacted_features/contributions-forks.md). + +Since we plan to move away from using personal Namespaces as a home for storing forks, we can assume that the main remaining use case does not need to support cross-Organization linking. +In this case the easiest thing to do is to keep all personal Namespaces in the default Organization. +Depending on the amount of workloads happening in personal Namespaces we may be required in the future to migrate them to different Cells. +This may necessitate that they all get moved to some Organization created just for the user. +If we go this route, there may be breakage similar to what will happen to when we move Groups or Projects into their own Organization, though the full impact may need further investigation. + +This decision, however, means that existing personal Namespaces that were used as forks to contribute to some upstream Project will become disconnected from the upstream as soon as the upstream moves into an Organization. +On GitLab.com 10% of all projects in personal Namespaces are forks. +This may be a slightly disruptive workflow but as long as the forks are mainly just storing branches used in merge requests then it may be reasonable to ask the affected users to recreate the fork in the context of the Organization. + +For existing Users, we suggest to keep their existing personal Namespaces in the default Organization. +New Users joining an Organization other than the default Organization will also have their personal Namespace hosted on the default Organization. Having all personal Namespaces in the default Organization means we don't need to worry about deletion of the parent organization and the impact of that on personal Namespaces, which would be the case if they existed in other organizations. +This implies that all Users will have an association to the default Organization via their personal Namespace, requiring them to switch to the default Organization to access their personal Namespace. + +We will further explore the idea of a `contribution space` to give Users a place to store forks when they want to contribute to a Project where they don't have permission to push a branch. +That discussion will be handled as part of the larger discussion of the [Cells impact on forks](../impacted_features/contributions-forks.md). + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md b/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md new file mode 100644 index 00000000000..d403d6ff963 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md @@ -0,0 +1,34 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Router Endpoints Classification' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Router Endpoints Classification + +Classification of all endpoints is essential to properly route requests hitting the load balancer of a GitLab installation to a Cell that can serve it. +Each Cell should be able to decode each request and classify which Cell it belongs to. + +GitLab currently implements hundreds of endpoints. +This document tries to describe various techniques that can be implemented to allow the Rails to provide this information efficiently. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/schema-changes.md b/doc/architecture/blueprints/cells/impacted_features/schema-changes.md new file mode 100644 index 00000000000..0d6ca4717d3 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/schema-changes.md @@ -0,0 +1,38 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Schema changes' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Schema changes + +When we introduce multiple Cells that own their own databases this will complicate the process of making schema changes to Postgres and Elasticsearch. +Today we already need to be careful to make changes comply with our zero downtime deployments. +For example, [when removing a column we need to make changes over 3 separate deployments](../../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). +We have tooling like `post_migrate` that helps with these kinds of changes to reduce the number of merge requests needed, but these will be complicated when we are dealing with deploying multiple Rails applications that will be at different versions at any one time. +This problem will be particularly tricky to solve for shared databases like our plan to share the `users` related tables among all Cells. + +A key benefit of Cells may be that it allows us to run different customers on different versions of GitLab. +We may choose to update our own Cell before all our customers giving us even more flexibility than our current canary architecture. +But doing this means that schema changes need to have even more versions of backward compatibility support which could slow down development as we need extra steps to make schema changes. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/secrets.md b/doc/architecture/blueprints/cells/impacted_features/secrets.md new file mode 100644 index 00000000000..681c229711d --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/secrets.md @@ -0,0 +1,43 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Secrets' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Secrets + +Where possible, each Cell should have its own distinct set of secrets. +However, there will be some secrets that will be required to be the same for all Cells in the cluster. + +## 1. Definition + +GitLab has a lot of [secrets](https://docs.gitlab.com/charts/installation/secrets.html) that need to be configured. +Some secrets are for inter-component communication, for example, `GitLab Shell secret`, and used only within a Cell. +Some secrets are used for features, for example, `ci_jwt_signing_key`. + +## 2. Data flow + +## 3. Proposal + +1. Secrets used for features will need to be consistent across all Cells, so that the UX is consistent. + 1. This is especially true for the `db_key_base` secret which is used for + encrypting data at rest in the database - so that Projects that are + transferred to another Cell will continue to work. We do not want to have + to re-encrypt such rows when we move Projects/Groups between Cells. +1. Secrets which are used for intra-Cell communication only should be uniquely generated + per Cell. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/snippets.md b/doc/architecture/blueprints/cells/impacted_features/snippets.md new file mode 100644 index 00000000000..f77879907df --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/snippets.md @@ -0,0 +1,56 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Snippets' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Snippets + +Snippets will be scoped to an Organization. Initially it will not be possible to aggregate snippet collections across Organizations. See also [issue #416954](https://gitlab.com/gitlab-org/gitlab/-/issues/416954). + +## 1. Definition + +Two different types of snippets exist: + +- [Project snippets](../../../../api/project_snippets.md). These snippets have URLs + like `/<group>/<project>/-/snippets/123` +- [Personal snippets](../../../../user/snippets.md). These snippets have URLs like + `/-/snippets/123` + +Snippets are backed by a Git repository. + +## 2. Data flow + +## 3. Proposal + +### 3.1. Scoped to an organization + +Both project and personal snippets will be scoped to an Organization. + +- Project snippets URLs will remain unchanged, as the URLs are routable. +- Personal snippets URLs will need to change to be `/-/organizations/<organization>/snippets/123`, + so that the URL is routeable + +Creation of snippets will also be scoped to a User's current Organization. Because of that, we recommend renaming `personal snippets` to `organization snippets` once the Organization is rolled out. A User can create many independent snippet collections across multiple Organizations. + +## 4. Evaluation + +Snippets are scoped to an Organization because Gitaly is confined to a Cell. + +## 4.1. Pros + +- No need to have clusterwide Gitaly. + +## 4.2. Cons + +- We will break [snippet discovery](/ee/user/snippets.md#discover-snippets). +- Snippet access may become subordinate to the visibility of the Organization. diff --git a/doc/architecture/blueprints/cells/impacted_features/template.md b/doc/architecture/blueprints/cells/impacted_features/template.md new file mode 100644 index 00000000000..3cece3dc99e --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/template.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Problem A' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: A + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/uploads.md b/doc/architecture/blueprints/cells/impacted_features/uploads.md new file mode 100644 index 00000000000..fdac3a9977c --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/uploads.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Uploads' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Uploads + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/user-profile.md b/doc/architecture/blueprints/cells/impacted_features/user-profile.md new file mode 100644 index 00000000000..fac8552e6d6 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/user-profile.md @@ -0,0 +1,53 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: User Profile' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: User Profile + +The existing User Profiles will initially be scoped to an Organization. Long-term, we should consider aggregating parts of the User activity across Organizations to enable Users a global view of their contributions. + +## 1. Definition + +Each GitLab account has a [User Profile](../../../../user/profile/index.md), which contains information about the User and their GitLab activity. + +## 2. Data flow + +## 3. Proposal + +User Profiles will be scoped to an Organization. We follow the same pattern as is used for `Your Work`, meaning that profiles are always seen in the context of an Organization. + +- User Profile URLs will reference the Organization with the following URL structure `/-/organizations/<organization>/username`. +- Users can set a Home Organization as their main Organization. +- The default User Profile URL `/<username>` will refer to the user's Home Organization, or the default Organization if the user's Home Organization is not set. +- Users who do not exist in the database at all display a 404 not found error when trying to access their User Profile. +- User who haven't contributed to an Organization display their User Profile with an empty state. +- When displaying a User Profile empty state, if the profile has a Home Organization set to another Organization, we display a call-to-action allowing navigation to the main Organization. +- Breadcrumbs on the User Profile will present as `[Organization Name] / [Username]`. + +See [issue #411931](https://gitlab.com/gitlab-org/gitlab/-/issues/411931) for design proposals. + +## 4. Evaluation + +We expect the [majority of Users to perform most of their activity in one single Organization](../../organization/index.md#data-exploration). +This is why we deem it acceptable to scope the User Profile to an Organization at first. +More discovery is necessary to understand which aspects of the current User Profile are relevant to showcase contributions in a global context. + +## 4.1. Pros + +- Viewing a User Profile scoped to an Organization allows you to focus on contributions that are most relevant to your Organization, filtering out the User's other activities. +- Existing User Profile URLs do not break. + +## 4.2. Cons + +- Users will lose the ability to display their entire activity, which may lessen the effectiveness of using their User Profile as a resume of achievements when working across multiple Organizations. diff --git a/doc/architecture/blueprints/cells/impacted_features/your-work.md b/doc/architecture/blueprints/cells/impacted_features/your-work.md new file mode 100644 index 00000000000..5a72d50fbea --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/your-work.md @@ -0,0 +1,60 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Your Work' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Your Work + +Your Work will be scoped to an Organization. +Counts presented in the individual dashboards will relate to the selected Organization. + +## 1. Definition + +When accessing `gitlab.com/dashboard/`, users can find a [focused view of items that they have access to](../../../../tutorials/left_sidebar/index.md#use-a-more-focused-view). +This overview contains dashboards relating to: + +- Projects +- Groups +- Issues +- Merge requests +- To-Do list +- Milestones +- Snippets +- Activity +- Workspaces +- Environments +- Operations +- Security + +## 2. Data flow + +## 3. Proposal + +Your Work will be scoped to an Organization, giving the user an overview of all the items they can access in the Organization they are currently viewing. + +- Issue, Merge request and To-Do list counts will refer to the selected Organization. +- The URL will reference the Organization with the following URL structure `/-/organizations/<organization>/dashboard`. +- The default URL `/dashboard` will refer to the [Home Organization](../impacted_features/user-profile.md#3-proposal). + +## 4. Evaluation + +Scoping Your Work to an Organization makes sense in the context of the [proposed Organization navigation](https://gitlab.com/gitlab-org/gitlab/-/issues/417778). +Considering that [we expect most users to work in a single Organization](../../organization/index.md#data-exploration), we deem this impact acceptable. + +## 4.1. Pros + +- Viewing Your Work scoped to an Organization allows Users to focus on content that is most relevant to their currently selected Organization. + +## 4.2. Cons + +- Users working across multiple Organizations will have to navigate to each Organization to access all of their work items. diff --git a/doc/architecture/blueprints/cells/index.md b/doc/architecture/blueprints/cells/index.md index 0e93b9d5d3b..28414f9b68c 100644 --- a/doc/architecture/blueprints/cells/index.md +++ b/doc/architecture/blueprints/cells/index.md @@ -1,5 +1,5 @@ --- -status: accepted +status: ongoing creation-date: "2022-09-07" authors: [ "@ayufan", "@fzimmer", "@DylanGriffith", "@lohrc", "@tkuah" ] coach: "@ayufan" @@ -18,9 +18,7 @@ Cells is a new architecture for our software as a service platform. This archite For more information about Cells, see also: -- [Glossary](glossary.md) -- [Goals](goals.md) -- [Cross-section impact](impact.md) +- [Goals, Glossary and Requirements](goals.md) ## Work streams @@ -101,6 +99,10 @@ The first 2-3 quarters are required to define a general split of data and build The purpose is to perform a targeted decomposition of `users` and `projects`, because `projects` will be stored locally in the Cell. +1. **User can create Organization on Cell 2.** + + The purpose is to create Organizations that are isolated from each other. + 1. **User can change profile avatar that is shared in cluster.** The purpose is to fix global uploads that are shared in cluster. @@ -166,6 +168,11 @@ For example: The routing service needs to be able to discover and monitor the health of all Cells. +1. **User can use single domain to interact with many Cells.** + + The routing service will intelligently route all requests to Cells based on the resource being + accessed versus the Cell containing the data. + 1. **Router endpoints classification.** The stateless routing service will fetch and cache information about endpoints from one of the Cells. @@ -258,28 +265,30 @@ One iteration describes one quarter's worth of work. - Data access layer: Initial Admin Area settings are shared across cluster. - Essential workflows: Allow to share cluster-wide data with database-level data access layer -1. [Iteration 2](https://gitlab.com/groups/gitlab-org/-/epics/9813) - FY24Q2 - In progress +1. [Iteration 2](https://gitlab.com/groups/gitlab-org/-/epics/9813) - Expected delivery: 16.2 FY24Q2 | Actual delivery: 16.4 FY24Q3 - In progress - Essential workflows: User accounts are shared across cluster. - Essential workflows: User can create Group. -1. [Iteration 3](https://gitlab.com/groups/gitlab-org/-/epics/10997) - FY24Q3 - Planned +1. [Iteration 3](https://gitlab.com/groups/gitlab-org/-/epics/10997) - Expected delivery: 16.7 FY24Q4 - Planned - Essential workflows: User can create Project. - Routing: Technology. - - Data access layer: Evaluate the efficiency of database-level access vs. API-oriented access layer + - Routing: Cell discovery. + - Data access layer: Evaluate the efficiency of database-level access vs. API-oriented access layer. + - Data access layer: Data access layer. -1. [Iteration 4](https://gitlab.com/groups/gitlab-org/-/epics/10998) - FY24Q4 +1. [Iteration 4](https://gitlab.com/groups/gitlab-org/-/epics/10998) - Expected delivery: 16.10 FY25Q1 - Planned - - Essential workflows: User can push to Git repository. - - Essential workflows: User can create issue, merge request, and merge it after it is green. + - Essential workflows: User can create organization on Cell 2. - Data access layer: Cluster-unique identifiers. - - Routing: Cell discovery. - - Routing: Router endpoints classification. - - Cell deployment: Extend GitLab Dedicated to support GCP + - Routing: User can use single domain to interact with many Cells. + - Cell deployment: Extend GitLab Dedicated to support GCP. -1. Iteration 5 - FY25Q1 +1. Iteration 5..N - starting FY25Q1 + - Essential workflows: User can push to Git repository. + - Essential workflows: User can create issue, merge request, and merge it after it is green. - Essential workflows: User can run CI pipeline. - Essential workflows: Instance-wide settings are shared across cluster. - Essential workflows: User can change profile avatar that is shared in cluster. @@ -287,22 +296,13 @@ One iteration describes one quarter's worth of work. - Essential workflows: User can manage Group and Project members. - Essential workflows: User can manage instance-wide runners. - Essential workflows: User is part of Organization and can only see information from the Organization. + - Routing: Router endpoints classification. - Routing: GraphQL and other ambiguous endpoints. - Data access layer: Allow to share cluster-wide data with database-level data access layer. - Data access layer: Cluster-wide deletions. - - Data access layer: Data access layer. - Data access layer: Database migrations. -1. Iteration 6 - FY25Q2 - - TBD - -1. Iteration 7 - FY25Q3 - - TBD - -1. Iteration 8 - FY25Q4 - - TBD - -## Technical Proposals +## Technical proposals The Cells architecture has long lasting implications to data processing, location, scalability and the GitLab architecture. This section links all different technical proposals that are being evaluated. @@ -315,34 +315,36 @@ This section links all different technical proposals that are being evaluated. The Cells architecture will impact many features requiring some of them to be rewritten, or changed significantly. Below is a list of known affected features with preliminary proposed solutions. -- [Cells: Admin Area](cells-feature-admin-area.md) -- [Cells: Backups](cells-feature-backups.md) -- [Cells: CI Runners](cells-feature-ci-runners.md) -- [Cells: Container Registry](cells-feature-container-registry.md) -- [Cells: Contributions: Forks](cells-feature-contributions-forks.md) -- [Cells: Database Sequences](cells-feature-database-sequences.md) -- [Cells: Data Migration](cells-feature-data-migration.md) -- [Cells: Explore](cells-feature-explore.md) -- [Cells: Git Access](cells-feature-git-access.md) -- [Cells: Global Search](cells-feature-global-search.md) -- [Cells: GraphQL](cells-feature-graphql.md) -- [Cells: Organizations](cells-feature-organizations.md) -- [Cells: Secrets](cells-feature-secrets.md) -- [Cells: Snippets](cells-feature-snippets.md) -- [Cells: User Profile](cells-feature-user-profile.md) -- [Cells: Your Work](cells-feature-your-work.md) +- [Cells: Admin Area](impacted_features/admin-area.md) +- [Cells: Backups](impacted_features/backups.md) +- [Cells: CI Runners](impacted_features/ci-runners.md) +- [Cells: Container Registry](impacted_features/container-registry.md) +- [Cells: Contributions: Forks](impacted_features/contributions-forks.md) +- [Cells: Database Sequences](impacted_features/database-sequences.md) +- [Cells: Data Migration](impacted_features/data-migration.md) +- [Cells: Explore](impacted_features/explore.md) +- [Cells: Git Access](impacted_features/git-access.md) +- [Cells: Global Search](impacted_features/global-search.md) +- [Cells: GraphQL](impacted_features/graphql.md) +- [Cells: Organizations](impacted_features/organizations.md) +- [Cells: Personal Namespaces](impacted_features/personal-namespaces.md) +- [Cells: Secrets](impacted_features/secrets.md) +- [Cells: Snippets](impacted_features/snippets.md) +- [Cells: User Profile](impacted_features/user-profile.md) +- [Cells: Your Work](impacted_features/your-work.md) ### Impacted features: Placeholders The following list of impacted features only represents placeholders that still require work to estimate the impact of Cells and develop solution proposals. -- [Cells: Agent for Kubernetes](cells-feature-agent-for-kubernetes.md) -- [Cells: GitLab Pages](cells-feature-gitlab-pages.md) -- [Cells: Personal Access Tokens](cells-feature-personal-access-tokens.md) -- [Cells: Personal Namespaces](cells-feature-personal-namespaces.md) -- [Cells: Router Endpoints Classification](cells-feature-router-endpoints-classification.md) -- [Cells: Schema changes (Postgres and Elasticsearch migrations)](cells-feature-schema-changes.md) -- [Cells: Uploads](cells-feature-uploads.md) +- [Cells: Agent for Kubernetes](impacted_features/agent-for-kubernetes.md) +- [Cells: CI/CD Catalog](impacted_features/ci-cd-catalog.md) +- [Cells: Data pipeline ingestion](impacted_features/data-pipeline-ingestion.md) +- [Cells: GitLab Pages](impacted_features/gitlab-pages.md) +- [Cells: Personal Access Tokens](impacted_features/personal-access-tokens.md) +- [Cells: Router Endpoints Classification](impacted_features/router-endpoints-classification.md) +- [Cells: Schema changes (Postgres and Elasticsearch migrations)](impacted_features/schema-changes.md) +- [Cells: Uploads](impacted_features/uploads.md) - ... ## Frequently Asked Questions @@ -367,6 +369,59 @@ For example, users on GitLab Dedicated don't have to have a different and unique Up until iteration 3, Cells communicate with each other only via a shared database that contains common data. In iteration 4 we are going to evaluate the option of Cells calling each other via API to provide more isolation and reliability. +### How are Cells provisioned? + +The GitLab.com cluster of Cells will use GitLab Dedicated instances. +Once a GitLab Dedicated instance gets provisioned it could join the GitLab.com cluster and become a Cell. +One requirement will be that the GitLab Dedicated instance does not contain any prior data. + +To reach shared resources, Cells will use [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect). + +See also the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641). + +### What is a Cells topology? + +See the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641). + +### How are users of an Organization routed to the correct Cell? + +TBD + +### How do users authenticate with Cells and Organizations? + +See the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/395736). + +### How are Cells rebalanced? + +TBD + +### How can Cells implement disaster recovery capabilities? + +TBD + +### How do I decide whether to move my feature to the cluster, Cell or Organization level? + +By default, features are required to be scoped to the Organization level. Any deviation from that rule should be validated and approved by Tenant Scale. + +The design goals of the Cells architecture describe that [all Cells are under a single domain](goals.md#all-cells-are-under-a-single-gitlabcom-domain) and as such, Cells are invisible to the user: + +- Cell-local features should be limited to those related to managing the Cell, but never be a feature where the Cell semantic is exposed to the customer. +- The Cells architecture wants to freely control the distribution of Organization and customer data across Cells without impacting users when data is migrated. + +Cluster-wide features are strongly discouraged because: + +- They might require storing a substantial amount of data cluster-wide which decreases [scalability headroom](goals.md#provides-100x-headroom). +- They might require implementation of non-trivial [data aggregation](goals.md#aggregation-of-cluster-wide-data) that reduces resilience to [single node failure](goals.md#high-resilience-to-a-single-cell-failure). +- They are harder to build due to the need of being able to run [mixed deployments](goals.md#cells-running-in-mixed-deployments). Cluster-wide features need to take this into account. +- They might affect our ability to provide an [on-premise like experience on GitLab.com](goals.md#on-premise-like-experience). +- Some features that are expected to be cluster-wide might in fact be better implemented using federation techniques that use trusted intra-cluster communication using the same user identity. User Profile is shared across the cluster. +- The Cells architecture limits what services can be considered cluster-wide. Services that might initially be cluster-wide are still expected to be split in the future to achieve full service isolation. No feature should be built to depend on such a service (like Elasticsearch). + +### Will Cells use the [reference architecture for 50,000 users](../../../administration/reference_architectures/50k_users.md)? + +The infrastructure team will properly size Cells depending on the load. +The Tenant Scale team sees an opportunity to use GitLab Dedicated as a base for Cells deployment. + ## Decision log - 2022-03-15: Google Cloud as the cloud service. For details, see [issue 396641](https://gitlab.com/gitlab-org/gitlab/-/issues/396641#note_1314932272). @@ -378,3 +433,4 @@ In iteration 4 we are going to evaluate the option of Cells calling each other v - [Database group investigation](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/doc/root-namespace-sharding.html) - [Shopify Pods architecture](https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale) - [Opstrace architecture](https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/architecture/overview.md) +- [Adding Diagrams to this blueprint](diagrams/index.md) diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md index c1ca0c60dcd..847532a36dc 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md @@ -27,10 +27,10 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Cells but instead they see -different data dependent on their chosen "organization". -[Organizations](glossary.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which cell, since an -organization can only be on a single cell. +different data dependent on their chosen Organization. +[Organizations](goals.md#organizations) will be a new entity introduced to enforce isolation in the +application and allow us to decide which request routes to which Cell, since an +Organization can only be on a single Cell. ## Differences diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md index 3b3d481914f..962f71673df 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md @@ -27,10 +27,10 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Cells but instead they see -different data dependent on their chosen "organization". -[Organizations](glossary.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which cell, since an -organization can only be on a single cell. +different data dependent on their chosen Organization. +[Organizations](goals.md#organizations) will be a new entity introduced to enforce isolation in the +application and allow us to decide which request routes to which Cell, since an +Organization can only be on a single Cell. ## Differences diff --git a/doc/architecture/blueprints/code_search_with_zoekt/diagrams/sharding_proposal_2023-08.drawio.png b/doc/architecture/blueprints/code_search_with_zoekt/diagrams/sharding_proposal_2023-08.drawio.png Binary files differnew file mode 100644 index 00000000000..c45745c9dd2 --- /dev/null +++ b/doc/architecture/blueprints/code_search_with_zoekt/diagrams/sharding_proposal_2023-08.drawio.png diff --git a/doc/architecture/blueprints/code_search_with_zoekt/index.md b/doc/architecture/blueprints/code_search_with_zoekt/index.md index 273d8da482c..5b34d84b47e 100644 --- a/doc/architecture/blueprints/code_search_with_zoekt/index.md +++ b/doc/architecture/blueprints/code_search_with_zoekt/index.md @@ -273,6 +273,57 @@ progress and reassess if it turns out to add too much complexity to GitLab. This is already implemented as the `::Zoekt::IndexedNamespace` implements a many-to-many relationship between namespaces and shards. +#### Sharding proposal with self-registering Zoekt nodes + +This proposal is mostly inspired by GitLab Runner's architecture with the main difference +that the communication is bidirectional. We've arrived to this after discussions in [Zoekt Sharding and Replication](https://gitlab.com/gitlab-org/gitlab/-/issues/419900). + +##### Alternatives we've considered + +We've considered different options for where to manage the Zoekt cluster state including Raft and Zoekt's own database. We decided that there are many benefits to having the whole cluster state managed by GitLab instead of Zoekt so we're opting to keep Zoekt nodes as naive as possible. + +The main benefits are: + +1. The deployment cycle for GitLab is faster than Zoekt which requires many version bumps across many projects +1. GitLab already has lots tooling that can be used for managing the state that we are already familiar with including Postgres, Redis, Sidekiq and others +1. The engineers mainly working on this project have much more experience with Rails than Go and spend more time writing Rails code than Go code as the other search features are mostly in Rails + +Some of those benefits could also be seen as downsides and maybe not the right choice for different projects owned by different teams. + +##### High level proposal + +<img src="diagrams/sharding_proposal_2023-08.drawio.png" height="600"> + +1. Zoekt nodes are started with 3 additional arguments: its own address, shard name, and GitLab URL. +1. We'd like to keep shard name separate so that one will be able to migrate a shard to a different address. +1. When Zoekt is running in k8s, we can pass `hostname --fqdn` (for example, `gitlab-zoekt-1.gitlab-zoekt.default.svc.cluster.local`) as an argument for the address. Customers running Zoekt on bare-metal will need to configure it separately. +1. Zoekt most likely will use [Internal API](../../../development/internal_api/index.md) to connect to GitLab. We might also want to use a separate GitLab URL to keep the traffic internal and to avoid extra traffic cost. +1. GitLab will maintain a lookup table with `last_seen_at` and shard's name (we could expand `::Zoekt::Shard`). We'll also need to introduce the concept of replicas and primaries. +1. Zoekt nodes (indexers in this case) will send periodic requests to get new jobs with its address and name to the configured GitLab URL. GitLab will either register a new node or update the existing record in the lookup table. +1. After the job is completed, `zoekt-indexer` will send a callback to GitLab to indicate that the job has been completed. +1. If after a specified time GitLab doesn't receive a request, it can reassign namespaces to different shards and mark the missing shard as unavailable. +1. When executing searches, we can round-robin requests to primaries and replicas. We might even want to implement retries. For example, if a request to primary fails, we send another request to replica right away or vice versa. Here is a related issue: [Consider circuit breaker for Zoekt code search](https://gitlab.com/gitlab-org/gitlab/-/issues/393445). +1. Initially, we might want to skip replication until we implement efficiently moving and copying index files between shards (rsync for example). +1. Rebalancing most likely will happen in a cron Sidekiq worker, which will consider if an indexed namespace has enough replicas as well as available storage. + +An example of command we might consider running in k8s: + +```shell +./gitlab-zoekt-indexer -index_dir=/data/index -shard_name=`hostname` -address=`hostname --fqdn` +``` + +When we add more replicas to the stateful set, it should automatically handle addresses and shard names. For example: + +- `gitlab-zoekt-0` / `gitlab-zoekt-0.gitlab-zoekt.default.svc.cluster.local` +- `gitlab-zoekt-1` / `gitlab-zoekt-1.gitlab-zoekt.default.svc.cluster.local` +- .. + +Possible jobs indexer can receive: + +- `index_repositories(ids: [1,2,3,4])` +- `delete_repositories(ids: [5,6])` +- `copy_index(from: 'gitlab-zoekt-0', to: 'gitlab-zoekt-1', repo_id: 4)` + #### Replication and service discovery using Consul If we plan to replicate at the Zoekt node level as described above we need to diff --git a/doc/architecture/blueprints/container_registry_metadata_database_self_managed_rollout/index.md b/doc/architecture/blueprints/container_registry_metadata_database_self_managed_rollout/index.md index 0987b317af8..84a95e3e7c3 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database_self_managed_rollout/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database_self_managed_rollout/index.md @@ -66,29 +66,46 @@ the setup and maintenance of the registry database for new and existing deploys. For the registry, we need to develop and validate import tooling which coordinates with the core import functionality which was used to migrate all -container images on GitLab.com. Additionally, we must validate that each supported -storage driver works as expected with the import process and provide estimated -import times for admins. - -We can structure our work to meet the standards outlined in support for -Experiment, Beta, and Alpha features. Doing so will help to prioritize core -functionality and to allow users who wish to be early adopters to begin using -the database and providing us with invaluable feedback. - -These levels of support could be advertised to self-managed users via a simple -chart, allowing them to tell at a glance the status of this project as it relates -to their situation. - -| Installation | GCS | AWS | Filesystem | Azure | OSS | Swift| -| ------ | ------ |------ | ------ | ------ |------ | ------ | -| Omnibus | GA | GA | Beta | Experimental | Experimental | Experimental | -| Charts | GA | GA |Beta | Experimental | Experimental | Experimental | - -### Justification of Structuring Support by Driver - -It's possible that we could simplify the proposed support matrix by structuring -it only by deployment environment and not differentiate by storage driver. The -following two sections briefly summarize several points for and against. +container images on GitLab.com. Additionally, we should provide estimated import +times for admins for each supported storage driver. + +During the beta phase, we can highlight key features of our work to provide a +quick reference for what features we have now, are planning, their statuses, and +an excutive summary of the overall state of the migration experience. +This could be advertised to self-managed users via a simple chart, allowing them +to tell at a glance the status of this project and determine if it is feature- +complete enough for their needs and level of risk tolerance. + +This should be documented in the container registry administration documentation, +rather than in this blueprint. Providing this information there will place it in +a familiar place for self-managed admins, will allow for logical cross-linking +from other sections of the same document, such as from the garbage collection +section. + +For example: + +The metadata database is in early beta for self-managed users. The core migration +process for existing registries has been implemented, and online garbage collection +is fully implemented. Certain database enabled features are only enabled for GitLab.com +and automatic database provisioning for the registry database is not available. +Please see the table below for the status of features related to the container +registry database. + +| Feature | Description | Status | Link | +| --------------------------- | ------------------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------- | +| Import Tool | Allows existing deployments to migrate to the database. | Completed | [Import Tool](https://gitlab.com/gitlab-org/container-registry/-/issues/884) | +| Automatic Import Validation | Tests that the import maintained data integrity of imported images. | Backlog | [Validate self-managed imports](https://gitlab.com/gitlab-org/container-registry/-/issues/938) | +| Foo Bar | Lorem ipsum dolor sit amet. | Scheduled for 16.5 | <LINK> | + +### Structuring Support by Driver + +The import operation heavily relies on the object storage driver implementation +to iterate over all registry metadata so that it can be stored in the database. +It's possible that implementation differences in the driver will make a +meaningful impact on the performance and reliability of the import process. + +The following two sections briefly summarize several points for and against +structuring support by driver. #### Arguments Opposed to Structuring Support by Driver diff --git a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/adaptive_concurrency_limit_flow.png b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/adaptive_concurrency_limit_flow.png Binary files differindex 0475a32e933..ce6bb1a8dfc 100644 --- a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/adaptive_concurrency_limit_flow.png +++ b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/adaptive_concurrency_limit_flow.png diff --git a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md index 89606cdc8fa..2bd121a34bb 100644 --- a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md +++ b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md @@ -96,8 +96,10 @@ to function flawlessly. The suggested approach considers Gitaly's specific constraints and distinguishing features, including cgroup utilization and upload-pack RPC, among others. -The proposed solution does not aim to replace the existing [Gitaly concurrency -limit][Gitlay-backpressure], but automatically tweak its parameters. This means +The proposed solution does not aim to replace the existing limits in Gitaly +for [RPC concurrency](../../../administration/gitaly/configure_gitaly.md#limit-rpc-concurrency) +and [pack object concurrency](../../../administration/gitaly/configure_gitaly.md#limit-pack-objects-concurrency), +but automatically tweak the parameters. This means that other aspects, such as queuing, in-queue timeout, queue length, partitioning, and scoping, will remain unchanged. The proposed solution only focuses on modifying the current **value** of the concurrency limit. diff --git a/doc/architecture/blueprints/gitaly_handle_upload_pack_in_http2_server/index.md b/doc/architecture/blueprints/gitaly_handle_upload_pack_in_http2_server/index.md new file mode 100644 index 00000000000..acee83b2649 --- /dev/null +++ b/doc/architecture/blueprints/gitaly_handle_upload_pack_in_http2_server/index.md @@ -0,0 +1,310 @@ +--- +status: proposed +creation-date: "2023-06-15" +authors: [ "@qmnguyen0711" ] +approvers: [ ] +owning-stage: "~devops::enablement" +--- + +<!-- vale gitlab.FutureTense = NO --> +# Gitaly - Handle upload-pack traffic in a pure HTTP/2 server + +## Summary + +All Git data transferring operations that use HTTP/SSH are handled by upload-pack RPCs in Gitaly. + +These RPCs use a unique application-layer protocol called Sidechannel, which takes over the handshaking process during client dialing of the Gitaly gRPC server. This +protocol allows for an out-of-band connection to transfer large data back to the client while serving the gRPC connection as normal. + +Although it provides a huge performance improvement over using pure gRPC streaming calls, this protocol is unconventional, confusing, sophisticated, and hard to extend +or integrate into the next architecture of Gitaly. To address this, a new "boring" tech solution is proposed in this blueprint, which involves exposing a new HTTP/2 server +to handle all upload-pack traffic. + +## Motivation + +This section will explore how Git data transfers work, giving special attention to the role of Sidechannel optimization, and discussing both its advantages and +disadvantages. Our main goal is to make Git data transfers simpler while maintaining the performance improvements that Sidechannel optimization has +provided. We are looking for a solution that won't require us to completely rewrite the system and won't affect other parts of the systems and other RPCs. + +### How Git data transfer works + +Please skip this part if you are familiar with how Git data transfer architecture at GitLab. + +Git data transfer is undeniably one of the crucial services that a Git server can offer. It is a fundamental feature of Git that was originally developed for Linux +kernel development. As Git gained popularity, it continued to be recognized as a distributed system. However, the emergence of centralized Git services like GitHub or +GitLab resulted in a shift in usage patterns. Consequently, handling Git data transfer in hosted Git servers has become challenging. + +Git supports transferring data in [packfiles](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) over multiple protocols, notably HTTP and SSH. For more information, see +[pack-protocol](https://git-scm.com/docs/pack-protocol) and [http-protocol](https://git-scm.com/docs/http-protocol). + +In short, the general flow involves the following steps: + +1. Reference Discovery: the server advertises its refs to the client. +1. Packfile negotiation: the client negotiates the packfile necessary for the transport with the server by sending the list of refs it "haves", "wants", etc. +1. Transfer Packfile data: the server composes the requested data and sends it back to the client using Pack protocol. + +Further details and optimizations can underlie these steps. Git servers have never had issues with reference discovery and Packfile negotiation. The most demanding aspect +of the process is the transfer of Packfile data, which is where the actual data exchange takes place. + +For GitLab, Gitaly manages all Git operations. However, it is not accessible to external sources because Workhorse and GitLab Shell handle all external communications. +Gitaly offers a gRPC server for them through specific RPCs such as [SmartHTTP](https://gitlab.com/gitlab-org/gitaly/-/blob/master/proto/smarthttp.proto) and +[SSH](https://gitlab.com/gitlab-org/gitaly/-/blob/master/proto/ssh.proto). In addition, it provides numerous other RPCs for GitLab Rails and other services. The following +diagram illustrates a clone using SSH: + +```mermaid +sequenceDiagram + participant User as User + participant UserGit as git fetch + participant SSHClient as User's SSH Client + participant SSHD as GitLab SSHD + participant GitLabShell as gitlab-shell + participant GitalyServer as Gitaly + participant GitalyGit as git upload-pack + + User ->> UserGit: Runs git fetch + UserGit ->> SSHClient: Spawns SSH client + Note over User,SSHClient: On user's local machine + + SSHClient ->> SSHD: SSH session + Note over SSHClient,SSHD: Session over Internet + + SSHD ->> GitLabShell: spawns gitlab-shell + GitLabShell ->> GitalyServer: gRPC SSHUploadPack + GitalyServer ->> GitalyGit: spawns git upload-pack + + Note over GitalyServer,GitalyGit: On Gitaly server + Note over SSHD,GitalyGit: On GitLab server +``` + +Source: shameless copy from [this doc](https://gitlab.com/gitlab-org/gitaly/-/blob/master/cmd/gitaly-ssh/README.md?plain=0) + +### Git data transfer optimization with Sidechannel + +In the past, we faced many performance problems when transferring huge amounts of data with gRPC. [Epic 463](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463) tracks +the work of this optimization. More context surrounds it but, in summary, there are two main problems: + +- The way gRPC is designed is ideal for transmitting a large number of messages that are relatively small in size. However, when it comes to cloning a medium-sized + repository, it can result in transferring gigabytes of data. Protobuf struggles when dealing with large data, as confirmed by various sources + (examples: <https://protobuf.dev/programming-guides/techniques/#large-data> and <https://github.com/protocolbuffers/protobuf/issues/7968>). It adds significant overhead + and requires substantial CPU usage during encoding and decoding. Additionally, protobuf cannot read or write messages partially, which causes memory usage to skyrocket + when the server receives multiple requests simultaneously. +- Secondly, `grpc-go` implementation also optimizes for a similar purpose. gRPC protocol is [built on HTTP/2](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md). + When the gRPC server writes into the wire, it wraps the data inside an HTTP/2 data frame. `grpc-go` implementation maintains an asynchronous control buffer. It allocates + new memory, copies the data over, and appends to the control buffer ([client](https://github.com/grpc/grpc-go/blob/master/internal/transport/http2_client.go), + [server](https://github.com/grpc/grpc-go/blob/master/internal/transport/http2_server.go)). So, even if we can overpass the protobuf problem with a + [custom codec](https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md), `grpc-go` is still an unsolved problem. + [Upstream discussion](https://github.com/grpc/grpc-go/issues/1455) (for read path only) about reusing memory is still pending. + [An attempt to add pooled memory]( https://github.com/grpc/grpc-go/commit/642675125e198ce612ea9caff4bf75d3a4a45667) was reverted because it conflicts with typical + usage patterns. + +We have developed a protocol called Sidechannel, which enables us to communicate raw binary data with clients through an out-of-band channel, bypassing the `grpc-go` +implementation. For more information on Sidechannel, see [this document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/sidechannel.md). In summary, Sidechannel works +as follows: + +- During the handshaking process of a gRPC server's acceptance of a client TCP connection, we use [Yamux](https://github.com/hashicorp/yamux) to multiplex the TCP + connection at the application layer. This allows the gRPC server to operate on a virtual multiplexed connection called Yamux stream. +- When it becomes necessary for the server to transfer data, it establishes an additional Yamux stream. The data is transferred using that channel. + +```mermaid +sequenceDiagram + participant Workhorse + participant Client handshaker + participant Server handshaker + participant Gitaly + Note over Workhorse,Client handshaker: Workhorse process + Note over Gitaly,Server handshaker: Gitaly process + Workhorse ->> Client handshaker: - + Client handshaker ->> Server handshaker: 1. Dial + Note over Server handshaker: Create Yamux session + Server handshaker ->> Client handshaker: - + Client handshaker ->> Workhorse: Yamux stream 1 + Note over Workhorse: Persist Yamus session + + par Yamux stream 1 + Workhorse ->>+ Gitaly: 2. PostUploadWithSidechannel + Note over Gitaly: Validate request + Gitaly ->> Workhorse: 3. Open Sidechannel + end + + Note over Workhorse: Create Yamux stream 2 + + par Yamux stream 2 + Workhorse ->> Gitaly: 4. Packfile Negotiation and stuff + Workhorse ->> Gitaly: 5. Half-closed signal + Gitaly ->> Workhorse: 6. Packfile data + Note over Workhorse: Close Yamux stream 2 + end + + par Continue Yamux stream 1 + Gitaly ->>- Workhorse: 7. PostUploadPackWithSidechannelResponse + end +``` + +Sidechannel solved the original problem for us so far. We observed a huge improvement in Git transfer utilization and a huge reduction in CPU and memory usage. Of course, it comes with some tradeoffs: + +- It's a clever trick, maybe too clever. `grpc-go` provides a handshaking hook solely for authentication purposes. It's not supposed to be used for connection modification. +- Because Sidechannel works on the connection level, all RPCs using that connection must establish the multiplexing, even if they are not targeted upload-pack ones. +- Yamux is an application-level multiplexer. It's heavily inspired by HTTP/2, especially the binary framing, flow control, etc. We can call it a slimmed-down version of HTTP/2. + We are stacking two framing protocols when running gRPC on top of Yamux. +- The detailed implementation of Sidechannel is sophisticated. Apart from the aforementioned handshaking, when the server dials back, the client must start an invisible + gRPC server for handshaking before handing it back to the handler. It also implements an asymmetric framing protocol inspired by `pktline`. This protocol is tailored + for upload-pack RPC and overcomes the lack of the "half-closed" ability of Yamux. + +All of those complexity adds up to the point that it becomes a burden for the maintenance and future evolution of Gitaly. When looking forward to the future Gitaly +Raft-based architecture, Sidechannel is usually a puzzle. Reasoning about the routing strategies and implementations must consider the compatibility with Sidechannel. +Sidechannel is an application-layer protocol so most client-side routing libraries don't work well with it. Besides, we must ensure the performance +gained by Sidechannel is reserved. Eventually, we can still find a way to fit Sidechannel in, but the choices are pretty limited, likely leading to another +clever hack. + +Replacing Sidechannel with a simpler and widely adopted technology that maintains the same performance characteristics would be beneficial. One potential solution to address +all the issues would be to transfer all upload-pack RPCs to a pure HTTP/2 server. + +### Goals + +- Find an alternative to Sidechannel for upload-pack RPCs that is easier to use, uncomplicated, and widely adopted. +- The implementation must use supported API and use cases of popular libraries, frameworks, or Go's standard lib. No more hacking in the transportation layer. +- The new solution must work well with Praefect and be friendly to future routing mechanisms, load-balancers, and proxies. +- Have the same performance and low resource utilization as Sidechannel. +- Allow gradual rollout and full backwards compatibility with clients. + +### Non-Goals + +- Re-implement everything we invested into gRPC, such as authentication, observability, concurrency limiting, metadata propagation, etc. We don't want to maintain or + replicate features between two systems simultaneously. +- Modify other RPCs or change how they are used in clients. +- Migrate all RPCs to a new HTTP/2 server. + +## Proposal + +A huge portion of this blueprint describes the historical contexts and why we should move on. The proposed solution is straightforward: Gitaly exposes a pure HTTP2 server +to handle all upload-pack RPCs. Other RPCs stay intact and are handled by the existing gRPC server. + +```mermaid +flowchart LR + Workhorse-- Other RPCs --> Port3000{{":3000"}} + Port3000 --o gRPCServer("gRPC Server") + Workhorse-- PostUploadPack --> Port3001{{":3001"}} + Port3001 --o HTTP2Server("HTTP/2 Server") + GitLabShell("GitLab Shell")-- SSHUploadPack --> Port3001 + GitLabRails("GitLab Rails") -- Other RPCs --> Port3000 + subgraph Gitaly + Port3000 + gRPCServer + Port3001 + HTTP2Server + end +``` + +As mentioned earlier, Sidechannel utilizes the Yamux multiplexing protocol, which can be seen as a streamlined version of HTTP/2. When HTTP/2 is used instead, the core +functionality remains unchanged, namely multiplexing, binary framing protocol, and flow control. This means that clients like Workhorse can efficiently exchange +large amounts of binary data over the same TCP connection without requiring a custom encoding and decoding layer like Protobuf. This was the intended use case for HTTP/2 +from the start and, in theory, it can deliver the same level of performance as Sidechannel. Moreover, this replacement can eliminate the overhead for other direct +RPCs so the situation of gRPC over Yamux over TCP goes away. + +In addition, Gitaly provides access to advanced HTTP/2 functionality through its officially-supported APIs. HTTP/2 is a first-class citizen and is officially supported by +Go's standard library. This protocol seamlessly integrates with off-the-shelf load balancers and proxies and is also supported by various libraries. + +At the end of the day, both UploadPack RPC and other normal RPCs can co-exist on the same port using the technique described in the following section. However, migrating all +of them at one go is risky in terms of performance and functionality. There might be some unexpected consequeneces. Therefore, it would be wiser to perform the +migration gradually on GitLab.com, starting with UploadPack RPC. Other RPCs can be migrated after careful consideration. Self-managed instances can use the existing +single port without any modification, which means this change is transparent to users. + +The next section describes the detailed implementation and the pros and cons of that approach. + +## Design and implementation details + +### Design + +In summary, the proposal is for Gitaly to expose an HTTP2 server. At first glance, it looks like we'll need to implement a new handler and a series of interceptors. Fortunately, +gRPC Server provides [ServeHTTP](https://github.com/grpc/grpc-go/blob/642dd63a85275a96d172f446911fd04111e2c74c/server.go#L1001-L1001), allowing us to handle HTTP/2 in the +gRPC way. It implements the `http.Handler` interface to be plugged into a HTTP/2 server. Because gRPC protocol is built on top of HTTP/2, the HTTP/2 server receives and routes +the request to the handlers accordingly. We can use the header for redirecting: + +```go +if r.ProtoMajor == 2 && strings.HasPrefix( + r.Header.Get("Content-Type"), "application/grpc") { + grpcServer.ServeHTTP(w, r) +} else { + yourMux.ServeHTTP(w, r) +} +``` + +This approach brings the following benefits: + +- `ServeHTTP` uses Go's HTTP/2 server implementation, which is totally separate from `grpc-go`'s HTTP/2 server. Digging into the implementation of both gRPC and HTTP/2 + implementation, the built-in implementation should solve the memory allocation issue mentioned in the above section. +- `ServeHTTP` implements the Go standard library's `http.Handler` interface. It allows writing the handler as a gRPC handler, in which we can transfer raw binary data and + return status codes, including error details. Clients can receive a precise reason instead of a broken pipe error in Sidechannel when something goes wrong with the data + transferring process. +- Most, if not all, interceptors could be re-used without modification. Other built-in tools, such as stats reporter, [channelz](https://grpc.io/blog/a-short-introduction-to-channelz/), + and client-side load-balancing, also work out of the box. Observability toolkits, such as logging, and metrics work well. +- An upload-pack request becomes a pure streaming call on one connection. No need to open another out-of-band transportation anymore. +- Clients (Workhorse and GitLab Shell) continue to use gRPC clients. A request using this approach is considered to be a normal gRPC call. Hence, it should work well + with Praefect, with minimal modifications. + +Of course, using `ServeHTTP` comes with a cost in which requests and responses are Protobuf structs. A custom codec can overcome these performance penalties. An ideal +solution is to implement the handler as an HTTP handler so that we can have raw access to the connection. That solution entails re-implementing all gRPC-specific +components though. As a result, the proposed solution makes a reasonable tradeoff to ease the evolution of this architecture. + +```mermaid +sequenceDiagram + participant Workhorse + participant HTTP2 Server + participant UploadPack Handler + Note over HTTP2 Server, UploadPack Handler: Gitaly process + + Workhorse ->> HTTP2 Server: 1. Dial + HTTP2 Server ->> Workhorse: Connection + + par PostUploadPackV3 + Workhorse ->> UploadPack Handler: 2. PostUploadPackV3Request + Note over UploadPack Handler: Validate request + UploadPack Handler ->> Workhorse: PostUploadPackV3Response (empty) + Workhorse ->> UploadPack Handler: 3. Send raw []byte + Workhorse ->> Workhorse: Close sending + UploadPack Handler ->> Workhorse: 4. Send raw []byte + UploadPack Handler ->> UploadPack Handler: Close connection + end +``` + +The proposed solution is to: + +- Implement a "raw codec" that passes-through input binary data. +- Create a new `PostUploadPackV3` gRPC call and implement a corresponding server handler similar to other gRPC handlers. +- Implement an HTTP/2 server that calls to gRPC's `ServeHTTP`. This server is treated as another gRPC server. In fact, Gitaly now starts up to 4 gRPC servers + (internally and externally) over different transportation channels. This HTTP2 server can blend into the fleet and use Gitaly's existing process management, and + graceful shutdowns and upgrades. + +I implemented two POC merge requests in Gitaly and Workhorse to demonstrate the solution: + +- Gitaly: [merge request 5885](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5885). +- Workhorse: [merge request 123764](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123764). + +These POC MRs make the tests green for a non-Praefect setup. The early benchmarks on the local environment show that this approach: + +- Is slightly faster than the Sidechannel approach. +- Reduces peak CPU consumption by **10-20%**. +- Maintains the same memory utilization. + +The POC is not complete, but it shows the simplification using "boring" techniques. The performance improvement is totally unexpected though. + +The existing Sidechannel solution can co-exist with the new one. That makes gradual adoption feasible. + +Apart from external traffic, a Gitaly server also handles internal requests. They come from Git processes spawned by Gitaly. One typical example is +`PackObjectsHookWithSidechannel` RPC, triggered by `git-upload-pack(1)` for generating Packfile. The proposed solution also benefits these internal RPCs. + +### Considerations + +A major issue with the proposed solution is that it opens up HTTP/2 through a separate port. While `ServeHTTP` enables us to integrate the gRPC handler into an existing +HTTP/2 server, it doesn't support the other way around. A new port exposure leads to some extra setup, such as firewall, NAT, etc, in a restrictive environment. A typical +setup doesn't need any infrastructure change. As mentioned above, the two-port situation occurs during the migration only. When it's done, we can unify them into one and +release to self-managed instances. Users don't need to change anything. + +As previously mentioned, the newly-introduced HTTP/2 server is managed by the Gitaly process. It operates consistently with the existing gRPC servers for starting, +restarting, shutting down, and upgrading. This also implies that load-balancing, service discovery, domain management, and other configurations will work seamlessly +if they are set up for the current gRPC server. The new server can utilize most of the configurations of the current gRPC servers, including TLS and authentication. The +only new configuration required is the address for the HTTP server to bind to. Therefore, exposing a new port should not be a hindrance. + +Another concern is that Workhorse and GitLab Shell must maintain a separate connection pool. At the moment, they maintain one connection to each Gitaly node. This number +will be doubled to two connections to each Gitaly node. This shouldn't be a big issue. Eventually, the traffic splits between two connections in which most heavy operations +are handled by the HTTP/2 server. GitLab Rails stay intact as it doesn't handle UploadPack. diff --git a/doc/architecture/blueprints/gitlab_events_platform/index.md b/doc/architecture/blueprints/gitlab_events_platform/index.md new file mode 100644 index 00000000000..7d6377aaf43 --- /dev/null +++ b/doc/architecture/blueprints/gitlab_events_platform/index.md @@ -0,0 +1,120 @@ +--- +status: proposed +creation-date: "2023-03-06" +authors: [ "@grzesiek", "@fabiopitino" ] +coach: "@ayufan" +approvers: [ "@jreporter", "@sgoldstein" ] +owning-stage: "~devops::ops section" +--- + +# GitLab Events Platform + +## Summary + +GitLab codebase has grown a lot since the [first commit](https://gitlab.com/gitlab-org/gitlab/-/commit/93efff945215) +made in 2011. We've been able to implement many features that got adopted by +millions of users. There is a demand for more features, but there is also an +opportunity of a paradigm change: instead of delivering features that cover +specific use-cases, we can start building a platform that our users will be +able to extend with automation as they see fit. We can build a flexible and +generic DevSecOps solution that will integrate with external and internal +workflows using a robust eventing system. + +In this design document we propose to add a few additional layers of +abstraction to make it possible to: + +1. Design a notion of events hierarchy that encodes their origin and schema. +1. Publish events from within the application code using Publishers. +1. Intercept and transform events from external sources using Gateways. +1. Subscribe to internal / external events using Subscribers. +1. Hide queueing and processing implementation details behind an abstraction. + +This will allow us to transform GitLab into a generic automation tooling, but +will also reduce the complexity of existing events-like features: + +1. [Webhooks](../../../user/project/integrations/webhook_events.md) +1. [Audit Events](../../../administration/audit_events.md) +1. [GitLab CI Events](https://about.gitlab.com/blog/2022/08/03/gitlab-ci-event-workflows/) +1. [Package Events](https://gitlab.com/groups/gitlab-org/-/epics/9677) +1. [GraphQL Events](https://gitlab.com/gitlab-org/gitlab/-/blob/dabf4783f5d758f69d947f5ff2391b4b1fb5f18a/app/graphql/graphql_triggers.rb) + +## Goals + +Build required abstractions and their implementation needed to better manage +internally and externally published events. + +## Challenges + +1. There is no solution allowing users to build subscribers and publishers. +1. There is no solution for managing subscriptions outside of the Ruby code. +1. There are many events-like features inside GitLab not using common abstractions. +1. Our current eventing solution `Gitlab::EventStore` is tightly coupled with Sidekiq. +1. There is no unified and resilient way to subscribe to externally published events. +1. Payloads associated with events differ a lot, similarly to how we define schemas. +1. Not all events are strongly typed, there is no solution to manage their hierarchy. +1. Events are not being versioned, it is easy to break schema contracts. +1. We want to build more features based on events, but because of missing + abstractions the value we could get from the implementations is limited. + +## Proposal + +### Publishers + +Publishing events from within our Rails codebase is an important piece of the +proposed architecture. Events should be strongly typed, ideally using Ruby classes. + +For example, we could emit events in the following way: + +```ruby +include Gitlab::Events::Emittable + +emit Gitlab::Events::Package::Published.new(package) +``` + +- Publishing events should be a non-blocking, and near zero-cost operation. +- Publishing events should take their origin and identity into the account. +- Publishing events should build their payload based on their lineage. +- `emit` can be a syntactic sugar over mechanism used in `GitLab::EventStore`. + +### Subscribers + +Subscribers will allow application developers to subscribe to arbitrary events, +published internally or externally. Subscribers could also allow application +developers to build subscription mechanisms that could be used by our users to, +for example, subscribe to project events to trigger pipelines. + +Events that subscribers will subscribe to will becomes contracts, hence we +should version them or use backwards-and-forward compatible solution (like +Protobuf). + +### Gateways + +Gateways can be used to intercept internal and external events and change their +type, augment lineage and transform their payloads. + +Gateways can be used, for example, to implement sink endpoints to intercept +Cloud Events, wrap into an internally used Ruby classes and allow developers / +users to subscribe to them. + +We also may be able to implement [cross-Cell](../cells) communication through a +generic events bus implemented using Gateways. + +There are also ideas around cross-instance communication to improve how GitLab +can coordinate complex deployments that involve multiple instances. + +### Processing + +Today in order to queue events, we either use PostgreSQL or Sidekiq. Both +mechanisms are being used interchangeably and are tightly coupled with existing +solution. + +The main purpose of building an abstraction for queuing and processing is to be +able to switch to a different queuing backend when needed. For example, we +could queue some of the events on Google Pub/Sub, and send those through a +dedicated Gateway on their way back to the application. + +### Observability + +In order to understand interactions between events, publishers and subscribers +we may need to deliver a proper instrumentation _via_ OpenTelemetry. This will +allow us to visualize these interactions with Distributed Tracing Backends. diff --git a/doc/architecture/blueprints/gitlab_steps/index.md b/doc/architecture/blueprints/gitlab_steps/index.md new file mode 100644 index 00000000000..d7878445cd0 --- /dev/null +++ b/doc/architecture/blueprints/gitlab_steps/index.md @@ -0,0 +1,142 @@ +--- +status: proposed +creation-date: "2023-08-23" +authors: [ "@ayufan" ] +coach: "@grzegorz" +approvers: [ "@dhershkovitch", "@DarrenEastman", "@marknuzzo", "@nicolewilliams" ] +owning-stage: "~devops::verify" +participating-stages: [ ] +--- + +# Step Runner for executing GitLab Steps + +## Summary + +This document describes architecture of a new component called Step Runner, the GitLab Steps syntax it uses, +and how the GitHub Actions support will be achieved. + +The competitive CI products [drone.io](https://drone.io), +[GitHub Actions](https://docs.github.com/en/actions/creating-actions) +have a composable CI jobs execution in form of steps, or actions. + +Their usage and our prior evaluation of [GitLab Runner Plugins](https://gitlab.com/gitlab-org/gitlab/-/issues/15067) +shows a need for a better way to define CI job execution. + +## Glossary + +- GitLab Steps: a name of GitLab CI feature to define and use reusable components + within a single job execution context. +- Step Runner: a RFC implementation for GitLab Steps that provides compatibility with the GitHub Actions. +- GitHub Actions: similar to GitLab Steps, a reusable execution component used on GitHub. +- CI Catalog: a public or private component catalog that could be used to discover and use shared components. +- GitLab Rails: a main application responsible for pipeline execution, running on GitLab.com or on-premise installation. + +## Motivation + +Even though the current [`.gitlab-ci.yml`](../../../ci/yaml/gitlab_ci_yaml.md) is reasonably flexible, it easily becomes very +complex when trying to support complex workflows. This complexity is represented +with repetetitve patterns, a purpose-specific syntax, or a complex sequence of commands +to execute. + +This is particularly challenging, because the [`.gitlab-ci.yml`](../../../ci/yaml/gitlab_ci_yaml.md) +is inflexible on more complex workflows that require fine-tuning or special behavior +for the CI job execution. Its prescriptive approach how to handle Git cloning, +when artifacts are downloaded, or how the shell script is being executed quite often +results in the need to work around the system for pipelines that are not "standard" +or when new features are requested. + +This proves especially challenging when trying to add a new syntax to the +[`.gitlab-ci.yml`](../../../ci/yaml/gitlab_ci_yaml.md) +to support a specific feature, like [`secure files`](../../../ci/secure_files/index.md) +or `release:` keyword. Adding these special features on a syntax level +results in a more complex config, which is harder to maintain, and more complex +to deal with technical debt when requirements change. + +An example of the `drone.io` and the `GitHub Actions` shows that a lot of workflows do not +have to be part of CI syntax. Instead, they can be provided in the form of reusable components +that are configured in a generic way in the CI config, and later downloaded and executed according +to inputs and parameters. + +The GitLab Steps is meant to fill that product-gap by following similar model to competitors +and to some extent staying compatible with them. The GitLab Steps is meant to replace all +purpose-specific syntax to handle specific features. By providing and using reusable components, +that are build outside of `.gitlab-ci.yml`, that are versioned, and requested when needed +this allows the customer much more flexibility, and allows us to iterate on a catalog much faster. + +The reusable components that are part of a CI job execution could be used from a publicily hosted +repository on GitLab.com, from on-premise repository of steps, or be fetched from local project. + +Each CI job would define a list of `steps:` to execute, that would reference GitLab Steps +or GitHub Actions. Those steps would be executed by the step runner directly in the context of +the target environment. GitLab Runner would be responsible to be connection between GitLab.com +(or on-premise installation) and Step Runner. + +### Goals + +GitLab Steps: + +- GitLab Steps defines a syntax and structure for GitLab specific Steps implementation. +- GitLab Steps are published in CI Catalog. +- GitLab Steps can be used across instances (federation). +- GitLab Steps do define `inputs` and `outputs`. +- GitLab Steps needs to explicitly request sensitive informations with expected permissions. + For example: secrets, variables, tokens. + +GitLab Inc. managed repository of GitLab Steps: + +- GitLab Inc. provides a repository of GitLab Steps that are a drop-in replacement + for all current purpose-specific syntax: `artifacts:`, `cache:`, `release:`, etc. +- GitLab Inc. will provide a generic step to execute `shell` steps supporting various + shells (`bash`, `powershell`). +- The usage of purpose-specific syntax might be eventually deprecated in favor of steps. + +Step Runner: + +- Step Runner is hosted in a separate project in `https://gitlab.com/gitlab-org`. +- Step Runner can be used to execute most of GitHub Actions. +- Step Runner is run as a process in a target environment. +- Step Runner can be used by user on their local machine to run steps of a specific CI job + from locally stored `.gitlab-ci.yml`. +- Step Runner is external component to GitLab Runner, the GitLab Runner does provision + environment, construct payload and pass execution to Step Runner. +- Step Runner is to replace all custom handling in GitLab Runner for `clone`, `artifacts`, + `caches`, `script` and `after_script`, and custom handling for all different shells (`bash`, `powershell`). +- Step Runner is responsible for parsing and compiling GitLab Steps and GitHub Actions. +- Step Runner is responsible for downloading, and managing repositories required by GitLab Steps and GitHub Actions. +- Step Runner does control and monitor execution flow of individual steps of execution. +- Step Runner is required to be executable from the command-line interface (CLI). It means that it can be configured either via config file, + or environment file, or be able to read `.gitlab-ci.yml`. +- Step Runner can expose gRPC or other programmable interface to run config or get trace from. + +Steps Execution: + +- Each Step is defined by a single published or locally defined GitLab Step, or GitHub Action. +- Each Step is executed depending on conditions that are defined by that step. +- Each Step is executed with least amount of information exposed. Exposed informations to step + are requested explicitly by the step. For example: only environment variables explicitly + requested by the step will be passed to the step. +- Each Step is considered untrusted. It means that even though some steps are trusted, the whole + CI job should be considered untrusted, since system cannot guarantee trust. +- Each Step describes its execution in a form of preconditions, versions used, and output produced. + This is meant to allow to sign steps execution for the purpose of creating reproducible builds. + +Backward compatibility: + +- All currently executable syntax (for example: `before_script:`, `script:`, `artifacts:`, `cache:`, etc.) + should be convertible by GitLab (Rails) + +## Non-Goals + +TBD + +## Proposal + +TBD + +## Design and implementation details + +TBD + +## References + +- [GitLab Issue #215511](https://gitlab.com/gitlab-org/gitlab/-/issues/215511) diff --git a/doc/architecture/blueprints/google_artifact_registry_integration/index.md b/doc/architecture/blueprints/google_artifact_registry_integration/index.md new file mode 100644 index 00000000000..adde0f7f587 --- /dev/null +++ b/doc/architecture/blueprints/google_artifact_registry_integration/index.md @@ -0,0 +1,159 @@ +--- +status: proposed +creation-date: "2023-08-31" +authors: [ "@jdrpereira", "@10io" ] +coach: "@grzesiek" +approvers: [ "@trizzi", "@crystalpoole" ] +owning-stage: "~devops::package" +participating-stages: [] +--- + +# Google Artifact Registry Integration + +## Summary + +GitLab and Google Cloud have recently [announced](https://about.gitlab.com/blog/2023/08/29/gitlab-google-partnership-s3c/) a partnership to combine the unique capabilities of their platforms. + +As highlighted in the announcement, one key goal is the ability to "_use Google's Artifact Registry with GitLab pipelines and packaging to create a security data plane_". The initial step toward this goal is to allow users to configure a new [Google Artifact Registry](https://cloud.google.com/artifact-registry) (abbreviated as GAR from now on) [project integration](../../../user/project/integrations/index.md) and display [container image artifacts](https://cloud.google.com/artifact-registry/docs/supported-formats) in the GitLab UI. + +## Motivation + +Please refer to the [announcement](https://about.gitlab.com/blog/2023/08/29/gitlab-google-partnership-s3c/) blog post for more details about the motivation and long-term goals of the GitLab and Google Cloud partnership. + +Regarding the scope of this design document, our primary focus is to fulfill the Product requirement of providing users with visibility over their container images in GAR. The motivation for this specific goal is rooted in foundational research on the use of external registries as a complement to the GitLab Container Registry ([internal](https://gitlab.com/gitlab-org/ux-research/-/issues/2602)). + +Since this marks the first step in the GAR integration, our aim is to achieve this goal in a way that establishes a foundation to facilitate reusability in the future. This groundwork could benefit potential future expansions, such as support for additional artifact formats (npm, Maven, etc.), and features beyond the Package stage (e.g., vulnerability scanning, deployments, etc.). + +### Goals + +- Allow GitLab users to configure a new [project integration](../../../user/project/integrations/index.md) for connecting to GAR. +- Limited to a single top-level GAR [repository](https://cloud.google.com/artifact-registry/docs/repositories) per GitLab project. +- Limited to GAR repositories in [Standard](https://cloud.google.com/artifact-registry/docs/repositories#mode) mode. Support for Remote and Virtual [repository modes](https://cloud.google.com/artifact-registry/docs/repositories#mode) (both in Preview) is a strech goal. +- Limited to GAR repositories of format [Container images](https://cloud.google.com/artifact-registry/docs/supported-formats#container). +- Use a Google Cloud [service account](https://cloud.google.com/iam/docs/service-account-overview) provided by the GitLab project owner/maintainer to interact with GAR. +- Allow GitLab users to list container images under the connected GAR repository, including sub-repositories. The list should be paginable and sortable. +- For each listed image, display its URI, list of tags, size, digest, upload time, media type, build time, and update time, as documented [here](https://cloud.google.com/artifact-registry/docs/reference/rest/v1/projects.locations.repositories.dockerImages#DockerImage). +- Listing container images under the connected GAR repository is restricted to users with [Reporter+](../../../user/permissions.md#roles) roles. + +### Non-Goals + +While some of these may become goals for future iterations, they are currently out of scope: + +- Create, update and delete operations. +- Connecting to multiple (top-level) GAR repositories under the same project. +- Support for [repository formats](https://cloud.google.com/artifact-registry/docs/supported-formats) beyond container images. +- Support for other [Identity and Access Management (IAM)](https://cloud.google.com/iam) permissions/credentials beyond [service accounts](https://cloud.google.com/iam/docs/service-account-overview). +- GAR [cleanup policies](https://cloud.google.com/artifact-registry/docs/repositories/cleanup-policy). +- Filtering the images list by their attributes (name or value). The current [GAR API](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#listdockerimagesrequest) does not support filtering. +- [Artifact analysis and vulnerability scanning](https://cloud.google.com/artifact-registry/docs/analysis). + +## Proposal + +### Design and Implementation Details + +#### Project Integration + +A new [project integration](../../../user/project/integrations/index.md) for GAR will be created. Once enabled, this will display a new "Google Artifact Registry" item in the "Operate" section of the sidebar. This is also where the [Harbor](../../../user/project/integrations/harbor.md) integration is displayed if enabled. + +The GAR integration can be enabled by project owner/maintainer(s), who must provide four configuration parameters during setup: + +- **GCP project ID**: The globally unique identifier for the GCP project where the target GAR repository lives. +- **Repository location**: The [GCP location](https://cloud.google.com/about/locations) where the target GAR repository lives. +- **Repository name**: The name of the target GAR repository. +- **GCP service account key**: The _content_ (not the file) of the [service account key](https://cloud.google.com/iam/docs/keys-create-delete) in JSON format ([sample](https://cloud.google.com/iam/docs/keys-create-delete#creating)). + +#### Authentication + +The integration is simplified by using a single GCP service account for the integration. Users retain the ability to [audit usage](https://cloud.google.com/iam/docs/audit-logging/examples-service-accounts#access-with-key) of this service account on the GCP side and revoke permissions if/when necessary. + +The service account key provided during the integration setup must be granted at least with the [`Artifact Registry Reader`](https://cloud.google.com/artifact-registry/docs/access-control#permissions) role in the target GCP project. + +Saving the (encrypted) service account key JSON content in the backend allows us to easily grab and use it to initialize the GAR client (more about that later). Providing the content of the key file instead of uploading it is similar to what we do with users' public SSH keys. + +As previously highlighted, access to the GAR integration features is restricted to users with [Reporter+](../../../user/permissions.md#roles) roles. + +#### Resource Mapping + +For the [GitLab Container Registry](../../../user/packages/container_registry/index.md), repositories within a specific project must have a path that matches the project full path. This is essentially how we establish a resource mapping between GitLab Rails and the registry, which serves multiple purposes, including granular authorization, scoping storage usage to a given project/group/namespace, and more. + +Regarding the GAR integration, since there is no equivalent entities for GitLab project/group/namespace resources on the GAR side, we aim to simplify matters by allowing users to attach any [GAR repository](https://cloud.google.com/artifact-registry/docs/repositories) to any GitLab project, regardless of their respective paths. Similarly, we do not plan to restrict the attachment of a particular GAR repository to a single GitLab project. Ultimately, it is up to users to determine how to organize both datasets in the way that best suits their needs. + +#### GAR API + +GAR provides three APIs: Docker API, REST API, and RPC API. + +The [Docker API](https://cloud.google.com/artifact-registry/docs/reference/docker-api) is based on the [Docker Registry HTTP API V2](https://docs.docker.com/registry/spec/api), now superseded by the [OCI Distribution Specification API](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) (from now on referred to as OCI API). This API is used for pushing/pulling images to/from GAR and also provides some discoverability operations. Please refer to [Alternative Solutions](#alternative-solutions) for the reasons why we don't intend to use it. + +Among the proprietary GAR APIs, the [REST API](https://cloud.google.com/artifact-registry/docs/reference/rest) provides basic functionality for managing repositories. This includes [`list`](https://cloud.google.com/artifact-registry/docs/reference/rest/v1/projects.locations.repositories.dockerImages/list) and [`get`](https://cloud.google.com/artifact-registry/docs/reference/rest/v1/projects.locations.repositories.dockerImages/get) operations for container image repositories, which could be used for this integration. Both operations return the same data structure, represented by the [`DockerImage`](https://cloud.google.com/artifact-registry/docs/reference/rest/v1/projects.locations.repositories.dockerImages#DockerImage) object, so both provide the same level of detail. + +Last but not least, there is also an [RPC API](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1), backed by gRPC and Protocol Buffers. This API provides the most functionality, covering all GAR features. From the available operations, we can make use of the [`ListDockerImagesRequest`](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#listdockerimagesrequest) and [`GetDockerImageRequest`](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#google.devtools.artifactregistry.v1.GetDockerImageRequest) operations. As with the REST API, both responses are composed of [`DockerImage`](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#google.devtools.artifactregistry.v1.DockerImage) objects. + +Between the two proprietary API options, we chose the RPC one because it provides support not only for the operations we need today but also offers better coverage of all GAR features, which will be beneficial in future iterations. Finally, we do not intend to make direct use of this API but rather use it through the official Ruby client SDK. Please see [Client SDK](#client-sdk) below for more details. + +#### Backend Integration + +##### Client SDK + +To interact with GAR we will make use of the official GAR [Ruby client SDK](https://cloud.google.com/ruby/docs/reference/google-cloud-artifact_registry/latest). + +*TODO: Add more details about the client SDK integration and its limitations (no filtering for example).* + +##### Database Changes + +*TODO: Describe any necessary changes to the database to support this integration.* + +##### CI/CD variables + +Similar to the [Harbor](../../../user/project/integrations/harbor.md#configure-gitlab) integration, once users activates the GAR integration, additional CI/CD variables will be automatically available if the integration is enabled. These will be set according to the requirements described in the [documentation](https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key): + +- `GCP_ARTIFACT_REGISTRY_URL`: This will be set to `https://LOCATION-docker.pkg.dev`, where `LOCATION` is the GCP project location configured for the integration. +- `GCP_ARTIFACT_REGISTRY_PROJECT_URI`: This will be set to `LOCATION-docker.pkg.dev/PROJECT-ID`. `PROJECT-ID` is the GCP project ID of the GAR repository configured for the integration. +- `GCP_ARTIFACT_REGISTRY_PASSWORD`: This will be set to the base64-encode version of the service account JSON key file configured for the integration. +- `GCP_ARTIFACT_REGISTRY_USER`: This will be set to `_json_key_base64`. + +These can then be used to log in using `docker login`: + +```shell +docker login -u $GCP_ARTIFACT_REGISTRY_USER -p $GCP_ARTIFACT_REGISTRY_PASSWORD $GCP_ARTIFACT_REGISTRY_URL +``` + +Similarly, these can be used to download images from the repository with `docker pull`: + +```shell +docker pull $GCP_ARTIFACT_REGISTRY_PROJECT_URI/REPOSITORY/myapp:latest +``` + +Finally, provided that the configured service account has the `Artifact Registry Writer` role, one can also push images to GAR: + +```shell +docker build -t $GCP_ARTIFACT_REGISTRY_REPOSITORY_URI/myapp:latest . +docker push $GCP_ARTIFACT_REGISTRY_REPOSITORY_URI/myapp:latest +``` + +For forward compatibility reasons, the repository name (`REPOSITORY` in the command above) must be appended to `GCP_ARTIFACT_REGISTRY_PROJECT_URI` by the user. In the first iteration we will only support a single GAR repository, and therefore we could technically provide an e.g. `GCP_ARTIFACT_REGISTRY_REPOSITORY_URI` variable with the repository name already included. However, once we add support for multiple repositories, there is no way we can tell what repository a user will want to target for a specific instruction. So it must be the user to tell that. + +#### UI/UX + +This integration will include a dedicated page named "Google Artifact Registry," listed under the "Operate" section of the sidebar. This page will enable users to view the list of all container images in the configured GAR repository. See the [UI/UX](ui_ux.md) page for additional details. + +#### GraphQL APIs + +*TODO: Describe any GraphQL APIs or changes to existing APIs that will be needed for this integration.* + +## Alternative Solutions + +### Use Docker/OCI API + +One alternative solution considered was to use the Docker/OCI API provided by GAR, as it is a common standard for container registries. This approach would have allowed GitLab to reuse [existing logic](https://gitlab.com/gitlab-org/gitlab/-/blob/20df77103147c0c8ff1c22a888516eba4bab3c46/lib/container_registry/client.rb) for connecting to container registries, which could potentially speed up development. However, there were several drawbacks to this approach: + +- **Authentication Complexity**: The API requires authentication tokens, which need to be requested at the [login endpoint](https://docs.docker.com/registry/spec/auth/token). These tokens have limited validity, adding complexity to the authentication process. Handling expiring tokens would have been necessary. + +- **Limited Focus**: The API is solely focused on container registry objects, which does not align with the goal of creating a flexible integration framework for adopting additional GAR artifacts (e.g. package registry formats) down the road. + +- **Discoverability Limitations**: The API has severe limitations when it comes to discoverability, lacking features like filtering or sorting. + +- **Multiple Requests**: To retrieve all the required information about each image, multiple requests to different endpoints (listing tags, obtaining image manifests, and image configuration blobs) would have been necessary, leading to a `1+N` performance issue. + +GitLab had previously faced significant challenges with the last two limitations, prompting the development of a custom [GitLab Container Registry API](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/api.md) to address them. Additionally, GitLab decided to [deprecate support](../../../update/deprecations.md#use-of-third-party-container-registries-is-deprecated) for connecting to third-party container registries using the Docker/OCI API due to these same limitations and the increased cost of maintaining two solutions in parallel. As a result, there is an ongoing effort to replace the use of the Docker/OCI API endpoints with custom API endpoints for all container registry functionalities in GitLab. + +Considering these factors, the decision was made to build the GAR integration from scratch using the proprietary GAR API. This approach provides more flexibility and control over the integration and can serve as a foundation for future expansions, such as support for other GAR artifact formats. diff --git a/doc/architecture/blueprints/google_artifact_registry_integration/ui_ux.md b/doc/architecture/blueprints/google_artifact_registry_integration/ui_ux.md new file mode 100644 index 00000000000..5cb862d50e7 --- /dev/null +++ b/doc/architecture/blueprints/google_artifact_registry_integration/ui_ux.md @@ -0,0 +1,17 @@ +--- +stage: package +group: container registry +description: 'UI/UX for Google Artifact Registry Integration' +--- + +# UI/UX for Google Artifact Registry Integration + +## Structure and Organization + +Unlike the GitLab Container Registry (and therefore the Docker Registry and OCI Distribution), GAR does not treat tags as the primary "artifacts" in a repository. Instead, the primary "artifacts" are the image manifests. For each manifest object (represented by [`DockerImage`](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#google.devtools.artifactregistry.v1.DockerImage)), there is a list of assigned tags (if any). Consequently, when listing the contents of a repository through the GAR API, the response comprises a collection of manifest objects (along with their associated tags as properties), rather than a collection of tag objects. Additionally, due to this design choice, untagged manifests are also present in the response. + +To maximize flexibility, extensibility, and maintain familiarity for GAR users, we plan to fully embrace the GAR API data structures while surfacing data in the GitLab UI. We won't attempt to emulate a "list of tags" response to match the UI/UX that we already have for the GitLab Container Registry. + +Considering the above, there will be a view that provides a pageable and sortable list of all images in the configured GAR repository. Additionally, there will be a detail view to display more information about a single image. You can find a list of available image attributes documented [here](https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1#google.devtools.artifactregistry.v1.DockerImage). + +## Designs diff --git a/doc/architecture/blueprints/modular_monolith/bounded_contexts.md b/doc/architecture/blueprints/modular_monolith/bounded_contexts.md index 0f71e24864e..8133106050d 100644 --- a/doc/architecture/blueprints/modular_monolith/bounded_contexts.md +++ b/doc/architecture/blueprints/modular_monolith/bounded_contexts.md @@ -41,11 +41,16 @@ The majority of the code is not properly namespaced and organized: In June 2023 we've started extracing gems out of the main codebase, into [`gems/` directory inside the monorepo](https://gitlab.com/gitlab-org/gitlab/-/blob/4c6e120069abe751d3128c05ade45ea749a033df/doc/development/gems.md). -This is our first step towards modularization: externalize code that can be -extracted to prevent coupling from being introduced into modules that have been -designed as separate components. +This is our first step towards modularization. -These gems as still part of the monorepo. +- We want to separate generic code from domain code (that powers the business logic). +- We want to cleanup `lib/` directory from generic code. +- We want to isolate code that could live in a separate project, to prevent it from depending on domain code. + +These gems as still part of the monorepo but could be extracted into dedicated repositories if needed. + +Extraction of gems is non blocking to modularization but the less generic code exists in `lib/` the +easier will be identifying and separating bounded context. ### 1. What makes a bounded context? @@ -103,17 +108,3 @@ With this static list we could: - Understand where to place new classes and modules. - Enforce if any top-level namespaces are used that are not in the list of bounded contexts. - Autoload non-standard Rails directories based on the given list. - -## Glossary - -- `modules` are Ruby modules and can be used to nest code hierarchically. -- `namespaces` are unique hierarchies of Ruby constants. For example, `Ci::` but also `Ci::JobArtifacts::` or `Ci::Pipeline::Chain::`. -- `packages` are Packwerk packages to group together related functionalities. These packages can be big or small depending on the design and architecture. Inside a package all constants (classes and modules) have the same namespace. For example: - - In a package `ci`, all the classes would be nested under `Ci::` namespace. There can be also nested namespaces like `Ci::PipelineProcessing::`. - - In a package `ci-pipeline_creation` all classes are nested under `Ci::PipelineCreation`, like `Ci::PipelineCreation::Chain::Command`. - - In a package `ci` a class named `MergeRequests::UpdateHeadPipelineService` would not be allowed because it would not match the package's namespace. - - This can be enforced easily with [Packwerk's based Rubocop Cops](https://github.com/rubyatscale/rubocop-packs/blob/main/lib/rubocop/cop/packs/root_namespace_is_pack_name.rb). -- `bounded context` is a top-level Packwerk package that represents a macro aspect of the domain. For example: `Ci::`, `MergeRequests::`, `Packages::`, etc. - - A bounded context is represented by a single Ruby module/namespace. For example, `Ci::` and not `Ci::JobArtifacts::`. - - A bounded context can be made of 1 or multiple Packwerk packages. Nested packages would be recommended if the domain is quite complex and we want to enforce privacy among all the implementation details. For example: `Ci::PipelineProcessing::` and `Ci::PipelineCreation::` could be separate packages of the same bounded context and expose their public API while keeping implementation details private. - - A new bounded context like `RemoteDevelopment::` can be represented a single package while large and complex bounded contexts like `Ci::` would need to be organized into smaller/nested packages. diff --git a/doc/architecture/blueprints/modular_monolith/hexagonal_monolith/index.md b/doc/architecture/blueprints/modular_monolith/hexagonal_monolith/index.md index eb4b428cf52..f0f689d48ca 100644 --- a/doc/architecture/blueprints/modular_monolith/hexagonal_monolith/index.md +++ b/doc/architecture/blueprints/modular_monolith/hexagonal_monolith/index.md @@ -25,12 +25,22 @@ Use [Packwerk](https://github.com/Shopify/packwerk) to enforce privacy and depen ## Details +```mermaid +flowchart TD + u([User]) -- interacts directly with --> AA[Application Adapter: WebUI, REST, GraphQL, git, ...] + AA --uses abstractions from--> D[Application Domain] + AA -- depends on --> Platform + D -- depends on --> Platform[Platform: gems, configs, framework, ...] +``` + ### Application domain -The application core (functional domains) is divided into separate top-level bounded contexts called after the -[feature category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) they represent. +The application core (functional domains) is composed of all the code that describes the business logic, policies and data +that is unique to GitLab product. +It is divided into separate top-level [bounded contexts](../bounded_contexts.md). A bounded-context is represented in the form of a Ruby module. -This follows the existing [guideline on naming namespaces](../../../../development/software_design.md#use-namespaces-to-define-bounded-contexts) but puts more structure to it. +This follows the existing [guideline on naming namespaces](../../../../development/software_design.md#use-namespaces-to-define-bounded-contexts) +but puts more structure to it. Modules should: @@ -52,6 +62,12 @@ If a feature category is only relevant in the context of a parent feature catego parent's bounded context. For example: Build artifacts existing in the context of Continuous Integration feature category and they may be merged under a single bounded context. +The application domain has no knowledge of outer layers like the application adapters and only depends on the +platform code. This makes the domain code to be the SSoT of the business logic, be reusable and testable regardless +whether the request came from the WebUI or REST API. + +If a dependency between an outer layer and an inner layer is required (domain code depending on the interface of an adapter), this can be solved using inversion of control techniques, especially dependency injection. + ### Application adapters >>> @@ -67,9 +83,14 @@ Application adapters would be: - Web UI (Rails controllers, view, JS and Vue client) - REST API endpoints - GraphQL Endpoints -- Action Cable -TODO: continue describing how adapters are organized and why they are separate from the domain code. +They are responsible for the interaction with the user. Each adapter should interpret the request, parse parameters +and invoke the right abstraction from the application domain, then present the result back to the user. + +Presentation logic, and possibly authentication, would be specific to the adapters layer. + +The application adapters layer depends on the platform code to run: the Rails framework, the gems that power the adapter, +the configurations and utilities. ### Platform code @@ -95,19 +116,76 @@ This means that aside from the Rails framework code, the rest of the platform co Eventually all code inside `gems/` could potentially be extracted in a separate repository or open sourced. Placing platform code inside `gems/` makes it clear that its purpose is to serve the application code. -### Why Packwerk? +### Enforcing boundaries + +Ruby does not have the concept of privacy of constants in a given module. Unlike other programming languages, even extracting +well documented gems doesn't prevent other developers from coupling code to implementation details because all constants +are public in Ruby. + +We can have a codebase perfectly organized in an hexagonal architecture but still having the application domain, the biggest +part of the codebase, being a non modularized [big ball of mud](https://en.wikipedia.org/wiki/Big_ball_of_mud). + +Enforcing boundaries is also vital to maintaining the structure long term. We don't want that after a big modularization +effort we slowly fall back into a big ball of mud gain by violating the boundaries. + +We explored the idea of [using Packwerk in a proof of concept](../proof_of_concepts.md#use-packwerk-to-enforce-module-boundaries) +to enforce module boundaries. -TODO: +[Packwerk](https://github.com/Shopify/packwerk) is a static analyzer that allows to gradually introduce packages in the +codebase and enforce privacy and explicit dependencies. Packwerk can detect if some Ruby code is using private implementation +details of another package or if it's using a package that wasn't declared explicitly as a dependency. -- boundaries not enforced at runtime. Ruby code will still work as being all loaded in the same memory space. -- can be introduced incrementally. Not everything requires to be moved to packs for the Rails autoloader to work. +Being a static analyzer it does not affect code execution, meaning that introducing Packwerk is safe and can be done +gradually. Companies like Gusto have been developing and maintaining a list of [development and engineering tools](https://github.com/rubyatscale) for organizations that want to move to using a Rails modular monolith around Packwerk. ### EE and JH extensions -TODO: +One of the unique challenges of modularizing the GitLab codebase is the presence of EE extensions (managed by GitLab) +and JH extensions (managed by JiHu). + +By moving related domain code (e.g. `Ci::`) under the same bounded context and Packwerk package, we would also need to +move `ee/` extensions in it. + +To have top-level bounded contexts to also match Packwerk packages it means that all code related to a specific domain +needs to be placed under the same package directory, including EE extensions, for example. + +The following is just an example of a possible directory structure: + +```shell +domains +├── ci +│ ├── package.yml # package definition. +│ ├── packwerk.yml # tool configurations for this package. +│ ├── package_todo.yml # existing violations. +│ ├── core # Core features available in Community Edition and always autoloaded. +│ │ ├── app +│ │ │ ├── models/... +│ │ │ ├── services/... +│ │ │ └── lib/... # domain-specific `lib` moved inside `app` together with other classes. +│ │ └── spec +│ │ └── models/... +│ ├── ee # EE extensions specific to the bounded context, conditionally autoloaded. +│ │ ├── models/... +│ │ └── spec +│ │ └── models/... +│ └── public # Public constants are placed here so they can be referenced by other packages. +│ ├── core +│ │ ├── app +│ │ │ └── models/... +│ │ └── spec +│ │ └── models/... +│ └── ee +│ ├── app +│ │ └── models/... +│ └── spec +│ └── models/... +├── merge_requests/ +├── repositories/ +└── ... +``` ## Challenges diff --git a/doc/architecture/blueprints/modular_monolith/index.md b/doc/architecture/blueprints/modular_monolith/index.md index ef50be643a6..f1e6c119552 100644 --- a/doc/architecture/blueprints/modular_monolith/index.md +++ b/doc/architecture/blueprints/modular_monolith/index.md @@ -93,12 +93,11 @@ There are many aspects and details required to make modularization of our monolith successful. We will work on the aspects listed below, refine them, and add more important details as we move forward towards the goal: -1. [Deliver modularization proof-of-concepts that will deliver key insights](proof_of_concepts.md) -1. [Align modularization plans to the organizational structure](bounded_contexts.md) +1. [Deliver modularization proof-of-concepts that will deliver key insights](proof_of_concepts.md). +1. Align modularization plans to the organizational structure by [defining bounded contexts](bounded_contexts.md). +1. Separate domains into modules that will reflect organizational structure (TODO) 1. Start a training program for team members on how to work with decoupled domains (TODO) 1. Build tools that will make it easier to build decoupled domains through inversion of control (TODO) -1. Separate domains into modules that will reflect organizational structure (TODO) -1. Build necessary services to align frontend and backend modularization (TODO) 1. [Introduce hexagonal architecture within the monolith](hexagonal_monolith/index.md) 1. Introduce clean architecture with one-way-dependencies and host application (TODO) 1. Build abstractions that will make it possible to run and deploy domains separately (TODO) @@ -107,6 +106,20 @@ add more important details as we move forward towards the goal: In progress. +## Glossary + +- `modules` are Ruby modules and can be used to nest code hierarchically. +- `namespaces` are unique hierarchies of Ruby constants. For example, `Ci::` but also `Ci::JobArtifacts::` or `Ci::Pipeline::Chain::`. +- `packages` are Packwerk packages to group together related functionalities. These packages can be big or small depending on the design and architecture. Inside a package all constants (classes and modules) have the same namespace. For example: + - In a package `ci`, all the classes would be nested under `Ci::` namespace. There can be also nested namespaces like `Ci::PipelineProcessing::`. + - In a package `ci-pipeline_creation` all classes are nested under `Ci::PipelineCreation`, like `Ci::PipelineCreation::Chain::Command`. + - In a package `ci` a class named `MergeRequests::UpdateHeadPipelineService` would not be allowed because it would not match the package's namespace. + - This can be enforced easily with [Packwerk's based Rubocop Cops](https://github.com/rubyatscale/rubocop-packs/blob/main/lib/rubocop/cop/packs/root_namespace_is_pack_name.rb). +- `bounded context` is a top-level Packwerk package that represents a macro aspect of the domain. For example: `Ci::`, `MergeRequests::`, `Packages::`, etc. + - A bounded context is represented by a single Ruby module/namespace. For example, `Ci::` and not `Ci::JobArtifacts::`. + - A bounded context can be made of 1 or multiple Packwerk packages. Nested packages would be recommended if the domain is quite complex and we want to enforce privacy among all the implementation details. For example: `Ci::PipelineProcessing::` and `Ci::PipelineCreation::` could be separate packages of the same bounded context and expose their public API while keeping implementation details private. + - A new bounded context like `RemoteDevelopment::` can be represented a single package while large and complex bounded contexts like `Ci::` would need to be organized into smaller/nested packages. + ## References [List of references](references.md) diff --git a/doc/architecture/blueprints/organization/index.md b/doc/architecture/blueprints/organization/index.md index 09448d6d90c..0955d53313d 100644 --- a/doc/architecture/blueprints/organization/index.md +++ b/doc/architecture/blueprints/organization/index.md @@ -244,6 +244,7 @@ Organizations will have an Owner role. Compared to Users, they can perform the f | View Groups overview | ✓ | ✓ (1) | | View Projects overview | ✓ | ✓ (1) | | View Users overview | ✓ | ✓ (2) | +| View Organization activity page | ✓ | ✓ (1) | | Transfer top-level Group into Organization if Owner of both | ✓ | | (1) Users can only see what they have access to. @@ -251,19 +252,53 @@ Organizations will have an Owner role. Compared to Users, they can perform the f [Roles](../../../user/permissions.md) at the Group and Project level remain as they currently are. +#### Relationship between Organization Owner and Instance Admin + +Users with the (Instance) Admin role can currently [administer a self-managed GitLab instance](../../../administration/index.md). +As functionality is moved to the Organization level, Organization Owners will be able to access more features that are currently only accessible to Admins. +On our SaaS platform, this helps us in empowering enterprises to manage their own Organization more efficiently without depending on the Instance Admin, which is currently a GitLab team member. +On SaaS, we expect the Instance Admin and the Organization Owner to be different users. +Self-managed instances are generally scoped to a single organization, so in this case it is possible that both roles are fulfilled by the same person. +There are situations that might require intervention by an Instance Admin, for instance when Users are abusing the system. +When that is the case, actions taken by the Instance Admin overrule actions of the Organization Owner. +For instance, the Instance Admin can ban or delete a User on behalf of the Organization Owner. + ### Routing -Today only Users, Projects, Namespaces and container images are considered routable entities which require global uniqueness on `https://gitlab.com/<path>/-/`. Initially, Organization routes will be [unscoped](../../../development/routing.md). Organizations will follow the path `https://gitlab.com/-/organizations/org-name/` as one of the design goals is that the addition of Organizations should not change existing Group and Project paths. +Today only Users, Projects, Namespaces and container images are considered routable entities which require global uniqueness on `https://gitlab.com/<path>/-/`. +Initially, Organization routes will be [unscoped](../../../development/routing.md). +Organizations will follow the path `https://gitlab.com/-/organizations/org-name/` as one of the design goals is that the addition of Organizations should not change existing Group and Project paths. + +## Impact of the Organization on Other Domains + +We want a minimal amount of infrequently written tables in the shared database. +If we have high write volume or large amounts of data in the shared database then this can become a single bottleneck for scaling and we lose the horizontal scalability objective of Cells. +With isolation being one of the main requirements to make Cells work, this means that existing features will mostly be scoped to an Organization rather than work across Organizations. +One exception to this are Users, which are stored in the cluster-wide shared database. +For a deeper exploration of the impact on select features, see the [list of features impacted by Cells](../cells/index.md#impacted-features). + +### Alignment between Organization and Fulfillment -### Impact of the Organization on Other Features +Fulfillment is supportive of an entity above top-level groups. Their perspective is outlined in issue [#1138](https://gitlab.com/gitlab-org/fulfillment-meta/-/issues/1138). -We want a minimal amount of infrequently written tables in the shared database. If we have high write volume or large amounts of data in the shared database then this can become a single bottleneck for scaling and we lose the horizontal scalability objective of Cells. With isolation being one of the main requirements to make Cells work, this means that existing features will mostly be scoped to an Organization rather than work across Organizations. One exception to this are Users, which are stored in the cluster-wide shared database. For a deeper exploration of the impact on select features, see the [list of features impacted by Cells](../cells/index.md#impacted-features). +#### Goals of Fulfillment + +- Fulfillment has a longstanding plan to move billing from the top-level Group to a level above. This would mean that a license applies to an Organization and all its top-level Groups. +- Fulfillment uses Zuora for billing and would like to have a 1-to-1 relationship between an Organization and their Zuora entity called BillingAccount. They want to move away from tying a license to a single top-level Group. +- If a customer needs multiple Organizations, they will need to have a separate BillingAccount per each. +- Ideally, a self-managed instance has a single Organization by default, which should be enough for most customers. +- Fulfillment prefers only one additional entity. + +### Open-source Contributions in Organizations + +Several aspects of the current open-source workflow will be impacted by the introduction of Organizations. +We are conducting deeper research around this specific problem in [issue 420804](https://gitlab.com/gitlab-org/gitlab/-/issues/420804). ## Iteration Plan The following iteration plan outlines how we intend to arrive at the Organization MVC. We are following the guidelines for [Experiment, Beta, and Generally Available features](../../../policy/experiment-beta-support.md). -### Iteration 1: Organization Prototype (FY24Q3) +### Iteration 1: Organization Prototype (FY24Q4) In iteration 1, we introduce the concept of an Organization as a way to group top-level Groups together. Support for Organizations does not require any [Cells](../cells/index.md) work, but having them will make all subsequent iterations of Cells simpler. The goal of iteration 1 will be to generate a prototype that can be used by GitLab teams to test basic functionality within an Organization. The prototype contains the following functionality: @@ -278,15 +313,16 @@ In iteration 1, we introduce the concept of an Organization as a way to group to - Users can navigate between the different Organizations they are part of. - Any User within or outside of an Organization can be invited to Groups and Projects contained by the Organization. -### Iteration 2: Organization MVC Experiment (FY24Q4) +### Iteration 2: Organization MVC Experiment (FY25Q1) In iteration 2, an Organization MVC Experiment will be released. We will test the functionality with a select set of customers and improve the MVC based on these learnings. The MVC Experiment contains the following functionality: - Users are listed in the User overview. Every Organization User can access the User overview and see Users that are part of the Groups and Projects they have access to. - Organizations can be deleted. +- Organization Owners can access the Activity page for the Organization. - Forking across Organizations will be defined. -### Iteration 3: Organization MVC Beta (FY24Q4) +### Iteration 3: Organization MVC Beta (FY25Q1) In iteration 3, the Organization MVC Beta will be released. Users will be able to transfer existing top-level Groups into an Organization. @@ -297,7 +333,7 @@ In iteration 3, the Organization MVC Beta will be released. Users will be able t - Top-level Groups can be transferred into an Organization. - The Organization URL path can be changed. -### Iteration 4: Organization MVC GA (FY25Q1) +### Iteration 4: Organization MVC GA (FY25Q2) In iteration 4, the Organization MVC will be rolled out. @@ -343,6 +379,10 @@ A force-option will only be considered if the we do not achieve the load distrib An alternative approach to building Organizations is to convert top-level Groups into Organizations. The main advantage of this approach is that features could be built on top of the Namespace framework and therewith leverage functionality that is already available at the Group level. We would avoid building the same feature multiple times. However, Organizations have been identified as a critical driver of Cells. Due to the urgency of delivering Cells, we decided to opt for the quickest and most straightforward solution to deliver an Organization, which is the lightweight design described above. More details on comparing the two Organization proposals can be found [here](https://gitlab.com/gitlab-org/tenant-scale-group/group-tasks/-/issues/56). +## Frequently Asked Questions + +See [Organization: Frequently Asked Questions](organization-faq.md). + ## Decision Log - 2023-05-10: [Billing is not part of the Organization MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/406614#note_1384055365) diff --git a/doc/architecture/blueprints/organization/organization-faq.md b/doc/architecture/blueprints/organization/organization-faq.md new file mode 100644 index 00000000000..97bd8f057cc --- /dev/null +++ b/doc/architecture/blueprints/organization/organization-faq.md @@ -0,0 +1,44 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Organization: FAQ' +--- + +# Organization: Frequently Asked Questions + +## Do we expect large SaaS customers to be licensed at the Organization level, for example to have the ability to include multiple top-level Groups under on license? + +Yes, this has been discussed with Fulfillment and is part of the post MVC roadmap for Organizations. +See also [Alignment between Organization and Fulfillment](index.md#alignment-between-organization-and-fulfillment). + +## Do we expect to be able to configure alternate GitLab domain names for Organizations (such as `customer.gitlab.com`)? + +There is no plan at this point to allow configuration of alternate GitLab domain names. +We have previously heard that sub-domains bring administrative challenges. +GitLab Dedicated will be a much better fit for that at this moment. + +## Do we expect Organizations to have visibility settings (public/private) of their own? Will visibility remain a property of top-level Groups? + +Organizations are public for now but will have their own independent visibility settings. +See also [When can Users see an Organization?](index.md#when-can-users-see-an-organization). + +## What would the migration of a feature from the top-level Group to the Organization look like? + +One of our requirements is that everything needs to be mapped to an Organization. +Only that way will we achieve the isolation we are striving for. +For SaaS, all existing Groups and Projects are already mapped to `Org_ID = 1` in the backend. +`Org_ID = 1` corresponds to the `Default Organization`, meaning that upon Organization rollout, all existing Groups and Projects will be part of the default Organization and will be seen in that context. +Because we want to achieve as much parity as possible between SaaS and self-managed, self-managed customers would also get everything mapped to the default Organization. +The difference between SaaS and self-managed is that for SaaS we expect users to create many Organizations, and for self-managed we do not. +We will control this via a `can_create_organization` application setting that will be enabled by default on SaaS and disabled by default for self-managed users. + +Consider whether your feature can support cascading, or in other words, whether the functionality is capable of existing on multiple nested levels without causing conflicts. +If your feature can support cascading: + +- Today, you should add your feature to the top-level Group for both SaaS and self-managed, and to the instance for self-managed. +- Once the Organization is ready, you would migrate your instance level feature over the Organization object at which point it would be available at both the Organization and top-level Group for all customers. + +If your feature cannot support cascading: + +- Today, you should add your feature to the top-level Group for SaaS only, and to the instance for self-managed. The top-level Group functionality would be hidden for self-managed users. +- Once the Organization is ready, you would migrate instance functionality to the Organization for self-managed customers, but hide it at the Organization level for SaaS. On SaaS, users would continue to manage their functionality at the top-level Group, and not at the Organization level. At some point in the future when 99% of paying customers have moved to their own Organization, you could clean things up by introducing a breaking change and unhiding it from the Organization level for all customers (SaaS and self-managed) and removing the functionality from the top-level Group. diff --git a/doc/architecture/blueprints/remote_development/index.md b/doc/architecture/blueprints/remote_development/index.md index c7d1ec29add..d64fbfc8b55 100644 --- a/doc/architecture/blueprints/remote_development/index.md +++ b/doc/architecture/blueprints/remote_development/index.md @@ -485,7 +485,11 @@ RestartRequested -left-> Running : status=Running ## Injecting environment variables and files into a workspace -Like CI, there is a need to inject environment variables and files into a workspace. These environment variables and files will be frozen in time during workspace creation to ensure the same values are injected into the workspace every time it starts/restarts. Thus, a new database table, on the lines of `ci_job_variables` will be required. This table will contain the following columns - +Like CI, there is a need to inject environment variables and files into a workspace. +These environment variables and files will be frozen in time during workspace creation to ensure the same values +are injected into the workspace every time it starts/restarts. +Thus, a new database table, on the lines of `ci_job_variables` will be required. +This table will contain the following columns - - `key` - To store the name of the environment variable or the file. - `encrypted_value` - To store the encrypted value of the environment variable or the file. @@ -493,29 +497,40 @@ Like CI, there is a need to inject environment variables and files into a worksp - `workspace_id` - To reference the workspace the environment variable or the file is to be injected into. - `variable_type` - To store whether this data is to be injected as an environment variable or a file. -To perform the encryption, a secret key would be required. This would be uniquely generated for each workpsace upon creation. -Having a unique secret key which is used for encrypting the corresponding workspace's environment variable and file data in the workspace, improves the security profile. +To perform the encryption, the GitLab instance level secret key is used. The data about the environment variables +and files will only be sent to the Agent when required i.e. -Because of the nature of reconciliation loop between Agent and Rails, it is not scalable to decrypt these values at Rails side for each request. -Instead, the `key`, `encrypted_value` and `encrypted_value_iv` of each environment variable of the workspace are sent to the Agent along with the workspace's `secret_key` -for the Agent to decrypt them in place. +- When new workspace creation request has been received from the user and an Agent initiates a Partial Reconciliation request +- When an Agent initiates a Full Reconciliation request -To optimize this further, the data about the environment variables and files along with the secret key will only be sent when required i.e. +More details about the implementation details can be found in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/10882). -- When new workspace creation request has been received from the user and an Agent initiates a Partial Reonciliation request -- When an Agent initiates a Full Reconciliation request +We need to keep in mind potential performance concerns of decrypting workspace variables on the Rails side, +and perform benchmarks of what scale we will reach unacceptably long request times for a reconcile request. -When a workspace is created from a project, it will inherit all the variables from the group/subgroup/project hierarchy which are defined under -[`Settings > CI/CD > Variables`](../../../ci/variables/index.md#define-a-cicd-variable-in-the-ui). This aspect will be generalized to allow for defining `Variables` -which will be inherited in both CI/CD and Workspaces. +e.g. a reconcile request for 100 workspaces with 20 encrypted values each == 2000 decryptions in a single request. -A user will also be able to define, at a user level, environment variables and files to be injected into each workspace created by them. +More details about the benchmarking can be found in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/421504). -When a new workspace is created, a new personal access token associated to the user who created the workspace will be generated. -This personal access token will be tied to the lifecycle of the workspace and will be injected into the workspace as an environment variable or a file -to allow for cloning private projects and supporting transparent Git operations from within the workspace out-of-the-box among other things. +When a workspace is created from a project, it will inherit all the variables from the group/subgroup/project hierarchy +which are defined under [`Settings > CI/CD > Variables`](../../../ci/variables/index.md#define-a-cicd-variable-in-the-ui). +This aspect will be generalized to allow for defining `Variables` which will be inherited in both CI/CD and Workspaces. +A user will also be able to define, at a user level, environment variables and files to be injected into each +workspace created by them. While creating a workspace, a user would be able to override any environment variable +or file that is inherited from the group/subgroup/project/user hierarchy. -More details about the implementation details can be found in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/10882). +## Git operations from within a workspace + +When a new workspace is created, a new personal access token associated to the user who created the workspace +will be generated. This personal access token will be tied to the lifecycle of the workspace and will be injected +into the workspace as a file to allow for cloning private projects and supporting transparent Git operations from +within the workspace out-of-the-box among other things using a +[custom Git credential helper](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). + +[Investigation](https://gitlab.com/gitlab-org/gitlab/-/issues/421289#note_1511631931) into using +ephemeral tokens(JWTs/OAuth/OIDC/etc.) instead of Personal Access Tokens revealed the need to have a common +JWT Authentication/Authorization layer at GitLab which can be tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/421983). +Once such a feature is available, Personal Access Tokens for each workspace would be replaced with JWT tokens. ## Workspace user traffic authentication and authorization diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index 29d7c05c553..7e5ce57dcdc 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -78,7 +78,7 @@ graph TD <!-- vale gitlab.Spelling = NO --> In this proposal, runners created in the GitLab UI are assigned -[authentication tokens](../../../security/token_overview.md#runner-authentication-tokens-also-called-runner-tokens) +[authentication tokens](../../../security/token_overview.md#runner-authentication-tokens) prefixed with `glrt-` (**G**it**L**ab **R**unner **T**oken). <!-- vale gitlab.Spelling = YES --> The prefix allows the existing `register` command to use the authentication token _in lieu_ @@ -97,8 +97,8 @@ token in the `--registration-token` argument: | Token type | Behavior | | ---------- | -------- | -| [Registration token](../../../security/token_overview.md#runner-authentication-tokens-also-called-runner-tokens) | Leverages the `POST /api/v4/runners` REST endpoint to create a new runner, creating a new entry in `config.toml`. | -| [Authentication token](../../../security/token_overview.md#runner-authentication-tokens-also-called-runner-tokens) | Leverages the `POST /api/v4/runners/verify` REST endpoint to ensure the validity of the authentication token. Creates an entry in `config.toml` file and a `system_id` value in a sidecar file if missing (`.runner_system_id`). | +| [Registration token](../../../security/token_overview.md#runner-authentication-tokens) | Leverages the `POST /api/v4/runners` REST endpoint to create a new runner, creating a new entry in `config.toml`. | +| [Runner authentication token](../../../security/token_overview.md#runner-authentication-tokens) | Leverages the `POST /api/v4/runners/verify` REST endpoint to ensure the validity of the authentication token. Creates an entry in `config.toml` file and a `system_id` value in a sidecar file if missing (`.runner_system_id`). | ### Transition period @@ -423,21 +423,21 @@ scope. | Component | Milestone | Changes | |------------------|----------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| GitLab Rails app | `%16.6` | Disable registration tokens for all groups by running database migration (only on GitLab.com) | | -| GitLab Rails app | `%16.6` | Disable registration tokens on the instance level by running database migration (except GitLab.com) | | -| GitLab Rails app | `%16.8` | Disable registration tokens on the instance level for GitLab.com | | +| GitLab Rails app | `%17.0` | Disable registration tokens for all groups by running database migration (only on GitLab.com) | | +| GitLab Rails app | `%17.0` | Disable registration tokens on the instance level by running database migration (except GitLab.com) | | +| GitLab Rails app | `%17.0` | Disable registration tokens on the instance level for GitLab.com | | | GitLab Rails app | `%16.3` | Implement new `:create_runner` PPGAT scope so that we don't require a full `api` scope. | -| GitLab Rails app | | Document gotchas when [automatically rotating runner tokens](../../../ci/runners/configure_runners.md#automatically-rotate-authentication-tokens) with multiple machines. | +| GitLab Rails app | | Document gotchas when [automatically rotating runner tokens](../../../ci/runners/configure_runners.md#automatically-rotate-runner-authentication-tokens) with multiple machines. | ### Stage 7 - Removals | Component | Milestone | Changes | |------------------|----------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| GitLab Rails app | `17.0` | Remove UI enabling registration tokens on the group and instance levels. | -| GitLab Rails app | `17.0` | Remove legacy UI showing registration with a registration token. | -| GitLab Runner | `17.0` | Remove runner model arguments from `register` command (for example `--run-untagged`, `--tag-list`, etc.) | -| GitLab Rails app | `17.0` | Create database migrations to drop `allow_runner_registration_token` setting columns from `application_settings` and `namespace_settings` tables. | -| GitLab Rails app | `17.0` | Create database migrations to drop:<br/>- `runners_registration_token`/`runners_registration_token_encrypted` columns from `application_settings`;<br/>- `runners_token`/`runners_token_encrypted` from `namespaces` table;<br/>- `runners_token`/`runners_token_encrypted` from `projects` table. | +| GitLab Rails app | `18.0` | Remove UI enabling registration tokens on the group and instance levels. | +| GitLab Rails app | `18.0` | Remove legacy UI showing registration with a registration token. | +| GitLab Runner | `18.0` | Remove runner model arguments from `register` command (for example `--run-untagged`, `--tag-list`, etc.) | +| GitLab Rails app | `18.0` | Create database migrations to drop `allow_runner_registration_token` setting columns from `application_settings` and `namespace_settings` tables. | +| GitLab Rails app | `18.0` | Create database migrations to drop:<br/>- `runners_registration_token`/`runners_registration_token_encrypted` columns from `application_settings`;<br/>- `runners_token`/`runners_token_encrypted` from `namespaces` table;<br/>- `runners_token`/`runners_token_encrypted` from `projects` table. | ## FAQ diff --git a/doc/architecture/blueprints/runway/img/runway-architecture.png b/doc/architecture/blueprints/runway/img/runway-architecture.png Binary files differnew file mode 100644 index 00000000000..e577eb7fd15 --- /dev/null +++ b/doc/architecture/blueprints/runway/img/runway-architecture.png diff --git a/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio b/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio new file mode 100644 index 00000000000..ed6366af0cc --- /dev/null +++ b/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio @@ -0,0 +1,189 @@ +<mxfile host="app.diagrams.net" modified="2023-08-04T01:11:28.014Z" agent="Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" etag="tMKdnwQyoIu5vVdj8cNn" version="21.6.6" type="device"> + <diagram name="Page-1" id="zu7ysp4j318SMqtlebVL"> + <mxGraphModel dx="1810" dy="1150" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0"> + <root> + <mxCell id="0" /> + <mxCell id="1" parent="0" /> + <mxCell id="Vbz_GmQMVEEN3iYocOr--46" value="GitLab Vault" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;fillColor=#dae8fc;strokeColor=#6c8ebf;" vertex="1" parent="1"> + <mxGeometry x="130" y="240" width="570" height="270" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--1" value="/runway" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1"> + <mxGeometry x="170" y="280" width="490" height="210" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--2" value="/service1" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;" vertex="1" parent="1"> + <mxGeometry x="200" y="330" width="200" height="130" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--3" value="/production" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="240" y="360" width="120" height="35" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--4" value="/staging" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="240" y="410" width="120" height="35" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--5" value="/service2" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;" vertex="1" parent="1"> + <mxGeometry x="425" y="330" width="200" height="130" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--6" value="/production" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="465" y="360" width="120" height="35" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--7" value="/staging" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="465" y="410" width="120" height="35" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--14" value="GCP Project: gitlab-runway-staging" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1"> + <mxGeometry x="120" y="540" width="590" height="180" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--10" value="" style="strokeColor=#dddddd;shadow=1;strokeWidth=1;rounded=1;absoluteArcSize=1;arcSize=2;" vertex="1" parent="1"> + <mxGeometry x="150" y="580" width="250" height="120" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--11" value="<font color="#000000">service1</font><br>Cloud Run" style="editableCssRules=.*;html=1;fontColor=#999999;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;labelPosition=right;align=left;spacingLeft=20;part=1;points=[];imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjM2NS40NjQ5OTY3NzA0MjQ5MyIgaGVpZ2h0PSIzNzkuMjIyOTk0NDYzNTc3OTUiIHZpZXdCb3g9IjAgMCA5Ni42OTU5OTkxNDU1MDc4MSAxMDAuMzM1OTk4NTM1MTU2MjUiPiYjeGE7PHN0eWxlIHR5cGU9InRleHQvY3NzIj4mI3hhOwkuc3Qwe2ZpbGw6IzQyODVmNDt9JiN4YTsJLnN0MXtmaWxsOiNhZWNiZmE7fSYjeGE7PC9zdHlsZT4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNMjkuNzk0IDEwMC4zMzZMNDYuOTIgNTAuMTY4aDQ5Ljc3NnpNMCA5OS42NzFsMTIuOTc2LTQ5LjUwMkgyOS4yMkwxNi44OTcgOTIuMDU0eiIvPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik0yOS43OTQgMEw0Ni45MiA1MC4xNjhoNDkuNzc2ek0wIC42NjZsMTIuOTc2IDQ5LjUwMkgyOS4yMkwxNi44OTcgOC4yODN6Ii8+JiN4YTs8L3N2Zz4=;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--10"> + <mxGeometry width="30" height="30" relative="1" as="geometry"> + <mxPoint x="15" y="15" as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--15" value="vault agent sidecar" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--10"> + <mxGeometry x="55.55555555555556" y="43.996666666666655" width="166.66666666666666" height="32" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--31" value="crun-service1@gitlab-runway-staging.." style="editableCssRules=.*;html=1;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;aspect=fixed;imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjE2LjQyMDAwMDA3NjI5Mzk0NSIgaGVpZ2h0PSIyMC4wNDk5OTkyMzcwNjA1NDciIGZpbGwtcnVsZT0iZXZlbm9kZCIgdmlld0JveD0iMCAwIDE2LjQyMDAwMDA3NjI5Mzk0NSAyMC4wNDk5OTkyMzcwNjA1NDciPiYjeGE7CTxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+JiN4YTsJLnN0MHtmaWxsOiM0Mjg1ZjQ7fSYjeGE7CS5zdDF7ZmlsbDojNjY5ZGY2O30mI3hhOwk8L3N0eWxlPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik04LjIxIDBMMCAzLjQydjUuNjNjMCA1LjA2IDMuNSA5LjggOC4yMSAxMSA0LjcxLTEuMTUgOC4yMS01Ljg5IDguMjEtMTAuOTVWMy40MnptMCAzLjc5YTIuNjMgMi42MyAwIDAgMSAxLjAwNSA1LjA2QTIuNjMgMi42MyAwIDAgMSA2LjM1IDQuNTZhMi42MyAyLjYzIDAgMCAxIDEuODYtLjc3em00LjExIDExLjE1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTMgOC42NCA4LjY0IDAgMCAxLTQuMTEtMi45M3YtMi4yNWMwLTEuNjcgMi43NC0yLjUyIDQuMTEtMi41MnM0LjExLjg1IDQuMTEgMi41MnoiLz4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNOC4yMSAwdjMuNzlhMi42MyAyLjYzIDAgMSAxIDAgNS4yNnYxLjEyYzEuMzcgMCA0LjExLjg1IDQuMTEgMi41MnYyLjI1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTNWMjBjNC43MS0xLjE1IDguMjEtNS44OSA4LjIxLTEwLjk1VjMuNDJ6Ii8+JiN4YTs8L3N2Zz4=;labelPosition=right;align=left;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--10"> + <mxGeometry x="13.88888888888889" y="90" width="16" height="20" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--18" value="" style="strokeColor=#dddddd;shadow=1;strokeWidth=1;rounded=1;absoluteArcSize=1;arcSize=2;" vertex="1" parent="1"> + <mxGeometry x="425" y="580" width="245" height="120" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--19" value="<font color="#000000">service2</font><br>Cloud Run" style="editableCssRules=.*;html=1;fontColor=#999999;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;labelPosition=right;align=left;spacingLeft=20;part=1;points=[];imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjM2NS40NjQ5OTY3NzA0MjQ5MyIgaGVpZ2h0PSIzNzkuMjIyOTk0NDYzNTc3OTUiIHZpZXdCb3g9IjAgMCA5Ni42OTU5OTkxNDU1MDc4MSAxMDAuMzM1OTk4NTM1MTU2MjUiPiYjeGE7PHN0eWxlIHR5cGU9InRleHQvY3NzIj4mI3hhOwkuc3Qwe2ZpbGw6IzQyODVmNDt9JiN4YTsJLnN0MXtmaWxsOiNhZWNiZmE7fSYjeGE7PC9zdHlsZT4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNMjkuNzk0IDEwMC4zMzZMNDYuOTIgNTAuMTY4aDQ5Ljc3NnpNMCA5OS42NzFsMTIuOTc2LTQ5LjUwMkgyOS4yMkwxNi44OTcgOTIuMDU0eiIvPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik0yOS43OTQgMEw0Ni45MiA1MC4xNjhoNDkuNzc2ek0wIC42NjZsMTIuOTc2IDQ5LjUwMkgyOS4yMkwxNi44OTcgOC4yODN6Ii8+JiN4YTs8L3N2Zz4=;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--18"> + <mxGeometry width="30" height="30" relative="1" as="geometry"> + <mxPoint x="15" y="15" as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--20" value="vault agent sidecar" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--18"> + <mxGeometry x="54.44444444444444" y="43.996666666666655" width="163.33333333333334" height="32" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--34" value="crun-service2@gitlab-runway-staging.." style="editableCssRules=.*;html=1;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;aspect=fixed;imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjE2LjQyMDAwMDA3NjI5Mzk0NSIgaGVpZ2h0PSIyMC4wNDk5OTkyMzcwNjA1NDciIGZpbGwtcnVsZT0iZXZlbm9kZCIgdmlld0JveD0iMCAwIDE2LjQyMDAwMDA3NjI5Mzk0NSAyMC4wNDk5OTkyMzcwNjA1NDciPiYjeGE7CTxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+JiN4YTsJLnN0MHtmaWxsOiM0Mjg1ZjQ7fSYjeGE7CS5zdDF7ZmlsbDojNjY5ZGY2O30mI3hhOwk8L3N0eWxlPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik04LjIxIDBMMCAzLjQydjUuNjNjMCA1LjA2IDMuNSA5LjggOC4yMSAxMSA0LjcxLTEuMTUgOC4yMS01Ljg5IDguMjEtMTAuOTVWMy40MnptMCAzLjc5YTIuNjMgMi42MyAwIDAgMSAxLjAwNSA1LjA2QTIuNjMgMi42MyAwIDAgMSA2LjM1IDQuNTZhMi42MyAyLjYzIDAgMCAxIDEuODYtLjc3em00LjExIDExLjE1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTMgOC42NCA4LjY0IDAgMCAxLTQuMTEtMi45M3YtMi4yNWMwLTEuNjcgMi43NC0yLjUyIDQuMTEtMi41MnM0LjExLjg1IDQuMTEgMi41MnoiLz4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNOC4yMSAwdjMuNzlhMi42MyAyLjYzIDAgMSAxIDAgNS4yNnYxLjEyYzEuMzcgMCA0LjExLjg1IDQuMTEgMi41MnYyLjI1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTNWMjBjNC43MS0xLjE1IDguMjEtNS44OSA4LjIxLTEwLjk1VjMuNDJ6Ii8+JiN4YTs8L3N2Zz4=;labelPosition=right;align=left;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--18"> + <mxGeometry x="14.998888888888871" y="90" width="16" height="20" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--29" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=0;exitY=0.5;exitDx=0;exitDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--15" target="Vbz_GmQMVEEN3iYocOr--4"> + <mxGeometry relative="1" as="geometry"> + <mxPoint x="150" y="650" as="targetPoint" /> + <Array as="points"> + <mxPoint x="206" y="642" /> + <mxPoint x="100" y="642" /> + <mxPoint x="100" y="428" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--54" value="read" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--29"> + <mxGeometry x="-0.1058" y="3" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--30" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--20" target="Vbz_GmQMVEEN3iYocOr--7"> + <mxGeometry width="50" height="50" relative="1" as="geometry"> + <mxPoint x="400" y="610" as="sourcePoint" /> + <mxPoint x="450" y="560" as="targetPoint" /> + <Array as="points"> + <mxPoint x="740" y="640" /> + <mxPoint x="740" y="540" /> + <mxPoint x="740" y="430" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--55" value="read" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--30"> + <mxGeometry x="-0.0904" y="1" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--35" value="GCP Project: gitlab-runway-production" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1"> + <mxGeometry x="120" y="760" width="590" height="180" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--36" value="" style="strokeColor=#dddddd;shadow=1;strokeWidth=1;rounded=1;absoluteArcSize=1;arcSize=2;" vertex="1" parent="1"> + <mxGeometry x="150" y="800" width="250" height="120" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--37" value="<font color="#000000">service1</font><br>Cloud Run" style="editableCssRules=.*;html=1;fontColor=#999999;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;labelPosition=right;align=left;spacingLeft=20;part=1;points=[];imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjM2NS40NjQ5OTY3NzA0MjQ5MyIgaGVpZ2h0PSIzNzkuMjIyOTk0NDYzNTc3OTUiIHZpZXdCb3g9IjAgMCA5Ni42OTU5OTkxNDU1MDc4MSAxMDAuMzM1OTk4NTM1MTU2MjUiPiYjeGE7PHN0eWxlIHR5cGU9InRleHQvY3NzIj4mI3hhOwkuc3Qwe2ZpbGw6IzQyODVmNDt9JiN4YTsJLnN0MXtmaWxsOiNhZWNiZmE7fSYjeGE7PC9zdHlsZT4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNMjkuNzk0IDEwMC4zMzZMNDYuOTIgNTAuMTY4aDQ5Ljc3NnpNMCA5OS42NzFsMTIuOTc2LTQ5LjUwMkgyOS4yMkwxNi44OTcgOTIuMDU0eiIvPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik0yOS43OTQgMEw0Ni45MiA1MC4xNjhoNDkuNzc2ek0wIC42NjZsMTIuOTc2IDQ5LjUwMkgyOS4yMkwxNi44OTcgOC4yODN6Ii8+JiN4YTs8L3N2Zz4=;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--36"> + <mxGeometry width="30" height="30" relative="1" as="geometry"> + <mxPoint x="15" y="15" as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--38" value="vault agent sidecar" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--36"> + <mxGeometry x="55.55555555555556" y="43.996666666666655" width="166.66666666666666" height="32" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--39" value="crun-service1@gitlab-runway-production.." style="editableCssRules=.*;html=1;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;aspect=fixed;imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjE2LjQyMDAwMDA3NjI5Mzk0NSIgaGVpZ2h0PSIyMC4wNDk5OTkyMzcwNjA1NDciIGZpbGwtcnVsZT0iZXZlbm9kZCIgdmlld0JveD0iMCAwIDE2LjQyMDAwMDA3NjI5Mzk0NSAyMC4wNDk5OTkyMzcwNjA1NDciPiYjeGE7CTxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+JiN4YTsJLnN0MHtmaWxsOiM0Mjg1ZjQ7fSYjeGE7CS5zdDF7ZmlsbDojNjY5ZGY2O30mI3hhOwk8L3N0eWxlPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik04LjIxIDBMMCAzLjQydjUuNjNjMCA1LjA2IDMuNSA5LjggOC4yMSAxMSA0LjcxLTEuMTUgOC4yMS01Ljg5IDguMjEtMTAuOTVWMy40MnptMCAzLjc5YTIuNjMgMi42MyAwIDAgMSAxLjAwNSA1LjA2QTIuNjMgMi42MyAwIDAgMSA2LjM1IDQuNTZhMi42MyAyLjYzIDAgMCAxIDEuODYtLjc3em00LjExIDExLjE1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTMgOC42NCA4LjY0IDAgMCAxLTQuMTEtMi45M3YtMi4yNWMwLTEuNjcgMi43NC0yLjUyIDQuMTEtMi41MnM0LjExLjg1IDQuMTEgMi41MnoiLz4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNOC4yMSAwdjMuNzlhMi42MyAyLjYzIDAgMSAxIDAgNS4yNnYxLjEyYzEuMzcgMCA0LjExLjg1IDQuMTEgMi41MnYyLjI1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTNWMjBjNC43MS0xLjE1IDguMjEtNS44OSA4LjIxLTEwLjk1VjMuNDJ6Ii8+JiN4YTs8L3N2Zz4=;labelPosition=right;align=left;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--36"> + <mxGeometry x="13.88888888888889" y="90" width="16" height="20" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--40" value="" style="strokeColor=#dddddd;shadow=1;strokeWidth=1;rounded=1;absoluteArcSize=1;arcSize=2;" vertex="1" parent="1"> + <mxGeometry x="425" y="800" width="245" height="120" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--41" value="<font color="#000000">service2</font><br>Cloud Run" style="editableCssRules=.*;html=1;fontColor=#999999;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;labelPosition=right;align=left;spacingLeft=20;part=1;points=[];imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjM2NS40NjQ5OTY3NzA0MjQ5MyIgaGVpZ2h0PSIzNzkuMjIyOTk0NDYzNTc3OTUiIHZpZXdCb3g9IjAgMCA5Ni42OTU5OTkxNDU1MDc4MSAxMDAuMzM1OTk4NTM1MTU2MjUiPiYjeGE7PHN0eWxlIHR5cGU9InRleHQvY3NzIj4mI3hhOwkuc3Qwe2ZpbGw6IzQyODVmNDt9JiN4YTsJLnN0MXtmaWxsOiNhZWNiZmE7fSYjeGE7PC9zdHlsZT4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNMjkuNzk0IDEwMC4zMzZMNDYuOTIgNTAuMTY4aDQ5Ljc3NnpNMCA5OS42NzFsMTIuOTc2LTQ5LjUwMkgyOS4yMkwxNi44OTcgOTIuMDU0eiIvPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik0yOS43OTQgMEw0Ni45MiA1MC4xNjhoNDkuNzc2ek0wIC42NjZsMTIuOTc2IDQ5LjUwMkgyOS4yMkwxNi44OTcgOC4yODN6Ii8+JiN4YTs8L3N2Zz4=;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--40"> + <mxGeometry width="30" height="30" relative="1" as="geometry"> + <mxPoint x="15" y="15" as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--42" value="vault agent sidecar" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--40"> + <mxGeometry x="54.44444444444444" y="43.996666666666655" width="163.33333333333334" height="32" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--43" value="crun-service2@gitlab-runway-production.." style="editableCssRules=.*;html=1;shape=image;verticalLabelPosition=middle;labelBackgroundColor=#ffffff;verticalAlign=middle;aspect=fixed;imageAspect=0;image=data:image/svg+xml,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnY9Imh0dHBzOi8vdmVjdGEuaW8vbmFubyIgd2lkdGg9IjE2LjQyMDAwMDA3NjI5Mzk0NSIgaGVpZ2h0PSIyMC4wNDk5OTkyMzcwNjA1NDciIGZpbGwtcnVsZT0iZXZlbm9kZCIgdmlld0JveD0iMCAwIDE2LjQyMDAwMDA3NjI5Mzk0NSAyMC4wNDk5OTkyMzcwNjA1NDciPiYjeGE7CTxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+JiN4YTsJLnN0MHtmaWxsOiM0Mjg1ZjQ7fSYjeGE7CS5zdDF7ZmlsbDojNjY5ZGY2O30mI3hhOwk8L3N0eWxlPiYjeGE7CTxwYXRoIGNsYXNzPSJzdDEiIGQ9Ik04LjIxIDBMMCAzLjQydjUuNjNjMCA1LjA2IDMuNSA5LjggOC4yMSAxMSA0LjcxLTEuMTUgOC4yMS01Ljg5IDguMjEtMTAuOTVWMy40MnptMCAzLjc5YTIuNjMgMi42MyAwIDAgMSAxLjAwNSA1LjA2QTIuNjMgMi42MyAwIDAgMSA2LjM1IDQuNTZhMi42MyAyLjYzIDAgMCAxIDEuODYtLjc3em00LjExIDExLjE1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTMgOC42NCA4LjY0IDAgMCAxLTQuMTEtMi45M3YtMi4yNWMwLTEuNjcgMi43NC0yLjUyIDQuMTEtMi41MnM0LjExLjg1IDQuMTEgMi41MnoiLz4mI3hhOwk8cGF0aCBjbGFzcz0ic3QwIiBkPSJNOC4yMSAwdjMuNzlhMi42MyAyLjYzIDAgMSAxIDAgNS4yNnYxLjEyYzEuMzcgMCA0LjExLjg1IDQuMTEgMi41MnYyLjI1YTguNjQgOC42NCAwIDAgMS00LjExIDIuOTNWMjBjNC43MS0xLjE1IDguMjEtNS44OSA4LjIxLTEwLjk1VjMuNDJ6Ii8+JiN4YTs8L3N2Zz4=;labelPosition=right;align=left;" vertex="1" parent="Vbz_GmQMVEEN3iYocOr--40"> + <mxGeometry x="14.998888888888871" y="90" width="16" height="20" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--44" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--38" target="Vbz_GmQMVEEN3iYocOr--3"> + <mxGeometry relative="1" as="geometry"> + <Array as="points"> + <mxPoint x="60" y="860" /> + <mxPoint x="60" y="378" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--53" value="read" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--44"> + <mxGeometry x="0.1399" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--45" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--42" target="Vbz_GmQMVEEN3iYocOr--6"> + <mxGeometry relative="1" as="geometry"> + <Array as="points"> + <mxPoint x="770" y="860" /> + <mxPoint x="770" y="378" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--56" value="read" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--45"> + <mxGeometry x="0.1104" y="-2" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--48" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.882;entryY=0;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--47" target="Vbz_GmQMVEEN3iYocOr--1"> + <mxGeometry relative="1" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--49" value="create/manage" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--48"> + <mxGeometry x="0.0275" y="-1" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--47" value="reliability tooling" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="610" y="80" width="120" height="60" as="geometry" /> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--51" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--50" target="Vbz_GmQMVEEN3iYocOr--2"> + <mxGeometry relative="1" as="geometry"> + <mxPoint x="230" y="160" as="sourcePoint" /> + <Array as="points"> + <mxPoint x="224" y="210" /> + <mxPoint x="300" y="210" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--52" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" edge="1" parent="1" source="Vbz_GmQMVEEN3iYocOr--50" target="Vbz_GmQMVEEN3iYocOr--5"> + <mxGeometry relative="1" as="geometry"> + <Array as="points"> + <mxPoint x="224" y="210" /> + <mxPoint x="525" y="210" /> + </Array> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--57" value="create/manage" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="Vbz_GmQMVEEN3iYocOr--52"> + <mxGeometry x="0.0102" relative="1" as="geometry"> + <mxPoint as="offset" /> + </mxGeometry> + </mxCell> + <mxCell id="Vbz_GmQMVEEN3iYocOr--50" value="runway provisioner" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1"> + <mxGeometry x="164" y="80" width="120" height="60" as="geometry" /> + </mxCell> + </root> + </mxGraphModel> + </diagram> +</mxfile> diff --git a/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio.png b/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio.png Binary files differnew file mode 100644 index 00000000000..b56e326c8c4 --- /dev/null +++ b/doc/architecture/blueprints/runway/img/runway_vault_4_.drawio.png diff --git a/doc/architecture/blueprints/runway/index.md b/doc/architecture/blueprints/runway/index.md new file mode 100644 index 00000000000..becb7914feb --- /dev/null +++ b/doc/architecture/blueprints/runway/index.md @@ -0,0 +1,251 @@ +--- +status: ongoing +creation-date: "2023-07-31" +authors: [ "@igorwwwwwwwwwwwwwwwwwwww", "@ggillies" ] +coach: "@andrewn" +approvers: [ "@marin", "@fzimmer" ] +--- + +<!-- Blueprints often contain forward-looking statements --> +<!-- vale gitlab.FutureTense = NO --> + +# Runway: A PaaS for GitLab + +## Summary + +Runway is an internal Platform as a Service for GitLab, which aims to enable teams to deploy and run their services quickly and safely. + +## Motivation + +<!-- +This section is for explicitly listing the motivation, goals and non-goals of +this blueprint. Describe why the change is important, all the opportunities, +and the benefits to users. + +The motivation section can optionally provide links to issues that demonstrate +interest in a blueprint within the wider GitLab community. Links to +documentation for competing products and services is also encouraged in cases +where they demonstrate clear gaps in the functionality GitLab provides. + +For concrete proposals we recommend laying out goals and non-goals explicitly, +but this section may be framed in terms of problem statements, challenges, or +opportunities. The latter may be a more suitable framework in cases where the +problem is not well-defined or design details not yet established. +--> + +The underlying motivation for this initiative is covered in [the Service Integration blueprint](../gitlab_ml_experiments/index.md). This blueprint can be considered the implementation of the strategic requirements put forward in that proposal. + +### Goals + +<!-- +List the specific goals / opportunities of the blueprint. + +- What is it trying to achieve? +- How will we know that this has succeeded? +- What are other less tangible opportunities here? +--> + +- Development teams aiming to deploy their service without having to worry too much about managing the infrastructure, scaling, monitoring. +- We are focusing on satellite services that are stateless and thus can be autoscaled to meet demand. +- We aim to integrate with existing GitLab features and tooling to provide a streamlined experience. + +### Non-Goals + +<!-- +Listing non-goals helps to focus discussion and make progress. This section is +optional. + +- What is out of scope for this blueprint? +--> + +- Hosting the GitLab monolith: The monolith is a complex application with very specialized requirements, and as such is out of scope. Deployment of the monolith is owned by the Delivery team, and there are other tools and initiatives that target this space, e.g. [Cells](../cells/index.md). +- Arbitrary GCP resources: While we may support a commonly used subset of GCP resources, we will be selective with what we support. If you need more flexibility, you may want to request a [GitLab Sandbox](https://about.gitlab.com/handbook/infrastructure-standards/realms/sandbox/) project instead. +- Arbitrary Kubernetes resources: As a managed platform, we aim not to expose too much of the underlying deployment mechanisms. This allows us to have a well-supported subset and gives us the flexibility to change providers. If you have specialized requirements, getting your own Kubernetes cluster may be a better option. + +## Proposal + +<!-- +This is where we get down to the specifics of what the proposal actually is, +but keep it simple! This should have enough detail that reviewers can +understand exactly what you're proposing, but should not include things like +API designs or implementation. The "Design Details" section below is for the +real nitty-gritty. + +You might want to consider including the pros and cons of the proposed solution so that they can be +compared with the pros and cons of alternatives. +--> + +Runway is a means for deploying a service, packaged up as a Docker image to a production environment. It leverages GitLab CI/CD as well as other GitLab product features to do this. + +## Design and implementation details + +<!-- +This section should contain enough information that the specifics of your +change are understandable. This may include API specs (though not always +required) or even code snippets. If there's any ambiguity about HOW your +proposal will be implemented, this is the place to discuss them. + +If you are not sure how many implementation details you should include in the +blueprint, the rule of thumb here is to provide enough context for people to +understand the proposal. As you move forward with the implementation, you may +need to add more implementation details to the blueprint, as those may become +an important context for important technical decisions made along the way. A +blueprint is also a register of such technical decisions. If a technical +decision requires additional context before it can be made, you probably should +document this context in a blueprint. If it is a small technical decision that +can be made in a merge request by an author and a maintainer, you probably do +not need to document it here. The impact a technical decision will have is +another helpful information - if a technical decision is very impactful, +documenting it, along with associated implementation details, is advisable. + +If it's helpful to include workflow diagrams or any other related images. +Diagrams authored in GitLab flavored markdown are preferred. In cases where +that is not feasible, images should be placed under `images/` in the same +directory as the `index.md` for the proposal. +--> + +The design of Runway aims to decouple individual components in a way that allows them to be changed and replaced over time. + +### Architecture + + + +[Diagram Source](https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/uploads/a6b6646efaa084937ef1f961ad902b59/runway-arch.key) + +[Initial Architecture Discussion](https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/7) + +### Provisioner + +Provisioner is the privileged code that creates service accounts and minimal resources needed by the rest of the system. + +This process is responsible for taking a request "create an experimentation space for me", and stamping out the minimum required infrastructure for that space. It also covers decommissioning when a space is no longer needed. + +- It is currently based on Terraform. +- Terraform runs via CI on the provisioner project. +- We store terraform state in the GitLab Terraform state backend. + +### Reconciler + +The Reconciler is the heart of the system. It is responsible for creating a desired view of the world (based on service definition and current version), finding the differences from the actual state, and then applying that diff. + +Deploying a new version of a service is a matter of invoking the Reconciler. + +This process is responsible for taking an artifact (e.g. a Docker image) from a service developer and bringing that into a runtime. This includes rollout strategies, rollbacks, canarying, multi-environment promotion, as well as diagnostic tools for failed deploys. Some of these capabilities may also be delegated to the runtime. There should also be a standard way for connecting an existing code base to a deployment. + +- It is currently based on Terraform. +- Terraform runs via CI on the deployment project, triggered as a downstream pipeline from the service project. +- We store terraform state in the GitLab Terraform state backend on the deployment project. + +The user-facing integration with the Reconciler is mediated via [`ci-tasks/service-project/runway.yml`](https://gitlab.com/gitlab-com/gl-infra/platform/runway/ci-tasks/-/blob/main/service-project/runway.yml), which is a version-locked CI task that service projects include into their CI config. + +### Runtime + +The runtime is responsible for actually scheduling and running the service workloads. Reconciler targets a runtime. Runtime will provide autoscaled compute resources with a degree of tenant isolation. It will also optionally expose an endpoint at which the workload can be reached. This endpoint will have a DNS name and be TLS encrypted. + +- It is currently based on Cloud Run. +- If we need more flexibility, [Knative](https://knative.dev/docs/) is a likely migration target. + +A service should expose an HTTP port, and it should be stateless (allowing instances to be auto-scaled). Other execution models (e.g. scheduled jobs) may be supported in the future. + +#### Images used by Runtime + +The images deployed by Runway are built by the teams responsible for the service. They are able to build the image in any fashion they wish and keep it inside the GitLab container registry of the service project. As part of the Runway deployment process, this image is mirrored to [GCP Artifact Registry](https://cloud.google.com/artifact-registry) before being consumed by Cloud Run. This is for two reasons: + +1. Cloud run is only able to consume images from GCP Artifact Registry. +1. This means that if for whatever reason the image tag is changed in the future (by error), the image running inside Runway is not affected. + +#### GCP Project Layout + +Runway currently uses shared GCP projects based off three environments (dev, staging, production). These GCP projects are + +- Dev: `runway-dev-527768b3` (managed by IT [HackyStack](https://handbook.gitlab.com/handbook/infrastructure-standards/realms/sandbox/)) +- Staging: `gitlab-runway-staging` (managed by [reliability](https://gitlab.com/gitlab-com/gl-infra/config-mgmt/-/tree/master/environments/runway-staging?ref_type=heads) +- Production: `gitlab-runway-production` (managed by [reliability](https://gitlab.com/gitlab-com/gl-infra/config-mgmt/-/tree/master/environments/runway-production?ref_type=heads) + +### Documents and Schemas used by Runway + +In order for runway to function, there are two JSON/YAML documents in use. They are: + +1. The Runway Inventory Model. This covers what service projects are currently onboarded into Runway. It's located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/blob/main/inventory.json?ref_type=heads). The schema used to validate the docuemnt is located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/blob/main/schemas/service-inventory/v1.0.0-beta/inventory.schema.json?ref_type=heads). There is no backwards compatibility guarenteed to changes to this document schema. This is because it's only used internally by the Runway team, and there is only a single document actually being used by Runway to provision/deprovision Runway services. + +1. The runway Service Model. This is used by Runway users to pass through configuration needed to Runway in order to deploy their service. It's located inside their Service project, at `.runway/runway.yml`. [An example is here](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/.runway/runway.yml?ref_type=heads). The schema used to validate the document is located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/blob/main/schemas/service-manifest/v1.0.0-beta/manifest.schema.json?ref_type=heads). We aim to continue to make improvements and changes to the model, but all changes to the model within the same `kind/apiVersion` must be backwards compatible. In order to +make breaking changes, a new `apiVersion` of the schema will be released. The overall goal is to copy the [Kubernetes model for making API changes](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md). + +There are also [GitLab CI templates](https://gitlab.com/gitlab-com/gl-infra/platform/runway/ci-tasks) used by Runway users in order to automate deployments via Runway through GitLab CI. Users will be encouraged to use tools such as [Renovate bot](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/renovate-bot.md) in order to make sure the CI templates and +version of Runway they are using is up to date. The Runway team will support all released versions of Runway, with the exception of when a security issue is identified. When this happens, Runway users will be expected to update to a version of Runway that contains a fix for the issue as soon as possible (once notification is received). + +### Secrets management + +For secrets management we aim to integrate with our existing HashiCorp Vault setup. We will sync secrets from Vault to whatever secrets store the Runtime integrates best with. For Cloud Run, we will use Google Secret Manager. For Kubernetes, we would use external-secrets to sync to Kubernetes Secret objects. + +The following high level diagram shows the proposed setup of secrets within Vault, for consumption via runway. The general idea is a top level namespace (`/runway`) will be made in Vault, with roles and policies such that: + +- Runway team members have full privileges over the namespace. +- The runway provisioner running in CI has the ability to create/modify/delete new service namespaces at `runway/env/$environment/service`. The environments currently needed are dev, staging, and production. +- The runway reconciler service accounts and GitLab team members will need read only access to `runway/env/$environment/service/$runway_service_id` in order to read secrets for deployment. +- The runway reconciler will mirror secrets in Vault into Google Secrets Manager for consumption in Cloud Run via its native secrets integration. + + + +[Diagram Source](img/runway_vault_4_.drawio) + +### Identity Management, Authentication, Authorization across Runway Components + +The goal of Runway is to not rely on long lived secrets or tokens inside the runway components themselves. In order to achieve this, all pieces of runway authenticate to each other in the following ways. + +#### Service projects to runway deployment projects + +This is handled by GitLab downstream pipeline triggers. Because of this, all permissions are handled within GitLab itself (and API calls to GitLab use short lived `CI_JOB_TOKEN`). We leverage [CI_JOB_TOKEN allowlists](../../../ci/jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist) to allow deployment projects and service projects to interact in API calls (e.g. updating environments in the service project). + +#### Deployment project to GCP Cloud + +GitLab CI pipelines in the deployment project are responsible for talking to GCP to provision and change the cloud resources for a Runway Service. This is done via [OpenID Connnect](../../../ci/cloud_services/google_cloud/index.md) leveraging setup done in the Runway provisioner, in order to make deployment projects authenticate as a GCP service account +with restricted permissions. + +#### Reconciler to GCP Cloud + +The reconciler (`runwayctl` wrapping around `terraform`), runs inside GitLab CI in the deployment project. This uses a specific service account setup for each Runway service, with only the permissions it needs to manipulate the GCP apis for its work. The authentication for this is handled by GitLab CI as described above. + +### Observability + +Observability will be covered [in a separate blueprint](https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/76). + +## Alternative Solutions + +<!-- +It might be a good idea to include a list of alternative solutions or paths considered, although it is not required. Include pros and cons for +each alternative solution/path. + +"Do nothing" and its pros and cons could be included in the list too. +--> + +### Unmanaged GCP project + +Instead of building a managed platform, we could give teams a GCP project and let them have at it. In fact, this is what we have done for a few services. It brings excellent isolation and flexibility. There are however several issues with this approach: + +- **Missing Infrastructure-as-Code:** Without any existing structure, teams may be inclined to create cloud resources via UI without using Terraform or similar IaC tools. This makes it very difficult to audit changes and re-provision the infrastructure across several environments. +- **Sprawling:** Without sane and safe defaults, every service will need to develop their own approach for deploying services. This also makes it very difficult for anyone else to contribute to these services. +- **Non-scalable design:** By being able to create arbitrary VMs, it can be tempting to co-locate services on a single machine without thinking about horizontal scalability. + +### Unmanaged Kubernetes cluster or namespace + +Instead of building a custom platform, we could decide that Kubernetes is the platform, and let teams have either their own cluster or their own namespace in a shared cluster. + +Some challenges with this approach: + +- **Baseline cost for GKE:** The management fee is $70 per month. If we aim to have many services across multiple environments, some of them receiving low traffic volume, then a separate cluster per service is not cost effective. This suggests we would want to go with a shared cluster. +- **Developer friendliness:** Kubernetes is growing in popularity but it's not the most developer friendly interface. There are a lot of concepts and abstractions to grapple with. A narrower interface a la "deploy this container and give me a port", while less flexible, has a much lower barrier to entry. + +It is plausible that Runway will evolve more in this direction. + +### GCP project per service + +GCP projects provide a very solid foundation for resource isolation and cost attribution. Ideally we would leverage such a model. However, the lack of shared resources comes at a significant baseline cost per service. + +If we want to make use of GKE, the per-cluster management fee of $70 per month would likely result in the minimum cost for a service being $140 per month. For small services, this is not cost effective. + +An alternative to consider would be running Kubernetes without GKE. This then means we need to manage that Kubernetes deployment ourselves, and stay on top of the differences to GKE. + +Another alternative to consider is to use Cloud Run for small services and GKE for larger ones. This however requires maintaining compatibility with Cloud Run, and potentially be limited to the lowest common denominator in terms of features. + +Some hybrid is possible (e.g. give a Runway service its own project), but as long as we have shared resources, we need to implement permission control on a per-resource level. diff --git a/doc/architecture/blueprints/transfer_data/index.md b/doc/architecture/blueprints/transfer_data/index.md new file mode 100644 index 00000000000..9eea86d91e7 --- /dev/null +++ b/doc/architecture/blueprints/transfer_data/index.md @@ -0,0 +1,141 @@ +--- +status: proposed +creation-date: "2023-09-07" +authors: [ "@vyaklushin" ] +approvers: [ "@ofernandez2", "@sean_carroll" ] +coach: ["@andrewn", "@grzesiek"] +owning-stage: "~group::source_code" +participating-stages: [] +--- + +# Transfer data + +## Summary + +GitLab already provides users transparency on their Usage Quotas. + +We currently display data about: + +- used license seats +- used storage +- CI/CD minutes usage + +But we don't collect and present transfer data (egress traffic caused by +various parts of the application). + +Collecting data about number of transferred bytes between clients, customers +and services will allow us to discover new efficiencies and reduce the +operational risk. We want to better understand the data transfer +patterns across the whole application stack. + +The goal of this blueprint to describe steps we need to do to achieve the result. + +### Goals + +Explore various solutions to store, process and present transfer data across the +whole application stack. + +## Proposal + +There are different types of transferred data. + +| Type | Description | +| --------------- | ----------------------------------------------------------- | +| `Repository` | Egress data related to Git `fetch` operations (pull, clone) | +| `Artifacts` | Artifacts transfer caused by direct and proxied egress | +| `Pages` | Pages egress (depends on Artifacts API) | +| `Packages` | Package registry egress | +| `Registry` | Container registry egress | +| `Uploads` | Object store egress | + +Each type has different implementations and can be measured separately but +collection of metadata / data transfer telemetry and consuming / visualizing it, +should be built on top of the same abstraction. + +## Overview + +```mermaid +flowchart TB + + A[Applications] -->|send logs| Pub(Google Pub/Sub) + Pub -->JSONParser + + + subgraph DataflowPipeline + direction TB + + JSONParser -->|selects only JSON lines| LogProcessor + LogProcessor -->|insert only data transfer logs|ClickHouse + end + +ClickHouse -->|query transfer logs| Rails +``` + +### Applications + +Every application produces logs in structured format. Logs related +to transfer data requests have metadata fields that include the +number of bytes transferred, namespace, project, and timestamp +of the egress event. + +### Google Pub/Sub + +Application logs are collected and sent to Google Pub/Sub. +Pub/Sub allows to subscribe to topics and read incoming logs. + +### Dataflow pipeline + +[Dataflow](https://cloud.google.com/dataflow/docs/overview) is a Google +Cloud unified stream and batch data processing that's serverless, fast +, and cost-effective. It's built on the open-source +[Apache Beam](https://beam.apache.org/) project. + +Dataflow pipeline provides a data processing abstraction that can be written +in Java, Python or Go. + +The Dataflow pipeline is a core of the processing logic. It relies on the streaming +implementation of Dataflow. The pipeline subscribes to Pub/Sub topics, +reads, processes logs, and inserts them into ClickHouse database. + +### ClickHouse + +ClickHouse is designed to provide a fast access to work with massive +data sets. It will allow customers to query aggregated data for +dynamic timeframes. + +ClickHouse is an abstract store for logs. The Dataflow pipeline will +transform different input sources into consistent structure to be +stored in ClickHouse. That allows to support various inputs and formats +without affecting ClickHouse-stored timeseries. + +ClickHouse table schema + +```sql +CREATE TABLE transfer_data +( + created_at DateTime, + bytes UInt64, + project String, + namespace String, + type String +) +ENGINE = MergeTree +PRIMARY KEY (project, namespace) +``` + +`created_at` - a timestamp of the event +`bytes` - a number of transferred bytes +`project` - a full project path +`namespace` - a root namespace of the project +`type` - a type of egress (`git`, `container_registry`, ...) + +### Rails + +Rails application uses [a gem to connect and query ClickHouse](../../../development/database/clickhouse/clickhouse_within_gitlab.md). +Customers will be able see their transfer data details in their dashboard. +They can request a transfer data report for their whole namespace or +for particular projects. + +## Implementation proposals + +- [Repository egress](repository.md) diff --git a/doc/architecture/blueprints/transfer_data/repository.md b/doc/architecture/blueprints/transfer_data/repository.md new file mode 100644 index 00000000000..7d3b6be8150 --- /dev/null +++ b/doc/architecture/blueprints/transfer_data/repository.md @@ -0,0 +1,67 @@ +--- +status: proposed +creation-date: "2023-09-07" +authors: [ "@vyaklushin" ] +approvers: [ "@ofernandez2", "@sean_carroll" ] +coach: ["@andrewn", "@grzesiek"] +owning-stage: "~group::source_code" +participating-stages: [] +--- + +# Repository egress + +Users generate Repository egress events for every fetch operation in Git. It +includes commands like `git clone`, `git fetch` and `git pull`, because all of +them request data from Gitaly that needs to be delivered to the end user. + +Two main clients of Gitaly traffic are +[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell)(for SSH traffic) +and +[Workhorse](https://gitlab.com/gitlab-org/gitlab/-/tree/master/workhorse) +(for HTTP traffic). + +Both clients send `git-upload-pack` command to Gitaly and stream back the +Git response that contains requested changes. + +## Current metrics + +| Service | Number of `git-upload-pack` events (per day) | +|--------------|----------------------------------------------| +| Workhorse | ~80 million | +| GitLab Shell | ~85 million | +| Gitaly | ~165 million (combined traffic) | + +Kibana links to see current metrics for each service: + +- [Workhorse](https://log.gprd.gitlab.net/goto/cf799060-e2b2-11ed-8afc-c9851e4645c0) +- [GitLab Shell](https://log.gprd.gitlab.net/goto/bd93f5c0-e2b2-11ed-a017-0d32180b1390) +- [Gitaly](https://log.gprd.gitlab.net/goto/9221c230-e2b4-11ed-8afc-c9851e4645c0) + +Total number of events: + +- 165 million per day +- 7.5 million per hour +- 120 thousand per minute + +## Logs structure + +### HTTP traffic + +Captured in Workhorse logs. + +| Fields | Description | +|---------------|-----------------------------| +| written_bytes | number of bytes transferred | +| uri | namespace and project name | +| timestamp | timestamp of Egress event | + +### SSH traffic + +Captured in GitLab Shell logs. + +| Fields | Description | +|---------------|-----------------------------| +| written_bytes | number of bytes transferred | +| project | full project name | +| root_namspace | root namespace | +| timestamp | timestamp of Egress event | diff --git a/doc/architecture/blueprints/work_items/index.md b/doc/architecture/blueprints/work_items/index.md index 9924b0db9f4..6f5b48fffcb 100644 --- a/doc/architecture/blueprints/work_items/index.md +++ b/doc/architecture/blueprints/work_items/index.md @@ -46,6 +46,9 @@ Work is underway to convert existing objects to Work Item Types or add new ones: Every Work Item type has the following common properties: +**NOTE:** +You can also refer to fields of [Work Item](../../../api/graphql/reference/index.md#workitem) to learn more. + - `id` - a unique Work Item global identifier; - `iid` - internal ID of the Work Item, relative to the parent workspace (currently workspace can only be a project) - Work Item type; @@ -63,20 +66,25 @@ All Work Item types share the same pool of predefined widgets and are customized ### Work Item widget types (updating) -| widget type | feature flag | -|---|---| -| assignees | | -| description | | -| hierarchy | | -| [iteration](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) | | -| [milestone](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) | | -| labels | | -| start and due date | | -| status\* | | -| weight | | -| [notes](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) | | - -\* status is not currently a widget, but a part of the root work item, similar to title +| Widget | Description | feature flag | +|---|---|---| +| [WorkItemWidgetAssignees](../../../api/graphql/reference/index.md#workitemwidgetassignees) | List of work item assignees | | +| [WorkItemWidgetAwardEmoji](../../../api/graphql/reference/index.md#workitemwidgetawardemoji) | Emoji reactions added to work item, including support for upvote/downvote counts | | +| [WorkItemWidgetCurrentUserTodos](../../../api/graphql/reference/index.md#workitemwidgetcurrentusertodos) | User todo state of work item | | +| [WorkItemWidgetDescription](../../../api/graphql/reference/index.md#workitemwidgetdescription) | Description of work item, including support for edited state, timestamp, and author | | +| [WorkItemWidgetHealthStatus](../../../api/graphql/reference/index.md#workitemwidgethealthstatus) | Health status assignment support for work item | | +| [WorkItemWidgetHierarchy](../../../api/graphql/reference/index.md#workitemwidgethierarchy) | Hierarchy of work items, including support for boolean representing presence of children. **Note:** Hierarchy is currently available only for OKRs. | `okrs_mvc` | +| [WorkItemWidgetIteration](../../../api/graphql/reference/index.md#workitemwidgetiteration) | Iteration assignment support for work item | | +| [WorkItemWidgetLabels](../../../api/graphql/reference/index.md#workitemwidgetlabels) | List of labels added to work items, including support for checking whether scoped labels are supported | +| [WorkItemWidgetLinkedItems](../../../api/graphql/reference/index.md#workitemwidgetlinkeditems) | List of work items added as related to a given work item, with possible relationship types being `relates_to`, `blocks`, and `blocked_by`. Includes support for individual counts of blocked status, blocked by, blocking, and related to. | `linked_work_items` | +| [WorkItemWidgetMilestone](../../../api/graphql/reference/index.md#workitemwidgetmilestone) | Milestone assignment support for work item | | +| [WorkItemWidgetNotes](../../../api/graphql/reference/index.md#workitemwidgetnotes) | List of discussions within a work item | | +| [WorkItemWidgetNotifications](../../../api/graphql/reference/index.md#workitemwidgetnotifications) | Notifications subscription status of a work item for current user | | +| [WorkItemWidgetProgress](../../../api/graphql/reference/index.md#workitemwidgetprogress) | Progress value of a work item. **Note:** Progress is currently available only for OKRs. | `okrs_mvc` | +| [WorkItemWidgetStartAndDueDate](../../../api/graphql/reference/index.md#workitemwidgetstartandduedate) | Set start and due dates for a work item | | +| [WorkItemWidgetStatus](../../../api/graphql/reference/index.md#workitemwidgetstatus) | Status of a work item when type is Requirement, with possible status types being `unverified`, `satisfied`, or `failed` | | +| [WorkItemWidgetTestReports](../../../api/graphql/reference/index.md#workitemwidgettestreports) | Test reports associated with a work item | | +| [WorkItemWidgetWeight](../../../api/graphql/reference/index.md#workitemwidgetweight) | Set weight of a work item | | ### Work item relationships diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index be5112251a4..7aeafce9352 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -535,7 +535,7 @@ and should only be disabled in an environment where all users with Developer rol To use the same cache for all branches: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Clear the **Use separate caches for protected branches** checkbox. @@ -630,7 +630,7 @@ The next time the pipeline runs, the cache is stored in a different location. You can clear the cache in the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipelines**. 1. In the upper-right corner, select **Clear runner caches**. diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 7164fae10a1..1820cf77841 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -15,8 +15,8 @@ GitLab CI/CD can be used with Bitbucket Cloud by: To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **Repository by URL**. @@ -40,7 +40,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: using the Personal Access Token we just generated for authentication. ```plaintext - https://gitlab.com/api/v4/projects/<PROJECT_ID>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + https://gitlab.example.com/api/v4/projects/:project_id/mirror/pull?private_token=<your_personal_access_token> ``` The web hook Trigger should be set to 'Repository Push'. diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 1fad7ad5a53..bc61990fcd8 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -34,8 +34,8 @@ repositories: `repo` and `admin:repo_hook` so that GitLab can access your project, update commit statuses, and create a web hook to notify GitLab of new commits. 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub**. @@ -63,8 +63,8 @@ To manually enable GitLab CI/CD for your repository: 1. Enter a **Token description** and update the scope to allow `repo` so that GitLab can access your project and update commit statuses. 1. In GitLab, create a project: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. Select **New project**. 1. Select **Run CI/CD for external repository** and **Repository by URL**. 1. In the **Git repository URL** field, enter the HTTPS URL for your GitHub repository. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index a9093632a8c..76996294a96 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -18,14 +18,14 @@ external repository to get the benefits of GitLab CI/CD. Connecting an external repository sets up [repository mirroring](../../user/project/repository/mirror/index.md) and creates a lightweight project with issues, merge requests, wiki, and snippets disabled. These features -[can be re-enabled later](../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). +[can be re-enabled later](../../user/project/settings/index.md#configure-project-features-and-permissions). ## Connect to an external repository To connect to an external repository: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. Select **New project**. 1. Select **Run CI/CD for external repository**. 1. Select **GitHub** or **Repository by URL**. @@ -33,7 +33,7 @@ To connect to an external repository: If the **Run CI/CD for external repository** option is not available, the GitLab instance might not have any import sources configured. Ask an administrator for your instance to check -the [import sources configuration](../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources). +the [import sources configuration](../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). ## Pipelines for external pull requests diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index f50b7be855a..2a73184eb7f 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -17,7 +17,7 @@ Ensure your own [runners are configured](../../runners/index.md). ## Prerequisites -- An [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). +- An [AWS account](https://repost.aws/knowledge-center/create-and-activate-aws-account). Sign in with an existing AWS account or create a new one. - In this guide, you create an infrastructure in [`us-east-2` region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). You can use any region, but do not change it after you begin. diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index f0183fc7ba2..dd36da86cca 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. In this tutorial, we'll show you how to use a GitLab CI/CD job with a JSON web token (JWT) to retrieve temporary credentials from AWS without needing to store secrets. To do this, you must configure OpenID Connect (OIDC) for ID federation between GitLab and AWS. For background and requirements for integrating GitLab using OIDC, see [Connect to cloud services](../index.md). @@ -85,9 +85,12 @@ assume role: - `ROLE_ARN`: The role ARN defined in this [step](#configure-a-role-and-trust). - `GITLAB_OIDC_TOKEN`: An OIDC [ID token](../../yaml/index.md#id_tokens). -## Working example +## Working examples -See this [reference project](https://gitlab.com/guided-explorations/aws/configure-openid-connect-in-aws) for provisioning OIDC in AWS using Terraform and a sample script to retrieve temporary credentials. +- See this [reference project](https://gitlab.com/guided-explorations/aws/configure-openid-connect-in-aws) for provisioning OIDC in AWS using Terraform and a sample script to retrieve temporary credentials. +- [OIDC and Multi-Account Deployment with GitLab and ECS](https://gitlab.com/guided-explorations/aws/oidc-and-multi-account-deployment-with-ecs). +- AWS Partner (APN) Blog: [Setting up OpenID Connect with GitLab CI/CD](https://aws.amazon.com/blogs/apn/setting-up-openid-connect-with-gitlab-ci-cd-to-provide-secure-access-to-environments-in-aws-accounts/). +- [GitLab at AWS re:Inforce 2023: Secure GitLab CD pipelines to AWS w/ OpenID and JWT](https://www.youtube.com/watch?v=xWQGADDVn8g). ## Troubleshooting diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index fa8ff506dd0..3a882cf6820 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. This tutorial demonstrates how to use a JSON web token (JWT) in a GitLab CI/CD job to retrieve temporary credentials from Azure without needing to store secrets. @@ -135,7 +135,7 @@ The CI/CD variables are: - `AZURE_CLIENT_ID`: The [application client ID you saved earlier](#create-azure-ad-application-and-service-principal). - `AZURE_TENANT_ID`: Your Azure Active Directory. You can - [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-how-to-find-tenant). + [find it by using the Azure CLI or Azure Portal](https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/how-to-find-tenant). - `GITLAB_OIDC_TOKEN`: An OIDC [ID token](../../yaml/index.md#id_tokens). ## Troubleshooting diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md index f2318b818d8..a733f3d59cb 100644 --- a/doc/ci/cloud_services/google_cloud/index.md +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT_V2` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and is scheduled to be removed in GitLab 16.5. Use [ID tokens](../../yaml/index.md#id_tokens) instead. +and is scheduled to be removed in GitLab 17.0. Use [ID tokens](../../yaml/index.md#id_tokens) instead. This tutorial demonstrates authenticating to Google Cloud from a GitLab CI/CD job using a JSON Web Token (JWT) token and Workload Identity Federation. This configuration diff --git a/doc/ci/cloud_services/index.md b/doc/ci/cloud_services/index.md index 6896fffcce7..87984c424c4 100644 --- a/doc/ci/cloud_services/index.md +++ b/doc/ci/cloud_services/index.md @@ -12,9 +12,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: `CI_JOB_JWT` and `CI_JOB_JWT_V2` were [deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and are scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. +and are scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. -GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) to +GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/) to give your build and deployment jobs access to cloud credentials and services. Historically, teams stored secrets in projects or applied permissions on the GitLab Runner instance to build and deploy. OIDC capable [ID tokens](../yaml/index.md#id_tokens) are configurable diff --git a/doc/ci/components/catalog.md b/doc/ci/components/catalog.md new file mode 100644 index 00000000000..2194e72d56c --- /dev/null +++ b/doc/ci/components/catalog.md @@ -0,0 +1,33 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# CI/CD catalog **(PREMIUM ALL EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. + +The CI/CD catalog is a list of [components repositories](index.md#components-repository), +each containing resources that you can add to your CI/CD pipelines. + +## Mark a components repository as a catalog resource + +After components are added to a components repository, they can immediately be [used](index.md#use-a-component-in-a-cicd-configuration) +to build pipelines in other projects. + +However, this repository is not discoverable. You must mark this project as a catalog resource +to allow it to be visible in the CI/CD Catalog so other users can discover it. + +To mark a project as a catalog resource: + +1. On the left sidebar, select **Search or go to** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Scroll down to **CI/CD Catalog resource** and select the toggle to mark the project as a catalog resource. + +Ensure the project has a clear [description](../../user/project/settings/index.md#edit-project-name-and-description), +as the project description is displayed in the component list in the catalog. + +NOTE: +This action is not reversible. diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 4a739bdfcf6..e73436522dc 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -2,29 +2,32 @@ stage: Verify group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference --- -# CI/CD Components (Experimental) +# CI/CD components **(FREE ALL EXPERIMENT)** > - Introduced as an [experimental feature](../../policy/experiment-beta-support.md) in GitLab 16.0, [with a flag](../../administration/feature_flags.md) named `ci_namespace_catalog_experimental`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/groups/gitlab-org/-/epics/9897) in GitLab 16.2. -> - [Feature flag `ci_namespace_catalog_experimental` removed.](https://gitlab.com/gitlab-org/gitlab/-/issues/394772) in GitLab 16.3. +> - [Feature flag `ci_namespace_catalog_experimental` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/394772) in GitLab 16.3. This feature is an experimental feature and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. Tell us about your use case by leaving comments in the epic. -## Components Repository +## Components repository -A components repository is a GitLab project with a repository that hosts one or more pipeline components. A pipeline component is a reusable single pipeline configuration unit. You can use them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). +A components repository is a GitLab project with a repository that hosts one or more pipeline components. +A pipeline component is a reusable single pipeline configuration unit. Use them to compose +an entire pipeline configuration or a small part of a larger pipeline. -### Create a components repository +A component can optionally take [input parameters](../yaml/inputs.md). + +## Create a components repository To create a components repository, you must: 1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. - -1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. For example: +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. + For example: ```yaml spec: @@ -39,14 +42,71 @@ To create a components repository, you must: ### Directory structure -A components repository can host one or more components. +A components repository can host one or more components, and must follow a mandatory file structure. + +Component configurations can be saved through the following directory structure, containing: + +- A `templates` directory at the top level of your components repository. All component configuration files + should be saved under this directory. +- Files ending in `.yml` containing the component configurations, one file per component. +- A Markdown `README.md` file explaining the details of all the components in the repository. + +For example, if the project contains a single component and a pipeline to test the component, +the file structure should be similar to: + +```plaintext +├── templates/ +│ └── only_template.yml +├── README.md +└── .gitlab-ci.yml +``` + +This example component could be referenced with a path similar to `gitlab.com/my-username/my-component/only_template@<version>`, +if the project is: + +- On GitLab.com +- Named `my-component` +- In a personal namespace named `my-username` + +The templates directory and the suffix of the configuration file should be excluded from the referenced path. + +If the project contains multiple components, then the file structure should be similar to: + +```plaintext +├── README.md +├── .gitlab-ci.yml +└── templates/ + └── all-scans.yml + └── secret-detection.yml +``` + +These components would be referenced with these paths: -Components repositories must follow a mandatory file structure, containing: +- `gitlab.com/my-username/my-component/all-scans` +- `gitlab.com/my-username/my-component/secret-detection` + +You can omit the filename in the path if the configuration file is named `template.yml`. +For example, the following component could be referenced with `gitlab.com/my-username/my-component/dast`: + +```plaintext +├── README.md +├── .gitlab-ci.yml +├── templates/ +│ └── dast +│ └── template.yml +``` + +#### Component configurations saved in any directory (deprecated) + +NOTE: +Saving component configurations through this directory structure is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/415855). + +Components configurations can be saved through the following directory structure, containing: - `template.yml`: The component configuration, one file per component. If there is only one component, this file can be in the root of the project. If there are multiple components, each file must be in a separate subdirectory. -- `README.md`: A documentation file explaining the details of the all the components in the repository. +- `README.md`: A documentation file explaining the details of all the components in the repository. For example, if the project is on GitLab.com, named `my-component`, and in a personal namespace named `my-username`: @@ -60,6 +120,9 @@ namespace named `my-username`: └── .gitlab-ci.yml ``` + The `.gitlab-ci.yml` file is not required for a CI/CD component to work, but + [testing the component](#test-the-component) in a pipeline in the project is recommended. + This component is referenced with the path `gitlab.com/my-username/my-component@<version>`. - Containing one default component and multiple sub-components, then the file structure @@ -95,279 +158,281 @@ Nesting of components is not possible. For example: │ └── nested_template.yml ``` -### Test a component +## Release a component + +To create a release for a CI/CD component, use either: + +- The [`release`](../yaml/index.md#release) keyword in a CI/CD pipeline. Like in the + [component testing example](#test-the-component), you can set a component to automatically + be released after all tests pass in pipelines for new tags. +- The [UI for creating a release](../../user/project/releases/index.md#create-a-release). + +All released versions of the components are displayed in the CI/CD Catalog +page for the given resource, providing users with information about official releases. -Testing components as part of the development workflow to ensure that quality maintains high standards is strongly recommended. +Components [can be used](#use-a-component-in-a-cicd-configuration) without being released, +but only with a commit SHA or a branch name. To enable the use of tags or the `~latest` version keyword, +you must create a release. -Testing changes in a CI/CD pipeline can be done, like any other project, by creating a `.gitlab-ci.yml` in the root directory. +## Use a component in a CI/CD configuration +You can add a component to a CI/CD configuration with the `include: component` keyword. For example: ```yaml include: - # include the component located in the current project from the current SHA - - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + - component: gitlab.example.com/my-namespace/my-component@1.0 inputs: stage: build - -stages: [build, test, release] - -# Expect `component-job` is added. -# This is an example of testing that the included component works as expected. -# You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. -ensure-job-added: - stage: test - image: badouralix/curl-jq - script: - - | - route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" - count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` - if [ "$count" != "1" ]; then - exit 1 - fi - -# If we are tagging a release with a specific convention ("v" + number) and all -# previous checks succeeded, we proceed with creating a release automatically. -create-release: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG =~ /^v\d+/ - script: echo "Creating release $CI_COMMIT_TAG" - release: - tag_name: $CI_COMMIT_TAG - description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" ``` -After committing and pushing changes, the pipeline tests the component then releases it if the test passes. +The component is identified by a unique address in the form `<fully-qualified-domain-name>/<component-path>@<specific-version>`, +where: -### Release a component +- `<fully-qualified-domain-name>` matches the GitLab host. You can only reference components + in the same GitLab instance as your project. +- `<component-path>` is the component project's full path and directory where the + component YAML file is located. +- `<specific-version>` is the version of the component. In order of highest priority first, + the version can be: + - A branch name, for example `main`. + - A commit SHA, for example `e3262fdd0914fa823210cdb79a8c421e2cef79d8`. + - A tag, for example: `1.0`. If a tag and branch exist with the same name, the tag + takes precedence over the branch. If a tag and commit SHA exist with the same name, + the commit SHA takes precedence over the tag. + - `~latest`, which is a special version that always points to the most recent released tag. + Available only if the component has been [released](#release-a-component). -Component repositories are released using the [`release`](../yaml/index.md#release) keyword within a CI pipeline. +For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`, +the path: -Like in the [example above](#test-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. +- `gitlab.com/gitlab-org/dast@main` targets the `template.yml` in the root directory + on the `main` branch. +- `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` targets the same file + for the specified commit SHA. +- `gitlab.com/gitlab-org/dast@1.0` targets the same file for the `1.0` tag. +- `gitlab.com/gitlab-org/dast@~latest` targets the same file for the latest release. +- `gitlab.com/gitlab-org/dast/api-scan@main` targets a different file, the `template.yml` + in the `/api-scan` directory in the component repository, for the `main` branch. -All released versions of the components repository are displayed in the Components Catalog page for the given resource, providing users with information about official releases. +## Best practices -### Use a component in a CI/CD configuration +### Avoid using global keywords -A pipeline component is identified by a unique address in the form `<fully-qualified-doman-name>/<component-path>@<version>` -containing: +Avoid using [global keywords](../yaml/index.md#global-keywords) in a component. +Using these keywords in a component affects all jobs in a pipeline, including jobs +directly defined in the main `.gitlab-ci.yml` or in other included components. -- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. -- **A specific version**: The version of the component can be (in order of highest priority first): - - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - - `~latest`, which is a special version that always points to the most recent released tag, - for example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example `gitlab.com/gitlab-org/dast@main`. -- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. +As an alternative to global keywords, instead: -For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: +- Add the configuration directly to each job, even if it creates some duplication + in the component configuration. +- Use the [`extends`](../yaml/index.md#extends) keyword in the component. -- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. -- The path `gitlab.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. +For example, using the `default` keyword is not recommended: -**Additional notes:** +```yaml +# Not recommended +default: + image: ruby:3.0 -- You can only reference components in the same GitLab instance as your project. -- If a tag and branch exist with the same name, the tag takes precedence over the branch. -- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, - the commit SHA takes precedence over the tag. +rspec-1: + script: bundle exec rspec dir1/ -### Best practices +rspec-2: + script: bundle exec rspec dir2/ +``` -#### Avoid using global keywords +Instead, you can: -When using [global keywords](../yaml/index.md#global-keywords) all jobs in the -pipeline are affected. Using these keywords in a component affects all jobs in a -pipeline, whether they are directly defined in the main `.gitlab-ci.yml` or -in any included components. +- Add the configuration to each job: -To make the composition of pipelines more deterministic, either: + ```yaml + rspec-1: + image: ruby:3.0 + script: bundle exec rspec dir1/ -- Duplicate the default configuration for each job. -- Use [`extends`](../yaml/index.md#extends) feature within the component. + rspec-2: + image: ruby:3.0 + script: bundle exec rspec dir2/ + ``` -```yaml -## -# BAD -default: - image: ruby:3.0 +- Use `extends` to reuse configuration: -rspec: - script: bundle exec rspec -``` + ```yaml + .rspec-image: + image: ruby:3.0 -```yaml -## -# GOOD -rspec: - image: ruby:3.0 - script: bundle exec rspec -``` + rspec-1: + extends: + - .rspec-image + script: bundle exec rspec dir1/ -#### Replace hard-coded values with inputs + rspec-2: + extends: + - .rspec-image + script: bundle exec rspec dir2/ + ``` -A typical hard-coded value found in CI templates is `stage:` value. Such hard coded values may force the user -of the component to know and adapt the pipeline to such implementation details. +### Replace hard-coded values with inputs -For example, if `stage: test` is hard-coded for a job in a component, the pipeline using the component must -define the `test` stage. Additionally, if the user of the component want to customize the stage value it has -to override the configuration: +Avoid hard-coding values in CI/CD components. Hard-coded values might force +component users to need to review the component's internal details and adapt their pipeline +to work with the component. -```yaml -## -# BAD: In order to use different stage name you need to override all the jobs -# included by the component. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +A common keyword with problematic hard-coded values is `stage`. If a component job's +stage is set to a specific value, the pipeline using the component **must** define +the exact same stage. Additionally, if the component user wants to use a different stage, +they must [override](../yaml/includes.md#override-included-configuration-values) the configuration. -stages: [verify, deploy] +The preferred method is to use the [`input` keyword](../yaml/inputs.md). +The component user can specify the exact value they need. -unit-test: - stage: verify +For example: -integration-test: - stage: verify -``` +- In the component configuration: -```yaml -## -# BAD: In order to use the component correctly you need to define the stage -# that is hard-coded in it. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 + ```yaml + spec: + inputs: + stage: + default: test + --- + unit-test: + stage: $[[ inputs.stage ]] + script: echo unit tests + + integration-test: + stage: $[[ inputs.stage ]] + script: echo integration tests + ``` -stages: [test, deploy] -``` +- In the project using the component: -To improve this we can use [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs) -allowing the user of a component to inject values that can be customized: + ```yaml + include: + - component: gitlab.com/gitlab-org/ruby-test@1.0 + inputs: + stage: verify -```yaml -## -# GOOD: We don't need to know the implementation details of a component and instead we can -# rely on the inputs. -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 - inputs: - stage: verify + stages: [verify, deploy] + ``` -stages: [verify, deploy] +### Replace custom CI/CD variables with inputs -## -# inside the component YAML: -spec: - inputs: - stage: - default: test ---- -unit-test: - stage: $[[ inputs.stage ]] - script: echo unit tests +When using CI/CD variables in a component, evaluate if the `inputs` keyword +should be used instead. Avoid requiring a user to define custom variables to change a component's +behavior. You should try to use `inputs` for any component customization. -integration-test: - stage: $[[ inputs.stage ]] - script: echo integration tests -``` +Inputs are explicitly defined in the component's specs, and are better validated than variables. +For example, if a required input is not passed to the component, GitLab returns a pipeline error. +By contrast, if a variable is not defined, its value is empty, and there is no error. -#### Prefer inputs over variables +For example, use `inputs` instead of variables to let users change a scanner's output format: -If variables are only used for YAML evaluation (for example `rules`) and not by the Runner -execution, it's advised to use inputs instead. -Inputs are explicitly defined in the component's contract and they are better validated -than variables. +- In the component configuration: -For example, if a required input is not passed an error is returned as soon as the component -is being used. By contrast, if a variable is not defined, it's value is empty. + ```yaml + spec: + inputs: + scanner-output: + default: json + --- + my-scanner: + script: my-scan --output $[[ inputs.scanner-output ]] + ``` -```yaml -## -# BAD: you need to configure an environment variable for a custom value that doesn't need -# to be used on the Runner -unit-test: - image: $MY_COMPONENT_X_IMAGE - script: echo unit tests - -integration-test: - image: $MY_COMPONENT_X_IMAGE - script: echo integration tests - -## -# Usage: -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 +- In the project using the component: -variables: - MY_COMPONENT_X_IMAGE: ruby:3.2 -``` + ```yaml + include: + - component: gitlab.example.com/my-scanner@1.0 + inputs: + scanner-output: yaml + ``` -```yaml -## -# GOOD: we define a customizable value and accept it as input -spec: - inputs: - image: - default: ruby:3.0 ---- -unit-test: - image: $[[ inputs.image ]] - script: echo unit tests +In other cases, CI/CD variables are still preferred, including: -integration-test: - image: $[[ inputs.image ]] - script: echo integration tests +- Using [predefined variables](../variables/predefined_variables.md) to automatically configure + a component to match a user's project. +- Requiring tokens or other sensitive values to be stored as [masked or protected variables in project settings](../variables/index.md#define-a-cicd-variable-in-the-ui). -## -# Usage: -include: - - component: gitlab.com/gitlab-org/ruby-test@1.0 - inputs: - image: ruby:3.2 -``` +### Use semantic versioning -#### Use semantic versioning +When tagging and releasing new versions of components, you should use [semantic versioning](https://semver.org). +Semantic versioning is the standard for communicating that a change is a major, minor, patch, +or other kind of change. -When tagging and releasing new versions of components we recommend using [semantic versioning](https://semver.org) -which is the standard for communicating bugfixes, minor and major or breaking changes. +You should use at least the `major.minor` format, as this is widely understood. For example, +`2.0` or `2.1`. -We recommend adopting at least the `MAJOR.MINOR` format. +Other examples of semantic versioning: -For example: `2.1`, `1.0.0`, `1.0.0-alpha`, `2.1.3`, `3.0.0-rc.1`. +- `1.0.0` +- `2.1.3` +- `1.0.0-alpha` +- `3.0.0-rc1` -## CI/CD Catalog **(PREMIUM ALL)** +### Test the component -The CI/CD Catalog is a list of [components repositories](#components-repository), -each containing resources that you can add to your CI/CD pipelines. +Testing CI/CD components as part of the development workflow is strongly recommended +and helps ensure consistent behavior. -### Mark the project as a catalog resource +Test changes in a CI/CD pipeline (like any other project) by creating a `.gitlab-ci.yml` +in the root directory. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/407249) in GitLab 16.1. +For example: -After components are added to a components repository, they can immediately be [used](#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. +```yaml +include: + # include the component located in the current project from the current SHA + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + inputs: + stage: build -However, this repository is not discoverable. You must mark this project as a catalog resource to allow it to be visible in the CI Catalog -so other users can discover it. +stages: [build, test, release] -To mark a project as a catalog resource: +# Expect `component-job` is added. +# This example tests that the included component works as expected. +# You can inspect data generated by the component, use GitLab API endpoints or third-party tools. +ensure-job-added: + stage: test + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` + if [ "$count" != "1" ]; then + exit 1 + fi -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. On the left sidebar, select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Scroll down to **CI/CD Catalog resource** and select the toggle to mark the project as a catalog resource. +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` -NOTE: -This action is not reversible. +After committing and pushing changes, the pipeline tests the component, then releases it if the test passes. -## Convert a CI template to component +## Convert a CI/CD template to a component -Any existing CI template, that you share with other projects via `include:` syntax, can be converted to a CI component. +Any existing CI/CD template that you use in projects by using the `include:` syntax +can be converted to a CI/CD component: -1. Decide whether you want the component to be part of an existing [components repository](#components-repository), - if you want to logically group components together. Create and setup a [components repository](#components-repository) otherwise. -1. Create a YAML file in the components repository according to the expected [directory structure](#directory-structure). -1. Copy the content of the template YAML file into the new component YAML file. -1. Refactor the component YAML to follow the [best practices](#best-practices) for components. -1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](#test-a-component). -1. Tag and [release the component](#release-a-component). +1. Decide if you want the component to be part of an existing [components repository](index.md#components-repository) + to be grouped with other components, or create and set up a new components repository. +1. Create a YAML file in the components repository according to the expected [directory structure](index.md#directory-structure). +1. Copy the content of the original template YAML file into the new component YAML file. +1. Refactor the new component's configuration to: + - Follow the [best practices](index.md#best-practices) for components. + - Improve the configuration, for example by enabling [merge request pipelines](../pipelines/merge_request_pipelines.md) + or making it [more efficient](../pipelines/pipeline_efficiency.md). +1. Leverage the `.gitlab-ci.yml` in the components repository to [test changes to the component](index.md#test-the-component). +1. Tag and [release the component](index.md#release-a-component). diff --git a/doc/ci/docker/authenticate_registry.md b/doc/ci/docker/authenticate_registry.md index 52cc3071fda..28edb5140dd 100644 --- a/doc/ci/docker/authenticate_registry.md +++ b/doc/ci/docker/authenticate_registry.md @@ -18,9 +18,9 @@ login`: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind variables: DOCKER_TLS_CERTDIR: "/certs" @@ -42,7 +42,7 @@ empty or remove it. If you are an administrator for GitLab Runner, you can mount a file with the authentication configuration to `~/.docker/config.json`. Then every job that the runner picks up is already authenticated. If you -are using the official `docker:20.10.16` image, the home directory is +are using the official `docker:24.0.5` image, the home directory is under `/root`. If you mount the configuration file, any `docker` command @@ -126,9 +126,9 @@ The same commands apply for any solution you implement. ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind variables: DOCKER_TLS_CERTDIR: "/certs" diff --git a/doc/ci/docker/buildah_rootless_tutorial.md b/doc/ci/docker/buildah_rootless_tutorial.md new file mode 100644 index 00000000000..fdb33fd4a8b --- /dev/null +++ b/doc/ci/docker/buildah_rootless_tutorial.md @@ -0,0 +1,149 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Tutorial: Use Buildah in a rootless container with GitLab Runner Operator on OpenShift **(FREE)** + +This tutorial teaches you how to successfully build images using the `buildah` tool, +with GitLab Runner deployed using [GitLab Runner Operator](https://gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator) +on an OpenShift cluster. + +This guide is an adaptation of [using Buildah to build images in a rootless OpenShift container](https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md) +documentation for GitLab Runner Operator. + +To complete this tutorial, you will: + +1. [Configure the Buildah image](#configure-the-buildah-image) +1. [Configure the service account](#configure-the-service-account) +1. [Configure the job](#configure-the-job) + +## Prerequisites + +- A runner already deployed to a `gitlab-runner` namespace. + +## Configure the Buildah image + +We start by preparing a custom image based on the `quay.io/buildah/stable:v1.23.1` image. + +1. Create the `Containerfile-buildah` file: + + ```shell + cat > Containerfile-buildah <<EOF + FROM quay.io/buildah/stable:v1.23.1 + + RUN touch /etc/subgid /etc/subuid \ + && chmod g=u /etc/subgid /etc/subuid /etc/passwd \ + && echo build:10000:65536 > /etc/subuid \ + && echo build:10000:65536 > /etc/subgid + + # Use chroot since the default runc does not work when running rootless + RUN echo "export BUILDAH_ISOLATION=chroot" >> /home/build/.bashrc + + # Use VFS since fuse does not work + RUN mkdir -p /home/build/.config/containers \ + && (echo '[storage]';echo 'driver = "vfs"') > /home/build/.config/containers/storage.conf + + # The buildah container will run as `build` user + USER build + WORKDIR /home/build + EOF + ``` + +1. Build and push the Buildah image to a Container Registry. Let's push to the + [GitLab Container Registry](../../user/packages/container_registry/index.md): + + ```shell + docker build -f Containerfile-buildah -t registry.example.com/group/project/buildah:1.23.1 . + docker push registry.example.com/group/project/buildah:1.23.1 + ``` + +## Configure the service account + +For these steps, you need to run the commands in a terminal connected to the OpenShift cluster. + +1. Run this command to create a service account named `buildah-sa`: + + ```shell + oc create -f - <<EOF + apiVersion: v1 + kind: ServiceAccount + metadata: + name: buildah-sa + namespace: gitlab-runner + EOF + ``` + +1. Give the created service account the ability to run with `anyuid` [SCC](https://docs.openshift.com/container-platform/4.3/authentication/managing-security-context-constraints.html): + + ```shell + oc adm policy add-scc-to-user anyuid -z buildah-sa -n gitlab-runner + ``` + +1. Use a [runner configuration template](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#customize-configtoml-with-a-configuration-template) + to configure Operator to use the service account we just created. Create a `custom-config.toml` file that contains: + + ```toml + [[runners]] + [runners.kubernetes] + service_account_overwrite_allowed = "buildah-*" + ``` + +1. Create a `ConfigMap` named `custom-config-toml` from the `custom-config.toml` file: + + ```shell + oc create configmap custom-config-toml --from-file config.toml=custom-config.toml -n gitlab-runner + ``` + +1. Set the `config` property of the `Runner` by updating its [Custom Resource Definition (CRD) file](https://docs.gitlab.com/runner/install/operator.html#install-gitlab-runner): + + ```yaml + apiVersion: apps.gitlab.com/v1beta2 + kind: Runner + metadata: + name: builah-runner + spec: + gitlabUrl: https://gitlab.example.com + token: gitlab-runner-secret + config: custom-config-toml + ``` + +## Configure the job + +The final step is to set up a GitLab CI/CD configuration file in you project to use +the image we built and the configured service account: + +```yaml +build: + stage: build + image: registry.example.com/group/project/buildah:1.23.1 + variables: + STORAGE_DRIVER: vfs + BUILDAH_FORMAT: docker + BUILDAH_ISOLATION: chroot + FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test" + KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: "buildah-sa" + before_script: + # Log in to the GitLab container registry + - buildah login -u "$CI_REGISTRY_USER" --password $CI_REGISTRY_PASSWORD $CI_REGISTRY + script: + - buildah images + - buildah build -t $FQ_IMAGE_NAME + - buildah images + - buildah push $FQ_IMAGE_NAME +``` + +The job should use the image that we built as the value of `image` keyword. + +The `KUBERNETES_SERVICE_ACCOUNT_OVERWRITE` variable should have the value of the +service account name that we created. + +Congratulations, you've successfully built an image with Buildah in a rootless container! + +## Troubleshooting + +There is a [known issue](https://github.com/containers/buildah/issues/4049) with running as non-root. +You might need to use a [workaround](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#configure-setfcap) +if you are using an OpenShift runner. diff --git a/doc/ci/docker/docker_layer_caching.md b/doc/ci/docker/docker_layer_caching.md index 861bea70cb7..b34b58ce964 100644 --- a/doc/ci/docker/docker_layer_caching.md +++ b/doc/ci/docker/docker_layer_caching.md @@ -29,9 +29,9 @@ This example `.gitlab-ci.yml` file shows how to use Docker caching: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index fb38de0f7de..269ce2c3212 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -80,7 +80,8 @@ For more information, see [security of the `docker` group](https://blog.zopyx.co "Docker-in-Docker" (`dind`) means: -- Your registered runner uses the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html) or the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html). +- Your registered runner uses the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html) or + the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html). - The executor uses a [container image of Docker](https://hub.docker.com/_/docker/), provided by Docker, to run your CI/CD jobs. @@ -90,7 +91,7 @@ the job script in context of the image in privileged mode. You should use Docker-in-Docker with TLS enabled, which is supported by [GitLab.com shared runners](../runners/index.md). -You should always pin a specific version of the image, like `docker:20.10.16`. +You should always pin a specific version of the image, like `docker:24.0.5`. If you use a tag like `docker:latest`, you have no control over which version is used. This can cause incompatibility problems when new versions are released. @@ -121,12 +122,12 @@ To use Docker-in-Docker with TLS enabled: --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:20.10.16" \ + --docker-image "docker:24.0.5" \ --docker-privileged \ --docker-volumes "/certs/client" ``` - - This command registers a new runner to use the `docker:20.10.16` image. + - This command registers a new runner to use the `docker:24.0.5` image (if none is specified at the job level). To start the build and service containers, it uses the `privileged` mode. If you want to use Docker-in-Docker, you must always use `privileged = true` in your Docker containers. @@ -143,7 +144,7 @@ To use Docker-in-Docker with TLS enabled: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = true disable_cache = false volumes = ["/certs/client", "/cache"] @@ -153,13 +154,13 @@ To use Docker-in-Docker with TLS enabled: ``` 1. You can now use `docker` in the job script. You should include the - `docker:20.10.16-dind` service: + `docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -202,7 +203,7 @@ Assuming that the runner's `config.toml` is similar to: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = true disable_cache = false volumes = ["/cache"] @@ -212,13 +213,13 @@ Assuming that the runner's `config.toml` is similar to: ``` You can now use `docker` in the job script. You should include the -`docker:20.10.16-dind` service: +`docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -276,13 +277,13 @@ To use Docker-in-Docker with TLS enabled in Kubernetes: ``` 1. You can now use `docker` in the job script. You should include the - `docker:20.10.16-dind` service: + `docker:24.0.5-dind` service: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - docker:20.10.16-dind + - docker:24.0.5-dind before_script: - docker info @@ -330,7 +331,7 @@ Docker-in-Docker is the recommended configuration, but you should be aware of th - **Storage drivers**: By default, earlier versions of Docker use the `vfs` storage driver, which copies the file system for each job. Docker 17.09 and later use `--storage-driver overlay2`, which is the recommended storage driver. See [Using the OverlayFS driver](#use-the-overlayfs-driver) for details. -- **Root file system**: Because the `docker:20.10.16-dind` container and the runner container do not share their +- **Root file system**: Because the `docker:24.0.5-dind` container and the runner container do not share their root file system, you can use the job's working directory as a mount point for child containers. For example, if you have files you want to share with a child container, you could create a subdirectory under `/builds/$CI_PROJECT_PATH` @@ -352,7 +353,7 @@ container. Docker is then available in the context of the image. If you bind the Docker socket and you are [using GitLab Runner 11.11 or later](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1261), -you can no longer use `docker:20.10.16-dind` as a service. Volume bindings also affect services, +you can no longer use `docker:24.0.5-dind` as a service. Volume bindings also affect services, making them incompatible. To make Docker available in the context of the image, you need to mount @@ -369,7 +370,7 @@ Your configuration should look similar to this example: executor = "docker" [runners.docker] tls_verify = false - image = "docker:20.10.16" + image = "docker:24.0.5" privileged = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] @@ -385,7 +386,7 @@ sudo gitlab-runner register -n \ --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "docker:20.10.16" \ + --docker-image "docker:24.0.5" \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock ``` @@ -408,7 +409,7 @@ mirror: ```yaml services: - - name: docker:20.10.16-dind + - name: docker:24.0.5-dind command: ["--registry-mirror", "https://registry-mirror.example.com"] # Specify the registry mirror to use ``` @@ -431,7 +432,7 @@ Docker: ... privileged = true [[runners.docker.services]] - name = "docker:20.10.16-dind" + name = "docker:24.0.5-dind" command = ["--registry-mirror", "https://registry-mirror.example.com"] ``` @@ -445,7 +446,7 @@ Kubernetes: ... privileged = true [[runners.kubernetes.services]] - name = "docker:20.10.16-dind" + name = "docker:24.0.5-dind" command = ["--registry-mirror", "https://registry-mirror.example.com"] ``` @@ -552,12 +553,12 @@ the implications of this method are: docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` -You do not need to include the `docker:20.10.16-dind` service, like you do when +You do not need to include the `docker:24.0.5-dind` service, like you do when you use the Docker-in-Docker executor: ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 before_script: - docker info @@ -638,30 +639,43 @@ To build Docker images without enabling privileged mode on the runner, you can use one of these alternatives: - [`kaniko`](using_kaniko.md). -- [`buildah`](https://github.com/containers/buildah). There is a [known issue](https://github.com/containers/buildah/issues/4049) with running as non-root, you might need this [workaround](https://docs.gitlab.com/runner/configuration/configuring_runner_operator.html#configure-setfcap) if you are using OpenShift Runner. +- [`buildah`](#buildah-example). -For example, with `buildah`: +### Buildah example -```yaml -# Some details from https://major.io/2019/05/24/build-containers-in-gitlab-ci-with-buildah/ +To use Buildah with GitLab CI/CD, you need [a runner](https://docs.gitlab.com/runner/) with one +of the following executors: + +- [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html). +- [Docker](https://docs.gitlab.com/runner/executors/docker.html). +- [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html). + +In this example, you use Buildah to: + +1. Build a Docker image. +1. Push it to [GitLab Container Registry](../../user/packages/container_registry/index.md). +In the last step, Buildah uses the `Dockerfile` under the +root directory of the project to build the Docker image. Finally, it pushes the image to the +project's Container Registry: + +```yaml build: stage: build image: quay.io/buildah/stable variables: - # Use vfs with buildah. Docker offers overlayfs as a default, but buildah + # Use vfs with buildah. Docker offers overlayfs as a default, but Buildah # cannot stack overlayfs on top of another overlayfs filesystem. STORAGE_DRIVER: vfs # Write all image metadata in the docker format, not the standard OCI format. # Newer versions of docker can handle the OCI format, but older versions, like # the one shipped with Fedora 30, cannot handle the format. BUILDAH_FORMAT: docker - # You may need this workaround for some errors: https://stackoverflow.com/a/70438141/1233435 - BUILDAH_ISOLATION: chroot FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test" before_script: - # Log in to the GitLab container registry - - export REGISTRY_AUTH_FILE=$HOME/auth.json + # GitLab Container Registry credentials taken from the + # [predefined CI/CD variables](../variables/index.md#predefined-cicd-variables) + # to authenticate to the registry. - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY script: - buildah images @@ -670,6 +684,9 @@ build: - buildah push $FQ_IMAGE_NAME ``` +If you are using GitLab Runner Operator deployed to an OpenShift cluster, try the +[tutorial for using Buildah to build images in rootless container](buildah_rootless_tutorial.md). + ## Use the GitLab Container Registry After you've built a Docker image, you can push it to the @@ -702,9 +719,9 @@ This issue can occur when the service's image name ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - registry.hub.docker.com/library/docker:20.10.16-dind + - registry.hub.docker.com/library/docker:24.0.5-dind ``` A service's hostname is [derived from the full image name](../../ci/services/index.md#accessing-the-services). @@ -713,15 +730,94 @@ To allow service resolution and access, add an explicit alias for the service na ```yaml default: - image: docker:20.10.16 + image: docker:24.0.5 services: - - name: registry.hub.docker.com/library/docker:20.10.16-dind + - name: registry.hub.docker.com/library/docker:24.0.5-dind alias: docker ``` +### `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` + +You might get the following error when trying to run a `docker` command +to access a `dind` service: + +```shell +$ docker ps +Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? +``` + +Make sure your job has defined these environment variables: + +- `DOCKER_HOST` +- `DOCKER_TLS_CERTDIR` (optional) +- `DOCKER_TLS_VERIFY` (optional) + +You may also want to update the image that provides the Docker +client. For example, the [`docker/compose` images are obsolete](https://hub.docker.com/r/docker/compose) and should be +replaced with [`docker`](https://hub.docker.com/_/docker). + +As described in [runner issue 30944](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/30944#note_1514250909), +this error can happen if your job previously relied on environment variables derived from the deprecated +[Docker `--link` parameter](https://docs.docker.com/network/links/#environment-variables), +such as `DOCKER_PORT_2375_TCP`. Your job fails with this error if: + +- Your CI/CD image relies on a legacy variable, such as `DOCKER_PORT_2375_TCP`. +- The [runner feature flag `FF_NETWORK_PER_BUILD`](https://docs.gitlab.com/runner/configuration/feature-flags.html) is set to `true`. +- `DOCKER_HOST` is not explicitly set. + ### Error: `Error response from daemon: Get "https://registry-1.docker.io/v2/": unauthorized: incorrect username or password` This error appears when you use the deprecated variable, `CI_BUILD_TOKEN`. To prevent users from receiving this error, you should: - Use [CI_JOB_TOKEN](../jobs/ci_job_token.md) instead. - Change from `gitlab-ci-token/CI_BUILD_TOKEN` to `$CI_REGISTRY_USER/$CI_REGISTRY_PASSWORD`. + +### Error: `error during connect: Post "https://docker:2376/v1.24/auth": dial tcp: lookup docker on 127.0.0.11:53: no such host` + +This error appears when the `dind` service has failed to start. Check +the job log to see if `mount: permission denied (are you root?)` +appears. For example: + +```plaintext +Service container logs: +2023-08-01T16:04:09.541703572Z Certificate request self-signature ok +2023-08-01T16:04:09.541770852Z subject=CN = docker:dind server +2023-08-01T16:04:09.556183222Z /certs/server/cert.pem: OK +2023-08-01T16:04:10.641128729Z Certificate request self-signature ok +2023-08-01T16:04:10.641173149Z subject=CN = docker:dind client +2023-08-01T16:04:10.656089908Z /certs/client/cert.pem: OK +2023-08-01T16:04:10.659571093Z ip: can't find device 'ip_tables' +2023-08-01T16:04:10.660872131Z modprobe: can't change directory to '/lib/modules': No such file or directory +2023-08-01T16:04:10.664620455Z mount: permission denied (are you root?) +2023-08-01T16:04:10.664692175Z Could not mount /sys/kernel/security. +2023-08-01T16:04:10.664703615Z AppArmor detection and --privileged mode might break. +2023-08-01T16:04:10.665952353Z mount: permission denied (are you root?) +``` + +This indicates the GitLab Runner does not have permission to start the +`dind` service: + +1. Check that `privileged = true` is set in the `config.toml`. +1. Make sure the CI job has the right Runner tags to use these +privileged runners. + +### Error: `cgroups: cgroup mountpoint does not exist: unknown` + +There is a known incompatibility introduced by Docker Engine 20.10. + +When the host uses Docker Engine 20.10 or newer, then the `docker:dind` service in a version older than 20.10 does +not work as expected. + +While the service itself will start without problems, trying to build the container image results in the error: + +```plaintext +cgroups: cgroup mountpoint does not exist: unknown +``` + +To resolve this issue, update the `docker:dind` container to version at least 20.10.x, +for example `docker:24.0.5-dind`. + +The opposite configuration (`docker:24.0.5-dind` service and Docker Engine on the host in version +19.06.x or older) works without problems. For the best strategy, you should to frequently test and update +job environment versions to the newest. This brings new features, improved security and - for this specific +case - makes the upgrade on the underlying Docker Engine on the runner's host transparent for the job. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 568f4977c2f..8ab13c7154d 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -58,7 +58,7 @@ project's Container Registry while tagging it with the Git tag: build: stage: build image: - name: gcr.io/kaniko-project/executor:v1.9.0-debug + name: gcr.io/kaniko-project/executor:v1.14.0-debug entrypoint: [""] script: - /kaniko/executor @@ -96,7 +96,7 @@ build: https_proxy: <your-proxy> no_proxy: <your-no-proxy> image: - name: gcr.io/kaniko-project/executor:v1.9.0-debug + name: gcr.io/kaniko-project/executor:v1.14.0-debug entrypoint: [""] script: - /kaniko/executor @@ -158,3 +158,25 @@ on what other GitLab CI patterns are demonstrated are available at the project p If you receive this error, it might be due to an outside proxy. Setting the `http_proxy` and `https_proxy` [environment variables](../../administration/packages/container_registry.md#running-the-docker-daemon-with-a-proxy) can fix the problem. + +### Error: `kaniko should only be run inside of a container, run with the --force flag if you are sure you want to continue` + +There is a known incompatibility introduced by Docker Engine 20.10. + +When the host uses Docker Engine 20.10 or newer, then the `gcr.io/kaniko-project/executor:debug` image in a version +older than v1.9.0 does not work as expected. + +When you try to build the image, Kaniko fails with: + +```plaintext +kaniko should only be run inside of a container, run with the --force flag if you are sure you want to continue +``` + +To resolve this issue, update the `gcr.io/kaniko-project/executor:debug` container to version at least v1.9.0, +for example `gcr.io/kaniko-project/executor:v1.14.0-debug`. + +The opposite configuration (`gcr.io/kaniko-project/executor:v1.14.0-debug` image and Docker Engine +on the host in version 19.06.x or older) works without problems. For the best strategy, you should +frequently test and update job environment versions to the newest. This brings new features, improved +security and - for this specific case - makes the upgrade on underlying Docker Engine on the runner's +host transparent for the job. diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 395a07f48c8..3081b8d1b39 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -30,7 +30,7 @@ When you disable GitLab CI/CD: To disable GitLab CI/CD in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. In the **Repository** section, turn off **CI/CD**. @@ -40,7 +40,7 @@ To disable GitLab CI/CD in your project: To enable GitLab CI/CD in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. In the **Repository** section, turn on **CI/CD**. diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index 2933b25e09b..754dcafb9f7 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -8,32 +8,24 @@ description: Require approvals prior to deploying to a Protected Environment # Deployment approvals **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343864) in GitLab 14.7 with a flag named `deployment_approvals`. Disabled by default. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/347342) in GitLab 14.8. Feature flag `deployment_approvals` removed. -It may be useful to require additional approvals before deploying to certain protected environments (for example, production). This pre-deployment approval requirement is useful to accommodate testing, security, or compliance processes that must happen before each deployment. +You can require additional approvals for deployments to protected +environments. Deployments are blocked until all required approvals are +given. -When a protected environment requires one or more approvals, all deployments to that environment become blocked and wait for the required approvals from the `Allowed to Deploy` list before running. +Use deployment approvals to accommodate testing, +security, or compliance processes. For example, you might want to +require approvals for deployments to production environments. -NOTE: -See the [epic](https://gitlab.com/groups/gitlab-org/-/epics/6832) for planned features. - -## Prerequisites - -- Basic knowledge of [GitLab Environments and Deployments](index.md). -- Basic knowledge of [Protected Environments](protected_environments.md). +## Configure deployment approvals -## Configure deployment approvals for a project +You can require approvals for deployments to protected environments in +a project. To configure deployment approvals for a project: -1. [Create a deployment job](#create-a-deployment-job). -1. [Require approvals for a protected environment](#require-approvals-for-a-protected-environment). - -### Create a deployment job - -Create a deployment job in the `.gitlab-ci.yml` file of the desired project. The job does **not** need to be manual (`when: manual`). - -Example: +1. Create a deployment job in the `.gitlab-ci.yml` file of your project: ```yaml stages: @@ -47,22 +39,15 @@ Example: name: ${CI_JOB_NAME} ``` -### Require approvals for a protected environment + The job does not need to be manual (`when: manual`). -There are two ways to configure the approval requirements: +1. Add the required [approval rules](#multiple-approval-rules). -- [Unified approval setting](#unified-approval-setting-deprecated) ... You can define who can execute **and** approve deployments. - This is useful when there is no separation of duties between executors and approvers in your organization. -- [Multiple approval rules](#multiple-approval-rules) ... You can define who can execute **or** approve deployments. - This is useful when there is a separation of duties between executors and approvers in your organization. - -NOTE: -Multiple approval rules is a more flexible option than the unified approval setting, thus both configurations shouldn't -co-exist and multiple approval rules takes the precedence over the unified approval setting if it happens. +The environments in your project require approval before deployment. <!--- start_remove The following content will be removed on remove_date: '2024-05-22' --> -#### Unified approval setting (deprecated) +### Unified approval setting (deprecated) > - UI configuration [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/378447) in GitLab > 15.11. @@ -94,7 +79,7 @@ Maintainer role. <!--- end_remove --> -#### Multiple approval rules +### Multiple approval rules > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 14.10 with a flag named `deployment_approval_rules`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 15.0. [Feature flag `deployment_approval_rules`](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) removed. @@ -107,7 +92,7 @@ Maintainer role. - **Allowed to deploy** sets which entities can execute the deployment job. - **Approvers** sets which entities can approve the deployment job. -After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. +After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. Once a deployment job is approved, it must be [run manually](../jobs/job_control.md#run-a-manual-job). A configuration that uses the REST API might look like: @@ -128,7 +113,7 @@ NOTE: To protect, update, or unprotect an environment, you must have at least the Maintainer role. -#### Migrate to multiple approval rules +### Migrate to multiple approval rules You can migrate a protected environment from unified approval rules to multiple approval rules. Unified approval rules allow all entities that can deploy to an @@ -137,7 +122,7 @@ create a new approval rule for each entity allowed to deploy to the environment. To migrate with the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Environment** list, select your environment. @@ -171,7 +156,7 @@ require `Administrator` to approve every deployment job in `Production`. By default, the user who triggers a deployment pipeline can't also approve the deployment job. To allow self-approval of a deployment job: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Approval options**, select the **Allow pipeline triggerer to approve deployment** checkbox. @@ -196,7 +181,7 @@ Prerequisites: To approve or reject a deployment to a protected environment using the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment's name. 1. In the deployment's row, select **Approval options** (**{thumb-up}**). @@ -233,7 +218,7 @@ granted. To view the approval details of a deployment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment's name. 1. In the deployment's row, select **Approval options** (**{thumb-up}**). @@ -249,7 +234,7 @@ The approval status details are shown: ### Using the UI -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment being deployed to. 1. Look for the `blocked` label. @@ -265,9 +250,9 @@ Use the [Deployments API](../../api/deployments.md#get-a-specific-deployment) to - When the [multiple approval rules](#multiple-approval-rules) is configured: - The `approval_summary` field contains the current approval status per rule. -## Related features +## Related topics -For details about other GitLab features aimed at protecting deployments, see [safe deployments](deployment_safety.md). +- [Deployment approvals feature epic](https://gitlab.com/groups/gitlab-org/-/epics/6832) <!-- ## Troubleshooting diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index e98205f45b9..0beaf2b8888 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -20,7 +20,7 @@ see which pipelines are green and which are red allowing you to diagnose if there is a block at a particular point, or if there's a more systemic problem you need to investigate. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Environments**. diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index a498e525b54..309e311e252 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -22,9 +22,9 @@ Manual and Timed rollouts are included automatically in projects controlled by [Auto DevOps](../../topics/autodevops/index.md), but they are also configurable through GitLab CI/CD in the `.gitlab-ci.yml` configuration file. -Manually triggered rollouts can be implemented with your [Continuous Delivery](../introduction/index.md#continuous-delivery) -methodology, while timed rollouts do not require intervention and can be part of your -[Continuous Deployment](../introduction/index.md#continuous-deployment) strategy. +Manually triggered rollouts can be implemented with Continuous Delivery, +while timed rollouts do not require intervention and can be part of your +Continuous Deployment strategy. You can also combine both of them in a way that the app is deployed automatically unless you eventually intervene manually if necessary. diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index c91018980f0..713d58de326 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -51,7 +51,7 @@ Deployments show up in this list only after a deployment job has created them. To search environments by name: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. In the search bar, enter your search term. - The length of your **search term should be 3 or more characters**. @@ -93,7 +93,7 @@ Prerequisites: To create a static environment in the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select **Create an environment**. 1. Complete the fields. @@ -339,13 +339,19 @@ It points to the commit you're rolling back to. For the rollback to succeed, the deployment process must be defined in the job's `script`. +Only the [deployment jobs](../jobs/index.md#deployment-jobs) are run. +In cases where a previous job generates artifacts that must be regenerated +on deploy, you must manually run the necessary jobs from the pipelines page. +For example, if you use Terraform and your `plan` and `apply` commands are separated +into multiple jobs, you must manually run the jobs to deploy or roll back. + #### Retry or roll back a deployment If there is a problem with a deployment, you can retry it or roll it back. To retry or roll back a deployment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the environment. 1. To the right of the deployment name: @@ -359,7 +365,7 @@ In this case, see [job retries for rollback deployments](deployment_safety.md#jo ### Environment URL -> - [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) to persist arbitrary URLs in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `soft_validation_on_external_url`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337417) in GitLab 15.3. [Feature flag `soft_validation_on_external_url`](https://gitlab.com/gitlab-org/gitlab/-/issues/367206) removed. The [environment URL](../yaml/index.md#environmenturl) is displayed in a few @@ -560,7 +566,7 @@ you can view its expiration date and time. To view an environment's expiration date and time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the name of the environment. @@ -573,7 +579,7 @@ you can override its expiration. To override an environment's expiration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the deployment name. 1. in the upper-right corner, select the thumbtack (**{thumbtack}**). @@ -600,7 +606,7 @@ Environments view, the stop and deploy jobs must be in the same To stop an environment in the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Next to the environment you want to stop, select **Stop**. 1. On the confirmation dialog, select **Stop environment**. @@ -664,7 +670,7 @@ Prerequisites: To delete an environment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select the **Stopped** tab. 1. Next to the environment you want to delete, select **Delete environment**. @@ -767,7 +773,7 @@ Limitations of GitLab Auto Rollback: GitLab Auto Rollback is turned off by default. To turn it on: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Automatic deployment rollbacks**. 1. Select the checkbox for **Enable automatic rollbacks**. @@ -910,6 +916,7 @@ the `review/feature-1` spec takes precedence over `review/*` and `*` specs. ## Related topics - [Dashboard for Kubernetes](kubernetes_dashboard.md) +- [Downstream pipelines for deployments](../pipelines/downstream_pipelines.md#downstream-pipelines-for-deployments) - [Deploy to multiple environments with GitLab CI/CD (blog post)](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) - [Review Apps](../review_apps/index.md) - [Protected environments](protected_environments.md) diff --git a/doc/ci/environments/kubernetes_dashboard.md b/doc/ci/environments/kubernetes_dashboard.md index 98b6731e469..0f9e1d808ec 100644 --- a/doc/ci/environments/kubernetes_dashboard.md +++ b/doc/ci/environments/kubernetes_dashboard.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Dashboard for Kubernetes (Beta) **(FREE ALL)** +# Dashboard for Kubernetes **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390769) in GitLab 16.1, with [flags](../../administration/feature_flags.md) named `environment_settings_to_graphql`, `kas_user_access`, `kas_user_access_project`, and `expose_authorized_cluster_agents`. This feature is in [Beta](../../policy/experiment-beta-support.md#beta). > - Feature flag `environment_settings_to_graphql` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124177) in GitLab 16.2. @@ -15,17 +15,12 @@ Use the Dashboard for Kubernetes to understand the status of your clusters with The dashboard works with every connected Kubernetes cluster, whether you deployed them with CI/CD or GitOps. -For Flux users, the synchronization status of a given environment is not displayed in the dashboard. -[Issue 391581](https://gitlab.com/gitlab-org/gitlab/-/issues/391581) proposes to add this functionality. - ## Configure a dashboard > - Filtering resources by namespace [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/403618) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `kubernetes_namespace_for_environment`. Disabled by default. > - Filtering resources by namespace [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127043) in GitLab 16.3. Feature flag `kubernetes_namespace_for_environment` removed. -> - Selecting the related Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. Disabled by default. - -FLAG: -On self-managed GitLab, by default selecting a Flux resource is not available. To make it available, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. On GitLab.com, this feature is not available. +> - Selecting the related Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. +> - Selecting the related Flux resource [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130648) in GitLab 16.4. Feature flag `flux_resource_for_environment` removed. Configure a dashboard to use it for a given environment. You can configure dashboard for an environment that already exists, or @@ -38,9 +33,9 @@ Prerequisites: ### The environment already exists -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. -1. Select the environment to be associated with the Kubernetes. +1. Select the environment to be associated with the agent for Kubernetes. 1. Select **Edit**. 1. Select a GitLab agent for Kubernetes. 1. Optional. From the **Kubernetes namespace** dropdown list, select a namespace. @@ -49,7 +44,7 @@ Prerequisites: ### The environment doesn't exist -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. 1. Select **New environment**. 1. Complete the **Name** field. @@ -62,17 +57,25 @@ Prerequisites: To view a configured dashboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Environments**. -1. Expand the environment associated with GitLab agent for Kubernetes. +1. Expand the environment associated with the agent for Kubernetes. 1. Expand **Kubernetes overview**. ### Flux sync status > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391581) in GitLab 16.3. -> - Customizing the name of the Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. Disabled by default. +> - Customizing the name of the Flux resource [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128857) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `flux_resource_for_environment`. +> - Customizing the name of the Flux resource [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130648) in GitLab 16.4. Feature flag `flux_resource_for_environment` removed. -A dashboard displays the sync status of your Flux deployments. +You can review the sync status of your Flux deployments from a dashboard. +To display the deployment status, your dashboard must be able to retrieve the `Kustomization` and `HelmRelease` resources, +which requires a namespace to be configured for the environment. + +By default, GitLab searches the `Kustomization` and `HelmRelease` resources for the name of the project slug. +You can specify the resource names with the **Flux resource** dropdown list in the environment settings. + +A dashboard displays one of the following status badges: | Status | Description | |---------|-------------| @@ -83,11 +86,6 @@ A dashboard displays the sync status of your Flux deployments. | **Unknown** | The sync status of the deployment couldn't be retrieved. | | **Unavailable** | The `Kustomization` or `HelmRelease` resource couldn't be retrieved. | -Deployments rely on Flux `Kustomization` and `HelmRelease` resources to gather -the status of a given environment, which requires a namespace to be configured for the environment. -By default, GitLab searches the `Kustomization` and `HelmRelease` resources for the name of the project slug. -You can customize the name GitLab looks for in the environment settings. - ## Troubleshooting When working with the Dashboard for Kubernetes, you might encounter the following issues. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index fd789ea798d..21b58ac5ab4 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -30,7 +30,7 @@ Prerequisites: To protect an environment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. Select **Protect an environment**. @@ -256,7 +256,7 @@ To protect a group-level environment, make sure your environments have the corre > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) in GitLab 15.1. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Protected environments**. 1. From the **Environment** list, select the [deployment tier of environments](index.md#deployment-tier-of-environments) you want to protect. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 737c95cf747..647669385d8 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -9,7 +9,7 @@ type: tutorial WARNING: Authenticating with `CI_JOB_JWT` was [deprecated in GitLab 15.9](../../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) -and the token is scheduled to be removed in GitLab 16.5. Use +and the token is scheduled to be removed in GitLab 17.0. Use [ID tokens to authenticate with HashiCorp Vault](../../secrets/id_token_authentication.md#automatic-id-token-authentication-with-hashicorp-vault) instead, as demonstrated on this page. diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md index f9360faedac..2be800efcbb 100644 --- a/doc/ci/examples/deployment/index.md +++ b/doc/ci/examples/deployment/index.md @@ -121,7 +121,7 @@ We also use two secure variables: To store API keys as secure variables: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index b284e2b2dc1..356a3d1d63e 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -96,7 +96,7 @@ As part of publishing a package, semantic-release increases the version number i 1. Under **select scopes**, select the **api** checkbox. 1. Select **Create project access token**. 1. Copy the value. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. 1. Select **Add variable**. diff --git a/doc/ci/index.md b/doc/ci/index.md index 21fe122f503..8d9108e4fc5 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -6,174 +6,86 @@ description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Inte type: index --- -# GitLab CI/CD **(FREE ALL)** - -GitLab CI/CD is a tool for software development using the continuous methodologies: - -- [Continuous Integration (CI)](introduction/index.md#continuous-integration) -- [Continuous Delivery (CD)](introduction/index.md#continuous-delivery) -- [Continuous Deployment (CD)](introduction/index.md#continuous-deployment) - -NOTE: -Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. -Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) -webcast to learn about continuous methods and how GitLab CI/CD can help you simplify and scale software development. - -Use GitLab CI/CD to catch bugs and errors early in -the development cycle. Ensure that all the code deployed to -production complies with the code standards you established for -your app. - -GitLab CI/CD can automatically build, test, deploy, and -monitor your applications by using [Auto DevOps](../topics/autodevops/index.md). - -For a complete overview of these methodologies and GitLab CI/CD, -read the [Introduction to CI/CD with GitLab](introduction/index.md). - -<div class="video-fallback"> - Video demonstration of continuous integration with GitLab CI/CD: <a href="https://www.youtube.com/watch?v=ljth1Q5oJoo">Continuous Integration with GitLab (overview demo)</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen> </iframe> -</figure> - -## Concepts - -GitLab CI/CD uses a number of concepts to describe and run your build and deploy. - -| Concept | Description | -|:--------------------------------------------------------|:--------------------------------------------------------------------------------------| -| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | -| [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments/index.md) | Deploy your application to different environments (for example, staging, production). | -| [Job artifacts](jobs/job_artifacts.md) | Output, use, and reuse job artifacts. | -| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | -| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | -| [Test cases](test_cases/index.md) | Create testing scenarios. | - -## Configuration - -GitLab CI/CD supports numerous configuration options: - -| Configuration | Description | -|:---------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#specify-a-custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | -| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | -| [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | -| [Merge request pipelines](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | -| [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | - -Certain operations can only be performed according to the -[user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. - -## Features - -GitLab CI/CD features, grouped by DevOps stage, include: - -| Feature | Description | -|:---------------------------------------------------------------------------------------------|:------------| -| **Configure** | | -| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | -| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. | -| **Verify** | | -| [CI services](services/index.md) | Link Docker containers with your base image. | -| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | -| [Unit test reports](testing/unit_test_reports.md) | Identify test failures directly on merge requests. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -| **Release** | | -| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | -| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature flags](../operations/feature_flags.md) | Deploy your features behind Feature flags. | -| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | -| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | -| **Secure** | | -| [Code Quality](testing/code_quality.md) | Analyze your source code quality. | -| [Container Scanning](../user/application_security/container_scanning/index.md) | Scan your container images for known vulnerabilities. | -| [Coverage-guided fuzz testing](../user/application_security/coverage_fuzzing/index.md) | Test your application's behavior by providing randomized input. | -| [Dynamic Application Security Testing](../user/application_security/dast/index.md) | Test your application's runtime behavior for vulnerabilities. | -| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) | Analyze your dependencies for known vulnerabilities. | -| [Infrastructure as Code scanning](../user/application_security/iac_scanning/index.md) | Scan your IaC configuration files for known vulnerabilities. | -| [License Scanning](../user/compliance/license_scanning_of_cyclonedx_files/index.md) | Search your project dependencies for their licenses. | Search your project dependencies for their licenses. | -| [Secret Detection](../user/application_security/secret_detection/index.md) | Search your application's source code for secrets. | -| [Static Application Security Testing](../user/application_security/sast/index.md) | Test your application's source code for known vulnerabilities. | -| [Web API fuzz testing](../user/application_security/api_fuzzing/index.md) | Test your application's API behavior by providing randomized input. | -| **Govern** | | -| [Compliance frameworks](../user/group/compliance_frameworks.md) | Enforce a GitLab CI/CD configuration on all projects in a group. | -| [Scan execution policies](../user/application_security/policies/scan-execution-policies.md) | Enforce security scans run on a specified schedule or with the project pipeline. | -| [Scan results policies](../user/application_security/policies/scan-result-policies.md) | Enforce action based on results of a pipeline security scan. | - -## Examples - -See the [CI/CD examples](examples/index.md) page for example project code and tutorials for -using GitLab CI/CD with various: - -- App frameworks -- Languages -- Platforms - -## Administration - -You can change the default behavior of GitLab CI/CD for: - -- An entire GitLab instance in the [CI/CD administration settings](../administration/cicd.md). -- Specific projects in the [pipelines settings](pipelines/settings.md). - -See also: - -- [Enable or disable GitLab CI/CD in a project](enable_or_disable_ci.md). +# Get started with GitLab CI/CD **(FREE ALL)** + +CI/CD is a continuous method of software development, where you continuously build, +test, deploy, and monitor iterative code changes. + +This iterative process helps reduce the chance that you develop new code based on +buggy or failed previous versions. GitLab CI/CD can catch bugs early in the development cycle, +and help ensure that all the code deployed to production complies with your established code standards. + +## Common terms + +If you're new to GitLab CI/CD, start by reviewing some of the commonly used terms. + +### The `.gitlab-ci.yml` file + +To use GitLab CI/CD, you start with a `.gitlab-ci.yml` file at the root of your project. +In this file, you specify the list of things you want to do, like test and deploy your application. +This file follows the YAML format and has its own special syntax. + +You can name this file anything you want, but `.gitlab-ci.yml` is the most common name. +Use the pipeline editor to edit the `.gitlab-ci.yml` file and test the syntax before you commit changes. + +**Get started:** + +- [Create your first `.gitlab-ci.yml` file](quick_start/index.md). +- [View all the possible keywords that you can use in the `.gitlab-ci.yml` file](yaml/index.md). + +### Runners + +Runners are the agents that run your jobs. These agents can run on physical machines or virtual instances. +In your `.gitlab-ci.yml` file, you can specify a container image you want to use when running the job. +The runner loads the image and runs the job either locally or in the container. + +If you use GitLab.com, free shared runners are already available for you. And you can register your own +runners on GitLab.com if you'd like. + +If you don't use GitLab.com, you can: + +- Register runners or use runners already registered for your self-managed instance. +- Create a runner on your local machine. + +**Get started:** + +- [Create a runner on your local machine](../tutorials/create_register_first_runner/index.md). +- [Learn more about runners](https://docs.gitlab.com/runner/). + +### Pipelines + +Pipelines are made up of jobs and stages: + +- **Jobs** define what you want to do. For example, test code changes, or deploy + to a staging environment. +- Jobs are grouped into **stages**. Each stage contains at least one job. + Typical stages might be `build`, `test`, and `deploy`. + +**Get started:** + +- [Learn more about pipelines](pipelines/index.md). + +### CI/CD variables + +CI/CD variables help you customize jobs by making values defined elsewhere accessible to jobs. +They can be hard-coded in your `.gitlab-ci.yml` file, project settings, or dynamically generated +[predefined variables](variables/predefined_variables.md). + +**Get started:** + +- [Learn more about CI/CD variables](variables/index.md). + +## Videos + +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab CI/CD demo](https://www.youtube-nocookie.com/embed/ljth1Q5oJoo). +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab CI/CD and the Web IDE](https://youtu.be/l5705U8s_nQ?t=369). +- Webcast: [Mastering continuous software development](https://about.gitlab.com/webcast/mastering-ci-cd/). ## Related topics -- [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/) -- [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/) -- [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) -- If you use VS Code to edit your GitLab CI/CD configuration, the - [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) helps you +- [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/). +- [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). +- Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) from 30 days to under 8 hours with GitLab. +- Use the [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) to [validate your configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration) - and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue) - -See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. - -### Major version changes (breaking) - -As GitLab CI/CD has evolved, certain breaking changes have -been necessary. - -For GitLab 15.0 and later, all breaking changes are documented on the following pages: - -- [Deprecations](../update/deprecations.md) -- [Removals](../update/removals.md) - -The breaking changes for [GitLab Runner](https://docs.gitlab.com/runner/) in earlier -major version releases are: - -- 14.0: No breaking changes. -- 13.0: - - [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). - - [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). - - [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). - - [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). - - [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). - - [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). - - [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). - - [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). - - [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). -- 12.0: - - [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). - - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). - - [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). - - [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). - - [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). - - [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). - - [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). + and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue). diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differdeleted file mode 100644 index 1f11db55f81..00000000000 --- a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png +++ /dev/null diff --git a/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png b/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png Binary files differdeleted file mode 100644 index 6e1066d4868..00000000000 --- a/doc/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png +++ /dev/null diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 35692ed7b7e..536dd224ac5 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,116 +1,11 @@ --- -stage: Verify -group: Pipeline Authoring -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." -type: concepts +redirect_to: 'index.md' +remove_date: '2023-11-24' --- -# CI/CD concepts **(FREE ALL)** +This document was moved to [another location](../index.md). -With the continuous method of software development, you continuously build, -test, and deploy iterative code changes. This iterative process helps reduce -the chance that you develop new code based on buggy or failed previous versions. -With this method, you strive to have less human intervention or even no intervention at all, -from the development of new code until its deployment. - -The three primary approaches for the continuous method are: - -- [Continuous Integration](#continuous-integration) -- [Continuous Delivery](#continuous-delivery) -- [Continuous Deployment](#continuous-deployment) - -Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more. -Watch our ["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) -webcast to learn about continuous methods and how built-in GitLab CI/CD can help you simplify and scale software development. - -- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i>Learn how to: [configure CI/CD](https://www.youtube.com/watch?v=opdLqwz6tcE). -- [Make the case for CI/CD in your organization](https://about.gitlab.com/devops-tools/github-vs-gitlab/). -- Learn how [Verizon reduced rebuilds](https://about.gitlab.com/blog/2019/02/14/verizon-customer-story/) from 30 days to under 8 hours with GitLab. - -## Continuous Integration - -Consider an application that has its code stored in a Git -repository in GitLab. Developers push code changes every day, -multiple times a day. For every push to the repository, you -can create a set of scripts to build and test your application -automatically. These scripts help decrease the chances that you introduce errors in your application. - -This practice is known as [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration). -Each change submitted to an application, even to development branches, -is built and tested automatically and continuously. These tests ensure the -changes pass all tests, guidelines, and code compliance -standards you established for your application. - -[GitLab itself](https://gitlab.com/gitlab-org/gitlab) is an -example of a project that uses Continuous Integration as a software -development method. For every push to the project, a set -of checks run against the code. - -## Continuous Delivery - -[Continuous Delivery](https://continuousdelivery.com/) is a step -beyond Continuous Integration. Not only is your application -built and tested each time a code change is pushed to the codebase, -the application is also deployed continuously. However, with continuous -delivery, you trigger the deployments manually. - -Continuous Delivery checks the code automatically, but it requires -human intervention to manually and strategically trigger the deployment -of the changes. - -## Continuous Deployment - -Continuous Deployment is another step beyond Continuous Integration, similar to -Continuous Delivery. The difference is that instead of deploying your -application manually, you set it to be deployed automatically. -Human intervention is not required. - -## GitLab CI/CD - -[GitLab CI/CD](../quick_start/index.md) is the part of GitLab that you use -for all of the continuous methods (Continuous Integration, -Delivery, and Deployment). With GitLab CI/CD, you can test, build, -and publish your software with no third-party application or integration needed. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Introduction to GitLab CI/CD](https://www.youtube.com/watch?v=l5705U8s_nQ&t=397) from an April 2020 GitLab meetup. - -### GitLab CI/CD workflow - -GitLab CI/CD fits in a common development workflow. - -You can start by discussing a code implementation in an issue -and working locally on your proposed changes. Then you can push your -commits to a feature branch in a remote repository that's hosted in GitLab. -The push triggers the CI/CD pipeline for your project. Then, GitLab CI/CD: - -- Runs automated scripts (sequentially or in parallel) to: - - Build and test your application. - - Preview the changes in a Review App, the same as you - would see on your `localhost`. - -After the implementation works as expected: - -- Get your code reviewed and approved. -- Merge the feature branch into the default branch. - - GitLab CI/CD deploys your changes automatically to a production environment. - -If something goes wrong, you can roll back your changes. - - - -This workflow shows the major steps in the GitLab process. -You don't need any external tools to deliver your software and -you can visualize all the steps in the GitLab UI. - -### A deeper look into the CI/CD workflow - -If you look deeper into the workflow, you can see -the features available in GitLab at each stage of the DevOps -lifecycle. - - - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -[Get a deeper look at GitLab CI/CD](https://youtu.be/l5705U8s_nQ?t=369). +<!-- This redirect file can be deleted after <2023-11-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index dee078c21e0..f1aa834a038 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -121,7 +121,7 @@ Prerequisite: To disable the job token scope allowlist: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit access _to_ this project** to disabled. @@ -144,7 +144,7 @@ Prerequisite: To add a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Verify **Limit access _to_ this project** is enabled. @@ -188,7 +188,7 @@ Prerequisite: To configure the job token scope: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Toggle **Limit access _from_ this project** to enabled. @@ -255,7 +255,7 @@ While troubleshooting CI/CD job token authentication issues, be aware that: - A [GraphQL example mutation](../../api/graphql/getting_started.md#update-project-settings) is available to toggle the scope settings per project. - [This comment](https://gitlab.com/gitlab-org/gitlab/-/issues/351740#note_1335673157) - demonstrates how to use graphQL with Bash and cURL to: + demonstrates how to use GraphQL with Bash and cURL to: - Enable the inbound token access scope. - Give access to project B from project A, or add B to A's allowlist. - To remove project access. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 762fb717505..90a64ea7569 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -48,7 +48,7 @@ Selecting an individual job shows you its job log, and allows you to: To view the full list of jobs that ran in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Jobs**. You can filter the list by [job status](#the-order-of-jobs-in-a-pipeline). diff --git a/doc/ci/jobs/job_artifacts.md b/doc/ci/jobs/job_artifacts.md index b080a4fd1e9..b6269918ed9 100644 --- a/doc/ci/jobs/job_artifacts.md +++ b/doc/ci/jobs/job_artifacts.md @@ -299,7 +299,7 @@ You can also delete individual artifacts from the [**Artifacts** page](#bulk-del You can delete multiple artifacts at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Artifacts**. 1. Select the checkboxes next to the artifacts you want to delete. You can select up to 50 artifacts. 1. Select **Delete selected**. @@ -341,7 +341,7 @@ Keeping the latest artifacts can use a large amount of storage space in projects with a lot of jobs or large artifacts. If the latest artifacts are not needed in a project, you can disable this behavior to save space: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Artifacts**. 1. Clear the **Keep artifacts from most recent successful jobs** checkbox. diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 770e1d38297..1065ee93389 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -867,7 +867,7 @@ linux:rspec: parallel: matrix: - PROVIDER: aws - - STACK: app1 + STACK: app1 script: echo "Running rspec on linux..." mac:rspec: @@ -877,7 +877,7 @@ mac:rspec: parallel: matrix: - PROVIDER: [gcp, vultr] - - STACK: [data] + STACK: [data] script: echo "Running rspec on mac..." production: diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index da5ebe21341..cef0554bf6c 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -1,254 +1,11 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference +redirect_to: '../../user/project/repository/managing_large_repositories.md' +remove_date: '2023-11-30' --- -# Optimize GitLab for large repositories **(FREE ALL)** +This document was moved to [another location](../../user/project/repository/managing_large_repositories.md). -Large repositories consisting of more than 50k files in a worktree -may require more optimizations beyond -[pipeline efficiency](../pipelines/pipeline_efficiency.md) -because of the time required to clone and check out. - -GitLab and GitLab Runner handle this scenario well -but require optimized configuration to efficiently perform its -set of operations. - -The general guidelines for handling big repositories are simple. -Each guideline is described in more detail in the sections below: - -- Always fetch incrementally. Do not clone in a way that results in recreating all of the worktree. -- Always use shallow clone to reduce data transfer. Be aware that this puts more burden - on GitLab instance due to higher CPU impact. -- Control the clone directory if you heavily use a fork-based workflow. -- Optimize `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. - -## Shallow cloning - -> Introduced in GitLab Runner 8.9. - -GitLab and GitLab Runner perform a [shallow clone](../pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) -by default. - -Ideally, you should always use `GIT_DEPTH` with a small number -like 10. This instructs GitLab Runner to perform shallow clones. -Shallow clones make Git request only the latest set of changes for a given branch, -up to desired number of commits as defined by the `GIT_DEPTH` variable. - -This significantly speeds up fetching of changes from Git repositories, -especially if the repository has a very long backlog consisting of number -of big files as we effectively reduce amount of data transfer. - -The following example makes the runner shallow clone to fetch only a given branch; -it does not fetch any other branches nor tags. - -```yaml -variables: - GIT_DEPTH: 10 - -test: - script: - - ls -al -``` - -## Git strategy - -> Introduced in GitLab Runner 8.9. - -By default, GitLab is configured to use the [`fetch` Git strategy](../runners/configure_runners.md#git-strategy), -which is recommended for large repositories. -This strategy reduces the amount of data to transfer and -does not really impact the operations that you might do on a repository from CI. - -## Git clone path - -> Introduced in GitLab Runner 11.10. - -[`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) allows you to -control where you clone your sources. This can have implications if you -heavily use big repositories with fork workflow. - -Fork workflow from GitLab Runner's perspective is stored as a separate repository -with separate worktree. That means that GitLab Runner cannot optimize the usage -of worktrees and you might have to instruct GitLab Runner to use that. - -In such cases, ideally you want to make the GitLab Runner executor be used only -for the given project and not shared across different projects to make this -process more efficient. - -The [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) has to be -within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path -from disk. - -## Git clean flags - -> Introduced in GitLab Runner 11.10. - -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) allows you to control -whether or not you require the `git clean` command to be executed for each CI -job. By default, GitLab ensures that you have your worktree on the given SHA, -and that your repository is clean. - -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) is disabled when set -to `none`. On very big repositories, this might be desired because `git -clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx --e .build/` (for example) allows you to control and disable removal of some -directories within the worktree between subsequent runs, which can speed-up -the incremental builds. This has the biggest effect if you re-use existing -machines and have an existing worktree that you can re-use for builds. - -For exact parameters accepted by -[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags), see the documentation -for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters -are dependent on Git version. - -## Git fetch extra flags - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. - -[`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) allows you -to modify `git fetch` behavior by passing extra flags. - -For example, if your project contains a large number of tags that your CI jobs don't rely on, -you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) -to the extra flags to make your fetches faster and more compact. - -Also in the case where you repository does _not_ contain a lot of -tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). -If your CI builds do not depend on Git tags it is worth trying. - -See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags) -for more information. - -## Fork-based workflow - -> Introduced in GitLab Runner 11.10. - -Following the guidelines above, let's imagine that we want to: - -- Optimize for a big project (more than 50k files in directory). -- Use forks-based workflow for contributing. -- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. -- Runner assigned only to project and all forks. - -Let's consider the following two examples, one using `shell` executor and -other using `docker` executor. - -### `shell` executor example - -Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "shell" - builds_dir = "/builds" - cache_dir = "/cache" - - [runners.custom_build_dir] - enabled = true -``` - -This `config.toml`: - -- Uses the `shell` executor, -- Specifies a custom `/builds` directory where all clones are stored. -- Enables the ability to specify `GIT_CLONE_PATH`, -- Runs at most 4 jobs at once. - -### `docker` executor example - -Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "docker" - builds_dir = "/builds" - cache_dir = "/cache" - - [runners.docker] - volumes = ["/builds:/builds", "/cache:/cache"] -``` - -This `config.toml`: - -- Uses the `docker` executor, -- Specifies a custom `/builds` directory on disk where all clones are stored. - We host mount the `/builds` directory to make it reusable between subsequent runs - and be allowed to override the cloning strategy. -- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. -- Runs at most 4 jobs at once. - -### Our `.gitlab-ci.yml` - -Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. - -Our pipeline is most performant if we use the following `.gitlab-ci.yml`: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME - -build: - script: ls -al -``` - -This YAML setting configures a custom clone path. This path makes it possible to re-use worktrees -between the parent project and forks because we use the same clone path for all forks. - -Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting -between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor. -When we use it to construct the path, this directory does not conflict -with other concurrent jobs running. - -### Store custom clone options in `config.toml` - -Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. -However, sometimes it is desirable to make these schemes part of the runner's configuration. - -In the above example of Forks, making this configuration discoverable for users may be preferred, -but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. -In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it -a configuration of the runner. - -We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) -with the following specification that is used by the runner if `.gitlab-ci.yml` does not override it: - -```toml -concurrent = 4 - -[[runners]] - url = "GITLAB_URL" - token = "TOKEN" - executor = "docker" - builds_dir = "/builds" - cache_dir = "/cache" - - environment = [ - "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" - ] - - [runners.docker] - volumes = ["/builds:/builds", "/cache:/cache"] -``` - -This makes the cloning configuration to be part of the given runner -and does not require us to update each `.gitlab-ci.yml`. - -## Git fetch caching step - -For very active repositories with a large number of references and files, consider using the -[Gitaly pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache). -The pack-objects cache: - -- Benefits all repositories on your GitLab server. -- Automatically works for forks. +<!-- This redirect file can be deleted after <2023-11-30>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 0ca0469f5e9..2b54c645807 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -24,7 +24,7 @@ configuration added with the [`includes` keyword](yaml/index.md#include). To check CI/CD configuration with the CI lint tool: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. @@ -45,7 +45,7 @@ Prerequisites: To simulate a pipeline: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. diff --git a/doc/ci/migration/examples/img/maven-freestyle-plugin.png b/doc/ci/migration/examples/img/maven-freestyle-plugin.png Binary files differnew file mode 100644 index 00000000000..bb3308a1595 --- /dev/null +++ b/doc/ci/migration/examples/img/maven-freestyle-plugin.png diff --git a/doc/ci/migration/examples/img/maven-freestyle-shell.png b/doc/ci/migration/examples/img/maven-freestyle-shell.png Binary files differnew file mode 100644 index 00000000000..73ecbb9401d --- /dev/null +++ b/doc/ci/migration/examples/img/maven-freestyle-shell.png diff --git a/doc/ci/migration/examples/jenkins-maven.md b/doc/ci/migration/examples/jenkins-maven.md new file mode 100644 index 00000000000..0ed08357eef --- /dev/null +++ b/doc/ci/migration/examples/jenkins-maven.md @@ -0,0 +1,232 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, howto +--- + +# Migrate a Maven build from Jenkins to GitLab CI/CD + +If you have a Maven build in Jenkins, you can use a [Java Spring](https://gitlab.com/gitlab-org/project-templates/spring) +project template to migrate to GitLab. The template uses Maven for its underlying dependency management. + +## Sample Jenkins configurations + +The following three Jenkins examples each use different methods to test, build, and install a +Maven project into a shell agent: + +- Freestyle with shell execution +- Freestyle with the Maven task plugin +- A declarative pipeline using a Jenkinsfile + +All three examples run the same three commands in order, in three different stages: + +- `mvn test`: Run any tests found in the codebase +- `mvn package -DskipTests`: Compile the code into an executable type defined in the POM + and skip running any tests because that was done in the first stage. +- `mvn install -DskipTests`: Install the compiled executable into the agent's local Maven + `.m2` repository and again skip running the tests. + +These examples use a single, persistent Jenkins agent, which requires Maven to be +pre-installed on the agent. This method of execution is similar to a GitLab Runner +using the [shell executor](https://docs.gitlab.com/runner/executors/shell.html). + +### Freestyle with shell execution + +If using Jenkins' built-in shell execution option to directly call `mvn` commands +from the shell on the agent, the configuration might look like: + + + +### Freestyle with Maven task plugin + +If using the Maven plugin in Jenkins to declare and execute any specific goals +in the [Maven build lifecycle](https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html), +the configuration might look like: + + + +This plugin requires Maven to be installed on the Jenkins agent, and uses a script wrapper +for calling Maven commands. + +### Using a declarative pipeline + +If using a declarative pipeline, the configuration might look like: + +```groovy +pipeline { + agent any + tools { + maven 'maven-3.6.3' + jdk 'jdk11' + } + stages { + stage('Build') { + steps { + sh "mvn package -DskipTests" + } + } + stage('Test') { + steps { + sh "mvn test" + } + } + stage('Install') { + steps { + sh "mvn install -DskipTests" + } + } + } +} +``` + +This example uses shell execution commands instead of plugins. + +By default, a declarative pipeline configuration is stored either in the Jenkins +pipeline configuration or directly in the Git repository in a `Jenksinfile`. + +## Convert Jenkins configuration to GitLab CI/CD + +While the examples above are all slightly different, they can all be migrated to GitLab CI/CD +with the same pipeline configuration. + +Prerequisites: + +- A GitLab Runner with a Shell executor +- Maven 3.6.3 and Java 11 JDK installed on the shell runner + +This example mimics the behavior and syntax of building, testing, and installing on Jenkins. + +In a GitLab CI/CD pipeline, the commands run in "jobs", which are grouped into stages. +The migrated configuration in the `.gitlab-ci.yml` configuration file consists of +two global keywords (`stages` and `variables`) followed by 3 jobs: + +```yaml +stages: + - build + - test + - install + +variables: + MAVEN_OPTS: >- + -Dhttps.protocols=TLSv1.2 + -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository + MAVEN_CLI_OPTS: >- + -DskipTests + +build-JAR: + stage: build + script: + - mvn $MAVEN_CLI_OPTS package + +test-code: + stage: test + script: + - mvn test + +install-JAR: + stage: install + script: + - mvn $MAVEN_CLI_OPTS install +``` + +In this example: + +- `stages` defines three stages that run in order. Like the Jenkins examples above, + the test job runs first, followed by the build job, and finally the install job. +- `variables` defines [CI/CD variables](../../variables/index.md) that can be used by all jobs: + - `MAVEN_OPTS` are Maven environment variables needed whenever Maven is executed: + - `-Dhttps.protocols=TLSv1.2` sets the TLS protocol to version 1.2 for any HTTP requests in the pipeline. + - `-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository` sets the location of the + local Maven repository to the GitLab project directory on the runner, so the job + can access and modify the repository. + - `MAVEN_CLI_OPTS` are specific arguments to be added to `mvn` commands: + - `-DskipTests` skips the `test` stage in the Maven build lifecycle. +- `test-code`, `build-JAR`, and `install-JAR` are the user-defined names for the jobs + to run in the pipeline: + - `stage` defines which stage the job runs in. A pipeline contains one or more stages + and a stage contains one or more jobs. This example has three stages, each with a single job. + - `script` defines the commands to run in that job, similar to `steps` in a `Jenkinsfile`. + Jobs can run multiple commands in sequence, which run in the image container, + but in this example the jobs run only one command each. + +### Run jobs in Docker containers + +Instead of using a persistent machine for handling this build process like the Jenkins samples, +this example uses an ephemeral Docker container to handle execution. Using a container +removes the need for maintaining a virtual machine and the Maven version installed on it. +It also increases flexibility for expanding and extending the functionality of the pipeline. + +Prerequisites: + +- A GitLab Runner with the Docker executor that can be used by the project. + If you are using GitLab.com, you can use the public shared runners. + +This migrated pipeline configuration consists of three global keywords (`stages`, `default`, and `variables`) +followed by 3 jobs. This configuration makes use of additional GitLab CI/CD features +for an improved pipeline compared to the [example above](#convert-jenkins-configuration-to-gitlab-cicd): + +```yaml +stages: + - build + - test + - install + +default: + image: maven:3.6.3-openjdk-11 + cache: + key: $CI_COMMIT_REF_SLUG + paths: + - .m2/ + +variables: + MAVEN_OPTS: >- + -Dhttps.protocols=TLSv1.2 + -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository + MAVEN_CLI_OPTS: >- + -DskipTests + +build-JAR: + stage: build + script: + - mvn $MAVEN_CLI_OPTS package + +test-code: + stage: test + script: + - mvn test + +install-JAR: + stage: install + script: + - mvn $MAVEN_CLI_OPTS install +``` + +In this example: + +- `stages` defines three stages that run in order. Like the Jenkins examples above, + the test job runs first, followed by the build job, and finally the install job. +- `default` defines standard configuration to reuse in all jobs by default: + - `image` defines the Docker image container to use and execute commands in. In this example, + it's an official Maven Docker image with everything needed already installed. + - `cache` is used to cache and reuse dependencies: + - `key` is the unique identifier for the specific cache archive. In this example, + it's a shortened version of the Git commit ref, autogenerated as a [predefined CI/CD variable](../../variables/predefined_variables.md). + Any job that runs for the same commit ref reuses the same cache. + - `paths` are the directories or files to include in the cache. In this example, + we cache the `.m2/` directory to avoid re-installing dependencies between job runs. +- `variables` defines [CI/CD variables](../../variables/index.md) that can be used by all jobs: + - `MAVEN_OPTS` are Maven environment variables needed whenever Maven is executed: + - `-Dhttps.protocols=TLSv1.2` sets the TLS protocol to version 1.2 for any HTTP requests in the pipeline. + - `-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository` sets the location of the + local Maven repository to the GitLab project directory on the runner, so the job + can access and modify the repository. + - `MAVEN_CLI_OPTS` are specific arguments to be added to `mvn` commands: + - `-DskipTests` skips the `test` stage in the Maven build lifecycle. +- `test-code`, `build-JAR`, and `install-JAR` are the user-defined names for the jobs + to run in the pipeline: + - `stage` defines which stage the job runs in. A pipeline contains one or more stages + and a stage contains one or more jobs. This example has three stages, each with a single job. + - `script` defines the commands to run in that job, similar to `steps` in a `Jenkinsfile`. + Jobs can run multiple commands in sequence, which run in the image container, + but in this example the jobs run only one command each. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 54ea3812029..d02c2f9c54e 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -7,215 +7,175 @@ type: index, howto # Migrating from Jenkins **(FREE ALL)** -A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. -We've collected several resources here that you might find informative if you're just getting started. -Think of this page as a "GitLab CI/CD for Jenkins Users" guide. +If you're migrating from Jenkins to GitLab CI/CD, you should be able +to create CI/CD pipelines that do everything you need. + +You can start by watching the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) +video for examples of: + +- Converting a Jenkins pipeline into a GitLab CI/CD pipeline. +- Using Auto DevOps to test your code automatically. + +## Get started The following list of recommended steps was created after observing organizations -that were able to quickly complete this migration: - -1. Start by reading the GitLab CI/CD [Quick Start Guide](../quick_start/index.md) and [important product differences](#important-product-differences). -1. Learn the importance of [managing the organizational transition](#manage-organizational-transition). -1. [Add runners](../runners/index.md) to your GitLab instance. -1. Educate and enable your developers to independently perform the following steps in their projects: - 1. Review the [Quick Start Guide](../quick_start/index.md) and [Pipeline Configuration Reference](../yaml/index.md). - 1. Use the [Jenkins Wrapper](#jenkinsfile-wrapper) to temporarily maintain fragile Jenkins jobs. - 1. Migrate the build and CI jobs and configure them to show results directly in your merge requests. They can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) the configuration as needed. - 1. Add [Review Apps](../review_apps/index.md). - 1. Migrate the deployment jobs using [cloud deployment templates](../cloud_deployment/index.md), adding [environments](../environments/index.md), and [deploy boards](../../user/project/deploy_boards.md). - 1. Work to unwrap any jobs still running with the use of the Jenkins wrapper. -1. Take stock of any common CI/CD job definitions then create and share [templates](#templates) for them. -1. Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) - to learn how to make your GitLab CI/CD pipelines faster and more efficient. - -Watch the [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y) video for examples of how to: - -- Convert a Jenkins pipeline into a GitLab CI/CD pipeline. -- Use Auto DevOps to test your code automatically. - -Otherwise, read on for important information that helps you get the ball rolling. Welcome -to GitLab! +that were able to quickly complete this migration. + +Before doing any migration work, you should [start with a migration plan](plan_a_migration.md). + +Engineers that need to migrate projects to GitLab CI/CD should: + +- Read about some [key GitLab CI/CD features](#key-gitlab-cicd-features). +- Follow tutorials to create: + - [Your first GitLab pipeline](../quick_start/index.md). + - [A more complex pipeline](../quick_start/tutorial.md) that builds, tests, + and deploys a static site. +- Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). +- Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners + or installing new runners. +- Migrate build and CI jobs and configure them to show results directly in merge requests. + You can use [Auto DevOps](../../topics/autodevops/index.md) as a starting point, + and [customize](../../topics/autodevops/customize.md) or [decompose](../../topics/autodevops/customize.md#use-individual-components-of-auto-devops) + the configuration as needed. +- Migrate deployment jobs by using [cloud deployment templates](../cloud_deployment/index.md), + [environments](../environments/index.md), and the [GitLab agent for Kubernetes](../../user/clusters/agent/index.md). +- Check if any CI/CD configuration can be reused across different projects, then create + and share [templates](#templates). +- Check the [pipeline efficiency documentation](../pipelines/pipeline_efficiency.md) + to learn how to make your GitLab CI/CD pipelines faster and more efficient. If you have questions that are not answered here, the [GitLab community forum](https://forum.gitlab.com/) can be a great resource. -## Manage organizational transition - -An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that come with the move, and successfully managing them. A few -things we have found that help this are: - -- Setting and communicating a clear vision of what your migration goals are helps - your users understand why the effort is worth it. The value is clear when - the work is done, but people need to be aware while it's in progress too. -- Sponsorship and alignment from the relevant leadership team helps with the point above. -- Spending time educating your users on what's different and sharing this document - with them helps ensure you are successful. -- Finding ways to sequence or delay parts of the migration can help a lot, but you - don't want to leave things in a non-migrated (or partially-migrated) state for too - long. To gain all the benefits of GitLab, moving your existing Jenkins setup over - as-is, including any current problems, isn't enough. You need to take advantage - of the improvements that GitLab offers, and this requires (eventually) updating - your implementation as part of the transition. - -## JenkinsFile Wrapper - -We are building a [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) which -you can use to run a complete Jenkins instance inside of a GitLab job, including plugins. This can help ease the process -of transition, by letting you delay the migration of less urgent pipelines for a period of time. - -If you are interested in helping GitLab test the wrapper, join our [public testing issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215675) for instructions and to provide your feedback. - -NOTE: -If you have a paid GitLab subscription, the JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). - -## Important product differences - -Some high level differences between the products worth mentioning are: - -- With GitLab you don't need a root `pipeline` keyword to wrap everything. -- The way pipelines are triggered and [trigger other pipelines](../yaml/index.md#trigger) - is different than Jenkins. GitLab pipelines can be triggered: - - - on push - - on [schedule](../pipelines/schedules.md) - - from the [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) - - by [API call](../triggers/index.md) - - by [webhook](../triggers/index.md#use-a-webhook) - - by [ChatOps](../chatops/index.md) - -- You can control which jobs run in which cases, depending on how they are triggered, - with the [`rules` syntax](../yaml/index.md#rules). -- GitLab [pipeline scheduling concepts](../pipelines/schedules.md) are also different from Jenkins. -- You can reuse pipeline configurations using the [`include` keyword](../yaml/index.md#include) - and [templates](#templates). Your templates can be kept in a central repository (with different - permissions), and then any project can use them. This central project could also - contain scripts or other reusable code. -- You can also use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration - in a single pipeline configuration. -- All jobs in a single stage always run in parallel, and all stages run in sequence. - Certain jobs might break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47063) - feature. +### Key GitLab CI/CD features + +GitLab CI/CD key features might be different or not exist in Jenkins. For example, +in GitLab: + +- Pipelines can be triggered with: + - A Git push + - A [Schedule](../pipelines/schedules.md) + - The [GitLab UI](../pipelines/index.md#run-a-pipeline-manually) + - An [API call](../triggers/index.md) + - A [webhook](../triggers/index.md#use-a-webhook) +- You can control which jobs run in which cases with the [`rules` syntax](../yaml/index.md#rules). +- You can reuse pipeline configurations: + - Use the [`extends` keyword](../yaml/index.md#extends) to reuse configuration + in a single pipeline configuration. + - Use the [`include` keyword](../yaml/index.md#include) to reuse configuration across + multiple pipelines and projects. +- Jobs are grouped into stages, and jobs in the same stage can run at the same time. + Stages run in sequence. Jobs can be configured to run outside of the stage ordering with the + [`needs` keyword](../yaml/index.md#needs). - The [`parallel`](../yaml/index.md#parallel) keyword can automatically parallelize tasks, - like tests that support parallelization. -- Usually all jobs in a single stage run in parallel, and all stages run in sequence. - Different [pipeline architectures](../pipelines/pipeline_architectures.md) allow you to change this behavior. -- The new [`rules` syntax](../yaml/index.md#rules) is the recommended method of - controlling when different jobs run. It is more powerful than the `only/except` syntax. -- One important difference is that jobs run independently of each other and have a - fresh environment in each job. Passing artifacts between jobs is controlled using the - [`artifacts`](../yaml/index.md#artifacts) and [`dependencies`](../yaml/index.md#dependencies) - keywords. When finished, use the planned [Workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/29265) - feature to persist a common workspace between serial jobs. -- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but - is in the YAML format (see [complete reference](../yaml/index.md)) instead of a Groovy DSL. It's most - analogous to the declarative Jenkinsfile format. -- Manual approvals or gates can be set up as [`when:manual` jobs](../jobs/job_control.md#create-a-job-that-must-be-run-manually). These can - also leverage [`protected environments`](../jobs/job_control.md#run-a-job-after-a-delay) - to control who is able to approve them. -- GitLab comes with a [container registry](../../user/packages/container_registry/index.md), so you can use - container images to set up your build environment. For example, set up one pipeline that builds your build environment - itself and publish that to the container registry. Then, have your pipelines use this instead of each building their - own environment, which is slower and may be less consistent. We have extensive documentation on [how to use the Container Registry](../../user/packages/container_registry/index.md). -- A central utilities repository can be a great place to put assorted scheduled jobs - or other manual jobs that function like utilities. Jenkins installations tend to - have a few of these. - -## Agents vs. runners - -Both Jenkins agents and GitLab runners are the hosts that run jobs. To convert the -Jenkins agent, uninstall it and then [install and register the runner](../runners/index.md). -Runners do not require much overhead, so you can size them similarly to the Jenkins -agents you were using. - -Some important differences in the way runners work in comparison to agents are: - -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). - They self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) for finer control, and - associate runners with specific jobs. For example, you can use a tag for jobs that + especially tests that support parallelization. +- Jobs run independently of each other and have a fresh environment for each job. + Passing artifacts between jobs is controlled using the [`artifacts`](../yaml/index.md#artifacts) + and [`dependencies`](../yaml/index.md#dependencies) keywords. +- The `.gitlab-ci.yml` configuration file exists in your Git repository, like a `Jenkinsfile`, + but is [a YAML file](#yaml-configuration-file), not Groovy. +- GitLab comes with a [container registry](../../user/packages/container_registry/index.md). + You can build and store custom container images to run your jobs in. + +## Runners + +Like Jenkins agents, GitLab runners are the hosts that run jobs. If you are using GitLab.com, +you can use the [shared runner fleet](../runners/index.md) to run jobs without provisioning +your own runners. + +To convert a Jenkins agent for use with GitLab CI/CD, uninstall the agent and then +[install and register a runner](../runners/index.md). Runners do not require much overhead, +so you might be able to use similar provisioning as the Jenkins agents you were using. + +Some key details about runners: + +- Runners can be [configured](../runners/runners_scope.md) to be shared across an instance, + a group, or dedicated to a single project. +- You can use the [`tags` keyword](../runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) + for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). Use autoscaling to provision runners only when needed and scale down when not needed, similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/index.md) -to run jobs without provisioning your own runners. We are investigating making them -[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) -as well. +## YAML configuration file -## Groovy vs. YAML +GitLab pipeline configuration files use the [YAML](https://yaml.org/) format instead of +the [Groovy](https://groovy-lang.org/) format that Jenkins uses. -Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. -GitLab works a bit differently, using the more highly structured [YAML](https://yaml.org/) format. -The scripting elements are in `script` blocks separate from the pipeline specification itself. - -Using YAML is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running. -It also avoids some of the problem of unconstrained complexity which can make your Jenkinsfile hard to understand -and manage. - -We do of course still value DRY (don't repeat yourself) principles. We want to ensure that -behaviors of your jobs can be codified once and applied as needed. You can use the `extends` syntax to -[reuse configuration in your jobs](../yaml/index.md#extends), and `include` can -be used to [reuse pipeline configurations](../yaml/index.md#include) in pipelines -in different projects: +Using YAML is a strength of GitLab CI/CD, as it is a simple format to understand +and start using. For example, a small configuration file with two jobs and some +shared configuration in a hidden job: ```yaml -.in-docker: +.test-config: tags: - - docker - image: alpine + - docker-runners + stage: test -rspec: +test-job: extends: - - .in-docker + - .docker-config script: - - rake rspec + - bundle exec rake rspec + +lint-job: + extends: + - .docker-config + script: + - yarn run prettier ``` -## Artifact publishing +In this example: + +- The commands to run in jobs are added with the [`script` keyword](../yaml/index.md#script). +- The [`extends` keyword](../yaml/index.md#extends) reduces duplication in the configuration + by adding the same `tags` and `stage` configuration defined in `.test-config` to both jobs. + +### Artifacts + +In GitLab, any job can use the [`artifacts` keyword](../yaml/index.md#artifacts) +to define a set of [artifacts](../jobs/job_artifacts.md) to be stored when a job completes. +Artifacts are files that can be used in later jobs, for example for testing or deployment. -Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define -a set of artifacts to be saved by using the `artifacts` keyword. This can be configured to point to a file -or set of files that can then be persisted from job to job. Read more on our detailed -[artifacts documentation](../jobs/job_artifacts.md): +For example: ```yaml pdf: script: xelatex mycv.tex artifacts: paths: - - ./mycv.pdf - - ./output/ + - mycv.pdf + - output/ expire_in: 1 week ``` -Additionally, we have package management features like built-in container and package registries that you -can leverage. You can see the complete list of packaging features in the -[Packages and registries](../../user/packages/index.md) documentation. +In this example: -## Integrated features +- The `mycv.pdf` file and all the files in `output/` are stored and could be used + in later jobs. +- To save resources, the artifacts expire and are deleted after one week. -You may have used plugins to get things like code quality, unit tests, and security scanning working in Jenkins. -GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into -your merge requests, pipeline details pages, and other locations. You may find that you actually don't -need to configure anything to have these appear. +### Scanning features -Our [CI/CD feature index](../index.md#features) has the full list of bundled features and links to the documentation for each. -Refer to this index if these features aren't working as expected, or if you'd like to see what's available. +You might have used plugins for things like code quality, security, or static application scanning +in Jenkins. Tools like these are already available in GitLab and can be used in your +pipeline. + +GitLab features including [code quality](../testing/code_quality.md), [security scanning](../../user/application_security/index.md), +[SAST](../../user/application_security/sast/index.md), and many others generate reports +when they complete. These reports can be displayed in merge requests and pipeline details pages. ### Templates -For advanced CI/CD teams, project templates can enable the reuse of pipeline configurations, -as well as encourage inner sourcing. +For organizations with many CI/CD pipelines, you can use project templates to configure +custom CI/CD configuration templates and reuse them across projects. + +Group maintainers can configure a group to use as the source for [custom project templates](../../administration/custom_project_templates.md). +These templates can be used by all projects in the group. -In self-managed GitLab instances, you can build an [Instance Template Repository](../../administration/settings/instance_template_repository.md). -Development teams across the whole organization can select templates from a dropdown list. -A group maintainer or a group owner is able to set a group to use as the source for the -[custom project templates](../../administration/custom_project_templates.md). This can -be used by all projects in the group. An instance administrator can set a group as -the source for [instance project templates](../../user/group/custom_project_templates.md), -which can be used by projects in that instance. +An instance administrator can set a group as the source for [instance project templates](../../user/group/custom_project_templates.md), +which can be used by all projects in that instance. ## Convert a declarative Jenkinsfile @@ -361,7 +321,48 @@ my_job: - if: $CI_COMMIT_BRANCH ``` +## Secrets Management + +Privileged information, often referred to as "secrets", is sensitive information +or credentials you need in your CI/CD workflow. You might use secrets to unlock protected resources +or sensitive information in tools, applications, containers, and cloud-native environments. + +Secrets management in Jenkins is usually handled with the `Secret` type field or the +Credentials Plugin. Credentials stored in the Jenkins settings can be exposed to +jobs as environment variables by using the Credentials Binding plugin. + +For secrets management in GitLab, you can use one of the supported integrations +for an external service. These services securely store secrets outside of your GitLab project, +though you must have a subscription for the service: + +- [HashiCorp Vault](../secrets/id_token_authentication.md#automatic-id-token-authentication-with-hashicorp-vault) +- [Azure Key Vault](../secrets/azure_key_vault.md). + +GitLab also supports [OIDC authentication](../secrets/id_token_authentication.md) +for other third party services that support OIDC. + +Additionally, you can make credentials available to jobs by storing them in CI/CD variables, though secrets +stored in plain text are susceptible to accidental exposure, [the same as in Jenkins](https://www.jenkins.io/doc/developer/security/secrets/#storing-secrets). +You should always store sensitive information in [masked](../variables/index.md#mask-a-cicd-variable) +and [protected](../variables/index.md#protect-a-cicd-variable) variables, which mitigates +some of the risk. + +Also, never store secrets as variables in your `.gitlab-ci.yml` file, which is public to all +users with access to the project. Storing sensitive information in variables should +only be done in [the project, group, or instance settings](../variables/index.md#define-a-cicd-variable-in-the-ui). + +Review the [security guidelines](../variables/index.md#cicd-variable-security) to improve +the safety of your CI/CD variables. + ## Additional resources -For help making your pipelines faster and more efficient, see the -[pipeline efficiency documentation](../pipelines/pipeline_efficiency.md). +- You can use the [JenkinsFile Wrapper](https://gitlab.com/gitlab-org/jfr-container-builder/) + to run a complete Jenkins instance inside of a GitLab CI/CD job, including plugins. Use this tool to + help ease the transition to GitLab CI/CD, by delaying the migration of less urgent pipelines. + + NOTE: + The JenkinsFile Wrapper is not packaged with GitLab and falls outside of the scope of support. + For more information, see the [Statement of Support](https://about.gitlab.com/support/statement-of-support/). +- If your tooling outputs packages that you want to make accessible, you can store them + in a [package registry](../../user/packages/index.md). +- Use [review Apps](../review_apps/index.md) to preview changes before merging them. diff --git a/doc/ci/migration/plan_a_migration.md b/doc/ci/migration/plan_a_migration.md new file mode 100644 index 00000000000..488b2abf3a2 --- /dev/null +++ b/doc/ci/migration/plan_a_migration.md @@ -0,0 +1,71 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, howto +--- + +# Plan a migration from another tool to GitLab CI/CD + +Before starting a migration from another tool to GitLab CI/CD, you should begin by +developing a migration plan. + +Review the advice on [managing organizational changes](#manage-organizational-changes) +first for advice on initial steps for larger migrations. + +Users involved in the migration itself should review the [questions to ask before starting a migration](#technical-questions-to-ask-before-starting-a-migration), +as an important technical step for setting expectations. CI/CD tools differ in approach, +structure, and technical specifics. While some concepts map one-to-one, others require +interactive conversion. + +It's important to focus on your desired end state instead of strictly translating +the behavior of your old tool. + +## Manage organizational changes + +An important part of transitioning to GitLab CI/CD is the cultural and organizational +changes that come with the move, and successfully managing them. + +A few things that organizations have reported as helping: + +- Set and communicate a clear vision of what your migration goals are, which helps + your users understand why the effort is worth it. The value is clear when + the work is done, but people need to be aware while it's in progress too. +- Sponsorship and alignment from the relevant leadership teams helps with the point above. +- Spend time educating your users on what's different, and share this guide + with them. +- Finding ways to sequence or delay parts of the migration can help a lot. Importantly though, + try not to leave things in a non-migrated (or partially-migrated) state for too + long. +- To gain all the benefits of GitLab, moving your existing configuration over as-is, + including any current problems, isn't enough. Take advantage of the improvements + that GitLab CI/CD offers, and update your implementation as part of the transition. + +## Technical questions to ask before starting a migration + +Asking some initial technical questions about your CI/CD needs helps quickly define +the migration requirements: + +- How many projects use this pipeline? +- What branching strategy is used? Feature branches? Mainline? Release branches? +- What tools do you use to build your code? For example, Maven, Gradle, or NPM? +- What tools do you use to test your code? For example JUnit, Pytest, or Jest? +- Do you use any security scanners? +- Where do you store any built packages? +- How do you deploy your code? +- Where do you deploy your code? + +### Jenkins + +If you are [migrating from Jenkins](jenkins.md), these additional questions can help with planning +the migration: + +- What plugins are used by jobs in Jenkins today? + - Do you know what these plugins do exactly? + - Do any plugin wrap a common build tool? For example, Maven, Gradle, or NPM? +- What is installed on the Jenkins agents? +- Are there any shared libraries in use? +- How are you authenticating from Jenkins? Are you using SSH keys, API tokens, or other secrets? +- Are there other projects that you need to access from your pipeline? +- Are there credentials in Jenkins to access outside services? For example Ansible Tower, + Artifactory, or other Cloud Providers or deployment targets? diff --git a/doc/ci/mobile_devops.md b/doc/ci/mobile_devops.md index 6969b3d4ef4..4e1a0f4683c 100644 --- a/doc/ci/mobile_devops.md +++ b/doc/ci/mobile_devops.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Mobile DevOps (Experiment) +# Mobile DevOps **(EXPERIMENT)** Use GitLab Mobile DevOps to quickly build, sign, and release native and cross-platform mobile apps for Android and iOS using GitLab CI/CD. Mobile DevOps is an experimental feature developed by @@ -189,8 +189,7 @@ fastlane init ``` This command creates a `fastlane` folder in the project with an `Appfile` and a stubbed-out `fastfile`. -This process asks you for login credentials to App Store Connect -to generate an app identifier and App Store app if they don't already exist. +During this process, you are prompted for App Store Connect login credentials to generate an app identifier and an App Store app if they don't already exist. The next step sets up fastlane match to manage code signing files for the project. Run the following command to generate a `Matchfile` with the configuration: @@ -295,7 +294,7 @@ Use the [Google Play integration](../user/project/integrations/google_play.md), to configure your CI/CD pipelines to connect to the [Google Play Console](https://play.google.com/console) to build and release Android apps. To enable the integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Google Play**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -352,7 +351,7 @@ Use the [Apple App Store integration](../user/project/integrations/apple_app_sto to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com/) to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. To enable the integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Apple App Store**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 5291a8abb49..1dc6bc05e71 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Pipeline Editor **(FREE ALL)** +# Pipeline editor **(FREE ALL)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4540) in GitLab 13.8. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/270059) in GitLab 13.10. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 4e822cf3edd..fa4a5e691a7 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -54,7 +54,7 @@ Prerequisite: To change the default quota that applies to all namespaces: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. @@ -76,7 +76,7 @@ Prerequisite: To set a compute quota for a namespace: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Groups**. 1. For the group you want to update, select **Edit**. @@ -107,7 +107,7 @@ Prerequisite: To view compute usage for your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find your group. The group must not be a subgroup. 1. Select **Settings > Usage Quotas**. 1. Select the **Pipelines** tab. @@ -167,7 +167,7 @@ You can purchase additional compute minutes for your group. You cannot transfer purchased compute minutes from one group to another, so be sure to select the correct group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Usage Quotas**. 1. Select **Pipelines**. 1. Select **Buy additional compute minutes**. diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index fca6e8407ef..6dc58882f37 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -439,6 +439,7 @@ upstream pipeline: Use [`needs:project`](../yaml/index.md#needsproject) to fetch artifacts from an upstream pipeline: +1. In GitLab 15.9 and later, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist) of the upstream project. 1. In the upstream pipeline, save the artifacts in a job with the [`artifacts`](../yaml/index.md#artifacts) keyword, then trigger the downstream pipeline with a trigger job: @@ -491,6 +492,7 @@ because the downstream pipeline attempts to fetch artifacts from the latest bran To fetch the artifacts from the upstream `merge request` pipeline instead of the `branch` pipeline, pass `CI_MERGE_REQUEST_REF_PATH` to the downstream pipeline using [variable inheritance](#pass-yaml-defined-cicd-variables): +1. In GitLab 15.9 and later, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist) of the upstream project. 1. In a job in the upstream pipeline, save the artifacts using the [`artifacts`](../yaml/index.md#artifacts) keyword. 1. In the job that triggers the downstream pipeline, pass the `$CI_MERGE_REQUEST_REF_PATH` variable: @@ -718,6 +720,80 @@ Use the [`trigger:forward` keyword](../yaml/index.md#triggerforward) to specify what type of variables to forward to the downstream pipeline. Forwarded variables are considered trigger variables, which have the [highest precedence](../variables/index.md#cicd-variable-precedence). +## Downstream pipelines for deployments + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369061) in GitLab 16.4. + +You can use the [`environment`](../yaml/index.md#environment) keyword with [`trigger`](../yaml/index.md#trigger). +You might want to use `environment` from a trigger job if your deployment and application projects are separately managed. + +```yaml +deploy: + trigger: + project: project-group/my-downstream-project + environment: production +``` + +A downstream pipeline can provision infrastructure, deploy to a designated environment, and return the deployment status +to the upstream project. + +You can [view the environment and deployment](../environments/index.md#view-environments-and-deployments) +from the upstream project. + +### Advanced example + +This example configuration has the following behaviors: + +- The upstream project dynamically composes an environment name based on a branch name. +- The upstream project passes the context of the deployment to the downstream project with `UPSTREAM_*` variables. + +The `.gitlab-ci.yml` in an upstream project: + +```yaml +stages: + - deploy + - cleanup + +.downstream-deployment-pipeline: + variables: + UPSTREAM_PROJECT_ID: $CI_PROJECT_ID + UPSTREAM_ENVIRONMENT_NAME: $CI_ENVIRONMENT_NAME + UPSTREAM_ENVIRONMENT_ACTION: $CI_ENVIRONMENT_ACTION + trigger: + project: project-group/deployment-project + branch: main + strategy: depend + +deploy-review: + stage: deploy + extends: .downstream-deployment-pipeline + environment: + name: review/$CI_COMMIT_REF_SLUG + on_stop: stop-review + +stop-review: + stage: cleanup + extends: .downstream-deployment-pipeline + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop + when: manual +``` + +The `.gitlab-ci.yml` in a downstream project: + +```yaml +deploy: + script: echo "Deploy to ${UPSTREAM_ENVIRONMENT_NAME} for ${UPSTREAM_PROJECT_ID}" + rules: + - if: $CI_PIPELINE_SOURCE == "pipeline" && $UPSTREAM_ENVIRONMENT_ACTION == "start" + +stop: + script: echo "Stop ${UPSTREAM_ENVIRONMENT_NAME} for ${UPSTREAM_PROJECT_ID}" + rules: + - if: $CI_PIPELINE_SOURCE == "pipeline" && $UPSTREAM_ENVIRONMENT_ACTION == "stop" +``` + ## Troubleshooting ### Trigger job fails and does not create multi-project pipeline @@ -747,3 +823,9 @@ You cannot trigger a multi-project pipeline with a tag when a branch exists with name. The downstream pipeline fails to create with the error: `downstream pipeline can not be created, Ref is ambiguous`. Only trigger multi-project pipelines with tag names that do not match branch names. + +### `403 Forbidden` error when downloading a job artifact from an upstream pipeline + +In GitLab 15.9 and later, CI/CD job tokens are scoped to the project that the pipeline executes under. Therefore, the job token in a downstream pipeline cannot be used to access an upstream project by default. + +To resolve this, [add the downstream project to the job token scope allowlist](../jobs/ci_job_token.md#add-a-project-to-the-job-token-scope-allowlist). diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 7cde8b50524..f0f0f6d29d2 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -139,7 +139,7 @@ operation of the pipeline. To execute a pipeline manually: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. Select **Run pipeline**. 1. In the **Run for branch name or tag** field, select the branch or tag to run the pipeline for. @@ -310,9 +310,9 @@ related objects, such as builds, logs, artifacts, and triggers. A strict security model is enforced when pipelines are executed on [protected branches](../../user/project/protected_branches.md). -The following actions are allowed on protected branches only if the user is +The following actions are allowed on protected branches if the user is [allowed to merge or push](../../user/project/protected_branches.md) -on that specific branch: +to that specific branch: - Run manual pipelines (using the [Web UI](#run-a-pipeline-manually) or [pipelines API](#pipelines-api)). - Run scheduled pipelines. @@ -321,15 +321,13 @@ on that specific branch: - Trigger manual actions on existing pipelines. - Retry or cancel existing jobs (using the Web UI or pipelines API). -**Variables** marked as **protected** are accessible only to jobs that -run on protected branches, preventing untrusted users getting unintended access to -sensitive information like deployment credentials and tokens. +**Variables** marked as **protected** are accessible to jobs that run in pipelines for protected branches. Only assign users the right to merge to protected branches if they have permission to access sensitive information like deployment credentials and tokens. **Runners** marked as **protected** can run jobs only on protected branches, preventing untrusted code from executing on the protected runner and preserving deployment keys and other credentials from being unintentionally accessed. To ensure that jobs intended to be executed on protected -runners do not use regular runners, they must be tagged accordingly. +runners do not use regular runners, they must be [tagged](../yaml/index.md#tags) accordingly. Review the [deployment safety](../environments/deployment_safety.md) page for additional security recommendations for securing your pipelines. @@ -349,7 +347,7 @@ Prerequisites: To trigger the pipeline when the upstream project is rebuilt: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline subscriptions**. 1. Select **Add project**. diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 356b97aacc0..7b8a2a16734 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -169,8 +169,13 @@ To use the UI to run a pipeline in the parent project for a merge request from a 1. Select **Run pipeline**. You must read and accept the warning, or the pipeline does not run. You can disable this feature by using [the projects API](../../api/projects.md#edit-project) -to disable the `ci_allow_fork_pipelines_to_run_in_parent_project` setting. -The setting is `enabled` by default. +to disable the `ci_allow_fork_pipelines_to_run_in_parent_project` setting (enabled by default). +When you disable this setting, new pipelines from forks in the parent project are prevented. + +WARNING: +Older pipelines created before the setting was disabled are not affected and continue to run. +If you rerun a job in an older pipeline, the job uses the same context as when the +pipeline was originally created. ## Available predefined variables diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index c2bf9743e4f..d7f03490c68 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -101,7 +101,7 @@ Prerequisites: To enable merge trains: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge method** section, verify that **Merge commit** is selected. 1. In the **Merge options** section: diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md index 51678e64b10..1a21a2a9a00 100644 --- a/doc/ci/pipelines/merged_results_pipelines.md +++ b/doc/ci/pipelines/merged_results_pipelines.md @@ -41,7 +41,7 @@ To use merged results pipelines: To enable merged results pipelines in a project, you must have at least the Maintainer role: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge options** section, select **Enable merged results pipelines**. 1. Select **Save changes**. diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index d0324f16ffb..305e48ca9f5 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -306,5 +306,4 @@ deploy_b: environment: production ``` -It's also possible to set jobs to run before or after triggering child pipelines, -for example if you have common setup steps or a unified deployment at the end. +Jobs can be set to run before or after triggering child pipelines in GitLab, allowing common setup steps or unified deployment. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md deleted file mode 100644 index 79435c0276d..00000000000 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -stage: Verify -group: Pipeline Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -redirect_to: '../testing/test_coverage_visualization.md' -remove_date: '2023-08-31' ---- - -This document was moved to [another location](../testing/test_coverage_visualization.md). - -<!-- This redirect file can be deleted after <2023-08-31>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 1d7d6c3289a..3d24be0f8e1 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -28,7 +28,7 @@ The easiest indicators to check for inefficient pipelines are the runtimes of th stages, and the total runtime of the pipeline itself. The total pipeline duration is heavily influenced by the: -- [Size of the repository](../large_repositories/index.md) +- [Size of the repository](../../user/project/repository/managing_large_repositories.md) - Total number of stages and jobs. - Dependencies between jobs. - The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents @@ -227,7 +227,7 @@ Methods to reduce Docker image size: - Use a small base image, for example `debian-slim`. - Do not install convenience tools such as vim or curl if they aren't strictly needed. - Create a dedicated development image. -- Disable man pages and docs installed by packages to save space. +- Disable man pages and documentation installed by packages to save space. - Reduce the `RUN` layers and combine software installation steps. - Use [multi-stage builds](https://blog.alexellis.io/mutli-stage-docker-builds/) to merge multiple Dockerfiles that use the builder pattern into one Dockerfile, which can reduce image size. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 75a7d373203..1003c03bebf 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -26,7 +26,7 @@ Otherwise, the pipeline is not created. No error message is displayed. To add a pipeline schedule: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline schedules**. 1. Select **New schedule** and fill in the form. - **Interval Pattern**: Select one of the preconfigured intervals, or enter a custom @@ -47,7 +47,7 @@ you must delete unused schedules before you can add another. The owner of a pipeline schedule can edit it: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline schedules**. 1. Next to the schedule, select **Edit** (**{pencil}**) and fill in the form. @@ -60,7 +60,7 @@ of the schedule. To trigger a pipeline schedule manually, so that it runs immediately instead of the next scheduled time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipeline schedules**. 1. On the right of the list, for the pipeline you want to run, select **Play** (**{play}**). @@ -79,7 +79,7 @@ including [protected environments](../environments/protected_environments.md) an To take ownership of a pipeline created by a different user: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Build > Pipeline schedules**. 1. On the right of the list, for the pipeline you want to become owner of, select **Take ownership**. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index b9c95c63098..02559da75a0 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -24,7 +24,7 @@ For public and internal projects, you can change who can see your: To change the visibility of your pipelines and related features: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Select or clear the **Public pipelines** checkbox. @@ -56,7 +56,7 @@ This setting has no effect when: To change the pipeline visibility for non-project members: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. For **CI/CD**, choose: @@ -72,7 +72,7 @@ is selected. You can set pending or running pipelines to cancel automatically when a pipeline for new changes runs on the same branch. You can enable this in the project settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General Pipelines**. 1. Select the **Auto-cancel redundant pipelines** checkbox. @@ -94,7 +94,7 @@ newer one, which may not be what you want. To avoid this scenario: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Select the **Prevent outdated deployment jobs** checkbox. @@ -112,7 +112,7 @@ directory. However, you can specify an alternate filename path, including locati To customize the path: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. In the **CI/CD configuration file** field, enter the filename. If the file: @@ -161,7 +161,7 @@ able to edit it. You can choose how your repository is fetched from GitLab when a job runs. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Under **Git strategy**, select an option: @@ -169,7 +169,7 @@ You can choose how your repository is fetched from GitLab when a job runs. for every job. However, the local working copy is always pristine. - `git fetch` is faster because it re-uses the local working copy (and falls back to clone if it doesn't exist). This is recommended, especially for - [large repositories](../large_repositories/index.md#git-strategy). + [large repositories](../../user/project/repository/managing_large_repositories.md#git-strategy). The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) in the `.gitlab-ci.yml` file. @@ -182,7 +182,7 @@ in the `.gitlab-ci.yml` file. You can limit the number of changes that GitLab CI/CD fetches when it clones a repository. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Under **Git strategy**, under **Git shallow clone**, enter a value. @@ -192,14 +192,14 @@ a repository. In GitLab versions 14.7 and later, newly created projects have a default `git depth` value of `20`. GitLab versions 14.6 and earlier have a default `git depth` value of `50`. -This value can be overridden by the [`GIT_DEPTH` variable](../large_repositories/index.md#shallow-cloning) +This value can be overridden by the [`GIT_DEPTH` variable](../../user/project/repository/managing_large_repositories.md#shallow-cloning) in the `.gitlab-ci.yml` file. ## Set a limit for how long jobs can run You can define how long a job can run before it times out. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. In the **Timeout** field, enter the number of minutes, or a human-readable value like `2 hours`. diff --git a/doc/ci/quick_start/tutorial.md b/doc/ci/quick_start/tutorial.md index acc47a07a02..41239615590 100644 --- a/doc/ci/quick_start/tutorial.md +++ b/doc/ci/quick_start/tutorial.md @@ -8,10 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w This tutorial walks you through configuring a progressively more complex CI/CD pipeline through small, iterative steps. The pipeline is always fully functional, -but it gains more functionality with each step. +but it gains more functionality with each step. The goal is to build, test, and deploy +a documentation site. -When you finish this tutorial, you will have a new project on GitLab.com and a working documentation site on -[Docusaurus](https://docusaurus.io/). +When you finish this tutorial, you will have a new project on GitLab.com and a working documentation site +using [Docusaurus](https://docusaurus.io/). To complete this tutorial, you will: @@ -36,8 +37,8 @@ Before adding the pipeline configuration, you must first set up a Docusaurus pro on GitLab.com: 1. Create a new project under your username (not a group): - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Create blank project**. 1. Enter the project details: diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index cf29ab62240..a0f6a8008d0 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -9,7 +9,7 @@ description: Control the job concurrency in GitLab CI/CD > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. -By default, pipelines in GitLab CI/CD run in parallel. The parallelization is an important factor to improve +By default, pipelines in GitLab CI/CD run concurrently. Concurrency is an important factor to improve the feedback loop in merge requests, however, there are some situations that you may want to limit the concurrency on deployment jobs to run them one by one. @@ -130,13 +130,13 @@ pipelines run almost at the same time: Depending on the process mode of the resource group: - If the process mode is set to `unordered`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - There is no guarantee on the job execution order, for example, `deploy-1` could run before or after `deploy-3` runs. - If the process mode is `oldest_first`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - `deploy-1` runs first, `deploy-2` runs second, and `deploy-3` runs last. - If the process mode is `newest_first`: - - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1`, `deploy-2`, and `deploy-3` do not run concurrently. - `deploy-3` runs first, `deploy-2` runs second and `deploy-1` runs last. ## Pipeline-level concurrency control with cross-project/parent-child pipelines diff --git a/doc/ci/review_apps/img/enable_review_app_v16.png b/doc/ci/review_apps/img/enable_review_app_v16.png Binary files differnew file mode 100644 index 00000000000..00e305d6a6b --- /dev/null +++ b/doc/ci/review_apps/img/enable_review_app_v16.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index d05861818e2..f831c53c024 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -76,14 +76,14 @@ Prerequisite: To use the review apps template: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project you want to create a review app job for. -1. Select **Build > Environments**. +1. Select **Operate > Environments**. 1. Select **Enable review apps**. 1. Copy the provided code snippet and paste it into your `.gitlab-ci.yml` file: -  +  You can edit this template as needed. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 3e26c120f74..fd1716cc58a 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -64,9 +64,9 @@ of shared runners on large GitLab instances. This ensures that you control access to your GitLab instances and secure [runner executors](https://docs.gitlab.com/runner/executors/). If certain executors run a job, the file system, the code the runner executes, -and the runner token may be exposed. This means that anyone that runs jobs +and the runner authentication token may be exposed. This means that anyone that runs jobs on a _shared runner_ can access another user's code that runs on the runner. -Users with access to the runner token can use it to create a clone of +Users with access to the runner authentication token can use it to create a clone of a runner and submit false jobs in a vector attack. For more information, see [Security Considerations](https://docs.gitlab.com/runner/security/). ### Prevent runners from revealing sensitive information @@ -76,7 +76,7 @@ on [protected branches](../../user/project/protected_branches.md), or jobs that To prevent runners from revealing sensitive information: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Find the runner you want to protect or unprotect. Make sure the runner is enabled. @@ -111,7 +111,7 @@ That new runner may then be used to obtain the values of secret variables or to To reset the registration token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. To the right of **New project runner**, select the vertical ellipsis (**{ellipsis_v}**). @@ -124,19 +124,19 @@ you use to provision and register new values. ### Reset the runner authentication token -If an authentication token is revealed, an attacker could use the token to [clone a runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). +If a runner authentication token is revealed, an attacker could use the token to [clone a runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). -To reset the authentication token: +To reset the runner authentication token: 1. Delete the runner: - [Delete a shared runner](runners_scope.md#delete-shared-runners). - [Delete a group runner](runners_scope.md#delete-a-group-runner). - [Delete a project runner](runners_scope.md#delete-a-project-runner). -1. Create a new runner so that it is assigned a new authentication token: - - [Create a shared runner](runners_scope.md#create-a-shared-runner-with-an-authentication-token). - - [Create a group runner](runners_scope.md#create-a-group-runner-with-an-authentication-token). - - [Create a project runner](runners_scope.md#create-a-project-runner-with-an-authentication-token). -1. Optional. To verify that the previous authentication token has been revoked, use the [Runners API](../../api/runners.md#verify-authentication-for-a-registered-runner). +1. Create a new runner so that it is assigned a new runner authentication token: + - [Create a shared runner](runners_scope.md#create-a-shared-runner-with-a-runner-authentication-token). + - [Create a group runner](runners_scope.md#create-a-group-runner-with-a-runner-authentication-token). + - [Create a project runner](runners_scope.md#create-a-project-runner-with-a-runner-authentication-token). +1. Optional. To verify that the previous runner authentication token has been revoked, use the [Runners API](../../api/runners.md#verify-authentication-for-a-registered-runner). ## Use tags to control which jobs a runner can run @@ -874,7 +874,7 @@ defaults to the number of CPUs available. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30942) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `enforce_runner_token_expires_at`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/377902) in GitLab 15.5. Feature flag `enforce_runner_token_expires_at` removed. -Each runner has an [authentication token](../../api/runners.md#registration-and-authentication-tokens) +Each runner has an [runner authentication token](../../api/runners.md#registration-and-authentication-tokens) to connect with the GitLab instance. To help prevent the token from being compromised, you can have the @@ -883,12 +883,12 @@ they are updated for each runner, regardless of the runner's status (`online` or No manual intervention should be required, and no running jobs should be affected. -If you need to manually update the authentication token, you can run a +If you need to manually update the runner authentication token, you can run a command to [reset the token](https://docs.gitlab.com/runner/commands/#gitlab-runner-reset-token). -### Automatically rotate authentication tokens +### Automatically rotate runner authentication tokens -You can specify an interval for authentication tokens to rotate. +You can specify an interval for runner authentication tokens to rotate. This rotation helps ensure the security of the tokens assigned to your runners. Prerequisites: @@ -897,11 +897,11 @@ Prerequisites: To automatically rotate runner authentication tokens: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**.. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment** 1. Set a **Runners expiration** time for runners, leave empty for no expiration. 1. Select **Save**. -Before the interval expires, runners automatically request a new authentication token. +Before the interval expires, runners automatically request a new runner authentication token. diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 5427911e1ce..7e597264e1e 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -17,10 +17,15 @@ Your jobs can run on: - [Windows runners](saas/windows_saas_runner.md) ([Beta](../../policy/experiment-beta-support.md#beta)) - [macOS runners](saas/macos_saas_runner.md) ([Beta](../../policy/experiment-beta-support.md#beta)) -Refer to the compute minutes [cost factor](../../ci/pipelines/cicd_minutes.md#cost-factor) for the cost factor applied to the machine type based on size. -The number of minutes you can use on these runners depends on the [maximum number of compute minutes](../pipelines/cicd_minutes.md) +For more information about the cost factor applied to the machine type based on size, see [cost factor](../../ci/pipelines/cicd_minutes.md#cost-factor). +The number of minutes you can use on these runners depends on the [maximum number of units of compute](../pipelines/cicd_minutes.md) in your [subscription plan](https://about.gitlab.com/pricing/). +[Untagged](../../ci/runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) jobs automatically run in containers +on the `small` Linux runners. + +The objective is to make 90% of CI/CD jobs start executing in 120 seconds or less. The error rate should be less than 0.5%. + ## How SaaS runners work When you use SaaS runners: @@ -30,9 +35,6 @@ When you use SaaS runners: - The virtual machine where your job runs has `sudo` access with no password. - The storage is shared by the operating system, the image with pre-installed software, and a copy of your cloned repository. This means that the available free disk space for your jobs to use is reduced. -- [Untagged](../../ci/runners/configure_runners.md#use-tags-to-control-which-jobs-a-runner-can-run) jobs automatically run in containers -on the `small` Linux runners. -- The objective is to make 90% of CI jobs start executing in 120 seconds or less. The error rate target will be less than 0.5%. NOTE: Jobs handled by SaaS runners on GitLab.com **time out after 3 hours**, regardless of the timeout configured in a project. @@ -67,3 +69,65 @@ takes over the task of securely deleting the virtual machine and associated data - Inbound communication from the public internet to the temporary VM is not allowed. - Firewall rules do not permit communication between VMs. - The only internal communication allowed to the temporary VMs is from the runner manager. + +## Supported image lifecycle + +For runners on macOS and Windows, you can only run jobs on supported images. You cannot bring your own image. Supported images have the following lifecycle: + +- Beta +- Generally Available +- Deprecated + +### Beta + +To gather feedback on an image prior to making the image Generally Available (GA) and to address +any issues, new images are released as Beta. Any jobs running on Beta images are not +covered by the service-level agreement. If you use Beta images, you can provide feedback +by creating an issue. + +### Generally Available + +A Generally Available (GA) image is released after the image completes a Beta phase +and is considered suitable for general use. To become GA, the +image must fulfill the following requirements: + +- Successful completion of a Beta phase by resolving all reported significant bugs +- Compatibility of installed software with the underlying OS + +Jobs running on GA images are covered by the defined service-level agreement. Over time, these images are deprecated. + +### Deprecated + +A maximum of two Generally Available (GA) images are supported at a time. After a new GA image is released, +the oldest GA image becomes deprecated. A deprecated image is no longer +updated and is deleted after 3 months in accordance with the [deprecation guidelines](../../development/deprecation_guidelines/index.md). + +## Major version changes (breaking) + +As GitLab CI/CD and Runner have evolved, certain breaking changes have been necessary. + +For GitLab 15.0 and later, all breaking changes are documented on the following page: + +- [Deprecations and removals](../../update/deprecations.md) + +The breaking changes for GitLab Runner in earlier major version releases are: + +- 14.0: No breaking changes. +- 13.0: + - [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). + - [Remove Fedora 29 package support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/16158). + - [Remove macOS 32-bit support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25466). + - [Removed `debug/jobs/list?v=1` endpoint](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6361). + - [Remove support for array of strings when defining services for Docker executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4922). + - [Remove `--docker-services` flag on register command](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6404). + - [Remove legacy build directory caching](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4180). + - [Remove `FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER` feature flag](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6581). + - [Remove support for Windows Server 1803](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6553). +- 12.0: + - [Use `refspec` to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4069). + - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4070). + - [Old metrics server configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4072). + - [Remove `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4073). + - [Remove Linux distributions that reach EOL](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1130). + - [Update command line API for helper images](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4013). + - [Remove old `git clean` flow](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4175). diff --git a/doc/ci/runners/new_creation_workflow.md b/doc/ci/runners/new_creation_workflow.md index 55ff5165ff7..c4c2fbebb12 100644 --- a/doc/ci/runners/new_creation_workflow.md +++ b/doc/ci/runners/new_creation_workflow.md @@ -14,15 +14,13 @@ As with all projects, the items mentioned on this page are subject to change or The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc. -In GitLab 16.0, we introduced a new runner creation workflow that uses authentication tokens to register -runners. The legacy workflow that uses registration tokens is deprecated and will be removed in GitLab 17.0. +In GitLab 16.0, we introduced a new runner creation workflow that uses runner authentication tokens to register +runners. The legacy workflow that uses registration tokens is deprecated and will be removed in GitLab 18.0. For information about the current development status of the new workflow, see [epic 7663](https://gitlab.com/groups/gitlab-org/-/epics/7663). For information about the technical design and reasons for the new architecture, see [Next GitLab Runner Token Architecture](../../architecture/blueprints/runner_tokens/index.md). -## Feedback - If you experience problems or have concerns about the new runner registration workflow, or if the following information is not sufficient, you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387993). @@ -32,8 +30,8 @@ you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab For the new runner registration workflow, you: 1. [Create a runner](register_runner.md) directly in the GitLab UI. -1. Receive an authentication token. -1. Use the authentication token instead of the registration token when you register +1. Receive a runner authentication token. +1. Use the runner authentication token instead of the registration token when you register a runner with this configuration. Runner managers registered in multiple hosts appear under the same runner in the GitLab UI, but with an identifying system ID. @@ -46,54 +44,63 @@ multiple runners. For more information, see [Reusing a GitLab Runner configurati ## Estimated time frame for planned changes - In GitLab 15.10 and later, you can use the new runner registration workflow. -- In GitLab 16.6, we plan to disable registration tokens. -- In GitLab 17.0, we plan to completely remove support for runner registration tokens. +- In GitLab 17.0, we plan to disable runner registration tokens. +- In GitLab 18.0, we plan to completely remove support for runner registration tokens. ## Prevent your runner registration workflow from breaking -Until GitLab 16.6, you can still use the legacy runner registration workflow. +Until GitLab 17.0, you can still use the legacy runner registration workflow. -In GitLab 16.6, the legacy runner registration workflow will be disabled automatically. You will be able to manually re-enable the legacy runner registration workflow for a limited time. For more information, see -[Using registration tokens after GitLab 16.6](#using-registration-tokens-after-gitlab-166). +In GitLab 17.0, the legacy runner registration workflow will be disabled automatically. You will be able to manually re-enable the legacy runner registration workflow for a limited time. For more information, see +[Using registration tokens after GitLab 17.0](#using-registration-tokens-after-gitlab-170). -If no action is taken before your GitLab instance is upgraded to GitLab 16.6, then your runner registration +If no action is taken before your GitLab instance is upgraded to GitLab 17.0, then your runner registration workflow will break. To avoid a broken workflow, you must: -1. [Create a shared runner](runners_scope.md#create-a-shared-runner-with-an-authentication-token) and obtain the authentication token. +1. [Create a shared runner](runners_scope.md#create-a-shared-runner-with-a-runner-authentication-token) and obtain the authentication token. 1. Replace the registration token in your runner registration workflow with the authentication token. -## Using registration tokens after GitLab 16.6 +## Using registration tokens after GitLab 17.0 -To continue using registration tokens after GitLab 16.6: +To continue using registration tokens after GitLab 17.0: -- On GitLab.com, you can manually re-enable the legacy runner registration process in the top-level group settings until GitLab 16.8. -- On GitLab self-managed, you can manually re-enable the legacy runner registration process in the Admin Area settings until GitLab 17.0. +- On GitLab.com, you can manually re-enable the legacy runner registration process in the top-level group settings until GitLab 18.0. +- On GitLab self-managed, you can manually re-enable the legacy runner registration process in the Admin Area settings until GitLab 18.0. Plans to implement a UI setting to re-enable registration tokens are proposed in [issue 411923](https://gitlab.com/gitlab-org/gitlab/-/issues/411923) +## Runners registered with a registration token will continue to work after 18.0 + +Existing runners will not be affected by these changes, they will still work even after the legacy registration method is removed. + ## Changes to the `gitlab-runner register` command syntax -The `gitlab-runner register` command will stop accepting registration tokens and instead accept new +The `gitlab-runner register` command will stop accepting registration tokens and instead accept new runner authentication tokens generated in the GitLab runners administration page. -These authentication tokens are recognizable by their `glrt-` prefix. +The runner authentication tokens are recognizable by their `glrt-` prefix. When you create a runner in the GitLab UI, you specify configuration values that were previously command-line options prompted by the `gitlab-runner register` command. These command-line options have been [deprecated](../../update/deprecations.md#registration-tokens-and-server-side-runner-arguments-in-post-apiv4runners-endpoint). -If you specify an authentication token with: +If you specify a runner authentication token with: - the `--token` command-line option, the `gitlab-runner register` command does not accept the configuration values. - the `--registration-token` command-line option, the `gitlab-runner register` command ignores the configuration values. +| Token | Registration command | +|----------------------------------------|-----------------------------------------------------------------------------------------------------------| +| Runner authentication token | `gitlab-runner register --token $RUNNER_AUTHENTICATION_TOKEN` | +| Runner registration token (deprecated) | `gitlab-runner register --registration-token $RUNNER_REGISTRATION_TOKEN <runner configuration arguments>` | + Authentication tokens have the prefix, `glrt-`. To ensure minimal disruption to your automation workflow, [legacy-compatible registration processing](https://docs.gitlab.com/runner/register/#legacy-compatible-registration-processing) -triggers if an authentication token is specified in the legacy parameter `--registration-token`. +triggers if a runner authentication token is specified in the legacy parameter `--registration-token`. Example command for GitLab 15.9: @@ -126,7 +133,7 @@ gitlab-runner register \ ## Impact on autoscaling In autoscaling scenarios such as GitLab Runner Operator or GitLab Runner Helm Chart, the -registration token is replaced with the authentication token generated from the UI. +registration token is replaced with the runner authentication token generated from the UI. This means that the same runner configuration is reused across jobs, instead of creating a runner for each job. The specific runner can be identified by the unique system ID that is generated when the runner @@ -138,33 +145,13 @@ Existing runners will continue to work as usual. This change only affects regist ## Creating runners programmatically -A new [POST /user/runners REST API](../../api/users.md#create-a-runner) was introduced in -GitLab 15.11, which allows a runner to be created in the context of an authenticated user. This should only be used in -scenarios where the runner configuration is dynamic, or not reusable. If the runner configuration is static, it is -preferable to reuse the authentication token of an existing runner. +In GitLab 15.11 and later, you can use the [POST /user/runners REST API](../../api/users.md#create-a-runner) +to create a runner as an authenticated user. This should only be used if the runner configuration is dynamic +or not reusable. If the runner configuration is static, you should reuse the runner authentication token of +an existing runner. -The following snippet shows how a group runner could be created and registered with a -[Group Access Token](../../user/group/settings/group_access_tokens.md) using the new creation flow. -The process is very similar when using [Project Access Tokens](../../user/project/settings/project_access_tokens.md) -or [Personal Access Tokens](../../user/profile/personal_access_tokens.md): - -```shell -# `GROUP_ID` contains the numerical ID of the group where the runner will be created -# `GITLAB_TOKEN` can be a Personal Access Token for a group owner, or a Group Access Token on the respective group -# created with `owner` access and `api` scope. -# -# The output will be parsed by `jq` to extract the token of the newly created runner -RUNNER_TOKEN=$(curl --silent --request POST "https://gitlab.com/api/v4/user/runners" \ - --header "private-token: $GITLAB_TOKEN" \ - --data runner_type=group_type --data group_id=$GROUP_ID --data 'description=My runner' --data 'tag_list=java,linux' \ - | jq -r '.token') - -gitlab-runner register \ - --non-interactive \ - --executor "shell" \ - --url "https://gitlab.com/" \ - --token "$RUNNER_TOKEN" -``` +For instructions about how to automate runner creation and registration, see the tutorial, +[Automate runner creation and registration](../../tutorials/automate_runner_creation/index.md). ## Installing GitLab Runner with Helm chart @@ -185,7 +172,7 @@ runUntagged: true protected: true ``` -If you store the runner token in `secrets`, you must also modify them. +If you store the runner authentication token in `secrets`, you must also modify them. In the legacy runner registration workflow, fields were specified with: @@ -212,3 +199,7 @@ data: runner-registration-token: "" # need to leave as an empty string for compatibility reasons runner-token: "REDACTED" ``` + +NOTE: +If your secret management solution doesn't allow you to set an empty string for `runner-registration-token`, +you can set it to any string - it will be ignored when `runner-token` is present. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index fff695eb606..c6387a60495 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -34,7 +34,7 @@ If you are using GitLab.com: - The shared runners consume the [compute minutes](../pipelines/cicd_minutes.md) included with your account. -### Create a shared runner with an authentication token +### Create a shared runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383139) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_admin` [flag](../../administration/feature_flags.md) > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/389269) in GitLab 16.0. @@ -44,23 +44,32 @@ Prerequisite: - You must be an administrator. -When you create a runner, it is assigned an authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +When you create a runner, it is assigned a runner authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. To create a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Select **New instance runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, to add a runner description + that displays in GitLab, enter a runner description. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for a limited period of time during registration. After you register the runner, +the authentication token is stored in the `config.toml`. ### Create a shared runner with a registration token (deprecated) @@ -75,7 +84,7 @@ Prerequisite: To create a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. Select **Register an instance runner**. @@ -90,7 +99,7 @@ Prerequisite: You can pause a runner so that it does not accept jobs from groups and projects in the GitLab instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. In the search box, enter the runner description or filter the runner list. @@ -110,7 +119,7 @@ jobs, you can [pause](#pause-or-resume-a-shared-runner) the runner instead. To delete a single or multiple shared runners: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. 1. In the search box, enter the runner description or filter the list of runners. @@ -134,7 +143,7 @@ For existing projects, an administrator must To enable shared runners for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable shared runners for this project** toggle. @@ -143,7 +152,7 @@ To enable shared runners for a project: To enable shared runners for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable shared runners for this group** toggle. @@ -156,7 +165,7 @@ or group. To disable shared runners for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. In the **Shared runners** area, turn off the **Enable shared runners for this project** toggle. @@ -170,7 +179,7 @@ Shared runners are automatically disabled for a project: To disable shared runners for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn off the **Enable shared runners for this group** toggle. @@ -222,7 +231,7 @@ to have access to a set of runners. Group runners process jobs by using a first in, first out queue. -### Create a group runner with an authentication token +### Create a group runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383143) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_namespace` [flag](../../administration/feature_flags.md). Disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/393919) in GitLab 16.0. @@ -233,22 +242,31 @@ Prerequisites: - You must have the Owner role for the group. You can create a group runner for your self-managed GitLab instance or for GitLab.com. -When you create a runner, it is assigned an authentication token that you use to register it. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +When you create a runner, it is assigned a runner authentication token that you use to register it. +The runner uses the token to authenticate with GitLab when it picks up jobs from the job queue. To create a group runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. Select **New group runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, add a runner description + that displays in GitLab. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for only a short period of time during registration. ### Create a group runner with a registration token (deprecated) @@ -256,7 +274,7 @@ The authentication token displays in the UI for only a short period of time duri WARNING: The ability to pass a runner registration token, and support for certain configuration arguments was -[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) in GitLab 15.6 and will be removed in GitLab 17.0. Authentication tokens +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) in GitLab 15.6 and will be removed in GitLab 18.0. Authentication tokens should be used instead. For more information, see [Migrating to the new runner registration workflow](new_creation_workflow.md). You must have the Owner role for the group. @@ -264,7 +282,7 @@ You must have the Owner role for the group. To create a group runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the upper-right corner, select **Register a group runner**. 1. Select **Show runner installation and registration instructions**. @@ -273,17 +291,20 @@ To create a group runner: Alternately, you can copy the registration token and follow the documentation for how to [register a runner](https://docs.gitlab.com/runner/register/). -### View and manage group runners +### View group runners + +> Ability for users with the Maintainer role to view group runners [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384179) in GitLab 16.4. + +Prerequisite: + +- You must have the Maintainer or Owner role for the group. -You can view and manage all runners for a group, its subgroups, and projects. +You can view all runners for a group and its subgroups and projects. You can do this for your self-managed GitLab instance or for GitLab.com. -You must have the Owner role for the group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. -From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. - #### Filter group runners to show only inherited > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337838/) in GitLab 15.5. @@ -297,7 +318,7 @@ By default, only those that are inherited are shown. To show all runners available in the instance, including shared runners and those in other groups: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. Above the list, turn off the **Show only inherited** toggle. @@ -310,7 +331,7 @@ Prerequisite: You can pause a runner so that it does not accept jobs from subgroups and projects in the GitLab instance. If you pause a group runner that is used by multiple projects, the runner pauses for all projects. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the search box, enter the runner description or filter the runner list. 1. In the runner list, to the right of the runner: @@ -331,7 +352,7 @@ jobs, you can [pause](#pause-or-resume-a-group-runner) the runner instead. To delete a single or multiple group runners: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. 1. In the search box, enter the runner description or filter the list of runners. 1. Delete the group runner: @@ -344,11 +365,15 @@ To delete a single or multiple group runners: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363012) in GitLab 15.1. +Prerequisite: + +- You must have the Owner role for the group. + You can clean up group runners that have been inactive for more than three months. Group runners are those that were created at the group level. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. 1. Turn on the **Enable stale runner cleanup** toggle. @@ -399,7 +424,7 @@ NOTE: Project runners do not get shared with forked projects automatically. A fork *does* copy the CI/CD settings of the cloned repository. -### Create a project runner with an authentication token +### Create a project runner with a runner authentication token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383143) in GitLab 15.10. Deployed behind the `create_runner_workflow_for_namespace` [flag](../../administration/feature_flags.md). Disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/393919) in GitLab 16.0. @@ -409,23 +434,33 @@ Prerequisites: - You must have the Maintainer role for the project. -You can create a project runner for your self-managed GitLab instance or for GitLab.com. When you create a runner, it is assigned an authentication token that you use to register to the runner. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. +You can create a project runner for your self-managed GitLab instance or for GitLab.com. When you create a runner, +it is assigned a runner authentication token that you use to register to the runner. The runner uses the token to +authenticate with GitLab when it picks up jobs from the job queue. To create a project runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand the **Runners** section. 1. Select **New project runner**. -1. Select a platform. -1. Optional. Enter configurations for the runner. -1. Select **Submit**. -1. Follow the on-screen instructions to register the runner from the command line. - -You can also [create a runner](../../api/users.md#create-a-runner) with the API to generate an authentication token. +1. Select the operating system where GitLab Runner is installed. +1. In the **Tags** section, in the **Tags** field, enter the job tags to specify jobs the runner can run. + If there are no job tags for this runner, select **Run untagged**. +1. Optional. In the **Runner description** field, add a description for the runner + that displays in GitLab. +1. Optional. In the **Configuration** section, add additional configurations. +1. Select **Create runner**. +1. Follow the on-screen instructions to register the runner from the command line. When prompted by the command line: + - For the `GitLab instance URL`, use the URL for your GitLab instance. For example, if your project + is hosted on `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is `https://gitlab.example.com`. + - For the `executor`, enter the type of [executor](https://docs.gitlab.com/runner/executors/). The executor is the + environment where the runner executes the job. + +You can also [use the API](../../api/users.md#create-a-runner) to create a runner. NOTE: -The authentication token displays in the UI for only a short period of time during registration. +The runner authentication token displays in the UI for only a short period of time during registration. ### Create a project runner with a registration token (deprecated) @@ -441,7 +476,7 @@ Prerequisite: To create a project runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to use the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -459,7 +494,7 @@ Prerequisite: You can pause a project runner so that it does not accept jobs from projects it's assigned to in the GitLab instance. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -479,7 +514,7 @@ When you delete a project runner, it is permanently deleted from the GitLab inst no longer be used by projects. If you want to temporarily stop the runner from accepting jobs, you can [pause](#pause-or-resume-a-project-runner) the runner instead. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -500,7 +535,7 @@ You must have at least the Maintainer role for: To enable a project runner for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -520,7 +555,7 @@ but can also be changed later. To lock or unlock a project runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to +1. On the left sidebar, select **Search or go to** and find the project where you want to enable the runner. 1. Select **Settings > CI/CD**. 1. Expand **Runners**. @@ -553,7 +588,7 @@ runners are considered. queued for longer than the median value, and half of the jobs queued for less than the median value. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Select **View metrics**. @@ -569,10 +604,10 @@ To determine which runners need to be upgraded: 1. View the list of runners: - For a group: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Build > Runners**. - For the instance: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **CI/CD > Runners**. @@ -602,7 +637,7 @@ Prerequisite: To determine the IP address of a shared runner: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **CI/CD > Runners**. 1. Find the runner in the table and view the **IP Address** column. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index dd5381f3cd6..c026ccf3d22 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -29,7 +29,7 @@ The `small` machine type is set as default. If no [tag](../../yaml/index.md#tags the jobs will run on this default runner. All SaaS runners on Linux currently run on -[`n2d-standard`](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) gerneral-purpose compute from GCP. +[`n2d-standard`](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) general-purpose compute from GCP. The machine type and underlying processor type can change. Jobs optimized for a specific processor design could behave inconsistently. ## Container images diff --git a/doc/ci/runners/saas/macos/codesigning.md b/doc/ci/runners/saas/macos/codesigning.md deleted file mode 100644 index 7e70e984c7c..00000000000 --- a/doc/ci/runners/saas/macos/codesigning.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../macos_saas_runner.md' -remove_date: '2023-09-05' ---- - -This document was moved to [another location](../macos_saas_runner.md). - -<!-- This redirect file can be deleted after <2023-09-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md deleted file mode 100644 index 7e70e984c7c..00000000000 --- a/doc/ci/runners/saas/macos/environment.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../macos_saas_runner.md' -remove_date: '2023-09-05' ---- - -This document was moved to [another location](../macos_saas_runner.md). - -<!-- This redirect file can be deleted after <2023-09-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 44f99ed6ccc..1445ae58bd4 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SaaS runners on macOS (Beta) **(PREMIUM SAAS)** +# SaaS runners on macOS **(PREMIUM SAAS BETA)** SaaS runners on macOS are in [Beta](../../../policy/experiment-beta-support.md#beta) for open source programs and customers in Premium and Ultimate plans. @@ -38,28 +38,30 @@ in your `.gitlab-ci.yml` file. Each image runs a specific version of macOS and Xcode. -| VM image | Status | -|---------------------------|---------------| -| `macos-12-xcode-13` | `maintenance` | -| `macos-12-xcode-14` | `maintenance` | -| (none, awaiting macOS 13) | `beta` | +| VM image | Status | +|----------------------------|--------| +| `macos-12-xcode-13` | `GA` | +| `macos-12-xcode-14` | `GA` | +| `macos-13-xcode-14` | `Beta` | -NOTE: -If your job requires tooling or dependencies not available in our available images, those can only be installed in the job execution. +## Image update policy for macOS -## Image update policy +macOS and Xcode follow a yearly release cadence, during which GitLab increments its versions synchronously. GitLab typically supports multiple versions of preinstalled tools. For more information, see +a [full list of preinstalled software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/job-images/-/tree/main/toolchain). -GitLab expects to release new images based on this cadence: +GitLab provides `stable` and `latest` macOS images that follow different update patterns: -macOS updates: +- **Stable image:** The `stable` images and installed components are updated every release. Images without the `:latest` prefix are considered stable images. +- **Latest image:** The `latest` images are typically updated on a weekly cadence and use a `:latest` prefix in the image name. Using the `latest` image results in more regularly updated components and shorter update times for Homebrew or asdf. The `latest` images are used to test software components before releasing the components to the `stable` images. +By definition, the `latest` images are always Beta. +A `latest` image is not available. -- **For new OS versions:** When Apple releases a new macOS version to developers (like macOS `12`), GitLab will plan to release an image based on the OS within the next 30 business days. The image is considered `beta` and the contents of the image (including tool versions) are subject to change until the first patch release (`12.1`). The long-term name will not include `beta` (for example, `macos-12-xcode-13`), so customers are moved automatically out of beta over time. GitLab will try to minimize breaking changes between the first two minor versions but makes no guarantees. Tooling often gets critical bug fixes after the first public release of an OS version. +### Image release process -- **After the first patch release (`12.1`):** - - The image moves to `maintenance` mode. The tools GitLab builds into the image with Homebrew and asdf are frozen. GitLab continues making Xcode updates, security updates, and any non-breaking changes deemed necessary. - - The image for the previous OS version (`11`) moves to `frozen` mode. GitLab then does only unavoidable changes: security updates, runner version upgrades, and setting the production password. +When Apple releases a new macOS version, GitLab releases both `stable` and `latest` images based on the OS in the next release. Both images are Beta. -Both macOS and Xcode follow a yearly release cadence. As time goes on, GitLab increments their versions synchronously (meaning we build macOS 11 with Xcode 12, macOS 12 with Xcode 13, and so on). +With the release of the first patch to macOS, the `stable` image becomes Generally Available (GA). +As only two GA images are supported at a time, the prior OS version becomes deprecated and is deleted after three months in accordance with the [supported image lifecycle](../index.md#supported-image-lifecycle). ## Example `.gitlab-ci.yml` file diff --git a/doc/ci/runners/saas/windows_saas_runner.md b/doc/ci/runners/saas/windows_saas_runner.md index 8ec44b8c275..108388f22c8 100644 --- a/doc/ci/runners/saas/windows_saas_runner.md +++ b/doc/ci/runners/saas/windows_saas_runner.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SaaS runners on Windows (Beta) **(FREE SAAS)** +# SaaS runners on Windows **(FREE SAAS BETA)** SaaS runner on Windows autoscale by launching virtual machines on the Google Cloud Platform. This solution uses an @@ -35,7 +35,7 @@ You can execute your job in one of the following Windows versions: | Version tag | Status | |----------------|---------------| -| `windows-1809` | `maintenance` | +| `windows-1809` | `Beta` | You can find a full list of available pre-installed software in the [pre-installed software documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). @@ -81,16 +81,15 @@ test: - echo "running scripts in the test job" ``` -## Limitations and known issues +## Known issues -- All the limitations mentioned in our [Beta definition](../../../policy/experiment-beta-support.md#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- For more information about support for Beta features, see [Beta](../../../policy/experiment-beta-support.md#beta). +- The average provisioning time for a new Windows virtual machine (VM) is five minutes, so + you might notice slower start times for builds on the Windows runner + fleet during the Beta. Updating the autoscaler to enable the pre-provisioning + of virtual machines is proposed in a future release. This update is intended to + significantly reduce the time it takes to provision a VM on the Windows fleet. + For more information, see [issue 32](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). - The Windows runner fleet may be unavailable occasionally for maintenance or updates. - The job may stay in a pending state for longer than the diff --git a/doc/ci/secrets/id_token_authentication.md b/doc/ci/secrets/id_token_authentication.md index 22a260e4bb6..697346474f8 100644 --- a/doc/ci/secrets/id_token_authentication.md +++ b/doc/ci/secrets/id_token_authentication.md @@ -193,7 +193,7 @@ ID token authentication is now always available, and JSON Web Token access is al To enable automatic ID token authentication: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Token Access**. 1. Turn on the **Limit JSON Web Token (JWT) access** toggle. diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index 37c453a5b9d..a666e0aca7b 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -29,7 +29,7 @@ tool. To add a secure file to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand the **Secure Files** section. 1. Select **Upload File**. diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index f92c50fcf3f..a2ffd095de3 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -493,5 +493,10 @@ Docker privileged mode applies to services. This means that the service image co ## Shared /builds directory -Services can access files from the build because all services have the job -directory mounted as a volume under `/builds`. +The build directory is mounted as a volume under `/builds` and is shared +between the job and services. The job checks the project out into +`/builds/$CI_PROJECT_PATH` after the services are running. As a result, if your +service needs files from the project or, for example, wants to put files there +to serve as artifacts, it may need to wait for that directory to exist and +have `$CI_COMMIT_SHA` checked out. Any changes made before the job finishes its +checkout process are removed by the checkout process. diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 0bc9ae7776e..d9dcbca0825 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -27,7 +27,7 @@ Prerequisite: To create a test case in a GitLab project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select **New test case**. You are taken to the new test case form. Here you can enter the new case's title, [description](../../user/markdown.md), attach a file, and assign [labels](../../user/project/labels.md). @@ -45,7 +45,7 @@ Prerequisites: To view a test case: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select the title of the test case you want to view. You are taken to the test case page. @@ -80,7 +80,7 @@ To archive a test case, on the test case's page, select **Archive test case**. To view archived test cases: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Test cases**. 1. Select **Archived**. diff --git a/doc/ci/testing/code_coverage.md b/doc/ci/testing/code_coverage.md index 90a07314083..fb846f52a72 100644 --- a/doc/ci/testing/code_coverage.md +++ b/doc/ci/testing/code_coverage.md @@ -72,7 +72,7 @@ Use this regex for commonly used test tools. To see the evolution of your project code coverage over time, you can view a graph or download a CSV file with this data. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Analyze > Repository analytics**. The historic data for each job is listed in the dropdown list above the graph. diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index 549aa10287e..6b16f15a9d9 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -59,7 +59,7 @@ Configuring your Load Performance Testing job can be broken down into several di ### Determine the test parameters -The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction) +The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/load-test-types/) you want to run, and how you want it to run (for example, the number of users, throughput, and so on). Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides), @@ -79,7 +79,7 @@ We strongly recommend [not running these tests against a production environment] ### Write the load performance test After the environment is prepared, you can write the k6 test itself. k6 is a flexible -tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction). +tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/load-test-types/). Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests. ### Configure the test in GitLab CI/CD @@ -151,7 +151,7 @@ The CI/CD YAML configuration example above works for testing against static envi but it can be extended to work with [review apps](../review_apps/index.md) or [dynamic environments](../environments/index.md) with a few extra steps. -The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/) +The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/environment-variables/env-file/) as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS` to configure the k6 Docker container to use the file. With this, k6 can then use any environment variables from the `.env` file in scripts using standard JavaScript, diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index 506f6fb2106..698118f457f 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -26,7 +26,7 @@ Prerequisite: To create a trigger token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline trigger tokens**. 1. Select **Add new token** @@ -154,7 +154,7 @@ users with the Owner and Maintainer role can view the values. To revoke a pipeline trigger token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Pipeline triggers**. 1. To the left of the trigger token you want to revoke, select **Revoke** (**{remove}**). diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 3d5bcc64889..e0b8c6213de 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -210,6 +210,42 @@ To illustrate its life cycle: 1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA. 1. When the pipeline finishes, its persistent ref is cleaned up in a background process. +### `get_sources` job section fails because of an HTTP/2 problem + +Sometimes, jobs fail with the following cURL error: + +```plaintext +++ git -c 'http.userAgent=gitlab-runner <version>' fetch origin +refs/pipelines/<id>:refs/pipelines/<id> ... +error: RPC failed; curl 16 HTTP/2 send again with decreased length +fatal: ... +``` + +You can work around this problem by configuring Git and `libcurl` to +[use HTTP/1.1](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpversion). +The configuration can be added to: + +- A job's [`pre_get_sources_script`](yaml/index.md#hookspre_get_sources_script): + + ```yaml + job_name: + hooks: + pre_get_sources_script: + - git config --local http.version "HTTP/1.1" + ``` + +- The [runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) + with [Git configuration environment variables](https://git-scm.com/docs/git-config#ENVIRONMENT): + + ```toml + [[runners]] + ... + environment = [ + "GIT_CONFIG_COUNT=1", + "GIT_CONFIG_KEY_1=http.version", + "GIT_CONFIG_VALUE_1=HTTP/1.1" + ] + ``` + ### Merge request pipeline messages The merge request pipeline widget shows information about the pipeline status in @@ -440,9 +476,9 @@ This flag reduces system resource usage on the `jobs/request` endpoint. When enabled, jobs created in the last hour can run in projects which are out of quota. Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). -## CI/CD troubleshooting rails console commands +## CI/CD troubleshooting Rails console commands -The following commands are run in the [rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). +The following commands are run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). WARNING: Any command that changes data directly could be damaging if not run correctly, or under the right conditions. diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 6280c9080ab..975157ff917 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -170,7 +170,7 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM ALL)** - **Protect variable** Optional. If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** Optional. If selected, the variable's **Value** is masked @@ -194,7 +194,7 @@ Prerequisite: To add an instance variable: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD** and expand the **Variables** section. 1. Select **Add variable** and fill in the details: @@ -371,7 +371,7 @@ For example: ```yaml variables: - SITE_URL: "https://example.gitlab.com" + SITE_URL: "https://gitlab.example.com" job: script: diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 6acb254e76f..b1a0fca9069 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -68,9 +68,9 @@ as it can cause the pipeline to behave unexpectedly. | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | -| `CI_JOB_JWT` (Deprecated) | 12.10 | all | A RS256 JSON web token to authenticate with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | -| `CI_JOB_JWT_V1` (Deprecated) | 14.6 | all | The same value as `CI_JOB_JWT`. [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | -| `CI_JOB_JWT_V2` (Deprecated) | 14.6 | all | A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. The `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 16.5. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT` (Deprecated) | 12.10 | all | A RS256 JSON web token to authenticate with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT_V1` (Deprecated) | 14.6 | all | The same value as `CI_JOB_JWT`. [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | +| `CI_JOB_JWT_V2` (Deprecated) | 14.6 | all | A newly formatted RS256 JSON web token to increase compatibility. Similar to `CI_JOB_JWT`, except the issuer (`iss`) claim is changed from `gitlab.com` to `https://gitlab.com`, `sub` has changed from `job_id` to a string that contains the project path, and an `aud` claim is added. The `aud` field is a constant value. Trusting JWTs in multiple relying parties can lead to [one RP sending a JWT to another one and acting maliciously as a job](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72555#note_769112331). [Deprecated in GitLab 15.9](../../update/deprecations.md#old-versions-of-json-web-tokens-are-deprecated) and scheduled to be removed in GitLab 17.0. Use [ID tokens](../yaml/index.md#id_tokens) instead. | | `CI_JOB_MANUAL` | 8.12 | all | Only available if the job was started manually. `true` when available. | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job. | | `CI_JOB_NAME_SLUG` | 15.4 | all | `CI_JOB_NAME_SLUG` in lowercase, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in paths. | @@ -165,13 +165,15 @@ These variables are available when: | `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request. For example `namespace/awesome-project`. | | `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | | `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | +| `CI_MERGE_REQUEST_SQUASH_ON_MERGE` | 16.4 | all | `true` when the [squash on merge](../../user/project/merge_requests/squash_and_merge.md) option is set. | | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_PROTECTED` | 16.4 | all | `true` when the source branch of the merge request is [protected](../../user/project/protected_branches.md). | | `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | | `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | | `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED` | 15.2 | all | The protection status for the target branch of the merge request. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED` | 15.2 | all | `true` when the target branch of the merge request is [protected](../../user/project/protected_branches.md). | | `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | | `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | | `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 52edeb67f0e..d25f9801f5b 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -22,6 +22,8 @@ There are two places defined variables can be used. On the: ### `.gitlab-ci.yml` file +> Support for `CI_ENVIRONMENT_*` variables except `CI_ENVIRONMENT_SLUG` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128694) in GitLab 16.4. + | Definition | Can be expanded? | Expansion place | Description | |:----------------------------------------------------------------------|:-----------------|:-----------------------|:------------| | [`after_script`](../yaml/index.md#after_script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | @@ -32,15 +34,15 @@ There are two places defined variables can be used. On the: | [`environment:name`](../yaml/index.md#environmentname) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- `CI_ENVIRONMENT_*` variables.<br/>- [Persisted variables](#persisted-variables). | | [`environment:url`](../yaml/index.md#environmenturl) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in the GitLab Runner `config.toml` and variables created in the job's `script`. | | [`environment:auto_stop_in`](../yaml/index.md#environmentauto_stop_in)| yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab.<br/><br/> The value of the variable being substituted should be a period of time in a human readable natural language form. See [possible inputs](../yaml/index.md#environmentauto_stop_in) for more information.| -| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`except:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`id_tokens:aud`](../yaml/index.md#id_tokens) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. Variable expansion [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/414293) in GitLab 16.1. | | [`image`](../yaml/index.md#image) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`include`](../yaml/index.md#include) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. <br/><br/>See [Use variables with include](../yaml/includes.md#use-variables-with-include) for more information on supported variables. | -| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`resource_group`](../yaml/index.md#resource_group) | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/>- `CI_ENVIRONMENT_URL`<br/>- [Persisted variables](#persisted-variables). | | [`rules:changes`](../yaml/index.md#ruleschanges) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | | [`rules:exists`](../yaml/index.md#rulesexists) | yes | GitLab | The variable expansion is made by the [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism) in GitLab. | -| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_*` variables, except `CI_ENVIRONMENT_NAME` which is supported.<br/>- [Persisted variables](#persisted-variables). | +| [`rules:if`](../yaml/index.md#rulesif) | no | Not applicable | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- `CI_ENVIRONMENT_SLUG` variable.<br/>- [Persisted variables](#persisted-variables). | | [`script`](../yaml/index.md#script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | | [`services:name`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`services`](../yaml/index.md#services) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index fa7e941ffe5..e931a8b3b4e 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -40,6 +40,55 @@ GitLab can display the results of one or more reports in the merge request For more information, see [Accessibility testing](../testing/accessibility_testing.md). +## `artifacts:reports:annotations` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38337) in GitLab 16.3. + +The `annotations` report is used to attach auxiliary data to a job. + +An annotations report is a JSON file with annotation sections. Each annotation +section can have any desired name and can have any number of annotations of the +same or differing types. + +Each annotation is a single key (the annotation type), containing the subkeys with +the data for that annotation. + +### Annotation types + +#### `external_link` + +An `external_link` annotation can be attached to a job to add a link to the job +output page. The value of an `external_link` annotation is an object with the +following keys: + +| Key | Description | +|---------|----------------------------------------------------| +| `label` | The human-readable label associated with the link. | +| `url` | The URL pointed to by the link. | + +### Example report + +The following is an example of what a job annotations report might look like: + +```json +{ + "my_annotation_section_1": [ + { + "external_link": { + "label": "URL 1", + "url": "https://url1.example.com/" + } + }, + { + "external_link": { + "label": "URL 2", + "url": "https://url2.example.com/" + } + } + ] +} +``` + ## `artifacts:reports:api_fuzzing` **(ULTIMATE ALL)** > - Introduced in GitLab 13.4. @@ -95,7 +144,7 @@ GitLab can display the results of coverage report in the merge request ## `artifacts:reports:codequality` > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -> - [Added support for multiple reports in diff annotations and full pipeline report](https://gitlab.com/gitlab-org/gitlab/-/issues/9014) in 15.7. +> - Support for multiple reports in diff annotations and full pipeline report [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9014) in 15.7. The `codequality` report collects [code quality issues](../testing/code_quality.md). The collected code quality report uploads to GitLab as an artifact. @@ -313,7 +362,7 @@ The collected Secret Detection report is uploaded to GitLab. GitLab can display the results of one or more reports in: - The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). -- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). +- The [pipeline security tab](../../user/application_security/index.md#pipeline-security-tab). - The [security dashboard](../../user/application_security/security_dashboard/index.md). ## `artifacts:reports:terraform` @@ -325,6 +374,6 @@ The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing re The collected Terraform plan report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in the merge request -[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). +[Terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 79eb42fd781..f0e5a475838 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -501,6 +501,44 @@ In this example, GitLab checks for the existence of `test-file.yml` in `my-group not the current project. Follow [issue 386040](https://gitlab.com/gitlab-org/gitlab/-/issues/386040) for information about work to improve this behavior. +### `include` with `rules:changes` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342209) in GitLab 16.4. + +Use [`rules:changes`](index.md#ruleschanges) to conditionally include other configuration files +based on changed files. For example: + +```yaml +include: + - local: builds1.yml + rules: + - changes: + - Dockerfile + - local: builds2.yml + rules: + - changes: + paths: + - Dockerfile + compare_to: 'refs/heads/branch1' + when: always + - local: builds3.yml + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: + paths: + - Dockerfile + +test: + stage: test + script: exit 0 +``` + +In this example: + +- `builds1.yml` is included when `Dockerfile` has changed. +- `builds2.yml` is included when `Dockerfile` has changed relative to `refs/heads/branch1`. +- `builds3.yml` is included when `Dockerfile` has changed and the pipeline source is a merge request event. + ## Use `include:local` with wildcard file paths > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. @@ -528,144 +566,6 @@ When the pipeline runs, GitLab: include: 'configs/**/*.yml' ``` -## Define inputs for configuration added with `include` (Beta) - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391331) in GitLab 15.11 as a Beta feature. - -FLAG: -`spec` and `with` are experimental [Open Beta features](../../policy/experiment-beta-support.md#beta) -and subject to change without notice. - -### Define input parameters with `spec:inputs` - -Use `spec:inputs` to define input parameters for CI/CD configuration intended to be added -to a pipeline with `include`. Use [`include:inputs`](#set-input-parameter-values-with-includeinputs) -to define the values to use when the pipeline runs. - -The specs must be declared at the top of the configuration file, in a header section. -Separate the header from the rest of the configuration with `---`. - -Use the interpolation format `$[[ input.input-id ]]` to reference the values outside of the header section. -The inputs are evaluated and interpolated once, when the configuration is fetched -during pipeline creation, but before the configuration is merged with the contents of the `.gitlab-ci.yml`. - -```yaml -spec: - inputs: - environment: - job-stage: ---- - -scan-website: - stage: $[[ inputs.job-stage ]] - script: ./scan-website $[[ inputs.environment ]] -``` - -When using `spec:inputs`: - -- Defined inputs are mandatory by default. -- Inputs can be made optional by specifying a `default`. Use `default: null` to have no default value. -- A string containing an interpolation block must not exceed 1 MB. -- The string inside an interpolation block must not exceed 1 KB. - -For example, a `custom_configuration.yml`: - -```yaml -spec: - inputs: - website: - user: - default: 'test-user' - flags: - default: null ---- - -# The pipeline configuration would follow... -``` - -In this example: - -- `website` is mandatory and must be defined. -- `user` is optional. If not defined, the value is `test-user`. -- `flags` is optional. If not defined, it has no value. - -### Specify functions to manipulate input values - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409462) in GitLab 16.3. - -You can specify predefined functions in the interpolation block to manipulate the input value. -The format supported is the following: - -```yaml -$[[ input.input-id | <function1> | <function2> | ... <functionN> ]] -``` - -Details: - -- Only [predefined interpolation functions](#predefined-interpolation-functions) are permitted. -- A maximum of 3 functions may be specified in a single interpolation block. -- The functions are executed in the sequence they are specified. - -```yaml -spec: - inputs: - test: - default: '0123456789' ---- - -test-job: - script: echo $[[ inputs.test | truncate(1,3) ]] -``` - -In this example: - -- The function [`truncate`](#truncate) applies to the value of `inputs.test`. -- Assuming the value of `inputs.test` is `0123456789`, then the output of `script` would be `echo 123`. - -### Predefined interpolation functions - -#### Truncate - -Use `truncate` to shorten the interpolated value. For example: - -- `truncate(<offset>,<length>)` - -| Name | Type | Description | -| ---- | ---- | ----------- | -| `offset` | Integer | Number of characters to offset by. | -| `length` | Integer | Number of characters to return after the offset. | - -Example: - -```yaml -$[[ inputs.test | truncate(3,5) ]] -``` - -Assuming the value of `inputs.test` is `0123456789`, then the output would be `34567`. - -### Set input parameter values with `include:inputs` - -> `include:with` [renamed to `include:inputs`](https://gitlab.com/gitlab-org/gitlab/-/issues/406780) in GitLab 16.0. - -Use `include:inputs` to set the values for the parameters when the included configuration -is added to the pipeline. - -For example, to include a `custom_configuration.yml` that has the same specs -as the [example above](#define-input-parameters-with-specinputs): - -```yaml -include: - - local: 'custom_configuration.yml' - inputs: - website: "My website" -``` - -In this example: - -- `website` has a value of `My website` for the included configuration. -- `user` has a value of `test-user`, because that is the default when not specified. -- `flags` has no value, because it is optional and has no default when not specified. - ## Troubleshooting ### `Maximum of 150 nested includes are allowed!` error diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index c4f7e6d6e01..6275d4293ea 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -9,9 +9,11 @@ type: reference This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. -- For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). -- For a collection of examples, see [GitLab CI/CD Examples](../examples/index.md). -- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +- For a collection of examples, see [GitLab CI/CD examples](../examples/index.md). +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the + [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +- To create your own `.gitlab-ci.yml` file, try a tutorial that demonstrates a + [simple](../quick_start/index.md) or [complex](../quick_start/tutorial.md) pipeline. When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. @@ -75,8 +77,11 @@ or import additional pipeline configuration. ### `default` -You can set global defaults for some keywords. Jobs that do not define one or more -of the listed keywords use the value defined in the `default` section. +> Support for `id_tokens` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/419750) in GitLab 16.4. + +You can set global defaults for some keywords. Each default keyword is copied to every job +that doesn't already have it defined. If the job already has a keyword defined, that default +is not used. **Keyword type**: Global keyword. @@ -87,6 +92,7 @@ of the listed keywords use the value defined in the `default` section. - [`before_script`](#before_script) - [`cache`](#cache) - [`hooks`](#hooks) +- [`id_tokens`](#id_tokens) - [`image`](#image) - [`interruptible`](#interruptible) - [`retry`](#retry) @@ -99,6 +105,7 @@ of the listed keywords use the value defined in the `default` section. ```yaml default: image: ruby:3.0 + retry: 2 rspec: script: bundle exec rspec @@ -108,16 +115,17 @@ rspec 2.7: script: bundle exec rspec ``` -In this example, `ruby:3.0` is the default `image` value for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image` section: +In this example: + +- `image: ruby:3.0` and `retry: 2` are the default keywords for all jobs in the pipeline. +- The `rspec` job does not have `image` or `retry` defined, so it uses the defaults of + `image: ruby:3.0` and `retry: 2`. +- The `rspec 2.7` job does not have `retry` defined, but it does have `image` explictly defined. + It uses the default `retry: 2`, but ignores the default `image` and uses the `image: ruby:2.7` + defined in the job. **Additional details**: -- When the pipeline is created, each default is copied to all jobs that don't have - that keyword defined. -- If a job already has one of the keywords configured, the configuration in the job - takes precedence and is not replaced by the default. - Control inheritance of default keywords in jobs with [`inherit:default`](#inheritdefault). ### `include` @@ -267,6 +275,11 @@ include: - When you include a YAML file from another private project, the user running the pipeline must be a member of both projects and have the appropriate permissions to run pipelines. A `not found or access denied` error may be displayed if the user does not have access to any of the included files. +- Be careful when including another project's CI/CD configuration file. No pipelines or notifications trigger when CI/CD configuration files change. + From a security perspective, this is similar to pulling a third-party dependency. For the `ref`, consider: + - Using a specific SHA hash, which should be the most stable option. + - Applying both [protected branch](../../user/project/protected_branches.md) and [protected tag](../../user/project/protected_tags.md#prevent-tag-creation-with-the-same-name-as-branches) rules to + the `ref` in the other project. Protected tags and branches are more likely to pass through change management before changing. #### `include:remote` @@ -293,9 +306,11 @@ include: - All [nested includes](includes.md#use-nested-includes) are executed without context as a public user, so you can only include public projects or templates. No variables are available in the `include` section of nested includes. -- Be careful when including a remote CI/CD configuration file. No pipelines or notifications - trigger when external CI/CD configuration files change. From a security perspective, - this is similar to pulling a third-party dependency. +- Be careful when including another project's CI/CD configuration file. No pipelines or notifications trigger + when the other project's files change. From a security perspective, this is similar to + pulling a third-party dependency. If you link to another GitLab project you own, consider the use of both + [protected branches](../../user/project/protected_branches.md) and [protected tags](../../user/project/protected_tags.md#prevent-tag-creation-with-the-same-name-as-branches) + to enforce change management rules. #### `include:template` @@ -1518,27 +1533,28 @@ is extracted from the job output. The coverage is shown in the UI if at least on line in the job output matches the regular expression. To extract the code coverage value from the match, GitLab uses -this smaller regular expression: `\d+(\.\d+)?`. +this smaller regular expression: `\d+(?:\.\d+)?`. **Possible inputs**: -- A regular expression. Must start and end with `/`. Must match the coverage number. +- An RE2 regular expression. Must start and end with `/`. Must match the coverage number. May match surrounding text as well, so you don't need to use a regular expression character group to capture the exact number. + Because it uses RE2 syntax, all groups must be non-capturing. **Example of `coverage`**: ```yaml job1: script: rspec - coverage: '/Code coverage: \d+\.\d+/' + coverage: '/Code coverage: \d+(?:\.\d+)?/' ``` In this example: 1. GitLab checks the job log for a match with the regular expression. A line like `Code coverage: 67.89% of lines covered` would match. -1. GitLab then checks the matched fragment to find a match to `\d+(\.\d+)?`. +1. GitLab then checks the matched fragment to find a match to `\d+(?:\.\d+)?`. The sample matching line above gives a code coverage of `67.89`. **Additional details**: @@ -1997,12 +2013,19 @@ at certain stages of job execution, like before retrieving the Git repository. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/381840) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. Use `hooks:pre_get_sources_script` to specify a list of commands to execute on the runner -before retrieving the Git repository and any submodules. You can use it -to adjust the Git client configuration first, for example. +before cloning the Git repository and any submodules. +You can use it for example to: -**Related topics**: +- Adjust the [Git configuration](../troubleshooting.md#get_sources-job-section-fails-because-of-an-http2-problem). +- Export [tracing variables](../../topics/git/useful_git_commands.md). -- [GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +**Possible inputs**: An array including: + +- Single line commands. +- Long commands [split over multiple lines](script.md#split-long-commands). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). + +CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). **Example of `hooks:pre_get_sources_script`**: @@ -2014,6 +2037,10 @@ job1: script: echo 'hello job1 script' ``` +**Related topics**: + +- [GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) + ### `id_tokens` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.7. @@ -2717,7 +2744,7 @@ linux:rspec: parallel: matrix: - PROVIDER: aws - - STACK: app1 + STACK: app1 script: echo "Running rspec on linux..." ``` @@ -3379,6 +3406,8 @@ only one of the jobs starts. The other jobs wait until the `resource_group` is f Resource groups behave similar to semaphores in other programming languages. +You can choose a [process mode](../resource_groups/index.md#process-modes) to strategically control the job concurrency for your deployment preferences. The default process mode is `unordered`. To change the process mode of a resource group, use the [API](../../api/resource_groups.md#edit-an-existing-resource-group) to send a request to edit an existing resource group. + You can define multiple resource groups per environment. For example, when deploying to physical devices, you might have multiple physical devices. Each device can be deployed to, but only one deployment can occur per device at any given time. @@ -3598,10 +3627,10 @@ to specific files. WARNING: You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, manual pipelines, -and so on do **not** have a Git `push` event associated with them. A `rules: changes` job -is **always** added to those pipelines if there is no `if` that limits the job to -branch or merge request pipelines. +evaluates to true for new branch pipelines or when there is no Git `push` event. Pipelines like tag pipelines, +scheduled pipelines, and manual pipelines, all do **not** +have a Git `push` event associated with them. In these cases, use [`rules: changes: compare_to`](#ruleschangescompare_to) +to specify the branch to compare against. **Keyword type**: Job keyword. You can use it only as part of a job. @@ -4368,6 +4397,9 @@ test: ### `trigger` +> - Support for `resource_group` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) support for `resource_group` in GitLab 13.9. +> - Support for `environment` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369061) in GitLab 16.4. + Use `trigger` to declare that a job is a "trigger job" which starts a [downstream pipeline](../pipelines/downstream_pipelines.md) that is either: @@ -4386,6 +4418,8 @@ The keywords available for use in trigger jobs are: - [`trigger`](#trigger). - [`variables`](#variables). - [`when`](#when) (only with a value of `on_success`, `on_failure`, or `always`). +- [`resource_group`](#resource_group). +- [`environment`](#environment). **Keyword type**: Job keyword. You can use it only as part of a job. @@ -4417,6 +4451,7 @@ trigger-multi-project-pipeline: to forward these variables to downstream pipelines. - [Job-level persisted variables](../variables/where_variables_can_be_used.md#persisted-variables) are not available in trigger jobs. +- Environment variables [defined in the runner's `config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) are not available to trigger jobs and are not passed to downstream pipelines. **Related topics**: diff --git a/doc/ci/yaml/inputs.md b/doc/ci/yaml/inputs.md new file mode 100644 index 00000000000..1af53d666ce --- /dev/null +++ b/doc/ci/yaml/inputs.md @@ -0,0 +1,174 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Define inputs for configuration added with `include` **(FREE ALL BETA)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/391331) in GitLab 15.11 as a Beta feature. + +FLAG: +`spec` and `inputs` are experimental [Open Beta features](../../policy/experiment-beta-support.md#beta) +and subject to change without notice. + +## Define input parameters with `spec:inputs` + +Use `spec:inputs` to define input parameters for CI/CD configuration intended to be added +to a pipeline with `include`. Use [`include:inputs`](#set-input-parameter-values-with-includeinputs) +to define the values to use when the pipeline runs. + +The specs must be declared at the top of the configuration file, in a header section. +Separate the header from the rest of the configuration with `---`. + +Use the interpolation format `$[[ input.input-id ]]` to reference the values outside of the header section. +The inputs are evaluated and interpolated once, when the configuration is fetched +during pipeline creation, but before the configuration is merged with the contents of the `.gitlab-ci.yml`. + +```yaml +spec: + inputs: + environment: + job-stage: +--- + +scan-website: + stage: $[[ inputs.job-stage ]] + script: ./scan-website $[[ inputs.environment ]] +``` + +When using `spec:inputs`: + +- Defined inputs are mandatory by default. +- Inputs can be made optional by specifying a `default`. Use `default: null` to have no default value. +- A string containing an interpolation block must not exceed 1 MB. +- The string inside an interpolation block must not exceed 1 KB. + +For example, a `custom_configuration.yml`: + +```yaml +spec: + inputs: + website: + user: + default: 'test-user' + flags: + default: null +--- + +# The pipeline configuration would follow... +``` + +In this example: + +- `website` is mandatory and must be defined. +- `user` is optional. If not defined, the value is `test-user`. +- `flags` is optional. If not defined, it has no value. + +## Set input parameter values with `include:inputs` + +> `include:with` [renamed to `include:inputs`](https://gitlab.com/gitlab-org/gitlab/-/issues/406780) in GitLab 16.0. + +Use `include:inputs` to set the values for the parameters when the included configuration +is added to the pipeline. + +For example, to include a `custom_configuration.yml` that has the same specs +as the [example above](#define-input-parameters-with-specinputs): + +```yaml +include: + - local: 'custom_configuration.yml' + inputs: + website: "My website" +``` + +In this example: + +- `website` has a value of `My website` for the included configuration. +- `user` has a value of `test-user`, because that is the default when not specified. +- `flags` has no value, because it is optional and has no default when not specified. + +### Use `include:inputs` with multiple files + +`inputs` must be specified separately for each included file. For example: + +```yaml +include: + - component: gitlab.com/org/my-component@1.0 + inputs: + stage: my-stage + - local: path/to/file.yml + inputs: + stage: my-stage +``` + +You can also include the same file multiple times, with different inputs. +For example: + +```yaml +include: + - local: path/to/my-super-linter.yml + inputs: + type: docs + job-name: lint-docs + lint-path: "doc/" + - local: path/to/my-super-linter.yml + inputs: + type: yaml + job-name: lint-yaml + lint-path: "data/yaml/" +``` + +## Specify functions to manipulate input values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409462) in GitLab 16.3. + +You can specify predefined functions in the interpolation block to manipulate the input value. +The format supported is the following: + +```yaml +$[[ input.input-id | <function1> | <function2> | ... <functionN> ]] +``` + +Details: + +- Only [predefined interpolation functions](#predefined-interpolation-functions) are permitted. +- A maximum of 3 functions may be specified in a single interpolation block. +- The functions are executed in the sequence they are specified. + +```yaml +spec: + inputs: + test: + default: '0123456789' +--- + +test-job: + script: echo $[[ inputs.test | truncate(1,3) ]] +``` + +In this example: + +- The function [`truncate`](#truncate) applies to the value of `inputs.test`. +- Assuming the value of `inputs.test` is `0123456789`, then the output of `script` would be `echo 123`. + +### Predefined interpolation functions + +#### `truncate` + +Use `truncate` to shorten the interpolated value. For example: + +- `truncate(<offset>,<length>)` + +| Name | Type | Description | +| ---- | ---- | ----------- | +| `offset` | Integer | Number of characters to offset by. | +| `length` | Integer | Number of characters to return after the offset. | + +Example: + +```yaml +$[[ inputs.test | truncate(3,5) ]] +``` + +Assuming the value of `inputs.test` is `0123456789`, then the output would be `34567`. diff --git a/doc/ci/yaml/signing_examples.md b/doc/ci/yaml/signing_examples.md index 72e007a749f..e97ade891c4 100644 --- a/doc/ci/yaml/signing_examples.md +++ b/doc/ci/yaml/signing_examples.md @@ -7,12 +7,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Use Sigstore for keyless signing and verification **(FREE SAAS)** The [Sigstore](https://www.sigstore.dev/) project provides a CLI called -[Cosign](https://docs.sigstore.dev/cosign/overview/) which can be used for keyless signing of container images built +[Cosign](https://docs.sigstore.dev/signing/quickstart/) which can be used for keyless signing of container images built with GitLab CI/CD. Keyless signing has many advantages, including eliminating the need to manage, safeguard, and rotate a private key. Cosign requests a short-lived key pair to use for signing, records it on a certificate transparency log, and then discards it. The key is generated through a token obtained from the GitLab server using the OIDC identity of the user who ran the pipeline. This token includes unique claims that certify the token was generated by a CI/CD pipeline. To learn more, -see Cosign [documentation](https://docs.sigstore.dev/cosign/overview/#example-working-with-containers) on keyless signatures. +see Cosign [documentation](https://docs.sigstore.dev/signing/quickstart/#example-working-with-containers) on keyless signatures. For details on the mapping between GitLab OIDC claims and Fulcio certificate extensions, see the GitLab column of [Mapping OIDC token claims to Fulcio OIDs](https://github.com/sigstore/fulcio/blob/main/docs/oid-info.md#mapping-oidc-token-claims-to-fulcio-oids). @@ -30,17 +30,21 @@ You can use Cosign to sign and verify container images and build artifacts. - You must use a version of Cosign that is `>= 2.0.1`. +**Limitations** + +- The `id_tokens` portion of the CI/CD config file must be located in the project that is being built and signed. AutoDevOps, CI files included from another repository, and child pipelines are not supported. Work to remove this limitation is being tracked in [issue 411317](https://gitlab.com/gitlab-org/gitlab/-/issues/411317). + **Best practices**: - Build and sign an image/artifact in the same job to prevent it from being tampered with before it is signed. - When signing container images, sign the digest (which is immutable) instead of the tag. GitLab [ID tokens](../secrets/id_token_authentication.md#id-tokens) can be used by Cosign for -[keyless signing](https://docs.sigstore.dev/cosign/overview/). The token must have +[keyless signing](https://docs.sigstore.dev/signing/quickstart/). The token must have `sigstore` set as the [`aud`](../secrets/id_token_authentication.md#token-payload) claim. The token can be used by Cosign automatically when it is set in the `SIGSTORE_ID_TOKEN` environment variable. -To learn more about how to install Cosign, see [Cosign Installation documentation](https://docs.sigstore.dev/cosign/installation/). +To learn more about how to install Cosign, see [Cosign Installation documentation](https://docs.sigstore.dev/system_config/installation/). ### Signing @@ -49,7 +53,7 @@ To learn more about how to install Cosign, see [Cosign Installation documentatio The example below demonstrates how to sign a container image in GitLab CI. The signature is automatically stored in the same container repository as the image. -To learn more about signing containers, see [Cosign Signing Containers documentation](https://docs.sigstore.dev/cosign/signing_with_containers/). +To learn more about signing containers, see [Cosign Signing Containers documentation](https://docs.sigstore.dev/signing/signing_with_containers/). ```yaml build_and_sign_image: @@ -77,7 +81,7 @@ build_and_sign_image: The example below demonstrates how to sign a build artifact in GitLab CI. You should save the `cosign.bundle` file produced by `cosign sign-blob`, which is used for signature verification. -To learn more about signing artifacts, see [Cosign Signing Blobs documentation](https://docs.sigstore.dev/cosign/signing_with_blobs/#keyless-signing-of-blobs-and-files). +To learn more about signing artifacts, see [Cosign Signing Blobs documentation](https://docs.sigstore.dev/signing/signing_with_blobs/). ```yaml build_and_sign_artifact: @@ -109,7 +113,7 @@ build_and_sign_artifact: | `--certificate-oidc-issuer` | The GitLab instance URL where the image/artifact was signed. For example, `https://gitlab.com`. | | `--bundle` | The `bundle` file produced by `cosign sign-blob`. Only used for verifying build artifacts. | -To learn more about verifying signed images/artifacts, see [Cosign Verifying documentation](https://docs.sigstore.dev/cosign/verify/#keyless-verification-using-openid-connect). +To learn more about verifying signed images/artifacts, see [Cosign Verifying documentation](https://docs.sigstore.dev/verifying/verify/). #### Container images @@ -149,7 +153,7 @@ You can use Sigstore and npm, together with GitLab CI/CD, to digitally sign buil ### About npm provenance -[npm CLI](https://docs.npmjs.com/cli) allows package maintainers to provide users with provenance attestations. Using npm +[npm CLI](https://docs.npmjs.com/cli/) allows package maintainers to provide users with provenance attestations. Using npm CLI provenance generation allows users to trust and verify that the package they are downloading and using is from you and the build system that built it. diff --git a/doc/cloud_seed/index.md b/doc/cloud_seed/index.md index a5067215ebb..70166b2d09f 100644 --- a/doc/cloud_seed/index.md +++ b/doc/cloud_seed/index.md @@ -21,8 +21,8 @@ services on a hyper-cloud based on a foundation of Terraform and infrastructure- We believe that it should be **trivial** to deploy web applications (and other workloads) from GitLab to major cloud providers. -To support this effort, Cloud Seed makes it simple and intuitive to consume appropriate Google Cloud services -within GitLab. +To support this effort, Cloud Seed makes it straightforward and intuitive to consume appropriate Google Cloud services +in GitLab. ## Why Google Cloud @@ -114,11 +114,11 @@ the underlying Google Cloud service that is used to provision the database insta The following databases and versions are supported: -- PostgreSQL: 14, 13, 12, 11, 10 and 9.6 +- PostgreSQL: 14, 13, 12, 11, 10, and 9.6 - MySQL: 8.0, 5.7 and 5.6 - SQL Server - - 2019: Standard, Enterprise, Express and Web - - 2017: Standard, Enterprise, Express and Web + - 2019: Standard, Enterprise, Express, and Web + - 2017: Standard, Enterprise, Express, and Web Google Cloud pricing applies. Please refer to the [Cloud SQL pricing page](https://cloud.google.com/sql/pricing). @@ -129,7 +129,7 @@ Google Cloud pricing applies. Please refer to the [Cloud SQL pricing page](https ### Create a database instance -From the `Project :: Infrastructure :: Google Cloud` page, select the **Database** tab. Here you will find three +From the `Project :: Infrastructure :: Google Cloud` page, select the **Database** tab. Here you find three buttons to create Postgres, MySQL, and SQL Server database instances. The database instance creation form has fields for GCP project, Git ref (branch or tag), database version and @@ -145,7 +145,7 @@ Successful creation of the database instance triggers a background worker to per ### Connect to the database -Once the database instance setup is complete, the database connection details are available as project variables. These +After the database instance setup is complete, the database connection details are available as project variables. These can be managed through the `Project :: Settings :: CI` page and are made available to pipeline executing in the appropriate environment. @@ -158,11 +158,9 @@ Console. Select an instance to view the details and manage the instance. There are several ways you can contribute to Cloud Seed: -- [Become a Cloud Seed user](https://docs.google.com/forms/d/e/1FAIpQLSeJPtFE8Vpqs_YTAKkFK42p5mO9zIYA2jr_PiP2h32cs8R39Q/viewform) - in GitLab - and [share feedback](https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/feedback/-/issues/new?template=general_feedback). +- Use Cloud Seed and [share feedback](https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/feedback/-/issues/new?template=general_feedback). - If you are familiar with Ruby on Rails or Vue.js, consider [contributing to GitLab](../development/contributing/index.md) as a developer. - - Much of Cloud Seed is an internal module within the GitLab codebase. + - Much of Cloud Seed is an internal module in the GitLab codebase. - If you are familiar with GitLab pipelines, consider contributing to the [Cloud Seed Library](https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/library) project. diff --git a/doc/user/project/web_ide_beta/index.md b/doc/development/activitypub/actor.md index a4c733be376..1d10e421df7 100644 --- a/doc/user/project/web_ide_beta/index.md +++ b/doc/development/activitypub/actor.md @@ -1,11 +1,11 @@ --- -redirect_to: '../web_ide/index.md' -remove_date: '2023-08-22' +redirect_to: 'actors/index.md' +remove_date: '2023-12-08' --- -This document was moved to [another location](../web_ide/index.md). +This document was moved to [another location](actors/index.md). -<!-- This redirect file can be deleted after <2023-08-22>. --> +<!-- This redirect file can be deleted after <2023-12-08>. --> <!-- Redirects that point to other docs in the same project expire in three months. --> <!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/activitypub/actors/group.md b/doc/development/activitypub/actors/group.md new file mode 100644 index 00000000000..dad02298170 --- /dev/null +++ b/doc/development/activitypub/actors/group.md @@ -0,0 +1,205 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Activities for group actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## Profile + +```javascript +{ + "@context": "https://www.w3.org/ns/activitystreams", + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "summary": GROUP_DESCRIPTION, + "url": GROUP_URL, + "outbox": GROUP_OUTBOX_URL, + "inbox": null, +} +``` + +## Outbox + +The various activities for a group are: + +- [The group was created](#the-group-was-created). +- All project activities for projects in that group, and its subgroups. +- [A user joined the group](#a-user-joined-the-group). +- [A user left the group](#a-user-left-the-group). +- [The group was deleted](#the-group-was-deleted). +- [A subgroup was created](#a-subgroup-was-created). +- [A subgroup was deleted](#a-subgroup-was-deleted). + +### The group was created + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Create", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + } +} +``` + +### A user joined the group + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Join", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + }, +} +``` + +### A user left the group + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Leave", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + }, +} +``` + +### The group was deleted + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Delete", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + } +} +``` + +### A subgroup was created + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Create", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + "context": { + "id": PARENT_GROUP_URL, + "type": "Group", + "name": PARENT_GROUP_NAME, + "url": PARENT_GROUP_URL, + } + } +} +``` + +### A subgroup was deleted + +```javascript +{ + "id": GROUP_OUTBOX_URL#event_id, + "type": "Delete", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": GROUP_URL, + "type": "Group", + "name": GROUP_NAME, + "url": GROUP_URL, + "context": { + "id": PARENT_GROUP_URL, + "type": "Group", + "name": PARENT_GROUP_NAME, + "url": PARENT_GROUP_URL, + } + } +} +``` diff --git a/doc/development/activitypub/actors/index.md b/doc/development/activitypub/actors/index.md new file mode 100644 index 00000000000..032cb26587a --- /dev/null +++ b/doc/development/activitypub/actors/index.md @@ -0,0 +1,148 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Implement an ActivityPub actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +ActivityPub is based on three standard documents: + +- [ActivityPub](https://www.w3.org/TR/activitypub/) defines the HTTP + requests happening to implement federation. +- [ActivityStreams](https://www.w3.org/TR/activitystreams-core/) defines the + format of the JSON messages exchanged by the users of the protocol. +- [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) + defines the various messages recognized by default. + +The first one is typically handled by controllers, while the two others are +related to what happen in serializers. + +To implement an ActivityPub actor, you must: + +- Implement the profile page of the resource. +- Implement the outbox page. +- Handle incoming requests on the inbox. + +All requests are made using +`application/ld+json; profile="https://www.w3.org/ns/activitystreams"` as `Accept` HTTP header. + +The actors we're implementing for the social features: + +- [Releases](releases.md) +- [Topics](topic.md) +- [Projects](project.md) +- [Groups](group.md) +- [Users](user.md) + +## Profile page + +Querying the profile page is used to retrieve: + +- General information about it, like name and description. +- URLs for the inbox and the outbox. + +To implement a profile page, create an ActivityStreams +serializer in `app/serializers/activity_pub/`, making your serializer +inherit from `ActivityStreamsSerializer`. See below in the serializers +section about the mandatory fields. + +To call your serializer in your controller: + +```ruby +opts = { + inbox: nil, + outbox: outbox_project_releases_url(project) +} + +render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) +``` + +- `outbox` is the endpoint where to find the activities feed for this +actor. +- `inbox` is where to POST to subscribe to the feed. Not yet implemented, so pass `nil`. + +## Outbox page + +The outbox is the list of activities for the resource. It's a feed for the +resource, and it allows ActivityPub clients to show public activities for +this actor without having yet subscribed to it. + +To implement an outbox page, create an ActivityStreams +serializer in `app/serializers/activity_pub/`, making your serializer +inherit from `ActivityStreamsSerializer`. See below in the serializers +section about the mandatory fields. + +You call your serializer in your controller like this: + +```ruby +serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) +render json: serializer.represent(releases) +``` + +This converts the response to an `OrderedCollection` +ActivityPub type, with all the correct fields. + +## Inbox + +Not yet implemented. + +The inbox is where the ActivityPub compatible third-parties makes their +requests, to subscribe to the actor or send it messages. + +## ActivityStreams serializers + +The serializers implement half the core of ActivityPub support: they're all +about [ActivityStreams](https://www.w3.org/TR/activitystreams-core/), the +message format used by ActivityPub. + +To leverage the features doing most of the formatting for you, your +serializer should inherit from `ActivityPub::ActivityStreamsSerializer`. + +To use it, call the `#represent` method. It requires you to provide +`inbox` and `outbox` options (as mentioned above) if it +is an actor profile page. You don't need those if your serializer +represents an object that is just meant to be embedded as part of actors, +like the object representing the contact information for a user. + +Each resource serialized (included other objects embedded in your +actor) must provide an `id` and a `type` field. + +`id` is a URL. It's meant to be a unique identifier for the resource, and +it must point to an existing page: ideally, an actor. Otherwise, you can +just reference the closest actor and use an anchor, like this: + +```plaintext +https://gitlab.com/user/project/-/releases#release-1 +``` + +`type` should be taken from ActivityStreams core vocabulary: + +- [Activity types](https://www.w3.org/TR/activitystreams-vocabulary/#activity-types) +- [Actor types](https://www.w3.org/TR/activitystreams-vocabulary/#actor-types) +- [Object types](https://www.w3.org/TR/activitystreams-vocabulary/#object-types) + +The properties you can use are all documented in +[the ActivityStreams vocabulary document](https://www.w3.org/TR/activitystreams-vocabulary). +Given the type you have chosen for your resource, find the +`properties` list, telling you all available properties, direct or +inherited. + +It's worth noting that Mastodon adds one more property, `preferredName`. +Mastodon expects it to be set on any actor, or that actor is not recognized by +Mastodon. diff --git a/doc/development/activitypub/actors/project.md b/doc/development/activitypub/actors/project.md new file mode 100644 index 00000000000..4f876b9e3fa --- /dev/null +++ b/doc/development/activitypub/actors/project.md @@ -0,0 +1,640 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Activities for project actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## Profile + +```javascript +{ + "@context": "https://www.w3.org/ns/activitystreams", + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + "outbox": PROJECT_OUTBOX_URL, + "inbox": null, +} +``` + +## Outbox + +For a project, we can map the events happening on the project activity +timeline on GitLab, when a user: + +- [Creates the repository](#user-creates-the-repository). +- [Pushes commits](#user-pushes-commits). +- [Pushes a tag](#user-pushes-a-tag). +- [Opens a merge request](#user-opens-a-merge-request). +- [Accepts a merge request](#user-accepts-a-merge-request). +- [Closes a merge request](#user-closes-a-merge-request). +- [Opens an issue](#user-opens-an-issue). +- [Closes an issue](#user-closes-an-issue). +- [Reopens an issue](#user-reopens-an-issue). +- [Comments on a merge request](#user-comments-on-a-merge-request). +- [Comments on an issue](#user-comments-on-an-issue). +- [Creates a wiki page](#user-creates-a-wiki-page). +- [Updates a wiki page](#user-updates-a-wiki-page). +- [Destroys a wiki page](#user-destroys-a-wiki-page). +- [Joins the project](#user-joins-the-project). +- [Leaves the project](#user-leaves-the-project). +- [Deletes the repository](#user-deletes-the-repository). + +There's also a Design tab in the project activities, but it's just empty in +all projects I follow and I don't see anything related to it in my projects +sidebar. Maybe it's a premium feature? If so, it's of no concern to us for +public following through ActivityPub. + +### User creates the repository + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Create", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + } +} +``` + +### User pushes commits + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Update", + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + "result": COMMITS_DIFF_URL, +} +``` + +### User pushes a tag + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Update", + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + "name": TAG_NAME, + "result": COMMIT_URL, +} +``` + +### User opens a merge request + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": MERGE_REQUEST_URL, + "type": "Application", + "name": MERGE_REQUEST_TITLE, + "url": MERGE_REQUEST_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, + "target": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +### User accepts a merge request + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Accept", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": MERGE_REQUEST_URL, + "type": "Application", + "name": MERGE_REQUEST_TITLE, + "url": MERGE_REQUEST_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, + "target": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +### User closes a merge request + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Remove", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": MERGE_REQUEST_URL, + "type": "Application", + "name": MERGE_REQUEST_TITLE, + "url": MERGE_REQUEST_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, + "origin": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +### User opens an issue + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": ISSUE_URL, + "type": "Page", + "name": ISSUE_TITLE, + "url": ISSUE_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + } + }, + "target": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + } +} +``` + +Why to add the project both as `object.context` and `target`? For multiple +consistency reasons: + +- The **Add** activity is more commonly used with a `target`. +- The **Remove** activity used to close the issue is more + commonly used with an `origin`. +- The **Update** activity used to reopen an issue specifies that + `target` and `origin` have no specific meaning, making `context` better + suited for that. +- We could use `context` only with **Update**, but merge requests + must be taken into consideration. + +Merge requests are very similar to issues, so we want their activities to +be similar. While the best type for issues is `page`, the type chosen for +merge request is `application`, both to distinguish it from issues and because +they contain code. + +To distinguish merge requests from projects (which are also `application`), +merge requests are an `application` with another `application` (the project) +as context. Given the merge request will have a `context` even with the **Add** +and **Remove** activities, the same is done with issues for consistency. + +An alternative that was considered, but dismissed: instead of **Add** for issues, +use **Create**. That would have allowed us to always use `context`, but +it creates more problems that it solves. **Accept** and **Reject** could work quite +well for closing merge requests, but what would we use to close issues? +**Delete** is incorrect, as the issue is not deleted, just closed. +Reopening the issue later would require an **Update** after a +**Delete**. + +Using **Create** for opening issues and **Remove** for closing +issues would be asymmetrical: + +- **Create** is mirrored by **Delete**. +- **Add** is mirrored by **Remove**. + +To minimize pain for those who will build on top of those resources, it's best +to duplicate the project information as `context` and `target` / `origin`. + +### User closes an issue + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Remove", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": ISSUE_URL, + "type": "Page", + "name": ISSUE_TITLE, + "url": ISSUE_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, + "origin": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +### User reopens an issue + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Update", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": ISSUE_URL, + "type": "Page", + "name": ISSUE_TITLE, + "url": ISSUE_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, +} +``` + +### User comments on a merge request + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": NOTE_URL, + "type": "Note", + "content": NOTE_NOTE, + }, + "target": { + "id": MERGE_REQUEST_URL, + "type": "Application", + "name": MERGE_REQUEST_TITLE, + "url": MERGE_REQUEST_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, +} +``` + +### User comments on an issue + +```javascript +{ + "id": PROJECT_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": NOTE_URL, + "type": "Note", + "content": NOTE_NOTE, + }, + "target": { + "id": ISSUE_URL, + "type": "Page", + "name": ISSUE_TITLE, + "url": ISSUE_URL, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, +} +``` + +### User creates a wiki page + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Create", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": WIKI_PAGE_URL, + "type": "Page", + "name": WIKI_PAGE_HUMAN_TITLE, + "url": WIKI_PAGE_URL, + } +} +``` + +### User updates a wiki page + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Update", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": WIKI_PAGE_URL, + "type": "Page", + "name": WIKI_PAGE_HUMAN_TITLE, + "url": WIKI_PAGE_URL, + } +} +``` + +### User destroys a wiki page + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Delete", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": WIKI_PAGE_URL, + "type": "Page", + "name": WIKI_PAGE_HUMAN_TITLE, + "url": WIKI_PAGE_URL, + } +} +``` + +### User joins the project + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "target": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +The GitLab project timeline does not mention who added a member to the +project, so this does the same. However, the **Add** activity requires an Actor. +For that reason, we use the same person as actor and object. + +In the **Members** page of a project contains a `source` attribute. +While there is sometimes mention of who added the user, this is used mainly +to distinguish if the user is a member attached to the project directly, or +through a group. It would not be a good "actor" (that would rather be an +`origin` for the membership). + +### User leaves the project + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Remove", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "target": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, +} +``` + +See [User joined the project](#user-joins-the-project). + +### User deletes the repository + +```javascript +{ + "id": PROJECT_OUTBOX_URL#event_id, + "type": "Delete", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + } +} +``` diff --git a/doc/development/activitypub/actors/releases.md b/doc/development/activitypub/actors/releases.md new file mode 100644 index 00000000000..009b98b6adf --- /dev/null +++ b/doc/development/activitypub/actors/releases.md @@ -0,0 +1,85 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Activities for following releases actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## Profile + +The profile is this actor is a bit different from other actors. We don't want to +show activities for a given release, but instead the releases for a given project. + +The profile endpoint is handled by `Projects::ReleasesController#index` +on the list of releases, and should reply with something like this: + +```javascript +{ + "@context": "https://www.w3.org/ns/activitystreams", + "id": PROJECT_RELEASES_URL, + "type": "Application", + "name": PROJECT_NAME + " releases", + "url": PROJECT_RELEASES_URL, + "content": PROJECT_DESCRIPTION, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + "outbox": PROJECT_RELEASES_OUTBOX_URL, + "inbox": null, +} +``` + +## Outbox + +The release actor is relatively simple: the only activity happening is the +**Create release** event. + +```javascript +{ + "id": PROJECT_RELEASES_OUTBOX_URL#release_id, + "type": "Create", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + }, + "object": { + "id": RELEASE_URL, + "type": "Application", + "name": RELEASE_TITLE, + "url": RELEASE_URL, + "content": RELEASE_DESCRIPTION, + "context": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "summary": PROJECT_DESCRIPTION, + "url": PROJECT_URL, + }, + }, +} +``` diff --git a/doc/development/activitypub/actors/topic.md b/doc/development/activitypub/actors/topic.md new file mode 100644 index 00000000000..f99a6e0569a --- /dev/null +++ b/doc/development/activitypub/actors/topic.md @@ -0,0 +1,91 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Activities for topic actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## Profile + +```javascript +{ + "@context": "https://www.w3.org/ns/activitystreams", + "id": TOPIC_URL, + "type": "Group", + "name": TOPIC_NAME, + "url": TOPIC_URL, + "summary": TOPIC_DESCRIPTION, + "outbox": TOPIC_OUTBOX_URL, + "inbox": null, +} +``` + +## Outbox + +Like the release actor, the topic specification is not complex. It generates an +activity only when a new project has been added to the given topic. + +```javascript +{ + "id": TOPIC_OUTBOX_URL#event_id, + "type": "Add", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "actor": { + "id": PROJECT_URL, + "type": "Application", + "name": PROJECT_NAME, + "url": PROJECT_URL, + }, + "object": { + "id": TOPIC_URL, + "type": "Group", + "name": TOPIC_NAME, + "url": TOPIC_URL, + }, + }, +} +``` + +## Possible difficulties + +There is hidden complexity here. + +The simpler way to build this endpoint is to take the projects associated +to a topic, and sort them by descending creation date. However, +if we do that, discrepancies will occur when implementing the +activity push part of the standard. + +Adding the project to a topic is not made at project creation time. It's +made when a project's topics are _edited_. That action can happen a very long time +after the project creation date. In that case, a push activity is +created and sent to federated instances when adding the topic to the +project. However, the list in the outbox endpoint that sorts projects by descending +creation date doesn't show the project, because it was created long ago. + +No special logic happens when a topic is added to a project, except: + +- Cleaning up the topic list. +- Creating the topic in database, if it doesn't exist yet. + +No event is generated. We should add such an event so the activity +push create an event, ideally in `Projects::UpdateService`. Then, the outbox endpoint +can list those events to be sure to match what was sent. When doing that, we should +verify that it doesn't affect other pages or endpoints dealing with events. diff --git a/doc/development/activitypub/actors/user.md b/doc/development/activitypub/actors/user.md new file mode 100644 index 00000000000..9fe4f8ec88e --- /dev/null +++ b/doc/development/activitypub/actors/user.md @@ -0,0 +1,47 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Activities for following user actor **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## Profile + +This activity is the first resource ActivityPub has in mind: + +```javascript +{ + "@context": "https://www.w3.org/ns/activitystreams", + "id": USER_PROFILE_URL, + "type": "Person", + "name": USER_NAME, + "url": USER_PROFILE_URL, + "outbox": USER_OUTBOX_URL, + "inbox": null, +} +``` + +## Outbox + +The user actor is special because it can be linked to all events happening on the platform. +It's a join of events on other resources: + +- All release activities. +- All project activities. +- All group activities. diff --git a/doc/development/activitypub/index.md b/doc/development/activitypub/index.md new file mode 100644 index 00000000000..d89f18080f0 --- /dev/null +++ b/doc/development/activitypub/index.md @@ -0,0 +1,216 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# ActivityPub **(EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023) in GitLab 16.5 [with two flags](../../administration/feature_flags.md) named `activity_pub` and `activity_pub_project`. Disabled by default. This feature is an [Experiment](../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../../administration/feature_flags.md) +named `activity_pub` and `activity_pub_project`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +Usage of ActivityPub in GitLab is governed by the +[GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). + +The goal of those documents is to provide an implementation path for adding +Fediverse capabilities to GitLab. + +This page describes the conceptual and high level point of view, while +sub-pages discuss implementation in more technical depth (as in, how to +implement this in the actual rails codebase of GitLab). + +This feature requires two feature flags: + +- `activity_pub`: Enables or disables all ActivityPub-related features. +- `activity_pub_project`: Enables and disable ActivityPub features specific to + projects. Requires the `activity_pub` flag to also be enabled. + +## What + +Feel free to jump to [the Why section](#why) if you already know what +ActivityPub and the Fediverse are. + +Among the push for [decentralization of the web](https://en.wikipedia.org/wiki/Decentralized_web), +several projects tried different protocols with different ideals behind their reasoning. +Some examples: + +- [Secure Scuttlebutt](https://en.wikipedia.org/wiki/Secure_Scuttlebutt) (or SSB for short) +- [Dat](https://en.wikipedia.org/wiki/Dat_%28software%29) +- [IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System), +- [Solid](https://en.wikipedia.org/wiki/Solid_%28web_decentralization_project%29) + +One gained traction recently: [ActivityPub](https://en.wikipedia.org/wiki/ActivityPub), +better known for the colloquial [Fediverse](https://en.wikipedia.org/wiki/Fediverse) built +on top of it, through applications like +[Mastodon](https://en.wikipedia.org/wiki/Mastodon_%28social_network%29) +(which could be described as some sort of decentralized Facebook) or +[Lemmy](https://en.wikipedia.org/wiki/Lemmy_%28software%29) (which could be +described as some sort of decentralized Reddit). + +ActivityPub has several advantages that makes it attractive +to implementers and could explain its current success: + +- **It's built on top of HTTP**. You don't need to install new software or + to tinker with TCP/UDP to implement ActivityPub, if you have a webserver + or an application that provides an HTTP API (like a rails application), + you already have everything you need. +- **It's built on top of JSON**. All communications are basically JSON + objects, which web developers are already used to, which simplifies adoption. +- **It's a W3C standard and already has multiple implementations**. Being + piloted by the W3C is a guarantee of stability and quality work. They + have profusely demonstrated in the past through their work on HTML, CSS + or other web standards that we can build on top of their work without + the fear of it becoming deprecated or irrelevant after a few years. + +### The Fediverse + +The core idea behind Mastodon and Lemmy is called the Fediverse. Rather +than full decentralization, those applications rely on federation, in the +sense that there still are servers and clients. It's not P2P like SSB, +Dat and IPFS, but instead a galaxy of servers chatting with each other +instead of having central servers controlled by a single entity. + +The user signs up to one of those servers (called **instances**), and they +can then interact with users either on this instance, or on other ones. +From the perspective of the user, they access a global network, and not +only their instance. They see the articles posted on other instances, they +can comment on them, upvote them, etc. + +What happens behind the scenes: +their instance knows where the user they reply to is hosted. It +contacts that other instance to let them know there is a message for them - +somewhat similar to SMTP. Similarly, when a user subscribes +to a feed, their instance informs the instance where the feed is +hosted of this subscription. That target instance then posts back +messages when new activities are created. This allows for a push model, rather +than a constant poll model like RSS. Of course, what was just described is +the happy path; there is moderation, validation and fault tolerance +happening all the way. + +### ActivityPub + +Behind the Fediverse is the ActivityPub protocol. It's a HTTP API +attempting to be as general a social network implementation as possible, +while giving options to be extendable. + +The basic idea is that an `actor` sends and receives `activities`. Activities +are structured JSON messages with well-defined properties, but are extensible +to cover any need. An actor is defined by four endpoints, which are +contacted with the +`application/ld+json; profile="https://www.w3.org/ns/activitystreams"` HTTP Accept header: + +- `GET /inbox`: used by the actor to find new activities intended for them. +- `POST /inbox`: used by instances to push new activities intended for the actor. +- `GET /outbox`: used by anyone to read the activities created by the actor. +- `POST /outbox`: used by the actor to publish new activities. + +Among those, Mastodon and Lemmy only use `POST /inbox` and `GET /outbox`, which +are the minimum needed to implement federation: + +- Instances push new activities for the actor on the inbox. +- Reading the outbox allows reading the feed of an actor. + +Additionally, Mastodon and Lemmy implement a `GET /` endpoint (with the +mentioned Accept header). This endpoint responds with general information about the +actor, like name and URL of the inbox and outbox. While not required by the +standard, it makes discovery easier. + +While a person is the main use case for an actor, an actor does not +necessarily map to a person. Anything can be an actor: a topic, a +subreddit, a group, an event. For GitLab, anything with activities (in the sense +of what GitLab means by "activity") can be an ActivityPub actor. This includes +items like projects, groups, and releases. In those more abstract examples, +an actor can be thought of as an actionable feed. + +ActivityPub by itself does not cover everything that is needed to implement +the Fediverse. Most notably, these are left for the implementers to figure out: + +- Finding a way to deal with spam. Spam is handled by authorizing or + blocking ("defederating") other instances. +- Discovering new instances. +- Performing network-wide searches. + +## Why + +Why would a social media protocol be useful for GitLab? People want a single, +global GitLab network to interact between various projects, without having to +register on each of their hosts. + +Several very popular discussions around this have already happened: + +- [Share events externally via ActivityPub](https://gitlab.com/gitlab-org/gitlab/-/issues/21582) +- [Implement cross-server (federated) merge requests](https://gitlab.com/gitlab-org/gitlab/-/issues/14116) +- [Distributed merge requests](https://gitlab.com/groups/gitlab-org/-/epics/260). + +The ideal workflow would be: + +1. Alice registers to her favorite GitLab instance, like `gitlab.example.org`. +1. She looks for a project on a given topic, and sees Bob's project, even though + Bob is on `gitlab.com`. +1. Alice selects **Fork**, and the `gitlab.com/Bob/project.git` is + forked to `gitlab.example.org/Alice/project.git`. +1. She makes her edits, and opens a merge request, which appears in Bob's + project on `gitlab.com`. +1. Alice and Bob discuss the merge request, each one from their own GitLab + instance. +1. Bob can send additional commits, which are picked up by Alice's instance. +1. When Bob accepts the merge request, his instance picks up the code from + Alice's instance. + +In this process, ActivityPub would help in: + +- Letting Bob know a fork happened. +- Sending the merge request to Bob. +- Enabling Alice and Bob to discuss the merge request. +- Letting Alice know the code was merged. + +It does _not_ help in these cases, which need specific implementations: + +- Implementing a network-wide search. +- Implementing cross-instance forks. (Not needed, thanks to Git.) + +Why use ActivityPub here rather than implementing cross-instance merge requests +in a custom way? Two reasons: + +1. **Building on top of a standard helps reach beyond GitLab**. + While the workflow presented above only mentions GitLab, building on top + of a W3C standard means other forges can follow GitLab + there, and build a massive Fediverse of code sharing. +1. **An opportunity to make GitLab more social**. To prepare the + architecture for the workflow above, smaller steps can be taken, allowing + people to subscribe to activity feeds from their Fediverse social + network. Anything that has a RSS feed could become an ActivityPub feed. + People on Mastodon could follow their favorite developer, project, or topic + from GitLab and see the news in their feed on Mastodon, hopefully raising + engagement with GitLab. + +## How + +The idea of this implementation path is not to take the fastest route to +the feature with the most value added (cross-instance merge requests), but +to go on with the smallest useful step at each iteration, making sure each step +brings something immediately. + +1. **Implement ActivityPub for social following**. + After this, the Fediverse can follow activities on GitLab instances. + 1. ActivityPub to subscribe to project releases. + 1. ActivityPub to subscribe to project creation in topics. + 1. ActivityPub to subscribe to project activities. + 1. ActivityPub to subscribe to group activities. + 1. ActivityPub to subscribe to user activities. +1. **Implement cross-instance search** to enable discovering projects on other instances. +1. **Implement cross-instance forks** to enable forking a project from an other instance. +1. **Implement ActivityPub for cross-instance discussions** to enable discussing + issues and merge requests from another instance: + 1. In issues. + 1. In merge requests. +1. **Implement ActivityPub to submit cross-instance merge requests** to enable + submitting merge requests to other instances. + +For now, see [how to implement an ActivityPub actor](actors/index.md). diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 6e47d2991dc..3ce303d429a 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -23,7 +23,7 @@ The following outline re-uses the [maturity metric](https://about.gitlab.com/dir - [Release management](#release-management) - [Enabled on GitLab.com](feature_flags/controls.md#enabling-a-feature-for-gitlabcom) - Complete - - [Configurable by the GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) + - [Validated by the Reference Architecture group and scaled out recommendations made](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/self-managed-excellence/#reference-architectures) - Lovable - Enabled by default for the majority of users diff --git a/doc/development/ai_architecture.md b/doc/development/ai_architecture.md index 84a2635b13c..28483b943d1 100644 --- a/doc/development/ai_architecture.md +++ b/doc/development/ai_architecture.md @@ -11,7 +11,7 @@ GitLab has created a common set of tools to support our product groups and their 1. Increase the velocity of feature teams by providing a set of high quality, ready to use tools 1. Ability to switch underlying technologies quickly and easily -AI is moving very quickly, and we need to be able to keep pace with changes in the area. We have built an [abstraction layer](../../ee/development/ai_features.md) to do this, allowing us to take a more "pluggable" approach to the underlying models, data stores, and other technologies. +AI is moving very quickly, and we need to be able to keep pace with changes in the area. We have built an [abstraction layer](../../ee/development/ai_features/index.md) to do this, allowing us to take a more "pluggable" approach to the underlying models, data stores, and other technologies. The following diagram from the [architecture blueprint](../architecture/blueprints/ai_gateway/index.md) shows a simplified view of how the different components in GitLab interact. The abstraction layer helps avoid code duplication within the REST APIs within the `AI API` block. @@ -27,6 +27,25 @@ There are two primary reasons for this: the best AI models are cloud-based as th The AI Gateway (formerly the [model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist)) is a standalone-service that will give access to AI features to all users of GitLab, no matter which instance they are using: self-managed, dedicated or GitLab.com. The SaaS-based AI abstraction layer will transition to connecting to this gateway, rather than accessing cloud-based providers directly. +Calls to the AI-gateway from GitLab-rails can be made using the +[Abstraction Layer](ai_features/index.md#abstraction-layer). +By default, these actions are performed asynchronously via a Sidekiq +job to prevent long-running requests in Puma. It should be used for +non-latency sensitive actions due to the added latency by Sidekiq. + +At the time of writing, the Abstraction Layer still directly calls the AI providers. This will be +changed [in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/424614). + +When a certain action is latency sensitive, we can decide to call the +AI-gateway directly. This avoids the latency added by Sidekiq. +[We already do this for `code_suggestions`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/code_suggestions.rb) +which get handled by API endpoints nested in +`/api/v4/code_suggestions`. For any new endpoints added, we should +nest them within the `/api/v4/ai_assisted` namespace. Doing this will +automatically route the requests on GitLab.com to the `ai-assisted` +fleet for GitLab.com, isolating the workload from the regular API and +making it easier to scale if needed. + ## Supported technologies As part of the AI working group, we have been investigating various technologies and vetting them. Below is a list of the tools which have been reviewed and already approved for use within the GitLab application. @@ -98,7 +117,7 @@ The following table documents functionality that Code Suggestions offers today, #### Code Suggestions Latency -Code Suggestions acceptance rates are _highly_ sensitive to latency. While writing code with an AI assistant, a user will pause only for a short duration before continuing on with manually typing out a block of code. As soon as the user has pressed a subsequent keypress, the existing suggestion will be invalidated and a new request will need to be issued to the code suggestions endpoint. In turn, this request will also be highly sensitive to latency. +Code Suggestions acceptance rates are _highly_ sensitive to latency. While writing code with an AI assistant, a user will pause only for a short duration before continuing on with manually typing out a block of code. As soon as the user has pressed a subsequent keypress, the existing suggestion will be invalidated and a new request will need to be issued to the Code Suggestions endpoint. In turn, this request will also be highly sensitive to latency. In a worst case with sufficient latency, the IDE could be issuing a string of requests, each of which is then ignored as the user proceeds without waiting for the response. This adds no value for the user, while still putting load on our services. diff --git a/doc/development/ai_features.md b/doc/development/ai_features.md index ffe151f3876..a952d8f2804 100644 --- a/doc/development/ai_features.md +++ b/doc/development/ai_features.md @@ -1,659 +1,11 @@ --- -stage: AI-powered -group: AI Framework -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'ai_features/index.md' +remove_date: '2023-12-01' --- -# AI features based on 3rd-party integrations +This document was moved to [another location](ai_features/index.md). -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117296) in GitLab 15.11. - -## Features - -- Async execution of the long running API requests - - GraphQL Action starts the request - - Background workers execute - - GraphQL subscriptions deliver results back in real time -- Abstraction for - - OpenAI - - Google Vertex AI - - Anthropic -- Rate Limiting -- Circuit Breaker -- Multi-Level feature flags -- License checks on group level -- Snowplow execution tracking -- Tracking of Token Spent on Prometheus -- Configuration for Moderation check of inputs -- Automatic Markdown Rendering of responses -- Centralised Group Level settings for experiment and 3rd party -- Experimental API endpoints for exploration of AI APIs by GitLab team members without the need for credentials - - OpenAI - - Google Vertex AI - - Anthropic - -## Feature flags - -Apply the following two feature flags to any AI feature work: - -- A general that applies to all AI features. -- A flag specific to that feature. The feature flag name [must be different](feature_flags/index.md#feature-flags-for-licensed-features) than the licensed feature name. - -See the [feature flag tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/405161) for the list of all feature flags and how to use them. - -## Implement a new AI action - -To implement a new AI action, connect to the preferred AI provider. You can connect to this API using either the: - -- Experimental REST API. -- Abstraction layer. - -All AI features are experimental. - -## Test AI features locally - -NOTE: -Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. - -1. Enable the required general feature flags: - - ```ruby - Feature.enable(:ai_related_settings) - Feature.enable(:openai_experimentation) - Feature.enable(:tofa_experimentation_main_flag) - Feature.enable(:anthropic_experimentation) - ``` - -1. Simulate the GDK to [simulate SaaS](ee_features.md#simulate-a-saas-instance) and ensure the group you want to test has an Ultimate license -1. Enable `Experimental features` and `Third-party AI services` - 1. Go to the group with the Ultimate license - 1. **Group Settings** > **General** -> **Permissions and group features** - 1. Enable **Experiment features** - 1. Enable **Third-party AI services** -1. Enable the specific feature flag for the feature you want to test -1. Set the required access token. To receive an access token: - 1. For Vertex, follow the [instructions below](#configure-gcp-vertex-access). - 1. For all other providers, like Anthropic or OpenAI, create an access request where `@m_gill`, `@wayne`, and `@timzallmann` are the tech stack owners. - -### Set up the embedding database - -NOTE: -Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. - -For features that use the embedding database, additional setup is needed. - -1. Enable [pgvector](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/pgvector.md#enable-pgvector-in-the-gdk) in GDK -1. Enable the embedding database in GDK - - ```shell - gdk config set gitlab.rails.databases.embedding.enabled true - ``` - -1. Run `gdk reconfigure` -1. Run database migrations to create the embedding database - -### Set up GitLab Duo Chat - -NOTE: -Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. - -1. [Enable Anthropic API features](#configure-anthropic-access). -1. [Enable OpenAI support](#configure-openai-access). -1. [Ensure the embedding database is configured](#set-up-the-embedding-database). -1. Enable feature specific feature flag. - - ```ruby - Feature.enable(:gitlab_duo) - Feature.enable(:tanuki_bot) - Feature.enable(:ai_redis_cache) - ``` - -1. Ensure that your current branch is up-to-date with `master`. -1. To access the GitLab Duo Chat interface, in the lower-left corner of any page, select **Help** and **Ask GitLab Duo Chat**. - -#### Tips for local development - -1. When responses are taking too long to appear in the user interface, consider restarting Sidekiq by running `gdk restart rails-background-jobs`. If that doesn't work, try `gdk kill` and then `gdk start`. -1. Alternatively, bypass Sidekiq entirely and run the chat service synchronously. This can help with debugging errors as GraphQL errors are now available in the network inspector instead of the Sidekiq logs. - -```diff -diff --git a/ee/app/services/llm/chat_service.rb b/ee/app/services/llm/chat_service.rb -index 5fa7ae8a2bc1..5fe996ba0345 100644 ---- a/ee/app/services/llm/chat_service.rb -+++ b/ee/app/services/llm/chat_service.rb -@@ -5,7 +5,7 @@ class ChatService < BaseService - private - - def perform -- worker_perform(user, resource, :chat, options) -+ worker_perform(user, resource, :chat, options.merge(sync: true)) - end - - def valid? -``` - -### Working with GitLab Duo Chat - -Prompts are the most vital part of GitLab Duo Chat system. Prompts are the instructions sent to the Large Language Model to perform certain tasks. - -The state of the prompts is the result of weeks of iteration. If you want to change any prompt in the current tool, you must put it behind a feature flag. - -If you have any new or updated prompts, ask members of AI Framework team to review, because they have significant experience with them. - -### Setup for GitLab documentation chat (legacy chat) - -To populate the embedding database for GitLab chat: - -1. Open a rails console -1. Run [this script](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/10588#note_1373586079) to populate the embedding database - -### Contributing to GitLab Duo Chat - -The Chat feature uses a [zero-shot agent](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/llm/chain/agents/zero_shot/executor.rb) that includes a system prompt explaining how the large language model should interpret the question and provide an -answer. The system prompt defines available tools that can be used to gather -information to answer the user's question. - -The zero-shot agent receives the user's question and decides which tools to use to gather information to answer it. -It then makes a request to the large language model, which decides if it can answer directly or if it needs to use one -of the defined tools. - -The tools each have their own prompt that provides instructions to the large language model on how to use that tool to -gather information. The tools are designed to be self-sufficient and avoid multiple requests back and forth to -the large language model. - -After the tools have gathered the required information, it is returned to the zero-shot agent, which asks the large language -model if enough information has been gathered to provide the final answer to the user's question. - -#### Adding a new tool - -To add a new tool: - -1. Create files for the tool in the `ee/lib/gitlab/llm/chain/tools/` folder. Use existing tools like `issue_identifier` or -`resource_reader` as a template. - -1. Write a class for the tool that includes: - - - Name and description of what the tool does - - Example questions that would use this tool - - Instructions for the large language model on how to use the tool to gather information - so the main prompts that - this tool is using. - -1. Test and iterate on the prompt using RSpec tests that make real requests to the large language model. - - Prompts require trial and error, the non-deterministic nature of working with LLM can be surprising. - - Anthropic provides good [guide](https://docs.anthropic.com/claude/docs/introduction-to-prompt-design) on working on prompts. - -1. Implement code in the tool to parse the response from the large language model and return it to the zero-shot agent. - -1. Add the new tool name to the `tools` array in `ee/lib/gitlab/llm/completions/chat.rb` so the zero-shot agent knows about it. - -1. Add tests by adding questions to the test-suite for which the new tool should respond to. Iterate on the prompts as needed. - -The key things to keep in mind are properly instructing the large language model through prompts and tool descriptions, -keeping tools self-sufficient, and returning responses to the zero-shot agent. With some trial and error on prompts, -adding new tools can expand the capabilities of the chat feature. - -There are available short [videos](https://www.youtube.com/playlist?list=PL05JrBw4t0KoOK-bm_bwfHaOv-1cveh8i) covering this topic. - -### Debugging - -To gather more insights about the full request, use the `Gitlab::Llm::Logger` file to debug logs. -The default logging level on production is `INFO` and **must not** be used to log any data that could contain personal identifying information. - -To follow the debugging messages related to the AI requests on the abstraction layer, you can use: - -```shell -export LLM_DEBUG=1 -gdk start -tail -f log/llm.log -``` - -### Configure GCP Vertex access - -In order to obtain a GCP service key for local development, please follow the steps below: - -- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group environment by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). -- In the GCP console, go to `IAM & Admin` > `Service Accounts` and click on the "Create new service account" button -- Name the service account something specific to what you're using it for. Select Create and Continue. Under `Grant this service account access to project`, select the role `Vertex AI User`. Select `Continue` then `Done` -- Select your new service account and `Manage keys` > `Add Key` > `Create new key`. This will download the **private** JSON credentials for your service account. -- Open the Rails console. Update the settings to: - -```ruby -Gitlab::CurrentSettings.update(vertex_ai_credentials: File.read('/YOUR_FILE.json')) - -# Note: These credential examples will not work locally for all models -Gitlab::CurrentSettings.update(vertex_ai_host: "<root-domain>") # Example: us-central1-aiplatform.googleapis.com -Gitlab::CurrentSettings.update(vertex_ai_project: "<project-id>") # Example: cloud-large-language-models -``` - -Internal team members can [use this snippet](https://gitlab.com/gitlab-com/gl-infra/production/-/snippets/2541742) for help configuring these endpoints. - -### Configure OpenAI access - -```ruby -Gitlab::CurrentSettings.update(openai_api_key: "<open-ai-key>") -``` - -### Configure Anthropic access - -```ruby -Feature.enable(:anthropic_experimentation) -Gitlab::CurrentSettings.update!(anthropic_api_key: <insert API key>) -``` - -### Testing GitLab Duo Chat with predefined questions - -Because success of answers to user questions in GitLab Duo Chat heavily depends on toolchain and prompts of each tool, it's common that even a minor change in a prompt or a tool impacts processing of some questions. To make sure that a change in the toolchain doesn't break existing functionality, you can use the following rspecs to validate answers to some predefined questions: - -```ruby -export OPENAI_API_KEY='<key>' -export ANTHROPIC_API_KEY='<key>' -REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb -``` - -When you need to update the test questions that require documentation embeddings, -make sure a new fixture is generated and committed together with the change. - -#### Populating embeddings and using embeddings fixture - -To seed your development database with the embeddings for GitLab Documentation, -you may use the pre-generated embeddings and a Rake test. - -```shell -RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:seed_pre_generated -``` - -The DBCleaner gem we use clear the database tables before each test runs. -Instead of fully populating the table `tanuki_bot_mvc` where we store embeddings for the documentations, -we can add a few selected embeddings to the table from a pre-generated fixture. - -For instance, to test that the question "How can I reset my password" is correctly -retrieving the relevant embeddings and answered, we can extract the top N closet embeddings -to the question into a fixture and only restore a small number of embeddings quickly. -To faciliate an extraction process, a Rake task been written. -You can add or remove the questions needed to be tested in the Rake task and run the task to generate a new fixture. - -```shell -RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:extract_embeddings -``` - -In the specs where you need to use the embeddings, -use the RSpec config hook `:ai_embedding_fixtures` on a context. - -```ruby -context 'when asking about how to use GitLab', :ai_embedding_fixtures do - # ...examples -end -``` - -## Experimental REST API - -Use the [experimental REST API endpoints](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/experimentation) to quickly experiment and prototype AI features. - -The endpoints are: - -- `https://gitlab.example.com/api/v4/ai/experimentation/openai/completions` -- `https://gitlab.example.com/api/v4/ai/experimentation/openai/embeddings` -- `https://gitlab.example.com/api/v4/ai/experimentation/openai/chat/completions` -- `https://gitlab.example.com/api/v4/ai/experimentation/anthropic/complete` -- `https://gitlab.example.com/api/v4/ai/experimentation/tofa/chat` - -These endpoints are only for prototyping, not for rolling features out to customers. -The experimental endpoint is only available to GitLab team members on production. Use the -[GitLab API token](../user/profile/personal_access_tokens.md) to authenticate. - -## Abstraction layer - -### GraphQL API - -To connect to the AI provider API using the Abstraction Layer, use an extendable GraphQL API called -[`aiAction`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/graphql/mutations/ai/action.rb). -The `input` accepts key/value pairs, where the `key` is the action that needs to be performed. -We only allow one AI action per mutation request. - -Example of a mutation: - -```graphql -mutation { - aiAction(input: {summarizeComments: {resourceId: "gid://gitlab/Issue/52"}}) { - clientMutationId - } -} -``` - -As an example, assume we want to build an "explain code" action. To do this, we extend the `input` with a new key, -`explainCode`. The mutation would look like this: - -```graphql -mutation { - aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log()" }}) { - clientMutationId - } -} -``` - -The GraphQL API then uses the [OpenAI Client](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/llm/open_ai/client.rb) -to send the response. - -Remember that other clients are available and you should not use OpenAI. - -#### How to receive a response - -As the OpenAI API requests are handled in a background job, we do not keep the request alive and -the response is sent through the `aiCompletionResponse` subscription: - -```mutation -subscription aiCompletionResponse($userId: UserID, $resourceId: AiModelID!) { - aiCompletionResponse(userId: $userId, resourceId: $resourceId) { - responseBody - errors - } -} -``` - -WARNING: -You should only subscribe to the subscription once the mutation is sent. If multiple subscriptions are active on the same page, they currently all receive updates as our identifier is the user and the resource. To mitigate this, you should only subscribe when the mutation is sent. You can use [`skip()`](You can use [`skip()`](https://apollo.vuejs.org/guide/apollo/subscriptions.html#skipping-the-subscription)) for this case. To prevent this problem in the future, we implement a [request identifier](https://gitlab.com/gitlab-org/gitlab/-/issues/408196). - -#### Current abstraction layer flow - -The following graph uses OpenAI as an example. You can use different providers. - -```mermaid -flowchart TD -A[GitLab frontend] -->B[AiAction GraphQL mutation] -B --> C[Llm::ExecuteMethodService] -C --> D[One of services, for example: Llm::GenerateSummaryService] -D -->|scheduled| E[AI worker:Llm::CompletionWorker] -E -->F[::Gitlab::Llm::Completions::Factory] -F -->G[`::Gitlab::Llm::OpenAi::Completions::...` class using `::Gitlab::Llm::OpenAi::Templates::...` class] -G -->|calling| H[Gitlab::Llm::OpenAi::Client] -H --> |response| I[::Gitlab::Llm::OpenAi::ResponseService] -I --> J[GraphqlTriggers.ai_completion_response] -J --> K[::GitlabSchema.subscriptions.trigger] -``` - -## CircuitBreaker - -The CircuitBreaker concern is a reusable module that you can include in any class that needs to run code with circuit breaker protection. The concern provides a `run_with_circuit` method that wraps a code block with circuit breaker functionality, which helps prevent cascading failures and improves system resilience. For more information about the circuit breaker pattern, see: - -- [What is Circuit breaker](https://martinfowler.com/bliki/CircuitBreaker.html). -- [The Hystrix documentation on CircuitBreaker](https://github.com/Netflix/Hystrix/wiki/How-it-Works#circuit-breaker). - -### Use CircuitBreaker - -To use the CircuitBreaker concern, you need to include it in a class. For example: - -```ruby -class MyService - include Gitlab::Llm::Concerns::CircuitBreaker - - def call_external_service - run_with_circuit do - # Code that interacts with external service goes here - - raise InternalServerError - end - end -end -``` - -The `call_external_service` method is an example method that interacts with an external service. -By wrapping the code that interacts with the external service with `run_with_circuit`, the method is executed within the circuit breaker. -The circuit breaker is created and configured by the `circuit` method, which is called automatically when the `CircuitBreaker` module is included. -The method should raise `InternalServerError` error which will be counted towards the error threshold if raised during the execution of the code block. - -The circuit breaker tracks the number of errors and the rate of requests, -and opens the circuit if it reaches the configured error threshold or volume threshold. -If the circuit is open, subsequent requests fail fast without executing the code block, and the circuit breaker periodically allows a small number of requests through to test the service's availability before closing the circuit again. - -### Configuration - -The circuit breaker is configured with two constants which control the number of errors and requests at which the circuit will open: - -- `ERROR_THRESHOLD` -- `VOLUME_THRESHOLD` - -You can adjust these values as needed for the specific service and usage pattern. -The `InternalServerError` is the exception class counted towards the error threshold if raised during the execution of the code block. -This is the exception class that triggers the circuit breaker when raised by the code that interacts with the external service. - -NOTE: -The `CircuitBreaker` module depends on the `Circuitbox` gem to provide the circuit breaker implementation. By default, the service name is inferred from the class name where the concern module is included. Override the `service_name` method if the name needs to be different. - -### Testing - -To test code that uses the `CircuitBreaker` concern, you can use `RSpec` shared examples and pass the `service` and `subject` variables: - -```ruby -it_behaves_like 'has circuit breaker' do - let(:service) { dummy_class.new } - let(:subject) { service.dummy_method } -end -``` - -## How to implement a new action - -### Register a new method - -Go to the `Llm::ExecuteMethodService` and add a new method with the new service class you will create. - -```ruby -class ExecuteMethodService < BaseService - METHODS = { - # ... - amazing_new_ai_feature: Llm::AmazingNewAiFeatureService - }.freeze -``` - -### Create a Service - -1. Create a new service under `ee/app/services/llm/` and inherit it from the `BaseService`. -1. The `resource` is the object we want to act on. It can be any object that includes the `Ai::Model` concern. For example it could be a `Project`, `MergeRequest`, or `Issue`. - -```ruby -# ee/app/services/llm/amazing_new_ai_feature_service.rb - -module Llm - class AmazingNewAiFeatureService < BaseService - private - - def perform - ::Llm::CompletionWorker.perform_async(user.id, resource.id, resource.class.name, :amazing_new_ai_feature) - success - end - - def valid? - super && Ability.allowed?(user, :amazing_new_ai_feature, resource) - end - end -end -``` - -### Authorization - -We recommend to use [policies](policies.md) to deal with authorization for a feature. Currently we need to make sure to cover the following checks: - -1. General AI feature flag is enabled -1. Feature specific feature flag is enabled -1. The namespace has the required license for the feature -1. User is a member of the group/project -1. `experiment_features_enabled` and `third_party_ai_features_enabled` flags are set on the `Namespace` - -For our example, we need to implement the `allowed?(:amazing_new_ai_feature)` call. As an example, you can look at the [Issue Policy for the summarize comments feature](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/policies/ee/issue_policy.rb). In our example case, we want to implement the feature for Issues as well: - -```ruby -# ee/app/policies/ee/issue_policy.rb - -module EE - module IssuePolicy - extend ActiveSupport::Concern - prepended do - with_scope :subject - condition(:ai_available) do - ::Feature.enabled?(:openai_experimentation) - end - - with_scope :subject - condition(:amazing_new_ai_feature_enabled) do - ::Feature.enabled?(:amazing_new_ai_feature, subject_container) && - subject_container.licensed_feature_available?(:amazing_new_ai_feature) - end - - rule do - ai_available & amazing_new_ai_feature_enabled & is_project_member - end.enable :amazing_new_ai_feature - end - end -end -``` - -### Pairing requests with responses - -Because multiple users' requests can be processed in parallel, when receiving responses, -it can be difficult to pair a response with its original request. The `requestId` -field can be used for this purpose, because both the request and response are assured -to have the same `requestId` UUID. - -### Caching - -AI requests and responses can be cached. Cached conversation is being used to -display user interaction with AI features. In the current implementation, this cache -is not used to skip consecutive calls to the AI service when a user repeats -their requests. - -```graphql -query { - aiMessages { - nodes { - id - requestId - content - role - errors - timestamp - } - } -} -``` - -This cache is especially useful for chat functionality. For other services, -caching is disabled. (It can be enabled for a service by using `cache_response: true` -option.) - -Caching has following limitations: - -- Messages are stored in Redis stream. -- There is a single stream of messages per user. This means that all services - currently share the same cache. If needed, this could be extended to multiple - streams per user (after checking with the infrastructure team that Redis can handle - the estimated amount of messages). -- Only the last 50 messages (requests + responses) are kept. -- Expiration time of the stream is 3 days since adding last message. -- User can access only their own messages. There is no authorization on the caching - level, and any authorization (if accessed by not current user) is expected on - the service layer. - -### Check if feature is allowed for this resource based on namespace settings - -There are two settings allowed on root namespace level that restrict the use of AI features: - -- `experiment_features_enabled` -- `third_party_ai_features_enabled`. - -To check if that feature is allowed for a given namespace, call: - -```ruby -Gitlab::Llm::StageCheck.available?(namespace, :name_of_the_feature) -``` - -Add the name of the feature to the `Gitlab::Llm::StageCheck` class. There are arrays there that differentiate -between experimental and beta features. - -This way we are ready for the following different cases: - -- If the feature is not in any array, the check will return `true`. For example, the feature was moved to GA and does not use a third-party setting. -- If feature is in GA, but uses a third-party setting, the class will return a proper answer based on the namespace third-party setting. - -To move the feature from the experimental phase to the beta phase, move the name of the feature from the `EXPERIMENTAL_FEATURES` array to the `BETA_FEATURES` array. - -### Implement calls to AI APIs and the prompts - -The `CompletionWorker` will call the `Completions::Factory` which will initialize the Service and execute the actual call to the API. -In our example, we will use OpenAI and implement two new classes: - -```ruby -# /ee/lib/gitlab/llm/open_ai/completions/amazing_new_ai_feature.rb - -module Gitlab - module Llm - module OpenAi - module Completions - class AmazingNewAiFeature - def initialize(ai_prompt_class) - @ai_prompt_class = ai_prompt_class - end - - def execute(user, issue, options) - options = ai_prompt_class.get_options(options[:messages]) - - ai_response = Gitlab::Llm::OpenAi::Client.new(user).chat(content: nil, **options) - - ::Gitlab::Llm::OpenAi::ResponseService.new(user, issue, ai_response, options: {}).execute( - Gitlab::Llm::OpenAi::ResponseModifiers::Chat.new - ) - end - - private - - attr_reader :ai_prompt_class - end - end - end - end -end -``` - -```ruby -# /ee/lib/gitlab/llm/open_ai/templates/amazing_new_ai_feature.rb - -module Gitlab - module Llm - module OpenAi - module Templates - class AmazingNewAiFeature - TEMPERATURE = 0.3 - - def self.get_options(messages) - system_content = <<-TEMPLATE - You are an assistant that writes code for the following input: - """ - TEMPLATE - - { - messages: [ - { role: "system", content: system_content }, - { role: "user", content: messages }, - ], - temperature: TEMPERATURE - } - end - end - end - end - end -end -``` - -Because we support multiple AI providers, you may also use those providers for the same example: - -```ruby -Gitlab::Llm::VertexAi::Client.new(user) -Gitlab::Llm::Anthropic::Client.new(user) -``` - -### Add Ai Action to GraphQL - -TODO - -## Security - -Refer to the [secure coding guidelines for Artificial Intelligence (AI) features](secure_coding_guidelines.md#artificial-intelligence-ai-features). +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/ai_features/duo_chat.md b/doc/development/ai_features/duo_chat.md new file mode 100644 index 00000000000..5c7359eca9f --- /dev/null +++ b/doc/development/ai_features/duo_chat.md @@ -0,0 +1,139 @@ +--- +stage: AI-powered +group: Duo Chat +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab Duo Chat + +## Set up GitLab Duo Chat + +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + +1. [Enable Anthropic API features](index.md#configure-anthropic-access). +1. [Enable OpenAI support](index.md#configure-openai-access). +1. [Ensure the embedding database is configured](index.md#set-up-the-embedding-database). +1. Enable feature specific feature flag. + + ```ruby + Feature.enable(:gitlab_duo) + Feature.enable(:tanuki_bot) + Feature.enable(:ai_redis_cache) + ``` + +1. Ensure that your current branch is up-to-date with `master`. +1. To access the GitLab Duo Chat interface, in the lower-left corner of any page, select **Help** and **Ask GitLab Duo Chat**. + +### Tips for local development + +1. When responses are taking too long to appear in the user interface, consider restarting Sidekiq by running `gdk restart rails-background-jobs`. If that doesn't work, try `gdk kill` and then `gdk start`. +1. Alternatively, bypass Sidekiq entirely and run the chat service synchronously. This can help with debugging errors as GraphQL errors are now available in the network inspector instead of the Sidekiq logs. + +```diff +diff --git a/ee/app/services/llm/chat_service.rb b/ee/app/services/llm/chat_service.rb +index 5fa7ae8a2bc1..5fe996ba0345 100644 +--- a/ee/app/services/llm/chat_service.rb ++++ b/ee/app/services/llm/chat_service.rb +@@ -5,7 +5,7 @@ class ChatService < BaseService + private + + def perform +- worker_perform(user, resource, :chat, options) ++ worker_perform(user, resource, :chat, options.merge(sync: true)) + end + + def valid? +``` + +## Working with GitLab Duo Chat + +Prompts are the most vital part of GitLab Duo Chat system. Prompts are the instructions sent to the Large Language Model to perform certain tasks. + +The state of the prompts is the result of weeks of iteration. If you want to change any prompt in the current tool, you must put it behind a feature flag. + +If you have any new or updated prompts, ask members of AI Framework team to review, because they have significant experience with them. + +## Contributing to GitLab Duo Chat + +The Chat feature uses a [zero-shot agent](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/llm/chain/agents/zero_shot/executor.rb) that includes a system prompt explaining how the large language model should interpret the question and provide an +answer. The system prompt defines available tools that can be used to gather +information to answer the user's question. + +The zero-shot agent receives the user's question and decides which tools to use to gather information to answer it. +It then makes a request to the large language model, which decides if it can answer directly or if it needs to use one +of the defined tools. + +The tools each have their own prompt that provides instructions to the large language model on how to use that tool to +gather information. The tools are designed to be self-sufficient and avoid multiple requests back and forth to +the large language model. + +After the tools have gathered the required information, it is returned to the zero-shot agent, which asks the large language +model if enough information has been gathered to provide the final answer to the user's question. + +### Adding a new tool + +To add a new tool: + +1. Create files for the tool in the `ee/lib/gitlab/llm/chain/tools/` folder. Use existing tools like `issue_identifier` or + `resource_reader` as a template. + +1. Write a class for the tool that includes: + + - Name and description of what the tool does + - Example questions that would use this tool + - Instructions for the large language model on how to use the tool to gather information - so the main prompts that + this tool is using. + +1. Test and iterate on the prompt using RSpec tests that make real requests to the large language model. + - Prompts require trial and error, the non-deterministic nature of working with LLM can be surprising. + - Anthropic provides good [guide](https://docs.anthropic.com/claude/docs/introduction-to-prompt-design) on working on prompts. + - GitLab [guide](prompts.md) on working with prompts. + +1. Implement code in the tool to parse the response from the large language model and return it to the zero-shot agent. + +1. Add the new tool name to the `tools` array in `ee/lib/gitlab/llm/completions/chat.rb` so the zero-shot agent knows about it. + +1. Add tests by adding questions to the test-suite for which the new tool should respond to. Iterate on the prompts as needed. + +The key things to keep in mind are properly instructing the large language model through prompts and tool descriptions, +keeping tools self-sufficient, and returning responses to the zero-shot agent. With some trial and error on prompts, +adding new tools can expand the capabilities of the Chat feature. + +There are available short [videos](https://www.youtube.com/playlist?list=PL05JrBw4t0KoOK-bm_bwfHaOv-1cveh8i) covering this topic. + +## Debugging + +To gather more insights about the full request, use the `Gitlab::Llm::Logger` file to debug logs. +The default logging level on production is `INFO` and **must not** be used to log any data that could contain personal identifying information. + +To follow the debugging messages related to the AI requests on the abstraction layer, you can use: + +```shell +export LLM_DEBUG=1 +gdk start +tail -f log/llm.log +``` + +## Testing GitLab Duo Chat with predefined questions + +Because success of answers to user questions in GitLab Duo Chat heavily depends on toolchain and prompts of each tool, it's common that even a minor change in a prompt or a tool impacts processing of some questions. To make sure that a change in the toolchain doesn't break existing functionality, you can use the following rspecs to validate answers to some predefined questions: + +```ruby +export OPENAI_API_KEY='<key>' +export ANTHROPIC_API_KEY='<key>' +REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb +``` + +When you need to update the test questions that require documentation embeddings, +make sure a new fixture is generated and committed together with the change. + +## GraphQL Subscription + +The GraphQL Subscription for Chat behaves slightly different because it's user-centric. A user could have Chat open on multiple browser tabs, or also on their IDE. +We therefore need to broadcast messages to multiple clients to keep them in sync. The `aiAction` mutation with the `chat` action behaves the following: + +1. All complete Chat messages (including messages from the user) are broadcasted with the `userId` and the `resourceId` from the mutation as identifier, ignoring the `clientSubscriptionId`. +1. Chunks from streamed Chat messages are broadcasted with the `userId`, `resourceId`, and `clientSubscriptionId` as identifier. + +To truly sync messages between all clients of a user, we need to remove the `resourceId` as well, which will be fixed by [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/420296). diff --git a/doc/development/ai_features/index.md b/doc/development/ai_features/index.md new file mode 100644 index 00000000000..e1d3ae36570 --- /dev/null +++ b/doc/development/ai_features/index.md @@ -0,0 +1,577 @@ +--- +stage: AI-powered +group: AI Framework +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# AI features based on 3rd-party integrations + +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117296) in GitLab 15.11. + +## Features + +- Async execution of the long running API requests + - GraphQL Action starts the request + - Background workers execute + - GraphQL subscriptions deliver results back in real time +- Abstraction for + - OpenAI + - Google Vertex AI + - Anthropic +- Rate Limiting +- Circuit Breaker +- Multi-Level feature flags +- License checks on group level +- Snowplow execution tracking +- Tracking of Token Spent on Prometheus +- Configuration for Moderation check of inputs +- Automatic Markdown Rendering of responses +- Centralised Group Level settings for experiment and 3rd party +- Experimental API endpoints for exploration of AI APIs by GitLab team members without the need for credentials + - OpenAI + - Google Vertex AI + - Anthropic + +## Feature flags + +Apply the following two feature flags to any AI feature work: + +- A general that applies to all AI features. +- A flag specific to that feature. The feature flag name [must be different](../feature_flags/index.md#feature-flags-for-licensed-features) than the licensed feature name. + +See the [feature flag tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/405161) for the list of all feature flags and how to use them. + +## Implement a new AI action + +To implement a new AI action, connect to the preferred AI provider. You can connect to this API using either the: + +- Experimental REST API. +- Abstraction layer. + +All AI features are experimental. + +## Test AI features locally + +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + +1. Enable the required general feature flags: + + ```ruby + Feature.enable(:ai_related_settings) + Feature.enable(:openai_experimentation) + ``` + +1. Simulate the GDK to [simulate SaaS](../ee_features.md#simulate-a-saas-instance) and ensure the group you want to test has an Ultimate license +1. Enable `Experimental features` and `Third-party AI services` + 1. Go to the group with the Ultimate license + 1. **Group Settings** > **General** -> **Permissions and group features** + 1. Enable **Experiment features** + 1. Enable **Third-party AI services** +1. Enable the specific feature flag for the feature you want to test +1. Set the required access token. To receive an access token: + 1. For Vertex, follow the [instructions below](#configure-gcp-vertex-access). + 1. For all other providers, like Anthropic or OpenAI, create an access request where `@m_gill`, `@wayne`, and `@timzallmann` are the tech stack owners. + +### Set up the embedding database + +NOTE: +Use [this snippet](https://gitlab.com/gitlab-org/gitlab/-/snippets/2554994) for help automating the following section. + +For features that use the embedding database, additional setup is needed. + +1. Enable [pgvector](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/pgvector.md#enable-pgvector-in-the-gdk) in GDK +1. Enable the embedding database in GDK + + ```shell + gdk config set gitlab.rails.databases.embedding.enabled true + ``` + +1. Run `gdk reconfigure` +1. Run database migrations to create the embedding database + +### Setup for GitLab documentation chat (legacy chat) + +To populate the embedding database for GitLab chat: + +1. Open a rails console +1. Run [this script](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/10588#note_1373586079) to populate the embedding database + +### Configure GCP Vertex access + +In order to obtain a GCP service key for local development, please follow the steps below: + +- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group environment by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). +- In the GCP console, go to `IAM & Admin` > `Service Accounts` and click on the "Create new service account" button +- Name the service account something specific to what you're using it for. Select Create and Continue. Under `Grant this service account access to project`, select the role `Vertex AI User`. Select `Continue` then `Done` +- Select your new service account and `Manage keys` > `Add Key` > `Create new key`. This will download the **private** JSON credentials for your service account. +- If you are using your own project, you may also need to enable the Vertex AI API: + 1. Go to **APIs & Services > Enabled APIs & services**. + 1. Select **+ Enable APIs and Services**. + 1. Search for `Vertex AI API`. + 1. Select **Vertex AI API**, then select **Enable**. +- Open the Rails console. Update the settings to: + +```ruby +Gitlab::CurrentSettings.update(vertex_ai_credentials: File.read('/YOUR_FILE.json')) + +# Note: These credential examples will not work locally for all models +Gitlab::CurrentSettings.update(vertex_ai_host: "<root-domain>") # Example: us-central1-aiplatform.googleapis.com +Gitlab::CurrentSettings.update(vertex_ai_project: "<project-id>") # Example: cloud-large-language-models +``` + +Internal team members can [use this snippet](https://gitlab.com/gitlab-com/gl-infra/production/-/snippets/2541742) for help configuring these endpoints. + +### Configure OpenAI access + +```ruby +Gitlab::CurrentSettings.update(openai_api_key: "<open-ai-key>") +``` + +### Configure Anthropic access + +```ruby +Gitlab::CurrentSettings.update!(anthropic_api_key: <insert API key>) +``` + +#### Populating embeddings and using embeddings fixture + +To seed your development database with the embeddings for GitLab Documentation, +you may use the pre-generated embeddings and a Rake test. + +```shell +RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:seed_pre_generated +``` + +The DBCleaner gem we use clear the database tables before each test runs. +Instead of fully populating the table `tanuki_bot_mvc` where we store embeddings for the documentations, +we can add a few selected embeddings to the table from a pre-generated fixture. + +For instance, to test that the question "How can I reset my password" is correctly +retrieving the relevant embeddings and answered, we can extract the top N closet embeddings +to the question into a fixture and only restore a small number of embeddings quickly. +To faciliate an extraction process, a Rake task been written. +You can add or remove the questions needed to be tested in the Rake task and run the task to generate a new fixture. + +```shell +RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:extract_embeddings +``` + +In the specs where you need to use the embeddings, +use the RSpec config hook `:ai_embedding_fixtures` on a context. + +```ruby +context 'when asking about how to use GitLab', :ai_embedding_fixtures do + # ...examples +end +``` + +### Working with GitLab Duo Chat + +View [guidelines](duo_chat.md) for working with GitLab Duo Chat. + +## Experimental REST API + +Use the [experimental REST API endpoints](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/experimentation) to quickly experiment and prototype AI features. + +The endpoints are: + +- `https://gitlab.example.com/api/v4/ai/experimentation/openai/completions` +- `https://gitlab.example.com/api/v4/ai/experimentation/openai/embeddings` +- `https://gitlab.example.com/api/v4/ai/experimentation/openai/chat/completions` +- `https://gitlab.example.com/api/v4/ai/experimentation/anthropic/complete` +- `https://gitlab.example.com/api/v4/ai/experimentation/vertex/chat` + +These endpoints are only for prototyping, not for rolling features out to customers. + +In your local dev environment, you can experiment with these endpoints locally with the feature flag enabled: + +```ruby +Feature.enable(:ai_experimentation_api) +``` + +On production, the experimental endpoints are only available to GitLab team members. Use a +[GitLab API token](../../user/profile/personal_access_tokens.md) to authenticate. + +## Abstraction layer + +### GraphQL API + +To connect to the AI provider API using the Abstraction Layer, use an extendable GraphQL API called +[`aiAction`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/graphql/mutations/ai/action.rb). +The `input` accepts key/value pairs, where the `key` is the action that needs to be performed. +We only allow one AI action per mutation request. + +Example of a mutation: + +```graphql +mutation { + aiAction(input: {summarizeComments: {resourceId: "gid://gitlab/Issue/52"}}) { + clientMutationId + } +} +``` + +As an example, assume we want to build an "explain code" action. To do this, we extend the `input` with a new key, +`explainCode`. The mutation would look like this: + +```graphql +mutation { + aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log()" }}) { + clientMutationId + } +} +``` + +The GraphQL API then uses the [OpenAI Client](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/llm/open_ai/client.rb) +to send the response. + +Remember that other clients are available and you should not use OpenAI. + +#### How to receive a response + +The API requests to AI providers are handled in a background job. We therefore do not keep the request alive and the Frontend needs to match the request to the response from the subscription. + +WARNING: +Determining the right response to a request can cause problems when only `userId` and `resourceId` are used. For example, when two AI features use the same `userId` and `resourceId` both subscriptions will receive the response from each other. To prevent this intereference, we introduced the `clientSubscriptionId`. + +To match a response on the `aiCompletionResponse` subscription, you can provide a `clientSubscriptionId` to the `aiAction` mutation. + +- The `clientSubscriptionId` should be unique per feature and within a page to not interfere with other AI features. We recommend to use a `UUID`. +- Only when the `clientSubscriptionId` is provided as part of the `aiAction` mutation, it will be used for broadcasting the `aiCompletionResponse`. +- If the `clientSubscriptionId` is not provided, only `userId` and `resourceId` are used for the `aiCompletionResponse`. + +As an example mutation for summarizing comments, we provide a `randomId` as part of the mutation: + +```graphql +mutation { + aiAction(input: {summarizeComments: {resourceId: "gid://gitlab/Issue/52"}, clientSubscriptionId: "randomId"}) { + clientMutationId + } +} +``` + +In our component, we then listen on the `aiCompletionResponse` using the `userId`, `resourceId` and `clientSubscriptionId` (`"randomId"`): + +```graphql +subscription aiCompletionResponse($userId: UserID, $resourceId: AiModelID, $clientSubscriptionId: String) { + aiCompletionResponse(userId: $userId, resourceId: $resourceId, clientSubscriptionId: $clientSubscriptionId) { + content + errors + } +} +``` + +Note that the [subscription for chat](duo_chat.md#graphql-subscription) behaves differently. + +To not have many concurrent subscriptions, you should also only subscribe to the subscription once the mutation is sent by using [`skip()`](https://apollo.vuejs.org/guide/apollo/subscriptions.html#skipping-the-subscription). + +#### Current abstraction layer flow + +The following graph uses OpenAI as an example. You can use different providers. + +```mermaid +flowchart TD +A[GitLab frontend] -->B[AiAction GraphQL mutation] +B --> C[Llm::ExecuteMethodService] +C --> D[One of services, for example: Llm::GenerateSummaryService] +D -->|scheduled| E[AI worker:Llm::CompletionWorker] +E -->F[::Gitlab::Llm::Completions::Factory] +F -->G[`::Gitlab::Llm::OpenAi::Completions::...` class using `::Gitlab::Llm::OpenAi::Templates::...` class] +G -->|calling| H[Gitlab::Llm::OpenAi::Client] +H --> |response| I[::Gitlab::Llm::OpenAi::ResponseService] +I --> J[GraphqlTriggers.ai_completion_response] +J --> K[::GitlabSchema.subscriptions.trigger] +``` + +## CircuitBreaker + +The CircuitBreaker concern is a reusable module that you can include in any class that needs to run code with circuit breaker protection. The concern provides a `run_with_circuit` method that wraps a code block with circuit breaker functionality, which helps prevent cascading failures and improves system resilience. For more information about the circuit breaker pattern, see: + +- [What is Circuit breaker](https://martinfowler.com/bliki/CircuitBreaker.html). +- [The Hystrix documentation on CircuitBreaker](https://github.com/Netflix/Hystrix/wiki/How-it-Works#circuit-breaker). + +### Use CircuitBreaker + +To use the CircuitBreaker concern, you need to include it in a class. For example: + +```ruby +class MyService + include Gitlab::Llm::Concerns::CircuitBreaker + + def call_external_service + run_with_circuit do + # Code that interacts with external service goes here + + raise InternalServerError + end + end +end +``` + +The `call_external_service` method is an example method that interacts with an external service. +By wrapping the code that interacts with the external service with `run_with_circuit`, the method is executed within the circuit breaker. +The circuit breaker is created and configured by the `circuit` method, which is called automatically when the `CircuitBreaker` module is included. +The method should raise `InternalServerError` error which will be counted towards the error threshold if raised during the execution of the code block. + +The circuit breaker tracks the number of errors and the rate of requests, +and opens the circuit if it reaches the configured error threshold or volume threshold. +If the circuit is open, subsequent requests fail fast without executing the code block, and the circuit breaker periodically allows a small number of requests through to test the service's availability before closing the circuit again. + +### Configuration + +The circuit breaker is configured with two constants which control the number of errors and requests at which the circuit will open: + +- `ERROR_THRESHOLD` +- `VOLUME_THRESHOLD` + +You can adjust these values as needed for the specific service and usage pattern. +The `InternalServerError` is the exception class counted towards the error threshold if raised during the execution of the code block. +This is the exception class that triggers the circuit breaker when raised by the code that interacts with the external service. + +NOTE: +The `CircuitBreaker` module depends on the `Circuitbox` gem to provide the circuit breaker implementation. By default, the service name is inferred from the class name where the concern module is included. Override the `service_name` method if the name needs to be different. + +### Testing + +To test code that uses the `CircuitBreaker` concern, you can use `RSpec` shared examples and pass the `service` and `subject` variables: + +```ruby +it_behaves_like 'has circuit breaker' do + let(:service) { dummy_class.new } + let(:subject) { service.dummy_method } +end +``` + +## How to implement a new action + +### Register a new method + +Go to the `Llm::ExecuteMethodService` and add a new method with the new service class you will create. + +```ruby +class ExecuteMethodService < BaseService + METHODS = { + # ... + amazing_new_ai_feature: Llm::AmazingNewAiFeatureService + }.freeze +``` + +### Create a Service + +1. Create a new service under `ee/app/services/llm/` and inherit it from the `BaseService`. +1. The `resource` is the object we want to act on. It can be any object that includes the `Ai::Model` concern. For example it could be a `Project`, `MergeRequest`, or `Issue`. + +```ruby +# ee/app/services/llm/amazing_new_ai_feature_service.rb + +module Llm + class AmazingNewAiFeatureService < BaseService + private + + def perform + ::Llm::CompletionWorker.perform_async(user.id, resource.id, resource.class.name, :amazing_new_ai_feature) + success + end + + def valid? + super && Ability.allowed?(user, :amazing_new_ai_feature, resource) + end + end +end +``` + +### Authorization + +We recommend to use [policies](../policies.md) to deal with authorization for a feature. Currently we need to make sure to cover the following checks: + +1. General AI feature flag is enabled +1. Feature specific feature flag is enabled +1. The namespace has the required license for the feature +1. User is a member of the group/project +1. `experiment_features_enabled` and `third_party_ai_features_enabled` flags are set on the `Namespace` + +For our example, we need to implement the `allowed?(:amazing_new_ai_feature)` call. As an example, you can look at the [Issue Policy for the summarize comments feature](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/policies/ee/issue_policy.rb). In our example case, we want to implement the feature for Issues as well: + +```ruby +# ee/app/policies/ee/issue_policy.rb + +module EE + module IssuePolicy + extend ActiveSupport::Concern + prepended do + with_scope :subject + condition(:ai_available) do + ::Feature.enabled?(:openai_experimentation) + end + + with_scope :subject + condition(:amazing_new_ai_feature_enabled) do + ::Feature.enabled?(:amazing_new_ai_feature, subject_container) && + subject_container.licensed_feature_available?(:amazing_new_ai_feature) + end + + rule do + ai_available & amazing_new_ai_feature_enabled & is_project_member + end.enable :amazing_new_ai_feature + end + end +end +``` + +### Pairing requests with responses + +Because multiple users' requests can be processed in parallel, when receiving responses, +it can be difficult to pair a response with its original request. The `requestId` +field can be used for this purpose, because both the request and response are assured +to have the same `requestId` UUID. + +### Caching + +AI requests and responses can be cached. Cached conversation is being used to +display user interaction with AI features. In the current implementation, this cache +is not used to skip consecutive calls to the AI service when a user repeats +their requests. + +```graphql +query { + aiMessages { + nodes { + id + requestId + content + role + errors + timestamp + } + } +} +``` + +This cache is especially useful for chat functionality. For other services, +caching is disabled. (It can be enabled for a service by using `cache_response: true` +option.) + +Caching has following limitations: + +- Messages are stored in Redis stream. +- There is a single stream of messages per user. This means that all services + currently share the same cache. If needed, this could be extended to multiple + streams per user (after checking with the infrastructure team that Redis can handle + the estimated amount of messages). +- Only the last 50 messages (requests + responses) are kept. +- Expiration time of the stream is 3 days since adding last message. +- User can access only their own messages. There is no authorization on the caching + level, and any authorization (if accessed by not current user) is expected on + the service layer. + +### Check if feature is allowed for this resource based on namespace settings + +There are two settings allowed on root namespace level that restrict the use of AI features: + +- `experiment_features_enabled` +- `third_party_ai_features_enabled`. + +To check if that feature is allowed for a given namespace, call: + +```ruby +Gitlab::Llm::StageCheck.available?(namespace, :name_of_the_feature) +``` + +Add the name of the feature to the `Gitlab::Llm::StageCheck` class. There are arrays there that differentiate +between experimental and beta features. + +This way we are ready for the following different cases: + +- If the feature is not in any array, the check will return `true`. For example, the feature was moved to GA and does not use a third-party setting. +- If feature is in GA, but uses a third-party setting, the class will return a proper answer based on the namespace third-party setting. + +To move the feature from the experimental phase to the beta phase, move the name of the feature from the `EXPERIMENTAL_FEATURES` array to the `BETA_FEATURES` array. + +### Implement calls to AI APIs and the prompts + +The `CompletionWorker` will call the `Completions::Factory` which will initialize the Service and execute the actual call to the API. +In our example, we will use OpenAI and implement two new classes: + +```ruby +# /ee/lib/gitlab/llm/open_ai/completions/amazing_new_ai_feature.rb + +module Gitlab + module Llm + module OpenAi + module Completions + class AmazingNewAiFeature + def initialize(ai_prompt_class) + @ai_prompt_class = ai_prompt_class + end + + def execute(user, issue, options) + options = ai_prompt_class.get_options(options[:messages]) + + ai_response = Gitlab::Llm::OpenAi::Client.new(user).chat(content: nil, **options) + + ::Gitlab::Llm::OpenAi::ResponseService.new(user, issue, ai_response, options: {}).execute( + Gitlab::Llm::OpenAi::ResponseModifiers::Chat.new + ) + end + + private + + attr_reader :ai_prompt_class + end + end + end + end +end +``` + +```ruby +# /ee/lib/gitlab/llm/open_ai/templates/amazing_new_ai_feature.rb + +module Gitlab + module Llm + module OpenAi + module Templates + class AmazingNewAiFeature + TEMPERATURE = 0.3 + + def self.get_options(messages) + system_content = <<-TEMPLATE + You are an assistant that writes code for the following input: + """ + TEMPLATE + + { + messages: [ + { role: "system", content: system_content }, + { role: "user", content: messages }, + ], + temperature: TEMPERATURE + } + end + end + end + end + end +end +``` + +Because we support multiple AI providers, you may also use those providers for the same example: + +```ruby +Gitlab::Llm::VertexAi::Client.new(user) +Gitlab::Llm::Anthropic::Client.new(user) +``` + +### Monitoring Ai Actions + +- Error ratio and response latency apdex for each Ai action can be found on [Sidekiq Service dashboard](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1) under "SLI Detail: llm_completion". +- Spent tokens, usage of each Ai feature and other statistics can be found on [periscope dashboard](https://app.periscopedata.com/app/gitlab/1137231/Ai-Features). + +### Add Ai Action to GraphQL + +TODO + +## Security + +Refer to the [secure coding guidelines for Artificial Intelligence (AI) features](../secure_coding_guidelines.md#artificial-intelligence-ai-features). diff --git a/doc/development/ai_features/prompts.md b/doc/development/ai_features/prompts.md new file mode 100644 index 00000000000..f4738055f6b --- /dev/null +++ b/doc/development/ai_features/prompts.md @@ -0,0 +1,28 @@ +--- +stage: AI-powered +group: AI Framework +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Working with AI prompts + +This documentation provides some tips and guidelines for working with AI prompts, particularly aimed at GitLab engineers. The tips are: + +1. Set the tone - Describe how the AI assistant should respond, e.g. "You're a helpful assistant specialized in DevSecOps". Giving context helps the AI provide better answers. This establishes expectations for how the AI should communicate. +1. Be specific - When describing a task, provide lots of details and context to help the AI understand. Give as much specific information as possible. For example, don't just say "summarize this text", provide context like "You are an AI assistant named GitLab Duo. Please read the following text and summarize it in 3 concise sentences focusing on the key points." The more details you provide, the better the AI will perform. +1. Give examples - Provide examples of potential questions and desired answers. This helps the AI give better responses. For instance, you can provide a sample question like "What is the main idea of this text?" and then give the ideal concise summary as an example response. Always give the instructions first, and then provide illustrative examples. +1. Guide the input - Use delimiters to clearly indicate where the user's input starts and ends. The AI needs to know what is input. Make it obvious to the model what text is the user input. +1. Step-by-step reasoning - Ask the AI to explain its reasoning step-by-step. This produces more accurate results. You can get better responses by explicitly asking the model to think through its reasoning step-by-step and show the full explanation. Say something like "Please explain your reasoning step-by-step for how you arrived at your summary:" +1. Allow uncertainty - Tell the AI to say "I don't know" if it is unsure, to avoid hallucinating answers. Give the model an explicit way out if it does not know the answer to avoid false responses. Say "If you do not know the answer, please respond with 'I don't know'". +1. Use positive phrasing - Say what the AI should do, not what it shouldn't do, even when prohibiting actions. Although tricky, use positive language as much as possible, even when restricting behavior. For example, say "Please provide helpful, honest responses" rather than "Do not provide harmful or dishonest responses". +1. Correct language - Use proper English grammar and syntax to help the AI understand. Having technically accurate language and grammar will enable the model to better comprehend the prompt. This is why working with technical writers is very helpful for crafting prompts. +1. Test different models - Prompts are provider specific. Test new models before fully switching. It's important to recognize prompts do not work equally across different AI providers. Make sure to test performance carefully when changing to a new model, don't assume it will work the same. +1. Build quality control - Automate testing prompts with RSpec or Rake task to catch differences. Develop automated checks to regularly test prompts and catch regressions. Use frameworks like RSpec or Rake tasks to build test cases with sample inputs and desired outputs. +1. Iterate - Refine prompts gradually, testing changes to see their impact. Treat prompt engineering as an iterative process. Make small changes, then test results before continuing. Build up prompts incrementally while continually evaluating effects. + +## Further Resources + +For more comprehensive prompt engineering guides, see: + +- [Prompt Engineering Guide 1](https://www.promptingguide.ai/) +- [Prompt Engineering Guide 2](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 440068d55c2..3662b21eb9e 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1058,6 +1058,17 @@ A taxonomic genus. See: [Wikipedia page on genera](https://wikipedia.org/wiki/Ge Multiple documentation references can be provided. The syntax for this property is a `HashMap` where the keys are textual descriptions, and the values are URLs. +### Subscription tier badges + +If a field or argument is available to higher subscription tiers than the other fields, +add the [tier badge](documentation/styleguide/index.md#product-tier-badges) inline. + +For example: + +```ruby +description: '**(ULTIMATE ALL)** Full path of a custom template.' +``` + ## Authorization See: [GraphQL Authorization](graphql_guide/authorization.md) diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index b8af1341919..aeab188fa76 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -55,7 +55,7 @@ With `Gitlab::Audit::Auditor` service, we can instrument audit events in two way ### Using block to record multiple events -This method is useful when events are emitted deep in the call stack. +You can use this method when events are emitted deep in the call stack. For example, we can record multiple audit events when the user updates a merge request approval rule. As part of this user flow, we would like to audit changes @@ -118,7 +118,7 @@ end ### Data volume considerations Because every audit event is persisted to the database, consider the amount of data we expect to generate, and the rate of generation, for new -audit events. For new audit events that will produce a lot of data in the database, consider adding a +audit events. For new audit events that produce a lot of data in the database, consider adding a [streaming-only audit event](#event-streaming) instead. If you have questions about this, feel free to ping `@gitlab-org/govern/compliance/backend` in an issue or merge request. @@ -225,6 +225,22 @@ To add a new audit event type: | `saved_to_database` | yes | Indicate whether to persist events to database and JSON logs | | `streamed` | yes | Indicate that events should be streamed to external services (if configured) | +### Generate documentation + +Audit event types documentation is automatically generated and [published](../../administration/audit_event_streaming/audit_event_types.md) +to the GitLab documentation site. + +If you add a new audit event type, run the +[`gitlab:audit_event_types:compile_docs` Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/audit_event_types/audit_event_types.rake) +to update the documentation: + +```shell +bundle exec rake gitlab:audit_event_types:compile_docs +``` + +Run the [`gitlab:audit_event_types:check_docs` Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/audit_event_types/audit_event_types.rake) +to check if the documentation is up-to-date. + ## Event streaming All events where the entity is a `Group` or `Project` are recorded in the audit log, and also streamed to one or more @@ -233,7 +249,7 @@ All events where the entity is a `Group` or `Project` are recorded in the audit - `Group`, events are streamed to the group's root ancestor's event streaming destinations. - `Project`, events are streamed to the project's root ancestor's event streaming destinations. -You can add streaming-only events that are not stored in the GitLab database. This is primarily intended to be used for actions that generate +You can add streaming-only events that are not stored in the GitLab database. Streaming-only events are primarily intended to be used for actions that generate a large amount of data. See [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76719/diffs#d56e47632f0384722d411ed3ab5b15e947bd2265_26_36) for an example. This feature is under heavy development. Follow the [parent epic](https://gitlab.com/groups/gitlab-org/-/epics/5925) for updates on feature @@ -243,5 +259,5 @@ development. We intentionally do not translate audit event messages because translated messages would be saved in the database and served to users, regardless of their locale settings. -This could mean, for example, that we use the locale for the currently authenticated user to record an audit event message and stream the message to an external streaming +For example, this could mean that we use the locale for the authenticated user to record an audit event message and stream the message to an external streaming destination in the wrong language for that destination. Users could find that confusing. diff --git a/doc/development/avoiding_required_stops.md b/doc/development/avoiding_required_stops.md index 0308e0c30c0..5c2197048b0 100644 --- a/doc/development/avoiding_required_stops.md +++ b/doc/development/avoiding_required_stops.md @@ -10,7 +10,7 @@ Required stops are any changes to GitLab [components](architecture.md) or dependencies that result in the need to upgrade to and stop at a specific `major.minor` version when [upgrading GitLab](../update/index.md). -While Development maintains a [maintainenance policy](../policy/maintenance.md) +While Development maintains a [maintenance policy](../policy/maintenance.md) that results in a three-release (3 month) backport window - GitLab maintains a much longer window of [version support](https://about.gitlab.com/support/statement-of-support/#version-support) that includes the current major version, as well as the two previous major @@ -31,8 +31,16 @@ across greater than 1-3 minor releases. Wherever possible, a required stop should be avoided. If it can't be avoided, the required stop should be aligned to a _scheduled_ required stop. +In cases where we are considering retroactively declaring an unplanned required stop, +please contact the [Distribution team product manager](https://about.gitlab.com/handbook/product/categories/#distributionbuild-group) to advise on next steps. If there +is uncertainty about whether we should declare a required stop, the Distribution product +manager may escalate to GitLab product leadership (VP or Chief Product Officer) to make +a final determination. This may happen, for example, if a change might require a stop for +a small subset of very large self-managed installations and there are well-defined workarounds +if customers run into issues. + Scheduled required stops are often implemented for the previous `major`.`minor` -release just prior to a `major` version release in order to accomodate multiple +release just prior to a `major` version release in order to accommodate multiple [planned deprecations](../update/terminology.md#deprecation) and known [breaking changes](../update/terminology.md#breaking-change). diff --git a/doc/development/build_test_package.md b/doc/development/build_test_package.md index 70aa328bf8a..c4ae0ac5b71 100644 --- a/doc/development/build_test_package.md +++ b/doc/development/build_test_package.md @@ -19,12 +19,23 @@ that will create: commit which triggered the pipeline). When you push a commit to either the GitLab CE or GitLab EE project, the -pipeline for that commit will have a `build-package` manual action you can -trigger. +pipeline for that commit will have a `trigger-omnibus` job in the `qa` stage you +can trigger manually (if it didn't trigger already). - + - +After the child pipeline started, you can select `trigger-omnibus` to go to +the child pipeline named `TRIGGERED_EE_PIPELINE`. + + + +Next, select the `Trigger:package` job in the `trigger-package` stage. + +The `Trigger:package` job when finished will upload its artifacts to GitLab, and +then you can `Browse` them and download the `.deb` file or you can use the +GitLab API to download the file straight to your VM. Keep in mind the expiry of +these artifacts is short, so they will be deleted automatically within a day or +so. ## Specifying versions of components diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 42f4b5dd6f3..16ad7b3eab6 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/cloud_connector/code_suggestions_for_sm.md b/doc/development/cloud_connector/code_suggestions_for_sm.md new file mode 100644 index 00000000000..bd8a39bc0d6 --- /dev/null +++ b/doc/development/cloud_connector/code_suggestions_for_sm.md @@ -0,0 +1,259 @@ +--- +stage: Data Stores +group: Cloud Connector +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Cloud Connector MVC: Code Suggestions for Self-Managed/GitLab Dedicated + +This document presents the systems and their interactions involved in delivering +the [Code Suggestions AI feature](../../user/project/repository/code_suggestions/index.md) +to self-managed and GitLab Dedicated customers. It was considered the MVC +or initial iteration toward a more extensive vision called [GitLab Cloud Connector](https://gitlab.com/groups/gitlab-org/-/epics/308) +(formerly "GitLab Plus"), which will allow self-managed customers to benefit from +features operated by us. + +In the context of this document, and any architectural discussions around Cloud Connector features, +it is important to understand that **Cloud Connector is not a system**. It is an umbrella term +for all the projects we engage in that make existing SaaS-only features available to +self-managed and GitLab Dedicated customers. Some of these may be sufficiently similar to share +solutions, including new systems that deliver these features, but this may not extend to all +features covered by the Cloud Connector umbrella project. + +For the remainder of this document we do not distinguish between self-managed and GitLab Dedicated +use cases, as at the time of this writing, they are identical. We are exploring any necessary +deviations needed specifically for GitLab Dedicated in [issue 410394](https://gitlab.com/gitlab-org/gitlab/-/issues/410394). + +## Problem statement + +Code Suggestions for self-managed users work almost identically to GitLab SaaS: + +1. The user: + 1. Connects their IDE to a GitLab instance. + 1. Requests a code suggestion through their IDE. +1. The GitLab instance: + 1. Authenticates the user and performs instance-local permission and configuration checks. + 1. Produces a [JSON Web Token](https://jwt.io/) to use with the [AI gateway](../../architecture/blueprints/ai_gateway/index.md). + 1. Forwards the request to the AI gateway with this access token. + 1. Returns the response back to the user's IDE. + +The unique challenge we had to solve in the context of self-managed instances is step 2b: +For GitLab SaaS, we can make an instance-local decision about whether a user is allowed to use Code Suggestions +(or any other feature for that matter) and contact the AI gateway, but for self-managed users we cannot. +This is because we cannot trust a self-managed instance to produce such a token as we have no control over them. +In the context of Cloud Connector, this means providing a system for permission delegation that takes into account +self-managed instance billing and license data, so that once we establish that an instance is eligible +for this feature based on its payment history, we can issue an access token to this instance. + +The architecture and data flow for this solution are outlined in the following section. + +## Architecture + +NOTE: +This section covers the architectural details relevant to Code Suggestions in the context of Cloud Connector. +A more high-level overview can be found in [AI Architecture](../ai_architecture.md). + +The Code Suggestions architecture for Cloud Connector serves as a blueprint to implement +similar features in the future. As mentioned above, the primary problem to solve is verifying that a request +coming from a self-managed instance is eligible to obtain Code Suggestions from the AI gateway. This can +be broken down further into two smaller problems: + +1. **Instance eligibility.** Code suggestions are a paid feature. The source of truth for subscription state is the + customers portal (CustomersDot). A self-managed GitLab instance must therefore involve + CustomersDot in the decision for whether an instance is allowed to request code suggestions + on behalf of a user before it forwards this request to the AI gateway. +1. **AI gateway authentication.** Because the AI gateway has no concept of self-managed instance or user identity, + the AI gateway must verify that the instance from which a request originates is legitimate. + This is handled almost identically to GitLab SaaS. The relevant differences are covered in later sections. + +Three systems are involved in the entire flow, with the following responsibilities: + +1. **GitLab Rails application:** + - Authenticates the current user requesting a code suggestion, as it would any other API request. + - Manages and checks instance-level configuration related to Code Suggestions for self-managed installations. + - [Runs a background cron job](#gitlabcustomersdot-token-synchronization) that regularly fetches an JWT access token for use with the AI gateway from CustomersDot. + - Calls the AI gateway with the given JWT access token, potentially enriching the call + with instance-local context. +1. **CustomersDot:** + - Provides a user interface for customers to purchase Code Suggestions. + - When a GitLab instance syncs with CustomersDot, checks whether it + has an active Code Suggestions purchase, and issues a cryptographically signed JWT access token scoped + to the Code Suggestions feature. This mechanism is extensible to other Cloud Connector features. + - Acts as an [OIDC Provider](https://openid.net/developers/how-connect-works/) by offering discovery and configuration endpoints + that serve the public keys required to validate a Code Suggestions JWT. The AI gateway + calls these endpoints (see below.) +1. **AI gateway:** + - Services Code Suggestions requests coming from the GitLab application. + - Synchronizes with CustomersDot OIDC endpoints to obtain token validation keys. + - Validates the JWT access token sent by a GitLab instance using these keys. + This establishes the necessary trust relationship between any GitLab instance + and the AI gateway, which is hosted by us. + +It is important to highlight that from the perspective of the AI gateway, all requests are equal. +As long as the access token can be verified to be authentic, the request will succeed, be that +from a GitLab SaaS or self-managed user. + +The following diagram shows how these systems interact at a high level: + +<!-- + Component architecture sources for PlantUML diagram source code: + https://gitlab.com/gitlab-org/gitlab/-/snippets/3597299 + + The comments do not render correctly in various tested browser, so it is included as a binary image instead. +--> + + +## Implementation details and data flow + +This section breaks down the three primary mechanisms used to deliver Code Suggestions +to self-managed instances: + +1. GitLab/CustomersDot token synchronization. +1. Proxying AI gateway requests. +1. AI gateway access token validation. + +The sequence diagram below shows how a typical flow might look like. + +```mermaid +sequenceDiagram + autonumber + participant U as User + participant VS as IDE + participant SM as GitLab + participant CD as CustomersDot + participant AI as AI gateway<br/>(aka Model Gateway) + + Note over SM,CD: AI gateway token synchronization + loop Sidekiq cron job + SM->>CD: sync subscription seat data + CD->>CD: verify eligibility + Note over CD,SM: Token validity tied to subscription + CD-->>SM: seat data + AI gateway access token (JWT) + SM->>SM: store seat data + JWT + end + Note over U,AI: Developer persona + U->>SM: create PAT + SM-->>U: PAT + U->>VS: configure with PAT + loop Use code suggestions + Note over VS,AI: All requests via AI abstraction layer + VS->>SM: get code suggestions with PAT + SM->>SM: auth user with PAT + SM->>AI: fetch code suggestions with JWT + alt Validation key missing + AI->>CD: fetch JWKS + AI->>AI: cache key set in Redis + else + AI->>AI: load key set + end + AI->>AI: validate JWT with<br/>cached JWKS keys + AI-->>SM: code suggestions + SM-->>VS: code suggestions + end +``` + +### GitLab/CustomersDot token synchronization + +The first problem we had to solve was to determine whether any given self-managed GitLab is +allowed to use Code Suggestions. The mechanism described below was built for Code +Suggestions but could serve any other Cloud Connector feature tied to a Cloud License +subscription. + +The source of truth for this from the perspective of the GitLab Rails application is CustomersDot, +which itself converses with Zuora (a third party subscription service) to determine which +subscriptions or add-ons are active for a given customer. + +Unlike with GitLab SaaS, CustomersDot does not call back into self-managed GitLab instances when +subscriptions or add-ons are purchased. Instead, a daily synchronization job is scheduled in +Sidekiq that compares purchases against actual seat usage using the `/api/v1/seat_links` +REST endpoint on CustomersDot. An instance authenticates itself by posting +its license key as part of the request, which CustomersDot uses to look up subscription data +connected to this license, therefore using the license key as a form of authentication token. + +If CustomersDot deems the instance eligible, it embeds a Code Suggestions token in the response +payload: + +```json +"service_tokens": { + "code_suggestions": { + "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJqdGkiOiI0M2FlOWI4...qmvMVhRS01YRc6a5LaBbhU_m5tw", + "expires_at": 1695121894 + } +} +``` + +- `token` is an [encoded JSON Web Token](https://jwt.io/) signed with RSA256, an asymmetric + secure signing algorithm. This allows other services who hold the corresponding public key + to validate the authenticity of this token. The token carries [claims](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-token-claims#registered-claims) that receivers + can verify, most importantly the `aud` (audience) claim. We use the audience claim to scope + access to particular features, such as Code Suggestions. Each paid feature requires a separate + token to be issued with a corresponding claim. +- `expires_at`: UNIX epoch timestamp of the token's expiration time. Tokens currently have + an expiration time of 3 days. This TTL was chosen to strike a balance between regularly + rolling over access tokens and some leeway in case the token sync fails. + +NOTE: +To sign tokens, CustomersDot maintains a private key. +For security reasons, we rotate this key on a regular basis. For more information, refer +to [the key rotation process for CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/architecture/add_ons/code_suggestions/authorization_for_self_managed.md#jwk-signing-key-rotation). + +Upon receiving a response, GitLab Rails stores this token in Postgres and removes any +other tokens it may have previously stored. When users request a code suggestion, GitLab +can then load this token and attach it to AI gateway requests, which is described next. + +### Proxying AI gateway requests + +Given the JWT access token described above, a GitLab instance is ready to serve Code Suggestions +requests to users. Upon receiving a request to `/api/v4/code_suggestions/completions`, GitLab: + +1. Authenticates this request with a user's Personal Access Token (PAT), as it would any REST API call. + Users configure this token in their IDE settings. +1. Verifies that the administrator has the Code Suggestions feature enabled and the instance + license tier includes this feature. +1. Loads the JWT access token from the database. +1. Forwards the request to the AI gateway with the token attached. + +As with GitLab SaaS requests, the downstream call uses Workhorse's `senddata` feature. This +mechanism yields control to Workhorse by responding with a `send-url` header field. Workhorse +then intercepts this response and calls into the AI gateway given the request URL, headers +and payload provided through `send-url`. This relieves the Puma process from stalling on downstream I/O, +removing a scalability bottleneck. + +This process is mostly identical to GitLab SaaS. The biggest difference is +how GitLab Rails resolves user permissions and loads the access token. GitLab SaaS can self-issue the access token +because billing is not handled by CustomersDot. + +### AI gateway access token validation + +The next problem we had to solve was authenticating the Code Suggestions requests that a self-managed GitLab sends +to the AI gateway on behalf of a user. The AI gateway does not and should not have any knowledge +of a customer's instance. Instead, the gateway verifies the request's authenticity by decoding the JWT +access token enclosed in the request using a public key it fetches from CustomersDot, which is +the original issuer of the token. + +The AI gateway accomplishes this by first requesting a [JSON Web Key Set](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) +from CustomersDot. The keyset obtained from the JWKS endpoint is then cached for 24 hours and used to decode any +JWTs attached to Code Suggestions requests. Note that token expiration is implicitly enforced +in that expired tokens fail to decode, in which case the AI gateway rejects this request. + +The steps to obtain the JWKS and verify tokens are detailed in [the AI service verification sequence diagram](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/architecture/add_ons/code_suggestions/authorization_for_self_managed.md#gitlab-hosted-ai-service-flow-to-verify-jwt-token) for CustomersDot. + +This process is mostly identical to GitLab SaaS. The only difference is that the AI gateway obtains validation keys +from CustomersDot instead of GitLab SaaS, which self-issues its own tokens. + +## Gaps and outlook + +Code Suggestions was the first Cloud Connector feature we looked at, but we expect many more +to be designed and built, some of which may require different technical approaches from what is +documented here. + +Some areas that are not currently well-defined or understood include: + +- Support for GitLab Dedicated and regional deployments. This is currently being investigated in + [issue 410394](https://gitlab.com/gitlab-org/gitlab/-/issues/410394). +- The impact on end-user experience when a GitLab instance is deployed to a geographic region that + has high latency overhead when connecting to a GitLab service in US-east. +- There are some known usability issues with relying solely on a daily Sidekiq job to fetch access + tokens. We are exploring ways to improve this in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/11289). +- Rate-limiting requests at the GitLab instance level was out of scope for the MVC. We are exploring + this idea in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/420123). diff --git a/doc/development/cloud_connector/img/code_suggestions_components.png b/doc/development/cloud_connector/img/code_suggestions_components.png Binary files differnew file mode 100644 index 00000000000..3f41d1d1a6b --- /dev/null +++ b/doc/development/cloud_connector/img/code_suggestions_components.png diff --git a/doc/development/code_review.md b/doc/development/code_review.md index a8c527ad30e..17c16e79232 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -161,12 +161,6 @@ The [Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette/) contains For more information, review [the roulette README](https://gitlab.com/gitlab-org/gitlab-roulette). -As an experiment, we want to introduce a `local` reviewer status for database reviews. Local reviewers are reviewers -focusing on work from a team/stage, but not outside of it. This helps to focus and build great domain -knowledge. We are not introducing changes to the reviewer roulette till we evaluate the impact and feedback from this -experiment. We ask to respect reviewers who decline reviews based on their focus on `local` reviews. For tracking purposes, -please use in your personal YAML file entry: `- reviewer database local` instead of `- reviewer database`. - ### Approval guidelines As described in the section on the responsibility of the maintainer below, you diff --git a/doc/development/code_suggestions/index.md b/doc/development/code_suggestions/index.md new file mode 100644 index 00000000000..38fd6200ace --- /dev/null +++ b/doc/development/code_suggestions/index.md @@ -0,0 +1,56 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Code Suggestions development guidelines + +## Code Suggestions development setup + +The recommended setup for locally developing and debugging Code Suggestions is to have all 3 different components running: + +- IDE Extension (e.g. VSCode Extension) +- Main application configured correctly +- [Model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist) + +This should enable everyone to see locally any change in an IDE being sent to the main application transformed to a prompt which is then sent to the respective model. + +### Setup instructions + +1. Install and run locally the [VSCode Extension](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CONTRIBUTING.md#configuring-development-environment) + 1. Add the ```"gitlab.debug": true,``` info to the Code Suggestions development config + 1. In VSCode navigate to the Extensions page and find "GitLab Workflow" in the list + 1. Open the extension settings by clicking a small cog icon and select "Extension Settings" option + 1. Check a "GitLab: Debug" checkbox. +1. Main Application + 1. Enable Feature Flags ```code_suggestions_completion_api``` and ```code_suggestions_tokens_api``` + 1. In your terminal, navigate to a `gitlab` inside your `gitlab-development-kit` directory + 1. Run `bundle exec rails c` to start a Rails console + 1. Call `Feature.enable(:code_suggestions_completion_api)` and `Feature.enable(:code_suggestions_tokens_api)` from the console + 1. Run the GDK with ```export CODE_SUGGESTIONS_BASE_URL=http://localhost:5052``` +1. [Setup Model Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally) + 1. Build tree sitter libraries ```poetry run scripts/build-tree-sitter-lib.py``` + 1. Extra .env Changes for all debugging insights + 1. LOG_LEVEL=DEBUG + 1. LOG_FORMAT_JSON=false + 1. LOG_TO_FILE=true + 1. Watch the new log file ```modelgateway_debug.log``` , e.g. ```tail -f modelgateway_debug.log | fblog -a prefix -a suffix -a current_file_name -a suggestion -a language -a input -a parameters -a score -a exception``` + +### Setup instructions to use staging Model Gateway + +When testing interactions with the Model Gateway, you might want to integrate your local GDK +with the deployed staging Model Gateway. To do this: + +1. You need a [cloud staging license](../../user/project/repository/code_suggestions/self_managed.md#update-gitlab) that has the Code Suggestions add-on, because add-ons are enabled on staging. Drop a note in the `#s_fulfillment` internal Slack channel to request an add-on to your license. See this [handbook page](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee-developer-licenses) for how to request a license for local development. +1. Set env variables to point customers-dot to staging, and the Model Gateway to staging: + + ```shell + export GITLAB_LICENSE_MODE=test + export CUSTOMER_PORTAL_URL=https://customers.staging.gitlab.com + export CODE_SUGGESTIONS_BASE_URL=https://codesuggestions.staging.gitlab.com + ``` + +1. Restart the GDK. +1. Ensure you followed the necessary [steps to enable the Code Suggestions feature](../../user/project/repository/code_suggestions/self_managed.md#gitlab-163-and-later). +1. Test out the Code Suggestions feature by opening the Web IDE for a project. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index d2063a6836d..4f941b798c1 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -61,7 +61,7 @@ To write and test your code, you will use the GitLab Development Kit. - To run a pre-configured GDK instance in the cloud, use [GDK with Gitpod](../../integration/gitpod.md). From a project repository: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. In the upper right, select **Edit > Gitpod**. 1. If you want to contribute to the [website](https://about.gitlab.com/) or the [handbook](https://about.gitlab.com/handbook/), go to the footer of any page and select **Edit in Web IDE** to open the [Web IDE](../../user/project/web_ide/index.md). diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 371df5b45ff..5a2343e883c 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -14,8 +14,7 @@ requiring downtime. ## Dropping columns -Removing columns is tricky because running GitLab processes may still be using -the columns. To work around this safely, you need three steps in three releases: +Removing columns is tricky because running GitLab processes expect these columns to exist, as ActiveRecord caches the tables schema, even if the columns are not referenced. This happens if the columns are not explicitly marked as ignored. To work around this safely, you need three steps in three releases: 1. [Ignoring the column](#ignoring-the-column-release-m) (release M) 1. [Dropping the column](#dropping-the-column-release-m1) (release M+1) diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index 10490df7b5e..a6d827df820 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -193,9 +193,10 @@ Because batched migrations are update heavy and there were few incidents in the These database indicators are checked to throttle a migration. On getting a stop signal, the migration is paused for a set time (10 minutes): -- WAL queue pending archival crossing a threshold. +- WAL queue pending archival crossing the threshold. - Active autovacuum on the tables on which the migration works on. - Patroni apdex SLI dropping below the SLO. +- WAL rate crossing the threshold. It's an ongoing effort to add more indicators to further enhance the database health check framework. For more details, see diff --git a/doc/development/database/clickhouse/clickhouse_within_gitlab.md b/doc/development/database/clickhouse/clickhouse_within_gitlab.md new file mode 100644 index 00000000000..297776429d7 --- /dev/null +++ b/doc/development/database/clickhouse/clickhouse_within_gitlab.md @@ -0,0 +1,237 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# ClickHouse within GitLab + +This document gives a high-level overview of how to develop features using ClickHouse in the GitLab Rails application. + +NOTE: +Most of the tooling and APIs are considered unstable. + +## GDK setup + +For instructions on how to set up a ClickHouse server locally, see the [ClickHouse installation documentation](https://clickhouse.com/docs/en/install). + +### Configure your Rails application + +1. Copy the example file and configure the credentials: + + ```shell + cp config/click_house.yml.example + config/click_house.yml + ``` + +1. Create the database using the `clickhouse-client` CLI tool: + + ```shell + clickhouse-client --password + ``` + + ```sql + create database gitlab_clickhouse_development; + ``` + +### Validate your setup + +Run the Rails console and invoke a simple query: + +```ruby +ClickHouse::Client.select('SELECT 1', :main) +# => [{"1"=>1}] +``` + +## Database schema and migrations + +For the ClickHouse database there are no established schema migration procedures yet. We have very basic tooling to build up the database schema in the test environment from scratch using timestamp-prefixed SQL files. + +You can create a table by placing a new SQL file in the `db/click_house/main` folder: + +```sql +// 20230811124511_create_issues.sql +CREATE TABLE issues +( + id UInt64 DEFAULT 0, + title String DEFAULT '' +) +ENGINE = MergeTree +PRIMARY KEY (id) +``` + +When you're working locally in your development environment, you can create or re-create your table schema by executing the respective `CREATE TABLE` statement. Alternatively, you can use the following snippet in the Rails console: + +```ruby +require_relative 'spec/support/database/click_house/hooks.rb' + +# Drops and re-creates all tables +ClickHouseTestRunner.new.ensure_schema +``` + +## Writing database queries + +For the ClickHouse database we don't use ORM (Object Relational Mapping). The main reason is that the GitLab application has many customizations for the `ActiveRecord` PostgresSQL adapter and the application generally assumes that all databases are using `PostgreSQL`. Since ClickHouse-related features are still in a very early stage of development, we decided to implement a simple HTTP client to avoid hard to discover bugs and long debugging time when dealing with multiple `ActiveRecord` adapters. + +Additionally, ClickHouse might not be used the same way as other adapters for `ActiveRecord`. The access patterns differ from traditional transactional databases, in that ClickHouse: + +- Uses nested aggregation `SELECT` queries with `GROUP BY` clauses. +- Doesn't use single `INSERT` statements. Data is inserted in batches via background jobs. +- Has different consistency characteristics, no transactions. +- Has very little database-level validations. + +Database queries are written and executed with the help of the `ClickHouse::Client` gem. + +A simple query from the `events` table: + +```ruby +rows = ClickHouse::Client.select('SELECT * FROM events', :main) +``` + +When working with queries with placeholders you can use the `ClickHouse::Query` object where you need to specify the placeholder name and its data type. The actual variable replacement, quoting and escaping will be done by the ClickHouse server. + +```ruby +raw_query = 'SELECT * FROM events WHERE id > {min_id:UInt64}' +placeholders = { min_id: Integer(100) } +query = ClickHouse::Client::Query.new(raw_query: raw_query, placeholders: placeholders) + +rows = ClickHouse::Client.select(query, :main) +``` + +When using placeholders the client can provide the query with redacted placeholder values which can be ingested by our logging system. You can see the redacted version of your query by calling the `to_redacted_sql` method: + +```ruby +puts query.to_redacted_sql +``` + +ClickHouse allows only one statement per request. This means that the common SQL injection vulnerability where the statement is closed with a `;` character and then another query is "injected" cannot be exploited: + +```ruby +ClickHouse::Client.select('SELECT 1; SELECT 2', :main) + +# ClickHouse::Client::DatabaseError: Code: 62. DB::Exception: Syntax error (Multi-statements are not allowed): failed at position 9 (end of query): ; SELECT 2. . (SYNTAX_ERROR) (version 23.4.2.11 (official build)) +``` + +### Subqueries + +You can compose complex queries with the `ClickHouse::Client::Query` class by specifying the query placeholder with the special `Subquery` type. The library will make sure to correctly merge the queries and the placeholders: + +```ruby +subquery = ClickHouse::Client::Query.new(raw_query: 'SELECT id FROM events WHERE id = {id:UInt64}', placeholders: { id: Integer(10) }) + +raw_query = 'SELECT * FROM events WHERE id > {id:UInt64} AND id IN ({q:Subquery})' +placeholders = { id: Integer(10), q: subquery } + +query = ClickHouse::Client::Query.new(raw_query: raw_query, placeholders: placeholders) +rows = ClickHouse::Client.select(query, :main) + +# ClickHouse will replace the placeholders +puts query.to_sql # SELECT * FROM events WHERE id > {id:UInt64} AND id IN (SELECT id FROM events WHERE id = {id:UInt64}) + +puts query.to_redacted_sql # SELECT * FROM events WHERE id > $1 AND id IN (SELECT id FROM events WHERE id = $2) + +puts query.placeholders # { id: 10 } +``` + +In case there are placeholders with the same name but different values the query will raise an error. + +### Writing query conditions + +When working with complex forms where multiple filter conditions are present, building queries by concatenating query fragments as string can get out of hands very quickly. For queries with several conditions you may use the `ClickHouse::QueryBuilder` class. The class uses the `Arel` gem to generate queries and provides a similar query interface like `ActiveRecord`. + +```ruby +builder = ClickHouse::QueryBuilder.new('events') + +query = builder + .where(builder.table[:created_at].lteq(Date.today)) + .where(id: [1,2,3]) + +rows = ClickHouse::Client.select(query, :main) +``` + +## Inserting data + +The ClickHouse client supports inserting data through the standard query interface: + +```ruby +raw_query = 'INSERT INTO events (id, target_type) VALUES ({id:UInt64}, {target_type:String})' +placeholders = { id: 1, target_type: 'Issue' } + +query = ClickHouse::Client::Query.new(raw_query: raw_query, placeholders: placeholders) +rows = ClickHouse::Client.execute(query, :main) +``` + +Inserting data this way is acceptable if: + +- The table contains settings or configuration data where we need to add one row. +- For testing, test data has to be prepared in the database. + +When inserting data, we should always try to use batch processing where multiple rows are inserted at once. Building large `INSERT` queries in memory is discouraged because of the increased memory usage. Additionally, values specified within such queries cannot be redacted automatically by the client. + +To compress data and reduce memory usage, insert CSV data. You can do this with the internal [`CsvBuilder`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/csv_builder) gem: + +```ruby +iterator = Event.find_each + +# insert from events table using only the id and the target_type columns +column_mapping = { + id: :id, + target_type: :target_type +} + +CsvBuilder::Gzip.new(iterator, column_mapping).render do |tempfile| + query = 'INSERT INTO events (id, target_type) FORMAT CSV' + ClickHouse::Client.insert_csv(query, File.open(tempfile.path), :main) +end +``` + +NOTE: +It's important to test and verify efficient batching of database records from PostgreSQL. Consider using the techniques described in the [Iterating tables in batches](../iterating_tables_in_batches.md). + +## Testing + +ClickHouse is enabled on CI/CD but to avoid significantly affecting the pipeline runtime we've decided to run the ClickHouse server for test cases tagged with `:click_house` only. + +The `:click_house` tag ensures that the database schema is properly set up before every test case. + +```ruby +RSpec.describe MyClickHouseFeature, :click_house do + it 'returns rows' do + rows = ClickHouse::Client.select('SELECT 1', :main) + expect(rows.size).to eq(1) + end +end +``` + +## Multiple databases + +By design, the `ClickHouse::Client` library supports configuring multiple databases. Because we're still at a very early stage of development, we only have one database called `main`. + +Multi database configuration example: + +```yaml +development: + main: + database: gitlab_clickhouse_main_development + url: 'http://localhost:8123' + username: clickhouse + password: clickhouse + + user_analytics: # made up database + database: gitlab_clickhouse_user_analytics_development + url: 'http://localhost:8123' + username: clickhouse + password: clickhouse +``` + +## Observability + +All queries executed via the `ClickHouse::Client` library expose the query with performance metrics (timings, read bytes) via `ActiveSupport::Notifications`. + +```ruby +ActiveSupport::Notifications.subscribe('sql.click_house') do |_, _, _, _, data| + puts data.inspect +end +``` + +Additionally, to view the executed ClickHouse queries in web interactions, on the performance bar, next to the `ch` label select the count. diff --git a/doc/development/database/foreign_keys.md b/doc/development/database/foreign_keys.md index 5dda3dd55a3..84ab32d0c0b 100644 --- a/doc/development/database/foreign_keys.md +++ b/doc/development/database/foreign_keys.md @@ -195,5 +195,5 @@ end ``` Using a foreign key as primary key saves space but can make -[batch counting](../internal_analytics/service_ping/implement.md#batch-counters) in [Service Ping](../service_ping/index.md) less efficient. +[batch counting](../internal_analytics/service_ping/implement.md#batch-counters) in [Service Ping](../internal_analytics/service_ping/index.md) less efficient. Consider using a regular `id` column if the table is relevant for Service Ping. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 1ee6aeaa213..70681994229 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -109,6 +109,7 @@ including the major methods: ## ClickHouse - [Introduction](clickhouse/index.md) +- [ClickHouse within GitLab](clickhouse/clickhouse_within_gitlab.md) - [Optimizing query execution](clickhouse/optimization.md) - [Rebuild GitLab features using ClickHouse 1: Activity data](clickhouse/gitlab_activity_data.md) - [Rebuild GitLab features using ClickHouse 2: Merge Request analytics](clickhouse/merge_request_analytics.md) @@ -118,3 +119,4 @@ including the major methods: - [Maintenance operations](maintenance_operations.md) - [Update multiple database objects](setting_multiple_values.md) +- [Batch iteration in a tree hierarchy proof of concept](poc_tree_iterator.md) diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 7037ab22983..4387e19b6df 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -11,6 +11,9 @@ To allow GitLab to scale further we The two databases are `main` and `ci`. GitLab supports being run with either one database or two databases. On GitLab.com we are using two separate databases. +For the purpose of building the [Cells](../../architecture/blueprints/cells/index.md) architecture, we are decomposing +the databases further, to introduce another database `gitlab_main_clusterwide`. + ## GitLab Schema For properly discovering allowed patterns between different databases @@ -23,17 +26,22 @@ that we cannot use PostgreSQL schema due to complex migration procedures. Instea the concept of application-level classification. Each table of GitLab needs to have a `gitlab_schema` assigned: -- `gitlab_main`: describes all tables that are being stored in the `main:` database (for example, like `projects`, `users`). -- `gitlab_ci`: describes all CI tables that are being stored in the `ci:` database (for example, `ci_pipelines`, `ci_builds`). -- `gitlab_geo`: describes all Geo tables that are being stored in the `geo:` database (for example, like `project_registry`, `secondary_usage_data`). -- `gitlab_shared`: describes all application tables that contain data across all decomposed databases (for example, `loose_foreign_keys_deleted_records`) for models that inherit from `Gitlab::Database::SharedModel`. -- `gitlab_internal`: describes all internal tables of Rails and PostgreSQL (for example, `ar_internal_metadata`, `schema_migrations`, `pg_*`). -- `gitlab_pm`: describes all tables that store `package_metadata` (it is an alias for `gitlab_main`). -- `...`: more schemas to be introduced with additional decomposed databases +| Database | Description | Notes | +| -------- | ----------- | ------- | +| `gitlab_main`| All tables that are being stored in the `main:` database. | Currently, this is being replaced with `gitlab_main_cell`, for the purpose of building the [Cells](../../architecture/blueprints/cells/index.md) architecture. `gitlab_main_cell` schema describes all tables that are local to a cell in a GitLab installation. For example, `projects` and `groups` | +| `gitlab_main_clusterwide` | All tables that are being stored cluster-wide in a GitLab installation, in the [Cells](../../architecture/blueprints/cells/index.md) architecture. For example, `users` and `application_settings` | | +| `gitlab_ci` | All CI tables that are being stored in the `ci:` database (for example, `ci_pipelines`, `ci_builds`) | | +| `gitlab_geo` | All Geo tables that are being stored in the `geo:` database (for example, like `project_registry`, `secondary_usage_data`) | | +| `gitlab_shared` | All application tables that contain data across all decomposed databases (for example, `loose_foreign_keys_deleted_records`) for models that inherit from `Gitlab::Database::SharedModel`. | | +| `gitlab_internal` | All internal tables of Rails and PostgreSQL (for example, `ar_internal_metadata`, `schema_migrations`, `pg_*`) | | +| `gitlab_pm` | All tables that store `package_metadata`| It is an alias for `gitlab_main`| + +More schemas to be introduced with additional decomposed databases The usage of schema enforces the base class to be used: -- `ApplicationRecord` for `gitlab_main` +- `ApplicationRecord` for `gitlab_main`/`gitlab_main_cell.` +- `MainClusterwide::ApplicationRecord` for `gitlab_main_clusterwide`. - `Ci::ApplicationRecord` for `gitlab_ci` - `Geo::TrackingBase` for `gitlab_geo` - `Gitlab::Database::SharedModel` for `gitlab_shared` @@ -465,6 +473,20 @@ You can see a real example of using this method for fixing a cross-join in #### Allowlist for existing cross-joins +The easiest way of identifying a cross-join is via failing pipelines. + +As an example, in [!130038](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130038/diffs) we moved the `notification_settings` table to the `gitlab_main_cell` schema, by marking it as such in the `db/docs/notification_settings.yml` file. + +The pipeline failed with the following [error](https://gitlab.com/gitlab-org/gitlab/-/jobs/4929130983): + +```ruby +Database::PreventCrossJoins::CrossJoinAcrossUnsupportedTablesError: + +Unsupported cross-join across 'users, notification_settings' querying 'gitlab_main_clusterwide, gitlab_main_cell' discovered when executing query 'SELECT "users".* FROM "users" WHERE "users"."id" IN (SELECT "notification_settings"."user_id" FROM ((SELECT "notification_settings"."user_id" FROM "notification_settings" WHERE "notification_settings"."source_id" = 119 AND "notification_settings"."source_type" = 'Project' AND (("notification_settings"."level" = 3 AND EXISTS (SELECT true FROM "notification_settings" "notification_settings_2" WHERE "notification_settings_2"."user_id" = "notification_settings"."user_id" AND "notification_settings_2"."source_id" IS NULL AND "notification_settings_2"."source_type" IS NULL AND "notification_settings_2"."level" = 2)) OR "notification_settings"."level" = 2))) notification_settings)' +``` + +To make the pipeline green, this cross-join query must be allow-listed. + A cross-join across databases can be explicitly allowed by wrapping the code in the `::Gitlab::Database.allow_cross_joins_across_databases` helper method. Alternative way is to mark a given relation as `relation.allow_cross_joins_across_databases`. @@ -494,6 +516,30 @@ def find_actual_head_pipeline end ``` +In model associations or scopes, this can be used as in the following example: + +```ruby +class Group < Namespace + has_many :users, -> { + allow_cross_joins_across_databases(url: "https://gitlab.com/gitlab-org/gitlab/-/issues/422405") + }, through: :group_members +end +``` + +WARNING: +Overriding an association can have unintended consequences and may even lead to data loss, as we noticed in [issue 424307](https://gitlab.com/gitlab-org/gitlab/-/issues/424307). Do not override existing ActiveRecord associations to mark a cross-join as allowed, as in the example below. + +```ruby +class Group < Namespace + has_many :users, through: :group_members + + # DO NOT override an association like this. + def users + super.allow_cross_joins_across_databases(url: "https://gitlab.com/gitlab-org/gitlab/-/issues/422405") + end +end +``` + The `url` parameter should point to an issue with a milestone for when we intend to fix the cross-join. If the cross-join is being used in a migration, we do not need to fix the code. See <https://gitlab.com/gitlab-org/gitlab/-/issues/340017> @@ -530,7 +576,42 @@ more information, look at the [transaction guidelines](transaction_guidelines.md#dangerous-example-third-party-api-calls) page. -#### Fixing cross-database errors +#### Fixing cross-database transactions + +A transaction across databases can be explicitly allowed by wrapping the code in the +`Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification.temporary_ignore_tables_in_transaction` helper method. + +For cross-database transactions in Rails callbacks, the `cross_database_ignore_tables` method can be used. + +These methods should only be used for existing code. + +The `temporary_ignore_tables_in_transaction` helper method can be used as follows: + +```ruby +class GroupMember < Member + def update_two_factor_requirement + return unless user + + # To mark and ignore cross-database transactions involving members and users/user_details/user_preferences + Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification.temporary_ignore_tables_in_transaction( + %w[users user_details user_preferences], url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/424288' + ) do + user.update_two_factor_requirement + end + end +end +``` + +The `cross_database_ignore_tables` method can be used as follows: + +```ruby +class Namespace < ApplicationRecord + include CrossDatabaseIgnoredTables + + # To mark and ignore cross-database transactions involving namespaces and routes/redirect_routes happening within Rails callbacks. + cross_database_ignore_tables %w[routes redirect_routes], url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/424277' +end +``` ##### Removing the transaction block @@ -616,6 +697,23 @@ or records that point to nowhere, which might lead to bugs. As such we created ["loose foreign keys"](loose_foreign_keys.md) which is an asynchronous process of cleaning up orphaned records. +### Allowlist for existing cross-database foreign keys + +The easiest way of identifying a cross-database foreign key is via failing pipelines. + +As an example, in [!130038](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130038/diffs) we moved the `notification_settings` table to the `gitlab_main_cell` schema, by marking it in the `db/docs/notification_settings.yml` file. + +`notification_settings.user_id` is a column that points to `users`, but the `users` table belongs to a different database, thus this is now treated as a cross-database foreign key. + +We have a spec to capture such cases of cross-database foreign keys in [`no_cross_db_foreign_keys_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/01d3a1e41513200368a22bbab5d4312174762ee0/spec/lib/gitlab/database/no_cross_db_foreign_keys_spec.rb), which would fail if such a cross-database foreign key is encountered. + +To make the pipeline green, this cross-database foreign key must be allow-listed. + +To do this, explicitly allow the existing cross-database foreign key to exist by adding it as an exception in the same spec (as in [this example](https://gitlab.com/gitlab-org/gitlab/-/blob/7d99387f399c548af24d93d564b35f2f9510662d/spec/lib/gitlab/database/no_cross_db_foreign_keys_spec.rb#L26)). +This way, the spec will not fail. + +Later, this foreign key can be converted to a loose foreign key, like we did in [!130080](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130080/diffs). + ## Testing for multiple databases In our testing CI pipelines, we test GitLab by default with multiple databases set up, using diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index e1b6868c68e..05b1081fc4d 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -72,8 +72,6 @@ The steps required are: Depending on the size of the table, a background migration for cleanup could be required in the next release. See the [`NOT NULL` constraints on large tables](not_null_constraints.md#not-null-constraints-on-large-tables) section for more information. - - Create an issue for the next milestone to validate the `NOT NULL` constraint. - 1. Release `N.M+1` (next release) 1. Make sure all existing records on GitLab.com have attribute set. If not, go back to step 1 from Release `N.M`. diff --git a/doc/development/database/poc_tree_iterator.md b/doc/development/database/poc_tree_iterator.md new file mode 100644 index 00000000000..453f77f0cde --- /dev/null +++ b/doc/development/database/poc_tree_iterator.md @@ -0,0 +1,475 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Batch iteration in a tree hierarchy (proof of concept) + +The group hierarchy in GitLab is represented with a tree, where the root element +is the top-level namespace, and the child elements are the subgroups or the +recently introduced `Namespaces::ProjectNamespace` records. + +The tree is implemented in the `namespaces` table ,via the `parent_id` column. +The column points to the parent namespace record. The top level namespace has no +`parent_id`. + +Partial hierarchy of `gitlab-org`: + +```mermaid +flowchart TD + A("gitlab-org (9979)") --- B("quality (2750817)") + B --- C("engineering-productivity (16947798)") + B --- D("performance-testing (9453799)") + A --- F("charts (5032027)") + A --- E("ruby (14018648)") +``` + +Efficiently iterating over the group hierarchy has several potential use cases. +This is true especially in background jobs, which need to perform queries on the group hierarchy, +where stable and safe execution is more important than fast runtime. Batch iteration +requires more network round-trips, but each batch provides similar performance +characteristics. + +A few examples: + +- For each subgroup, do something. +- For each project in the hierarchy, do something. +- For each issue in the hierarchy, do something. + +## Problem statement + +A group hierarchy could grow so big that a single query would not be able to load +it in time. The query would fail with statement timeout error. + +Addressing scalability issues related to very large groups requires us to store +the same data in different formats (de-normalization). However, if we're unable +to load the group hierarchy, then de-normalization could not be implemented. + +One de-normalization technique would be to store all descendant group IDs for a +given group. This would speed up queries where we need to load the group and its +subgroups. Example: + +```mermaid +flowchart TD + A(1) --- B(2) + A --- C(3) + C --- D(4) +``` + +| GROUP_ID | DESCENDANT_GROUP_IDS | +|----------|------------------------| +| 1 | `[2,3,4]` | +| 2 | `[]` | +| 3 | `[4]` | +| 4 | `[]` | + +With this structure, determining all the subgroups would require us to read only +one row from the database, instead of 4 rows. For a hierarchy as big as 1000 groups, +this could make a huge difference. + +The reading of the hierarchy problem is solved with this de-normalization. However, +we still need to find a way to persist this data in a table. Because a group and +its hierarchy could grow very large, we cannot expect a single query to work here. + +```sql +SELECT id FROM namespaces WHERE traversal_ids && ARRAY[9970] +``` + +The query above could time out for large groups, so we need to process the data in batches. + +Implementing batching logic in a tree is not something we've looked at before, +and it's fairly complex to implement. An `EachBatch` or `find_in_batches` based +solution would not work because: + +- The data (group IDs) are not sorted in the hierarchy. +- Groups in sub groups don't know about the top-level group ID. + +## Algorithm + +The batching query is implemented as a recursive CTE SQL query, where one batch +would read a maximum of N rows. Due to the tree structure, reading N rows might +not necessarily mean that we're reading N group IDs. If the tree is structured in +a non-optimal way, a batch could return less (but never more) group IDs. + +The query implements a [depth-first](https://en.wikipedia.org/wiki/Depth-first_search) +tree walking logic, where the DB scans the first branch of the tree until the leaf +element. We're implementing depth-first algorithm because, when a batch is finished, +the query must return enough information for the next batch (cursor). In GitLab, +we limit the depth of the tree to 20, which means that in the worst case, the +query would return a cursor containing 19 elements. + +Implementing a [breadth-first](https://en.wikipedia.org/wiki/Breadth-first_search) +tree walking algorithm would be impractical, because a group can have unbounded +number of descendants, thus we might end up with a huge cursor. + +1. Create an initializer row that contains: + 1. The currently processed group ID (top-level group ID) + 1. Two arrays (tree depth and the collected IDs) + 1. A counter for tracking the number of row reads in the query. +1. Recursively process the row and do one of the following (whenever the condition matches): + - Load the first child namespace and update the currently processed namespace + ID if we're not at the leaf node. (Walking down a branch) + - Load the next namespace record on the current depth if there are any rows left. + - Walk up one node and process rows at one level higher. +1. Continue the processing until the number of reads reaches our `LIMIT` (batch size). +1. Find the last processed row which contains the data for the cursor, and all the collected record IDs. + +```sql +WITH RECURSIVE result AS ( + ( + SELECT + 9970 AS current_id, /* current namespace id we're processing */ + ARRAY[9970]::int[] AS depth, /* cursor */ + ARRAY[9970]::int[] AS ids, /* collected ids */ + 1::bigint AS reads, + 'initialize' AS action + ) UNION ALL + ( + WITH cte AS ( /* trick for referencing the result cte multiple times */ + select * FROM result + ) + SELECT * FROM ( + ( + SELECT /* walk down the branch */ + namespaces.id, + cte.depth || namespaces.id, + cte.ids || namespaces.id, + cte.reads + 1, + 'walkdown' + FROM namespaces, cte + WHERE + namespaces.parent_id = cte.current_id + ORDER BY namespaces.id ASC + LIMIT 1 + ) UNION ALL + ( + SELECT /* find next element on the same level */ + namespaces.id, + cte.depth[:array_length(cte.depth, 1) - 1] || namespaces.id, + cte.ids || namespaces.id, + cte.reads + 1, + 'next' + FROM namespaces, cte + WHERE + namespaces.parent_id = cte.depth[array_length(cte.depth, 1) - 1] AND + namespaces.id > cte.depth[array_length(cte.depth, 1)] + ORDER BY namespaces.id ASC + LIMIT 1 + ) UNION ALL + ( + SELECT /* jump up one node when finished with the current level */ + cte.current_id, + cte.depth[:array_length(cte.depth, 1) - 1], + cte.ids, + cte.reads + 1, + 'jump' + FROM cte + WHERE cte.depth <> ARRAY[]::int[] + LIMIT 1 + ) + ) next_row LIMIT 1 + ) +) +SELECT current_id, depth, ids, action +FROM result +``` + +```plaintext + current_id | depth | ids | action +------------+--------------+------------------------+------------ + 24 | {24} | {24} | initialize + 25 | {24,25} | {24,25} | walkdown + 26 | {24,26} | {24,25,26} | next + 112 | {24,112} | {24,25,26,112} | next + 113 | {24,113} | {24,25,26,112,113} | next + 114 | {24,113,114} | {24,25,26,112,113,114} | walkdown + 114 | {24,113} | {24,25,26,112,113,114} | jump + 114 | {24} | {24,25,26,112,113,114} | jump + 114 | {} | {24,25,26,112,113,114} | jump +``` + +NOTE: +Using this query to find all the namespace IDs in a group hierarchy is likely slower +than other querying methods, such as the current `self_and_descendants` implementation +based on the `traversal_ids` column. The query above should be only used when +implementing batch iteration over the group hierarchy. + +Rudimentary batching implementation in Ruby: + +```ruby +class NamespaceEachBatch + def initialize(namespace_id:, cursor: nil) + @namespace_id = namespace_id + @cursor = cursor || { current_id: namespace_id, depth: [namespace_id] } + end + + def each_batch(of: 500) + current_cursor = cursor.dup + + first_iteration = true + loop do + new_cursor, ids = load_batch(cursor: current_cursor, of: of, first_iteration: first_iteration) + first_iteration = false + current_cursor = new_cursor + + yield ids + + break if new_cursor[:depth].empty? + end + end + + private + + # yields array of namespace ids + def load_batch(cursor:, of:, first_iteration: false) + recursive_cte = Gitlab::SQL::RecursiveCTE.new(:result, + union_args: { remove_order: false, remove_duplicates: false }) + + ids = first_iteration ? namespace_id.to_s : "" + + recursive_cte << Namespace.select( + Arel.sql(Integer(cursor.fetch(:current_id)).to_s).as('current_id'), + Arel.sql("ARRAY[#{cursor.fetch(:depth).join(',')}]::int[]").as('depth'), + Arel.sql("ARRAY[#{ids}]::int[]").as('ids'), + Arel.sql("1::bigint AS count") + ).from('(VALUES (1)) AS does_not_matter').limit(1) + + cte = Gitlab::SQL::CTE.new(:cte, Namespace.select('*').from('result')) + + union_query = Namespace.with(cte.to_arel).from_union( + walk_down, + next_elements, + up_one_level, + remove_duplicates: false, + remove_order: false + ).select('current_id', 'depth', 'ids', 'count').limit(1) + + recursive_cte << union_query + + scope = Namespace.with + .recursive(recursive_cte.to_arel) + .from(recursive_cte.alias_to(Namespace.arel_table)) + .limit(of) + row = Namespace.from(scope.arel.as('namespaces')).order(count: :desc).limit(1).first + + [ + { current_id: row[:current_id], depth: row[:depth] }, + row[:ids] + ] + end + + attr_reader :namespace_id, :cursor + + def walk_down + Namespace.select( + Arel.sql('namespaces.id').as('current_id'), + Arel.sql('cte.depth || namespaces.id').as('depth'), + Arel.sql('cte.ids || namespaces.id').as('ids'), + Arel.sql('cte.count + 1').as('count') + ).from('cte, LATERAL (SELECT id FROM namespaces WHERE parent_id = cte.current_id ORDER BY id LIMIT 1) namespaces') + end + + def next_elements + Namespace.select( + Arel.sql('namespaces.id').as('current_id'), + Arel.sql('cte.depth[:array_length(cte.depth, 1) - 1] || namespaces.id').as('depth'), + Arel.sql('cte.ids || namespaces.id').as('ids'), + Arel.sql('cte.count + 1').as('count') + ).from('cte, LATERAL (SELECT id FROM namespaces WHERE namespaces.parent_id = cte.depth[array_length(cte.depth, 1) - 1] AND namespaces.id > cte.depth[array_length(cte.depth, 1)] ORDER BY id LIMIT 1) namespaces') + end + + def up_one_level + Namespace.select( + Arel.sql('cte.current_id').as('current_id'), + Arel.sql('cte.depth[:array_length(cte.depth, 1) - 1]').as('depth'), + Arel.sql('cte.ids').as('ids'), + Arel.sql('cte.count + 1').as('count') + ).from('cte') + .where('cte.depth <> ARRAY[]::int[]') + .limit(1) + end +end + +iterator = NamespaceEachBatch.new(namespace_id: 9970) +all_ids = [] +iterator.each_batch do |ids| + all_ids.concat(ids) +end + +# Test +puts all_ids.count +puts all_ids.sort == Namespace.where('traversal_ids && ARRAY[9970]').pluck(:id).sort +``` + +Example batch query: + +```sql +SELECT + "namespaces".* +FROM ( WITH RECURSIVE "result" AS (( + SELECT + 15847356 AS current_id, + ARRAY[9970, + 12061481, + 12128714, + 12445111, + 15847356]::int[] AS depth, + ARRAY[]::int[] AS ids, + 1::bigint AS count + FROM ( + VALUES (1)) AS does_not_matter + LIMIT 1) + UNION ALL ( WITH "cte" AS MATERIALIZED ( + SELECT + * + FROM + result +) + SELECT + current_id, + depth, + ids, + count + FROM (( + SELECT + namespaces.id AS current_id, + cte.depth || namespaces.id AS depth, + cte.ids || namespaces.id AS ids, + cte.count + 1 AS count + FROM + cte, + LATERAL ( + SELECT + id + FROM + namespaces + WHERE + parent_id = cte.current_id + ORDER BY + id + LIMIT 1 +) namespaces +) + UNION ALL ( + SELECT + namespaces.id AS current_id, + cte.depth[:array_length( + cte.depth, 1 +) - 1] || namespaces.id AS depth, + cte.ids || namespaces.id AS ids, + cte.count + 1 AS count + FROM + cte, + LATERAL ( + SELECT + id + FROM + namespaces + WHERE + namespaces.parent_id = cte.depth[array_length( + cte.depth, 1 +) - 1] + AND namespaces.id > cte.depth[array_length( + cte.depth, 1 +)] + ORDER BY + id + LIMIT 1 +) namespaces +) + UNION ALL ( + SELECT + cte.current_id AS current_id, + cte.depth[:array_length( + cte.depth, 1 +) - 1] AS depth, + cte.ids AS ids, + cte.count + 1 AS count + FROM + cte + WHERE ( + cte.depth <> ARRAY[]::int[] +) + LIMIT 1 +) +) namespaces + LIMIT 1 +)) +SELECT + "namespaces".* +FROM + "result" AS "namespaces" +LIMIT 500) namespaces +ORDER BY + "count" DESC +LIMIT 1 +``` + +Execution plan: + +```plaintext + Limit (cost=16.36..16.36 rows=1 width=76) (actual time=436.963..436.970 rows=1 loops=1) + Buffers: shared hit=3721 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + -> Sort (cost=16.36..16.39 rows=11 width=76) (actual time=436.961..436.968 rows=1 loops=1) + Sort Key: namespaces.count DESC + Sort Method: top-N heapsort Memory: 27kB + Buffers: shared hit=3721 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + -> Limit (cost=15.98..16.20 rows=11 width=76) (actual time=0.005..436.394 rows=500 loops=1) + Buffers: shared hit=3718 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + CTE result + -> Recursive Union (cost=0.00..15.98 rows=11 width=76) (actual time=0.003..432.924 rows=500 loops=1) + Buffers: shared hit=3718 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + -> Limit (cost=0.00..0.01 rows=1 width=76) (actual time=0.002..0.003 rows=1 loops=1) + I/O Timings: read=0.000 write=0.000 + -> Result (cost=0.00..0.01 rows=1 width=76) (actual time=0.001..0.002 rows=1 loops=1) + I/O Timings: read=0.000 write=0.000 + -> Limit (cost=0.76..1.57 rows=1 width=76) (actual time=0.862..0.862 rows=1 loops=499) + Buffers: shared hit=3718 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + CTE cte + -> WorkTable Scan on result (cost=0.00..0.20 rows=10 width=76) (actual time=0.000..0.000 rows=1 loops=499) + I/O Timings: read=0.000 write=0.000 + -> Append (cost=0.56..17.57 rows=21 width=76) (actual time=0.862..0.862 rows=1 loops=499) + Buffers: shared hit=3718 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 + -> Nested Loop (cost=0.56..7.77 rows=10 width=76) (actual time=0.675..0.675 rows=0 loops=499) + Buffers: shared hit=1693 read=357 dirtied=1 + I/O Timings: read=327.812 write=0.000 + -> CTE Scan on cte (cost=0.00..0.20 rows=10 width=76) (actual time=0.001..0.001 rows=1 loops=499) + I/O Timings: read=0.000 write=0.000 + -> Limit (cost=0.56..0.73 rows=1 width=4) (actual time=0.672..0.672 rows=0 loops=499) + Buffers: shared hit=1693 read=357 dirtied=1 + I/O Timings: read=327.812 write=0.000 + -> Index Only Scan using index_namespaces_on_parent_id_and_id on public.namespaces namespaces_1 (cost=0.56..5.33 rows=29 width=4) (actual time=0.671..0.671 rows=0 loops=499) + Index Cond: (namespaces_1.parent_id = cte.current_id) + Heap Fetches: 7 + Buffers: shared hit=1693 read=357 dirtied=1 + I/O Timings: read=327.812 write=0.000 + -> Nested Loop (cost=0.57..9.45 rows=10 width=76) (actual time=0.208..0.208 rows=1 loops=442) + Buffers: shared hit=2025 read=66 dirtied=7 + I/O Timings: read=84.778 write=0.000 + -> CTE Scan on cte cte_1 (cost=0.00..0.20 rows=10 width=72) (actual time=0.000..0.000 rows=1 loops=442) + I/O Timings: read=0.000 write=0.000 + -> Limit (cost=0.57..0.89 rows=1 width=4) (actual time=0.203..0.203 rows=1 loops=442) + Buffers: shared hit=2025 read=66 dirtied=7 + I/O Timings: read=84.778 write=0.000 + -> Index Only Scan using index_namespaces_on_parent_id_and_id on public.namespaces namespaces_2 (cost=0.57..3.77 rows=10 width=4) (actual time=0.201..0.201 rows=1 loops=442) + Index Cond: ((namespaces_2.parent_id = (cte_1.depth)[(array_length(cte_1.depth, 1) - 1)]) AND (namespaces_2.id > (cte_1.depth)[array_length(cte_1.depth, 1)])) + Heap Fetches: 35 + Buffers: shared hit=2025 read=66 dirtied=6 + I/O Timings: read=84.778 write=0.000 + -> Limit (cost=0.00..0.03 rows=1 width=76) (actual time=0.003..0.003 rows=1 loops=59) + I/O Timings: read=0.000 write=0.000 + -> CTE Scan on cte cte_2 (cost=0.00..0.29 rows=9 width=76) (actual time=0.002..0.002 rows=1 loops=59) + Filter: (cte_2.depth <> '{}'::integer[]) + Rows Removed by Filter: 0 + I/O Timings: read=0.000 write=0.000 + -> CTE Scan on result namespaces (cost=0.00..0.22 rows=11 width=76) (actual time=0.005..436.240 rows=500 loops=1) + Buffers: shared hit=3718 read=423 dirtied=8 + I/O Timings: read=412.590 write=0.000 +``` diff --git a/doc/development/database_review.md b/doc/development/database_review.md index bb0bfbc759b..ba0423a1a0d 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -173,10 +173,11 @@ Include in the MR description: ##### Query Plans - The query plan for each raw SQL query included in the merge request along with the link to the query plan following each raw SQL snippet. -- Provide a link to the plan generated using the `explain` command in the [postgres.ai](database/database_lab.md) chatbot. - - If it's not possible to get an accurate picture in Database Lab, you may need to seed a development environment, and instead provide links - from [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com). Be sure to paste both the plan - and the query used in the form. +- Provide a link to the plan generated using the `explain` command in the [postgres.ai](database/database_lab.md) chatbot. The `explain` command runs + `EXPLAIN ANALYZE`. + - If it's not possible to get an accurate picture in Database Lab, you may need to + seed a development environment, and instead provide output + from `EXPLAIN ANALYZE`. Create links to the plan using [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com). Be sure to paste both the plan and the query used in the form. - When providing query plans, make sure it hits enough data: - To produce a query plan with enough data, you can use the IDs of: - The `gitlab-org` namespace (`namespace_id = 9970`), for queries involving a group. @@ -192,6 +193,13 @@ Include in the MR description: plan _before_ and _after_ the change. This helps spot differences quickly. - Include data that shows the performance improvement, preferably in the form of a benchmark. +- When evaluating a query plan, we need the final query to be + executed against the database. We don't need to analyze the intermediate + queries returned as `ActiveRecord::Relation` from finders and scopes. + PostgreSQL query plans are dependent on all the final parameters, + including limits and other things that may be added before final execution. + One way to be sure of the actual query executed is to check + `log/development.log`. #### Preparation when adding foreign keys to existing tables diff --git a/doc/development/development_seed_files.md b/doc/development/development_seed_files.md new file mode 100644 index 00000000000..2bf3688fd48 --- /dev/null +++ b/doc/development/development_seed_files.md @@ -0,0 +1,26 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Development seed files + +Development seed files are listed under `gitlab/db/fixtures/development/` and `gitlab/ee/db/fixtures/development/` +folders. These files are used to populate the database with records to help verifying if feature functionalities, like charts, are working as expected on local host. + +The task `rake db:seed_fu` can be used to run all development seeds with the exception of the ones under a flag which is usually passed as an environment variable. + +The following table summarizes the seeds and tasks that can be used to generate +data for features. + +| Feature | Command | Seed | +|-------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| +| DevOps Adoption | `FILTER=devops_adoption bundle exec rake db:seed_fu` | [31_devops_adoption.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/db/fixtures/development/31_devops_adoption.rb) | +| Value Streams Dashboard | `FILTER=cycle_analytics SEED_VSA=1 bundle exec rake db:seed_fu` | [17_cycle_analytics.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/fixtures/development/17_cycle_analytics.rb) | +| Value Stream Analytics | `FILTER=customizable_cycle_analytics SEED_CUSTOMIZABLE_CYCLE_ANALYTICS=1 bundle exec rake db:seed_fu` | [30_customizable_cycle_analytics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/db/fixtures/development/30_customizable_cycle_analytics.rb) | +| CI/CD analytics | `FILTER=ci_cd_analytics SEED_CI_CD_ANALYTICS=1 bundle exec rake db:seed_fu` | [38_ci_cd_analytics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/fixtures/development/38_ci_cd_analytics.rb?ref_type=heads) | +| Contributions Analytics<br><br>Productivity Analytics<br><br>Code review Analytics<br><br>Merge Request Analytics | `FILTER=productivity_analytics SEED_PRODUCTIVITY_ANALYTICS=1 bundle exec rake db:seed_fu` | [90_productivity_analytics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/db/fixtures/development/90_productivity_analytics.rb) | +| Repository Analytics | `FILTER=14_pipelines NEW_PROJECT=1 bundle exec rake db:seed_fu` | [14_pipelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/fixtures/development/14_pipelines.rb?ref_type=heads) | +| Issue Analytics<br><br>Insights | `NEW_PROJECT=1 bin/rake gitlab:seed:insights:issues` | [insights Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/seed/insights.rake) | +| DORA metrics | `SEED_DORA=1 FILTER=dora_metrics bundle exec rake db:seed_fu` | [92_dora_metrics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/db/fixtures/development/92_dora_metrics.rb) | diff --git a/doc/development/documentation/alpha_beta.md b/doc/development/documentation/alpha_beta.md index 61f07e79e12..4579c57b448 100644 --- a/doc/development/documentation/alpha_beta.md +++ b/doc/development/documentation/alpha_beta.md @@ -1,49 +1,11 @@ --- -info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects -stage: none -group: unassigned +redirect_to: 'experiment_beta.md' +remove_date: '2023-11-29' --- -# Documenting Experiment and Beta features +This document was moved to [another location](experiment_beta.md). -Some features are not generally available and are instead considered -[Experiment or Beta](../../policy/experiment-beta-support.md). - -When you document a feature in one of these three statuses: - -- Add `(Experiment)` or `(Beta)` in parentheses after the page or topic title. -- Do not include `(Experiment)` or `(Beta)` in the left nav. -- Ensure the version history lists the feature's status. - -These features are usually behind a feature flag, which follow [these documentation guidelines](feature_flags.md). - -If you add details of how users should enroll, or how to contact the team with issues, -the `FLAG:` note should be above these details. - -For example: - -```markdown -## Great new feature (Experiment) - -> [Introduced](link) in GitLab 15.10. This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). - -FLAG: -On self-managed GitLab, by default this feature is not available. -To make it available, an administrator can enable the feature flag named `example_flag`. -On GitLab.com, this feature is not available. This feature is not ready for production use. - -Use this great new feature when you need to do this new thing. - -This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). To join -the list of users testing this feature, do this thing. If you find a bug, -[open an issue](link). -``` - -When the feature is ready for production, remove: - -- The text in parentheses. -- Any language about the feature not being ready for production in the body - description. -- The feature flag information if available. - -Ensure the version history is up-to-date by adding a note about the production release. +<!-- This redirect file can be deleted after <2023-11-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/documentation/experiment_beta.md b/doc/development/documentation/experiment_beta.md new file mode 100644 index 00000000000..fab78082cb5 --- /dev/null +++ b/doc/development/documentation/experiment_beta.md @@ -0,0 +1,49 @@ +--- +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects +stage: none +group: unassigned +--- + +# Documenting Experiment and Beta features + +Some features are not generally available and are instead considered +[Experiment or Beta](../../policy/experiment-beta-support.md). + +When you document a feature in one of these three statuses: + +- Add the tier badge after the page or topic title. +- Do not include `(Experiment)` or `(Beta)` in the left nav. +- Ensure the version history lists the feature's status. + +These features are usually behind a feature flag, which follow [these documentation guidelines](feature_flags.md). + +If you add details of how users should enroll, or how to contact the team with issues, +the `FLAG:` note should be above these details. + +For example: + +```markdown +## Great new feature **(EXPERIMENT)** + +> [Introduced](link) in GitLab 15.10. This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, an administrator can enable the feature flag named `example_flag`. +On GitLab.com, this feature is not available. This feature is not ready for production use. + +Use this great new feature when you need to do this new thing. + +This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). To join +the list of users testing this feature, do this thing. If you find a bug, +[open an issue](link). +``` + +When the feature is ready for production, remove: + +- The text in parentheses. +- Any language about the feature not being ready for production in the body + description. +- The feature flag information if available. + +Ensure the version history is up-to-date by adding a note about the production release. diff --git a/doc/development/documentation/help.md b/doc/development/documentation/help.md index fb730aca6f0..a921429bf49 100644 --- a/doc/development/documentation/help.md +++ b/doc/development/documentation/help.md @@ -28,9 +28,6 @@ For example: 1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff for 14.4. -The exact cutoff date for each release is flexible, and can be sooner or later -than expected due to holidays, weekends or other events. In general, MRs merged -by the 17th should be present in the release on the 22nd, though it is not guaranteed. If it is important that a documentation update is present in that month's release, merge it as early as possible. diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index a5d565ffa79..a53330a7e63 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -146,7 +146,7 @@ Sort the table by required attributes first, then alphabetically. | Attribute | Type | Required | Description | |------------------------------|---------------|----------|-----------------------------------------------------| | `title` | string | Yes | Title of the issue. | -| `assignee_ids` **(PREMIUM)** | integer array | No | IDs of the users to assign the issue to. | +| `assignee_ids` **(PREMIUM ALL)** | integer array | No | IDs of the users to assign the issue to. | | `confidential` | boolean | No | Sets the issue to confidential. Default is `false`. | ``` @@ -155,7 +155,7 @@ Rendered example: | Attribute | Type | Required | Description | |------------------------------|---------------|----------|-----------------------------------------------------| | `title` | string | Yes | Title of the issue. | -| `assignee_ids` **(PREMIUM)** | integer array | No | IDs of the users to assign the issue to. | +| `assignee_ids` **(PREMIUM ALL)** | integer array | No | IDs of the users to assign the issue to. | | `confidential` | boolean | No | Sets the issue to confidential. Default is `false`. | For information about writing attribute descriptions, see the [GraphQL API description style guide](../api_graphql_styleguide.md#description-style-guide). @@ -181,7 +181,7 @@ Sort the table alphabetically. ```markdown | Attribute | Type | Description | |------------------------------|---------------|-------------------------------------------| -| `assignee_ids` **(PREMIUM)** | integer array | IDs of the users to assign the issue to. | +| `assignee_ids` **(PREMIUM ALL)** | integer array | IDs of the users to assign the issue to. | | `confidential` | boolean | Whether the issue is confidential or not. | | `title` | string | Title of the issue. | ``` @@ -190,7 +190,7 @@ Rendered example: | Attribute | Type | Description | |------------------------------|---------------|-------------------------------------------| -| `assignee_ids` **(PREMIUM)** | integer array | IDs of the users to assign the issue to. | +| `assignee_ids` **(PREMIUM ALL)** | integer array | IDs of the users to assign the issue to. | | `confidential` | boolean | Whether the issue is confidential or not. | | `title` | string | Title of the issue. | diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index adc9d727844..483145d1f44 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -16,6 +16,7 @@ Review apps are enabled for the following projects: - [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) - [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) - [GitLab Charts](https://gitlab.com/gitlab-org/charts/gitlab) +- [GitLab Operator](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator) Alternatively, check the [`gitlab-docs` development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/README.md#development-when-contributing-to-gitlab-documentation) or [the GDK documentation](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md) diff --git a/doc/development/documentation/site_architecture/automation.md b/doc/development/documentation/site_architecture/automation.md new file mode 100644 index 00000000000..5b2b02ad97e --- /dev/null +++ b/doc/development/documentation/site_architecture/automation.md @@ -0,0 +1,77 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Automated pages + +Most pages in the GitLab documentation are written manually in Markdown. +However, some pages are created by automated processes. + +Two primary categories of automation exist in the GitLab documentation: + +- Content that is generated by using a standard process and structured data (for example, YAML or JSON files). +- Content that is generated by any other means. + +Automation helps with consistency and speed. But content that is automated in a +non-standard way causes difficulty with: + +- Frontend changes. +- Site troubleshooting and maintenance. +- The contributor experience. + +Ideally, any automation should be done in a standard way, which helps alleviate some of the downsides. + +## Pages generated from structured data + +Some functionality on the docs site uses structured data: + +- Hierarchical global navigation (YAML) +- Survey banner (YAML) +- Badges (YAML) +- Homepage content lists (YAML) +- Redirects (YAML) +- Versions menu (JSON) + +## Pages generated otherwise + +Other pages are generated by using non-standard processes. These pages often use solutions +that are coded across multiple repositories. + +| Page | Details | Owner | +|---|---|---| +| [All feature flags in GitLab](../../../user/feature_flags.md) | [Generated during docs build](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/raketasks.md#generate-the-feature-flag-tables) | [Technical Writing](https://about.gitlab.com/handbook/product/ux/technical-writing/) | +| [GitLab Runner feature flags](https://docs.gitlab.com/runner/configuration/feature-flags.html) | [Page source](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/ec6e1797d2173a95c8ac7f726bd62f6f110b7211/docs/configuration/feature-flags.md?plain=1#L39) | [Runner](https://about.gitlab.com/handbook/engineering/development/ops/verify/runner/) | +| [Deprecations and removals by version](../../../update/deprecations.md) | [Deprecating GitLab features](../../deprecation_guidelines/index.md) | | +| [GraphQL API resources](../../../api/graphql/reference/index.md) | [GraphQL API style guide](../../api_graphql_styleguide.md#documentation-and-schema) | [Import and Integrate](https://about.gitlab.com/handbook/engineering/development/dev/manage/import-and-integrate/) | +| [Audit event types](../../../administration/audit_event_streaming/audit_event_types.md) | [Audit event development guidelines](../../audit_event_guide/index.md) | [Compliance](https://about.gitlab.com/handbook/engineering/development/sec/govern/compliance/) | +| DAST vulnerability check documentation ([Example](../../../user/application_security/dast/checks/798.19.md)) | [How to generate the Markdown](https://gitlab.com/gitlab-org/security-products/dast-cwe-checks/-/blob/main/doc/how-to-generate-the-markdown-documentation.md) | [Dynamic Analysis](https://about.gitlab.com/handbook/product/categories/#dynamic-analysis-group) | +| Blueprints ([Example](../../../architecture/blueprints/ci_data_decay/pipeline_partitioning.md)) | | | +| [The docs homepage](../../../index.md) | | [Technical Writing](https://about.gitlab.com/handbook/product/ux/technical-writing/) | + +## Make an automation request + +If you want to automate a page on the docs site: + +1. Review [issue 823](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/823) + and consider adding feedback there. +1. If that issue does not describe what you need, contact + [the DRI for the docs site backend](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects). + +Because automation adds extra complexity and a support burden, we +review it on a case-by-case basis. + +## Document the automation + +If you do add automation, you must document: + +- The list of files that are included. +- The `.gitlab-ci.yml` updates and any pipeline requirements. +- The steps needed to troubleshoot. + +Other GitLab team members should be able to easily find information about how to maintain the automation. +You should announce the change widely, including, at a minimum: + +- In Slack, in `#whats-happening-at-gitlab`. +- In the Technical Writer team meeting agenda. diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 2ba69ca0987..767fdf907d6 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -66,12 +66,12 @@ graph TD C["14.2 MR merged"] D["13.12 MR merged"] E["12.10 MR merged"] - F{{"Container registry on `gitlab-docs` project"}} - A--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.4` image"-->F - B--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.3` image"-->F - C--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.2` image"-->F - D--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:13.12` image"-->F - E--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:12.10` image"-->F + F{{"Container registry on gitlab-docs project"}} + A--"image:docs-single<br>job runs and pushes<br>gitlab-docs:14.4 image"-->F + B--"image:docs-single<br>job runs and pushes<br>gitlab-docs:14.3 image"-->F + C--"image:docs-single<br>job runs and pushes<br>gitlab-docs:14.2 image"-->F + D--"image:docs-single<br>job runs and pushes<br>gitlab-docs:13.12 image"-->F + E--"image:docs-single<br>job runs and pushes<br>gitlab-docs:12.10 image"-->F ``` ### Rebuild stable documentation images @@ -104,23 +104,23 @@ For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/ ```mermaid graph TD - A["Latest `gitlab`, `gitlab-runner`<br>`omnibus-gitlab`, and `charts`"] - subgraph "Container registry on `gitlab-docs` project" - B["14.4 versioned docs<br>`gitlab-docs:14.4`"] - C["14.3 versioned docs<br>`gitlab-docs:14.3`"] - D["14.2 versioned docs<br>`gitlab-docs:14.2`"] - E["13.12 versioned docs<br>`gitlab-docs:13.12`"] - F["12.10 versioned docs<br>`gitlab-docs:12.10`"] + A["Latest gitlab, gitlab-runner<br>omnibus-gitlab, and charts"] + subgraph "Container registry on gitlab-docs project" + B["14.4 versioned docs<br>gitlab-docs:14.4"] + C["14.3 versioned docs<br>gitlab-docs:14.3"] + D["14.2 versioned docs<br>gitlab-docs:14.2"] + E["13.12 versioned docs<br>gitlab-docs:13.12"] + F["12.10 versioned docs<br>gitlab-docs:12.10"] end - G[["Scheduled pipeline<br>`image:docs-latest` job<br>combines all these"]] + G[["Scheduled pipeline<br>image:docs-latest job<br>combines all these"]] A--"Default branches<br>pulled down"-->G - B--"`gitlab-docs:14.4` image<br>pulled down"-->G - C--"`gitlab-docs:14.3` image<br>pulled down"-->G - D--"`gitlab-docs:14.2` image<br>pulled down"-->G - E--"`gitlab-docs:13.12` image<br>pulled down"-->G - F--"`gitlab-docs:12.10` image<br>pulled down"-->G + B--"gitlab-docs:14.4 image<br>pulled down"-->G + C--"gitlab-docs:14.3 image<br>pulled down"-->G + D--"gitlab-docs:14.2 image<br>pulled down"-->G + E--"gitlab-docs:13.12 image<br>pulled down"-->G + F--"gitlab-docs:12.10 image<br>pulled down"-->G H{{"Container registry on gitlab-docs project"}} - G--"Latest `gitlab-docs:latest` image<br>pushed up"-->H + G--"Latest gitlab-docs:latest image<br>pushed up"-->H ``` ## Pages deploy job @@ -144,7 +144,7 @@ graph LR A{{"Container registry on gitlab-docs project"}} B[["Scheduled pipeline<br>`pages` and<br>`pages:deploy` job"]] C([docs.gitlab.com]) - A--"`gitlab-docs:latest`<br>pulled"-->B + A--"gitlab-docs:latest<br>pulled"-->B B--"Unpacked documentation uploaded"-->C ``` diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 94bc6bba240..0fa5819acae 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -62,20 +62,45 @@ the documentation helps others efficiently accomplish tasks and solve problems. ## Writing for localization -The GitLab documentation is not localized, but we follow guidelines that -help benefit translation. For example, we: +The GitLab documentation is not localized, but we follow guidelines that help us write for a global audience. -- Write in [active voice](word_list.md#active-voice). -- Write in [present tense](word_list.md#future-tense). -- Avoid words that can be translated incorrectly, like: - - [since and because](word_list.md#since) - - [once and after](word_list.md#once) - - [it](word_list.md#it) -- Avoid [-ing](word_list.md#-ing-words) words. +[The GitLab voice](#the-gitlab-voice) dictates that we write clearly and directly with translation in mind. +Our style guide, [word list](word_list.md), and [Vale rules](../testing.md) ensure consistency in the documentation. -[The GitLab voice](#the-gitlab-voice) dictates that we write clearly and directly, -and with translation in mind. [The word list](word_list.md) and our Vale rules -also aid in consistency, which is important for localization. +When documentation is translated into other languages, the meaning of each word must be clear. +The increasing use of machine translation, GitLab Duo Chat, and other AI tools +means that consistency is even more important. + +The following rules can help documentation be translated more efficiently. + +Avoid: + +- Phrases that hide the subject like [**there is** and **there are**](word_list.md#there-is-there-are). +- Ambiguous pronouns like [**it**](word_list.md#it). +- Words that end in [**-ing**](word_list.md#-ing-words). +- Words that can be confused with one another like [**since**](word_list.md#since) and **because**. +- Latin abbreviations like [**e.g.**](word_list.md#eg) and [**i.e.**](word_list.md#ie). +- Culture-specific references like **kill two birds with one stone**. + +Use: + +- Standard [text for links](#text-for-links). +- [Lists](#lists) and [tables](#tables) instead of complex sentences and paragraphs. +- Common abbreviations like [**AI**](word_list.md#ai-artificial-intelligence) and + [**CI/CD**](word_list.md#cicd) and abbreviations you've previously spelled out. + +Also, keep the following guidance in mind: + +- Be consistent with [feature names](#feature-names) and how to interact with them. +- Break up noun strings. For example, instead of **project integration custom settings**, + use **custom settings for project integrations**. +- Format [dates and times](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/term-collections/date-time-terms) + consistently and for an international audience. +- Use [images](#images), including screenshots, sparingly. +- For [UI text](#ui-text), allow for up to 30% expansion and contraction in translation. + To see how much a string expands or contracts in another language, paste the string + into [Google Translate](https://translate.google.com/) and review the results. + You can ask a colleague who speaks the language to verify if the translation is clear. ## Markdown @@ -240,12 +265,13 @@ create an issue or an MR to propose a change to the user interface text. #### Feature names -- Feature names are typically lowercase. -- Some features require title case, typically nouns that name GitLab-specific capabilities or tools. Features requiring - title case should be: - - Added as a proper name to markdownlint [configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.yml), - so that it can be consistently applied across all documentation. - - Added to the [word list](word_list.md). +Feature names should be lowercase. + +However, in a few rare cases, features can be title case. These exceptions are: + +- Added as a proper name to [markdownlint](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.yml), + so they can be consistently applied across all documentation. +- Added to the [word list](word_list.md). If the term is not in the word list, ask a GitLab Technical Writer for advice. @@ -449,7 +475,7 @@ For example: cp <your_source_directory> <your_destination_directory> ``` -If the placeholder is not in a code block, use [`<` and `>`] and wrap the placeholder +If the placeholder is not in a code block, use `<` and `>` and wrap the placeholder in a single backtick. For example: ```plaintext @@ -738,7 +764,7 @@ For example: ```html <html> -<small>Footnotes +<small>Footnotes: <ol> <li>This is the footnote.</li> <li>This is the other footnote.</li> @@ -755,7 +781,7 @@ This text renders as this output: | App B | Description text. <sup>2</sup> | <html> -<small>Footnotes +<small>Footnotes: <ol> <li>This is the footnote.</li> <li>This is the other footnote.</li> @@ -984,7 +1010,7 @@ To be consistent, use these templates when you write navigation steps in a task To open project settings: ```markdown -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. ``` @@ -992,7 +1018,7 @@ To open project settings: To open group settings: ```markdown -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. ``` @@ -1000,7 +1026,7 @@ To open group settings: To open either project or group settings: ```markdown -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. ``` @@ -1020,14 +1046,14 @@ To create a group: To open the Admin Area: ```markdown -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. ``` To open the **Your work** menu item: ```markdown -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. ``` @@ -1049,15 +1075,15 @@ To save the selection in some dropdown lists: To view all your projects: ```markdown -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. ``` To view all your groups: ```markdown -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. ``` ### Optional steps @@ -1089,7 +1115,7 @@ Use the phrase **Complete the fields**. For example: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Push rules**. 1. Complete the fields. @@ -1641,7 +1667,7 @@ The H1 tier badge should be the badge that applies to the lowest tier for the fe #### Available product tier badges -Tier badges must include two components, in this order: a subscription tier and an offering. +Tier badges should include two components, in this order: a subscription tier and an offering. These components are surrounded by bold and parentheses, for example `**(ULTIMATE SAAS)**`. Subscription tiers: @@ -1661,10 +1687,14 @@ You can also add a third component for the feature's status: - `EXPERIMENT` - `BETA` +For example, `**(FREE ALL EXPERIMENT)**`. + +A tier or status can stand alone. An offering should always have a tier. + #### Add a tier badge To add a tier badge to a topic title, add the two relevant components -after the title text. You must include the subscription tier first, and then the offering. +after the title text. You should include the subscription tier first, and then the offering. For example: ```markdown @@ -1677,6 +1707,12 @@ Optionally, you can add the feature status as the last part of the badge: # Topic title **(FREE ALL EXPERIMENT)** ``` +Or add the status by itself: + +```markdown +# Topic title **(EXPERIMENT)** +``` + ##### Inline tier badges Do not add tier badges inline with other text, except for [API attributes](../restful_api_styleguide.md). diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index d65df0b56c8..509cabbe631 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -22,7 +22,9 @@ For guidance not on this page, we defer to these style guides: - [Google Developer Documentation Style Guide](https://developers.google.com/style) <!-- vale off --> -<!-- markdownlint-disable --> + +<!-- Disable trailing punctuation in heading rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md026---trailing-punctuation-in-heading --> +<!-- markdownlint-disable MD026 --> ## `&` @@ -85,7 +87,7 @@ is passive. `Zombies select the button` is active. ## Admin Area -Use title case for **Admin Area** to refer to the area of the UI that you access when you select **Main menu > Admin**. +Use title case for **Admin Area**. This area of the UI says **Admin Area** at the top of the page and on the menu. ## administrator @@ -233,7 +235,6 @@ Use **text box** to refer to the UI field. Do not use **field** or **box**. For - In the **Variable name** text box, enter a value. - ## bullet Don't refer to individual items in an ordered or unordered list as **bullets**. Use **list item** instead. If you need to be less ambiguous, you can use: @@ -263,6 +264,9 @@ See also [contractions](index.md#contractions). Use **Chat** with a capital `c` for **Chat** or **GitLab Duo Chat**. +On first use on a page, use **GitLab Duo Chat**. +Thereafter, use **Chat** by itself. + ## checkbox Use one word for **checkbox**. Do not use **check box**. @@ -306,9 +310,30 @@ This version is different than the larger, more monolithic **Linux package** tha You can also use **cloud-native GitLab** for short. It should be hyphenated and lowercase. +## Code explanation + +Use sentence case for **Code explanation**. + +On first mention on a page, use **GitLab Duo Code explanation**. +Thereafter, use **Code explanation** by itself. + +## Code review summary + +Use sentence case for **Code review summary**. + +On first mention on a page, use **GitLab Duo Code review summary**. +Thereafter, use **Code review summary** by itself. + ## Code Suggestions -Use title case for **Code Suggestions**. +Use title case for **Code Suggestions**. On first mention on a page, use **GitLab Duo Code Suggestions**. + +**Code Suggestions** should always be plural, and is capitalized even if it's generic. + +Examples: + +- Use Code Suggestions to display suggestions as you type. (This phrase describes the feature.) +- As you type, Code Suggestions are displayed. (This phrase is generic but still uses capital letters.) ## collapse @@ -438,6 +463,13 @@ Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`Inclusi Use **prevent** instead of **disallow**. ([Vale](../testing.md#vale) rule: [`Substitutions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Substitutions.yml)) +## Discussion summary + +Use sentence case for **Discussion summary**. + +On first mention on a page, use **GitLab Duo Discussion summary**. +Thereafter, use **Discussion summary** by itself. + ## Docker-in-Docker, `dind` Use **Docker-in-Docker** when you are describing running a Docker container by using the Docker executor. @@ -580,7 +612,7 @@ Instead of: However, you can make an exception when you are writing a task and you need to refer to all of the fields at once. For example: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. Complete the fields. @@ -621,6 +653,13 @@ For **GB** and **MB**, follow the [Microsoft guidance](https://learn.microsoft.c Use title case for **Geo**. +## Git suggestions + +Use sentence case for **Git suggestions**. + +On first mention on a page, use **GitLab Duo Git suggestions**. +Thereafter, use **Git suggestions** by itself. + ## GitLab Do not make **GitLab** possessive (GitLab's). This guidance follows [GitLab Trademark Guidelines](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/brand/brand-activation/trademark-guidelines/). @@ -633,6 +672,24 @@ Do not use **Dedicated** by itself. Always use **GitLab Dedicated**. Do not use **Duo** by itself. Always use **GitLab Duo**. +On first use on a page, use **GitLab Duo `<featurename>`**. For example: + +- GitLab Duo Chat +- GitLab Duo Code Suggestions +- GitLab Duo Suggested Reviewers +- GitLab Duo Value stream forecasting +- GitLab Duo Discussion summary +- GitLab Duo Merge request summary +- GitLab Duo Code review summary +- GitLab Duo Code explanation +- GitLab Duo Vulnerability summary +- GitLab Duo Test generation +- GitLab Duo Git suggestions +- GitLab Duo Root cause analysis +- GitLab Duo Issue description generation + +After the first use, use the feature name without **GitLab Duo**. + ## GitLab Flavored Markdown When possible, spell out [**GitLab Flavored Markdown**](../../../user/markdown.md). @@ -757,7 +814,7 @@ Do not use **in order to**. Use **to** instead. ([Vale](../testing.md#vale) rule For the plural of **index**, use **indexes**. -However, for ElasticSearch, use [**indices**](https://www.elastic.co/blog/what-is-an-elasticsearch-index). +However, for Elasticsearch, use [**indices**](https://www.elastic.co/blog/what-is-an-elasticsearch-index). ## Installation from source @@ -792,6 +849,13 @@ Use lowercase for **issue**. Use lowercase for **issue board**. +## Issue description generation + +Use sentence case for **Issue description generation**. + +On first mention on a page, use **GitLab Duo Issue description generation**. +Thereafter, use **Issue description generation** by itself. + ## issue weights Use lowercase for **issue weights**. @@ -860,13 +924,13 @@ When writing about licenses: Use: - - Add a license to your instance. - - Purchase a subscription. +- Add a license to your instance. +- Purchase a subscription. Instead of: - - Buy a license. - - Purchase a license. +- Buy a license. +- Purchase a license. ## limitations @@ -956,6 +1020,13 @@ the user account becomes a **member**. Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it out on first use. +## Merge request summary + +Use sentence case for **Merge request summary**. + +On first mention on a page, use **GitLab Duo Merge request summary**. +Thereafter, use **Merge request summary** by itself. + ## milestones Use lowercase for **milestones**. @@ -1277,6 +1348,13 @@ Do not use **roles** and [**permissions**](#permissions) interchangeably. Each u Roles are not the same as [**access levels**](#access-level). +## Root cause analysis + +Use sentence case for **Root cause analysis**. + +On first mention on a page, use **GitLab Duo Root cause analysis**. +Thereafter, use **Root cause analysis** by itself. + ## roll back Use **roll back** for changing a GitLab version to an earlier one. @@ -1454,6 +1532,17 @@ To describe tiers: | In the Premium tier or higher | In the Premium and Ultimate tier | | In the Premium tier or lower | In the Free and Premium tier | +## Suggested Reviewers + +Use title case for **Suggested Reviewers**. On first mention on a page, use **GitLab Duo Suggested Reviewers**. + +**Suggested Reviewers** should always be plural, and is capitalized even if it's generic. + +Examples: + +- Suggested Reviewers can recommend a person to review your merge request. (This phrase describes the feature.) +- As you type, Suggested Reviewers are displayed. (This phrase is generic but still uses capital letters.) + ## that Do not use **that** when describing a noun. For example: @@ -1482,6 +1571,13 @@ talking about non-specific modules. For example: - You can publish a Terraform module to your project's Terraform Module Registry. +## Test generation + +Use sentence case for **Test generation**. + +On first mention on a page, use **GitLab Duo Test generation**. +Thereafter, use **Test generation** by itself. + ## text box Use **text box** instead of **field** or **box** when referring to the UI element. @@ -1620,10 +1716,23 @@ When you add a **user account** to a group or project, the user account becomes Do not use **utilize**. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) +## Value stream forecasting + +Use sentence case for **Value stream forecasting**. On first mention on a page, use **GitLab Duo Value stream forecasting**. + +Thereafter, use **Value stream forecasting** by itself. + ## via Do not use Latin abbreviations. Use **with**, **through**, or **by using** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) +## Vulnerability summary + +Use sentence case for **Vulnerability summary**. + +On first mention on a page, use **GitLab Duo Vulnerability summary**. +Thereafter, use **Vulnerability summary** by itself. + ## we Try to avoid **we** and focus instead on how the user can accomplish something in GitLab. diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 0c65e008436..c0f1d0028f9 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -507,7 +507,21 @@ To configure markdownlint in your editor, install one of the following as approp To configure Vale in your editor, install one of the following as appropriate: -- Sublime Text [`SublimeLinter-vale` package](https://packagecontrol.io/packages/SublimeLinter-vale). +- Sublime Text [`SublimeLinter-vale` package](https://packagecontrol.io/packages/SublimeLinter-vale). To have Vale + suggestions appears as blue instead of red (which is how errors appear), add `vale` configuration to your + [SublimeLinter](http://sublimelinter.readthedocs.org) configuration: + + ```json + "vale": { + "styles": [{ + "mark_style": "outline", + "scope": "region.bluish", + "types": ["suggestion"] + }] + } + ``` + +- [LSP for Sublime Text](https://lsp.sublimetext.io) package [`LSP-vale-ls`](https://packagecontrol.io/packages/LSP-vale-ls). - Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index 87ce4d770f5..7fb4201ac40 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -43,7 +43,7 @@ Prerequisites: To create an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. In the upper-right corner, select **New issue**. 1. Complete the fields. (If you have reference content that lists each field, link to it here.) diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 9219fcd6710..2bf8ad81ba4 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -49,7 +49,7 @@ version of the product: 1. Enable **Allow use of licensed EE features** to make licensed EE features available to projects only if the project namespace's plan includes the feature. - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Account and limit**. @@ -57,7 +57,7 @@ version of the product: 1. Select **Save changes**. 1. Ensure the group you want to test the EE feature for is actually using an EE plan: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Groups**. 1. Identify the group you want to modify, and select **Edit**. @@ -463,7 +463,7 @@ end When it's not possible/logical to modify the implementation of a method, then wrap it in a self-descriptive method and use that method. -For example, in GitLab-FOSS, the only user created by the system is `User.ghost` +For example, in GitLab-FOSS, the only user created by the system is `Users::Internal.ghost` but in EE there are several types of bot-users that aren't really users. It would be incorrect to override the implementation of `User#ghost?`, so instead we add a method `#internal?` to `app/models/user.rb`. The implementation: diff --git a/doc/development/event_store.md b/doc/development/event_store.md index c54e6ae2d07..918da8fb738 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -300,6 +300,21 @@ executed synchronously every time the given event is published. For complex conditions it's best to subscribe to all the events and then handle the logic in the `handle_event` method of the subscriber worker. +### Delayed dispatching of events + +A subscription can specify a delay when to receive an event: + +```ruby +store.subscribe ::MergeRequests::UpdateHeadPipelineWorker, + to: ::Ci::PipelineCreatedEvent, + delay: 1.minute +``` + +The `delay` parameter switches the dispatching of the event to use `perform_in` method +on the subscriber Sidekiq worker, instead of `perform_async`. + +This technique is useful when publishing many events and leverage the Sidekiq deduplication. + ## Testing ### Testing the publisher diff --git a/doc/development/experiment_guide/implementing_experiments.md b/doc/development/experiment_guide/implementing_experiments.md index 6fe58a1da54..61a46397390 100644 --- a/doc/development/experiment_guide/implementing_experiments.md +++ b/doc/development/experiment_guide/implementing_experiments.md @@ -281,7 +281,7 @@ about contexts now. We can assume we run the experiment in one or a few places, but track events potentially in many places. The tracking call remains the same, with the arguments you would usually use when -[tracking events using snowplow](../snowplow/index.md). The easiest example +[tracking events using snowplow](../internal_analytics/snowplow/index.md). The easiest example of tracking an event in Ruby would be: ```ruby diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 65b50bedb0c..a9f82e85493 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -531,10 +531,43 @@ We aim to have full coverage for all the views. One of the advantages of testing in feature tests is that we can check different states, not only single components in isolation. -Make sure to add assertions, when the view you are working on: +You can find some examples on how to approach accessibility checks below. -- Has an empty state, -- Has significant changes in page structure, for example an alert is shown, or a new section is rendered. +#### Empty state + +Some views have an empty state that result in a page structure that's different from the default view. +They may also offer some actions, for example to create a first issue or to enable a feature. +In this case, add assertions for both an empty state and a default view. + +#### Ensure compliance before user interactions + +Often we test against a number of steps we expect our users to perform. +In this case, make sure to include the check early on, before any of them has been simulated. +This way we ensure there are no barriers to what we expect of users. + +#### Ensure compliance after changed page structure + +User interactions may result in significant changes in page structure. For example, a modal is shown, or a new section is rendered. +In that case, add an assertion after any such change. +We want to make sure that users are able to interact with all available components. + +#### Separate file for extensive test suites + +For some views, feature tests span multiple files. +Take a look at our [feature tests for a merge request](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/features/merge_request). +The number of user interactions that needs to be covered is too big to fit into one test file. +As a result, multiple feature tests cover one view, with different user privileges, or data sets. +If we were to include accessibility checks in all of them, there is a chance we would cover the same states of a view multiple times and significantly increase the run time. +It would also make it harder to determine the coverage for accessibility, if assertions would be scattered across many files. + +In that case, consider creating one test file dedicated to accessibility. +Place it in the same directory and name it `accessibility_spec.rb`, for example `spec/features/merge_request/accessibility_spec.rb`. + +#### Shared examples + +Often feature tests include shared examples for a number of scenarios. +If they differ only by provided data, but are based on the same user interaction, you can check for accessibility compliance outside the shared examples. +This way we only run the check once and save resources. ### How to add accessibility tests diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 0c85a21fdf4..810d9af2de7 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -6,16 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Architecture -When developing a feature that requires architectural design, or changing the fundamental design of an existing feature, discuss it with a Frontend Architecture Expert. +When building new features, consider reaching out to relevant stakeholders as early as possible in the process. -A Frontend Architect is an expert who makes high-level Frontend design decisions -and decides on technical standards, including coding standards and frameworks. - -Architectural decisions should be accessible to everyone, so document -them in the relevant Merge Request discussion or by updating our documentation -when appropriate. - -You can find the Frontend Architecture experts on the [team page](https://about.gitlab.com/company/team/). +Architectural decisions should be accessible to everyone. Document +them in the relevant Merge Request discussions or by updating our documentation +when appropriate by adding an entry to this section. ## Widget Architecture @@ -23,8 +18,3 @@ The [Plan stage](https://about.gitlab.com/handbook/engineering/development/dev/p is refactoring the right sidebar to consist of **widgets**. They have a specific architecture to be reusable and to expose an interface that can be used by external Vue applications on the page. Learn more about the [widget architecture](widgets.md). - -## Examples - -You can find [documentation about the desired architecture](vue.md) for a new -feature built with Vue.js. diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index f90a2b37a1c..876855b807c 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Axios -We use [Axios](https://github.com/axios/axios) to communicate with the server in Vue applications and most new code. +In older parts of our codebase using the REST API, we used [Axios](https://github.com/axios/axios) to communicate with the server, but you should not use Axios in new applications. Instead rely on `apollo-client` to query the GraphQL API. For more details, see [our GraphQL documentation](graphql.md). -In order to guarantee all defaults are set you *should not use Axios directly*, you should import Axios from `axios_utils`. +To guarantee all defaults are set you should import Axios from `axios_utils`. Do not use Axios directly. ## CSRF token diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index ac8b0b8a1ab..476a8acabd0 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Product Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -74,15 +74,15 @@ export const pageViewsOverTime = { dimensions: [], filters: [ { - member: 'SnowplowTrackedEvents.event', + member: 'TrackedEvents.event', operator: 'equals', values: ['page_view'] } ], - measures: ['SnowplowTrackedEvents.pageViewsCount'], + measures: ['TrackedEvents.pageViewsCount'], timeDimensions: [ { - dimension: 'SnowplowTrackedEvents.derivedTstamp', + dimension: 'TrackedEvents.derivedTstamp', granularity: 'day', }, ], @@ -123,6 +123,7 @@ import { pageViewsOverTime } from './visualizations'; export const dashboard = { slug: 'my_dashboard', // Used to set the URL path for the dashboard. title: 'My dashboard title', // The title to display. + description: 'This is a description of the dashboard', // A description of the dashboard // Each dashboard consists of an array of panels to display. panels: [ { @@ -143,7 +144,7 @@ export const dashboard = { // Here we override the Cube.js query to get page views per week instead of days. queryOverrides: { timeDimensions: { - dimension: 'SnowplowTrackedEvents.derivedTstamp', + dimension: 'TrackedEvents.derivedTstamp', granularity: 'week', }, }, diff --git a/doc/development/fe_guide/dark_mode.md b/doc/development/fe_guide/dark_mode.md index 5e8f844172a..144418f72cd 100644 --- a/doc/development/fe_guide/dark_mode.md +++ b/doc/development/fe_guide/dark_mode.md @@ -5,7 +5,7 @@ group: Development info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -This page is about developing dark mode for GitLab. For more information on how to enable dark mode, see [Change the syntax highlighting theme]](../../user/profile/preferences.md#change-the-syntax-highlighting-theme). +This page is about developing dark mode for GitLab. For more information on how to enable dark mode, see [Change the syntax highlighting theme](../../user/profile/preferences.md#change-the-syntax-highlighting-theme). # How dark mode works diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index f087fbd8235..e0d3a80d9c8 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -1,218 +1,10 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'design_patterns.md' +remove_date: '2023-12-07' --- -# Design Anti-patterns +This document was moved to [another location](design_patterns.md). -Anti-patterns may seem like good approaches at first, but it has been shown that they bring more ills than benefits. These should -generally be avoided. - -Throughout the GitLab codebase, there may be historic uses of these anti-patterns. [Use discretion](https://about.gitlab.com/handbook/engineering/development/principles/#balance-refactoring-and-velocity) -when figuring out whether or not to refactor, when touching code that uses one of these legacy patterns. - -NOTE: -For new features, anti-patterns are not necessarily prohibited, but it is **strongly suggested** to find another approach. - -## Shared Global Object (Anti-pattern) - -A shared global object is an instance of something that can be accessed from anywhere and therefore has no clear owner. - -Here's an example of this pattern applied to a Vuex Store: - -```javascript -const createStore = () => new Vuex.Store({ - actions, - state, - mutations -}); - -// Notice that we are forcing all references to this module to use the same single instance of the store. -// We are also creating the store at import-time and there is nothing which can automatically dispose of it. -// -// As an alternative, we should export the `createStore` and let the client manage the -// lifecycle and instance of the store. -export default createStore(); -``` - -### What problems do Shared Global Objects cause? - -Shared Global Objects are convenient because they can be accessed from anywhere. However, -the convenience does not always outweigh their heavy cost: - -- **No ownership.** There is no clear owner to these objects and therefore they assume a non-deterministic - and permanent lifecycle. This can be especially problematic for tests. -- **No access control.** When Shared Global Objects manage some state, this can create some very buggy and difficult - coupling situations because there is no access control to this object. -- **Possible circular references.** Shared Global Objects can also create some circular referencing situations since submodules - of the Shared Global Object can reference modules that reference itself (see - [this MR for an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33366)). - -Here are some historic examples where this pattern was identified to be problematic: - -- [Reference to global Vuex store in IDE](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36401) -- [Docs update to discourage singleton Vuex store](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36952) - -### When could the Shared Global Object pattern be actually appropriate? - -Shared Global Object's solve the problem of making something globally accessible. This pattern -could be appropriate: - -- When a responsibility is truly global and should be referenced across the application - (for example, an application-wide Event Bus). - -Even in these scenarios, consider avoiding the Shared Global Object pattern because the -side-effects can be notoriously difficult to reason with. - -### References - -For more information, see [Global Variables Are Bad on the C2 wiki](https://wiki.c2.com/?GlobalVariablesAreBad). - -## Singleton (Anti-pattern) - -The classic [Singleton pattern](https://en.wikipedia.org/wiki/Singleton_pattern) is an approach to ensure that only one -instance of a thing exists. - -Here's an example of this pattern: - -```javascript -class MyThing { - constructor() { - // ... - } - - // ... -} - -MyThing.instance = null; - -export const getThingInstance = () => { - if (MyThing.instance) { - return MyThing.instance; - } - - const instance = new MyThing(); - MyThing.instance = instance; - return instance; -}; -``` - -### What problems do Singletons cause? - -It is a big assumption that only one instance of a thing should exist. More often than not, -a Singleton is misused and causes very tight coupling amongst itself and the modules that reference it. - -Here are some historic examples where this pattern was identified to be problematic: - -- [Test issues caused by singleton class in IDE](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30398#note_331174190) -- [Implicit Singleton created by module's shared variables](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/merge_requests/97#note_417515776) -- [Complexity caused by Singletons](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29461#note_324585814) - -Here are some ills that Singletons often produce: - -1. **Non-deterministic tests.** Singletons encourage non-deterministic tests because the single instance is shared across - individual tests, often causing the state of one test to bleed into another. -1. **High coupling.** Under the hood, clients of a singleton class all share a single specific - instance of an object, which means this pattern inherits all the [problems of Shared Global Object](#what-problems-do-shared-global-objects-cause) - such as no clear ownership and no access control. These leads to high coupling situations that can - be buggy and difficult to untangle. -1. **Infectious.** Singletons are infectious, especially when they manage state. Consider the component - [RepoEditor](https://gitlab.com/gitlab-org/gitlab/-/blob/27ad6cb7b76430fbcbaf850df68c338d6719ed2b/app%2Fassets%2Fjavascripts%2Fide%2Fcomponents%2Frepo_editor.vue#L0-1) - used in the Web IDE. This component interfaces with a Singleton [Editor](https://gitlab.com/gitlab-org/gitlab/-/blob/862ad57c44ec758ef3942ac2e7a2bd40a37a9c59/app%2Fassets%2Fjavascripts%2Fide%2Flib%2Feditor.js#L21) - which manages some state for working with Monaco. Because of the Singleton nature of the Editor class, - the component `RepoEditor` is now forced to be a Singleton as well. Multiple instances of this component - would cause production issues because no one truly owns the instance of `Editor`. - -### Why is the Singleton pattern popular in other languages like Java? - -This is because of the limitations of languages like Java where everything has to be wrapped -in a class. In JavaScript we have things like object and function literals where we can solve -many problems with a module that exports utility functions. - -### When could the Singleton pattern be actually appropriate?** - -Singletons solve the problem of enforcing there to be only 1 instance of a thing. It's possible -that a Singleton could be appropriate in the following rare cases: - -- We need to manage some resource that **MUST** have just 1 instance (that is, some hardware restriction). -- There is a real [cross-cutting concern](https://en.wikipedia.org/wiki/Cross-cutting_concern) (for example, logging) and a Singleton provides the simplest API. - -Even in these scenarios, consider avoiding the Singleton pattern. - -### What alternatives are there to the Singleton pattern? - -#### Utility Functions - -When no state needs to be managed, we can export utility functions from a module without -messing with any class instantiation. - -```javascript -// bad - Singleton -export class ThingUtils { - static create() { - if(this.instance) { - return this.instance; - } - - this.instance = new ThingUtils(); - return this.instance; - } - - bar() { /* ... */ } - - fuzzify(id) { /* ... */ } -} - -// good - Utility functions -export const bar = () => { /* ... */ }; - -export const fuzzify = (id) => { /* ... */ }; -``` - -#### Dependency Injection - -[Dependency Injection](https://en.wikipedia.org/wiki/Dependency_injection) is an approach which breaks -coupling by declaring a module's dependencies to be injected from outside the module (for example, through constructor parameters, a bona-fide Dependency Injection framework, and even in Vue `provide/inject`). - -```javascript -// bad - Vue component coupled to Singleton -export default { - created() { - this.mediator = MyFooMediator.getInstance(); - }, -}; - -// good - Vue component declares dependency -export default { - inject: ['mediator'] -}; -``` - -```javascript -// bad - We're not sure where the singleton is in it's lifecycle so we init it here. -export class Foo { - constructor() { - Bar.getInstance().init(); - } - - stuff() { - return Bar.getInstance().doStuff(); - } -} - -// good - Lets receive this dependency as a constructor argument. -// It's also not our responsibility to manage the lifecycle. -export class Foo { - constructor(bar) { - this.bar = bar; - } - - stuff() { - return this.bar.doStuff(); - } -} -``` - -In this example, the lifecycle and implementation details of `mediator` are all managed -**outside** the component (most likely the page entrypoint). +<!-- This redirect file can be deleted after <2023-12-07>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 3c273ab18e9..44238ff5dc5 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -6,12 +6,226 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Design Patterns +This page covers suggested design patterns and also anti-patterns. + +NOTE: +When adding a design pattern to this document, be sure to clearly state the **problem it solves**. +When adding a design anti-pattern, clearly state **the problem it prevents**. + +## Patterns + The following design patterns are suggested approaches for solving common problems. Use discretion when evaluating if a certain pattern makes sense in your situation. Just because it is a pattern, doesn't mean it is a good one for your problem. +## Anti-patterns + +Anti-patterns may seem like good approaches at first, but it has been shown that they bring more ills than benefits. These should +generally be avoided. + +Throughout the GitLab codebase, there may be historic uses of these anti-patterns. [Use discretion](https://about.gitlab.com/handbook/engineering/development/principles/#balance-refactoring-and-velocity) +when figuring out whether or not to refactor, when touching code that uses one of these legacy patterns. + NOTE: -When adding a design pattern to this document, be sure to clearly state the **problem it solves**. +For new features, anti-patterns are not necessarily prohibited, but it is **strongly suggested** to find another approach. + +### Shared Global Object + +A shared global object is an instance of something that can be accessed from anywhere and therefore has no clear owner. + +Here's an example of this pattern applied to a Vuex Store: + +```javascript +const createStore = () => new Vuex.Store({ + actions, + state, + mutations +}); + +// Notice that we are forcing all references to this module to use the same single instance of the store. +// We are also creating the store at import-time and there is nothing which can automatically dispose of it. +// +// As an alternative, we should export the `createStore` and let the client manage the +// lifecycle and instance of the store. +export default createStore(); +``` + +#### What problems do Shared Global Objects cause? + +Shared Global Objects are convenient because they can be accessed from anywhere. However, +the convenience does not always outweigh their heavy cost: + +- **No ownership.** There is no clear owner to these objects and therefore they assume a non-deterministic + and permanent lifecycle. This can be especially problematic for tests. +- **No access control.** When Shared Global Objects manage some state, this can create some very buggy and difficult + coupling situations because there is no access control to this object. +- **Possible circular references.** Shared Global Objects can also create some circular referencing situations since submodules + of the Shared Global Object can reference modules that reference itself (see + [this MR for an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33366)). + +Here are some historic examples where this pattern was identified to be problematic: + +- [Reference to global Vuex store in IDE](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36401) +- [Docs update to discourage singleton Vuex store](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36952) + +#### When could the Shared Global Object pattern be actually appropriate? + +Shared Global Object's solve the problem of making something globally accessible. This pattern +could be appropriate: + +- When a responsibility is truly global and should be referenced across the application + (for example, an application-wide Event Bus). + +Even in these scenarios, consider avoiding the Shared Global Object pattern because the +side-effects can be notoriously difficult to reason with. + +#### References + +For more information, see [Global Variables Are Bad on the C2 wiki](https://wiki.c2.com/?GlobalVariablesAreBad). + +### Singleton + +The classic [Singleton pattern](https://en.wikipedia.org/wiki/Singleton_pattern) is an approach to ensure that only one +instance of a thing exists. + +Here's an example of this pattern: + +```javascript +class MyThing { + constructor() { + // ... + } + + // ... +} + +MyThing.instance = null; + +export const getThingInstance = () => { + if (MyThing.instance) { + return MyThing.instance; + } + + const instance = new MyThing(); + MyThing.instance = instance; + return instance; +}; +``` + +#### What problems do Singletons cause? + +It is a big assumption that only one instance of a thing should exist. More often than not, +a Singleton is misused and causes very tight coupling amongst itself and the modules that reference it. + +Here are some historic examples where this pattern was identified to be problematic: + +- [Test issues caused by singleton class in IDE](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30398#note_331174190) +- [Implicit Singleton created by module's shared variables](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/merge_requests/97#note_417515776) +- [Complexity caused by Singletons](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29461#note_324585814) + +Here are some ills that Singletons often produce: + +1. **Non-deterministic tests.** Singletons encourage non-deterministic tests because the single instance is shared across + individual tests, often causing the state of one test to bleed into another. +1. **High coupling.** Under the hood, clients of a singleton class all share a single specific + instance of an object, which means this pattern inherits all the [problems of Shared Global Object](#what-problems-do-shared-global-objects-cause) + such as no clear ownership and no access control. These leads to high coupling situations that can + be buggy and difficult to untangle. +1. **Infectious.** Singletons are infectious, especially when they manage state. Consider the component + [RepoEditor](https://gitlab.com/gitlab-org/gitlab/-/blob/27ad6cb7b76430fbcbaf850df68c338d6719ed2b/app%2Fassets%2Fjavascripts%2Fide%2Fcomponents%2Frepo_editor.vue#L0-1) + used in the Web IDE. This component interfaces with a Singleton [Editor](https://gitlab.com/gitlab-org/gitlab/-/blob/862ad57c44ec758ef3942ac2e7a2bd40a37a9c59/app%2Fassets%2Fjavascripts%2Fide%2Flib%2Feditor.js#L21) + which manages some state for working with Monaco. Because of the Singleton nature of the Editor class, + the component `RepoEditor` is now forced to be a Singleton as well. Multiple instances of this component + would cause production issues because no one truly owns the instance of `Editor`. + +#### Why is the Singleton pattern popular in other languages like Java? + +This is because of the limitations of languages like Java where everything has to be wrapped +in a class. In JavaScript we have things like object and function literals where we can solve +many problems with a module that exports utility functions. + +#### When could the Singleton pattern be actually appropriate?** + +Singletons solve the problem of enforcing there to be only 1 instance of a thing. It's possible +that a Singleton could be appropriate in the following rare cases: + +- We need to manage some resource that **MUST** have just 1 instance (that is, some hardware restriction). +- There is a real [cross-cutting concern](https://en.wikipedia.org/wiki/Cross-cutting_concern) (for example, logging) and a Singleton provides the simplest API. + +Even in these scenarios, consider avoiding the Singleton pattern. + +#### What alternatives are there to the Singleton pattern? + +##### Utility Functions + +When no state needs to be managed, we can export utility functions from a module without +messing with any class instantiation. + +```javascript +// bad - Singleton +export class ThingUtils { + static create() { + if(this.instance) { + return this.instance; + } + + this.instance = new ThingUtils(); + return this.instance; + } + + bar() { /* ... */ } + + fuzzify(id) { /* ... */ } +} + +// good - Utility functions +export const bar = () => { /* ... */ }; + +export const fuzzify = (id) => { /* ... */ }; +``` + +##### Dependency Injection + +[Dependency Injection](https://en.wikipedia.org/wiki/Dependency_injection) is an approach which breaks +coupling by declaring a module's dependencies to be injected from outside the module (for example, through constructor parameters, a bona-fide Dependency Injection framework, and even in Vue `provide/inject`). + +```javascript +// bad - Vue component coupled to Singleton +export default { + created() { + this.mediator = MyFooMediator.getInstance(); + }, +}; + +// good - Vue component declares dependency +export default { + inject: ['mediator'] +}; +``` + +```javascript +// bad - We're not sure where the singleton is in it's lifecycle so we init it here. +export class Foo { + constructor() { + Bar.getInstance().init(); + } + + stuff() { + return Bar.getInstance().doStuff(); + } +} + +// good - Lets receive this dependency as a constructor argument. +// It's also not our responsibility to manage the lifecycle. +export class Foo { + constructor(bar) { + this.bar = bar; + } -## TBD + stuff() { + return this.bar.doStuff(); + } +} +``` -Stay tuned! +In this example, the lifecycle and implementation details of `mediator` are all managed +**outside** the component (most likely the page entrypoint). diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md deleted file mode 100644 index 232689080ea..00000000000 --- a/doc/development/fe_guide/development_process.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Frontend Development Process - -You can find more about the organization of the frontend team in the [handbook](https://about.gitlab.com/handbook/engineering/frontend/). - -## Development Checklist - -The idea is to remind us about specific topics during the time we build a new feature or start something. This is a common practice in other industries (like pilots) that also use standardized checklists to reduce problems early on. - -Copy the content over to your issue or merge request and if something doesn't apply, remove it from your current list. - -This checklist is intended to help us during development of bigger features/refactorings. It is not a "use it always and every point always matches" list. - -Use your best judgment when to use it and contribute new points through merge requests if something comes to your mind. - -```markdown -### Frontend development - -#### Planning development - -- [ ] Check the current set weight of the issue, does it fit your estimate? -- [ ] Are all [departments](https://about.gitlab.com/handbook/engineering/#engineering-teams) that are needed from your perspective already involved in the issue? (For example is UX missing?) -- [ ] Is the specification complete? Are you missing decisions? How about error handling/defaults/edge cases? Take your time to understand the needed implementation and go through its flow. -- [ ] Are all necessary UX specifications available that you will need to implement? Are there new UX components/patterns in the designs? Then contact the UI component team early on. How should error messages or validation be handled? -- [ ] **Library usage** Use Vuex as soon as you have even a medium state to manage, use Vue router if you need to have different views internally and want to link from the outside. Check what libraries we already have for which occasions. -- [ ] **Plan your implementation:** - - [ ] **Architecture plan:** Create a plan aligned with GitLab's architecture, how you are going to do the implementation, for example Vue application setup and its components (through [onion skinning](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35873#note_39994091)), Store structure and data flow, which existing Vue components can you reuse. It's a good idea to go through your plan with another engineer to refine it. - - [ ] **Backend:** The best way is to kickoff the implementation in a call and discuss with the assigned Backend engineer what you will need from the backend and also when. Can you reuse existing API's? How is the performance with the planned architecture? Maybe create together a JSON mock object to already start with development. - - [ ] **Communication:** It also makes sense to have for bigger features an own slack channel (normally called #f_{feature_name}) and even weekly demo calls with all people involved. - - [ ] **Dependency Plan:** Are there big dependencies in the plan between you and others, then maybe create an execution diagram to show what is blocking which part and the order of the different parts. - - [ ] **Task list:** Create a simple checklist of the subtasks that are needed for the implementation, also consider creating even sub issues. (for example show a comment, delete a comment, update a comment, etc.). This helps you and also everyone else following the implementation -- [ ] **Keep it small** To make it easier for you and also all reviewers try to keep merge requests small and merge into a feature branch if needed. To accomplish that you need to plan that from the start. Different methods are: - - [ ] **Skeleton based plan** Start with an MR that has the skeleton of the components with placeholder content. In following MRs you can fill the components with interactivity. This also makes it easier to spread out development on multiple people. - - [ ] **Cookie Mode** Think about hiding the feature behind a cookie flag if the implementation is on top of existing features - - [ ] **New route** Are you refactoring something big then you might consider adding a new route where you implement the new feature and when finished delete the current route and rename the new one. (for example 'merge_request' and 'new_merge_request') -- [ ] **Setup** Is there any specific setup needed for your implementation (for example a kubernetes cluster)? Then let everyone know if it is not already mentioned where they can find documentation (if it doesn't exist - create it) -- [ ] **Security** Are there any new security relevant implementations? Then contact the security team for an app security review. If you are not sure ask our [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) - -#### During development - -- [ ] Check off tasks on your created task list to keep everyone updated on the progress -- [ ] [Share your work early with reviewers/maintainers](#share-your-work-early) -- [ ] Share your work with UXer and Product Manager with Screenshots and/or [GIF images](https://about.gitlab.com/handbook/product/making-gifs/). They are easy to create for you and keep them up to date. -- [ ] If you are blocked on something let everyone on the issue know through a comment. -- [ ] Are you unable to work on this issue for a longer period of time, also let everyone know. -- [ ] **Documentation** Update/add docs for the new feature, see `docs/`. Ping one of the documentation experts/reviewers - -#### Finishing development + Review - -- [ ] **Keep it in the scope** Try to focus on the actual scope and avoid a scope creep during review and keep new things to new issues. -- [ ] **Performance** Have you checked performance? For example do the same thing with 500 comments instead of 1. Document the tests and possible findings in the MR so a reviewer can directly see it. -- [ ] Have you tested with a variety of our [supported browsers](../../install/requirements.md#supported-web-browsers)? You can use [browserstack](https://www.browserstack.com/) to be able to access a wide variety of browsers and operating systems. -- [ ] Did you check the mobile view? -- [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk start`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) -- [ ] **Tests** Not only greenfield tests - Test also all bad cases that come to your mind. -- [ ] If you have multiple MRs then also smoke test against the final merge. -- [ ] Are there any big changes on how and especially how frequently we use the API then let production know about it -- [ ] Smoke test of the RC on dev., staging., canary deployments and .com -- [ ] Follow up on issues that came out of the review. Create issues for discovered edge cases that should be covered in future iterations. -``` - -### Code deletion checklist - -When your merge request deletes code, it's important to also delete all -related code that is no longer used. -When deleting Haml and Vue code, check whether it contains the following types of -code that is unused: - -- CSS. - - For example, we've deleted a Vue component that contained the `.mr-card` class, which is now unused. - The `.mr-card` CSS rule set should then be deleted from `merge_requests.scss`. - -- Ruby variables. - - Deleting unused Ruby variables is important so we don't continue instantiating them with - potentially expensive code. - - For example, we've deleted a Haml template that used the `@total_count` Ruby variable. - The `@total_count` variable was no longer used in the remaining templates for the page. - The instantiation of `@total_count` in `issues_controller.rb` should then be deleted so that we - don't make unnecessary database calls to calculate the count of issues. - -- Ruby methods. - -### Merge Request Review - -With the purpose of being [respectful of others' time](https://about.gitlab.com/handbook/values/#be-respectful-of-others-time), follow these guidelines when asking for a review: - -- Make sure your Merge Request: - - milestone is set - - at least the labels suggested by danger-bot are set - - has a clear description - - includes before/after screenshots if there is a UI change - - pipeline is green - - includes tests - - includes a changelog entry (when necessary) -- Before assigning to a maintainer, assign to a reviewer. -- If you assigned a merge request or pinged someone directly, be patient because we work in different timezones and asynchronously. Unless the merge request is urgent (like fixing a broken default branch), don't DM or reassign the merge request before waiting for a 24-hour window. -- If you have a question regarding your merge request/issue, make it on the merge request/issue. When we DM each other, we no longer have a SSOT and [no one else is able to contribute](https://about.gitlab.com/handbook/values/#public-by-default). -- When you have a big **Draft** merge request with many changes, you're advised to get the review started before adding/removing significant code. Make sure it is assigned well before the release cut-off, as the reviewers/maintainers would always prioritize reviewing finished MRs before the **Draft** ones. -- Make sure to remove the `Draft:` title before the last round of review. - -### Share your work early - -1. Before writing code, ensure your vision of the architecture is aligned with - GitLab architecture. -1. Add a diagram to the issue and ask a frontend maintainer in the Slack channel `#frontend_maintainers` about it. - -  - -1. Don't take more than one week between starting work on a feature and - sharing a Merge Request with a reviewer or a maintainer. - -### Vue features - -1. Follow the steps in [Vue.js Best Practices](vue.md) -1. Follow the style guide. -1. Only a handful of people are allowed to merge Vue related features. - Reach out to one of Vue experts early in this process. diff --git a/doc/development/fe_guide/getting_started.md b/doc/development/fe_guide/getting_started.md new file mode 100644 index 00000000000..bb59bf7b8ee --- /dev/null +++ b/doc/development/fe_guide/getting_started.md @@ -0,0 +1,54 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Getting started + +This page will guide you through the Frontend development process and show you what a normal Merge Request cycle looks like. You can find more about the organization of the frontend team in the [handbook](https://about.gitlab.com/handbook/engineering/frontend/). + +There are a lot of things to consider for a first merge request and it can feel overwhelming. The [Frontend onboarding course](onboarding_course/index.md) provides a 6-week structured curriculum to learn how to contribute to the GitLab frontend. + +## Development life cycle + +### Step 1: Preparing the issue + +Before tackling any work, read through the issue that has been assigned to you and make sure that all [required departments](https://about.gitlab.com/handbook/engineering/#engineering-teams) have been involved as they should. Read through the comments as needed and if unclear, post a comment in the issue summarizing **what you think the work is** and ping your Engineering or Product Manager to confirm. Then once everything is clarified, apply the correct worfklow labels to the issue and create a merge request branch. If created directly from the issue, the issue and the merge request will be linked by default. + +### Step 2: Plan your implementation + +Before writing code, make sure to ask yourself the following questions and have clear answers before you start developing: + +- What API data is required? Is it already available in our API or should I ask a Backend counterpart? + - If this is GraphQL, write a query proposal and ask your BE counterpart to confirm they are in agreement. +- Can I use [GitLab UI components](https://gitlab-org.gitlab.io/gitlab-ui/?path=/docs/base-accordion--docs)? Which components are appropriate and do they have all of the functionality that I need? +- Are there existing components or utils in the GitLab project that I could use? +- [Should this change live behind a Feature Flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags)? +- In which directory should this code live? +- Should I build part of this feature as reusable? If so, where should it live in the codebase and how do I make it discoverable? + - Note: For now this is still being considered, but the `vue_shared` folder is still the preferred directory for GitLab-wide components. +- What kinds of tests will it require? Consider unit tests **and** [Feature Tests](../testing_guide/frontend_testing.md#get-started-with-feature-tests)? Should I reach out to a [SET](https://handbook.gitlab.com/job-families/engineering/software-engineer-in-test/) for guidance or am I comfortable implementing the tests? +- How big will this change be? Try to keep diffs to **500+- at the most**. + +If all of these questions have an answer, then you can safely move on to writing code. + +### Step 3: Writing code + +Make sure to communicate with your team as you progress or if you are unable to work on a planned issue for a long period of time. + +If you require assistance, make sure to push your branch and share your Merge Request either directly to a teammate or in the Slack channel `#frontend` to get advice on how to move forward. You can [mark your Merge Request as a draft](../../user/project/merge_requests/drafts.md), which will clearly communicate that it is not ready for a full on review. Always remember to have a [low level of shame](https://handbook.gitlab.com/handbook/values/#low-level-of-shame) and **ask for help when you need it**. + +As you write code, make sure to test your change thoroughly. It is the author's responsibility to test their code, ensure that it works as expected, and ensure that it did not break existing behaviours. Reviewers may help in that regard, but **do not expect it**. Make sure to check different browsers, mobile viewports and unexpected user flows. + +### Step 4: Review + +When it's time to send your code to review, it can be quite stressful. It is recommended to read through [the code review guidelines](../code_review.md) to get a better sense of what to expect. One of the most valuable pieces of advice that is **essential** is simply: + +> ... to avoid unnecessary back-and-forth with reviewers, ... perform a self-review of your own merge request, and follow the Code Review guidelines. + +This is key to having a great merge request experience because you will catch small mistakes and leave comments in areas where your reviewer might be uncertain and have questions. This speeds up the process tremendously. + +### Step 5: Verifying + +After your code has merged (congratulations!), make sure to verify that it works on the production environment and does not cause any errors. diff --git a/doc/development/fe_guide/guides.md b/doc/development/fe_guide/guides.md new file mode 100644 index 00000000000..dc2fffcf10a --- /dev/null +++ b/doc/development/fe_guide/guides.md @@ -0,0 +1,13 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Guides + +This section contains guides to help our developers. +For example, you can find information about how to accomplish a specific task, +or how get proficient with a tool. + +Guidelines related to one specific technology, like Vue, should not be added to this section. Instead, add them to the `Tech Stack` section. diff --git a/doc/development/fe_guide/img/boards_diagram.png b/doc/development/fe_guide/img/boards_diagram.png Binary files differdeleted file mode 100644 index 856c9b05bbf..00000000000 --- a/doc/development/fe_guide/img/boards_diagram.png +++ /dev/null diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 405e830406e..70f7aad207b 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -86,135 +86,48 @@ Now that our values have been defined, we can base our goals on these values and We have detailed description on how we see GitLab frontend in the future in [Frontend Goals](frontend_goals.md) section -### Frontend onboarding course +### First time contributors -The [Frontend onboarding course](onboarding_course/index.md) provides a 6-week structured curriculum to learn how to contribute to the GitLab frontend. +If you're a first-time contributor, see [Contribute to GitLab development](../contributing/index.md). -### Browser Support +When you're ready to create your first merge request, or need to review the GitLab frontend workflow, see [Getting started](getting_started.md). -For supported browsers, see our [requirements](../../install/requirements.md#supported-web-browsers). +For a guided introduction to frontend development at GitLab, you can watch the [Frontend onboarding course](onboarding_course/index.md) which provides a six-week structured curriculum. -Use [BrowserStack](https://www.browserstack.com/) to test with our supported browsers. -Sign in to BrowserStack with the credentials saved in the **Engineering** vault of the GitLab -[shared 1Password account](https://about.gitlab.com/handbook/security/#1password-guide). +### Helpful links -## Initiatives +#### Initiatives You can find current frontend initiatives with a cross-functional impact on epics with the label [frontend-initiative](https://gitlab.com/groups/gitlab-org/-/epics?state=opened&page=1&sort=UPDATED_AT_DESC&label_name[]=frontend-initiative). -## Principles - -[High-level guidelines](principles.md) for contributing to GitLab. - -## Development Process - -How we [plan and execute](development_process.md) the work on the frontend. - -## Architecture - -How we go about [making fundamental design decisions](architecture.md) in the GitLab frontend team -or make changes to our frontend development guidelines. - -## Testing +#### Testing How we write [frontend tests](../testing_guide/frontend_testing.md), run the GitLab test suite, and debug test related issues. -## Pajamas Design System +#### Pajamas Design System Reusable components with technical and usage guidelines can be found in our [Pajamas Design System](https://design.gitlab.com/). -## Design Patterns - -JavaScript [design patterns](design_patterns.md) in the GitLab codebase. - -## Design Anti-patterns - -JavaScript [design anti-patterns](design_anti_patterns.md) we try to avoid. - -## Vue.js Best Practices - -Vue specific [design patterns and practices](vue.md). - -## Vuex - -[Vuex](vuex.md) specific design patterns and practices. - -## Axios - -[Axios](axios.md) specific practices and gotchas. - -## GraphQL - -How to use [GraphQL](graphql.md). - -## HAML - -How to use [HAML](haml.md). - -## ViewComponent - -How we use [ViewComponent](view_component.md). - -## Icons and Illustrations - -How we use SVG for our [Icons and Illustrations](icons.md). - -## Dependencies - -General information about frontend [dependencies](dependencies.md) and how we manage them. - -## Keyboard Shortcuts - -How we implement [keyboard shortcuts](keyboard_shortcuts.md) that can be customized and disabled. - -## Editors - -GitLab text editing experiences are provided by the [source editor](source_editor.md) and -the [rich text editor](content_editor.md). - -## Frontend FAQ +#### Frontend FAQ Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful information. -## Style Guides - -See the relevant style guides for our guidelines and for information on linting: - -- [JavaScript](style/javascript.md). Our guide is based on -the excellent [Airbnb](https://github.com/airbnb/javascript) style guide with a few small -changes. -- [SCSS](style/scss.md): [our SCSS conventions](https://gitlab.com/gitlab-org/frontend/gitlab-stylelint-config) which are enforced through [`stylelint`](https://stylelint.io). -- [HTML](style/html.md). Guidelines for writing HTML code consistent with the rest of the codebase. -- [Vue](style/vue.md). Guidelines and conventions for Vue code may be found here. - -## [Tooling](tooling.md) - -Our code is automatically formatted with [Prettier](https://prettier.io) to follow our guidelines. Read our [Tooling guide](tooling.md) for more detail. - -## [Performance](performance.md) - -Best practices for monitoring and maximizing frontend performance. - -## [Security](security.md) - -Frontend security practices. - -## Accessibility - -Our [accessibility standards and resources](accessibility.md). - -## Logging - -Best practices for [client-side logging](logging.md) for GitLab frontend development. - -## [Internationalization (i18n) and Translations](../i18n/externalization.md) +#### [Internationalization (i18n) and Translations](../i18n/externalization.md) Frontend internationalization support is described in [this document](../i18n/index.md). The [externalization part of the guide](../i18n/externalization.md) explains the helpers/methods available. -## [Troubleshooting](troubleshooting.md) +#### [Troubleshooting](troubleshooting.md) Running into a Frontend development problem? Check out [this guide](troubleshooting.md) to help resolve your issue. + +#### Browser support + +For supported browsers, see our [requirements](../../install/requirements.md#supported-web-browsers). + +Use [BrowserStack](https://www.browserstack.com/) to test with our supported browsers. +Sign in to BrowserStack with the credentials saved in the **Engineering** vault of the GitLab +[shared 1Password account](https://about.gitlab.com/handbook/security/#1password-guide). diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md deleted file mode 100644 index 6d1a3238b73..00000000000 --- a/doc/development/fe_guide/principles.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Principles - -These principles ensure that your frontend contribution starts off in the right direction. - -## Discuss architecture before implementation - -Discuss your architecture design in an issue before writing code. This helps decrease the review time and also provides good practice for writing and thinking about system design. - -## Be consistent - -There are multiple ways of writing code to accomplish the same results. We should be as consistent as possible in how we write code across our codebases. This makes it easier for us to maintain our code across GitLab. - -## Improve code [iteratively](https://about.gitlab.com/handbook/values/#iteration) - -Whenever you see existing code that does not follow our current style guide, update it proactively. You don't need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. diff --git a/doc/development/fe_guide/sentry.md b/doc/development/fe_guide/sentry.md new file mode 100644 index 00000000000..af4a006c7ea --- /dev/null +++ b/doc/development/fe_guide/sentry.md @@ -0,0 +1,34 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sentry + +As part of the [Frontend Observability Working Group](https://google.com) we're looking to provide documentation on how to use Sentry effectively. +If left unchecked, Sentry can get noisy and become unreliable. +This page aims to help guide us toward more sensible Sentry usage. + +## Which errors we should report to Sentry explicitly and which should be only shown to users (e.g. as alerts) + +If we send all errors to Sentry, it gets very noisy, very quickly. +We want to filter out the errors that we either don't care about, or have no control over. +For example, if a user fills out a form incorrectly, this is not something we want to send to Sentry. +If that form fails because it's hitting a dead endpoint, this is an error we want Sentry to know about. + +## How to catch errors correctly so Sentry can display them reliably + +TBD + +## How to catch special cases you want to track (like we did with the pipeline graph) + +TBD + +## How to navigate Sentry and find errors + +TBD + +## How to debug Sentry errors effectively + +TBD diff --git a/doc/development/fe_guide/tech_stack.md b/doc/development/fe_guide/tech_stack.md new file mode 100644 index 00000000000..9c0d50ea7bd --- /dev/null +++ b/doc/development/fe_guide/tech_stack.md @@ -0,0 +1,11 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tech Stack + +For an exhaustive list of all the technology that we use, simply check our [latest `package.json` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/package.json?ref_type=heads). + +Each navigation item in this section is a guide for that specific technology. diff --git a/doc/development/fe_guide/tips_and_tricks.md b/doc/development/fe_guide/tips_and_tricks.md new file mode 100644 index 00000000000..dcacdb8387b --- /dev/null +++ b/doc/development/fe_guide/tips_and_tricks.md @@ -0,0 +1,31 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tips and tricks + +## Code deletion checklist + +When your merge request deletes code, it's important to also delete all +related code that is no longer used. +When deleting Haml and Vue code, check whether it contains the following types of +code that is unused: + +- CSS. + + For example, we've deleted a Vue component that contained the `.mr-card` class, which is now unused. + The `.mr-card` CSS rule set should then be deleted from `merge_requests.scss`. + +- Ruby variables. + + Deleting unused Ruby variables is important so we don't continue instantiating them with + potentially expensive code. + + For example, we've deleted a Haml template that used the `@total_count` Ruby variable. + The `@total_count` variable was no longer used in the remaining templates for the page. + The instantiation of `@total_count` in `issues_controller.rb` should then be deleted so that we + don't make unnecessary database calls to calculate the count of issues. + +- Ruby methods. diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index c4a5c996ab1..2f013f698dc 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -95,7 +95,7 @@ Consult these topics for information on contributing to specific GitLab features - [Shell commands](shell_commands.md) in the GitLab codebase - [Value Stream Analytics development guide](value_stream_analytics.md) - [Application limits](application_limits.md) -- [AI features](ai_features.md) +- [AI features](ai_features/index.md) ### Import and Export @@ -163,8 +163,8 @@ The following integration guides are internal. Some integrations require access ## Analytics Instrumentation guides - [Analytics Instrumentation guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) -- [Service Ping guide](service_ping/index.md) -- [Snowplow guide](snowplow/index.md) +- [Service Ping guide](internal_analytics/service_ping/index.md) +- [Snowplow guide](internal_analytics/snowplow/index.md) ## Experiment guide diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 8c0f7faab28..af40fd8b945 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -420,9 +420,18 @@ The actor is a second parameter of the `Feature.enabled?` call. The same actor type must be used consistently for all invocations of `Feature.enabled?`. ```ruby +# Bad Feature.enabled?(:feature_flag, project) Feature.enabled?(:feature_flag, group) Feature.enabled?(:feature_flag, user) + +# Good +Feature.enabled?(:feature_flag, group_a) +Feature.enabled?(:feature_flag, group_b) + +# Also good - using separate flags for each actor type +Feature.enabled?(:feature_flag_group, group) +Feature.enabled?(:feature_flag_user, user) ``` See [Feature flags in the development of GitLab](controls.md#process) for details on how to use ChatOps diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index a1f5111d6f4..f8de93a2243 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -16,6 +16,6 @@ When implementing new features, please refer to these existing features to avoid - [Route Maps](../ci/review_apps/index.md#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-helm-chart-values): `.gitlab/auto-deploy-values.yaml`. - [Insights](../user/project/insights/index.md#configure-project-insights): `.gitlab/insights.yml`. -- [Service Desk Templates](../user/project/service_desk/index.md#customize-emails-sent-to-the-requester): `.gitlab/service_desk_templates/`. +- [Service Desk Templates](../user/project/service_desk/configure.md#customize-emails-sent-to-the-requester): `.gitlab/service_desk_templates/`. - [Secret Detection Custom Rulesets](../user/application_security/secret_detection/index.md#disable-predefined-analyzer-rules): `.gitlab/secret-detection-ruleset.toml` - [Static Analysis Custom Rulesets](../user/application_security/sast/customize_rulesets.md#create-the-configuration-file): `.gitlab/sast-ruleset.toml` diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index c346d55f639..39833a441ee 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -56,7 +56,7 @@ they are still not 100% standardized. You can see them below: CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store. -In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage](../administration/repository_storage_types.md) layout, +In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage](../administration/repository_storage_paths.md) layout, instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the hash of the project ID instead, if project migrates to the new approach (introduced in 10.2). diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 4f6a9feb191..60677abf292 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -59,7 +59,7 @@ listed here that also do not work properly in FIPS mode: - [Container Scanning](../user/application_security/container_scanning/index.md) support for scanning images in repositories that require authentication. - [Code Quality](../ci/testing/code_quality.md) does not support operating in FIPS-compliant mode. - [Dependency scanning](../user/application_security/dependency_scanning/index.md) support for Gradle. -- [Dynamic Application Security Testing (DAST)](../user/application_security/dast/proxy-based.md) supports a reduced set of analyzers. The proxy-based analyzer is not available in FIPS mode today, however browser-based DAST, DAST API, and DAST API Fuzzing images are available. +- [Dynamic Application Security Testing (DAST)](../user/application_security/dast/proxy-based.md) supports a reduced set of analyzers. The proxy-based analyzer and on-demand scanning is not available in FIPS mode today, however browser-based DAST, DAST API, and DAST API Fuzzing images are available. - [Solutions for vulnerabilities](../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability) for yarn projects. - [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) diff --git a/doc/development/gems.md b/doc/development/gems.md index c061b33b5e4..132bf931da8 100644 --- a/doc/development/gems.md +++ b/doc/development/gems.md @@ -238,11 +238,10 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na the gem name with `gitlab-`. For example, `gitlab-sidekiq-fetcher`. 1. Locally create the gem or fork as necessary. 1. [Publish an empty `0.0.1` version of the gem to rubygems.org](https://guides.rubygems.org/publishing/#publishing-to-rubygemsorg) to ensure the gem name is reserved. -1. Add the [`gitlab_rubygems`](https://rubygems.org/profiles/gitlab_rubygems) and [`gitlab-qa`](https://rubygems.org/profiles/gitlab-qa) users as owners of the new gem by running: +1. Add the [`gitlab_rubygems`](https://rubygems.org/profiles/gitlab_rubygems) user as owner of the new gem by running: ```shell gem owner <gem-name> --add gitlab_rubygems - gem owner <gem-name> --add gitlab-qa ``` 1. Optional. Add some or all of the following users as co-owners: @@ -251,8 +250,8 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na - [Stan Hu](https://rubygems.org/profiles/stanhu) 1. Optional. Add any other relevant developers as co-owners. 1. Visit `https://rubygems.org/gems/<gem-name>` and verify that the gem was published - successfully and `gitlab_rubygems` & `gitlab-qa` are also owners. -1. Create a project in the [`gitlab-org/ruby/gems` group](https://gitlab.com/gitlab-org/ruby/gems/). To create this project: + successfully and `gitlab_rubygems` is also an owner. +1. Create a project in the [`gitlab-org/ruby/gems` group](https://gitlab.com/gitlab-org/ruby/gems/) (or in a subgroup of it): 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). 1. Use the [shared CI/CD config](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/gem-release.yml) @@ -264,7 +263,7 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na file: '/ci/gem-release.yml' ``` - This job will handle building and publishing the gem (it uses a `gilab-qa` Rubygems.org + This job will handle building and publishing the gem (it uses a `gitlab_rubygems` Rubygems.org API token inherited from the `gitlab-org/ruby/gems` group, in order to publish the gem package), as well as creating the tag, release and populating its release notes by using the diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 961bfca0d9b..65d2338cd65 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -46,7 +46,7 @@ reliable decide if an object is no longer needed. ### Git alternates in GitLab: pool repositories -GitLab organizes this object borrowing by [creating special **pool repositories**](../administration/repository_storage_types.md) +GitLab organizes this object borrowing by [creating special **pool repositories**](../administration/repository_storage_paths.md) which are hidden from the user. We then use Git alternates to let a collection of project repositories borrow from a single pool repository. We call such a collection of project @@ -101,7 +101,7 @@ are as follows: ### Assumptions -- All repositories in a pool must use [hashed storage](../administration/repository_storage_types.md). +- All repositories in a pool must use [hashed storage](../administration/repository_storage_paths.md). This is so that we don't have to ever worry about updating paths in `object/info/alternates` files. - All repositories in a pool must be on the same Gitaly storage shard. diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index d38be071f39..45554ae465d 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -243,11 +243,13 @@ To avoid mismatching users, the search by GitHub user ID is not done when import Enterprise. Because this process is quite expensive we cache the result of these lookups in -Redis. For every user looked up we store three keys: +Redis. For every user looked up we store five keys: - A Redis key mapping GitHub usernames to their Email addresses. - A Redis key mapping a GitHub Email addresses to a GitLab user ID. - A Redis key mapping a GitHub user ID to GitLab user ID. +- A Redis key mapping a GitHub username to an ETAG header. +- A Redis key indicating whether an email lookup has been done for a project. We cache two types of lookups: @@ -260,9 +262,12 @@ The expiration time of these keys is 24 hours. When retrieving the cache of a positive lookup, we refresh the TTL automatically. The TTL of false lookups is never refreshed. +If a lookup for email returns an empty or negative lookup, a [Conditional Request](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#conditional-requests) is made with a cached ETAG in the header once for every project. +Conditional Requests do not count towards the GitHub API rate limit. + Because of this caching layer, it's possible newly registered GitLab accounts aren't linked to their corresponding GitHub accounts. This, however, is resolved -after the cached keys expire. +after the cached keys expire or if a new project is imported. The user cache lookup is shared across projects. This means that the greater the number of projects that are imported, fewer GitHub API calls are needed. @@ -287,7 +292,7 @@ The code for this resides in: - `lib/gitlab/github_import/label_finder.rb` - `lib/gitlab/github_import/milestone_finder.rb` -- `lib/gitlab/github_import/caching.rb` +- `lib/gitlab/cache/import/caching.rb` ## Logs diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 7648e84f5e8..c6d7b231b72 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -9,8 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document describes various guidelines and best practices for GitLab projects using the [Go language](https://go.dev/). -## Overview - GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), but we're also using Go for projects where it makes sense. Go is a very powerful language, with many advantages, and is best suited for projects with a lot of @@ -73,7 +71,7 @@ of possible security breaches in our code: Remember to run [SAST](../../user/application_security/sast/index.md) and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) -**(ULTIMATE)** on your project (or at least the +**(ULTIMATE ALL)** on your project (or at least the [`gosec` analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), and to follow our [Security requirements](../code_review.md#security). @@ -142,6 +140,12 @@ become available, you can share job templates like this Go GitLab linter plugins are maintained in the [`gitlab-org/language-tools/go/linters`](https://gitlab.com/gitlab-org/language-tools/go/linters/) namespace. +### Help text style guide + +If your Go project produces help text for users, consider following the advice given in the +[Help text style guide](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/help_text_style_guide.md) in the +`gitaly` project. + ## Dependencies Dependencies should be kept to the minimum. The introduction of a new @@ -340,21 +344,34 @@ which documents and centralizes at the same time all the possible command line interactions with the program. Don't use `os.GetEnv`, it hides variables deep in the code. -## Daemons +## Libraries + +### LabKit -### Logging +[LabKit](https://gitlab.com/gitlab-org/labkit) is a place to keep common +libraries for Go services. For examples using of using LabKit, see [`workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse) +and [`gitaly`](https://gitlab.com/gitlab-org/gitaly). LabKit exports three related pieces of functionality: -The usage of a logging library is strongly recommended for daemons. Even -though there is a `log` package in the standard library, we generally use -[Logrus](https://github.com/sirupsen/logrus). Its plugin ("hooks") system -makes it a powerful logging library, with the ability to add notifiers and -formatters at the logger level directly. +- [`gitlab.com/gitlab-org/labkit/correlation`](https://gitlab.com/gitlab-org/labkit/tree/master/correlation): + for propagating and extracting correlation ids between services. +- [`gitlab.com/gitlab-org/labkit/tracing`](https://gitlab.com/gitlab-org/labkit/tree/master/tracing): + for instrumenting Go libraries for distributed tracing. +- [`gitlab.com/gitlab-org/labkit/log`](https://gitlab.com/gitlab-org/labkit/tree/master/log): + for structured logging using Logrus. + +This gives us a thin abstraction over underlying implementations that is +consistent across Workhorse, Gitaly, and possibly other Go servers. For +example, in the case of `gitlab.com/gitlab-org/labkit/tracing` we can switch +from using `Opentracing` directly to using `Zipkin` or the Go kit's own tracing wrapper +without changes to the application code, while still keeping the same +consistent configuration mechanism (that is, the `GITLAB_TRACING` environment +variable). #### Structured (JSON) logging Every binary ideally must have structured (JSON) logging in place as it helps -with searching and filtering the logs. At GitLab we use structured logging in -JSON format, as all our infrastructure assumes that. When using +with searching and filtering the logs. LabKit provides an abstraction over [Logrus](https://github.com/sirupsen/logrus). +We use structured logging in JSON format, because all our infrastructure assumes that. When using [Logrus](https://github.com/sirupsen/logrus) you can turn on structured logging by using the build in [JSON formatter](https://github.com/sirupsen/logrus#formatters). This follows the same logging type we use in our [Ruby applications](../logging.md#use-structured-json-logging). @@ -375,26 +392,6 @@ There are a few guidelines one should follow when using the have to log multiple keys, always use `WithFields` instead of calling `WithField` more than once. -### Tracing and Correlation - -[LabKit](https://gitlab.com/gitlab-org/labkit) is a place to keep common -libraries for Go services. Currently it's vendored into two projects: -Workhorse and Gitaly, and it exports two main (but related) pieces of -functionality: - -- [`gitlab.com/gitlab-org/labkit/correlation`](https://gitlab.com/gitlab-org/labkit/tree/master/correlation): - for propagating and extracting correlation ids between services. -- [`gitlab.com/gitlab-org/labkit/tracing`](https://gitlab.com/gitlab-org/labkit/tree/master/tracing): - for instrumenting Go libraries for distributed tracing. - -This gives us a thin abstraction over underlying implementations that is -consistent across Workhorse, Gitaly, and, in future, other Go servers. For -example, in the case of `gitlab.com/gitlab-org/labkit/tracing` we can switch -from using `Opentracing` directly to using `Zipkin` or the Go kit's own tracing wrapper -without changes to the application code, while still keeping the same -consistent configuration mechanism (that is, the `GITLAB_TRACING` environment -variable). - ### Context Since daemons are long-running applications, they should have mechanisms to diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 25651639170..59362dc33c0 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -221,7 +221,7 @@ Assets that need to be served to the user are stored under the `app/assets` dire However, you cannot access the content of any file from within `app/assets` from the application code, as we do not include that folder in production installations as a [space saving measure](https://gitlab.com/gitlab-org/omnibus-gitlab/-/commit/ca049f990b223f5e1e412830510a7516222810be). ```ruby -support_bot = User.support_bot +support_bot = Users::Internal.support_bot # accessing a file from the `app/assets` folder support_bot.avatar = Rails.root.join('app', 'assets', 'images', 'bot_avatars', 'support_bot.png').open @@ -244,3 +244,59 @@ Use `app/assets` for storing any asset that needs to be precompiled and served t Use `lib/assets` for storing any asset that does not need to be served to the end user directly, but is still required to be accessed by the application code. MR for reference: [!37671](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37671) + +## Do not override `has_many through:` or `has_one through:` associations + +Associations with the `:through` option should not be overridden as we could accidentally +destroy the wrong object. + +This is because the `destroy()` method behaves differently when acting on +`has_many through:` and `has_one through:` associations. + +```ruby +group.users.destroy(id) +``` + +The code example above reads as if we are destroying a `User` record, but behind the scenes, it is destroying a `Member` record. This is because the `users` association is defined on `Group` as a `has_many through:` association: + +```ruby +class Group < Namespace + has_many :group_members, -> { where(requested_at: nil).where.not(members: { access_level: Gitlab::Access::MINIMAL_ACCESS }) }, dependent: :destroy, as: :source + + has_many :users, through: :group_members +end +``` + +And Rails has the following [behavior](https://api.rubyonrails.org/classes/ActiveRecord/Associations/ClassMethods.html#method-i-has_many) on using `destroy()` on such associations: + +> If the :through option is used, then the join records are destroyed instead, not the objects themselves. + +This is why a `Member` record, which is the join record connecting a `User` and `Group`, is being destroyed. + +Now, if we override the `users` association, so like: + +```ruby +class Group < Namespace + has_many :group_members, -> { where(requested_at: nil).where.not(members: { access_level: Gitlab::Access::MINIMAL_ACCESS }) }, dependent: :destroy, as: :source + + has_many :users, through: :group_members + + def users + super.where(admin: false) + end +end +``` + +The overridden method now changes the above behavior of `destroy()`, such that if we execute + +```ruby +group.users.destroy(id) +``` + +a `User` record will be deleted, which can lead to data loss. + +In short, overriding a `has_many through:` or `has_one through:` association can prove dangerous. +To prevent this from happening, we are introducing an +automated check in [!131455](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131455). + +For more information, see [issue 424536](https://gitlab.com/gitlab-org/gitlab/-/issues/424536). diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index f4ace7491eb..68c2778eabe 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -204,9 +204,9 @@ Example: ```javascript // Bad. Not necessary in Frontend environment. -expect(findText()).toBe(__('Lorem ipsum dolar sit')); +expect(findText()).toBe(__('Lorem ipsum dolor sit')); // Good. -expect(findText()).toBe('Lorem ipsum dolar sit'); +expect(findText()).toBe('Lorem ipsum dolor sit'); ``` #### Recommendations diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index cf50e417278..65cde363e98 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -31,7 +31,7 @@ are very appreciative of the work done by translators and proofreaders! - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - Xiaogang Wen - [GitLab](https://gitlab.com/xiaogang_cn), [Crowdin](https://crowdin.com/profile/xiaogang_gitlab) - - Shuang Zhang - [GitLab](https://gitlab.com/tonygodspeed92), [Crowdin](https://crowdin.com/profile/tonygodspeed92) + - Qi Zhao - [GitLab](https://gitlab.com/zhaoqi01), [Crowdin](https://crowdin.com/profile/zhaoqi01) - Chinese Traditional 繁體中文 - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) - Yi-Jyun Pan - [GitLab](https://gitlab.com/pan93412), [Crowdin](https://crowdin.com/profile/pan93412) diff --git a/doc/development/img/build_package_v12_6.png b/doc/development/img/build_package_v12_6.png Binary files differdeleted file mode 100644 index 32a3ebedba4..00000000000 --- a/doc/development/img/build_package_v12_6.png +++ /dev/null diff --git a/doc/development/img/trigger_build_package_v12_6.png b/doc/development/img/trigger_build_package_v12_6.png Binary files differdeleted file mode 100644 index ca6797ebf65..00000000000 --- a/doc/development/img/trigger_build_package_v12_6.png +++ /dev/null diff --git a/doc/development/img/trigger_omnibus_v16_3.png b/doc/development/img/trigger_omnibus_v16_3.png Binary files differnew file mode 100644 index 00000000000..d5348333faa --- /dev/null +++ b/doc/development/img/trigger_omnibus_v16_3.png diff --git a/doc/development/img/triggered_ee_pipeline_v16_3.png b/doc/development/img/triggered_ee_pipeline_v16_3.png Binary files differnew file mode 100644 index 00000000000..be91f35b9ab --- /dev/null +++ b/doc/development/img/triggered_ee_pipeline_v16_3.png diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index dd73256ce11..f84375b3b77 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -12,7 +12,7 @@ which are part of our [main Rails project](https://gitlab.com/gitlab-org/gitlab) Also see our [direction page](https://about.gitlab.com/direction/manage/import_and_integrate/integrations/) for an overview of our strategy around integrations. -This guide is a work in progress. You're welcome to ping `@gitlab-org/manage/integrations` +This guide is a work in progress. You're welcome to ping `@gitlab-org/manage/import-and-integrate` if you need clarification or spot any outdated information. ## Add a new integration @@ -39,9 +39,9 @@ if you need clarification or spot any outdated information. has_one :foo_bar_integration, class_name: 'Integrations::FooBar' ``` -### Define properties +### Define fields -Integrations can define arbitrary properties to store their configuration with the class method `Integration.prop_accessor`. +Integrations can define arbitrary fields to store their configuration with the class method `Integration.field`. The values are stored as an encrypted JSON hash in the `integrations.encrypted_properties` column. For example: @@ -49,25 +49,26 @@ For example: ```ruby module Integrations class FooBar < Integration - prop_accessor :url - prop_accessor :tags + field :url + field :tags end end ``` -`Integration.prop_accessor` installs accessor methods on the class. Here we would have `#url`, `#url=` and `#url_changed?`, to manage the `url` field. Fields stored in `Integration#properties` should be accessed by these accessors directly on the model, just like other ActiveRecord attributes. +`Integration.field` installs accessor methods on the class. +Here we would have `#url`, `#url=`, and `#url_changed?` to manage the `url` field. +These accessors should access the fields stored in `Integration#properties` directly on the model, just like other `ActiveRecord` attributes. -You should always access the properties through their `getters`, and not interact with the `properties` hash directly. +You should always access the fields through their `getters` and not interact with the `properties` hash directly. You **must not** write to the `properties` hash, you **must** use the generated setter method instead. Direct writes to this hash are not persisted. You should also define validations for all your properties. +To see how these fields are exposed in the frontend form for the integration, +see [Customize the frontend form](#customize-the-frontend-form). -Also refer to the section [Customize the frontend form](#customize-the-frontend-form) below to see how these properties -are exposed in the frontend form for the integration. - -There is an alternative approach using `Integration.data_field`, which you may see in other integrations. -With data fields the values are stored in a separate table per integration. At the moment we don't recommend using this for new integrations. +Other approaches include using `Integration.prop_accessor` or `Integration.data_field`, which you might see in earlier versions of integrations. +You should not use these approaches for new integrations. ### Define trigger events @@ -94,7 +95,7 @@ The following events are supported for integrations: | [Pipeline event](../../user/project/integrations/webhook_events.md#pipeline-events) | | `pipeline` | A pipeline status changes. | [Push event](../../user/project/integrations/webhook_events.md#push-events) | ✓ | `push` | A push is made to the repository. | [Tag push event](../../user/project/integrations/webhook_events.md#tag-events) | ✓ | `tag_push` | New tags are pushed to the repository. -| Vulnerability event **(ULTIMATE)** | | `vulnerability` | A new, unique vulnerability is recorded. +| Vulnerability event **(ULTIMATE ALL)** | | `vulnerability` | A new, unique vulnerability is recorded. | [Wiki page event](../../user/project/integrations/webhook_events.md#wiki-page-events) | ✓ | `wiki_page` | A wiki page is created or updated. #### Event examples @@ -191,8 +192,8 @@ This method should return an array of hashes for each field, where the keys can | Key | Type | Required | Default | Description |:---------------|:--------|:---------|:-----------------------------|:-- -| `type:` | string | true | | The type of the form field. Can be `text`, `textarea`, `password`, `checkbox`, or `select`. -| `name:` | string | true | | The property name for the form field. This must match a `prop_accessor` [defined on the class](#define-properties). +| `type:` | symbol | true | `:text` | The type of the form field. Can be `:text`, `:textarea`, `:password`, `:checkbox`, or `:select`. +| `name:` | string | true | | The property name for the form field. | `required:` | boolean | false | `false` | Specify if the form field is required or optional. | `title:` | string | false | Capitalized value of `name:` | The label for the form field. | `placeholder:` | string | false | | A placeholder for the form field. @@ -200,19 +201,19 @@ This method should return an array of hashes for each field, where the keys can | `api_only:` | boolean | false | `false` | Specify if the field should only be available through the API, and excluded from the frontend form. | `if:` | boolean or lambda | false | `true` | Specify if the field should be available. The value can be a boolean or a lambda. -### Additional keys for `type: 'checkbox'` +### Additional keys for `type: :checkbox` | Key | Type | Required | Default | Description |:------------------|:-------|:---------|:------------------|:-- | `checkbox_label:` | string | false | Value of `title:` | A custom label that displays next to the checkbox. -### Additional keys for `type: 'select'` +### Additional keys for `type: :select` | Key | Type | Required | Default | Description |:-----------|:------|:---------|:--------|:-- | `choices:` | array | true | | A nested array of `[label, value]` tuples. -### Additional keys for `type: 'password'` +### Additional keys for `type: :password` | Key | Type | Required | Default | Description |:----------------------------|:-------|:---------|:------------------|:-- @@ -226,30 +227,20 @@ This example defines a required `url` field, and optional `username` and `passwo ```ruby module Integrations class FooBar < Integration - prop_accessor :url, :username, :password - - def fields - [ - { - type: 'text', - name: 'url', - title: s_('FooBarIntegration|Server URL'), - placeholder: 'https://example.com/', - required: true - }, - { - type: 'text', - name: 'username', - title: s_('FooBarIntegration|Username'), - }, - { - type: 'password', - name: 'password', - title: s_('FoobarIntegration|Password' - non_empty_password_title: s_('FooBarIntegration|Enter new password') - } - ] - end + field :url, + type: :text, + title: s_('FooBarIntegration|Server URL'), + placeholder: 'https://example.com/', + required: true + + field :username, + type: :text, + title: s_('FooBarIntegration|Username') + + field :password, + type: 'password', + title: s_('FoobarIntegration|Password' + non_empty_password_title: s_('FooBarIntegration|Enter new password') end end ``` diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index 65194a04a62..f52098394dc 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -24,7 +24,7 @@ brew services start jenkins GitLab does not allow requests to localhost or the local network by default. When running Jenkins on your local machine, you need to enable local access. 1. Log into your GitLab instance as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**, and select the following checkboxes: @@ -55,8 +55,8 @@ To set up the Jenkins project you intend to run your build on, read You can configure your integration between Jenkins and GitLab: -- With the [recommended approach for Jenkins integration](../../integration/jenkins.md#configure-a-jenkins-integration-recommended). -- [Using a webhook](../../integration/jenkins.md#configure-a-webhook). +- With the [recommended approach for Jenkins integration](../../integration/jenkins.md#with-a-jenkins-server-url). +- [Using a webhook](../../integration/jenkins.md#with-a-webhook). ## Test your setup diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 8fda6042fcf..8c6e3145000 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -181,7 +181,7 @@ See also [Docker Tagging: Best practices for tagging and versioning Docker image ## Command line -A scanner is a command line tool that takes environment variables as inputs, +A scanner is a command-line tool that takes environment variables as inputs, and generates a file that is uploaded as a report (based on the job definition). It also generates text output on the standard output and standard error streams, and exits with a status code. diff --git a/doc/development/internal_analytics/index.md b/doc/development/internal_analytics/index.md index c7cf907ca92..d24ecf5a99c 100644 --- a/doc/development/internal_analytics/index.md +++ b/doc/development/internal_analytics/index.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/internal_event_tracking/architecture.md b/doc/development/internal_analytics/internal_event_tracking/architecture.md index de5672a4895..0265e39745a 100644 --- a/doc/development/internal_analytics/internal_event_tracking/architecture.md +++ b/doc/development/internal_analytics/internal_event_tracking/architecture.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md index 7e4222ead2e..591c6672810 100644 --- a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md +++ b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/internal_event_tracking/index.md b/doc/development/internal_analytics/internal_event_tracking/index.md index 73e9e2d1a4c..e35d5f6f084 100644 --- a/doc/development/internal_analytics/internal_event_tracking/index.md +++ b/doc/development/internal_analytics/internal_event_tracking/index.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/internal_event_tracking/introduction.md b/doc/development/internal_analytics/internal_event_tracking/introduction.md index ebb3caa198a..e776691fdf0 100644 --- a/doc/development/internal_analytics/internal_event_tracking/introduction.md +++ b/doc/development/internal_analytics/internal_event_tracking/introduction.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/internal_event_tracking/migration.md b/doc/development/internal_analytics/internal_event_tracking/migration.md new file mode 100644 index 00000000000..4b8a726768f --- /dev/null +++ b/doc/development/internal_analytics/internal_event_tracking/migration.md @@ -0,0 +1,155 @@ +--- +stage: Analyze +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Migrating existing tracking to internal event tracking + +GitLab Internal Events Tracking exposes a unified API on top of the existing tracking options. Currently RedisHLL and Snowplow are supported. + +This page describes how you can switch from tracking using a single method to using Internal Events Tracking. + +## Migrating from tracking with Snowplow + +If you are already tracking events in Snowplow, you can start collecting metrics also from self-managed instances by switching to Internal Events Tracking. + +Notice that the Snowplow event you trigger after switching to Internal Events Tracking looks slightly different from your current event. + +Please make sure that you are okay with this change before you migrate. + +### Backend + +If you are already tracking Snowplow events using `Gitlab::Tracking.event` and you want to migrate to Internal Events Tracking you might start with something like this: + +```ruby +Gitlab::Tracking.event(name, 'ci_templates_unique', namespace: namespace, + project: project, context: [context], user: user, label: label) +``` + +The code above can be replaced by something like this: + +```ruby +Gitlab::InternalEvents.track_event('ci_templates_unique', namespace: namespace, project: project, user: user) +``` + +In addition, you have to create definitions for the metrics that you would like to track. + +To generate metric definitions, you can use the generator like this: + +```shell +bin/rails g gitlab:analytics:internal_events \ + --time_frames=7d 28d\ + --group=project_management \ + --stage=plan \ + --section=dev \ + --event=ci_templates_unique \ + --unique=user.id \ + --mr=https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121544 +``` + +### Frontend + +If you are using the `Tracking` mixin in the Vue component, you can replace it with the `InternalEvents` mixin. + +For example, if your current Vue component look like this: + +```vue +import Tracking from '~/tracking'; +... +mixins: [Tracking.mixin()] +... +... +this.track('some_label', options) +``` + +After converting it to Internal Events Tracking, it should look like this: + +```vue +import { InternalEvents } from '~/tracking'; +... +mixins: [InternalEvents.mixin()] +... +... +this.track_event('action') +``` + +You can use [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123901/diffs) as an example. It migrates the `devops_adoption_app` component to use Internal Events Tracking. + +If you are using `data-track-action` in the component, you have to change it to `data-event-tracking` to migrate to Internal Events Tracking. + +For example, if a button is defined like this: + +```vue + <gl-button + :href="diffFile.external_url" + :title="externalUrlLabel" + :aria-label="externalUrlLabel" + target="_blank" + data-track-action="click_toggle_external_button" + data-track-label="diff_toggle_external_button" + data-track-property="diff_toggle_external" + icon="external-link" +/> +``` + +This can be converted to Internal Events Tracking like this: + +```vue + <gl-button + :href="diffFile.external_url" + :title="externalUrlLabel" + :aria-label="externalUrlLabel" + target="_blank" + data-event-tracking="click_toggle_external_button" + icon="external-link" +/> +``` + +Notice that we just need action to pass in the `data-event-tracking` attribute which will be passed to both Snowplow and RedisHLL. + +## Migrating from tracking with RedisHLL + +### Backend + +If you are currently tracking a metric in `RedisHLL` like this: + +```ruby + Gitlab::UsageDataCounters::HLLRedisCounter.track_event(:git_write_action, values: current_user.id) +``` + +To start using Internal Events Tracking, follow these steps: + +1. Create an event definition that describes `git_write_action` ([guide](../snowplow/event_dictionary_guide.md#create-a-new-event-definition)). +1. Find metric definitions that list `git_write_action` in the events section (`20210216182041_action_monthly_active_users_git_write.yml` and `20210216184045_git_write_action_weekly.yml`). +1. Change the `data_source` from `redis_hll` to `internal_events` in the metric definition files. +1. Add an `events` section to both metric definition files. + + ```yaml + events: + - name: git_write_action + unique: user.id + ``` + + Use `project.id` or `namespace.id` instead of `user.id` if your metric is counting something other than unique users. +1. Call `InternalEvents.tract_event` instead of `HLLRedisCounter.track_event`: + + ```diff + - Gitlab::UsageDataCounters::HLLRedisCounter.track_event(:git_write_action, values: current_user.id) + + Gitlab::InternalEvents.track_event('project_created', user: current_user) + ``` + +1. Optional. Add additional values to the event. You typically want to add `project` and `namespace` as it is useful information to have in the data warehouse. + + ```diff + - Gitlab::UsageDataCounters::HLLRedisCounter.track_event(:git_write_action, values: current_user.id) + + Gitlab::InternalEvents.track_event('project_created', user: current_user, project: project, namespace: namespace) + ``` + +1. Update your test to use the `internal event tracking` shared example. + +### Frontend + +If you are calling `trackRedisHllUserEvent` in the frontend to track the frontend event, you can convert this to Internal events by using mixin, raw JavaScript or data tracking attribute, + +[Quick start guide](quick_start.md#frontend-tracking) has example for each methods. diff --git a/doc/development/internal_analytics/internal_event_tracking/quick_start.md b/doc/development/internal_analytics/internal_event_tracking/quick_start.md index 84926657a3b..19c76ecc045 100644 --- a/doc/development/internal_analytics/internal_event_tracking/quick_start.md +++ b/doc/development/internal_analytics/internal_event_tracking/quick_start.md @@ -1,15 +1,39 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Quick start for internal event tracking +# Quick start for Internal Event Tracking In an effort to provide a more efficient, scalable, and unified tracking API, GitLab is deprecating existing RedisHLL and Snowplow tracking. Instead, we're implementing a new `track_event` method. -With this approach, we can both update RedisHLL counters and send Snowplow events simultaneously, streamlining the tracking process. +With this approach, we can update both RedisHLL counters and send Snowplow events without worrying about the underlying implementation. -## Create and trigger events +In order to instrument your code with Internal Events Tracking need three things: + +1. Define an event +1. Define one or more metrics +1. Trigger the event + +## Defining event and metrics + +To create event and metric definitions you can use the `internal_events` generator. + +This example will create an event definition for an event called `project_created` and two metric definitions which will be aggregated every 7 and 28 days. + +```shell +bin/rails g gitlab:analytics:internal_events \ +--time_frames=7d 28d \ +--group=project_management \ +--stage=plan --section=dev \ +--event=project_created \ +--unique=user.id \ +--mr=https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121544 +``` + +## Trigger events + +Triggering an event and thereby updating a metric is slightly different on backend and frontend. Please refer to the relevant section below. ### Backend tracking @@ -18,9 +42,9 @@ To trigger an event, call the `Gitlab::InternalEvents.track_event` method with t ```ruby Gitlab::InternalEvents.track_event( "i_code_review_user_apply_suggestion", - user_id: user_id, - namespace_id: namespace_id, - project_id: project_id + user: user, + namespace: namespace, + project: project ) ``` @@ -46,7 +70,7 @@ To implement Vue component tracking: ```javascript export default { mixins: [trackingMixin], - + data() { return { expanded: false, @@ -97,7 +121,7 @@ This attribute ensures that if we want to track GitLab internal events for a but </gl-button> ``` -For Haml +#### Haml ```haml = render Pajamas::ButtonComponent.new(button_options: { class: 'js-settings-toggle', data: { event_tracking: 'action' }}) do @@ -111,3 +135,7 @@ Sometimes we want to send internal events when the component is rendered or load = render Pajamas::ButtonComponent.new(button_options: { data: { event_tracking_load: 'true', event_tracking: 'i_devops' } }) do = _("New project") ``` + +### Limitations + +The only values we allow for `unique` are `user.id`, `project.id`, and `namespace.id`, as they are logged as part of the standard context. We currently don't have anywhere to put a value like `merge_request.id`. That will change with self-describing events. diff --git a/doc/development/internal_analytics/service_ping/implement.md b/doc/development/internal_analytics/service_ping/implement.md index 9dbfa02854d..c6da26f86c2 100644 --- a/doc/development/internal_analytics/service_ping/implement.md +++ b/doc/development/internal_analytics/service_ping/implement.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/index.md b/doc/development/internal_analytics/service_ping/index.md index 22e66a247c9..f532bb1ac31 100644 --- a/doc/development/internal_analytics/service_ping/index.md +++ b/doc/development/internal_analytics/service_ping/index.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/metrics_dictionary.md b/doc/development/internal_analytics/service_ping/metrics_dictionary.md index 8103db5113f..e677118fff6 100644 --- a/doc/development/internal_analytics/service_ping/metrics_dictionary.md +++ b/doc/development/internal_analytics/service_ping/metrics_dictionary.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -46,9 +46,9 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `performance_indicator_type` | no | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, `umau` or `customer_health_score`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). | | `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | | `milestone` | yes | The milestone when the metric is introduced and when it's available to self-managed instances with the official GitLab release. | -| `milestone_removed` | no | The milestone when the metric is removed. | +| `milestone_removed` | no | The milestone when the metric is removed. Required for removed metrics. | | `introduced_by_url` | no | The URL to the merge request that introduced the metric to be available for self-managed instances. | -| `removed_by_url` | no | The URL to the merge request that removed the metric. | +| `removed_by_url` | no | The URL to the merge request that removed the metric. Required for removed metrics. | | `repair_issue_url` | no | The URL of the issue that was created to repair a metric with a `broken` status. | | `options` | no | `object`: options information needed to calculate the metric value. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | diff --git a/doc/development/internal_analytics/service_ping/metrics_instrumentation.md b/doc/development/internal_analytics/service_ping/metrics_instrumentation.md index dc225a40d1b..ada7cc566a1 100644 --- a/doc/development/internal_analytics/service_ping/metrics_instrumentation.md +++ b/doc/development/internal_analytics/service_ping/metrics_instrumentation.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md index bb3d6797011..4980a8cf63d 100644 --- a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md +++ b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md b/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md index d7811c52bb1..63177f093e2 100644 --- a/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md +++ b/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/review_guidelines.md b/doc/development/internal_analytics/service_ping/review_guidelines.md index 8a46de7086e..c816c905097 100644 --- a/doc/development/internal_analytics/service_ping/review_guidelines.md +++ b/doc/development/internal_analytics/service_ping/review_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/service_ping/troubleshooting.md b/doc/development/internal_analytics/service_ping/troubleshooting.md index 8f5e94506cd..e685635c5f7 100644 --- a/doc/development/internal_analytics/service_ping/troubleshooting.md +++ b/doc/development/internal_analytics/service_ping/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -58,7 +58,7 @@ checking the configuration file of your GitLab instance: - Using the Admin Area: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage statistics**. @@ -116,7 +116,7 @@ To work around this bug, you have two options: sudo gitlab-ctl reconfigure ``` - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage statistics**. diff --git a/doc/development/internal_analytics/service_ping/usage_data.md b/doc/development/internal_analytics/service_ping/usage_data.md index b6ec3e00670..8742bc03fbb 100644 --- a/doc/development/internal_analytics/service_ping/usage_data.md +++ b/doc/development/internal_analytics/service_ping/usage_data.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/event_dictionary_guide.md b/doc/development/internal_analytics/snowplow/event_dictionary_guide.md index 6e8947e0210..c0d5e3efdfa 100644 --- a/doc/development/internal_analytics/snowplow/event_dictionary_guide.md +++ b/doc/development/internal_analytics/snowplow/event_dictionary_guide.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/implementation.md b/doc/development/internal_analytics/snowplow/implementation.md index 5ad97cf528c..5d328f22ca5 100644 --- a/doc/development/internal_analytics/snowplow/implementation.md +++ b/doc/development/internal_analytics/snowplow/implementation.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/index.md b/doc/development/internal_analytics/snowplow/index.md index 8265bceaf06..17d3f3f2cfc 100644 --- a/doc/development/internal_analytics/snowplow/index.md +++ b/doc/development/internal_analytics/snowplow/index.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -32,7 +32,7 @@ instances do not have a collector configured and do not collect data via Snowplo You can configure your self-managed GitLab instance to use a custom Snowplow collector. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Snowplow**. diff --git a/doc/development/internal_analytics/snowplow/infrastructure.md b/doc/development/internal_analytics/snowplow/infrastructure.md index 462dee2c39b..b856ccec78e 100644 --- a/doc/development/internal_analytics/snowplow/infrastructure.md +++ b/doc/development/internal_analytics/snowplow/infrastructure.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/review_guidelines.md b/doc/development/internal_analytics/snowplow/review_guidelines.md index 03d1812cbfc..a0bdad8fafb 100644 --- a/doc/development/internal_analytics/snowplow/review_guidelines.md +++ b/doc/development/internal_analytics/snowplow/review_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/schemas.md b/doc/development/internal_analytics/snowplow/schemas.md index 21142f68d39..2cc09500c36 100644 --- a/doc/development/internal_analytics/snowplow/schemas.md +++ b/doc/development/internal_analytics/snowplow/schemas.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_analytics/snowplow/troubleshooting.md b/doc/development/internal_analytics/snowplow/troubleshooting.md index b531c6dcd56..5eabba04792 100644 --- a/doc/development/internal_analytics/snowplow/troubleshooting.md +++ b/doc/development/internal_analytics/snowplow/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Analytics Instrumentation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 538b66124ba..81fd78d1d27 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -178,6 +178,40 @@ Example response: - GitLab Shell +## Authorized Certs + +This endpoint is called by the GitLab Shell to get the namespace that has a particular CA SSH certificate +configured. It also accepts `user_identifier` to return a GitLab user for specified identifier. + +| Attribute | Type | Required | Description | +|:----------------------|:-------|:---------|:------------| +| `key` | string | yes | The fingerprint of the SSH certificate. | +| `user_identifier` | string | yes | The identifier of the user to whom the SSH certificate has been issued (username or primary email). | + +```plaintext +GET /internal/authorized_certs +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shell-Api-Request: <JWT token>" "http://localhost:3001/api/v4/internal/authorized_certs?key=<key>&user_identifier=<user_identifier>" +``` + +Example response: + +```json +{ + "success": true, + "namespace": "gitlab-org", + "username": "root" +} +``` + +### Known consumers + +- GitLab Shell + ## Get user for user ID or key This endpoint is used when a user performs `ssh git@gitlab.com`. It @@ -492,21 +526,24 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ Called from GitLab agent server (`kas`) to increase the usage metric counters. -| Attribute | Type | Required | Description | -|:--------------------------------------------------------------------------|:--------------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `counters` | hash | no | Hash of counters | -| `counters["k8s_api_proxy_request"]` | integer | no | The number to increase the `k8s_api_proxy_request` counter by | -| `counters["gitops_sync"]` | integer | no | The number to increase the `gitops_sync` counter by | -| `counters["flux_git_push_notifications_total"]` | integer | no | The number to increase the `flux_git_push_notifications_total` counter by | -| `counters["k8s_api_proxy_requests_via_ci_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_ci_access` counter by | -| `counters["k8s_api_proxy_requests_via_user_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_user_access` counter by | -| `unique_counters` | hash | no | Array of unique numbers | -| `unique_counters["agent_users_using_ci_tunnel"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel to track the `agent_users_using_ci_tunnel` metric event | -| `unique_counters["k8s_api_proxy_requests_unique_users_via_ci_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_users_via_ci_access` metric event | -| `unique_counters["k8s_api_proxy_requests_unique_agents_via_ci_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_agents_via_ci_access` metric event | -| `unique_counters["k8s_api_proxy_requests_unique_users_via_user_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_users_via_user_access` metric event | -| `unique_counters["k8s_api_proxy_requests_unique_agents_via_user_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_agents_via_user_access` metric event | -| `unique_counters["flux_git_push_notified_unique_projects"]` | integer array | no | The set of unique projects ids that have been notified to reconcile their Flux workloads to track the `flux_git_push_notified_unique_projects` metric event | +| Attribute | Type | Required | Description | +|:--------------------------------------------------------------------------|:--------------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `counters` | hash | no | Hash of counters | +| `counters["k8s_api_proxy_request"]` | integer | no | The number to increase the `k8s_api_proxy_request` counter by | +| `counters["gitops_sync"]` | integer | no | The number to increase the `gitops_sync` counter by | +| `counters["flux_git_push_notifications_total"]` | integer | no | The number to increase the `flux_git_push_notifications_total` counter by | +| `counters["k8s_api_proxy_requests_via_ci_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_ci_access` counter by | +| `counters["k8s_api_proxy_requests_via_user_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_user_access` counter by | +| `counters["k8s_api_proxy_requests_via_pat_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_pat_access` counter by | +| `unique_counters` | hash | no | Array of unique numbers | +| `unique_counters["agent_users_using_ci_tunnel"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel to track the `agent_users_using_ci_tunnel` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_users_via_ci_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_users_via_ci_access` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_agents_via_ci_access"]` | integer array | no | The set of unique agent ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_agents_via_ci_access` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_users_via_user_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_users_via_user_access` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_agents_via_user_access"]` | integer array | no | The set of unique agent ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_agents_via_user_access` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_users_via_pat_access"]` | integer array | no | The set of unique user ids that have used the KAS Kubernetes API proxy via PAT to track the `k8s_api_proxy_requests_unique_users_via_pat_access` metric event | +| `unique_counters["k8s_api_proxy_requests_unique_agents_via_pat_access"]` | integer array | no | The set of unique agent ids that have used the KAS Kubernetes API proxy via PAT to track the `k8s_api_proxy_requests_unique_agents_via_pat_access` metric event | +| `unique_counters["flux_git_push_notified_unique_projects"]` | integer array | no | The set of unique projects ids that have been notified to reconcile their Flux workloads to track the `flux_git_push_notified_unique_projects` metric event | ```plaintext POST /internal/kubernetes/usage_metrics @@ -819,7 +856,7 @@ Example response: ## Subscription add-on purchases (excluding storage and compute packs) -The subscription add-on purchase endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) to apply subscription add-on purchases like code suggestions for personal namespaces, or top-level groups within GitLab.com. It is not used to apply storage and compute pack purchases. +The subscription add-on purchase endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) to apply subscription add-on purchases like Code Suggestions for personal namespaces, or top-level groups within GitLab.com. It is not used to apply storage and compute pack purchases. ### Create a subscription add-on purchase @@ -831,9 +868,9 @@ POST /namespaces/:id/subscription_add_on_purchase/:add_on_name | Attribute | Type | Required | Description | |:------------|:--------|:---------|:------------| -| `quantity` | integer | yes | Amount of units in the subscription add-on purchase (Example: Number of seats for a code suggestions add-on) | +| `quantity` | integer | yes | Amount of units in the subscription add-on purchase (Example: Number of seats for a Code Suggestions add-on) | | `expires_on` | date | yes | Expiration date of the subscription add-on purchase | -| `purchase_xid` | string | yes | Identifier for the subscription add-on purchase (Example: Subscription name for a code suggestions add-on) | +| `purchase_xid` | string | yes | Identifier for the subscription add-on purchase (Example: Subscription name for a Code Suggestions add-on) | Example request: @@ -864,9 +901,9 @@ PUT /namespaces/:id/subscription_add_on_purchase/:add_on_name | Attribute | Type | Required | Description | |:------------|:--------|:---------|:------------| -| `quantity` | integer | no | Amount of units in the subscription add-on purchase (Example: Number of seats for a code suggestions add-on) | +| `quantity` | integer | no | Amount of units in the subscription add-on purchase (Example: Number of seats for a Code Suggestions add-on) | | `expires_on` | date | yes | Expiration date of the subscription add-on purchase | -| `purchase_xid` | string | no | Identifier for the subscription add-on purchase (Example: Subscription name for a code suggestions add-on) | +| `purchase_xid` | string | no | Identifier for the subscription add-on purchase (Example: Subscription name for a Code Suggestions add-on) | Example request: @@ -1365,7 +1402,7 @@ Example request: ```shell curl --verbose --request PATCH "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ - --data '{ "Operations": [{"op":"Update","path":"name.formatted","value":"New Name"}] }' \ + --data '{ "Operations": [{"op":"replace","path":"id","value":"1234abcd"}] }' \ --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" ``` diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md index 1c12e541149..cf45cf941d0 100644 --- a/doc/development/internal_users.md +++ b/doc/development/internal_users.md @@ -43,7 +43,7 @@ Other examples of internal users: - [GitLab Admin Bot](https://gitlab.com/gitlab-org/gitlab/-/blob/278bc9018dd1515a10cbf15b6c6cd55cb5431407/app/models/user.rb#L950-960) - [Alert Bot](../operations/incident_management/alerts.md#trigger-actions-from-alerts) - [Ghost User](../user/profile/account/delete_account.md#associated-records) -- [Support Bot](../user/project/service_desk/index.md#support-bot-user) +- [Support Bot](../user/project/service_desk/configure.md#support-bot-user) - Visual Review Bot - Resource access tokens, including: - [Project access tokens](../user/project/settings/project_access_tokens.md). diff --git a/doc/development/merge_request_concepts/performance.md b/doc/development/merge_request_concepts/performance.md index 665b84b2c40..7e26bf982b2 100644 --- a/doc/development/merge_request_concepts/performance.md +++ b/doc/development/merge_request_concepts/performance.md @@ -195,20 +195,18 @@ costly, time-consuming query to the replicas. Read about [complex queries on the relation object](../database/iterating_tables_in_batches.md#complex-queries-on-the-relation-object) for considerations on how to use CTEs. We have found in some situations that CTEs can become -problematic in use (similar to the n+1 problem above). In particular, hierarchical recursive +problematic in use (similar to the N+1 problem above). In particular, hierarchical recursive CTE queries such as the CTE in [AuthorizedProjectsWorker](https://gitlab.com/gitlab-org/gitlab/-/issues/325688) are very difficult to optimize and don't scale. We should avoid them when implementing new features that require any kind of hierarchical structure. CTEs have been effectively used as an optimization fence in many simpler cases, such as this [example](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43242#note_61416277). -Beginning in PostgreSQL 12, CTEs are inlined then [optimized by default](https://paquier.xyz/postgresql-2/postgres-12-with-materialize/). -Keeping the old behavior requires marking CTEs with the keyword `MATERIALIZED`. +With the currently supported PostgreSQL versions, the optimization fence behavior must be enabled +with the `MATERIALIZED` keyword. By default CTEs are inlined then [optimized by default](https://paquier.xyz/postgresql-2/postgres-12-with-materialize/). When building CTE statements, use the `Gitlab::SQL::CTE` class [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56976) in GitLab 13.11. -By default, this `Gitlab::SQL::CTE` class forces materialization through adding the `MATERIALIZED` keyword for PostgreSQL 12 and higher. -`Gitlab::SQL::CTE` automatically omits materialization when PostgreSQL 11 is running -(this behavior is implemented using a custom Arel node `Gitlab::Database::AsWithMaterialized` under the surface). +By default, this `Gitlab::SQL::CTE` class forces materialization through adding the `MATERIALIZED` keyword. WARNING: Upgrading to GitLab 14.0 requires PostgreSQL 12 or later. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 65dc4de30d1..9be322812e3 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1290,6 +1290,48 @@ class BuildMetadata end ``` +Additionally, you can expose the keys in a `JSONB` column as +ActiveRecord attributes. Do this when you need complex validations, +or ActiveRecord change tracking. This feature is provided by the +[`jsonb_accessor`](https://github.com/madeintandem/jsonb_accessor) gem, +and does not replace `JsonSchemaValidator`. + +```ruby +module Organizations + class OrganizationSetting < ApplicationRecord + belongs_to :organization + + validates :settings, json_schema: { filename: "organization_settings" } + + jsonb_accessor :settings, + restricted_visibility_levels: [:integer, { array: true }] + + validates_each :restricted_visibility_levels do |record, attr, value| + value&.each do |level| + unless Gitlab::VisibilityLevel.options.value?(level) + record.errors.add(attr, format(_("'%{level}' is not a valid visibility level"), level: level)) + end + end + end + end +end +``` + +You can now use `restricted_visibility_levels` as an ActiveRecord attribute: + +```ruby +> s = Organizations::OrganizationSetting.find(1) +=> #<Organizations::OrganizationSetting:0x0000000148d67628> +> s.settings +=> {"restricted_visibility_levels"=>[20]} +> s.restricted_visibility_levels +=> [20] +> s.restricted_visibility_levels = [0] +=> [0] +> s.changes +=> {"settings"=>[{"restricted_visibility_levels"=>[20]}, {"restricted_visibility_levels"=>[0]}], "restricted_visibility_levels"=>[[20], [0]]} +``` + ## Encrypted attributes > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227779) in GitLab 14.0. diff --git a/doc/development/packages/debian_repository.md b/doc/development/packages/debian_repository.md index b3e9bedfdd4..2d8ba98f9ad 100644 --- a/doc/development/packages/debian_repository.md +++ b/doc/development/packages/debian_repository.md @@ -17,11 +17,13 @@ This guide explains: There are two types of [Debian packages](https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html): binary and source. - **Binary** - These are usually `.deb` files and contain executables, config files, and other data. A binary package must match your OS or architecture since it is already compiled. These are usually installed using `dpkg`. Dependencies must already exist on the system when installing a binary package. -- **Source** - These are usual made up of `.dsc` files and `.gz` files. A source package is compiled on your system. These are fetched and installed with [`apt`](https://manpages.debian.org/bullseye/apt/apt.8.en.html), which then uses `dpkg` after the package is compiled. When you use `apt`, it will fetch and install the necessary dependencies. +- **Source** - These are usually made up of `.dsc` files and compressed `.tar` files. A source package may be compiled on your system. -The `.deb` file follows the naming convention `<PackageName>_<VersionNumber>-<DebianRevisionNumber>_<DebianArchitecture>.deb` +Packages are fetched with [`apt`](https://manpages.debian.org/bullseye/apt/apt.8.en.html) and installed with `dpkg`. When you use `apt`, it also fetches and installs any dependencies. -It includes a `control file` that contains metadata about the package. You can view the control file by using `dpkg --info <deb_file>` +The `.deb` file follows the naming convention `<PackageName>_<VersionNumber>-<DebianRevisionNumber>_<DebianArchitecture>.deb`. + +It includes a `control file` that contains metadata about the package. You can view the control file by using `dpkg --info <deb_file>`. The [`.changes` file](https://www.debian.org/doc/debian-policy/ch-controlfields.html#debian-changes-files-changes) is used to tell the Debian repository how to process updates to packages. It contains a variety of metadata for the package, including architecture, distribution, and version. In addition to the metadata, they contain three lists of checksums: `sha1`, `sha256`, and `md5` in the `Files` section. Refer to [sample_1.2.3~alpha2_amd64.changes](https://gitlab.com/gitlab-org/gitlab/-/blob/dd1e70d3676891025534dc4a1e89ca9383178fe7/spec/fixtures/packages/debian/sample_1.2.3~alpha2_amd64.changes) for an example of how these files are structured. @@ -40,8 +42,8 @@ When it comes to Debian, packages don't exist on their own. They belong to a _di ## What does a Debian Repository look like? - A [Debian repository](https://wiki.debian.org/DebianRepository) is made up of many releases. -- Each release is given a **codename**. For the public Debian repository, these are things like "bullseye" and "jesse". - - There is also the concept of **suites** which are essentially aliases of codenames synonymous with release channels like "stable" and "edge". +- Each release is given a stable **codename**. For the public Debian repository, these are names like "bullseye" and "jessie". + - There is also the concept of **suites** which are essentially aliases of codenames synonymous with release channels like "stable" and "edge". Over time they change and point to different _codenames_. - Each release has many **components**. In the public repository, these are "main", "contrib", and "non-free". - Each release has many **architectures** such as "amd64", "arm64", or "i386". - Each release has a signed **Release** file (see below about [GPG signing](#what-are-gpg-keys-and-what-are-signed-releases)) diff --git a/doc/development/performance.md b/doc/development/performance.md index fd7e9a85fba..428d5637aa9 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -250,7 +250,7 @@ the timeout. Once profiling stops, the profile is written out to disk at `$STACKPROF_FILE_PREFIX/stackprof.$PID.$RAND.profile`. It can then be inspected -further through the `stackprof` command line tool, as described in the +further through the `stackprof` command-line tool, as described in the [Reading a Stackprof profile section](#reading-a-stackprof-profile). Currently supported profiling targets are: diff --git a/doc/development/permissions.md b/doc/development/permissions.md index aa58447b818..35fd0f1c440 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/permissions/authorizations.md b/doc/development/permissions/authorizations.md index 8d8944562a8..7580b7c473b 100644 --- a/doc/development/permissions/authorizations.md +++ b/doc/development/permissions/authorizations.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/permissions/custom_roles.md b/doc/development/permissions/custom_roles.md index 337c8f6d96b..d317d943cd3 100644 --- a/doc/development/permissions/custom_roles.md +++ b/doc/development/permissions/custom_roles.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/permissions/predefined_roles.md b/doc/development/permissions/predefined_roles.md index 9afc5966e93..50e8fbfd5b3 100644 --- a/doc/development/permissions/predefined_roles.md +++ b/doc/development/permissions/predefined_roles.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -96,6 +96,20 @@ NOTE: In [GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/issues/351211) and later, projects in personal namespaces have a maximum role of Owner. Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/219299) in GitLab 14.8 and earlier, projects in personal namespaces have a maximum role of Maintainer. +#### Guest role + +A user with the Guest role in GitLab can view project plans, blockers and other +progress indicators. While unable to modify data they have not created, Guests +can contribute to a project by creating and linking project work items. Guests +can also view high-level project information such as: + +- Analytics. +- Incident information. +- Issues and epics. +- Licenses. + +For more information, see [project member permissions](../../user/permissions.md#project-members-permissions). + ### Confidential issues [Confidential issues](../../user/project/issues/confidential_issues.md) can be accessed diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index 316ae3c83cd..a5b654e96e2 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -278,8 +278,8 @@ See `.review:rules:start-review-app-pipeline` in [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for the specific list of rules. -If you want to force a Review App to be deployed regardless of your changes, you can add the -`pipeline:run-review-app` label to the merge request. +If you want to deploy a Review App in a merge request, you can either trigger the `start-review-app-pipeline` manual job in the CI/CD pipeline, or add the +`pipeline:run-review-app` label to the merge request and run a new CI/CD pipeline. Consult the [Review Apps](../testing_guide/review_apps.md) dedicated page for more information. diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 1a4e4e738a8..0e2c1c991fd 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -152,7 +152,7 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k | `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | | `.ruby-cache` | Allows a job to use a default `cache` definition suitable for Ruby tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | -| `.coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | +| `.ruby-gems-coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | | `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | | `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | | `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | @@ -164,7 +164,7 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k | `.use-pg15-ee` | Same as `.use-pg15` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | -| `.use-docker-in-docker` | Allows a job to use Docker in Docker. | +| `.use-docker-in-docker` | Allows a job to use Docker in Docker. For more details, see the [handbook about CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). | ## `rules`, `if:` conditions and `changes:` patterns @@ -198,7 +198,7 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if-merge-request` | Matches if the pipeline is for a merge request. | | | `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | | `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | -| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | +| `if-merge-request-labels-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | | `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | | `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | @@ -396,7 +396,6 @@ For this scenario, you have to: - `GITALY_SERVER_VERSION` - `GITLAB_ELASTICSEARCH_INDEXER_VERSION` - `GITLAB_KAS_VERSION` - - `GITLAB_METRICS_EXPORTER_VERSION` - `GITLAB_PAGES_VERSION` - `GITLAB_SHELL_VERSION` - `scripts/trigger-build.rb` @@ -417,3 +416,25 @@ For this scenario, you have to: - `spec/simplecov_env.rb` Additionally, `scripts/utils.sh` is always downloaded from the API when this pattern is used (this file contains the code for `.fast-no-clone-job`). + +#### Runner tags + +On GitLab.com, both unprivileged and privileged runners are +available. For projects in the `gitlab-org` group and forks of those +projects, only one of the following tags should be added to a job: + +- `gitlab-org`: Jobs randomly use privileged and unprivileged runners. +- `gitlab-org-docker`: Jobs must use a privileged runner. If you need [Docker-in-Docker support](../../ci/docker/using_docker_build.md#use-docker-in-docker), +use `gitlab-org-docker` instead of `gitlab-org`. + +The `gitlab-org-docker` tag is added by the `.use-docker-in-docker` job +definition above. + +To ensure compatibility with forks, avoid using both `gitlab-org` and +`gitlab-org-docker` simultaneously. No instance runners +have both `gitlab-org` and `gitlab-org-docker` tags. For forks of +`gitlab-org` projects, jobs will get stuck if both tags are supplied because +no matching runners are available. + +See [the GitLab Repositories handbook page](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration) +for more information. diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 2dbed640fbb..7db498adc02 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -40,7 +40,7 @@ This works well for the following reasons: - `.ruby-cache` - `.static-analysis-cache` - `.rubocop-cache` - - `.coverage-cache` + - `.ruby-gems-coverage-cache` - `.ruby-node-cache` - `.qa-cache` - `.yarn-cache` diff --git a/doc/development/policies.md b/doc/development/policies.md index 38f1fc54bf4..faf5b32985f 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md index 32295cc0e43..772206c2d73 100644 --- a/doc/development/rails_update.md +++ b/doc/development/rails_update.md @@ -4,11 +4,11 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Rails update guidelines +# Rails upgrade guidelines We strive to run GitLab using the latest Rails releases to benefit from performance, security updates, and new features. -## Rails update approach +## Rails upgrade approach 1. [Prepare an MR for GitLab](#prepare-an-mr-for-gitlab). 1. [Prepare an MR for Gitaly](#prepare-an-mr-for-gitaly). @@ -40,7 +40,7 @@ We strive to run GitLab using the latest Rails releases to benefit from performa ### Create patch releases and backports for security patches -If the Rails update was over a patch release and it contains important security fixes, +If the Rails upgrade was over a patch release and it contains important security fixes, make sure to release it in a GitLab patch release to self-managed customers. Consult with our [release managers](https://about.gitlab.com/community/release-managers/) for how to proceed. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index b15c4eca5ae..d879668649d 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -30,7 +30,9 @@ See also [Mass inserting Rails models](mass_insert.md). **LARGE_PROJECTS**: Create large projects (through import) from a predefined set of URLs. -### Seeding issues for all or a given project +### Seeding Data + +#### Seeding issues for all projects or a single project You can seed issues for all or a given project with the `gitlab:seed:issues` task: @@ -197,6 +199,14 @@ bundle exec rake "gitlab:seed:ci_variables_instance" bundle exec rake "gitlab:seed:ci_variables_instance[25, CI_VAR_]" ``` +#### Seed a project for merge train development + +Seeds a project with merge trains configured and 20 merge requests(each with 3 commits). The command: + +```shell +rake gitlab:seed:merge_trains:project +``` + ### Automation If you're very sure that you want to **wipe the current database** and refill @@ -520,6 +530,11 @@ The following command combines the intent of [Update GraphQL documentation and s bundle exec rake gitlab:graphql:update_all ``` +## Update audit event types documentation + +For information on updating audit event types documentation, see +[Generate documentation](audit_event_guide/index.md#generate-documentation). + ## Update OpenAPI client for Error Tracking feature NOTE: diff --git a/doc/development/redis.md b/doc/development/redis.md index ebc7c0271a1..19a6b8e75d4 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -56,7 +56,7 @@ the entry, instead of relying on the key changing. ### Multi-key commands -GitLab supports Redis Cluster only for the Redis [rate-limiting](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/rate_limiting.rb) type, introduced in [epic 823](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/823). +GitLab supports Redis Cluster for [cache-related workloads](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/cache.rb) type, introduced in [epic 878](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/878). This imposes an additional constraint on naming: where GitLab is performing operations that require several keys to be held on the same Redis server - for @@ -81,6 +81,12 @@ Developers are highly encouraged to use [hash-tags](https://redis.io/docs/refere where appropriate to facilitate future adoption of Redis Cluster in more Redis types. For example, the Namespace model uses hash-tags for its [config cache keys](https://gitlab.com/gitlab-org/gitlab/-/blob/1a12337058f260d38405886d82da5e8bb5d8da0b/app/models/namespace.rb#L786). +To perform multi-key commands, developers may use the [`Gitlab::Redis::CrossSlot::Pipeline`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/cross_slot.rb) wrapper. +However, this does not work for [transactions](https://redis.io/docs/interact/transactions/) as Redis Cluster does not support cross-slot transactions. + +For `Rails.cache`, we handle the `MGET` command found in `read_multi_get` by [patching it](https://gitlab.com/gitlab-org/gitlab/-/blob/c2bad2aac25e2f2778897bd4759506a72b118b15/lib/gitlab/patch/redis_cache_store.rb#L10) to use the `Gitlab::Redis::CrossSlot::Pipeline` wrapper. +The minimum size of the pipeline is set to 1000 commands and it can be adjusted by using the `GITLAB_REDIS_CLUSTER_PIPELINE_BATCH_LIMIT` environment variable. + ## Redis in structured logging For GitLab Team Members: There are <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> diff --git a/doc/development/rubocop_development_guide.md b/doc/development/rubocop_development_guide.md index 1fdc0fbe78c..f6c11a0c7e3 100644 --- a/doc/development/rubocop_development_guide.md +++ b/doc/development/rubocop_development_guide.md @@ -28,6 +28,12 @@ discussions, nitpicking, or back-and-forth in reviews. The [GitLab Ruby style guide](backend/ruby_style_guide.md) includes a non-exhaustive list of styles that commonly come up in reviews and are not enforced. +By default, we should not +[disable a RuboCop rule inline](https://docs.rubocop.org/rubocop/configuration.html#disabling-cops-within-source-code), because it negates agreed-upon code standards that the rule is attempting to apply to the codebase. + +If you must use inline disable, provide the reason on the MR and ensure the reviewers agree +before merging. + Additionally, we have dedicated [test-specific style guides and best practices](testing_guide/index.md). @@ -53,7 +59,7 @@ A cop is in a _grace period_ if it is enabled and has `Details: grace period` de On the default branch, offenses from cops in the [grace period](rake_tasks.md#run-rubocop-in-graceful-mode) do not fail the RuboCop CI job. Instead, the job notifies the `#f_rubocop` Slack channel. However, on other branches, the RuboCop job fails. -A grace period can safely be lifted as soon as there are no warnings for 2 weeks in the `#f_rubocop` channel on Slack. +A grace period can safely be lifted as soon as there are no warnings for 1 week in the `#f_rubocop` channel on Slack. ## Enabling a new cop @@ -61,7 +67,7 @@ A grace period can safely be lifted as soon as there are no warnings for 2 weeks 1. [Generate TODOs for the new cop](rake_tasks.md#generate-initial-rubocop-todo-list). 1. [Set the new cop to `grace period`](#cop-grace-period). 1. Create an issue to fix TODOs and encourage community contributions (via ~"quick win" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=quick%20wins&label_name%5B%5D=static%20code%20analysis&first_page_size=20). -1. Create an issue to remove `grace period` after 2 weeks of silence in the `#f_rubocop` Slack channel. [See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903). +1. Create an issue to remove `grace period` after 1 week of silence in the `#f_rubocop` Slack channel. [See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903). ## Silenced offenses diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index 21c19c31b0a..d3629bc8dba 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -272,5 +272,5 @@ These experimental branches are not intended to be merged; they can be closed on and merged back independently. - **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week -before the 22nd. This gives us extra time to act if something breaks. If in doubt, it is better to +before release day. This gives us extra time to act if something breaks. If in doubt, it is better to postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/development/principles/#prioritizing-technical-decisions). diff --git a/doc/development/sec/token_revocation_api.md b/doc/development/sec/token_revocation_api.md index e2006ba519c..860faf30cf6 100644 --- a/doc/development/sec/token_revocation_api.md +++ b/doc/development/sec/token_revocation_api.md @@ -108,8 +108,8 @@ For example, to configure these values in the ```ruby ::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN') -::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens') -::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://gitlab.example.com/revocation_service/v1/revoke_tokens') +::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://gitlab.example.com/revocation_service/v1/revocable_token_types') ::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true) ``` diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 186239cc547..806fbd8d1f6 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1279,7 +1279,7 @@ Credentials can be: - Login details like username and password. - Private keys. -- Tokens (PAT, runner tokens, JWT token, CSRF tokens, project access tokens, etc). +- Tokens (PAT, runner authentication tokens, JWT token, CSRF tokens, project access tokens, etc). - Session cookies. - Any other piece of information that can be used for authentication or authorization purposes. diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md deleted file mode 100644 index c1077793fb9..00000000000 --- a/doc/development/service_ping/implement.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/implement.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/implement.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md deleted file mode 100644 index d0806ed375b..00000000000 --- a/doc/development/service_ping/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/index.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/index.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md deleted file mode 100644 index fecab4916f5..00000000000 --- a/doc/development/service_ping/metrics_dictionary.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/metrics_dictionary.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/metrics_dictionary.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md deleted file mode 100644 index 5a4dfc325e2..00000000000 --- a/doc/development/service_ping/metrics_instrumentation.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/metrics_instrumentation.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/metrics_instrumentation.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md deleted file mode 100644 index 520b18139ff..00000000000 --- a/doc/development/service_ping/metrics_lifecycle.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/metrics_lifecycle.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/metrics_lifecycle.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/performance_indicator_metrics.md b/doc/development/service_ping/performance_indicator_metrics.md deleted file mode 100644 index eda7224732d..00000000000 --- a/doc/development/service_ping/performance_indicator_metrics.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/performance_indicator_metrics.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/performance_indicator_metrics.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md deleted file mode 100644 index d5805f615e2..00000000000 --- a/doc/development/service_ping/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/review_guidelines.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md deleted file mode 100644 index 31b04c1a6bc..00000000000 --- a/doc/development/service_ping/troubleshooting.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/troubleshooting.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/troubleshooting.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md deleted file mode 100644 index 94ae90273d0..00000000000 --- a/doc/development/service_ping/usage_data.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/service_ping/usage_data.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/service_ping/usage_data.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 25f62fbcc98..321bd7aeadd 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -88,7 +88,7 @@ cat: illegal option -- l usage: cat [-benstuv] [file ...] ``` -In the example above, the argument parser of `cat` assumes that `-l` is an option. The solution in the example above is to make it clear to `cat` that `-l` is really an argument, not an option. Many Unix command line tools follow the convention of separating options from arguments with `--`. +In the example above, the argument parser of `cat` assumes that `-l` is an option. The solution in the example above is to make it clear to `cat` that `-l` is really an argument, not an option. Many Unix command-line tools follow the convention of separating options from arguments with `--`. ```shell # Example (continued) diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md deleted file mode 100644 index 2bea681bf59..00000000000 --- a/doc/development/snowplow/event_dictionary_guide.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/event_dictionary_guide.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/event_dictionary_guide.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md deleted file mode 100644 index a9e4e252a53..00000000000 --- a/doc/development/snowplow/implementation.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/implementation.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/implementation.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md deleted file mode 100644 index c0e53fe3b1b..00000000000 --- a/doc/development/snowplow/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/index.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/index.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/infrastructure.md b/doc/development/snowplow/infrastructure.md deleted file mode 100644 index 6374af40ffe..00000000000 --- a/doc/development/snowplow/infrastructure.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/infrastructure.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/infrastructure.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md deleted file mode 100644 index f4752e08dde..00000000000 --- a/doc/development/snowplow/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/review_guidelines.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md deleted file mode 100644 index 7e00ddd976d..00000000000 --- a/doc/development/snowplow/schemas.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/schemas.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/schemas.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md deleted file mode 100644 index ed1f5033239..00000000000 --- a/doc/development/snowplow/troubleshooting.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_analytics/snowplow/troubleshooting.md' -remove_date: '2023-08-20' ---- - -This document was moved to [another location](../internal_analytics/snowplow/troubleshooting.md). - -<!-- This redirect file can be deleted after <2023-08-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 65787f7a355..49739d7c8e9 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -911,7 +911,8 @@ so we need to set some guidelines for their use going forward: ### Common test setup NOTE: -`before_all` does not work with the `:delete` strategy. For more information, see [issue 420379](https://gitlab.com/gitlab-org/gitlab/-/issues/420379). +`let_it_be` and `before_all` do not work with DatabaseCleaner's deletion strategy. This includes migration specs, Rake task specs, and specs that have the `:delete` RSpec metadata tag. +For more information, see [issue 420379](https://gitlab.com/gitlab-org/gitlab/-/issues/420379). In some cases, there is no need to recreate the same object for tests again for each example. For example, a project and a guest of that project @@ -1060,7 +1061,7 @@ Failing with an error along the lines of: ```shell expected { "assignee_id" => nil, "...1 +0000 } to include {"customer_relations_contacts" => [{:created_at => "2023-08-04T13:30:20Z", :first_name => "Sidney Jones3" }]} - + Diff: @@ -1,35 +1,69 @@ -"customer_relations_contacts" => [{:created_at=>"2023-08-04T13:30:20Z", :first_name=>"Sidney Jones3" }], @@ -1225,7 +1226,7 @@ specs, so created repositories accumulate in this directory over the lifetime of the process. Deleting them is expensive, but this could lead to pollution unless carefully managed. -To avoid this, [hashed storage](../../administration/repository_storage_types.md) +To avoid this, [hashed storage](../../administration/repository_storage_paths.md) is enabled in the test suite. This means that repositories are given a unique path that depends on their project's ID. Because the project IDs are not reset between specs, each spec gets its own repository on disk, diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 4627d5d29cb..12f90e0d88c 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -266,7 +266,7 @@ ensuring we now sign in at the beginning of each test. Next, let's test something other than Login. Let's test Issues, which are owned by the Plan stage and the Project Management Group, so [create a file](#identify-the-devops-stage) in -`qa/specs/features/browser_ui/3_create/issues` called `issues_spec.rb`. +`qa/specs/features/browser_ui/2_plan/issue` called `issues_spec.rb`. ```ruby # frozen_string_literal: true @@ -274,12 +274,7 @@ stage and the Project Management Group, so [create a file](#identify-the-devops- module QA RSpec.describe 'Plan' do describe 'Issues', product_group: :project_management do - let(:issue) do - Resource::Issue.fabricate_via_api! do |issue| - issue.title = 'My issue' - issue.description = 'This is an issue specific to this test' - end - end + let(:issue) { create(:issue) } before do Flow::Login.sign_in diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 96bd02d235b..ab4dd9acb63 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -169,18 +169,14 @@ Page::Main::Menu.perform do |menu| end #=> Good -issue = Resource::Issue.fabricate_via_api! do |issue| - issue.name = 'issue-name' -end +issue = create(:issue, name: 'issue-name') Project::Issues::Index.perform do |index| expect(index).to have_issue(issue) end #=> Bad -issue = Resource::Issue.fabricate_via_api! do |issue| - issue.name = 'issue-name' -end +issue = create(:issue, name: 'issue-name') Project::Issues::Index.perform do |index| expect(index).to have_issue(issue) @@ -371,7 +367,7 @@ If the _only_ action in the test that requires administrator access is to toggle In line with [using the API](#prefer-api-over-ui), use a `Commit` resource whenever possible. -`ProjectPush` uses raw shell commands via the Git Command Line Interface (CLI) whereas the `Commit` resource makes an HTTP request. +`ProjectPush` uses raw shell commands from the Git command-line interface (CLI), and the `Commit` resource makes an HTTP request. ```ruby # Using a commit resource diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 4e7ef6f29a2..30fb1abd7df 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -55,17 +55,17 @@ graph TB A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline<br>and wait for it to be done| B1 B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline<br>and wait for it to be done| C1 -subgraph "`gitlab-org/gitlab` pipeline" +subgraph " `gitlab-org/gitlab` pipeline" A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs] A2[`qa` stage<br>`e2e:package-and-test` job] end -subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline" +subgraph " `gitlab-org/build/omnibus-gitlab-mirror` pipeline" B1[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end -subgraph "`gitlab-org/gitlab-qa-mirror` pipeline" - C1>End-to-end jobs run] +subgraph " `gitlab-org/gitlab-qa-mirror` pipeline" + C1[End-to-end jobs run] end ``` diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index becdc375c63..bf3f1c25f5e 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -392,7 +392,7 @@ In this case, the result is similar to calling `Resource::Shirt.fabricate!`. ### Factories -You may also use FactoryBot invocations to create resources within your tests. +You may also use [FactoryBot](https://github.com/thoughtbot/factory_bot/) invocations to create, build, and fetch resources within your tests. ```ruby # create a project via the API to use in the test @@ -403,9 +403,81 @@ let(:issue) { create(:issue, project: project) } # create a private project via the API with a specific name let(:project) { create(:project, :private, name: 'my-project-name', add_name_uuid: false) } + +### + +# instantiate an Issue but don't create it via API yet +let(:issue) { build(:issue) } + +# instantiate a Project and perform some actions before creating +let(:project) do + build(:project) do |p| + p.name = 'Test' + p.add_name_uuid = false + end +end + +# fetch an existing issue via the API with attributes +let(:existing_issue) { build(:issue, project: project, iid: issue.iid).reload! } +``` + +All factories are defined in [`qa/qa/factories`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/factories/) and are representative of +their respective `QA::Resource::Base` class. + +For example, a factory `:issue` can be found in `qa/resource/issue.rb`. A factory `:project` can be found in `qa/resource/project.rb`. + +#### Create a new Factory + +Given a resource: + +```ruby +# qa/resource/shirt.rb +module QA + module Resource + class Shirt < Base + attr_accessor :name + attr_reader :read_only + + attribute :brand + + def api_post_body + { name: name, brand: brand } + end + end + end +end ``` -All factories are defined in [`qa/qa/factories`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/factories/). +Define a factory with defaults and overrides: + +```ruby +# qa/factories/shirts.rb +module QA + FactoryBot.define do + factory :shirt, class: 'QA::Resource::Shirt' do + brand { 'BrandName' } + + trait :with_name do + name { 'Shirt Name' } + end + end + end +end +``` + +In the test, create the resource via the API: + +```ruby +let(:my_shirt) { create(:shirt, brand: 'AnotherBrand') } #<Resource::Shirt @brand="AnotherBrand" @name=nil> +let(:named_shirt) { create(:shirt, :with_name) } #<Resource::Shirt @brand="Brand Name" @name="Shirt Name"> +let(:invalid_shirt) { create(:shirt, read_only: true) } # NoMethodError + +it 'creates a shirt' do + expect(my_shirt.brand).to eq('AnotherBrand') + expect(named_shirt.name).to eq('Shirt Name') + expect(invalid_shirt).to raise_error(NoMethodError) # tries to call Resource::Shirt#read_only= +end +``` ### Resources cleanup diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 2d36377967e..cd5e32bc0ad 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -38,6 +38,18 @@ it's reset to a pristine test after each test. - [Example 4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103434#note_1172316521): A test for a database query passes in a fresh database, but in a CI/CD pipeline where the database is used to process previous test sequences, the test fails. This likely means that the query itself needs to be updated to work in a non-clean database. +- [Example 5](https://gitlab.com/gitlab-org/gitlab/-/issues/416663#note_1457867234): Unrelated database connections + in asynchronous requests checked back in, causing the tests to accidentally + use these unrelated database connections. The failure was resolved in this + [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125742). +- [Example 6](https://gitlab.com/gitlab-org/gitlab/-/issues/418757#note_1502138269): The maximum time to live + for a database connection causes these connections to be disconnected, which + in turn causes tests that rely on the transactions on these connections to + in turn causes tests that rely on the transactions on these connections to + fail. The issue was fixed in this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128567). +- [Example 7](https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues/3389#note_1534827164): + A TCP socket used in a test was not closed before the next test, which also used + the same port with another TCP socket. ### Dataset-specific @@ -172,10 +184,12 @@ it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' end ``` -This means it is skipped unless run with `--tag quarantine`: +This means it is skipped in CI. By default, the quarantined tests will run locally. + +We can skip them in local development as well by running with `--tag ~quarantine`: ```shell -bin/rspec --tag quarantine +bin/rspec --tag ~quarantine ``` After the long-term quarantining MR has reached production, you should revert the fast-quarantine MR you created earlier. diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 8da4350074d..3800e22b2f9 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -51,7 +51,7 @@ The default timeout for Jest is set in If your test exceeds that time, it fails. If you cannot improve the performance of the tests, you can increase the timeout -for the whole suite using [`jest.setTimeout`](https://jestjs.io/docs/28.x/jest-object#jestsettimeouttimeout) +for the whole suite using [`jest.setTimeout`](https://jestjs.io/docs/next/jest-object#jestsettimeouttimeout) ```javascript jest.setTimeout(500); @@ -63,7 +63,7 @@ describe('Component', () => { }); ``` -or for a specific test by providing a third argument to [`it`](https://jestjs.io/docs/28.x/api#testname-fn-timeout) +or for a specific test by providing a third argument to [`it`](https://jestjs.io/docs/next/api#testname-fn-timeout) ```javascript describe('Component', () => { diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index b4ae23336d5..ba13ca0c05a 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -6,21 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Using review apps in the development of GitLab -Review apps are deployed using the `start-review-app-pipeline` job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a review app. +Review apps are deployed using the `start-review-app-pipeline` manual job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a review app.  For any of the following scenarios, the `start-review-app-pipeline` job would be automatically started: -- for merge requests with CI configuration changes -- for merge requests with frontend changes -- for merge requests with changes to `{,ee/,jh/}{app/controllers}/**/*` -- for merge requests with changes to `{,ee/,jh/}{app/models}/**/*` -- for merge requests with changes to `{,ee/,jh/}lib/{,ee/,jh/}gitlab/**/*` -- for merge requests with QA changes - for scheduled pipelines - the MR has the `pipeline:run-review-app` label set +For all other scenarios, the `start-review-app-pipeline` job can be triggered manually. + ## E2E test runs on review apps On every pipeline in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-blocking` jobs are automatically started. @@ -278,7 +274,7 @@ find a way to limit it to only us.** - [Review apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) - [Stability issues](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/212) -### Helpful command line tools +### Helpful command-line tools - [K9s](https://github.com/derailed/k9s) - enables CLI dashboard across pods and enabling filtering by labels - [Stern](https://github.com/wercker/stern) - enables cross pod log tailing based on label/field selectors diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 955fc88c713..5aa6aecd9db 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -354,36 +354,4 @@ Analytics::CycleAnalytics::ReaggregationWorker.new.perform #### Value stream analytics -Seed issues and merge requests for value stream analytics: - - ```shell - // Seed 10 issues for the project specified by <project-id> - $ VSA_SEED_PROJECT_ID=<project-id> VSA_ISSUE_COUNT=10 SEED_VSA=true FILTER=cycle_analytics rake db:seed_fu - ``` - -#### DORA metrics - -Seed DORA daily metrics for value stream, insights and CI/CD analytics: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. On the project's homepage, in the upper-left corner, copy the **Project ID**. You need it in a later step. -1. [Create an environment for your selected project from the UI](../ci/environments/index.md#create-a-static-environment) named `production`. -1. Open the rails console: - - ```shell - rails c - ``` - -1. In the rails console, find the created environment by searching for the project ID: - - ```shell - e = Environment.find_by(project_id: <project-id>, name: "production") - ``` - -1. To seed data for the past 100 days for the environment, run the following command: - - ```shell - 100.times { |i| Dora::DailyMetrics.create(environment_id: e.id, date: (i + 1).days.ago, deployment_frequency: rand(50), incidents_count: rand(5), lead_time_for_changes_in_seconds: rand(50000), time_to_restore_service_in_seconds: rand(100000)) } - ``` - -DORA metric data should now be available for your selected project and any group or subgroup it belongs to. +For instructions on how to seed data for value stream analytics, see [development seed files](../development/development_seed_files.md). diff --git a/doc/development/work_items.md b/doc/development/work_items.md index 90e454bec85..2b28b2cd4f2 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -45,14 +45,15 @@ Here are some problems with current issues usage and why we are looking into wor ## Work item terminology -To avoid confusion and ensure communication is efficient, we will use the following terms exclusively when discussing work items. +To avoid confusion and ensure [communication is efficient](https://handbook.gitlab.com/handbook/communication/#mecefu-terms), we will use the following terms exclusively when discussing work items. This list is the [single source of truth (SSoT)](https://handbook.gitlab.com/handbook/values/#single-source-of-truth) for Work Item terminology. | Term | Description | Example of misuse | Should be | | --- | --- | --- | --- | | work item type | Classes of work item; for example: issue, requirement, test case, incident, or task | _Epics will eventually become issues_ | _Epics will eventually become a **work item type**_ | | work item | An instance of a work item type | | | -| work item view | The new frontend view that renders work items of any type | | | -| legacy issue view | The existing view used to render issues and incidents | | | +| work item view | The new frontend view that renders work items of any type | _This should be rendered in the new view_ | _This should be rendered in the work item view_ | +| legacy object | An object that has been or will be converted to a Work Item Type | _Epics will be migrated from a standalone/old/former object to a work item type_ | _Epics will be converted from a legacy object to a work item type_ | +| legacy issue view | The existing view used to render issues and incidents | _Issues continue to be rendered in the old view_ | _Issues continue to be rendered in the legacy issue view_ | | issue | The existing issue model | | | | issuable | Any model currently using the issuable module (issues, epics and MRs) | _Incidents are an **issuable**_ | _Incidents are a **work item type**_ | | widget | A UI element to present or allow interaction with specific work item data | | | @@ -84,9 +85,10 @@ end We already use the concept of WITs within `issues` table through `issue_type` column. There are `issue`, `incident`, and `test_case` issue types. To extend this -so that in future we can allow users to define custom WITs, we will move the -`issue_type` to a separate table: `work_item_types`. The migration process of `issue_type` -to `work_item_types` will involve creating the set of WITs for all root-level groups. +so that in future we can allow users to define custom WITs, we will +move the `issue_type` to a separate table: `work_item_types`. The migration process of `issue_type` +to `work_item_types` will involve creating the set of WITs for all root-level groups as described in +[this epic](https://gitlab.com/groups/gitlab-org/-/epics/6536). NOTE: At first, defining a WIT will only be possible at the root-level group, which would then be inherited by subgroups. @@ -99,7 +101,7 @@ assume the following base types: `issue: 0`, `incident: 1`, `test_case: 2`. The respective `work_item_types` records: -| `group_id` | `base_type` | `title` | +| `namespace_id` | `base_type` | `title` | | -------------- | ----------- | --------- | | 11 | 0 | Issue | | 11 | 1 | Incident | @@ -191,6 +193,164 @@ Until the architecture of WIT widgets is finalized, we are holding off on the cr types. If a new work item type is absolutely necessary, please reach out to a member of the [Project Management Engineering Team](https://gitlab.com/gitlab-org/gitlab/-/issues/370599). +### Creating a new work item type in the database + +We have completed the removal of the `issue_type` column from the issues table, in favor of using the new +`work_item_types` table as described in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6536)). + +After the introduction of the `work_item_types` table, we added more `work_item_types`, and we want to make it +easier for other teams to do so. To introduce a new `work_item_type`, you must: + +1. Write a database migration to create a new record in the `work_item_types` table. +1. Update `Gitlab::DatabaseImporters::WorkItems::BaseTypeImporter`. + +The following MRs demonstrate how to introduce new `work_item_types`: + +- [MR example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127482) +- [MR example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127917) + +#### Write a database migration + +First, write a database migration that creates the new record in the `work_item_types` table. + +Keep the following in mind when you write your migration: + +- **Important:** Exclude new type from existing APIs. + - We probably want to exclude newly created work items of this type from showing + up in existing features (like issue lists) until we fully release a feature. For this reason, + we have to add a new type to + [this exclude list](https://gitlab.com/gitlab-org/gitlab/-/blob/a0a52dd05b5d3c6ca820b672f9c0626840d2429b/app/models/work_items/type.rb#L84), + unless it is expected that users can create new issues and work items with the new type as soon as the migration + is executed. +- Use a regular migration, not a post-deploy. + - We believe it would be beneficial to use + [regular migrations](migration_style_guide.md#choose-an-appropriate-migration-type) + to add new work item types instead of a + [post deploy migration](database/post_deployment_migrations.md). + This way, follow-up MRs that depend on the type being created can assume it exists right away, + instead of having to wait for the next release. +- Migrations should avoid failures. + - We expect data related to `work_item_types` to be in a certain state when running the migration that will create a new + type. At the moment, we write migrations that check the data and don't fail in the event we find + it in an inconsistent state. There's a discussion about how much we can rely on the state of data based on seeds and + migrations in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423483). We can only + have a successful pipeline if we write the migration so it doesn't fail if data exists in an inconsistent + state. We probably need to update some of the database jobs in order to change this. +- Add widget definitions for the new type. + - The migration adds the new work item type as well as the widget definitions that are required for each work item. + The widgets you choose depend on the feature the new work item supports, but there are some that probably + all new work items need, like `Description`. +- Optional. Create hierarchy restrictions. + - In one of the example MRs we also insert records in the `work_item_hierarchy_restrictions` table. This is only + necessary if the new work item type is going to use the `Hierarchy` widget. In this table, you must add what + work item type can have children and of what type. Also, you should specify the hierarchy depth for work items of the same + type. + +##### Example of adding a ticket work item + +The `Ticket` work item type already exists in the database, but we'll use it as an example migration. +Note that for a new type you need to use a new name and ENUM value. + +```ruby +class AddTicketWorkItemType < Gitlab::Database::Migration[2.1] + disable_ddl_transaction! + restrict_gitlab_migration gitlab_schema: :gitlab_main + + ISSUE_ENUM_VALUE = 0 + # Enum value comes from the model where the enum is defined in + # https://gitlab.com/gitlab-org/gitlab/-/blob/1253f12abddb69cd1418c9e13e289d828b489f36/app/models/work_items/type.rb#L30. + # A new work item type should simply pick the next integer value. + TICKET_ENUM_VALUE = 8 + TICKET_NAME = 'Ticket' + # Widget definitions also have an enum defined in + # https://gitlab.com/gitlab-org/gitlab/-/blob/1253f12abddb69cd1418c9e13e289d828b489f36/app/models/work_items/widget_definition.rb#L17. + # We need to provide both the enum and name as we plan to support custom widget names in the future. + TICKET_WIDGETS = { + 'Assignees' => 0, + 'Description' => 1, + 'Hierarchy' => 2, + 'Labels' => 3, + 'Milestone' => 4, + 'Notes' => 5, + 'Start and due date' => 6, + 'Health status' => 7, + 'Weight' => 8, + 'Iteration' => 9, + 'Notifications' => 14, + 'Current user todos' => 15, + 'Award emoji' => 16 + }.freeze + + class MigrationWorkItemType < MigrationRecord + self.table_name = 'work_item_types' + end + + class MigrationWidgetDefinition < MigrationRecord + self.table_name = 'work_item_widget_definitions' + end + + class MigrationHierarchyRestriction < MigrationRecord + self.table_name = 'work_item_hierarchy_restrictions' + end + + def up + existing_ticket_work_item_type = MigrationWorkItemType.find_by(base_type: TICKET_ENUM_VALUE, namespace_id: nil) + + return say('Ticket work item type record exists, skipping creation') if existing_ticket_work_item_type + + new_ticket_work_item_type = MigrationWorkItemType.create( + name: TICKET_NAME, + namespace_id: nil, + base_type: TICKET_ENUM_VALUE, + icon_name: 'issue-type-issue' + ) + + return say('Ticket work item type create record failed, skipping creation') if new_ticket_work_item_type.new_record? + + widgets = TICKET_WIDGETS.map do |widget_name, widget_enum_value| + { + work_item_type_id: new_ticket_work_item_type.id, + name: widget_name, + widget_type: widget_enum_value + } + end + + MigrationWidgetDefinition.upsert_all( + widgets, + unique_by: :index_work_item_widget_definitions_on_default_witype_and_name + ) + + issue_type = MigrationWorkItemType.find_by(base_type: ISSUE_ENUM_VALUE, namespace_id: nil) + return say('Issue work item type not found, skipping hierarchy restrictions creation') unless issue_type + + # This part of the migration is only necessary if the new type uses the `Hierarchy` widget. + restrictions = [ + { parent_type_id: new_ticket_work_item_type.id, child_type_id: new_ticket_work_item_type.id, maximum_depth: 1 }, + { parent_type_id: new_ticket_work_item_type.id, child_type_id: issue_type.id, maximum_depth: 1 } + ] + + MigrationHierarchyRestriction.upsert_all( + restrictions, + unique_by: :index_work_item_hierarchy_restrictions_on_parent_and_child + ) + end + + def down + # There's the remote possibility that issues could already be + # using this issue type, with a tight foreign constraint. + # Therefore we will not attempt to remove any data. + end +end +``` + +<!-- markdownlint-disable-next-line MD044 --> +#### Update Gitlab::DatabaseImporters::WorkItems::BaseTypeImporter + +The [BaseTypeImporter](https://gitlab.com/gitlab-org/gitlab/-/blob/f816a369d7d6bbd1d8d53d6c0bca4ca3389fdba7/lib/gitlab/database_importers/work_items/base_type_importer.rb) +is where we can clearly visualize the structure of the types we have and what widgets are associated with each of them. +`BaseTypeImporter` is the single source of truth for fresh GitLab installs and also our test suite. This should always +reflect what we change with migrations. + ### Custom work item types With the WIT widget metadata and the workflow around mapping WIT to specific diff --git a/doc/editor_extensions/index.md b/doc/editor_extensions/index.md index 11220bc716c..f9742d30803 100644 --- a/doc/editor_extensions/index.md +++ b/doc/editor_extensions/index.md @@ -4,7 +4,7 @@ group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Editor and IDE Extensions +# Editor and IDE extensions GitLab has plugins and extensions to extend GitLab functionality to the following editors: @@ -13,7 +13,7 @@ GitLab has plugins and extensions to extend GitLab functionality to the followin - [Visual Studio](visual_studio/index.md) - [Neovim](neovim/index.md) -GitLab also supports developers in their command line interface with [`glab`](gitlab_cli/index.md) the GitLab CLI. +GitLab also supports developers in their command-line interface with [`glab`](gitlab_cli/index.md) the GitLab CLI. ## Features diff --git a/doc/editor_extensions/jetbrains_ide/index.md b/doc/editor_extensions/jetbrains_ide/index.md index dcf13570b11..7809ad69866 100644 --- a/doc/editor_extensions/jetbrains_ide/index.md +++ b/doc/editor_extensions/jetbrains_ide/index.md @@ -30,7 +30,7 @@ integrates GitLab with JetBrains IDEs. The following JetBrains IDEs are supporte ## Supported features -GitLab for JetBrains supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions.md). +GitLab for JetBrains supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions/index.md). ## Download the extension diff --git a/doc/editor_extensions/neovim/index.md b/doc/editor_extensions/neovim/index.md index e2e410ae82c..220cae8f334 100644 --- a/doc/editor_extensions/neovim/index.md +++ b/doc/editor_extensions/neovim/index.md @@ -13,7 +13,7 @@ integrates GitLab with Neovim. The following Neovim versions are supported: ## Supported features -GitLab for Neovim supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions.md). +GitLab for Neovim supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions/index.md). ## Install and configure the extension diff --git a/doc/editor_extensions/visual_studio/index.md b/doc/editor_extensions/visual_studio/index.md index 744f7759bf5..76a1abe058a 100644 --- a/doc/editor_extensions/visual_studio/index.md +++ b/doc/editor_extensions/visual_studio/index.md @@ -14,7 +14,7 @@ integrates GitLab with Visual Studio. The following Visual Studio versions are s ## Supported features -GitLab for Visual Studio supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions.md). +GitLab for Visual Studio supports [GitLab Duo Code Suggestions](../../user/project/repository/code_suggestions/index.md). ## Download the extension diff --git a/doc/editor_extensions/visual_studio_code/index.md b/doc/editor_extensions/visual_studio_code/index.md index 7c49879be13..2aa745c17a8 100644 --- a/doc/editor_extensions/visual_studio_code/index.md +++ b/doc/editor_extensions/visual_studio_code/index.md @@ -22,7 +22,8 @@ do more day-to-day tasks in Visual Studio Code, such as: and paste snippets to, and from, your editor. - [Browse repositories](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#browse-a-repository-without-cloning) without cloning them. -- [Receive Code Suggestions](../../user/project/repository/code_suggestions.md). +- [Receive Code Suggestions](../../user/project/repository/code_suggestions/index.md). +- [View Security findings](https://marketplace.visualstudio.com/items?itemName=gitlab.gitlab-workflow#security-findings). ## Download the extension @@ -35,7 +36,7 @@ you can [configure](https://marketplace.visualstudio.com/items?itemName=GitLab.g - [Features to display or hide](https://gitlab.com/gitlab-org/gitlab-vscode-extension#extension-settings). - [Self-signed certificate](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#self-signed-certificates) information. -- [Code Suggestions](../../user/project/repository/code_suggestions.md). +- [Code Suggestions](../../user/project/repository/code_suggestions/index.md). ## Report issues with the extension diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index 255d78ab915..55300d52b71 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -25,7 +25,7 @@ If you are unfamiliar with the command line, use the <!-- Original source for this list: doc/user/project/repository/web_editor.md#upload-a-file --> <!-- For why we duplicated the info, see https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111072#note_1267429478 --> -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **Upload file**. 1. Complete the fields. To create a merge request with the uploaded file, ensure the **Start a new merge request with these changes** toggle is turned on. diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index 42488becbd6..b39f39f293e 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -32,38 +32,12 @@ Amazon provides a managed Kubernetes service offering known as [Amazon Elastic K ## Available Infrastructure as Code for GitLab Cloud Native Hybrid -The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Linux package installation + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. GET is where GitLab is investing its resources as the primary option for Infrastructure as Code, and is being actively used in production as a part of [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). +The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) is a set of opinionated Terraform +and Ansible scripts. These scripts help with the deployment of Linux package or Cloud Native Hybrid environments on selected cloud providers and are used +by GitLab developers for [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md) (for example). -For more information about the project, see [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md). - -The [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) is developed by AWS, GitLab, and the community that contributes to AWS Quick Starts, whether directly to the GitLab Quick Start or to the underlying Quick Start dependencies GitLab inherits (for example, EKS Quick Start). - -GET is recommended for most deployments. The AWS Quick Start can be used if the IaC language of choice is CloudFormation, integration with AWS services like Control Tower is desired, or preference for a UI-driven configuration experience or when any aspect in the below table is an overriding concern. - -NOTE: -This automation is in **[Open Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#open-beta)**. GitLab is working with AWS on resolving [the outstanding issues](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues?q=is%3Aissue+is%3Aopen+%5BHL%5D) before it is fully released. You can subscribe to this issue to be notified of progress and release announcements: [AWS Quick Start for GitLab Cloud Native Hybrid on EKS Status: Beta](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues/11).<br><br> -The Beta version deploys Aurora PostgreSQL, but the release version will deploy Amazon RDS PostgreSQL due to [known issues](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) with Aurora. All performance testing results will also be redone after this change has been made. - -| | [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) | [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| Overview and Vision | [AWS Quick Start](https://aws.amazon.com/solutions/implementations/amazon-eks/) | [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md) | -| Licensing | [Open Source (Apache 2.0)](https://github.com/aws-quickstart/quickstart-eks-gitlab/blob/main/LICENSE.txt) | [GitLab Enterprise Edition license](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/LICENSE) ([GitLab Premium tier](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/README.md)) | -| GitLab Support | [GitLab Beta Support](../../policy/experiment-beta-support.md#beta) | [GitLab GA Support](../../policy/experiment-beta-support.md#generally-available-ga) | -| GitLab Reference Architecture Compliant | Yes | Yes | -| GitLab Performance Tool (GPT) Tested | Yes | Yes | -| Amazon Well Architected Compliant | Yes<br />(via Quick Start program) | Critical portions <br />reviewed by AWS | -| Target Cloud Platforms | AWS | AWS, Google, Azure | -| IaC Languages | CloudFormation (Quick Starts) | Terraform, Ansible | -| Community Contributions and Participation (EcoSystem) | <u>GitLab QSG</u>: Getting Started<br /><u>For QSG Dependencies (for example, EKS)</u>: Substantial | Getting Started | -| Compatible with AWS Meta-Automation Services (via CloudFormation) | - [AWS Service Catalog](https://aws.amazon.com/servicecatalog/) (Direct Import)<br>- [ServiceNow via an AWS Service Catalog Connector](https://docs.aws.amazon.com/servicecatalog/latest/adminguide/integrations-servicenow.html#integrations-servicenow)<br>- [Jira Service Manager via an AWS Service Catalog Connector](https://docs.aws.amazon.com/servicecatalog/latest/adminguide/integrations-jiraservicedesk.html#integrations-jiraservicedesk)<br>- [AWS Control Tower](https://docs.aws.amazon.com/controltower/) ([Integration](https://aws.amazon.com/blogs/infrastructure-and-automation/deploy-aws-quick-start-to-multiple-accounts-using-aws-control-tower/))<br>- Quick Starts<br>- [AWS SaaS Factory](https://aws.amazon.com/partners/programs/saas-factory/) | No | -| Results in a Ready-to-Use instance | Yes | Manual Actions or <br />Supplemental IaC Required | -| **<u>Configuration Features</u>** | | | -| Can deploy Linux package (non-Kubernetes) | No | Yes | -| Can deploy a single instance by using the Linux package (non-Kubernetes) | No | Yes | -| Complete Internal Encryption | 85%, Targeting 100% | Manual | -| AWS GovCloud Support | Yes | TBD | -| No Code Form-Based Deployment User Experience Available | Yes | No | -| Full IaC User Experience Available | Yes | Yes | +You can use the GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment on AWS. However, it's not required and may not support every valid +permutation. That said, the scripts are presented as-is and you can adapt them accordingly. ### Two and Three Zone High Availability diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 8a3cd720079..60591991ea7 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -248,7 +248,7 @@ in this section whenever you need to update GitLab. To determine the version of GitLab you're currently running: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Dashboard**. 1. Find the version under the **Components** table. diff --git a/doc/install/cloud_native/index.md b/doc/install/cloud_native/index.md deleted file mode 100644 index d971cd419e9..00000000000 --- a/doc/install/cloud_native/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'https://docs.gitlab.com/charts/' -remove_date: '2023-09-09' ---- - -This document was moved to [another location](https://docs.gitlab.com/charts/). - -<!-- This redirect file can be deleted after <2023-09-09>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/install/install_methods.md b/doc/install/install_methods.md index fb48107e81b..5945e418274 100644 --- a/doc/install/install_methods.md +++ b/doc/install/install_methods.md @@ -17,7 +17,7 @@ or use one of the following methods. | [Helm chart](https://docs.gitlab.com/charts/) | A chart for installing a cloud-native version of GitLab and its components on Kubernetes. | Use if your infrastructure is on Kubernetes and you're familiar with how it works. Management, observability, and some concepts are different than traditional deployments.<br/>- Administration and troubleshooting requires Kubernetes knowledge.<br/>- It can be more expensive for smaller installations. The default installation requires more resources than a single node Linux package deployment, because most services are deployed in a redundant fashion.<br/><br/> | | [Docker](docker.md) | The GitLab packages in a Docker container. | Use if you're familiar with Docker. | | [Source](installation.md) | GitLab and its components from scratch. | Use if none of the previous methods are available for your platform. Can use for unsupported systems like \*BSD.| -| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#documentation) | A set of automation tools. | Use to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. Has some [limitations](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#missing-features-to-be-aware-of) and manual setup for production environments. | +| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#documentation) | A set of opinionated Terraform and Ansible scripts. | Use to deploy a [reference architecture](../administration/reference_architectures/index.md) on selected major cloud providers. Has some [limitations](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#missing-features-to-be-aware-of) and manual setup for production environments. | | [GitLab Operator](https://docs.gitlab.com/operator/) | An installation and management method that follows the [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). | Use to run GitLab in an [OpenShift](openshift_and_gitlab/index.md) environment. | ## Unsupported Linux distributions and Unix-like operating systems diff --git a/doc/install/next_steps.md b/doc/install/next_steps.md index ecc456cd3ec..4ad6a011cd6 100644 --- a/doc/install/next_steps.md +++ b/doc/install/next_steps.md @@ -42,7 +42,7 @@ installation. - [Back up and restore GitLab](../administration/backup_restore/index.md): Learn the different ways you can back up or restore GitLab. -- [Upgrade GitLab](../update/index.md): Every 22nd of the month, a new feature-rich GitLab version +- [Upgrade GitLab](../update/index.md): Every month, a new feature-rich GitLab version is released. Learn how to upgrade to it, or to an interim release that contains a security fix. - [Release and maintenance policy](../policy/maintenance.md): Learn about GitLab policies governing version naming, as well as release pace for major, minor, patch, diff --git a/doc/install/requirements.md b/doc/install/requirements.md index b095f265808..81244594a59 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -50,11 +50,13 @@ The following is the recommended minimum Memory hardware guidance for a handful - 8 GB RAM supports up to 1000 users - More users? Consult the [reference architectures page](../administration/reference_architectures/index.md) -In addition to the above, we generally recommend having at least 2 GB of swap on your server, -even if you currently have enough available RAM. Having swap helps to reduce the chance of errors occurring -if your available memory changes. We also recommend configuring the kernel's swappiness setting -to a low value like `10` to make the most of your RAM while still having the swap -available when needed. +For smaller installations, you should: + +- Have at least 2 GB of swap on your server, even if you have enough available RAM. Having swap helps to reduce the chance of + errors occurring if your available memory changes. +- Configure the kernel's swappiness setting to a low value like `10` to make the most of your RAM while still having the swap available when needed. + +For larger installations that follow our reference architectures, you [shouldn't configure swap](../administration/reference_architectures/index.md#no-swap). NOTE: Although excessive swapping is undesired and degrades performance, it is an @@ -98,7 +100,7 @@ The following managed PostgreSQL services are known to be incompatible and shoul | GitLab version | Managed service | |----------------|-------------------------------------------------------| -| 14.4+ | Amazon Aurora (see [14.4.0](../update/index.md#1440)) | +| 14.4+ | Amazon Aurora (see [14.4.0](../update/versions/gitlab_14_changes.md#1440)) | NOTE: Support for [PostgreSQL 9.6 and 10 was removed in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#postgresql-11-is-now-the-minimum-required-version-to-install-gitlab) so that GitLab can benefit from PostgreSQL 11 improvements, such as partitioning. diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md index f23bfa47eba..066c04081a5 100644 --- a/doc/integration/advanced_search/elasticsearch.md +++ b/doc/integration/advanced_search/elasticsearch.md @@ -5,7 +5,7 @@ group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Elasticsearch **(PREMIUM SELF)** +# Elasticsearch **(PREMIUM ALL)** This page describes how to enable advanced search. When enabled, advanced search provides faster search response times and [improved search features](../../user/search/advanced_search.md). @@ -14,16 +14,14 @@ advanced search provides faster search response times and [improved search featu ### Elasticsearch version requirements -> Support for Elasticsearch 6.8 was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350275) in GitLab 15.0. +> Support for Elasticsearch 6.8 [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350275) in GitLab 15.0. Advanced search works with the following versions of Elasticsearch. -| GitLab version | Elasticsearch version | -|-----------------------|--------------------------| -| GitLab 15.0 or later | Elasticsearch 7.x - 8.x | -| GitLab 13.9 - 14.10 | Elasticsearch 6.8 - 7.x | -| GitLab 13.3 - 13.8 | Elasticsearch 6.4 - 7.x | -| GitLab 12.7 - 13.2 | Elasticsearch 6.x - 7.x | +| GitLab version | Elasticsearch version | +|-----------------------|-----------------------------| +| GitLab 15.0 and later | Elasticsearch 7.x and later | +| GitLab 14.0 to 14.10 | Elasticsearch 6.8 to 7.x | Advanced search follows the [Elasticsearch end-of-life policy](https://www.elastic.co/support/eol). When we change Elasticsearch supported versions in GitLab, we announce them in [deprecation notes](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations) in monthly release posts @@ -31,10 +29,10 @@ before we remove them. ### OpenSearch version requirements -| GitLab version | OpenSearch version | -|-------------------------|---------------------------| -| GitLab 15.0 to 15.5.2 | OpenSearch 1.x | -| GitLab 15.5.3 and later | OpenSearch 1.x and later | +| GitLab version | OpenSearch version | +|-------------------------|--------------------------| +| GitLab 15.5.3 and later | OpenSearch 1.x and later | +| GitLab 15.0 to 15.5.2 | OpenSearch 1.x | If your version of Elasticsearch or OpenSearch is incompatible, to prevent data loss, indexing pauses and a message is logged in the @@ -47,7 +45,7 @@ If you are using a compatible version and after connecting to OpenSearch, you ge Elasticsearch requires additional resources to those documented in the [GitLab system requirements](../../install/requirements.md). -Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221177) in GitLab 13.10) uses the total repository size to estimate the advanced search storage requirements. +Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task uses the total repository size to estimate the advanced search storage requirements. ## Install Elasticsearch @@ -68,10 +66,14 @@ The search index updates after you: ## Upgrade to a new Elasticsearch major version -> - Elasticsearch 6.8 support is removed with GitLab 15.0. -> - Upgrading from GitLab 14.10 to 15.0 requires that you are using any version of Elasticsearch 7.x. +> Support for Elasticsearch 6.8 [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350275) in GitLab 15.0. -You are not required to change the GitLab configuration when you upgrade Elasticsearch. +You don't have to change the GitLab configuration when you upgrade Elasticsearch. + +You should pause indexing during an Elasticsearch upgrade so changes can still be tracked. +When the Elasticsearch cluster is fully upgraded and active, [resume indexing](#unpause-indexing). + +When you upgrade to GitLab 15.0 and later, you must use Elasticsearch 7.x and later. ## Elasticsearch repository indexer @@ -82,7 +84,7 @@ Depending on your GitLab version, there are different installation procedures fo - For Linux package installations, the Go indexer is included. - For self-compiled installations, see [Install the indexer from source](#install-the-indexer-from-source). - If you're using the GitLab Development Kit, see [Elasticsearch in the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/elasticsearch.md). -- If you're running a Helm deployment of GitLab 11.10 and later, [the indexer is already included](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/213). +- If you're using the GitLab Helm chart, [the indexer is already included](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/213). ### Install the indexer from source @@ -166,7 +168,7 @@ Prerequisite: To enable advanced search: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. @@ -210,7 +212,7 @@ You can only use the **Index all projects** setting to perform initial indexing, not to re-create an index from scratch. To enable advanced search with **Index all projects**: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Select the **Elasticsearch indexing** checkbox, then select **Save changes**. @@ -402,7 +404,7 @@ You can improve the language support for Chinese and Japanese languages by utili To enable languages support: 1. Install the desired plugins, refer to [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/plugins/7.9/installation.html) for plugins installation instructions. The plugins must be installed on every node in the cluster, and each node must be restarted after installation. For a list of plugins, see the table later in this section. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Locate **Custom analyzers: language support**. @@ -424,7 +426,7 @@ For guidance on what to install, see the following Elasticsearch language plugin To disable the Elasticsearch integration: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Clear the **Elasticsearch indexing** and **Search with Elasticsearch enabled** checkboxes. @@ -441,7 +443,7 @@ To disable the Elasticsearch integration: ## Unpause Indexing -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Expand **Advanced Search**. @@ -462,14 +464,10 @@ You can use zero-downtime reindexing to configure index settings or mappings tha ### Trigger the reindex via the advanced search administration -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34069) in GitLab 13.2. -> - A scheduled index deletion and the ability to cancel it was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38914) in GitLab 13.3. -> - Support for retries during reindexing was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55681) in GitLab 13.12. - To trigger the reindexing process: 1. Sign in to your GitLab instance as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Expand **Elasticsearch zero-downtime reindexing**. @@ -485,9 +483,7 @@ While the reindexing is running, you can follow its progress under that same sec #### Elasticsearch zero-downtime reindexing -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55681) in GitLab 13.12. - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Expand **Elasticsearch zero-downtime reindexing**, and you'll @@ -536,7 +532,7 @@ Sometimes, you might want to abandon the unfinished reindex job and resume the i bundle exec rake gitlab:elastic:mark_reindex_failed RAILS_ENV=production ``` -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Advanced Search**. 1. Expand **Advanced Search**. @@ -545,12 +541,7 @@ Sometimes, you might want to abandon the unfinished reindex job and resume the i ## Index integrity > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112369) in GitLab 15.10 [with a flag](../../administration/feature_flags.md) named `search_index_integrity`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/392981) in GitLab 16.0. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/392981) in GitLab 16.3. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `search_index_integrity`. -On GitLab.com, this feature is available. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/392981) in GitLab 16.4. Feature flag `search_index_integrity` removed. Index integrity detects and fixes missing repository data. This feature is automatically used when code searches @@ -558,8 +549,6 @@ scoped to a group or project return no results. ## Advanced search migrations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6. - With reindex migrations running in the background, there's no need for a manual intervention. This usually happens in situations where new features are added to advanced search, which means adding or changing the way content is indexed. @@ -685,7 +674,7 @@ The following are some available Rake tasks: | [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Performs an Elasticsearch import that indexes the snippets data. | | [`sudo gitlab-rake gitlab:elastic:index_users`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Imports all users into Elasticsearch. | | [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Displays which projects are not indexed. | -| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Schedules a zero-downtime cluster reindexing task. This feature should be used with an index that was created after GitLab 13.0. | +| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Schedules a zero-downtime cluster reindexing task. | | [`sudo gitlab-rake gitlab:elastic:mark_reindex_failed`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Mark the most recent re-index job as failed. | | [`sudo gitlab-rake gitlab:elastic:list_pending_migrations`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | List pending migrations. Pending migrations include those that have not yet started, have started but not finished, and those that are halted. | | [`sudo gitlab-rake gitlab:elastic:estimate_cluster_size`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Get an estimate of cluster size based on the total repository size. | @@ -751,7 +740,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic - A good guideline is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30 GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This generally helps the cluster stay in good health. - Number of Elasticsearch shards: - Small shards result in small segments, which increases overhead. Aim to keep the average shard size between at least a few GB and a few tens of GB. - - Another consideration is the number of documents. To determine the number of shards to use, sum the numbers in the **Main menu > Admin > Dashboard > Statistics** pane (the number of documents to be indexed), divide by 5 million, and add 5. For example: + - Another consideration is the number of documents. To determine the number of shards to use, sum the numbers in the Admin Area under **Dashboard > Statistics** (the number of documents to be indexed), divide by 5 million, and add 5. For example: - If you have fewer than about 2,000,000 documents, use the default of 5 shards - 10,000,000 documents: `10000000/5000000 + 5` = 7 shards - 100,000,000 documents: `100000000/5000000 + 5` = 25 shards @@ -828,7 +817,7 @@ Make sure to prepare for this task by having a ``` This enqueues a Sidekiq job for each project that needs to be indexed. - You can view the jobs in **Main menu > Admin > Monitoring > Background Jobs > Queues Tab** + You can view the jobs in the Admin Area under **Monitoring > Background Jobs > Queues Tab** and select `elastic_commit_indexer`, or you can query indexing status using a Rake task: ```shell @@ -893,7 +882,7 @@ Make sure to prepare for this task by having a A force merge should be called after enabling the refreshing above. - For Elasticsearch 6.x, the index should be in read-only mode before proceeding with the force merge: + For Elasticsearch 6.x and later, ensure the index is in read-only mode before proceeding with the force merge: ```shell curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ @@ -909,7 +898,7 @@ Make sure to prepare for this task by having a curl --request POST 'localhost:9200/gitlab-production/_forcemerge?max_num_segments=5' ``` - After this, if your index is in read-only mode, switch back to read-write: + Then, change the index back to read-write mode: ```shell curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ diff --git a/doc/integration/advanced_search/elasticsearch_troubleshooting.md b/doc/integration/advanced_search/elasticsearch_troubleshooting.md index d13d47a1633..df1e1f49083 100644 --- a/doc/integration/advanced_search/elasticsearch_troubleshooting.md +++ b/doc/integration/advanced_search/elasticsearch_troubleshooting.md @@ -513,3 +513,9 @@ $ jq '.class' sidekiq/current | sort | uniq -c | sort -nr In this case, `free -m` on the overloaded GitLab node would also show unexpectedly high `buff/cache` usage. + +## `Couldn't load task status` error when reindexing + +When you reindex, you might get a `Couldn't load task status` error. A `sliceId must be greater than 0 but was [-1]` error might also appear on the Elasticsearch host. As a workaround, consider [reindexing from scratch](../../integration/advanced_search/elasticsearch_troubleshooting.md#last-resort-to-recreate-an-index) or upgrading to GitLab 16.3. + +For more information, see [issue 422938](https://gitlab.com/gitlab-org/gitlab/-/issues/422938). diff --git a/doc/integration/akismet.md b/doc/integration/akismet.md index 19c86743eba..94c25a0547b 100644 --- a/doc/integration/akismet.md +++ b/doc/integration/akismet.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -30,7 +30,7 @@ To use Akismet: 1. Sign in or create a new account. 1. Select **Show** to reveal the API key, and copy the API key's value. 1. Sign in to GitLab as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Reporting**. 1. Expand **Spam and Anti-bot Protection**. diff --git a/doc/integration/alicloud.md b/doc/integration/alicloud.md index 9054341960f..3984deb303a 100644 --- a/doc/integration/alicloud.md +++ b/doc/integration/alicloud.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/arkose.md b/doc/integration/arkose.md index cd0b80e5a66..575cf8c4271 100644 --- a/doc/integration/arkose.md +++ b/doc/integration/arkose.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/auth0.md b/doc/integration/auth0.md index e6a6853cdef..a6577a3eb89 100644 --- a/doc/integration/auth0.md +++ b/doc/integration/auth0.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/azure.md b/doc/integration/azure.md index 08ca19a950b..71f6c2cf3a0 100644 --- a/doc/integration/azure.md +++ b/doc/integration/azure.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md index 7fad67116f3..0e880c79b7a 100644 --- a/doc/integration/bitbucket.md +++ b/doc/integration/bitbucket.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/cas.md b/doc/integration/cas.md deleted file mode 100644 index d2a29161a53..00000000000 --- a/doc/integration/cas.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -stage: Manage -group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-15' -redirect_to: '../administration/auth/index.md' ---- - -# CAS OmniAuth provider (removed) **(FREE SELF)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/369127) -in GitLab 15.3 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/369128) -in 16.0. diff --git a/doc/integration/datadog.md b/doc/integration/datadog.md index 5e89cfbe6a0..4d26235e65f 100644 --- a/doc/integration/datadog.md +++ b/doc/integration/datadog.md @@ -27,7 +27,7 @@ project, group, or instance level: 1. *For project-level or group-level integrations:* In GitLab, go to your project or group. 1. *For instance-level integrations:* 1. Sign in to GitLab as a user with administrator access. - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Integrations**. 1. Scroll to **Add an integration**, and select **Datadog**. diff --git a/doc/integration/ding_talk.md b/doc/integration/ding_talk.md index 005bf0a3244..91727ba5398 100644 --- a/doc/integration/ding_talk.md +++ b/doc/integration/ding_talk.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 2d0f0892f30..a4c7f75ea19 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -25,7 +25,7 @@ References are displayed as issue links. To disable the GitLab issue tracker for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Under **Issues**, turn off the toggle. diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index 0ff640306c3..299b1e53ff2 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/github.md b/doc/integration/github.md index df9a0db961b..0de7e463944 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 92709d974aa..8162d011395 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index 9b431846f2b..288c2615d66 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -45,7 +45,7 @@ With the Gitpod integration enabled for your GitLab instance, to enable it for y For self-managed GitLab instances, a GitLab administrator must: 1. Enable the Gitpod integration in GitLab: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Gitpod** configuration section. @@ -61,7 +61,7 @@ GitLab users can then [enable the Gitpod integration for themselves](#enable-git You can launch Gitpod directly from GitLab in one of these ways: - **From a project repository:** - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. In the upper right, select **Edit > Gitpod**. - **From a merge request:** diff --git a/doc/integration/google.md b/doc/integration/google.md index 5bba50c940a..5ef47c79e2d 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 736f5bc080c..4f76301adeb 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -8,33 +8,34 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/246756) to GitLab Free in 13.7. -You can trigger a build in Jenkins when you push code to your repository or -create a merge request in GitLab. The Jenkins pipeline status displays on merge -requests widgets and on the GitLab project's home page. +[Jenkins](https://www.jenkins.io/) is an open source automation server that supports +building, deploying and automating projects. -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of the Jenkins integration for GitLab, see -[GitLab workflow with Jira issues and Jenkins pipelines](https://youtu.be/Jn-_fyra7xQ). - -Use the Jenkins integration when: +You should use a Jenkins integration with GitLab when: - You plan to migrate your CI from Jenkins to [GitLab CI/CD](../ci/index.md) in the future, but need an interim solution. - You're invested in [Jenkins plugins](https://plugins.jenkins.io/) and choose to keep using Jenkins to build your apps. -NOTE: -This documentation focuses only on how to configure a Jenkins *integration* with -GitLab. Learn how to set up Jenkins [on your local machine](../development/integrations/jenkins.md) -in the developer documentation, and how to migrate from Jenkins to GitLab CI/CD in the -[Migrating from Jenkins](../ci/migration/jenkins.md) documentation. +After you have configured a Jenkins integration, you trigger a build in Jenkins +when you push code to your repository or create a merge request in GitLab. The +Jenkins pipeline status displays on merge request widgets and the GitLab +project's home page. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview of the Jenkins integration for GitLab, see +[GitLab workflow with Jira issues and Jenkins pipelines](https://youtu.be/Jn-_fyra7xQ). + +To configure a Jenkins integration with GitLab: -The Jenkins integration requires configuration in both GitLab and Jenkins. +- Grant Jenkins access to the GitLab project. +- Configure the Jenkins server. +- Configure the Jenkins project. +- Configure the GitLab project. ## Grant Jenkins access to the GitLab project -To grant Jenkins access to the GitLab project: - 1. Create a personal, project, or group access token. - [Create a personal access token](../user/profile/personal_access_tokens.md#create-a-personal-access-token) @@ -46,20 +47,22 @@ To grant Jenkins access to the GitLab project: to use the token for all Jenkins integrations in all projects of that group. 1. Set the access token scope to **API**. -1. Copy the access token value to [configure the Jenkins server](#configure-the-jenkins-server). +1. Copy the access token value to configure the Jenkins server. ## Configure the Jenkins server -Install and configure the Jenkins plugin. The plugin must be installed and configured to -authorize the connection to GitLab. +Install and configure the Jenkins plugin to authorize the connection to GitLab. 1. On the Jenkins server, select **Manage Jenkins > Manage Plugins**. -1. Install the [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +1. Select the **Available** tab. Search for `gitlab-plugin` and select it to install. + See the [Jenkins GitLab documentation](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) + for other ways to install the plugin. 1. Select **Manage Jenkins > Configure System**. 1. In the **GitLab** section, select **Enable authentication for '/project' end-point**. 1. Select **Add**, then choose **Jenkins Credential Provider**. 1. Select **GitLab API token** as the token type. -1. In **API Token**, [paste the value you copied from GitLab](#grant-jenkins-access-to-the-gitlab-project) and select **Add**. +1. In **API Token**, [paste the access token value you copied from GitLab](#grant-jenkins-access-to-the-gitlab-project) + and select **Add**. 1. Enter the GitLab server's URL in **GitLab host URL**. 1. To test the connection, select **Test Connection**. @@ -72,21 +75,21 @@ For more information, see Set up the Jenkins project you intend to run your build on. -1. On your Jenkins instance, go to **New Item**. +1. On your Jenkins instance, select **New Item**. 1. Enter the project's name. 1. Select **Freestyle** or **Pipeline** and select **OK**. - We recommend a Freestyle project, because the Jenkins plugin updates the build status on - GitLab. In a Pipeline project, you must configure a script to update the status on GitLab. + You should select a freestyle project, because the Jenkins plugin updates the build status on + GitLab. In a pipeline project, you must configure a script to update the status on GitLab. 1. Choose your GitLab connection from the dropdown list. 1. Select **Build when a change is pushed to GitLab**. 1. Select the following checkboxes: - **Accepted Merge Request Events** - **Closed Merge Request Events** 1. Specify how the build status is reported to GitLab: - - If you created a **Freestyle** project, in the **Post-build Actions** section, choose - **Publish build status to GitLab**. - - If you created a **Pipeline** project, you must use a Jenkins Pipeline script to update the status on - GitLab. + - If you created a freestyle project, in the **Post-build Actions** section, + choose **Publish build status to GitLab**. + - If you created a pipeline project, you must use a Jenkins Pipeline script to + update the status on GitLab. Example Jenkins Pipeline script: @@ -106,18 +109,19 @@ Set up the Jenkins project you intend to run your build on. } ``` - For more Jenkins Pipeline script examples, go to the [Jenkins GitLab plugin repository on GitHub](https://github.com/jenkinsci/gitlab-plugin#scripted-pipeline-jobs). + For more Jenkins Pipeline script examples, see the + [Jenkins GitLab plugin repository on GitHub](https://github.com/jenkinsci/gitlab-plugin#scripted-pipeline-jobs). ## Configure the GitLab project Configure the GitLab integration with Jenkins in one of the following ways. -### Configure a Jenkins integration (recommended) +### With a Jenkins server URL -GitLab recommends this approach for Jenkins integrations because it is easier to configure -than the [webhook integration](#configure-a-webhook). +You should use this approach for Jenkins integrations if you can provide GitLab +with your Jenkins server URL and authentication information. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Jenkins**. 1. Select the **Active** checkbox. @@ -128,61 +132,62 @@ than the [webhook integration](#configure-a-webhook). 1. Enter the **Jenkins server URL**. 1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](../user/project/integrations/index.md#manage-ssl-verification). 1. Enter the **Project name**. - The project name should be URL-friendly, where spaces are replaced with underscores. To ensure the project name is valid, copy it from your browser's address bar while viewing the Jenkins project. -1. If your Jenkins server requires - authentication, enter the **Username** and **Password**. +1. If your Jenkins server requires authentication, enter the **Username** and **Password**. 1. Optional. Select **Test settings**. 1. Select **Save changes**. -### Configure a webhook +### With a webhook -If you are unable to provide GitLab with your Jenkins server login, you can use this option -to integrate GitLab and Jenkins. +If you cannot [provide GitLab with your Jenkins server URL and authentication information](#with-a-jenkins-server-url), you can configure a webhook to integrate GitLab and Jenkins. 1. In the configuration of your Jenkins job, in the GitLab configuration section, select **Advanced**. 1. Under **Secret Token**, select **Generate**. 1. Copy the token, and save the job configuration. -1. In GitLab, create a webhook for your project, enter the trigger URL - (such as `https://JENKINS_URL/project/YOUR_JOB`) and paste the token in **Secret Token**. +1. In GitLab: + - [Create a webhook for your project](../user/project/integrations/webhooks.md#configure-a-webhook-in-gitlab). + - Enter the trigger URL (such as `https://JENKINS_URL/project/YOUR_JOB`). + - Paste the token in **Secret Token**. 1. To test the webhook, select **Test**. ## Related topics -- For a real use case, read the blog post - [Continuous integration: From Jenkins to GitLab using Docker](https://about.gitlab.com/blog/2017/07/27/docker-my-precious/). -- See the ['GitLab vs. Jenkins' comparison page](https://about.gitlab.com/devops-tools/jenkins-vs-gitlab/) - for information on how moving to a single application for the entire software development - lifecycle can decrease hours spent on maintaining toolchains by 10% or more. +- [GitLab Jenkins Integration](https://about.gitlab.com/solutions/jenkins/) +- [How to set up Jenkins on your local machine](../development/integrations/jenkins.md) +- [How to migrate from Jenkins to GitLab CI/CD](../ci/migration/jenkins.md) ## Troubleshooting ### Error during GitLab configuration - "Connection failed. Please check your settings" -If you get this error message while configuring GitLab, the following are possible causes: +While configuring GitLab, you might get an error that states "Connection failed. Please check your settings". + +This issue has multiple possible causes and solutions: -- GitLab is unable to reach your Jenkins instance at the address. If your GitLab instance is self-managed, try pinging the - Jenkins instance at the domain provided on the GitLab instance. -- The Jenkins instance is at a local address and is not included in the - [GitLab installation's allowlist](../security/webhooks.md#allow-outbound-requests-to-certain-ip-addresses-and-domains). -- The credentials for the Jenkins instance do not have sufficient access or are invalid. -- The **Enable authentication for `/project` end-point** checkbox is not selected in your [Jenkins plugin configuration](#configure-the-jenkins-server). +| Cause | Workaround | +|------------------------------------------------------------------|-------------| +| GitLab is unable to reach your Jenkins instance at the address. | If your GitLab instance is self-managed, ping the Jenkins instance at the domain provided on the GitLab instance. | +| The Jenkins instance is at a local address and is not included in the [GitLab installation's allowlist](../security/webhooks.md#allow-outbound-requests-to-certain-ip-addresses-and-domains).| Add the instance to the GitLab installation's allowlist. | +| The credentials for the Jenkins instance do not have sufficient access or are invalid.| Grant the credentials sufficient access or create valid credentials. | +|The **Enable authentication for `/project` end-point** checkbox is not selected in your [Jenkins plugin configuration](#configure-the-jenkins-server)| Select the checkbox. | ### Error in merge requests - "Could not connect to the CI server" -You might get the `Could not connect to the CI server` error if GitLab did not -receive a build status update from Jenkins via the [Commit Status API](../api/commits.md#commit-status). +You might get an error that states `Could not connect to the CI server` in a merge +request if GitLab did not receive a build status update from Jenkins through the +[Commit Status API](../api/commits.md#commit-status). -This issue occurs when Jenkins is not properly -configured or there is an error reporting the status via the API. +This issue occurs when Jenkins is not properly configured or there is an error +reporting the status through the API. -To fix this issue, ensure you: +To fix this issue: 1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access. -1. [Configure the Jenkins project](#configure-the-jenkins-project), including the - 'Publish build status to GitLab' post-build action. +1. [Configure the Jenkins project](#configure-the-jenkins-project), and make sure + that, if you create a freestyle project, you choose the "Publish build status to GitLab" + post-build action. ### Merge request event does not trigger a Jenkins pipeline diff --git a/doc/integration/jira/configure.md b/doc/integration/jira/configure.md index 8ef7b7bc442..dd43c6417e8 100644 --- a/doc/integration/jira/configure.md +++ b/doc/integration/jira/configure.md @@ -26,7 +26,7 @@ You can configure these settings at the [group level](../../administration/setti To configure your project settings in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Jira**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -89,7 +89,7 @@ To copy the API token, select **Copy**. To migrate from Jira Server to Jira Cloud in GitLab and maintain your Jira integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Jira**. 1. In **Web URL**, enter the new Jira site URL (for example, `https://myjirasite.atlassian.net`). diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 985f67fdf98..3fa2a58b787 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -100,7 +100,7 @@ You must enable OAuth authentication to: To create an OAuth application: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Applications**. 1. Select **New application**. @@ -139,10 +139,11 @@ To create branches from Jira Cloud, [install the app manually](#install-the-gitl - The instance must be on GitLab version 15.7 or later. - You must set up [OAuth authentication](#set-up-oauth-authentication-for-self-managed-instances). - Your network must allow inbound and outbound connections between GitLab and Jira. For self-managed instances that are behind a - firewall and cannot be directly accessed from the internet, you can: + firewall and cannot be directly accessed from the internet: - Open your firewall and only allow inbound traffic from [Atlassian IP addresses](https://support.atlassian.com/organization-administration/docs/ip-addresses-and-domains-for-atlassian-cloud-products/#Outgoing-Connections). - Set up an internet-facing reverse proxy in front of your self-managed instance. To secure this proxy further, only allow inbound traffic from [Atlassian IP addresses](https://support.atlassian.com/organization-administration/docs/ip-addresses-and-domains-for-atlassian-cloud-products/#Outgoing-Connections). + - Add [GitLab IP addresses](../../user/gitlab_com/index.md#ip-range) to the allowlist of your firewall. ### Set up your instance @@ -150,7 +151,7 @@ To create branches from Jira Cloud, [install the app manually](#install-the-gitl To set up your self-managed instance for the GitLab for Jira Cloud app in GitLab 15.7 and later: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Jira App**. @@ -245,7 +246,7 @@ You might want to use a proxy if you're managing multiple GitLab instances but o To configure your GitLab instance to serve as a proxy: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Jira App**. @@ -315,7 +316,7 @@ To resolve this issue, disable the **Jira Connect Proxy URL** setting. - In GitLab 15.8 and later: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **GitLab for Jira App**. diff --git a/doc/integration/jira/dvcs/troubleshooting.md b/doc/integration/jira/dvcs/troubleshooting.md index 801bdab4139..570237779b4 100644 --- a/doc/integration/jira/dvcs/troubleshooting.md +++ b/doc/integration/jira/dvcs/troubleshooting.md @@ -132,12 +132,12 @@ Failed to execute request [https://gitlab.com/api/v4/projects/:id/merge_requests {"message":"403 Forbidden"} ``` -If you find a `{"message":"403 Forbidden"}` error, it is possible that this specific project has some [GitLab features disabled](../../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). +If you find a `{"message":"403 Forbidden"}` error, it is possible that this specific project has some [GitLab features disabled](../../../user/project/settings/index.md#configure-project-features-and-permissions). In the example above, the merge requests feature is disabled. To resolve the issue, enable the relevant feature: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Use the toggles to enable the features as needed. @@ -146,7 +146,7 @@ To resolve the issue, enable the relevant feature: To find webhook logs in a DVCS-linked project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > Webhooks**. 1. Scroll down to **Project Hooks**. 1. Next to the log that points to your Jira instance, select **Edit**. diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index c7a60ec2ff8..a723ae2a249 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -58,7 +58,6 @@ The [Jira issue integration](configure.md) posts GitLab data in the form of comm The GitLab for Jira Cloud app and Jira DVCS connector share this data through the [Jira development panel](development_panel.md). This method provides more fine-grained access control because access can be restricted to certain user groups or roles. -## Third-party Jira integrations +## Related topics -Developers have built several third-party Jira integrations for GitLab that are -listed on the [Atlassian Marketplace](https://marketplace.atlassian.com/search?product=jira&query=gitlab). +- [Third-party Jira integrations](https://marketplace.atlassian.com/search?product=jira&query=gitlab) diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md index 06d64d19db5..ae4b726327c 100644 --- a/doc/integration/jira/issues.md +++ b/doc/integration/jira/issues.md @@ -52,7 +52,7 @@ You can [disable comments](#disable-comments-on-jira-issues) on issues. With this integration, you can prevent merge requests from being merged if they do not refer to a Jira issue. To enable this feature: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge checks** section, select **Require an associated issue from Jira**. 1. Select **Save**. @@ -76,7 +76,7 @@ When you don't configure custom rules, the [default behavior](https://gitlab.com To define a regex pattern for Jira issue keys: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Jira**. 1. Go to the **Jira issue matching** section. @@ -92,7 +92,7 @@ and you've set a `JIRA#` prefix, GitLab matches `JIRA#ALPHA-1` rather than `ALPH To define a prefix for Jira issue keys: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Jira**. 1. Go to the **Jira issue matching** section. @@ -136,7 +136,7 @@ provided your GitLab administrator [has configured the integration](configure.md To view Jira issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Plan > Jira issues**. The issues are sorted by **Created date** by default, with the most recently created issues listed at the top. diff --git a/doc/integration/jira/troubleshooting.md b/doc/integration/jira/troubleshooting.md index 70c6a306299..c2b79f116eb 100644 --- a/doc/integration/jira/troubleshooting.md +++ b/doc/integration/jira/troubleshooting.md @@ -55,7 +55,7 @@ To change all Jira projects to use instance-level integration settings: ```ruby jira_integration_instance_id = Integrations::Jira.find_by(instance: true).id - Integrations::Jira.where(active: true, instance: false, template: false, inherit_from_id: nil).find_each do |integration| + Integrations::Jira.where(active: true, instance: false, inherit_from_id: nil).find_each do |integration| integration.update_attribute(:inherit_from_id, jira_integration_instance_id) end ``` diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index c7dbc5caf35..9139c374780 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- @@ -106,7 +106,7 @@ set up GitLab to create a new account when a Kerberos user tries to sign in. If you're an administrator, you can link a Kerberos account to an existing GitLab account. To do so: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Select a user, then select the **Identities** tab. @@ -148,7 +148,7 @@ With that information at hand: ``` 1. As an administrator, you can confirm the new, blocked account: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users** and review the **Blocked** tab. 1. You can enable the user. diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index fea25a398ba..389568ccf6d 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -224,7 +224,7 @@ sudo chown mattermost:mattermost /var/opt/gitlab/mattermost/config.json sudo gitlab-ctl start mattermost ``` -## Mattermost Command Line Tools (CLI) +## Mattermost command-line tool (CLI) [`mmctl`](https://docs.mattermost.com/manage/mmctl-command-line-tool.html) is a CLI tool for the Mattermost server which is installed locally and uses the Mattermost API, but may also be used remotely. You must configure Mattermost either for local connections or authenticate as an administrator with local login credentials (not through GitLab SSO). The executable is located at `/opt/gitlab/embedded/bin/mmctl`. @@ -308,42 +308,6 @@ This setting can also be configured in `/var/opt/gitlab/mattermost/config.json`. ## Upgrading GitLab Mattermost -Below is a list of Mattermost versions for GitLab 14.0 and later: - -| GitLab version | Mattermost version | -|:---------------|:-------------------| -| 16.2 | 7.10 | -| 16.1 | 7.10 | -| 16.0 | 7.10 | -| 15.11 | 7.9 | -| 15.10 | 7.8 | -| 15.9 | 7.7 | -| 15.8 | 7.5 | -| 15.7 | 7.5 | -| 15.6 | 7.4 | -| 15.5 | 7.3 | -| 15.4 | 7.2 | -| 15.3 | 7.1 | -| 15.2 | 7.0 | -| 15.1 | 6.7 | -| 15.0 | 6.6 | -| 14.10 | 6.5 | -| 14.9 | 6.4 | -| 14.8 | 6.3 | -| 14.7 | 6.2 | -| 14.6 | 6.1 | -| 14.5 | 5.39 | -| 14.4 | 5.39 | -| 14.3 | 5.38 | -| 14.2 | 5.37 | -| 14.1 | 5.36 | -| 14.0 | 5.35 | - -- GitLab 14.5 remained on Mattermost 5.39. -- GitLab 14.6 updates to Mattermost 6.1 instead of 6.0. -- GitLab 15.8 remained on Mattermost 7.5. -- GitLab 16.1 and 16.2 remained on Mattermost 7.10. - NOTE: When upgrading the Mattermost version, it is essential to check the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) @@ -368,7 +332,38 @@ If this is not the case, there are two options: For a complete list of upgrade notices and special considerations for older versions, see the [Mattermost documentation](https://docs.mattermost.com/administration/important-upgrade-notes.html). -## Upgrading GitLab Mattermost to 14.6 +### GitLab Mattermost versions shipped with the Linux package + +Below is a list of Mattermost version changes for GitLab 14.0 and later: + +| GitLab version | Mattermost version | Notes | +| :------------- | :----------------- | ---------------------------------------------------------------------------------------- | +| 16.4 | 8.1 | | +| 16.3 | 8.0 | | +| 16.0 | 7.10 | | +| 15.11 | 7.9 | | +| 15.10 | 7.8 | | +| 15.9 | 7.7 | | +| 15.7 | 7.5 | | +| 15.6 | 7.4 | | +| 15.5 | 7.3 | | +| 15.4 | 7.2 | | +| 15.3 | 7.1 | | +| 15.2 | 7.0 | | +| 15.1 | 6.7 | | +| 15.0 | 6.6 | | +| 14.10 | 6.5 | | +| 14.9 | 6.4 | | +| 14.8 | 6.3 | | +| 14.7 | 6.2 | | +| 14.6 | 6.1 | Updates to 6.1 instead of 6.0. [See upgrade notes](#upgrading-gitlab-mattermost-to-146). | +| 14.4 | 5.39 | | +| 14.3 | 5.38 | | +| 14.2 | 5.37 | | +| 14.1 | 5.36 | | +| 14.0 | 5.35 | | + +### Upgrading GitLab Mattermost to 14.6 GitLab 14.6 ships with Mattermost 6.1 including potentially long running database migrations for Mattermost 6.0. For information about upgrading and for ways to reduce the downtime caused by those migrations, read the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) for both versions. If you need to perform any manual migrations, [connect to the bundled PostgreSQL database](#connecting-to-the-bundled-postgresql-database). @@ -409,4 +404,3 @@ For help and support around your GitLab Mattermost deployment, see: - [Troubleshooting Mattermost issues](https://docs.mattermost.com/install/troubleshooting.html). - [Mattermost GitLab Issues Support Handbook](https://docs.mattermost.com/process/support.html?highlight=omnibus#gitlab-issues). -- [GitLab Mattermost issue tracker](https://gitlab.com/gitlab-org/gitlab-mattermost/-/issues) for verified bugs with repro steps. diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index 00a50489793..5595a45fe3e 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index fc849adc2b3..af525cc8770 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -75,7 +75,7 @@ To create a new application for a group: To create an application for your GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Applications**. 1. Select **New application**. @@ -85,6 +85,8 @@ The user authorization step is automatically skipped for this application. ## View all authorized applications +> `k8s_proxy` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422408) in GitLab 16.4 [with a flag](../administration/feature_flags.md) named `k8s_proxy_pat`. Enabled by default. + To see all the application you've authorized with your GitLab credentials: 1. On the left sidebar, select your avatar. @@ -95,7 +97,7 @@ The GitLab OAuth 2 applications support scopes, which allow application to perfo different actions. See the following table for all available scopes. | Scope | Description | -| ------------------ | ----------- | +|--------------------| ----------- | | `api` | Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry. | | `read_user` | Grants read-only access to the authenticated user's profile through the /user API endpoint, which includes username, public email, and full name. Also grants access to read-only API endpoints under /users. | | `read_api` | Grants read access to the API, including all groups and projects, the container registry, and the package registry. | @@ -108,6 +110,7 @@ different actions. See the following table for all available scopes. | `profile` | Grants read-only access to the user's profile data using [OpenID Connect](openid_connect_provider.md). | | `email` | Grants read-only access to the user's primary email address using [OpenID Connect](openid_connect_provider.md). | | `create_runner` | Grants permission to create runners. | +| `k8s_proxy` | Grants permission to perform Kubernetes API calls using the agent for Kubernetes. | At any time you can revoke any access by selecting **Revoke**. @@ -115,16 +118,27 @@ At any time you can revoke any access by selecting **Revoke**. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21745) in GitLab 14.3, with the ability to opt out. > - Ability to opt-out of expiring access token [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340848) in GitLab 15.0. +> - Database validation on `expires_in` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112765) in GitLab 15.10. If your GitLab instance has any remaining OAuth Access Tokens without `expires_in` set when you are upgrading to 15.10 or later, the database migration will raise an error. For workaround instructions, see the [GitLab 15.10.0 upgrade documentation](../update/versions/gitlab_15_changes.md#15100). WARNING: The ability to opt out of expiring access tokens was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/340848) in GitLab 14.3 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340848) in 15.0. All existing integrations must be updated to support access token refresh. -Access tokens expire after two hours. Integrations that use access tokens must generate new ones at least every -two hours. +Access tokens expire after two hours. Integrations that use access tokens must +generate new ones using the `refresh_token` attribute. Refresh tokens may be +used even after the `access_token` itself expires. +See [OAuth 2.0 token documentation](../api/oauth2.md) for more detailed +information on how to refresh expired access tokens. + +This expiration setting is set in the GitLab codebase using the +`access_token_expires_in` configuration from +[Doorkeeper](https://github.com/doorkeeper-gem/doorkeeper), the library that +provides GitLab as an OAuth provider functionality. The expiration setting is +not configurable. -When applications are deleted, all grants and tokens associated with the application are also deleted. +When applications are deleted, all grants and tokens associated with the +application are also deleted. ## Hashed OAuth application secrets diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 30aa913ab8c..6add3534593 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -257,7 +257,7 @@ By default, sign-in is enabled for all the OAuth providers configured in `config To enable or disable an OmniAuth provider: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Sign-in restrictions**. diff --git a/doc/integration/openid_connect_provider.md b/doc/integration/openid_connect_provider.md index ec3c2d53495..c9c208bf1c1 100644 --- a/doc/integration/openid_connect_provider.md +++ b/doc/integration/openid_connect_provider.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -48,22 +48,22 @@ Similar URLs can be used for other GitLab instances. The following user information is shared with clients: -| Claim | Type | Description | -|:---------------------|:----------|:------------| -| `sub` | `string` | The ID of the user | -| `auth_time` | `integer` | The timestamp for the user's last authentication | -| `name` | `string` | The user's full name | -| `nickname` | `string` | The user's GitLab username | -| `preferred_username` | `string` | The user's GitLab username | -| `email` | `string` | The user's email address<br>This is the user's *primary* email address if the application has access to the `email` claim and the user's *public* email address otherwise | -| `email_verified` | `boolean` | Whether the user's email address was verified | -| `website` | `string` | URL for the user's website | -| `profile` | `string` | URL for the user's GitLab profile | -| `picture` | `string` | URL for the user's GitLab avatar | -| `groups` | `array` | Paths for the groups the user is a member of, either directly or through an ancestor group. | -| `groups_direct` | `array` | Paths for the groups the user is a direct member of. | -| `https://gitlab.org/claims/groups/owner` | `array` | Names of the groups the user is a direct member of with Owner role | -| `https://gitlab.org/claims/groups/maintainer` | `array` | Names of the groups the user is a direct member of with Maintainer role | -| `https://gitlab.org/claims/groups/developer` | `array` | Names of the groups the user is a direct member of with Developer role | - -The claims `sub`, `sub_legacy`, `email`, `email_verified` and `groups_direct` are included in the ID token. All other claims are available from the `/oauth/userinfo` endpoint used by OIDC clients. +| Claim | Type | Description | Included in ID Token | Included in `userinfo` endpoint | +|:---------------------|:----------|:------------|:---------------------|:------------------------------| +| `sub` | `string` | The ID of the user | **{check-circle}** Yes | **{check-circle}** Yes | +| `auth_time` | `integer` | The timestamp for the user's last authentication | **{check-circle}** Yes | **{dotted-circle}** No | +| `name` | `string` | The user's full name | **{check-circle}** Yes | **{check-circle}** Yes | +| `nickname` | `string` | The user's GitLab username | **{check-circle}** Yes| **{check-circle}** Yes | +| `preferred_username` | `string` | The user's GitLab username | **{check-circle}** Yes | **{check-circle}** Yes | +| `email` | `string` | The user's email address<br>This is the user's *primary* email address | **{check-circle}** Yes | **{check-circle}** Yes | +| `email_verified` | `boolean` | Whether the user's email address was verified | **{check-circle}** Yes | **{check-circle}** Yes | +| `website` | `string` | URL for the user's website | **{check-circle}** Yes | **{check-circle}** Yes | +| `profile` | `string` | URL for the user's GitLab profile | **{check-circle}** Yes | **{check-circle}** Yes| +| `picture` | `string` | URL for the user's GitLab avatar | **{check-circle}** Yes| **{check-circle}** Yes | +| `groups` | `array` | Paths for the groups the user is a member of, either directly or through an ancestor group. | **{dotted-circle}** No | **{check-circle}** Yes | +| `groups_direct` | `array` | Paths for the groups the user is a direct member of. | **{check-circle}** Yes | **{dotted-circle}** No | +| `https://gitlab.org/claims/groups/owner` | `array` | Names of the groups the user is a direct member of with Owner role | **{dotted-circle}** No | **{check-circle}** Yes | +| `https://gitlab.org/claims/groups/maintainer` | `array` | Names of the groups the user is a direct member of with Maintainer role | **{dotted-circle}** No | **{check-circle}** Yes | +| `https://gitlab.org/claims/groups/developer` | `array` | Names of the groups the user is a direct member of with Developer role | **{dotted-circle}** No | **{check-circle}** Yes | + +The claims `email` and `email_verified` are only added if the application has access to the `email` claim and the user's **public** email address, otherwise they are not included. All other claims are available from the `/oauth/userinfo` endpoint used by OIDC clients. diff --git a/doc/integration/recaptcha.md b/doc/integration/recaptcha.md index 9e44efc3e60..cd94cdfa43c 100644 --- a/doc/integration/recaptcha.md +++ b/doc/integration/recaptcha.md @@ -17,7 +17,7 @@ To use reCAPTCHA, first create a site and private key. 1. Go to the [Google reCAPTCHA page](https://www.google.com/recaptcha/admin). 1. To get reCAPTCHA v2 keys, fill in the form and select **Submit**. 1. Sign in to your GitLab server as an administrator. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Reporting**. 1. Expand **Spam and Anti-bot Protection**. diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 3a90a4570e6..a1df213d9dc 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/saml.md b/doc/integration/saml.md index e4ad8c33aa1..7af1dbeeff8 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference @@ -116,7 +116,7 @@ For more information on: omniauth: enabled: true allowSingleSignOn: ['saml'] - blockAutoCreatedUsers: true + blockAutoCreatedUsers: false ``` 1. Optional. You can automatically link SAML users with existing GitLab users if their diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index 859b5682879..c392864e346 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/integration/sourcegraph.md b/doc/integration/sourcegraph.md index 076b7f3937e..729dbade1d1 100644 --- a/doc/integration/sourcegraph.md +++ b/doc/integration/sourcegraph.md @@ -49,7 +49,7 @@ You can skip this step if you already have your GitLab repositories searchable i ### Configure your GitLab instance with Sourcegraph -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Sourcegraph** configuration section. diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index ed83db0d4eb..6491df1ec56 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 36c5ed03197..82982a03016 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -1,5 +1,5 @@ --- -stage: Analytics +stage: Analyze group: Observability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -8,6 +8,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w Error Tracking allows developers to discover and view errors generated by their application. Because error information is surfaced where the code is developed, this increases efficiency and awareness. Users can choose between [GitLab Integrated error tracking](#integrated-error-tracking) and [Sentry based](#sentry-error-tracking) backends. +To leave feedback about Error Tracking bugs or functionality, please comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2362) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). + ## How error tracking works For error tracking to work, you need: @@ -172,7 +174,7 @@ To enable the Sentry integration: `event:read`, and `event:write` (for resolving events). 1. In GitLab, enable and configure Error Tracking: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor > Error Tracking**. 1. Under **Enable error tracking**, select the **Active** checkbox. 1. Under **Error tracking backend**, select **Sentry**. @@ -206,3 +208,15 @@ To rectify the following error, specify the deprecated DSN in **Sentry.io > Proj ```plaintext ERROR: Sentry failure builds=0 error=raven: dsn missing private key ``` + +## Troubleshooting + +When working with Error Tracking, you might encounter the following issues. + +### Error `Connection failed. Check auth token and try again` + +If the Monitor feature is disabled in the [project settings](../user/project/settings/index.md#configure-project-features-and-permissions), +you might see an error when you try to [enable Sentry integration for a project](#enable-sentry-integration-for-a-project). +The resulting request to `/project/path/-/error_tracking/projects.json?api_host=https:%2F%2Fsentry.example.com%2F&token=<token>` returns a 404 status. + +To fix this issue, enable the Monitor feature for the project. diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 21470f66750..5fd497a79e1 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -39,7 +39,7 @@ with GitLab, so it's up to developers to use a compatible client library and To create and enable a feature flag: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Select **New feature flag**. 1. Enter a name that starts with a letter and contains only lowercase letters, digits, underscores (`_`), @@ -181,7 +181,7 @@ For example: To create a user list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Select **View user lists** 1. Select **New user list**. @@ -197,7 +197,7 @@ When viewing a list, you can rename it by selecting **Edit** (**{pencil}**). To add users to a user list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Select **Edit** (**{pencil}**) next to the list you want to add users to. 1. Select **Add Users**. @@ -211,7 +211,7 @@ To add users to a user list: To remove users from a user list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Select **Edit** (**{pencil}**) next to the list you want to change. 1. Select **Remove** (**{remove}**) next to the ID you want to remove. @@ -224,7 +224,7 @@ To remove the feature flag from the code during cleanup, find any project refere To search for code references of a feature flag: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Edit the feature flag you want to remove. 1. Select **More actions** (**{ellipsis_v}**). @@ -235,7 +235,7 @@ To search for code references of a feature flag: In [GitLab 13.0 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/8621), to disable a feature flag for a specific environment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. For the feature flag you want to disable, select **Edit** (**{pencil}**). 1. To disable the flag: @@ -250,7 +250,7 @@ to disable a feature flag for a specific environment: To disable a feature flag for all environments: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. For the feature flag you want to disable, slide the Status toggle to **Disabled**. @@ -265,7 +265,7 @@ Then prepare your application with a client library. To get the access credentials that your application needs to communicate with GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Feature flags**. 1. Select **Configure** to view the following: - **API URL**: URL where the client (application) connects to get a list of feature flags. diff --git a/doc/operations/img/tracing_list_v16_3.png b/doc/operations/img/tracing_list_v16_3.png Binary files differnew file mode 100644 index 00000000000..93c336d4bd7 --- /dev/null +++ b/doc/operations/img/tracing_list_v16_3.png diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index e558462cad7..f92739580bd 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -179,7 +179,7 @@ To assign an alert: 1. Display the list of current alerts: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Alerts**. 1. Select your desired alert to display its details. @@ -219,7 +219,7 @@ Prerequisites: To configure the actions: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand the **Alerts** section, then select the **Alert settings** tab. 1. Select the **Create an incident** checkbox. diff --git a/doc/operations/incident_management/escalation_policies.md b/doc/operations/incident_management/escalation_policies.md index 0cbbf42372e..604e223cf55 100644 --- a/doc/operations/incident_management/escalation_policies.md +++ b/doc/operations/incident_management/escalation_policies.md @@ -22,7 +22,7 @@ Prerequisite: To create an escalation policy: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Escalation Policies**. 1. Select **Add an escalation policy**. 1. Enter the policy's name and description, and @@ -46,7 +46,7 @@ the paged users is created on the alert. To update an escalation policy: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Escalation Policies**. 1. Select **Edit escalation policy** (**{pencil}**). 1. Edit the information. @@ -56,7 +56,7 @@ To update an escalation policy: To delete an escalation policy: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Escalation Policies**. 1. Select **Delete escalation policy** (**{remove}**). 1. On the confirmation dialog, select **Delete escalation policy**. diff --git a/doc/operations/incident_management/incident_timeline_events.md b/doc/operations/incident_management/incident_timeline_events.md index 0c47ff46bc9..f5322e0a7cc 100644 --- a/doc/operations/incident_management/incident_timeline_events.md +++ b/doc/operations/incident_management/incident_timeline_events.md @@ -23,7 +23,7 @@ They are grouped with dates and are listed in ascending order of the time when t To view the event timeline of an incident: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. Select the **Timeline** tab. @@ -42,7 +42,7 @@ Prerequisites: To create a timeline event: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. Select the **Timeline** tab. @@ -66,7 +66,7 @@ Prerequisites: To create a timeline event from a comment on the incident: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. Create a comment or choose an existing comment. @@ -83,7 +83,7 @@ of an incident.  -### When labels change (Experiment) +### When labels change **(EXPERIMENT)** > [Introduced]([issue-link](https://gitlab.com/gitlab-org/gitlab/-/issues/365489)) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `incident_timeline_events_from_labels`. Disabled by default. @@ -106,7 +106,7 @@ Prerequisites: To delete a timeline event: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. Select the **Timeline** tab. diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 4aae8809620..43df69a3b39 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -154,7 +154,7 @@ Prerequisites: To configure the timer: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand the **Incidents** section, then select the **Incident settings** tab. 1. Select **Activate "time to SLA" countdown timer**. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index 5eed1921168..8691d78c4c9 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -101,7 +101,7 @@ parameters. All fields are optional. If the incoming alert does not contain a va | `monitoring_tool` | String | The name of the associated monitoring tool. | | `hosts` | String or Array | One or more hosts, as to where this incident occurred. | | `severity` | String | The severity of the alert. Case-insensitive. Can be one of: `critical`, `high`, `medium`, `low`, `info`, `unknown`. Defaults to `critical` if missing or value is not in this list. | -| `fingerprint` | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. | +| `fingerprint` | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. When the `generic_alert_fingerprinting` feature is enabled, the fingerprint is generated automatically based on the payload (excluding the `start_time`, `end_time`, and `hosts` parameters). | | `gitlab_environment_name` | String | The name of the associated GitLab [environment](../../ci/environments/index.md). Required to [display alerts on a dashboard](../../user/operations_dashboard/index.md#adding-a-project-to-the-dashboard). | You can also add custom fields to the alert's payload. The values of extra @@ -144,7 +144,7 @@ Prerequisites: - You must have at least the Maintainer role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand the **Alerts** section, and select **Add new integration**. 1. From the **Select integration type** dropdown list, select **Prometheus**. @@ -418,6 +418,11 @@ receives a payload with the end time of the alert set. For HTTP Endpoints without [custom mappings](#map-fields-in-custom-alerts), the expected field is `end_time`. With custom mappings, you can select the expected field. +GitLab determines which alert to resolve based on the `fingerprint` value that can be provided as +part of the payload. +For more information on alert properties and mappings, see +[Customize the alert payload outside of GitLab](#customize-the-alert-payload-outside-of-gitlab). + You can also configure the associated [incident to be closed automatically](../incident_management/manage_incidents.md#automatically-close-incidents-via-recovery-alerts) when the alert resolves. ## Link to your Opsgenie Alerts **(PREMIUM ALL)** diff --git a/doc/operations/incident_management/linked_resources.md b/doc/operations/incident_management/linked_resources.md index 15317f61057..57fcb31e3ba 100644 --- a/doc/operations/incident_management/linked_resources.md +++ b/doc/operations/incident_management/linked_resources.md @@ -27,7 +27,7 @@ Linked resources for an incident are listed under the **Summary** tab. To view the linked resources of an incident: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. @@ -41,7 +41,7 @@ Prerequisites: To add a linked resource: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. In the **Linked resources** section, select the plus icon (**{plus-square}**). @@ -92,7 +92,7 @@ Prerequisites: To remove a linked resource: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select an incident. 1. In the **Linked resources** section, select **Remove** (**{close}**). diff --git a/doc/operations/incident_management/manage_incidents.md b/doc/operations/incident_management/manage_incidents.md index f88d9b51891..28c4adec250 100644 --- a/doc/operations/incident_management/manage_incidents.md +++ b/doc/operations/incident_management/manage_incidents.md @@ -24,7 +24,7 @@ Prerequisites: To create an incident from the incidents list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. 1. Select **Create incident**. @@ -38,7 +38,7 @@ Prerequisites: To create an incident from the issues list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, and select **New issue**. 1. From the **Type** dropdown list, select **Incident**. Only fields relevant to incidents are available on the page. @@ -57,7 +57,7 @@ Prerequisites: To create an incident from an alert: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Alerts**. 1. Select your desired alert. 1. Select **Create incident**. @@ -88,7 +88,7 @@ Prerequisites: To set up a webhook with PagerDuty: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor** 1. Expand **Incidents**. 1. Select the **PagerDuty integration** tab. @@ -104,7 +104,7 @@ check if a GitLab incident is created from the incident. To view the [incidents list](incidents.md#incidents-list): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Incidents**. To view an incident's [details page](incidents.md#incident-details), select it from the list. @@ -239,7 +239,7 @@ Prerequisites: To configure the setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand the **Incidents** section. 1. Select the **Automatically close associated incident** checkbox. diff --git a/doc/operations/incident_management/oncall_schedules.md b/doc/operations/incident_management/oncall_schedules.md index 808dc30581d..255cc79b3ce 100644 --- a/doc/operations/incident_management/oncall_schedules.md +++ b/doc/operations/incident_management/oncall_schedules.md @@ -28,7 +28,7 @@ Prerequisite: To create an on-call schedule: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. Select **Add a schedule**. 1. Enter the schedule's name and description and select a time zone. @@ -43,7 +43,7 @@ create [rotations](#rotations) for your schedule. To update a schedule: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. Select **Edit schedule** (**{pencil}**). 1. Edit the information. @@ -56,7 +56,7 @@ interval (if one is set) to the corresponding times in the new time zone. To delete a schedule: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. Select **Delete escalation policy** (**{remove}**). 1. On the confirmation dialog, select **Delete schedule**. @@ -67,7 +67,7 @@ Add rotations to an existing schedule to put your team members on-call. To create a rotation: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. Select the **Add a rotation** link. 1. Enter the following information: @@ -85,7 +85,7 @@ To create a rotation: To edit a rotation: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. In the **Rotations** section, select **Edit rotation** (**{pencil}**). 1. Edit the information. @@ -95,7 +95,7 @@ To edit a rotation: To delete a rotation: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > On-call Schedules**. 1. In the **Rotations** section, select **Delete rotation** (**{remove}**). 1. On the confirmation dialog, select **Delete rotation**. diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 05cc8c8e221..845c17d974a 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -26,7 +26,7 @@ Email notifications are available in projects for triggered alerts. Project members with the **Owner** or **Maintainer** roles have the option to receive a single email notification for new alerts. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand **Alerts**. 1. On the **Alert settings** tab, select the diff --git a/doc/operations/incident_management/slack.md b/doc/operations/incident_management/slack.md index 82d5d8b3150..100aaa0d9bc 100644 --- a/doc/operations/incident_management/slack.md +++ b/doc/operations/incident_management/slack.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Incident management for Slack (Beta) **(FREE SAAS)** +# Incident management for Slack **(FREE SAAS BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344856) in GitLab 15.7 [with a flag](../../administration/feature_flags.md) named `incident_declare_slash_command`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/378072) in GitLab 15.10 in [Beta](../../policy/experiment-beta-support.md#beta). diff --git a/doc/operations/incident_management/status_page.md b/doc/operations/incident_management/status_page.md index 374571d225e..682b2ffd705 100644 --- a/doc/operations/incident_management/status_page.md +++ b/doc/operations/incident_management/status_page.md @@ -45,7 +45,7 @@ Prerequisite: To provide GitLab with the AWS account information needed to push content to your Status Page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand **Status page**. 1. Select the **Active** checkbox. @@ -96,7 +96,7 @@ the issue can potentially [publish comments to your GitLab Status Page](#publish After creating the CI/CD variables, configure the Project you want to use for Incident issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Monitor**. 1. Expand **Status page**. 1. Fill in your cloud provider's credentials and make sure to select the **Active** checkbox. diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md deleted file mode 100644 index a5f580ac503..00000000000 --- a/doc/operations/metrics/alerts.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Set up alerts for Prometheus metrics (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md deleted file mode 100644 index 3eec1272307..00000000000 --- a/doc/operations/metrics/dashboards/default.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# GitLab-defined metrics dashboards (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/develop.md b/doc/operations/metrics/dashboards/develop.md deleted file mode 100644 index c6d4721863a..00000000000 --- a/doc/operations/metrics/dashboards/develop.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Developing templates for custom dashboards (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md deleted file mode 100644 index df94f979f5f..00000000000 --- a/doc/operations/metrics/dashboards/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Custom dashboards (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md deleted file mode 100644 index 8dcf4e6ad28..00000000000 --- a/doc/operations/metrics/dashboards/panel_types.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Panel types for dashboards (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md deleted file mode 100644 index 7abcbf88d59..00000000000 --- a/doc/operations/metrics/dashboards/settings.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Dashboard settings (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/templating_variables.md b/doc/operations/metrics/dashboards/templating_variables.md deleted file mode 100644 index ceeeb229a81..00000000000 --- a/doc/operations/metrics/dashboards/templating_variables.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Templating variables for metrics dashboards (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md deleted file mode 100644 index b4ea05a0cea..00000000000 --- a/doc/operations/metrics/dashboards/variables.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Using variables (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md deleted file mode 100644 index 95c02319b19..00000000000 --- a/doc/operations/metrics/dashboards/yaml.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Dashboard YAML properties (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/dashboards/yaml_number_format.md b/doc/operations/metrics/dashboards/yaml_number_format.md deleted file mode 100644 index e75beadfab9..00000000000 --- a/doc/operations/metrics/dashboards/yaml_number_format.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../index.md' ---- - -# Unit formats reference (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md deleted file mode 100644 index 68f115b66db..00000000000 --- a/doc/operations/metrics/embed.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Embedding metric charts within GitLab Flavored Markdown (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md deleted file mode 100644 index 82967aa663f..00000000000 --- a/doc/operations/metrics/embed_grafana.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- -# Embed Grafana panels in Markdown (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. -Use [embed charts](https://gitlab.com/groups/gitlab-org/opstrace/-/epics/33) instead. diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md deleted file mode 100644 index 65f84c9c825..00000000000 --- a/doc/operations/metrics/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Monitor your environment's metrics (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md new file mode 100644 index 00000000000..608a9c998f8 --- /dev/null +++ b/doc/operations/tracing.md @@ -0,0 +1,75 @@ +--- +stage: Analyze +group: Observability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Distributed tracing **(ULTIMATE SAAS EXPERIMENT)** + +> Introduced in GitLab 16.3 [with flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. Disabled by default. + +FLAG: +On GitLab.com, by default this feature is not available. To make it available, +an administrator can [enable the feature flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. +The feature is not ready for production use. + +With distributed tracing, you can troubleshoot application performance issues by inspecting how a request moves through different services and systems, the timing of each operation, and any errors or logs as they occur. Tracing is particularly useful in the context of microservice applications, which group multiple independent services collaborating to fulfill user requests. + +This feature is an [Experiment](../policy/experiment-beta-support.md). For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/observability/). To leave feedback about tracing bugs or functionality, please comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2363) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). + +## Configure distributed tracing for a project + +To configure distributed tracing: + +1. [Create an access token and enable tracing.](#create-an-access-token-and-enable-tracing) +1. [Configure your application to use the OpenTelemetry exporter.](#configure-your-application-to-use-the-opentelemetry-exporter) + +### Create an access token and enable tracing + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To enable tracing in a project: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > Access Tokens**. +1. Create an access token with the following scopes: `read_api`, `read_observability`, `write_observability`. +1. Copy the value of the access token. +1. Navigate to your project. +1. Select **Monitor > Tracing**. +1. Select **Enable**. + +## Configure your application to use the OpenTelemetry exporter + +Next, configure your application to send traces to GitLab. + +To do this, set the following environment variables: + +```shell +OTEL_EXPORTER = "otlphttp" +OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "https://observe.gitlab.com/v3/<namespace-id>/<gitlab-project-id>/ingest/traces" +OTEL_EXPORTER_OTLP_TRACES_HEADERS = "PRIVATE-TOKEN=<gitlab-access-token>" +``` + +Use the following values: + +- `namespace-id`: The top-level namespace ID where your project is located. +- `gitlab-project-id`: The project ID. +- `gitlab-access-token`: The access token you [created previously](#create-an-access-token-and-enable-tracing). + +When your application is configured, run it, and the OpenTelemetry exporter attempts to send +traces to GitLab. + +## View your traces + +If your traces are exported successfully, you can see them in the project. + +To view the list of traces: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Traces**. + +To see the details of a trace, select it from the list. + + diff --git a/doc/policy/alpha-beta-support.md b/doc/policy/alpha-beta-support.md deleted file mode 100644 index bfea5b3a2d6..00000000000 --- a/doc/policy/alpha-beta-support.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'experiment-beta-support.md' -remove_date: '2023-09-07' ---- - -This document was moved to [another location](experiment-beta-support.md). - -<!-- This redirect file can be deleted after <2023-09-07>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/policy/experiment-beta-support.md b/doc/policy/experiment-beta-support.md index 68cc96f118b..a87a72d7910 100644 --- a/doc/policy/experiment-beta-support.md +++ b/doc/policy/experiment-beta-support.md @@ -13,6 +13,9 @@ All other features are considered to be Generally Available (GA). ## Experiment Support is not provided for features listed as "Experimental" or "Alpha" or any similar designation. Issues regarding such features should be opened in the GitLab issue tracker. Teams should release features as GA from the start unless there are strong reasons to release them as Experiment or Beta versions first. +All Experimental features must [initiate Production Readiness Review](https://about.gitlab.com/handbook/engineering/infrastructure/production/readiness/#process) and complete the [experiment section in the readiness template](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/.gitlab/issue_templates/production_readiness.md#experiment). + +Experimental features are: - Not ready for production use. - No support available. @@ -20,7 +23,7 @@ Support is not provided for features listed as "Experimental" or "Alpha" or any - Can be removed at any time. - Data loss may occur. - Documentation may not exist or just be in a blog format. -- Offer an easy way to choose to opt-in to experimental features with minimal friction. For example, needing to flip a feature flag is too much friction, but a group or project-level setting that is in the UI is not. +- Offer a way to opt-in with minimal friction. For example, needing to flip a feature flag is too much friction, but a group or project-level setting in the UI is not. - Link out to the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/) in the opt-in. - Documentation reflects that the feature is subject to the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). - [UI reflects experiment status](https://design.gitlab.com/usability/feature-management#highlighting-feature-versions). @@ -33,6 +36,9 @@ Support is not provided for features listed as "Experimental" or "Alpha" or any ## Beta Commercially-reasonable efforts are made to provide limited support for features designated as "Beta," with the expectation that issues require extra time and assistance from development to troubleshoot. +All Beta features must complete all sections up to and including the [beta section in the readiness template](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/.gitlab/issue_templates/production_readiness.md#beta) by following the [Production Readiness Review process](https://about.gitlab.com/handbook/engineering/infrastructure/production/readiness/#process). + +Beta features are: - May not be ready for production use. - Support on a commercially-reasonable effort basis. @@ -50,7 +56,9 @@ Commercially-reasonable efforts are made to provide limited support for features ## Generally Available (GA) -Generally Available features means that they passed the [Production Readiness Review](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/.gitlab/issue_templates/production_readiness.md) for GitLab.com, and are: +Generally Available features must complete the [Production Readiness Review](https://about.gitlab.com/handbook/engineering/infrastructure/production/readiness) and complete all sections up to and including the [GA section in the readiness template](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/.gitlab/issue_templates/production_readiness.md#general-availability). + +GA features are: - Ready for production use at any scale. - Fully documented and supported. diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index a6de6f594fb..bb27fcb27e0 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -36,8 +36,8 @@ The following table describes the version types and their release cadence: | Version type | Description | Cadence | |:-------------|:------------|:--------| -| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 16.0 on May 22, 2023. GitLab schedules major releases on May 22 each year, by default. | -| Minor | For when new backward-compatible functionality is introduced to the public API, a minor feature is introduced, or when a set of smaller features is rolled out. | Monthly on the 22nd. | +| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 17.0, scheduled for May 16th, 2024. GitLab [schedules major releases](https://about.gitlab.com/releases/) for May each year, by default. | +| Minor | For when new backward-compatible functionality is introduced to the public API, a minor feature is introduced, or when a set of smaller features is rolled out. | Monthly. | | Patch | For backward-compatible bug fixes that fix incorrect behavior. See [Patch releases](#patch-releases). | As needed. | ## Upgrade recommendations diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 11a6e06fcf5..5ffed097a51 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -156,7 +156,7 @@ These commands don't work for artifacts stored on [object storage](../administration/object_storage.md). WARNING: -Prior to GitLab 14.9, this task incorrectly deletes [pipeline artifacts](../ci/pipelines/pipeline_artifacts.md). +Prior to GitLab 14.9, this task incorrectly deletes [test coverage-related artifacts](../ci/testing/test_coverage_visualization.md). [The bug fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81022) was also back-ported to 14.6.6, 14.7.5, and 14.8.3. Upgrade to a release with the bug fix to avoid data loss. diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md deleted file mode 100644 index 77c7d2c2c1a..00000000000 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: 'index.md' ---- - -# Sample Prometheus data Rake task (removed) **(FREE SELF)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md deleted file mode 100644 index d3b734eba12..00000000000 --- a/doc/raketasks/import.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' ---- - -# Import bare repositories (removed) **(FREE SELF)** - -The Rake task for importing bare repositories was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108507) -in GitLab 15.8 and [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118676) in GitLab 16.0. diff --git a/doc/raketasks/list_repos.md b/doc/raketasks/list_repos.md index 258d41aa190..b404e4dec60 100644 --- a/doc/raketasks/list_repos.md +++ b/doc/raketasks/list_repos.md @@ -4,7 +4,14 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# List repository directories Rake task **(FREE SELF)** +<!--- start_remove The following content will be removed on remove_date: '2024-05-16' --> + +# List repository directories Rake task (deprecated) **(FREE SELF)** + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/384361) in GitLab 16.4 and is planned for removal in 17.0. +[If migrating GitLab, use backup and restore](../administration/operations/moving_repositories.md#recommended-approach-in-all-cases) +instead. You can print a list of all Git repositories on disk managed by GitLab. @@ -34,3 +41,5 @@ sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015' cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015' ``` + +<!--- end_remove -->
\ No newline at end of file diff --git a/doc/raketasks/x509_signatures.md b/doc/raketasks/x509_signatures.md index ab35f432fc2..ec85c400d49 100644 --- a/doc/raketasks/x509_signatures.md +++ b/doc/raketasks/x509_signatures.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # X.509 signatures Rake task **(FREE SELF)** -When [signing commits with X.509](../user/project/repository/x509_signed_commits/index.md), +When [signing commits with X.509](../user/project/repository/signed_commits/x509.md), the trust anchor might change and the signatures stored within the database must be updated. ## Update all X.509 signatures diff --git a/doc/security/asset_proxy.md b/doc/security/asset_proxy.md index cde377cbb73..16051d22bd6 100644 --- a/doc/security/asset_proxy.md +++ b/doc/security/asset_proxy.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md index fdf3e5055b0..2c9969ab707 100644 --- a/doc/security/crime_vulnerability.md +++ b/doc/security/crime_vulnerability.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference diff --git a/doc/security/email_verification.md b/doc/security/email_verification.md index da844e3a2eb..7ebdfc32d2e 100644 --- a/doc/security/email_verification.md +++ b/doc/security/email_verification.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/hardening.md b/doc/security/hardening.md index 21b8594fc6e..9c222e5c758 100644 --- a/doc/security/hardening.md +++ b/doc/security/hardening.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/hardening_application_recommendations.md b/doc/security/hardening_application_recommendations.md index e9c09abdea1..5a11c53ffee 100644 --- a/doc/security/hardening_application_recommendations.md +++ b/doc/security/hardening_application_recommendations.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -14,7 +14,7 @@ web interface. ## System hooks -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **System Hooks**. @@ -33,7 +33,7 @@ encouraged for communications through system hooks. ## Push rules -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Push Rules**. @@ -48,7 +48,7 @@ The adjustments help limit pushes to established and authorized users. ## Deploy keys -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Deploy Keys**. @@ -61,7 +61,7 @@ the documentation on [deploy keys](../user/project/deploy_keys/index.md) and ## General -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. @@ -180,7 +180,7 @@ For more detailed information, see ## Integrations -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Integrations**. @@ -192,7 +192,7 @@ process or authenticated user. ## Metrics and profiling -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Metrics and profiling**. @@ -206,11 +206,11 @@ security patches come out frequently, this helps you stay up to date. restrict data gathering and statistics reporting to a software vendor, you may have to disable the **Enable service ping** feature. For more information on what data is collected to help you make an informed decision, see -[service ping](../development/service_ping/index.md). +[service ping](../development/internal_analytics/service_ping/index.md). ## Network -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. diff --git a/doc/security/hardening_cicd_recommendations.md b/doc/security/hardening_cicd_recommendations.md index 16b649cbdd7..72a3699868b 100644 --- a/doc/security/hardening_cicd_recommendations.md +++ b/doc/security/hardening_cicd_recommendations.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/hardening_configuration_recommendations.md b/doc/security/hardening_configuration_recommendations.md index 1cc3294f68b..e8cae41c535 100644 --- a/doc/security/hardening_configuration_recommendations.md +++ b/doc/security/hardening_configuration_recommendations.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/hardening_general_concepts.md b/doc/security/hardening_general_concepts.md index a227f0134d0..3c50196f9bc 100644 --- a/doc/security/hardening_general_concepts.md +++ b/doc/security/hardening_general_concepts.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/hardening_operating_system_recommendations.md b/doc/security/hardening_operating_system_recommendations.md index 33f88d43d22..80eea9b5085 100644 --- a/doc/security/hardening_operating_system_recommendations.md +++ b/doc/security/hardening_operating_system_recommendations.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/identity_verification.md b/doc/security/identity_verification.md index a4f7baad0e2..b6932d88820 100644 --- a/doc/security/identity_verification.md +++ b/doc/security/identity_verification.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/security/index.md b/doc/security/index.md index 5365228537f..d3bff521fcb 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: index @@ -24,7 +24,7 @@ type: index - [Proxying images](asset_proxy.md) - [CI/CD variables](../ci/variables/index.md#cicd-variable-security) - [Token overview](token_overview.md) -- [Maximum decompressed file size for imported archives](../administration/settings/account_and_limit_settings.md#maximum-decompressed-file-size-for-imported-archives) +- [Maximum decompressed file size for imported archives](../administration/settings/import_and_export_settings.md#maximum-decompressed-file-size-for-imported-archives) - [Responding to security incidents](responding_to_security_incidents.md) To harden your GitLab instance and minimize the risk of unwanted user account creation, consider access control features like [Sign up restrictions](../administration/settings/sign_up_restrictions.md) and [Authentication options](../topics/authentication/index.md). For more detailed information, refer to [Hardening](hardening.md). diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 21e4ad8b108..a0d7b425a23 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts diff --git a/doc/security/password_length_limits.md b/doc/security/password_length_limits.md index d8e9728f455..c7ebd713240 100644 --- a/doc/security/password_length_limits.md +++ b/doc/security/password_length_limits.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, howto @@ -24,7 +24,7 @@ The user password length is set to a minimum of 8 characters by default. To change the minimum password length using GitLab UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Sign-up restrictions**. diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md index e814f4e5069..71e7510513e 100644 --- a/doc/security/password_storage.md +++ b/doc/security/password_storage.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference diff --git a/doc/security/passwords_for_integrated_authentication_methods.md b/doc/security/passwords_for_integrated_authentication_methods.md index a141241f97c..c7d94120887 100644 --- a/doc/security/passwords_for_integrated_authentication_methods.md +++ b/doc/security/passwords_for_integrated_authentication_methods.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference diff --git a/doc/security/project_import_decompressed_archive_size_limits.md b/doc/security/project_import_decompressed_archive_size_limits.md index 48767740625..a4749d0e5f9 100644 --- a/doc/security/project_import_decompressed_archive_size_limits.md +++ b/doc/security/project_import_decompressed_archive_size_limits.md @@ -1,9 +1,9 @@ --- -redirect_to: '../administration/settings/account_and_limit_settings.md#maximum-decompressed-file-size-for-imported-archives' +redirect_to: '../administration/settings/import_and_export_settings.md#maximum-decompressed-file-size-for-imported-archives' remove_date: '2023-11-02' --- -This document was moved to [another location](../administration/settings/account_and_limit_settings.md#maximum-decompressed-file-size-for-imported-archives). +This document was moved to [another location](../administration/settings/import_and_export_settings.md#maximum-decompressed-file-size-for-imported-archives). <!-- This redirect file can be deleted after <2023-11-02>. --> <!-- Redirects that point to other docs in the same project expire in three months. --> diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index ee10d66a8ad..936e2931ff5 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, howto @@ -121,6 +121,9 @@ The **rate limit** is 20 calls per minute per IP address. ### Project Jobs API endpoint +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/382985) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `ci_enforce_rate_limits_jobs_api`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/384186) in GitLab 16.0. Feature flag `ci_enforce_rate_limits_jobs_api` removed. + There is a rate limit for the endpoint `project/:id/jobs`, which is enforced to reduce timeouts when retrieving jobs. The **rate limit** is 600 calls per minute per authenticated user. @@ -186,7 +189,7 @@ To remove a blocked IP: keys *rack::attack* ``` -By default, the [`keys` command is disabled](https://docs.gitlab.com/omnibus/settings/redis.html#renamed-commands). + By default, the [`keys` command is disabled](https://docs.gitlab.com/omnibus/settings/redis.html#renamed-commands). 1. Optionally, add [the IP to the allowlist](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-rack-attack) to prevent it being denylisted again. diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index fa15efe7cb7..4a59d2f9a21 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto @@ -20,7 +20,7 @@ The user's new password must meet all [password requirements](../user/profile/us To reset a user's password in the UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. For the user whose password you want to update, select **Edit**. diff --git a/doc/security/responding_to_security_incidents.md b/doc/security/responding_to_security_incidents.md index 0cd7170d35b..b5e38ce55ca 100644 --- a/doc/security/responding_to_security_incidents.md +++ b/doc/security/responding_to_security_incidents.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, howto diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index 87cbf12471f..90affd089f3 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -20,7 +20,7 @@ limit the allowed SSH key algorithms. GitLab allows you to restrict the allowed SSH key technology as well as specify the minimum key length for each technology: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General** . 1. Expand the **Visibility and access controls** section: diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index cb01c7d5160..f605e95dfbf 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference @@ -20,9 +20,10 @@ You can create [Personal access tokens](../user/profile/personal_access_tokens.m You can limit the scope and expiration date of your personal access tokens. By default, they inherit permissions from the user who created them. -You can use the [personal access tokens API](../api/personal_access_tokens.md) to -programmatically take action, such as -[rotating a personal access token](../api/personal_access_tokens.md#rotate-a-personal-access-token). +You can use the personal access tokens API to programmatically take action, +such as [rotating a personal access token](../api/personal_access_tokens.md#rotate-a-personal-access-token). + +You will receive an email when personal access tokens are 7 days or less from expiration. ## OAuth2 tokens @@ -55,6 +56,8 @@ You can use the [project access tokens API](../api/project_access_tokens.md) to programmatically take action, such as [rotating a project access token](../api/project_access_tokens.md#rotate-a-project-access-token). +All project maintainers receive an email when project access tokens are 7 days or less from expiration. + ## Group access tokens [Group access tokens](../user/group/settings/group_access_tokens.md#group-access-tokens) @@ -72,6 +75,8 @@ You can use the [group access tokens API](../api/group_access_tokens.md) to programmatically take action, such as [rotating a group access token](../api/group_access_tokens.md#rotate-a-group-access-token). +All group owners receive an email when group access tokens are 7 days or less from expiration. + ## Deploy tokens [Deploy tokens](../user/project/deploy_tokens/index.md) allow you to download (`git clone`) or push and pull packages and container registry images of a project without having a user and a password. Deploy tokens cannot be used with the GitLab API. @@ -86,39 +91,50 @@ This is useful, for example, for cloning repositories to your Continuous Integra Project maintainers and owners can add or enable a deploy key for a project repository -## Runner registration tokens (deprecated) +## Runner authentication tokens -WARNING: -The ability to pass a runner registration token has been [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) and is -planned for removal in 17.0, along with support for certain configuration arguments. This change is a breaking change. GitLab plans to introduce a new -[GitLab Runner token architecture](../architecture/blueprints/runner_tokens/index.md), which introduces -a new method for registering runners and eliminates the -runner registration token. +In GitLab 16.0 and later, you can use a runner authentication token to register +runners instead of a runner registration token. Runner registration tokens have +been [deprecated](../update/deprecations.md#registration-tokens-and-server-side-runner-arguments-in-gitlab-runner-register-command). -Runner registration tokens are used to [register](https://docs.gitlab.com/runner/register/) a [runner](https://docs.gitlab.com/runner/) with GitLab. Group or project owners or instance administrators can obtain them through the GitLab user interface. The registration token is limited to runner registration and has no further scope. +After you create a runner and its configuration, you receive a runner authentication token +that you use to register the runner. The runner authentication token is stored locally in the +[`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) file, which +you use to configure the runner. -You can use the runner registration token to add runners that execute jobs in a project or group. The runner has access to the project's code, so be careful when assigning project and group-level permissions. +The runner uses the runner authentication token to authenticate with GitLab when +it picks up jobs from the job queue. After the runner authenticates with GitLab, +the runner receives a [job token](../ci/jobs/ci_job_token.md), which it uses to +execute the job. -## Runner authentication tokens (also called runner tokens) +The runner authentication token stays on the runner machine. The execution environments +for the following executors only have access to the job token and not the runner authentication token: -Once created, the runner receives an authentication token, which it uses to authenticate with GitLab when picking up jobs from the job queue. The authentication token is stored locally in the runner's [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) file. +- Docker Machine +- Kubernetes +- VirtualBox +- Parallels +- SSH -After authentication with GitLab, the runner receives a [job token](../ci/jobs/ci_job_token.md), which it uses to execute the job. +Malicious access to a runner's file system may expose the `config.toml` file and the +runner authentication token. The attacker could use the runner authentication +to [clone the runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). -In case of Docker Machine/Kubernetes/VirtualBox/Parallels/SSH executors, the execution environment has no access to the runner authentication token, because it stays on the runner machine. They have access to the job token only, which is needed to execute the job. +You can use the `runners` API to +programmatically [rotate or revoke a runner authentication token](../api/runners.md#reset-runners-authentication-token-by-using-the-current-token). -Malicious access to a runner's file system may expose the `config.toml` file and thus the authentication token, allowing an attacker to [clone the runner](https://docs.gitlab.com/runner/security/#cloning-a-runner). +## Runner registration tokens (deprecated) -In GitLab 16.0 and later, you can use an authentication token to register runners instead of a -registration token. Runner registration tokens have been [deprecated](../update/deprecations.md#registration-tokens-and-server-side-runner-arguments-in-gitlab-runner-register-command). +WARNING: +The ability to pass a runner registration token has been [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) and is +planned for removal in GitLab 18.0, along with support for certain configuration arguments. This change is a breaking change. GitLab has implemented a new +[GitLab Runner token architecture](../ci/runners/new_creation_workflow.md), which introduces +a new method for registering runners and eliminates the +runner registration token. -To generate an authentication token, you create a runner in the GitLab UI and use the authentication token -instead of the registration token. +Runner registration tokens are used to [register](https://docs.gitlab.com/runner/register/) a [runner](https://docs.gitlab.com/runner/) with GitLab. Group or project owners or instance administrators can obtain them through the GitLab user interface. The registration token is limited to runner registration and has no further scope. -| Process | Registration command | -| ------------------ | --------------------- | -| Registration token (deprecated) | `gitlab-runner register --registration-token $RUNNER_REGISTRATION_TOKEN <runner configuration arguments>` | -| Authentication token | `gitlab-runner register --token $RUNNER_AUTHENTICATION_TOKEN` | +You can use the runner registration token to add runners that execute jobs in a project or group. The runner has access to the project's code, so be careful when assigning project and group-level permissions. ## CI/CD job tokens diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index 906bf4cd062..0c569e9843d 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -1,50 +1,55 @@ --- type: howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Enforce two-factor authentication **(FREE ALL)** -Two-factor authentication (2FA) provides an additional level of security to your -users' GitLab account. When enabled, users are prompted for a code generated by an application in -addition to supplying their username and password to sign in. +[Two-factor authentication (2FA)](../user/profile/account/two_factor_authentication.md) +is an authentication method that requires the user to provide two different factors +to prove their identity: + +- Username and password. +- A second authentication method, such as a code generated by an application. + +2FA makes it harder for an unauthorized person to access an account because +they would need both factors. NOTE: If you are [using and enforcing SSO](../user/group/saml_sso/index.md#sso-enforcement), you might already be enforcing 2FA on the identity provider (IDP) side. Enforcing 2FA on GitLab as well might be unnecessary. -Read more about [two-factor authentication (2FA)](../user/profile/account/two_factor_authentication.md). - ## Enforce 2FA for all users **(FREE SELF)** -Users on GitLab can enable it without any administrator's intervention. If you -want to enforce everyone to set up 2FA, you can choose from two different ways: +Administrators can enforce 2FA for all users in two different ways: + +- Enforce on next sign in. +- Suggest on next sign in, but allow a grace period before enforcing. -- Enforce on next login. -- Suggest on next login, but allow a grace period before enforcing. + After the configured grace period has elapsed, users can sign in but + cannot leave the 2FA configuration area at `/-/profile/two_factor_auth`. -After the configured grace period has elapsed, users can sign in but -cannot leave the 2FA configuration area at `/-/profile/two_factor_auth`. +You can use the UI or the API to enforce 2FA for all users. -To enable 2FA for all users: +### Use the UI -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. -1. Expand the **Sign-in restrictions** section, where you can configure both. +1. Expand the **Sign-in restrictions** section: + - Select **Enforce two-factor authentication** to enable this feature. + - In **Two-factor grace period**, enter a number of hours. If you want to + enforce 2FA on next sign-in attempt, enter `0`. -If you want 2FA enforcement to take effect during the next sign-in attempt, -change the grace period to `0`. +### Use the API -### Disable 2FA enforcement through Rails console +Use the [application settings API](../api/settings.md) to modify the following settings: -Using the [Rails console](../administration/operations/rails_console.md), enforcing 2FA for -all user can be disabled. Connect to the Rails console and run: +- `require_two_factor_authentication`. +- `two_factor_grace_period`. -```ruby -Gitlab::CurrentSettings.update!('require_two_factor_authentication': false) -``` +For more information, see the [list of settings that can be accessed through API calls](../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). ## Enforce 2FA for all users in a group **(FREE ALL)** @@ -56,59 +61,67 @@ Prerequisites: To enforce 2FA only for certain groups: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Select **All users in this group must set up two-factor authentication**. +1. Optional. In **Delay 2FA enforcement (hours)**, enter the number of hours you + want the grace period to last for. + If there are multiple different grace periods in a top level group and its subgroups + and projects, the shortest grace period is used. 1. Select **Save changes**. -You can also specify a grace period in the **Delay 2FA enforcement** option. - -If you want to enforce 2FA only for certain groups, you can enable it in the -group settings and specify a grace period as above. - -The following are important notes about 2FA: - -- Projects belonging to a 2FA-enabled group that - [is shared](../user/project/members/share_project_with_groups.md) - with a 2FA-disabled group will *not* require members of the 2FA-disabled group to use - 2FA for the project. For example, if project *P* belongs to 2FA-enabled group *A* and - is shared with 2FA-disabled group *B*, members of group *B* can access project *P* - without 2FA. To ensure this scenario doesn't occur, - [prevent sharing of projects](../user/group/access_and_permissions.md#prevent-a-project-from-being-shared-with-groups) - for the 2FA-enabled group. -- If you add additional members to a project within a group or subgroup that has - 2FA enabled, 2FA is **not** required for those individually added members. -- If there are multiple 2FA requirements (for example, group + all users, or multiple - groups) the shortest grace period is used. -- It is possible to prevent subgroups from setting up their own 2FA requirements: - 1. Go to the top-level group's **Settings > General**. - 1. Expand the **Permissions and group features** section. - 1. Uncheck the **Allow subgroups to set up their own two-factor authentication rule** field. - - This action causes all subgroups with 2FA requirements to stop requiring that from their members. -- Access tokens are not required to provide a second factor for authentication because they are API-based. - Tokens generated before 2FA is enforced remain valid. +Access tokens are not required to provide a second factor for authentication because +they are API-based. Tokens generated before 2FA is enforced remain valid. + +### 2FA in subgroups + +You can enable and enforce 2FA for individual subgroups in the same way as a top +level group. + +You can prevent subgroups from setting up their own 2FA requirements: + +1. Go to the top level group's **Settings > General**. +1. Expand the **Permissions and group features** section. +1. Clear the **Allow subgroups to set up their own two-factor authentication rule** checkbox. + +This action causes all subgroups with 2FA requirements to stop requiring 2FA from +their members. + +### 2FA in projects + +If a project belonging to a group that enables or enforces 2FA is [shared](../user/project/members/share_project_with_groups.md) +with a group that does not enable or enforce 2FA, members of the non-2FA group can access that project +without using 2FA. For example: + +- Group *A* has 2FA enabled and enforced. Group *B* does not have 2FA enabled. +- If a project, *P*, that belongs to group *A* is shared with group *B*, members + of group *B* can access project *P* without 2FA. + +To ensure this does not occur, [prevent sharing of projects](../user/group/access_and_permissions.md#prevent-a-project-from-being-shared-with-groups) +for the 2FA group. + +If you add members to a project in a group or subgroup that has 2FA +enabled, 2FA is **not** required for those individually added members. ## Disable 2FA **(FREE SELF)** +You can disable 2FA for a single user or all users. + +This is a permanent and irreversible action. Users must reactivate 2FA to use it again. + WARNING: Disabling 2FA for users does not disable the [enforce 2FA for all users](#enforce-2fa-for-all-users) or [enforce 2FA for all users in a group](#enforce-2fa-for-all-users-in-a-group) settings. You must also disable any enforced 2FA settings so users aren't asked to set up 2FA again when they next sign in to GitLab. -WARNING: -This is a permanent and irreversible action. Users must reactivate 2FA to use it again. - ### For a single user -To disable 2FA for non-administrator users, you should use the [API endpoint](../api/users.md#disable-two-factor-authentication) -instead of the Rails console. -Using the [Rails console](../administration/operations/rails_console.md), 2FA for a single user can be disabled. -Connect to the Rails console and run: +#### Administrators -**In GitLab 13.5 and later:** +In GitLab 13.5 and later, use the [Rails console](../administration/operations/rails_console.md) +to disable 2FA for a single administrator: ```ruby admin = User.find_by_username('<USERNAME>') @@ -117,20 +130,33 @@ user_to_disable = User.find_by_username('<USERNAME>') TwoFactor::DestroyService.new(admin, user: user_to_disable).execute ``` -The target user is notified that 2FA has been disabled. +The administrator is notified that 2FA has been disabled. + +#### Non-administrators + +In GitLab 15.2 and later, you can use either the Rails console or the +[API endpoint](../api/users.md#disable-two-factor-authentication) to disable 2FA +for a non-administrator. + +You can disable 2FA for your own account. + +You cannot use the API endpoint to disable 2FA for administrators. ### For all users -There may be some special situations where you want to disable 2FA for everyone -even when forced 2FA is disabled. There is a Rake task for that: +To disable 2FA for all users even when forced 2FA is disabled, use the following Rake task. -```shell -# Omnibus installations -sudo gitlab-rake gitlab:two_factor:disable_for_all_users +- For installations that use the Linux package: -# Installations from source -sudo -u git -H bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production -``` + ```shell + sudo gitlab-rake gitlab:two_factor:disable_for_all_users + ``` + +- For self-compiled installations: + + ```shell + sudo -u git -H bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production + ``` ## 2FA for Git over SSH operations **(PREMIUM ALL)** diff --git a/doc/security/unlock_user.md b/doc/security/unlock_user.md index 5e21cad8f3e..b2c8624b057 100644 --- a/doc/security/unlock_user.md +++ b/doc/security/unlock_user.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto @@ -25,7 +25,7 @@ If 2FA is enabled, users are locked after five failed sign-in attempts within 10 ## Unlock a user from the Admin Area -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Use the search bar to find the locked user. @@ -46,13 +46,13 @@ To unlock a locked user: sudo -u git -H bundle exec rails console -e production ``` -1. Find the user to unlock. You can search by email or ID. +1. Find the user to unlock. You can search by email: ```ruby user = User.find_by(email: 'admin@local.host') ``` - or + Or you can search by ID: ```ruby user = User.where(id: 1).first @@ -64,7 +64,7 @@ To unlock a locked user: user.unlock_access! ``` -1. Exit the console with <kbd>Control</kbd>+<kbd>d</kbd> +1. Exit the console with <kbd>Control</kbd>+<kbd>d</kbd>. The user should now be able to sign in. diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md index 899fed0b584..56445903d6c 100644 --- a/doc/security/user_email_confirmation.md +++ b/doc/security/user_email_confirmation.md @@ -1,6 +1,6 @@ --- type: howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -11,7 +11,7 @@ GitLab can be configured to require confirmation of a user's email address when the user signs up. When this setting is enabled, the user is unable to sign in until they confirm their email address. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Sign-up restrictions** and look for the **Email confirmation settings** options. diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index e0f1342b9c9..6ddda281a03 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -47,7 +47,7 @@ Prerequisite: To configure authentication settings for all media files: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Scroll to **Project visibility** and select **Require authentication to view media files**. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index 78c32341bf6..f8bd50bc4b3 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, reference, howto @@ -50,7 +50,7 @@ To prevent exploitation of insecure internal web services, all webhook and integ To allow access to these addresses: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. @@ -64,7 +64,7 @@ Prerequisite: [System hooks](../administration/system_hooks.md) can make requests to the local network by default. To prevent system hook requests to the local network: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. @@ -80,7 +80,7 @@ Prerequisite: To filter requests by blocking many requests: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. @@ -106,7 +106,7 @@ Prerequisite: To allow outbound requests to certain IP addresses and domains: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 93a68fa0338..3b2ef601136 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -76,9 +76,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Contribution Analytics](../user/group/contribution_analytics/index.md) - [Merge Request Analytics](../user/analytics/merge_request_analytics.md) - [Code Review Analytics](../user/analytics/code_review_analytics.md) - - [Audit Events](../administration/audit_events.md), including - [Group events](../administration/audit_events.md#group-events) and - [Project events](../administration/audit_events.md#project-events) + - [Audit Events](../administration/audit_events.md) - Rake tasks: - [Displaying GitLab license information](../administration/raketasks/maintenance.md#show-gitlab-license-information) - Reference Architecture information: diff --git a/doc/subscriptions/community_programs.md b/doc/subscriptions/community_programs.md index 26d221cf62d..a1d50dcb239 100644 --- a/doc/subscriptions/community_programs.md +++ b/doc/subscriptions/community_programs.md @@ -22,7 +22,7 @@ To meet GitLab for Open Source Program requirements, first add an OSI-approved o To add a license to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the overview page, select **Add LICENSE**. If the license you want is not available as a license template, manually copy the entire, unaltered [text of your chosen license](https://opensource.org/licenses/) into the `LICENSE` file. GitLab defaults to **All rights reserved** if users do not perform this action.  @@ -45,7 +45,7 @@ Benefits of the GitLab Open Source Program apply to all projects in a GitLab nam #### Screenshot 1: License overview -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select your project avatar. If you haven't specified an avatar for your project, the avatar displays as a single letter. 1. Take a screenshot of the project overview that clearly displays the license you've chosen for your project. @@ -53,7 +53,7 @@ Benefits of the GitLab Open Source Program apply to all projects in a GitLab nam #### Screenshot 2: License contents -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1.Select **Code > Repository** and locate the project's `LICENSE` file. 1. Take a screenshot of the contents of the file. Make sure the screenshot includes the title of the license. @@ -63,7 +63,7 @@ Benefits of the GitLab Open Source Program apply to all projects in a GitLab nam To be eligible for the GitLab Open Source Program, projects must be publicly visible. To check your project's public visibility settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. From the **Project visibility** dropdown list, select **Public**. diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index cb2a324bc9b..ff85c658c50 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -48,7 +48,7 @@ Prerequisite: To see the status of your GitLab SaaS subscription: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Billing**. The following information is displayed: @@ -80,10 +80,13 @@ Every user is included in seat usage, with the following exceptions: - Users who are pending approval. - Members with the [Guest role on an Ultimate subscription](#free-guest-users). +- Members with the [minimal access role](../../user/permissions.md#users-with-minimal-access). +- [Banned members](../../user/group/moderate_users.md#ban-a-user). +- [Blocked users](../../administration/moderate_users.md#block-a-user). - GitLab-created service accounts: - [Ghost User](../../user/profile/account/delete_account.md#associated-records). - Bots such as: - - [Support Bot](../../user/project/service_desk/index.md#support-bot-user). + - [Support Bot](../../user/project/service_desk/configure.md#support-bot-user). - [Bot users for projects](../../user/project/settings/project_access_tokens.md#bot-users-for-projects). - [Bot users for groups](../../user/group/settings/group_access_tokens.md#bot-users-for-groups). @@ -99,7 +102,7 @@ In this case, they would see only the features available to that subscription. To view a list of seats being used: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Usage Quotas**. 1. On the **Seats** tab, view usage information. @@ -113,7 +116,7 @@ The counts for **Max seats used** and **Seats owed** are updated once per day. To view your subscription information and a summary of seat counts: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Billing**. The usage statistics are updated once per day, which may cause @@ -141,7 +144,7 @@ For example: To export seat usage data as a CSV file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Billing**. 1. Under **Seats currently in use**, select **See usage**. 1. Select **Export list**. @@ -185,13 +188,13 @@ GitLab [bills you for the overage](../quarterly_reconciliation.md). To add seats to a subscription: 1. Log in to the [Customers Portal](https://customers.gitlab.com/). -1. Navigate to the **Manage Purchases** page. +1. Go to the **Manage Purchases** page. 1. Select **Add more seats** on the relevant subscription card. 1. Enter the number of additional users. -1. Select **Proceed to checkout**. -1. Review the **Subscription Upgrade Detail**. The system lists the total price for all users on the +1. Review the **Purchase summary** section. The system lists the total price for all users on the system and a credit for what you've already paid. You are only be charged for the net change. -1. Select **Confirm Upgrade**. +1. Enter your payment information. +1. Select **Purchase seats**. The following is emailed to you: @@ -202,7 +205,7 @@ The following is emailed to you: To remove a billable user from your subscription: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Billing**. 1. In the **Seats currently in use** section, select **See usage**. 1. In the row for the user you want to remove, on the right side, select the ellipsis and **Remove user**. @@ -241,10 +244,11 @@ amounts at which the alert displays. To change the namespace linked to a subscription: -1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) with a +1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) with a [linked](../customers_portal.md#link-a-gitlabcom-account) GitLab.com account. -1. Go to the **Manage Purchases** page. -1. Select **Change linked namespace**. +1. Do one of the following: + - If the subscription is not linked to a namespace, select **Link subscription to a group**. + - If the subscription is already linked to a namespace, select **Subscription actions** (**{ellipsis_v}**) > **Change linked group**. 1. Select the desired group from the **New Namespace** dropdown list. For a group to appear here, you must have the Owner role for that group. 1. If the [total number of users](#view-seat-usage) in your group exceeds the number of seats in your subscription, you are prompted to pay for the additional users. Subscription charges are calculated based on @@ -258,18 +262,17 @@ To change the namespace linked to a subscription: 1. Select **Confirm changes**. -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a demo, see [Linking GitLab Subscription to the Namespace](https://youtu.be/qAq8pyFP-a0). - Only one namespace can be linked to a subscription. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a demo, see [Linking GitLab Subscription to the Namespace](https://youtu.be/8iOsN8ajBUw). + ## Upgrade your GitLab SaaS subscription tier To upgrade your [GitLab tier](https://about.gitlab.com/pricing/): -1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. Select **Upgrade** on the relevant subscription card on the - [Manage purchases](https://customers.gitlab.com/subscriptions) page. +1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. Select **Upgrade** on the relevant subscription card. 1. Select the desired upgrade. 1. Confirm the active form of payment, or add a new form of payment. 1. Check the **I accept the Privacy Policy and Terms of Service** checkbox. @@ -305,12 +308,12 @@ Before you renew your subscription: Starting 30 days before a subscription expires, GitLab notifies group owners of the date of expiry with a banner in the GitLab user interface. - +You can only renew your subscription 15 days before it is due to expire. To renew your subscription: -1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select **Renew**. -The **Renew** button remains disabled (grayed-out) until 15 days before a subscription expires. -You can hover your mouse on the **Renew** button to see the date when it becomes active. +1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select **Renew**. +The **Renew** button displays only 15 days before a subscription expires. If there are more than 15 days before +the subscription expires, select **Subscription actions** (**{ellipsis_v}**), then select **Renew subscription** to view the date when you can renew. 1. Review your renewal details and complete the payment process. 1. Select **Confirm purchase**. @@ -340,10 +343,8 @@ expiration date without a gap in available service. Subscriptions purchased thro To view or change automatic subscription renewal (at the same tier as the previous period), sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in), and: -- If a **Turn on auto-renew** button is displayed, your subscription was canceled - previously. Select it to resume automatic renewal. -- If a **Cancel subscription** button is displayed, your subscription is set to automatically - renew at the end of the subscription period. Select it to cancel automatic renewal. +- If the subscription card displays `Expires on DATE`, your subscription is not set to automatically renew. To enable automatic renewal, in **Subscription actions** (**{ellipsis_v}**), select **Turn on auto-renew**. +- If the subscription card displays `Autorenews on DATE`, your subscription is set to automatically renew at the end of the subscription period. To cancel automatic renewal, in **Subscription actions** (**{ellipsis_v}**), select **Cancel subscription**. If you have difficulty during the renewal process, contact the [Support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. @@ -430,7 +431,7 @@ main quota. You can find pricing for additional storage on the To purchase additional storage for your group on GitLab SaaS: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Usage Quotas**. 1. Select **Storage** tab. 1. Select **Purchase more storage**. diff --git a/doc/subscriptions/gitlab_dedicated/index.md b/doc/subscriptions/gitlab_dedicated/index.md index a2e04812876..fd20e353056 100644 --- a/doc/subscriptions/gitlab_dedicated/index.md +++ b/doc/subscriptions/gitlab_dedicated/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Dedicated is a fully isolated, single-tenant SaaS service that is: - Hosted and managed by GitLab, Inc. -- Deployed on AWS in a cloud region of your choice (see the [regions that are not supported](#aws-regions-not-supported)). +- Deployed on AWS in a cloud region of your choice. See [unavailable AWS regions](#unavailable-aws-regions). GitLab Dedicated removes the overhead of platform management to increase your operational efficiency, reduce risk, and enhance the speed and agility of your organization. Each GitLab Dedicated instance is highly available with disaster recovery and deployed into the cloud region of your choice. GitLab teams fully manage the maintenance and operations of each isolated instance, so customers can access our latest product improvements while meeting the most complex compliance standards. @@ -19,7 +19,7 @@ It's the offering of choice for enterprises and organizations in highly regulate ### Data residency -GitLab Dedicated allows you to select the cloud region where your data will be stored. Upon [onboarding](../../administration/dedicated/index.md#onboarding), choose the cloud region where you want to deploy your Dedicated instance. Some AWS regions have limited features and as a result, we are not able to deploy production instances to those regions. See below for the [full list of regions](#aws-regions-not-supported) not supported. +GitLab Dedicated allows you to select the cloud region where your data will be stored. Upon [onboarding](../../administration/dedicated/index.md#onboarding), choose the cloud region where you want to deploy your Dedicated instance. Some AWS regions have limited features and as a result, we are not able to deploy production instances to those regions. See below for the [list of unavailable AWS regions](#unavailable-aws-regions). ### Availability and scalability @@ -29,7 +29,7 @@ GitLab Dedicated leverages the GitLab [Cloud Native Hybrid reference architectur When [onboarding](../../administration/dedicated/index.md#onboarding) to GitLab Dedicated, you can provide a Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster. Regular backups of all GitLab Dedicated datastores (including Database and Git repositories) are taken and tested regularly and stored in your desired secondary region. GitLab Dedicated also provides the ability to store copies of these backups in a separate cloud region of choice for greater redundancy. -For more information, read about the [recovery plan for GitLab Dedicated](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/#disaster-recovery-plan) as well as RPO and RTO targets. +For more information, read about the [recovery plan for GitLab Dedicated](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/#disaster-recovery-plan) as well as RPO and RTO targets. These targets are available only when both the primary and secondary regions are supported by GitLab Dedicated. See below for a [list of unavailable AWS regions](#unavailable-aws-regions) for GitLab Dedicated. ### Security @@ -129,7 +129,8 @@ The following GitLab application features are not available: The following features will not be supported: - Mattermost -- Server-side Git hooks. Use [push rules](../../user/project/repository/push_rules.md) instead. +- [Server-side Git hooks](../../administration/server_hooks.md). + GitLab Dedicated is a SaaS service, and access to the underlying infrastructure is only available to GitLab Inc. team members. Due to the nature of server side configuration, there is a possible security concern of running arbitrary code on Dedicated services, as well as the possible impact that can have on the service SLA. Use the alternative [push rules](../../user/project/repository/push_rules.md) or [webhooks](../../user/project/integrations/webhooks.md) instead. ### GitLab Dedicated service features @@ -143,9 +144,12 @@ The following operational features are not available: - Observability Dashboard using Switchboard - Pre-Production Instance -### AWS regions not supported +### Unavailable AWS regions -The following AWS regions are not available: +The following is an incomplete list of unavailable AWS regions. Regions must support `io2` volumes and meet other requirements. GitLab team members can view our evaluation of additional +regions in this confidential issue: `https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/1405`. + +Contact [GitLab Support](https://about.gitlab.com/support/) if you have any questions. - Jakarta (`ap-southeast-3`) - Bahrain (`me-south-1`) @@ -156,6 +160,15 @@ The following AWS regions are not available: - Zurich (`eu-central-2`) - GovCloud (US-East) (`us-gov-east-1`) - GovCloud (US-West) (`us-gov-west-1`) +- Melbourne (`ap-southeast-4`) +- Hyderabad (`ap-south-2`) +- Osaka (`ap-northeast-3`) +- Beijing (`cn-north-1`) +- Ningxia (`cn-northwest-1`) +- Spain (`eu-south-2`) +- Tel Aviv (`il-central-1`) +- UAE (`me-central-1`) +- São Paulo (`sa-east-1`) ## Planned features diff --git a/doc/subscriptions/quarterly_reconciliation.md b/doc/subscriptions/quarterly_reconciliation.md index 7e7cc93e284..1962ce02647 100644 --- a/doc/subscriptions/quarterly_reconciliation.md +++ b/doc/subscriptions/quarterly_reconciliation.md @@ -79,6 +79,15 @@ seats, and an invoice is generated for a prorated amount. If a credit card is on file, a payment is automatically applied. Otherwise, an invoice is sent and subject to your payment terms. +### Troubleshooting failed payment + +If your credit card is declined during the reconciliation process, an email will be sent with the subject `Your GitLab subscription failed to reconcile`. Please follow these instructions to update your payment information, and the reconciliation will be automatically retried: + +1. Log in to your account at `https://customers.gitlab.com`. +1. Go to **Payment Methods**. +1. Select **Add New Payment Method**. +1. Make sure that the payment method is set as **Default**. + ## Quarterly reconciliation eligibility ### You are automatically enrolled in quarterly reconciliation if diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index fca6ea57a95..ea6b87134ac 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -36,7 +36,7 @@ Prorated charges are not possible without a quarterly usage report. You can view users for your license and determine if you've gone over your subscription. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Users**. @@ -57,7 +57,7 @@ billable user, with the following exceptions: - GitLab-created service accounts: - [Ghost User](../../user/profile/account/delete_account.md#associated-records). - Bots such as: - - [Support Bot](../../user/project/service_desk/index.md#support-bot-user). + - [Support Bot](../../user/project/service_desk/configure.md#support-bot-user). - [Bot users for projects](../../user/project/settings/project_access_tokens.md#bot-users-for-projects). - [Bot users for groups](../../user/group/settings/group_access_tokens.md#bot-users-for-groups). - Other [internal users](../../development/internal_users.md#internal-users). @@ -217,7 +217,7 @@ Example of a license sync request: You can manually synchronize your subscription details at any time. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. 1. In the **Subscription details** section, select **Sync subscription details**. @@ -228,7 +228,7 @@ A job is queued. When the job finishes, the subscription details are updated. If you are an administrator, you can view the status of your subscription: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. @@ -253,7 +253,7 @@ It also displays the following information: If you are an administrator, you can export your license usage into a CSV: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Subscription**. 1. In the upper-right corner, select **Export license usage file**. diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 6137ade4559..8efa5afcb80 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -30,7 +30,6 @@ This page gathers all the resources for the topic **Authentication** in GitLab. - **Integrations:** - [OmniAuth](../../integration/omniauth.md) - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) diff --git a/doc/topics/autodevops/cicd_variables.md b/doc/topics/autodevops/cicd_variables.md index a5897e0b233..21d9dd0b3d3 100644 --- a/doc/topics/autodevops/cicd_variables.md +++ b/doc/topics/autodevops/cicd_variables.md @@ -136,7 +136,7 @@ Prerequisite: To configure secret variables: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. 1. Create a CI/CD variable with the prefix `K8S_SECRET_`. For example, you diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md index af396e159da..6ac8f87940e 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md @@ -13,7 +13,7 @@ You can choose to target AWS ECS as a deployment platform instead of using Kuber To get started on Auto DevOps to AWS ECS, you must add a specific CI/CD variable. To do so, follow these steps: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Specify which AWS platform to target during the Auto DevOps deployment diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md index 53b20f880a9..d0a7814348c 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md @@ -132,7 +132,7 @@ While Auto DevOps is enabled by default, Auto DevOps can be disabled at both the instance level (for self-managed instances) and the group level. Complete these steps to enable Auto DevOps if it's disabled: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the application project. +1. On the left sidebar, select **Search or go to** and find the application project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select **Default to Auto DevOps pipeline** to display more options. diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md index a939fe108a1..d50bdcfa056 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md @@ -136,7 +136,7 @@ While Auto DevOps is enabled by default, Auto DevOps can be disabled at both the instance level (for self-managed instances) and the group level. Complete these steps to enable Auto DevOps if it's disabled: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the application project. +1. On the left sidebar, select **Search or go to** and find the application project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select **Default to Auto DevOps pipeline** to display more options. diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 0668ea0df14..4191ba257ca 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -101,7 +101,7 @@ Prerequisites: To enable Auto DevOps for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select the **Default to Auto DevOps pipeline** checkbox. @@ -132,7 +132,7 @@ Prerequisites: To enable Auto DevOps for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select the **Default to Auto DevOps pipeline** checkbox. @@ -144,7 +144,7 @@ clear the **Default to Auto DevOps pipeline** checkbox. After enabling Auto DevOps at the group level, you can trigger the Auto DevOps pipeline for any project that belongs to that group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Make sure the project doesn't contain a `.gitlab-ci.yml` file. 1. Select **Build > Pipelines**. 1. To trigger the Auto DevOps pipeline, select **Run pipeline**. @@ -164,7 +164,7 @@ Prerequisites: To enable Auto DevOps for your instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. diff --git a/doc/topics/autodevops/prepare_deployment.md b/doc/topics/autodevops/prepare_deployment.md index c5a758e5d70..1dced373461 100644 --- a/doc/topics/autodevops/prepare_deployment.md +++ b/doc/topics/autodevops/prepare_deployment.md @@ -43,7 +43,7 @@ To define the base domain, either: - In the project, group, or instance level: go to your cluster settings and add it there. - In the project or group level: add it as an environment variable: `KUBE_INGRESS_BASE_DOMAIN`. -- In the instance level: go to **Main menu > Admin > Settings > CI/CD > Continuous Integration and Delivery** and add it there. +- In the instance level: go to the Admin Area, then **Settings > CI/CD > Continuous Integration and Delivery** and add it there. The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence as other environment [variables](../../ci/variables/index.md#cicd-variable-precedence). diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 47d79fcfc6b..a0aaba99a59 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -41,7 +41,7 @@ that works best for your needs: You can choose the deployment method when enabling Auto DevOps or later: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Choose the deployment strategy. @@ -60,7 +60,7 @@ To define the base domain, either: - In the project, group, or instance level: go to your cluster settings and add it there. - In the project or group level: add it as an environment variable: `KUBE_INGRESS_BASE_DOMAIN`. -- In the instance level: go to **Main menu > Admin > Settings > CI/CD > Continuous Integration and Delivery** and add it there. +- In the instance level: go to the Admin Area, then **Settings > CI/CD > Continuous Integration and Delivery** and add it there. The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of [precedence as other environment variables](../../ci/variables/index.md#cicd-variable-precedence). diff --git a/doc/topics/git/how_to_install_git/index.md b/doc/topics/git/how_to_install_git/index.md index 40c5147e20b..52e8e2e1259 100644 --- a/doc/topics/git/how_to_install_git/index.md +++ b/doc/topics/git/how_to_install_git/index.md @@ -34,16 +34,16 @@ Prerequisites: To install Git on macOS: -1. Open a terminal and install the XCode Command Line Tools: +1. Open a terminal and install Xcode Command Line Tools: ```shell xcode-select --install ``` - Alternatively, you can install the entire [XCode](https://developer.apple.com/xcode/) + Alternatively, you can install the entire [Xcode](https://developer.apple.com/xcode/) package through the macOS App Store. -1. Select **Install** to download and install XCode Command Line Tools. +1. Select **Install** to download and install Xcode Command Line Tools. 1. Install Homebrew according to the [official Homebrew installation instructions](https://brew.sh/index.html). 1. Install Git by running `brew install git` from your terminal. 1. In a terminal, verify that Git works on your computer: diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index 05b14b21f20..e349cf0bb92 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -8,104 +8,116 @@ type: index # Git **(FREE ALL)** Git is a [free and open source](https://git-scm.com/about/free-and-open-source) -distributed version control system designed to handle everything from small to -large projects with speed and efficiency. +distributed version control system. It handles projects of all sizes quickly and +efficiently, while providing support for rolling back changes when needed. -[GitLab](https://about.gitlab.com) is a Git-based fully integrated platform for -software development. Besides Git functionalities, GitLab has a lot of -powerful [features](https://about.gitlab.com/features/) to enhance your -[workflow](https://about.gitlab.com/topics/version-control/what-is-gitlab-flow/). +GitLab is built on top of (and with) Git, and provides you a Git-based, fully-integrated +platform for software development. GitLab adds many powerful +[features](https://about.gitlab.com/features/) on top of Git to enhance your workflow. -We've gathered some resources to help you to get the best from Git with GitLab. +These resources can help you to get the best from using Git with GitLab. -More information is also available on the [Git website](https://git-scm.com). +## Learn about Git -## Getting started +New to Git? These resources can help you understand basic Git concepts before +you dive in: -The following resources can help you get started with Git: - -- [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4), - a video introduction to Git. -- [Make your first Git commit](../../tutorials/make_first_git_commit/index.md) -- [Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) -- [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) -- [How to install Git](how_to_install_git/index.md) - [Git concepts](terminology.md) -- [Start using Git on the command line](../../gitlab-basics/start-using-git.md) -- [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) -- Commits: - - [Revert a commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) - - [Cherry-picking a commit](../../user/project/merge_requests/cherry_pick_changes.md) - - [Squash-and-merge](../../user/project/merge_requests/squash_and_merge.md) - - [Signing commits](../../user/project/repository/gpg_signed_commits/index.md) -- [Git stash](stash.md) -- [Git file blame](../../user/project/repository/git_blame.md) -- [Git file history](../../user/project/repository/git_history.md) -- [Git tags](../../user/project/repository/tags/index.md) - -### Concepts - -The following are resources on version control concepts: - -- [Why Git is Worth the Learning Curve](https://about.gitlab.com/blog/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) -- [The future of SaaS hosted Git repository pricing](https://about.gitlab.com/blog/2016/05/11/git-repository-pricing/) -- [Git website on version control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) -- [GitLab University presentation about Version Control](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit?usp=sharing) - -### Work with Git on the command line - -You can do many Git tasks from the command line: - -- [Cherry-pick](cherry_picking.md). -- [Getting started with Git](../../tutorials/make_first_git_commit/index.md). -- [Git add](git_add.md). -- [Git stash](stash.md). -- [Rollback commits](rollback_commits.md). -- [Unstage](unstage.md). - -## Git tips - -The following resources may help you become more efficient at using Git: +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> + Video tutorial: [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4) +- PDF download: [GitLab Git Cheat Sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) -- [Useful Git commands](useful_git_commands.md) collected by the GitLab support team. -- [Git Tips & Tricks](https://about.gitlab.com/blog/2016/12/08/git-tips-and-tricks/) -- [Eight Tips to help you work better with Git](https://about.gitlab.com/blog/2015/02/19/8-tips-to-help-you-work-better-with-git/) +The official Git documentation also offers information on +[Git basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics). -## Troubleshooting Git +## Begin using Git -If you have problems with Git, the following may help: +After you learn how Git works, you're ready to try it out. These resources are +appropriate for when you're ready to start learning Git by doing: -- [Numerous _undo_ possibilities in Git](numerous_undo_possibilities_in_git/index.md) -- Learn a few [Git troubleshooting](troubleshooting_git.md) techniques - -## Branching strategies +- [How to install Git](how_to_install_git/index.md) +- [Start using Git on the command line](../../gitlab-basics/start-using-git.md) +- Tutorial: [Make your first Git commit](../../tutorials/make_first_git_commit/index.md) +- Tutorial: [How to update Git commit messages](../../tutorials/update_commit_messages/index.md) +- The [GitLab CLI](https://gitlab.com/gitlab-org/cli/) + +A typical Git user encounters these concepts soon after starting to use Git: + +- [`git add`](git_add.md) to start tracking files with Git. +- [Tags](../../user/project/repository/tags/index.md) and + [branches](../../user/project/repository/branches/index.md). +- [How to undo mistakes](numerous_undo_possibilities_in_git/index.md), + including [`git reset`](rollback_commits.md). +- View a chronological list of changes to a file with + [Git history](../../user/project/repository/git_history.md). +- View a line-by-line editing history of a file with + [`git blame`](../../user/project/repository/git_blame.md). +- [Sign commits](../../user/project/repository/signed_commits/gpg.md) + for increased accountability and trust. + +## Learn more complex commands + +When you're comfortable with basic Git commands, you're ready to dive into the +more complex features of Git. These commands aren't required when creating +straightforward changes. When you begin managing multiple branches or need more complex +change management, you're ready for these features: + +- To stop tracking changes to a file, because you don't want to commit them, + [unstage the changes](unstage.md). +- [Stash your changes](stash.md) when your current work isn't ready to create a commit locally, + but you need to switch branches to work on something else. +- If you create many small commits locally, you can use + [squash and merge](../../user/project/merge_requests/squash_and_merge.md) + to combine them into fewer commits before pushing them. +- [Cherry-pick](../../user/project/merge_requests/cherry_pick_changes.md) the contents + of a commit from one branch to another. +- [Revert an existing commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) + if it contains changes you no longer want. + +## Learn branching and workflow strategies + +When you're comfortable with the creation and handling of individual branches, +you're ready to learn about Git workflows and branching strategies: - [Feature branch workflow](../../gitlab-basics/feature_branch_workflow.md) -- [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) -- [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) - -## Advanced use - -The following are advanced topics for those who want to get the most out of Git: - - [Introduction to Git rebase, force-push, and merge conflicts](git_rebase.md) -- [Server Hooks](../../administration/server_hooks.md) -- [Git Attributes](../../user/project/git_attributes.md) -- Git Submodules: [Using Git submodules with GitLab CI](../../ci/git_submodules.md) -- [Partial Clone](partial_clone.md) - -## API - -[Gitignore templates](../../api/templates/gitignores.md) API allow for -Git-related queries from GitLab. - -## Git Large File Storage (LFS) - -The following relate to Git Large File Storage: - -- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [Migrate an existing Git repository with Git LFS](lfs/migrate_to_git_lfs.md) -- [Removing objects from LFS](lfs/index.md#removing-objects-from-lfs) -- [GitLab Git LFS user documentation](lfs/index.md) -- [GitLab Git LFS administrator documentation](../../administration/lfs/index.md) -- [Towards a production quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) +- From the official Git documentation: + - [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) + - [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) + +## Learn advanced topics in Git management + +Git and GitLab, combined together, provide advanced features for repository management: + +- Enforce commit policies and run tasks with [Git server hooks](../../administration/server_hooks.md). +- Define which file types to treat as binary, and set the languages to use for + syntax highlighting with [the `.gitattributes` file](../../user/project/git_attributes.md). +- To keep a Git repository as a subdirectory in another repository, + [use Git submodules with GitLab CI](../../ci/git_submodules.md). +- When working with extremely large repositories, you can use a [partial clone](partial_clone.md) + of a repository instead of a complete clone. +- GitLab APIs for [`.gitignore` files](../../api/templates/gitignores.md), + [commits](../../api/commits.md), [tags](../../api/tags.md), + and [repositories](../../api/repositories.md). + +### Git Large File Storage (LFS) + +Many Git projects must manage large binary assets, such as videos and images. +Implementing Git Large File Storage can help manage these assets while keeping +your repository small: + +- [User documentation](lfs/index.md) for Git LFS at GitLab +- [Administrator documentation](../../administration/lfs/index.md) for Git LFS at GitLab +- Blog post: [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) +- [Migrate an existing Git repository](lfs/migrate_to_git_lfs.md) with Git LFS +- [Remove objects](lfs/index.md#removing-objects-from-lfs) from Git LFS +- Blog post: [Towards a production-quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) + +## Related topics + +- Official [Git documentation](https://git-scm.com), including + [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) +- [Git troubleshooting](troubleshooting_git.md) techniques +- [Git commands](useful_git_commands.md) collected by the GitLab support team +- Blog post: [Git Tips & Tricks](https://about.gitlab.com/blog/2016/12/08/git-tips-and-tricks/) +- Blog post: [Eight Tips to help you work better with Git](https://about.gitlab.com/blog/2015/02/19/8-tips-to-help-you-work-better-with-git/) diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index d9c906ccd81..23d972e9aa7 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -32,7 +32,7 @@ Prerequisites: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand the **Visibility, project features, permissions** section. 1. Turn on the **Git Large File Storage (LFS)** toggle. @@ -299,3 +299,23 @@ You might choose to do this if you are using an appliance like a Nexus Repositor GitLab can't verify LFS objects. Pushes then fail if you have GitLab LFS support enabled. To stop push failure, LFS support can be disabled in the [Project settings](../../../user/project/settings/index.md), which also disables GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). + +### I/O timeout when pushing LFS objects + +You might get an error that states: + +```shell +LFS: Put "http://your-instance.com/root/project.git/gitlab-lfs/objects/cc29e205d04a4062d0fb131700e8bfc8e54c44d0176a8dca22f40b24ef26d325/15": read tcp your-instance-ip:54544->your-instance-ip:443: i/o timeout +error: failed to push some refs to 'ssh://your-instance.com:2222/root/project.git' +``` + +When network conditions are unstable, the Git LFS client might time out when trying to upload files +if network conditions are unstable. + +The workaround is to set the client activity timeout a higher value. + +For example, to set the timeout to 60 seconds: + +```shell +git config lfs.activitytimeout 60 +``` diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index 43d5d746539..0cc7259a104 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -197,7 +197,7 @@ The root causes vary, so multiple potential solutions exist, and you may need to apply more than one: - If this error occurs when cloning a large repository, you can - [decrease the cloning depth](../../ci/large_repositories/index.md#shallow-cloning) + [decrease the cloning depth](../../user/project/repository/managing_large_repositories.md#shallow-cloning) to a value of `1`. For example: ```shell diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 82a88b53dcf..301f73a268d 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -217,7 +217,9 @@ always fails because it uses `pool.ntp.org`. This error can be ignored but you c ## Enabling the Package Metadata Database -Enabling the Package Metadata Database is required to enable [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files). +Enabling the Package Metadata Database is required to enable +[Continuous Vulnerability Scanning](../../user/application_security/continuous_vulnerability_scanning/index.md) +and [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files/index.md). This process requires the use of License and/or Advisory Data under what is collectively called the Package Metadata Database, which is licensed under the [EE License](https://storage.googleapis.com/prod-export-license-bucket-1a6c642fc4de57d4/LICENSE). Note the following in relation to use of the Package Metadata Database: @@ -225,8 +227,6 @@ Note the following in relation to use of the Package Metadata Database: - The Package Metadata Database may contain links to third-party websites or resources. We provide these links only as a convenience and are not responsible for any third-party data, content, products, or services from those websites or resources or links displayed on such websites. - The Package Metadata Database is based in part on information made available by third parties, and GitLab is not responsible for the accuracy or completeness of content made available. -Enabling the Package Metadata Database is also required to enable Continuous Vulnerability Scans for Dependency Scanning (see [epic 9534](https://gitlab.com/groups/gitlab-org/-/epics/9534) tracking this work for more info). - Package metadata is stored in the following Google Cloud Provider (GCP) buckets: - License Scanning - prod-export-license-bucket-1a6c642fc4de57d4 @@ -351,3 +351,29 @@ The directory for package metadata changed with the release of 16.2 from `vendor ```shell sed -i '.bckup' -e 's#vendor/package_metadata_db#vendor/package_metadata/licenses#g' [FILE ...] ``` + +### Troubleshooting + +#### Missing database data + +If license or advisory data is missing from the dependency list or MR pages, one possible cause of this is that the database has not been synchronized with the export data. + +`package_metadata` synchronization is triggered by using cron jobs ([advisory sync](https://gitlab.com/gitlab-org/gitlab/-/blob/16-3-stable-ee/config/initializers/1_settings.rb#L864-866) and [license sync](https://gitlab.com/gitlab-org/gitlab/-/blob/16-3-stable-ee/config/initializers/1_settings.rb#L855-857)) and imports only the package registry types enabled in [admin settings](../../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync). + +The file structure in `vendor/package_metadata` must coincide with the package registry type enabled above. For example, to sync `maven` license or advisory data, the package metadata directory under the Rails directory must have the following structure: + +- For licenses:`$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses/v2/maven/**/*.ndjson`. +- For advisories:`$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/advisories/v2/maven/**/*.ndjson`. + +After a successful run, data under the `pm_` tables in the database should be populated (check using [Rails console](../../administration/operations/rails_console.md)): + +- For licenses: `sudo gitlab-rails runner "puts \"Package model has #{PackageMetadata::Package.where(purl_type: 'maven').size} packages\""` +- For advisories: `sudo gitlab-rails runner "puts \"Advisory model has #{PackageMetadata::AffectedPackage.where(purl_type: 'maven').size} packages\""` + +Additionally, checkpoint data should exist for the particular package registry being synchronized. For Maven, for example, there should be a checkpoint created after a successful sync run: + +- For licenses: `sudo gitlab-rails runner "puts \"maven data has been synced up to #{PackageMetadata::Checkpoint.where(data_type: 'licenses', purl_type: 'maven')}\""` +- For advisories: `sudo gitlab-rails runner "puts \"maven data has been synced up to #{PackageMetadata::Checkpoint.where(data_type: 'advisories', purl_type: 'maven')}\""` + +Finally, you can check the [`application_json.log`](../../administration/logs/index.md#application_jsonlog) logs to verify that the +sync job has run and is without error by searching for `DEBUG` messages where the class is `PackageMetadata::SyncService`. Example: `{"severity":"DEBUG","time":"2023-06-22T16:41:00.825Z","correlation_id":"a6e80150836b4bb317313a3fe6d0bbd6","class":"PackageMetadata::SyncService","message":"Evaluating data for licenses:gcp/prod-export-license-bucket-1a6c642fc4de57d4/v2/pypi/1694703741/0.ndjson"}`. diff --git a/doc/tutorials/automate_runner_creation/index.md b/doc/tutorials/automate_runner_creation/index.md new file mode 100644 index 00000000000..fa1373f1e3a --- /dev/null +++ b/doc/tutorials/automate_runner_creation/index.md @@ -0,0 +1,220 @@ +--- +stage: none +group: Tutorials +info: For assistance with this tutorial, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. +--- + +# Tutorial: Automate runner creation and registration **(FREE ALL)** + +This tutorial describes how to automate runner creation and registration. + +To automate runner creation and registration: + +1. [Create a personal access token](#create-a-personal-access-token). +1. [Create a runner configuration](#create-a-runner-configuration). +1. [Automate GitLab Runner installation and registration](#automate-runner-installation-and-registration). +1. [View runners with the same configuration](#view-runners-with-the-same-configuration). + +NOTE: +The instructions in this tutorial describe runner creation and registration +with runner authentication tokens, which have replaced the deprecated registration +method that uses registration tokens. For more information, see +[The new runner registration workflow](../../ci/runners/new_creation_workflow.md#the-new-runner-registration-workflow). + +## Prerequisites + +- GitLab Runner must be installed on your GitLab instance. +- To create shared runners, you must be an administrator. +- To create group runners, you must be an administrator or have the Owner role for the group. +- To create project runners, you must be an administrator or have the Maintainer role for the project. + +## Create an access token + +Create an access token so that you can use the REST API to create runners. + +You can create: + +- A personal access token to use with shared, group, and project runners. +- A group or project access token to use with group and project runners. + +The access token is only visible once in the GitLab UI. After you leave the page, +you no longer have access to the token. You should use a secrets management solution +to store the token, like HashiCorp Vault or the Keeper Secrets Manager Terraform plugin. + +### Create a personal access token + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. On the left sidebar, select **Access Tokens**. +1. Select **Add new token**. +1. Enter a name and expiry date for the token. + - The token expires on that date at midnight UTC. + - If you do not enter an expiry date, the expiry date is automatically set to 365 days later than the current date. + - By default, this date can be a maximum of 365 days later than the current date. +1. In the **Select scopes** section, select the **create_runner** checkbox. +1. Select **Create personal access token**. + +### Create a project or group access token + +WARNING: +Project access tokens are treated as [internal users](../../development/internal_users.md). +If an internal user creates a project access token, that token is able to access +all projects that have visibility level set to [Internal](../../user/public_access.md). + +To create a project access token: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Settings > Access Tokens**. +1. Select **Add new token** +1. Enter a name. The token name is visible to any user with permissions to view + the group or project. +1. Enter an expiry date for the token. + - The token expires on that date at midnight UTC. + - If you do not enter an expiry date, the expiry date is automatically set + to 365 days later than the current date. + - By default, this date can be a maximum of 365 days later than the current date. + - An instance-wide [maximum lifetime](../../administration/settings/account_and_limit_settings.md#limit-the-lifetime-of-access-tokens) + setting can limit the maximum allowable lifetime on self-managed instances. +1. From the **Select a role** dropdown list: + - For the project access token, select **Maintainer**. + - For the group access token, select **Owner**. +1. In the **Select scopes** section, select the **create_runner** checkbox. +1. Select **Create project access token**. + +## Create a runner configuration + +A runner configuration is where you configure runners to your requirements. + +After you create a runner configuration, you receive a runner authentication +to register the runner. One or many runners can be linked to the +same configuration when these runners are registered with the same runner authentication +token. The runner configuration is stored in the `config.toml` file. + +To create a runner configuration, you can use: + +- The GitLab REST API. +- The `gitlab_user_runner` Terraform resource. + +### With the GitLab REST API + +Prerequisites: + +- The URL for your GitLab instance. For example, if your project is hosted on + `gitlab.example.com/yourname/yourproject`, your GitLab instance URL is + `https://gitlab.example.com`. +- For group or project runners, the ID number of the group or project. The ID number + is displayed in the project or group overview page, under the project or group + name. + +Use the access token in the [`POST /user/runners`](../../api/users.md#create-a-runner) +REST endpoint to create a runner: + +1. Use `curl` to invoke the endpoint to create a runner: + + ::Tabs + + :::TabTitle Project + + ```shell + curl --silent --request POST --url "https://gitlab.example.com/api/v4/user/runners" + --data "runner_type=project_type" + --data "project_id=<project_id>" + --data "description=<your_runner_description>" + --data "tag_list=<your_comma_separated_job_tags>" + --header "PRIVATE-TOKEN: <project_access_token>" + ``` + + :::TabTitle Group + + ```shell + curl --silent --request POST --url "https://gitlab.example.com/api/v4/user/runners" + --data "runner_type=group_type" + --data "group_id=<group_id>" + --data "description=<your_runner_description>" + --data "tag_list=<your_comma_separated_job_tags>" + --header "PRIVATE-TOKEN: <group_access_token>" + ``` + + :::TabTitle Shared + + ```shell + curl --silent --request POST --url "https://gitlab.example.com/api/v4/user/runners" + --data "runner_type=instance_type" + --data "description=<your_runner_description>" + --data "tag_list=<your_comma_separated_job_tags>" + --header "PRIVATE-TOKEN: <personal_access_token>" + ``` + + ::EndTabs + +1. Save the returned `token` value in a secure location or your secrets management + solution. The `token` value is returned only once in the API response. + +## With the `gitlab_user_runner` Terraform resource + +To create the runner configuration with Terraform, use the +[`gitlab_user_runner` Terraform resource](https://gitlab.com/gitlab-org/terraform-provider-gitlab/-/blob/main/docs/resources/user_runner.md?ref_type=heads) +from the [GitLab Terraform provider](https://gitlab.com/gitlab-org/terraform-provider-gitlab). + +Here's an example configuration block: + +```terraform +resource "gitlab_user_runner" "example_runner" { + runner_type = "instance_type" + description = "my-runner" + tag_list = ["shell", "docker"] +} +``` + +## Automate runner installation and registration + +If you host the runner on a virtual machine instance in a public cloud, you can automate +runner installation and registration. + +After you create a runner and its configuration, you can use the same runner +authentication token to register multiple runners with the same configuration. +For example, you can deploy multiple shared runners with the same executor type +and job tags to the target compute host. Each runner registered with the same runner +authentication token has a unique `system_id`, which GitLab Runner +generates randomly and stores in your local file system. + +Here's an example of an automation workflow you can use to register and deploy your +runners to Google Compute Engine: + +1. Use [Terraform infrastructure as code](../../user/infrastructure/iac/index.md) + to install the runner application to a virtual machine hosted on Google Cloud + Platform (GCP). +1. In the [GCP Terraform provider](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance), + use the `metadata` key to add the runner authentication token to the runner + configuration file on the GCP virtual machine. +1. To register the runner with the target GitLab instance, use a `cloud-init` script + populated from the GCP Terraform provider. Here's an example: + + ```shell + #!/bin/bash + apt update + curl --location "https://packages.gitlab.com/install/repositories/runner/ + gitlab-runner/script.deb.sh" | bash + GL_NAME=$(curl 169.254.169.254/computeMetadata/v1/instance/name + --header "Metadata-Flavor:Google") + GL_EXECUTOR=$(curl 169.254.169.254/computeMetadata/v1/instance/attributes/ + gl_executor --header "Metadata-Flavor:Google") + apt update + apt install -y gitlab-runner + gitlab-runner register --non-interactive --name="$GL_NAME" --url="https://gitlab.com" + --token="$RUNNER_TOKEN" --request-concurrency="12" --executor="$GL_EXECUTOR" + --docker-image="alpine:latest" + systemctl restart gitlab-runner + ``` + +## View runners with the same configuration + +Now that you've automated your runner creation and automation, you can view +the runners that use the same configuration in the GitLab UI. + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **CI/CD > Runners**. +1. In the search box, enter the runner description or search the list of runners. +1. To view the runners that use the same configuration, in the **Details** tab, + next to **Runners**, select **Show details**. diff --git a/doc/tutorials/boards_for_teams/index.md b/doc/tutorials/boards_for_teams/index.md index c316e42d218..fd61a4c03c5 100644 --- a/doc/tutorials/boards_for_teams/index.md +++ b/doc/tutorials/boards_for_teams/index.md @@ -99,7 +99,7 @@ projects you create later. To create each label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your **Paperclip Software Factory** group. +1. On the left sidebar, select **Search or go to** and find your **Paperclip Software Factory** group. 1. Select **Manage > Labels**. 1. Select **New label**. 1. In the **Title** field, enter the name of the label. Start with `Frontend`. @@ -124,7 +124,7 @@ to manage issues from all the projects that you might create later in this group To create a new group issue board: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your **Paperclip Software Factory** group. +1. On the left sidebar, select **Search or go to** and find your **Paperclip Software Factory** group. 1. Select **Plan > Issue boards**. 1. Create the UX workflow and Frontend workflow boards. diff --git a/doc/tutorials/build_application.md b/doc/tutorials/build_application.md index c22cba7e0e8..2b1f63874b1 100644 --- a/doc/tutorials/build_application.md +++ b/doc/tutorials/build_application.md @@ -21,6 +21,7 @@ Use CI/CD pipelines to automatically build, test, and deploy your code. | [Find CI/CD examples and templates](../ci/examples/index.md#cicd-examples) | Use these examples and templates to set up CI/CD for your use case. | | | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Understand CI/CD rules](https://www.youtube.com/watch?v=QjQc-zeL16Q) (8m 56s) | Learn more about how to use CI/CD rules. | | | [Use Auto DevOps to deploy an application](../topics/autodevops/cloud_deployments/auto_devops_with_gke.md) | Deploy an application to Google Kubernetes Engine (GKE). | | +| [Using Buildah in a rootless container with GitLab Runner Operator on OpenShift](../ci/docker/buildah_rootless_tutorial.md) | Learn how to setup GitLab Runner Operator on OpenShift to build Docker images with Buildah in a rootless container | | ## Configure GitLab Runner @@ -30,6 +31,7 @@ Set up runners to run jobs in a pipeline. |-------|-------------|--------------------| | [Create, register, and run your own project runner](create_register_first_runner/index.md) | Learn the basics of how to create and register a project runner that runs jobs for your project. | **{star}** | | [Configure GitLab Runner to use the Google Kubernetes Engine](configure_gitlab_runner_to_use_gke/index.md) | Learn how to configure GitLab Runner to use the GKE to run jobs. | | +| [Automate the creation of runners](https://about.gitlab.com/blog/2023/07/06/how-to-automate-creation-of-runners/) | Learn how to automate runner creation as an authenticated user to optimize your runner fleet. | | ## Publish a static website diff --git a/doc/tutorials/compliance_pipeline/index.md b/doc/tutorials/compliance_pipeline/index.md index 5396248d05a..710a2eb59ab 100644 --- a/doc/tutorials/compliance_pipeline/index.md +++ b/doc/tutorials/compliance_pipeline/index.md @@ -46,7 +46,7 @@ projects with the compliance framework applied. To create the compliance pipeline project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial group` group. +1. On the left sidebar, select **Search or go to** and find the `Tutorial group` group. 1. Select **New project**. 1. Select **Create blank project**. 1. In the **Project name** field, enter `Tutorial compliance project`. @@ -54,7 +54,7 @@ To create the compliance pipeline project: To add compliance pipeline configuration to `Tutorial compliance project`: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial compliance project` project. +1. On the left sidebar, select **Search or go to** and find the `Tutorial compliance project` project. 1. Select **Build > Pipeline editor**. 1. Select **Configure pipeline**. 1. In the pipeline editor, replace the default configuration with: @@ -74,7 +74,7 @@ The compliance framework is configured in the [new group](#create-a-new-group). To configure the compliance framework: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial group` group. +1. On the left sidebar, select **Search or go to** and find the `Tutorial group` group. 1. Select **Settings > General**. 1. Expand **Compliance frameworks**. 1. Select **Add framework**. @@ -87,7 +87,7 @@ To configure the compliance framework: For convenience, make the new compliance framework the default for all new projects in the group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial group` group. +1. On the left sidebar, select **Search or go to** and find the `Tutorial group` group. 1. Select **Settings > General**. 1. Expand **Compliance frameworks**. 1. In the row for `Tutorial compliance framework`, select **Options** (**{ellipsis_v}**). @@ -100,7 +100,7 @@ compliance pipeline configuration in their pipelines. To create a new project for running the compliance pipeline configuration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial group` group. +1. On the left sidebar, select **Search or go to** and find the `Tutorial group` group. 1. Select **Create new** (**{plus}**) and **New project/repository**. 1. Select **Create blank project**. 1. In the **Project name** field, enter `Tutorial project`. @@ -114,7 +114,7 @@ pipeline configuration in `Tutorial compliance project`. To run the compliance pipeline configuration in `Tutorial project`: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial project` project. +1. On the left sidebar, select **Search or go to** and find the `Tutorial project` project. 1. Select **Build > Pipelines**. 1. Select **Run pipeline**. 1. On the **Run pipeline** page, select **Run pipeline**. @@ -132,7 +132,7 @@ compliance pipeline configuration to refer to it. To create the regular pipeline configuration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial project` project. +1. On the left sidebar, select **Search or go to** and find the `Tutorial project` project. 1. Select **Build > Pipeline editor**. 1. Select **Configure pipeline**. 1. In the pipeline editor, replace the default configuration with: @@ -148,7 +148,7 @@ To create the regular pipeline configuration: To combine the new project pipeline configuration with the compliance pipeline configuration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial compliance project` project. +1. On the left sidebar, select **Search or go to** and find the `Tutorial compliance project` project. 1. Select **Build > Pipeline editor**. 1. In the existing configuration, add: @@ -162,7 +162,7 @@ To combine the new project pipeline configuration with the compliance pipeline c To confirm the regular pipeline configuration is combined with the compliance pipeline configuration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `Tutorial project` project. +1. On the left sidebar, select **Search or go to** and find the `Tutorial project` project. 1. Select **Build > Pipelines**. 1. Select **Run pipeline**. 1. On the **Run pipeline** page, select **Run pipeline**. diff --git a/doc/tutorials/configure_gitlab_runner_to_use_gke/index.md b/doc/tutorials/configure_gitlab_runner_to_use_gke/index.md index d9a6dbbb3f0..8fa2dbe3479 100644 --- a/doc/tutorials/configure_gitlab_runner_to_use_gke/index.md +++ b/doc/tutorials/configure_gitlab_runner_to_use_gke/index.md @@ -80,15 +80,11 @@ Now that you have a cluster, you're ready to install and configure the Kubernete 1. Install the Operator Lifecycle Manager (OLM), a tool that manages the Kubernetes Operators that run on the cluster: - <!-- markdownlint-disable --> - ```shell curl --silent --location "https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/install.sh" \ | bash -s v0.24.0 ``` - <!-- markdownlint-enable --> - 1. Install the Kubernetes Operator Catalog: ```shell @@ -196,7 +192,6 @@ To check if runners are running in the GKE cluster, you can either: ``` - Check the job log in GitLab: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) - to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Jobs** and find the job. 1. To view the job log, select the job status. diff --git a/doc/tutorials/convert_personal_namespace_to_group/index.md b/doc/tutorials/convert_personal_namespace_to_group/index.md index 093dd04882a..39bfebb7f4e 100644 --- a/doc/tutorials/convert_personal_namespace_to_group/index.md +++ b/doc/tutorials/convert_personal_namespace_to_group/index.md @@ -56,7 +56,7 @@ Before you start the transfer process, make sure you: To transfer a project to a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. Under **Transfer project**, choose the group to transfer the project to. @@ -84,7 +84,7 @@ Finally, rename the new group's URL to the username of the original personal nam To [change your group path](../../user/group/manage.md#change-a-groups-path) (group URL): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. Under **Change group URL**, enter the user's original username. diff --git a/doc/tutorials/create_register_first_runner/index.md b/doc/tutorials/create_register_first_runner/index.md index 05bf5cd8288..a16899509e7 100644 --- a/doc/tutorials/create_register_first_runner/index.md +++ b/doc/tutorials/create_register_first_runner/index.md @@ -53,7 +53,7 @@ In this file, you define: - The structure and order of jobs that the runner should execute. - The decisions the runner should make when specific conditions are encountered. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Project overview**. 1. Select the plus icon (**{plus}**), then select **New file**. 1. In the **Filename** field, enter `.gitlab-ci.yml`. @@ -85,7 +85,7 @@ to GitLab so that it can pick up jobs from the project pipeline. To create a project runner: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand the **Runners** section. 1. Select **New project runner**. @@ -107,7 +107,7 @@ To create a project runner: ### Check the runner configuration file -After you register the runner, the configuration and authentication token is saved to your `config.toml`. The runner uses the +After you register the runner, the configuration and runner authentication token is saved to your `config.toml`. The runner uses the token to authenticate with GitLab when picking up jobs from the job queue. You can use the `config.toml` to @@ -130,7 +130,7 @@ Here's what your `config.toml` should look like after you register and start the Next, trigger a pipeline in your project so you can view your runner execute a job. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. Select **Run pipeline**. 1. Select a job to view the job log. The output should look similar to this example, which shows diff --git a/doc/tutorials/dependency_scanning.md b/doc/tutorials/dependency_scanning.md index 90bc2ec96a2..6eb2592c2f4 100644 --- a/doc/tutorials/dependency_scanning.md +++ b/doc/tutorials/dependency_scanning.md @@ -138,7 +138,7 @@ need to upgrade the `fastify` package. To fix the vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. In the upper right, select **Edit > GitPod** and open GitPod in a new tab. 1. If you are prompted to, select **Continue with GitLab**, then select **Authorize**. diff --git a/doc/tutorials/export_sbom.md b/doc/tutorials/export_sbom.md new file mode 100644 index 00000000000..f0bbf6febf6 --- /dev/null +++ b/doc/tutorials/export_sbom.md @@ -0,0 +1,95 @@ +--- +stage: Secure +group: Composition Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tutorial: Export dependency list in SBOM format **(ULTIMATE ALL)** + +Dependency Scanning output can be exported to the CycloneDX JSON format. + +This tutorial shows you how to generate a CycloneDX JSON SBOM for a pipeline, and then to upload it as a CI job artifact. + +## Prerequisites + +Set up Dependency Scanning. For detailed instructions, follow [the Dependency Scanning tutorial](dependency_scanning.md). + +## Create configuration files + +1. Create a [snippet](../api/snippets.md) with the following code. + + Filename: `export.sh` + + ```shell + #! /bin/sh + + function create_export { + curl --silent \ + --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \ + -X 'POST' --data "export_type=sbom" \ + "http://gitlab.example.com/api/v4/pipelines/$CI_PIPELINE_ID/dependency_list_exports" \ + | jq '.id' + } + + function check_status { + curl --silent \ + --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \ + --write-out "%{http_code}" --output /dev/null \ + http://gitlab.example.com/api/v4/dependency_list_exports/$1 + } + + function download { + curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \ + --output "gl-sbom-merged-$CI_PIPELINE_ID.cdx.json" \ + "http://gitlab.example.com/api/v4/dependency_list_exports/$1/download" + } + + function export_sbom { + local ID=$(create_export) + + for run in $(seq 0 3); do + local STATUS=$(check_status $ID) + # Status is 200 when JSON is generated. + # Status is 202 when generate JSON job is running. + if [ $STATUS -eq "200" ]; then + download $ID + + exit 0 + elif [ $STATUS -ne "202" ]; then + exit 1 + fi + + echo "Waiting for JSON to be generated" + sleep 5 + done + + exit 1 + } + + export_sbom + ``` + + The above script works in the following steps: + + 1. Create a CycloneDX SBOM export for the current pipeline. + 1. Check the status of that export, and stop when it's ready. + 1. Download the CycloneDX SBOM file. + +1. Update `.gitlab-ci.yml` with the following code. + + ```yaml + export-merged-sbom: + before_script: + - apk add --update jq curl + stage: .post + script: + - curl --output export.sh --url "https://gitlab.example.com/api/v4/snippets/<SNIPPET_ID>/raw" + - /bin/sh export.sh + artifacts: + paths: + - "gl-sbom-merged-*.cdx.json" + ``` + +1. Go to **Build > Pipelines** and confirm that the latest pipeline completed successfully. + +In the job artifacts, `gl-sbom-merged-<pipeline_id>.cdx.json` file should be present. diff --git a/doc/tutorials/fuzz_testing/index.md b/doc/tutorials/fuzz_testing/index.md index e4c494e9b85..252a3501a32 100644 --- a/doc/tutorials/fuzz_testing/index.md +++ b/doc/tutorials/fuzz_testing/index.md @@ -59,7 +59,7 @@ a random buffer as a parameter. To create the two fuzz target files: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `fuzz-testing-demo` project. +1. On the left sidebar, select **Search or go to** and find the `fuzz-testing-demo` project. 1. Create a file in the root directory of the project. 1. Name the file `fuzz-sayhello.js` and add the following code: diff --git a/doc/tutorials/hugo/index.md b/doc/tutorials/hugo/index.md index 97c79e77392..5f466234a36 100644 --- a/doc/tutorials/hugo/index.md +++ b/doc/tutorials/hugo/index.md @@ -139,7 +139,7 @@ You'll see that GitLab has run your `test` and `pages` jobs. To view your site, on the left-hand navigation, select **Settings > Pages** -The `pages` job in your pipeline has deployed the contents of your `public` directory to GitLab Pages. Under **Access pages**, you should see the link in the format: `https://<your-namespace>.gitlab.io/<project-name>`. +The `pages` job in your pipeline has deployed the contents of your `public` directory to GitLab Pages. Under **Access pages**, you should see the link in the format: `https://<your-namespace>.gitlab.io/<project-path>`. You won't see this link if you haven't yet run your pipeline. diff --git a/doc/tutorials/issue_triage/index.md b/doc/tutorials/issue_triage/index.md index 38e4285c2ce..2634837edc8 100644 --- a/doc/tutorials/issue_triage/index.md +++ b/doc/tutorials/issue_triage/index.md @@ -102,7 +102,7 @@ However, they aren't mutually exclusive. To create each label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Select **New label**. 1. In the **Title** field, enter the name of the label. Start with `type::bug`. @@ -152,7 +152,7 @@ To set up your issue board: 1. Decide on the scope of the board. For example, create one that you'll use to assign severity to issues. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your +1. On the left sidebar, select **Search or go to** and find your **Issue triage tutorial** project. 1. Select **Plan > Issue boards**. 1. In the upper-left corner of the issue board page, select the dropdown list with the current board name. diff --git a/doc/tutorials/left_sidebar/img/admin_area_v16_4.png b/doc/tutorials/left_sidebar/img/admin_area_v16_4.png Binary files differnew file mode 100644 index 00000000000..ca6a229f69c --- /dev/null +++ b/doc/tutorials/left_sidebar/img/admin_area_v16_4.png diff --git a/doc/tutorials/left_sidebar/img/project_selected_v16_0.png b/doc/tutorials/left_sidebar/img/project_selected_v16_0.png Binary files differdeleted file mode 100644 index 534b06ac5de..00000000000 --- a/doc/tutorials/left_sidebar/img/project_selected_v16_0.png +++ /dev/null diff --git a/doc/tutorials/left_sidebar/img/project_selected_v16_4.png b/doc/tutorials/left_sidebar/img/project_selected_v16_4.png Binary files differnew file mode 100644 index 00000000000..a15068a14d9 --- /dev/null +++ b/doc/tutorials/left_sidebar/img/project_selected_v16_4.png diff --git a/doc/tutorials/left_sidebar/img/search_projects_v16_0.png b/doc/tutorials/left_sidebar/img/search_projects_v16_0.png Binary files differdeleted file mode 100644 index 12182f24009..00000000000 --- a/doc/tutorials/left_sidebar/img/search_projects_v16_0.png +++ /dev/null diff --git a/doc/tutorials/left_sidebar/img/search_projects_v16_4.png b/doc/tutorials/left_sidebar/img/search_projects_v16_4.png Binary files differnew file mode 100644 index 00000000000..966836f8d31 --- /dev/null +++ b/doc/tutorials/left_sidebar/img/search_projects_v16_4.png diff --git a/doc/tutorials/left_sidebar/img/sidebar_middle_v16_1.png b/doc/tutorials/left_sidebar/img/sidebar_middle_v16_1.png Binary files differdeleted file mode 100644 index 67e89c2c0a4..00000000000 --- a/doc/tutorials/left_sidebar/img/sidebar_middle_v16_1.png +++ /dev/null diff --git a/doc/tutorials/left_sidebar/img/sidebar_middle_v16_4.png b/doc/tutorials/left_sidebar/img/sidebar_middle_v16_4.png Binary files differnew file mode 100644 index 00000000000..2556a4985d9 --- /dev/null +++ b/doc/tutorials/left_sidebar/img/sidebar_middle_v16_4.png diff --git a/doc/tutorials/left_sidebar/img/sidebar_top_v16_1.png b/doc/tutorials/left_sidebar/img/sidebar_top_v16_1.png Binary files differdeleted file mode 100644 index fcb56370fc7..00000000000 --- a/doc/tutorials/left_sidebar/img/sidebar_top_v16_1.png +++ /dev/null diff --git a/doc/tutorials/left_sidebar/img/sidebar_top_v16_4.png b/doc/tutorials/left_sidebar/img/sidebar_top_v16_4.png Binary files differnew file mode 100644 index 00000000000..3bfe781f2b2 --- /dev/null +++ b/doc/tutorials/left_sidebar/img/sidebar_top_v16_4.png diff --git a/doc/tutorials/left_sidebar/img/your_work_v16_0.png b/doc/tutorials/left_sidebar/img/your_work_v16_0.png Binary files differdeleted file mode 100644 index f7b5ed4217d..00000000000 --- a/doc/tutorials/left_sidebar/img/your_work_v16_0.png +++ /dev/null diff --git a/doc/tutorials/left_sidebar/img/your_work_v16_4.png b/doc/tutorials/left_sidebar/img/your_work_v16_4.png Binary files differnew file mode 100644 index 00000000000..65e91b0077b --- /dev/null +++ b/doc/tutorials/left_sidebar/img/your_work_v16_4.png diff --git a/doc/tutorials/left_sidebar/index.md b/doc/tutorials/left_sidebar/index.md index fee10fbe783..d30ade980dc 100644 --- a/doc/tutorials/left_sidebar/index.md +++ b/doc/tutorials/left_sidebar/index.md @@ -28,20 +28,19 @@ At the top of the left sidebar are several shortcuts. Use these shortcuts to show and hide the left sidebar, create new items, search, and view your profile. You can also view your list of issues, merge requests, and to-do items. - + +NOTE: If you have hidden the left sidebar, you can display it temporarily by hovering your cursor over the left edge of the GitLab window. The next area of the left sidebar changes based on the information you're viewing. For example, you might be viewing a project, exploring projects or groups, or viewing your profile. -Use this area to switch to other areas of the left sidebar. +To switch to other areas of the left sidebar, use **Search or go to**. - + The rest of the left sidebar is populated based on the option you choose. For example, -if you're in a project, the sidebar is project-specific: - - +if you're in a project, the sidebar is project-specific. ## Find your project @@ -49,19 +48,15 @@ Now let's go over a few common tasks you'll use the left sidebar for. To start, we will find the project we want to work on. -1. To explore all available projects, on the left sidebar, select **Explore**: - -  - -1. On the right, above the list of projects, type search criteria. - The search finds projects with a matching description. +1. To explore all available projects, on the left sidebar, select **Search or go to**. +1. Choose from the list of frequently visited projects, or + type a colon `:` followed by the project name: -  +  -1. When you find the project you want, select the project name. - The left sidebar now shows project-specific options. +The left sidebar now shows project-specific options. -  + ## Pin frequently used items @@ -82,12 +77,12 @@ The items you pin while you're viewing a project are different than the items yo ## Use a more focused view On the left sidebar, you can also choose a more focused view into the areas you have access to. -Change the view to **Your work**: +Select **Search or go to** and then select **Your work**: - + ## Go to the Admin Area -The Admin Area is also available on the left sidebar: +The Admin Area is also available on the left sidebar when you select **Search or go to**: - + diff --git a/doc/tutorials/manage_user/index.md b/doc/tutorials/manage_user/index.md index 91309d7cb42..df343fe5f08 100644 --- a/doc/tutorials/manage_user/index.md +++ b/doc/tutorials/manage_user/index.md @@ -79,8 +79,8 @@ You will now create subgroups to reflect this organization structure. > Subgroups and projects must have visibility settings that are at least as restrictive as the visibility setting of their parent group. For example, you cannot have a private parent group and a public subgroup. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. 1. Select **Development**. You should see an **Owner** label next to the group name as you have the Owner role. 1. On the parent group's overview page, in the upper-right corner, select **New subgroup**. @@ -106,7 +106,7 @@ for the organization. You will now manually create the users for your organization. These are test users. To create the first test user, Alex Smith: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Users**. 1. Select **New user**. @@ -149,7 +149,7 @@ You can give users access to all projects in a group by adding them to that grou First, you will add all the users to the parent group, Development. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select **Manage > Members**. 1. Select **Invite members**. 1. Complete the fields for the product manager, Alex Smith. @@ -189,7 +189,7 @@ subgroups with the same role. You can filter a subgroup to show which users are direct members of that subgroup, and which members have inherited membership of that subgroup from the parent group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select the **User Experience** subgroup. 1. On the left sidebar, select **Subgroup information > Members**. 1. On the **Members** page, select the **Filter members** field. @@ -209,7 +209,7 @@ them from the parent group. Go back to the parent group and remove everyone except Alex Smith: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the parent group. +1. On the left sidebar, select **Search or go to** and find the parent group. 1. Select **Manage > Members**. 1. On the member row you want to remove, select the vertical ellipsis (**{ellipsis_v}**) and then select **Remove member**. @@ -228,7 +228,7 @@ You will now add users directly to the different subgroups. ### Add users to the Product Management subgroup -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select the **Product Management** subgroup. 1. On the left sidebar, select **Subgroup information > Members**. @@ -267,7 +267,7 @@ add users to the Engineering subgroup. You are now going to invite some users to the Engineering subgroup. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select the **Engineering** subgroup. 1. On the left sidebar, select **Subgroup information > Members**. The only members are you and Alex, both with the Owner role. These are inherited roles. @@ -312,7 +312,7 @@ included in both nested subgroups due to inherited permissions. Therefore, you will add these users to the appropriate nested subgroup directly rather than to the User Experience subgroup. -1. 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. 1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select the **User Experience** subgroup, and then the **UX Design** subgroup. 1. On the left sidebar, select **Subgroup information > Members**. You and Alex Smith are currently the only members. These are inherited roles. @@ -371,7 +371,7 @@ need to work on, and that piece of work is for the whole organization. To organi that work, you are going to create a project in the Development parent group, and add different users to that project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Development** group. +1. On the left sidebar, select **Search or go to** and find the **Development** group. 1. Select **Create new** (**{plus}**) and **New project/repository**. 1. Select **Create blank project**. 1. Enter the project details: @@ -400,7 +400,7 @@ directly to the project. ## Add users to the project and parent group -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Release 2.0** project. +1. On the left sidebar, select **Search or go to** and find the **Release 2.0** project. 1. On the left sidebar, select **Manage > Members**. 1. Select **Invite members**. Invite the following users: @@ -433,7 +433,7 @@ projects by assigning roles. You can also change the visibility of individual features in a project. You cannot do this for groups. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the **Release 2.0** project. +1. On the left sidebar, select **Search or go to** and find the **Release 2.0** project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. In **Project visibility**, you can who can see the project in the public access diff --git a/doc/tutorials/move_personal_project_to_group/index.md b/doc/tutorials/move_personal_project_to_group/index.md index 29849752f5f..73d22de6983 100644 --- a/doc/tutorials/move_personal_project_to_group/index.md +++ b/doc/tutorials/move_personal_project_to_group/index.md @@ -57,7 +57,7 @@ Before you move your project to a group: Now you're ready to move your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. Under **Transfer project**, choose the group to transfer the project to. @@ -78,7 +78,7 @@ your related resources and tools, such as websites and package managers. You can now view your project in your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Look for your project under **Subgroups and projects**. Start enjoying the benefits of a group! For example, as the group Owner, you can diff --git a/doc/tutorials/plan_and_track.md b/doc/tutorials/plan_and_track.md index 7dcafbd26ce..c5c2919cac7 100644 --- a/doc/tutorials/plan_and_track.md +++ b/doc/tutorials/plan_and_track.md @@ -12,8 +12,6 @@ issues, epics, and more. | Topic | Description | Good for beginners | |-------|-------------|--------------------| | [GitLab Agile Project Management](https://levelup.gitlab.com/courses/gitlab-agile-project-management) | Learn how to use planning features to manage your projects in this self-paced course. | **{star}** | -| [Create a project from a template](https://gitlab.com/projects/new#create_from_template) | Choose a project template and create a project with files to get you started. | | -| [Migrate to GitLab](../user/project/import/index.md) | If you are coming to GitLab from another platform, you can import or convert your projects. | | | [Build a protected workflow for your project](protected_workflow/index.md) | Set up a workflow for your teams, and enforce protections with approval rules. | | | [Run an agile iteration](agile_sprint/index.md) | Use group, projects, and iterations to run an agile development iteration. | | [Set up a single project for issue triage](issue_triage/index.md) | Use labels to set up a project for issue triage. | **{star}** | diff --git a/doc/tutorials/protected_workflow/index.md b/doc/tutorials/protected_workflow/index.md index b055faddc75..5ff798fce6b 100644 --- a/doc/tutorials/protected_workflow/index.md +++ b/doc/tutorials/protected_workflow/index.md @@ -62,7 +62,7 @@ Then you'll add these new groups as members of the `engineering` group. First, create the new subgroup: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) +1. On the left sidebar, select **Search or go to** and search for `engineering`. Select the group named `Engineering`:  @@ -75,7 +75,7 @@ First, create the new subgroup: Next, add the subgroup as a member of the `engineering` group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) +1. On the left sidebar, select **Search or go to** and search for `engineering`. Select the group named `Engineering`. 1. On the left sidebar, select **Manage > Members**. 1. On the top right, select **Invite a group**. @@ -105,7 +105,7 @@ for projects owned by `engineering`. As a result: To add a user to the `frontend` subgroup: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) +1. On the left sidebar, select **Search or go to** and search for `frontend`. Select the `Frontend` group. 1. Select **Manage > Members**. 1. Select **Invite members**. @@ -127,7 +127,7 @@ smaller subgroups you just created. To create the new `excelsior` project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) and +1. On the left sidebar, select **Search or go to** and search for `engineering`. Select the group named `Engineering`. 1. On the overview page for the `engineering` group, on the left sidebar, at the top, select **Create new...** (**{plus}**) and **In this group > New project/repository**. @@ -162,7 +162,7 @@ GitLab Premium or Ultimate. To add a CODEOWNERS file to your `excelsior` project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) and +1. On the left sidebar, select **Search or go to** and search for `Excelsior`. Select the project named `Excelsior`. 1. Next to the branch name, select the plus icon (**{plus}**), then **New file**:  diff --git a/doc/tutorials/scan_execution_policy/index.md b/doc/tutorials/scan_execution_policy/index.md new file mode 100644 index 00000000000..d0122908e19 --- /dev/null +++ b/doc/tutorials/scan_execution_policy/index.md @@ -0,0 +1,197 @@ +--- +stage: Govern +group: Security Policies +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tutorial: Set up a scan execution policy **(ULTIMATE ALL)** + +This tutorial shows you how to create and apply a +[scan execution policy](../../user/application_security/policies/scan-execution-policies.md). +These policies enforce application security tools as part of the CI/CD pipeline. In this tutorial, +you create a policy to enforce secret detection in the CI/CD pipeline of two projects. + +In this tutorial, you: + +- [Create project A](#create-project-a). +- [Create the scan execution policy](#create-the-scan-execution-policy). +- [Test the scan execution policy with project A](#test-the-scan-execution-policy-with-project-a). +- [Create project B](#create-project-b). +- [Link project B to the security policy project](#link-project-b-to-the-security-policy-project). +- [Test the scan execution policy with project B](#test-the-scan-execution-policy-with-project-b). + +Prerequisite: + +- Permission to create new projects in an existing group. + +## Create project A + +In a standard workflow, you might already have an existing project. In this +tutorial, you're starting with nothing, so the first step is to create a project. + +To create project A: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **New project**. +1. Select **Create blank project**. +1. Complete the fields. For **Project name**, enter `go-example-a`. +1. Select **Create project**. +1. Select **Add (`+`) > New file**. +1. Enter `helloworld.go` in the filename. +1. Copy and paste the following example Go code into the file. + + ```go + package main + import "fmt" + func main() { + fmt.Println("Hello world") + } + ``` + +1. Select **Commit changes**. + +The next step is to create a scan execution policy. When the first security policy is created, a +policy project is created. The policy project stores the security policies created in any projects +that are linked to it. Keeping policies separate from the projects they protect makes your security +configuration reusable and easier to maintain. + +## Create the scan execution policy + +To create the scan execution policy: + +1. On the left sidebar, select **Search or go to** and search for the `go-example-a` project. +1. Go to **Secure > Policies**. +1. Select **New policy**. +1. In the **Scan execution policy** section, select **Select policy**. +1. Complete the fields. + - **Name**: Enforce secret detection. + - **Policy status**: Enabled. + - **Actions**: Run a Secret Detection scan. + - **Conditions**: Triggers every time a pipeline runs for all branches. +1. Select **Configure with a merge request**. + + The policy project `go-example-a - Security project` is created, and a merge request is created. + +1. Optional. Review the generated policy YAML in the merge request's **Changes** tab. +1. Go to the **Overview** tab and select **Merge**. +1. On the left sidebar, select **Search or go to** and search for the `go-example-a` project. +1. Go to **Secure > Policies**. + +You now have a scan execution policy that runs a secret detection scan on every MR, for any branch. +Test the policy by creating a merge request in project A. + +## Test the scan execution policy with project A + +To test the scan execution policy: + +1. On the left sidebar, select **Search or go to** and find the project named `go-example-a`. +1. Go to **Code > Repository**. +1. Select the `helloworld.go` file. +1. Select **Edit > Edit single file**. +1. Add the following line immediately after the `fmt.Println("hello world")` line: + + ```plaintext + var GitLabFeedToken = "feed_token=eFLISqaBym4EjAefkl58" + ``` + +1. In the **Target Branch** field, enter `feature-a`. +1. Select **Commit changes**. +1. When the merge request page opens, select **Create merge request**. + + Let's check if the scan execution policy worked. Remember that we specified that secret detection + is to run every time a pipeline runs, for any branch. + +1. In the merge request just created, go the **Pipelines** tab and select the created pipeline. + + Here you can see that a secret detection job ran. Let's check if it detected the test secret. + +1. Select the secret detection job. + + Near the bottom of the job's log, the following output confirms that the example secret was detected. + + ```plaintext + [INFO] [secrets] [2023-09-04T03:46:36Z] ▶ 3:46AM INF 1 commits scanned. + [INFO] [secrets] [2023-09-04T03:46:36Z] ▶ 3:46AM INF scan completed in 60ms + [INFO] [secrets] [2023-09-04T03:46:36Z] ▶ 3:46AM WRN leaks found: 1 + ``` + +You've seen the policy work for one project. Create another project and apply the same policy. + +## Create project B + +To create project B: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **New project**. +1. Select **Create blank project**. +1. Complete the fields. For **Project name**, enter `go-example-b`. +1. Select **Create project**. +1. Select **Add (`+`) > New file**. +1. Enter `helloworld.go` in the filename. +1. Copy and paste the following example Go code into the file. + + ```go + package main + import "fmt" + func main() { + fmt.Println("Hello world") + } + ``` + +1. Select **Commit changes**. + +Now that you have another project, you link it to the same policy project. + +## Link project B to the security policy project + +To link project B to the security policy project: + +1. On the left sidebar, select **Search or go to** and find the `go-example-b` project. +1. Go to **Secure > Policies**. +1. Select **Edit policy project**. +1. Select the dropdown list, then search for the security policy project created at the start of + this tutorial. +1. Select **Save**. + +Linking project B to the same policy project resulted in the same policy being applied. A scan +execution policy runs a secret detection scan on every MR, for any branch. Let's test the +policy by creating an MR in project B. + +## Test the scan execution policy with project B + +To test the scan execution policy: + +1. On the left sidebar, select **Search or go to** and find the `go-example-b` project. +1. Go to **Code > Repository**. +1. Select the `helloworld.go` file. +1. Select **Edit > Edit single file**. +1. Add the following line immediately after the `fmt.Println("hello world")` line: + + ```plaintext + var AdobeClient = "4ab4b080d9ce4072a6be2629c399d653" + ``` + +1. In the **Target Branch** field, enter `feature-b`. +1. Select **Commit changes**. +1. When the merge request page opens, select **Create merge request**. + + Let's check if the scan execution policy worked. Remember that we specified that secret detection + is to run every time a pipeline runs, for any branch. + +1. In the merge request just created, go the **Pipelines** tab and select the created pipeline. + +1. In the merge request just created, select the pipeline's ID. + + Here you can see that a secret detection job ran. Let's check if it detected the test secret. + +1. Select the secret detection job. + + Near the bottom of the job's log, the following output confirms that the example secret was detected. + + ```plaintext + [INFO] [secrets] [2023-09-04T04:22:28Z] ▶ 4:22AM INF 1 commits scanned. + [INFO] [secrets] [2023-09-04T04:22:28Z] ▶ 4:22AM INF scan completed in 58.2ms + [INFO] [secrets] [2023-09-04T04:22:28Z] ▶ 4:22AM WRN leaks found: 1 + ``` + +Congratulations. You've learned how to create a scan execution policy and enforce it on projects. diff --git a/doc/tutorials/scan_result_policy/index.md b/doc/tutorials/scan_result_policy/index.md index 1c23d6a36eb..dabc311135a 100644 --- a/doc/tutorials/scan_result_policy/index.md +++ b/doc/tutorials/scan_result_policy/index.md @@ -35,7 +35,7 @@ To set up a scan result policy: Next, you'll add a scan result policy to your test project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `sast-scan-result-policy` project. +1. On the left sidebar, select **Search or go to** and find the `sast-scan-result-policy` project. 1. Select **Secure > Policies**. 1. Select **New policy**. 1. In **Scan result policy**, select **Select policy**. @@ -60,7 +60,7 @@ Next, you'll add a scan result policy to your test project: The application creates a new project to store the policies linked to it, and creates a merge request to define the policy. 1. Select **Merge**. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `sast-scan-result-policy` project. +1. On the left sidebar, select **Search or go to** and find the `sast-scan-result-policy` project. 1. Select **Secure > Policies**. You can see the list of policies added in the previous steps. @@ -69,7 +69,7 @@ Next, you'll add a scan result policy to your test project: Nice work, you've created a scan result policy. To test it, create some vulnerabilities and check the result: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the `sast-scan-result-policy` project. +1. On the left sidebar, select **Search or go to** and find the `sast-scan-result-policy` project. 1. Select **Code > Repository**. 1. From the **Add** (**{plus}**) dropdown list, select **New file**. 1. In the **Filename** field enter `main.ts`. diff --git a/doc/tutorials/secure_application.md b/doc/tutorials/secure_application.md index 54235d0a6dc..606ca4d2909 100644 --- a/doc/tutorials/secure_application.md +++ b/doc/tutorials/secure_application.md @@ -11,8 +11,10 @@ GitLab can check your application for security vulnerabilities and that it meets | Topic | Description | Good for beginners | |-------|-------------|--------------------| | [Set up dependency scanning](dependency_scanning.md) | Learn how to detect vulnerabilities in an application's dependencies. | **{star}** | +| [Export Dependency List in SBOM format](export_sbom.md) | Learn how to export an application's dependencies to the CycloneDX SBOM format. | **{star}** | | [Create a compliance pipeline](compliance_pipeline/index.md) | Learn how to create compliance pipelines for your groups. | **{star}** | | [Set up a scan result policy](scan_result_policy/index.md) | Learn how to configure a scan result policy that takes action based on scan results. | **{star}** | +| [Set up a scan execution policy](scan_execution_policy/index.md) | Learn how to create a scan execution policy to enforce security scanning of your project. | **{star}** | | [Scan a Docker container for vulnerabilities](container_scanning/index.md) | Learn how to use container scanning templates to add container scanning to your projects. | **{star}** | | [Get started with GitLab application security](../user/application_security/get-started-security.md) | Follow recommended steps to set up security tools. | | | [GitLab Security Essentials](https://levelup.gitlab.com/courses/security-essentials) | Learn about the essential security capabilities of GitLab in this self-paced course. | | diff --git a/doc/tutorials/update_commit_messages/index.md b/doc/tutorials/update_commit_messages/index.md index f6d92b5c13f..90626d95c32 100644 --- a/doc/tutorials/update_commit_messages/index.md +++ b/doc/tutorials/update_commit_messages/index.md @@ -12,7 +12,7 @@ that your commit message didn't completely align with a project's [commit message guidelines](../../development/contributing/merge_request_workflow.md#commit-messages-guidelines). Updating the message can be tricky if you don't have much practice with using Git -from the command line interface (CLI). But don't worry, even if you have only ever worked in +from the command-line interface (CLI). But don't worry, even if you have only ever worked in the GitLab UI, we'll walk you through the steps to use the CLI. This tutorial explains how to rewrite commit messages in both cases: @@ -33,7 +33,7 @@ You must have: - A GitLab project with a Git branch containing commits that you want to update. - Git [installed on your local machine](../../topics/git/how_to_install_git/index.md). -- The ability to get to your local machine's command line interface (CLI). In macOS, +- The ability to get to your local machine's command-line interface (CLI). In macOS, you can use Terminal. In Windows, you can use PowerShell. Linux users are probably already familiar with their system's CLI. - Familiarity with your system's default editor. This tutorial assumes your editor is Vim, diff --git a/doc/tutorials/website_project_with_analytics/index.md b/doc/tutorials/website_project_with_analytics/index.md index a0a78b68585..c75ec27fd24 100644 --- a/doc/tutorials/website_project_with_analytics/index.md +++ b/doc/tutorials/website_project_with_analytics/index.md @@ -131,20 +131,20 @@ To create an Insights report, in the `My website` project: ```yaml bugsCharts: - title: "Charts for bugs" - charts: - - title: "Monthly bugs created" - description: "Open bugs created per month" - type: bar - query: - data_source: issuables - params: - issuable_type: issue - issuable_state: opened - filter_labels: - - bug - group_by: month - period_limit: 12 + title: "Charts for bugs" + charts: + - title: "Monthly bugs created" + description: "Open bugs created per month" + type: bar + query: + data_source: issuables + params: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 12 ``` 1. Select **Commit changes**. diff --git a/doc/update/background_migrations.md b/doc/update/background_migrations.md index cbdcfef3bae..5a4b19016f8 100644 --- a/doc/update/background_migrations.md +++ b/doc/update/background_migrations.md @@ -30,7 +30,7 @@ are created by GitLab developers and run automatically on upgrade. However, such limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to prevent integer overflow for some tables. -Some installations [may need to run GitLab 14.0 for at least a day](index.md#1400) +Some installations [may need to run GitLab 14.0 for at least a day](versions/gitlab_14_changes.md#1400) to complete the database changes introduced by that upgrade. Batched background migrations are handled by Sidekiq and @@ -67,7 +67,7 @@ Prerequisites: To check the status of batched background migrations: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Monitoring > Background Migrations**. 1. Select **Queued** or **Finalizing** to see incomplete migrations, @@ -190,26 +190,34 @@ the number of batched background migrations executed in parallel: ApplicationSetting.update_all(database_max_running_batched_background_migrations: 4) ``` -### Fix and retry failed batched background migrations +### Resolve failed batched background migrations + +If a batched background migration fails, [fix and retry](#fix-and-retry-the-migration) it. +If the migration continues to fail with an error, either: + +- [Finish the failed migration manually](#finish-a-failed-migration-manually) +- [Mark the failed migration finished](#mark-a-failed-migration-finished) + +#### Fix and retry the migration > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67504) in GitLab 14.3. -If you [check the status](#check-the-status-of-batched-background-migrations) of batched background migrations, -some migrations might display in the **Failed** tab with a **failed** status: +All failed batched background migrations must be resolved to upgrade to a newer +version of GitLab. If you [check the status](#check-the-status-of-batched-background-migrations) +of batched background migrations, some migrations might display in the **Failed** tab +with a **failed** status:  -You must resolve all failed batched background migrations to upgrade to a newer -version of GitLab. - To determine why the batched background migration failed, -[view the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs) or: +[view the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs) +or view error information in the UI. Prerequisites: - You must have administrator access to the instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Monitoring > Background Migrations**. 1. Select the **Failed** tab. This displays a list of failed batched background migrations. @@ -219,13 +227,13 @@ Prerequisites: If you are a GitLab customer, consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) to debug why the batched background migrations failed. -To correct the problem, you can retry the failed batched background migrations: +To correct the problem, you can retry the failed migration. Prerequisites: - You must have administrator access to the instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Monitoring > Background Migrations**. 1. Select the **Failed** tab. This displays a list of failed batched background migrations. @@ -235,6 +243,111 @@ To monitor the retried batched background migrations, you can [check the status of batched background migrations](#check-the-status-of-batched-background-migrations) on a regular interval. +#### Finish a failed migration manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62634) in GitLab 14.1. + +To manually finish a batched background migration that failed with an error, +use the information in the failure error logs or the database: + +::Tabs + +:::TabTitle From the failure error logs + +1. [View the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs) + and look for an `An error has occurred, all later migrations canceled` error message, like this: + + ```plaintext + StandardError: An error has occurred, all later migrations canceled: + + Expected batched background migration for the given configuration to be marked as + 'finished', but it is 'active': + {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", + :table_name=>"push_event_payloads", + :column_name=>"event_id", + :job_arguments=>[["event_id"], + ["event_id_convert_to_bigint"]] + } + ``` + +1. Run the following command, replacing the values in angle brackets with the correct arguments: + + ```shell + sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>'] + ``` + + When dealing with multiple arguments, such as `[["id"],["id_convert_to_bigint"]]`, escape the + comma between each argument with a backslash <code>\</code> to prevent an invalid character error. + For example, to finish the migration from the previous step: + + ```shell + sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] + ``` + +:::TabTitle From the database + + 1. [Check the status](#check-the-status-of-batched-background-migrations) of the + migration in the database. + 1. Use the query results to construct a migration command, replacing the values + in angle brackets with the correct arguments: + + ```shell + sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>'] + ``` + + For example, if the query returns this data: + + - `job_class_name`: `CopyColumnUsingBackgroundMigrationJob` + - `table_name`: `events` + - `column_name`: `id` + - `job_arguments`: `[["id"], ["id_convert_to_bigint"]]` + + When dealing with multiple arguments, such as `[["id"],["id_convert_to_bigint"]]`, escape the + comma between each argument with a backslash <code>\</code> to prevent an invalid character error. + The command should be: + + ```shell + sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]'] + ``` + +::EndTabs + +#### Mark a failed migration finished + +WARNING: +[Contact GitLab Support](https://about.gitlab.com/support/#contact-support) before using +these instructions. This action can cause data loss, and make your instance fail +in ways that are difficult to recover from. + +There can be cases where the background migration fails: when jumping too many version upgrades, +or backward-incompatible database schema changes. (For an example, see [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216)). +Failed background migrations prevent further application upgrades. + +When the background migration is determined to be "safe" to skip, the migration can be manually marked finished: + +WARNING: +Make sure you create a backup before proceeding. + +```ruby +# Start the rails console + +connection = ApplicationRecord.connection # or Ci::ApplicationRecord.connection, depending on which DB was the migration scheduled + +Gitlab::Database::SharedModel.using_connection(connection) do + migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( + Gitlab::Database.gitlab_schemas_for_connection(connection), + 'BackfillUserDetailsFields', + :users, + :id, + [] + ) + + # mark all jobs completed + migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value) + migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) +end +``` + ## Background migrations In GitLab 13, background migrations were not batched. In GitLab 14 and later, this @@ -303,6 +416,8 @@ sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::Ba ## Troubleshooting +<!-- Linked from lib/gitlab/database/migrations/batched_background_migration_helpers.rb --> + ### Database migrations failing because of batched background migration not finished When updating to GitLab version 14.2 or later, database migrations might fail with a message like: @@ -319,8 +434,8 @@ Expected batched background migration for the given configuration to be marked a } ``` -First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/index.md#1420). -If you have, you can [manually finish the batched background migration](#manually-finishing-a-batched-background-migration). +First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/versions/gitlab_14_changes.md#1420). +If you have, you can [manually finish the batched background migration](#finish-a-failed-migration-manually)). If you haven't, choose one of the following methods: 1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required @@ -334,7 +449,7 @@ version and manually ensuring that the batched migrations complete successfully. 1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations and make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, -you can [manually finish them](#manually-finishing-a-batched-background-migration). +you can [manually finish them](#finish-a-failed-migration-manually). #### Roll forward and finish the migrations on the upgraded version @@ -344,7 +459,7 @@ To run all the batched background migrations, it can take a significant amount o depending on the size of your GitLab installation. 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations in the -database, and [manually run them](#manually-finishing-a-batched-background-migration) with the appropriate +database, and [manually run them](#finish-a-failed-migration-manually) with the appropriate arguments until the status query returns no rows. 1. When the status of all of all them is marked as complete, re-run migrations for your installation. 1. [Complete the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade: @@ -368,89 +483,9 @@ version and wait for the batched background migrations to finish. 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migration from the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, -or [manually finish it](#manually-finishing-a-batched-background-migration). +or [manually finish it](#finish-a-failed-migration-manually). 1. Re-run migrations for your installation, so the remaining post-deployment migrations finish. -### Manually finishing a batched background migration - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62634) in GitLab 14.1 - -If you need to manually finish a batched background migration due to an -error, you can run: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>'] -``` - -Replace the values in angle brackets with the correct -arguments. For example, if you receive an error similar to this: - -```plaintext -StandardError: An error has occurred, all later migrations canceled: - -Expected batched background migration for the given configuration to be marked as -'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", - :table_name=>"push_event_payloads", - :column_name=>"event_id", - :job_arguments=>[["event_id"], - ["event_id_convert_to_bigint"]] - } -``` - -Plug the arguments from the error message into the command: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] -``` - -If you need to manually run a batched background migration to continue an upgrade, you can -[check the status](#check-the-status-of-batched-background-migrations) in the database and get the -arguments from the query results. For example, if the query returns this: - -```plaintext - job_class_name | table_name | column_name | job_arguments ----------------------------------------+------------+-------------+------------------------------------ - CopyColumnUsingBackgroundMigrationJob | events | id | [["id"], ["id_convert_to_bigint"]] - ``` - -The results from the query can be plugged into the command: - -```shell -sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]'] -``` - -#### Mark a batched migration finished - -There can be cases where the background migration fails: when jumping too many version upgrades, -or backward-incompatible database schema changes. (For an example, see [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216)). -Failed background migrations prevent further application upgrades. - -When the background migration is determined to be "safe" to skip, the migration can be manually marked finished: - -WARNING: -Make sure you create a backup before proceeding. - -```ruby -# Start the rails console - -connection = ApplicationRecord.connection # or Ci::ApplicationRecord.connection, depending on which DB was the migration scheduled - -Gitlab::Database::SharedModel.using_connection(connection) do - migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( - Gitlab::Database.gitlab_schemas_for_connection(connection), - 'BackfillUserDetailsFields', - :users, - :id, - [] - ) - - # mark all jobs completed - migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value) - migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) -end -``` - ### The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job @@ -516,26 +551,26 @@ of Sidekiq jobs that perform various database or file updates. `BackfillDraftStatusOnMergeRequests` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match the migration's target. To clean up this stuck migration, see the - [14.2.0 version-specific instructions](index.md#1420). + [14.2.0 version-specific instructions](versions/gitlab_14_changes.md#1420). - GitLab 14.4 introduced an issue where a background migration named `PopulateTopicsTotalProjectsCountCache` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match the migration's target. To clean up this stuck migration, see the - [14.4.0 version-specific instructions](index.md#1440). + [14.4.0 version-specific instructions](versions/gitlab_14_changes.md#1440). - GitLab 14.5 introduced an issue where a background migration named `UpdateVulnerabilityOccurrencesLocation` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match the migration's target. To clean up this stuck migration, see the - [14.5.0 version-specific instructions](index.md#1450). + [14.5.0 version-specific instructions](versions/gitlab_14_changes.md#1450). - GitLab 14.8 introduced an issue where a background migration named `PopulateTopicsNonPrivateProjectsCount` can be permanently stuck in a **pending** state across upgrades. To clean up this stuck migration, see the - [14.8.0 version-specific instructions](index.md#1480). + [14.8.0 version-specific instructions](versions/gitlab_14_changes.md#1480). - GitLab 14.9 introduced an issue where a background migration named `ResetDuplicateCiRunnersTokenValuesOnProjects` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match the migration's target. To clean up this stuck migration, see the - [14.9.0 version-specific instructions](index.md#1490). + [14.9.0 version-specific instructions](versions/gitlab_14_changes.md#1490). For other background migrations stuck in pending, run the following check. If it returns non-zero and the count does not decrease over time, follow the rest diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index 08a5a3a7549..153119da1cc 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -46,6 +46,108 @@ For deprecation reviewers (Technical Writers only): {::options parse_block_html="true" /} <div class="js-deprecation-filters"></div> +<div class="milestone-wrapper" data-milestone="18.0"> + +## GitLab 18.0 + +<div class="deprecation breaking-change" data-milestone="18.0"> + +### GitLab Runner registration token in Runner Operator + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.6</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/382077). +</div> + +The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and Kubernetes Vanilla Operator to install a runner on Kubernetes is deprecated. Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments, +will be removed in GitLab 18.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md). +The configuration arguments disabled for authentication tokens are: + +- `--locked` +- `--access-level` +- `--run-untagged` +- `--tag-list` + +This change is a breaking change. You should use an [authentication token](../ci/runners/register_runner.md) in the `gitlab-runner register` command instead. + +</div> + +<div class="deprecation breaking-change" data-milestone="18.0"> + +### Registration tokens and server-side runner arguments in `gitlab-runner register` command + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.6</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/380872). +</div> + +Registration tokens and certain configuration arguments in the command `gitlab-runner register` that [registers](https://docs.gitlab.com/runner/register/) a runner, are deprecated. +Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments, +will be removed in GitLab 18.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md). +The configuration arguments disabled for authentication tokens are: + +- `--locked` +- `--access-level` +- `--run-untagged` +- `--maximum-timeout` +- `--paused` +- `--tag-list` +- `--maintenance-note` + +This change is a breaking change. You should [create a runner in the UI](../ci/runners/register_runner.md) to add configurations, and use the authentication token in the `gitlab-runner register` command instead. + +</div> + +<div class="deprecation breaking-change" data-milestone="18.0"> + +### Support for REST API endpoints that reset runner registration tokens + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.7</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383341). +</div> + +The support for runner registration tokens is deprecated. As a consequence, the REST API endpoints to reset a registration token are also deprecated and will +return the HTTP `410 Gone` status code in GitLab 18.0. +The deprecated endpoints are: + +- `POST /runners/reset_registration_token` +- `POST /projects/:id/runners/reset_registration_token` +- `POST /groups/:id/runners/reset_registration_token` + +We plan to implement a new method to bind runners to a GitLab instance +as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). +This new architecture introduces a new method for registering runners and will eliminate the legacy +[runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens). +From GitLab 18.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods. + +</div> + +<div class="deprecation breaking-change" data-milestone="18.0"> + +### `runnerRegistrationToken` parameter for GitLab Runner Helm Chart + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.6</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381111). +</div> + +The [`runnerRegistrationToken`](https://docs.gitlab.com/runner/install/kubernetes.html#required-configuration) parameter to use the GitLab Helm Chart to install a runner on Kubernetes is deprecated. + +We plan to implement a new method to bind runners to a GitLab instance leveraging `runnerToken` +as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). + +From GitLab 18.0 and later, the methods to register runners introduced by the new GitLab Runner token architecture will be the only supported methods. + +</div> +</div> + <div class="milestone-wrapper" data-milestone="17.0"> ## GitLab 17.0 @@ -131,6 +233,30 @@ These three variables will be removed in GitLab 17.0. <div class="deprecation breaking-change" data-milestone="17.0"> +### Default CI/CD job token (`CI_JOB_TOKEN`) scope changed + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.9</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383084). +</div> + +In GitLab 14.4 we introduced the ability to [limit your project's CI/CD job token](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#limit-your-projects-job-token-access) (`CI_JOB_TOKEN`) access to make it more secure. You can prevent job tokens **from your project's** pipelines from being used to **access other projects**. When enabled with no other configuration, your pipelines cannot access other projects. To use the job token to access other projects from your pipeline, you must list those projects explicitly in the **Limit CI_JOB_TOKEN access** setting's allowlist, and you must be a maintainer in all the projects. + +The job token functionality was updated in 15.9 with a better security setting to [allow access to your project with a job token](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#allow-access-to-your-project-with-a-job-token). When enabled with no other configuration, job tokens **from other projects** cannot **access your project**. Similar to the older setting, you can optionally allow other projects to access your project with a job token if you list those projects explicitly in the **Allow access to this project with a CI_JOB_TOKEN** setting's allowlist. With this new setting, you must be a maintainer in your own project, but only need to have the Guest role in the other projects. + +The **Limit** setting was deprecated in 16.0 in preference of the better **Allow access** setting and **Limit** setting was disabled by default for all new projects. From this point forward, if the **Limit** setting is disabled in any project, it will not be possible to re-enable this setting in 16.0 or later. + +In 17.0, we will remove the **Limit** setting completely, and set the **Allow access** setting to enabled for all projects. This change ensures a higher level of security between projects. If you currently use the **Limit** setting, you should update your projects to use the **Allow access** setting instead. If other projects access your project with a job token, you must add them to the **Allow access** allowlist. + +To prepare for this change, users on GitLab.com or self-managed GitLab 15.9 or later can enable the **Allow access** setting now and add the other projects. It will not be possible to disable the setting in 17.0 or later. + +In 16.3, the names of these settings were changed to clarify their meanings: the deprecated **Limit CI_JOB_TOKEN access** setting is now called **Limit access _from_ this project**, and the newer **Allow access to this project with a CI_JOB_TOKEN** setting is now called **Limit access _to_ this project**. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### Deprecate Windows CMD in GitLab Runner <div class="deprecation-notes"> @@ -189,6 +315,20 @@ The GitLab Runner Kubernetes executor setting, `terminationGracePeriodSeconds`, <div class="deprecation breaking-change" data-milestone="17.0"> +### Deprecate change vulnerability status from the Developer role + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.4</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424133). +</div> + +The ability for Developers to change the status of vulnerabilities is now deprecated. We plan to make a breaking change in the upcoming GitLab 17.0 release to remove this ability from the Developer role. Users who wish to continue to grant this permission to developers can [create a custom role](https://docs.gitlab.com/ee/user/permissions.html#custom-roles) for their developers and add in the `admin_vulnerability` permission to give them this access. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### Deprecate field `hasSolutions` from GraphQL VulnerabilityType <div class="deprecation-notes"> @@ -273,6 +413,25 @@ To avoid any disruptions, you should replace `filepath` with `direct_asset_path` <div class="deprecation breaking-change" data-milestone="17.0"> +### Geo: Legacy replication details routes for designs and projects deprecated + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.4</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424002). +</div> + +As part of the migration of legacy data types to the [Geo self-service framework](https://docs.gitlab.com/ee/development/geo/framework.html), the following replication details routes are deprecated: + +- Designs `/admin/geo/replication/designs` replaced by `/admin/geo/sites/<Geo Node/Site ID>/replication/design_management_repositories` +- Projects `/admin/geo/replication/projects` replaced by `/admin/geo/sites/<Geo Node/Site ID>/replication/projects` + +From GitLab 16.4 to 17.0, lookups for the legacy routes will automatically be redirected to the new routes. We will remove the redirections in 17.0. Please update any bookmarks or scripts that may use the legacy routes. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### GitLab Helm chart values `gitlab.kas.privateApi.*` are deprecated <div class="deprecation-notes"> @@ -310,29 +469,6 @@ are deprecated and will be removed from the GraphQL API. For installation instru <div class="deprecation breaking-change" data-milestone="17.0"> -### GitLab Runner registration token in Runner Operator - -<div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.6</span> -- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/382077). -</div> - -The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and Kubernetes Vanilla Operator to install a runner on Kubernetes is deprecated. Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments, -will be removed in GitLab 17.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md). -The configuration arguments disabled for authentication tokens are: - -- `--locked` -- `--access-level` -- `--run-untagged` -- `--tag-list` - -This change is a breaking change. You should use an [authentication token](../ci/runners/register_runner.md) in the `gitlab-runner register` command instead. - -</div> - -<div class="deprecation breaking-change" data-milestone="17.0"> - ### GraphQL deprecation of `dependencyProxyTotalSizeInBytes` field <div class="deprecation-notes"> @@ -381,6 +517,20 @@ Use `totalIssueWeight` instead, introduced in GitLab 16.2. <div class="deprecation breaking-change" data-milestone="17.0"> +### GraphQL networkPolicies resource deprecated + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">14.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/421440). +</div> + +The `networkPolicies` [GraphQL resource](https://docs.gitlab.com/ee/api/graphql/reference/#projectnetworkpolicies) has been deprecated and will be removed in GitLab 17.0. Since GitLab 15.0 this field has returned no data. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### GraphQL type, `RunnerMembershipFilter` renamed to `CiRunnerMembershipFilter` <div class="deprecation-notes"> @@ -410,6 +560,61 @@ In GitLab 17.0, the `DISABLED_WITH_OVERRIDE` value of the `SharedRunnersSetting` <div class="deprecation breaking-change" data-milestone="17.0"> +### HashiCorp Vault integration will no longer use CI_JOB_JWT by default + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.9</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798). +</div> + +As part of our effort to improve the security of your CI workflows using JWT and OIDC, the native HashiCorp integration is also being updated in GitLab 16.0. Any projects that use the [`secrets:vault`](https://docs.gitlab.com/ee/ci/yaml/#secretsvault) keyword to retrieve secrets from Vault will need to be [configured to use the ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). ID tokens were introduced in 15.7. + +To prepare for this change, use the new [`id_tokens`](https://docs.gitlab.com/ee/ci/yaml/#id_tokens) +keyword and configure the `aud` claim. Ensure the bound audience is prefixed with `https://`. + +In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication) +setting, which prevents the old tokens from being exposed to any jobs and enables +[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). + +In GitLab 16.0 and later: + +- This setting will be removed. +- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`, + and will not have any `CI_JOB_JWT*` tokens available. +- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*` + tokens available until GitLab 17.0. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + +### Internal Container Registry API tag deletion endpoint + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.4</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/container-registry/-/issues/1094). +</div> + +The [Docker Registry HTTP API V2 Spec](https://docs.docker.com/registry/spec/api/), later replaced by the [OCI Distribution Spec](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) did not include a tag delete operation, and an unsafe and slow workaround (involving deleting manifests, not tags) had to be used to achieve the same end. + +Tag deletion is an important function, so we added a tag deletion operation to the GitLab Container Registry, extending the V2 API beyond the scope of the Docker and OCI distribution spec. + +Since then, the OCI Distribution Spec has had some updates and it now has a tag delete operation, using the [`DELETE /v2/<name>/manifests/<tag>` endpoint](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#deleting-tags). + +This leaves the container registry with two endpoints that provide the exact same functionality. `DELETE /v2/<name>/tags/reference/<tag>` is the custom GitLab tag delete endpoint and `DELETE /v2/<name>/manifests/<tag>`, the OCI compliant tag delete endpoint introduced in GitLab 16.4. + +Support for the custom GitLab tag delete endpoint is deprecated in GitLab 16.4, and it will be removed in GitLab 17.0. + +This endpoint is used by the **internal** Container Registry application API, not the public [GitLab Container Registry API](https://docs.gitlab.com/ee/api/container_registry.html). No action should be required by the majority of container registry users. All the GitLab UI and API functionality related to tag deletions will remain intact as we transition to the new OCI-compliant endpoint. + +If you do access the internal container registry API and use the original tag deletion endpoint, you must update to the new endpoint. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### Maintainer role providing the ability to change Package settings using GraphQL API <div class="deprecation-notes"> @@ -432,6 +637,48 @@ settings for the group using either the GitLab UI or GraphQL API. <div class="deprecation breaking-change" data-milestone="17.0"> +### Old versions of JSON web tokens are deprecated + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.9</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798). +</div> + +[ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html) with OIDC support +were introduced in GitLab 15.7. These tokens are more configurable than the old JSON web tokens (JWTs), are OIDC compliant, +and only available in CI/CD jobs that explictly have ID tokens configured. +ID tokens are more secure than the old `CI_JOB_JWT*` JSON web tokens which are exposed in every job, +and as a result these old JSON web tokens are deprecated: + +- `CI_JOB_JWT` +- `CI_JOB_JWT_V1` +- `CI_JOB_JWT_V2` + +To prepare for this change, configure your pipelines to use [ID tokens](https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens) +instead of the deprecated tokens. For OIDC compliance, the `iss` claim now uses +the fully qualified domain name, for example `https://example.com`, previously +introduced with the `CI_JOB_JWT_V2` token. + +In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication) +setting, which prevents the old tokens from being exposed to any jobs and enables +[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). + +In GitLab 16.0 and later: + +- This setting will be removed. +- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`, + and will not have any `CI_JOB_JWT*` tokens available. +- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*` + tokens available until GitLab 17.0. + +In GitLab 17.0, the deprecated tokens will be completely removed and will no longer +be available in CI/CD jobs. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### OmniAuth Facebook is deprecated <div class="deprecation-notes"> @@ -544,33 +791,6 @@ This change is a breaking change. You should [create a runner in the UI](../ci/r <div class="deprecation breaking-change" data-milestone="17.0"> -### Registration tokens and server-side runner arguments in `gitlab-runner register` command - -<div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.6</span> -- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/380872). -</div> - -Registration tokens and certain configuration arguments in the command `gitlab-runner register` that [registers](https://docs.gitlab.com/runner/register/) a runner, are deprecated. -Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments, -will be removed in GitLab 17.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md). -The configuration arguments disabled for authentication tokens are: - -- `--locked` -- `--access-level` -- `--run-untagged` -- `--maximum-timeout` -- `--paused` -- `--tag-list` -- `--maintenance-note` - -This change is a breaking change. You should [create a runner in the UI](../ci/runners/register_runner.md) to add configurations, and use the authentication token in the `gitlab-runner register` command instead. - -</div> - -<div class="deprecation breaking-change" data-milestone="17.0"> - ### Required Pipeline Configuration is deprecated <div class="deprecation-notes"> @@ -678,56 +898,43 @@ we'll be introducing support in [this epic](https://gitlab.com/groups/gitlab-org <div class="deprecation breaking-change" data-milestone="17.0"> -### Support for REST API endpoints that reset runner registration tokens +### The GitLab legacy requirement IID is deprecated in favor of work item IID <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.7</span> +- Announced in GitLab <span class="milestone">15.9</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383341). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390263). </div> -The support for runner registration tokens is deprecated. As a consequence, the REST API endpoints to reset a registration token are also deprecated and will -return the HTTP `410 Gone` status code in GitLab 17.0. -The deprecated endpoints are: - -- `POST /runners/reset_registration_token` -- `POST /projects/:id/runners/reset_registration_token` -- `POST /groups/:id/runners/reset_registration_token` - -We plan to implement a new method to bind runners to a GitLab instance -as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). -The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). -This new architecture introduces a new method for registering runners and will eliminate the legacy -[runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens). -From GitLab 17.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods. +We will be transitioning to a new IID as a result of moving requirements to a [work item type](https://docs.gitlab.com/ee/development/work_items.html#work-items-and-work-item-types). Users should begin using the new IID as support for the legacy IID and existing formatting will end in GitLab 17.0. The legacy requirement IID remains available until its removal in GitLab 17.0. </div> <div class="deprecation breaking-change" data-milestone="17.0"> -### The GitLab legacy requirement IID is deprecated in favor of work item IID +### The Visual Reviews tool is deprecated <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.9</span> +- Announced in GitLab <span class="milestone">15.8</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390263). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387751). </div> -We will be transitioning to a new IID as a result of moving requirements to a [work item type](https://docs.gitlab.com/ee/development/work_items.html#work-items-and-work-item-types). Users should begin using the new IID as support for the legacy IID and existing formatting will end in GitLab 17.0. The legacy requirement IID remains available until its removal in GitLab 17.0. +Due to limited customer usage and capabilities, the Visual Reviews feature for Review Apps is deprecated and will be removed. There is no planned replacement and users should stop using Visual Reviews before GitLab 17.0. </div> <div class="deprecation breaking-change" data-milestone="17.0"> -### The Visual Reviews tool is deprecated +### The `ci_job_token_scope_enabled` projects API attribute is deprecated <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.8</span> +- Announced in GitLab <span class="milestone">16.4</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387751). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423091). </div> -Due to limited customer usage and capabilities, the Visual Reviews feature for Review Apps is deprecated and will be removed. There is no planned replacement and users should stop using Visual Reviews before GitLab 17.0. +GitLab 16.1 introduced [API endpoints for the job token scope](https://gitlab.com/gitlab-org/gitlab/-/issues/351740). In the [projects API](https://docs.gitlab.com/ee/api/projects.html), the `ci_job_token_scope_enabled` attribute is deprecated, and will be removed in 17.0. You should use the [job token scope APIs](https://docs.gitlab.com/ee/api/project_job_token_scopes.html) instead. </div> @@ -830,26 +1037,6 @@ removed in 17.0. <div class="deprecation breaking-change" data-milestone="17.0"> -### `runnerRegistrationToken` parameter for GitLab Runner Helm Chart - -<div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.6</span> -- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381111). -</div> - -The [`runnerRegistrationToken`](https://docs.gitlab.com/runner/install/kubernetes.html#required-configuration) parameter to use the GitLab Helm Chart to install a runner on Kubernetes is deprecated. - -We plan to implement a new method to bind runners to a GitLab instance leveraging `runnerToken` -as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). -The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). - -From GitLab 17.0 and later, the methods to register runners introduced by the new GitLab Runner token architecture will be the only supported methods. - -</div> - -<div class="deprecation breaking-change" data-milestone="17.0"> - ### `sidekiq` delivery method for `incoming_email` and `service_desk_email` is deprecated <div class="deprecation-notes"> @@ -909,92 +1096,42 @@ Previous work helped [align the vulnerabilities calls for pipeline security tabs </div> </div> -<div class="milestone-wrapper" data-milestone="16.5"> +<div class="milestone-wrapper" data-milestone="16.6"> -## GitLab 16.5 +## GitLab 16.6 -<div class="deprecation " data-milestone="16.5"> +<div class="deprecation breaking-change" data-milestone="16.6"> -### Adding non-LDAP synced members to a locked LDAP group is deprecated +### Job token allowlist covers public and internal projects <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">16.0</span> -- Removal in GitLab <span class="milestone">16.5</span> -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213311). -</div> - -Enabling the `ldap_settings_unlock_groups_by_owners` feature flag allowed non-LDAP synced users to be added to a locked LDAP group. This [feature](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) has always been disabled by default and behind a feature flag. We are removing this feature to keep continuity with our SAML integration, and because allowing non-synced group members defeats the "single source of truth" principle of using a directory service. Once this feature is removed, any LDAP group members that are not synced with LDAP will lose access to that group. - +- Announced in GitLab <span class="milestone">16.3</span> +- Removal in GitLab <span class="milestone">16.6</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/420678). </div> -<div class="deprecation breaking-change" data-milestone="16.5"> +Starting in 16.6, projects that are **public** or **internal** will no longer authorize job token requests from projects that are **not** on the project's allowlist when [**Limit access to this project**](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#allow-access-to-your-project-with-a-job-token) is enabled. -### HashiCorp Vault integration will no longer use CI_JOB_JWT by default +If you have [public or internal](https://docs.gitlab.com/ee/user/public_access.html#change-project-visibility) projects with the **Limit access to this project** setting enabled, you must add any projects which make job token requests to your project's allowlist for continued authorization. -<div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.9</span> -- Removal in GitLab <span class="milestone">16.5</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798). +</div> </div> -As part of our effort to improve the security of your CI workflows using JWT and OIDC, the native HashiCorp integration is also being updated in GitLab 16.0. Any projects that use the [`secrets:vault`](https://docs.gitlab.com/ee/ci/yaml/#secretsvault) keyword to retrieve secrets from Vault will need to be [configured to use the ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). ID tokens were introduced in 15.7. - -To prepare for this change, use the new [`id_tokens`](https://docs.gitlab.com/ee/ci/yaml/#id_tokens) -keyword and configure the `aud` claim. Ensure the bound audience is prefixed with `https://`. - -In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication) -setting, which prevents the old tokens from being exposed to any jobs and enables -[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). - -In GitLab 16.0 and later: - -- This setting will be removed. -- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`, - and will not have any `CI_JOB_JWT*` tokens available. -- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*` - tokens available until GitLab 16.5. +<div class="milestone-wrapper" data-milestone="16.5"> -</div> +## GitLab 16.5 -<div class="deprecation breaking-change" data-milestone="16.5"> +<div class="deprecation " data-milestone="16.5"> -### Old versions of JSON web tokens are deprecated +### Adding non-LDAP synced members to a locked LDAP group is deprecated <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.9</span> -- Removal in GitLab <span class="milestone">16.5</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798). +- Announced in GitLab <span class="milestone">16.0</span> +- Removal in GitLab <span class="milestone">16.5</span> +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213311). </div> -[ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html) with OIDC support -were introduced in GitLab 15.7. These tokens are more configurable than the old JSON web tokens (JWTs), are OIDC compliant, -and only available in CI/CD jobs that explictly have ID tokens configured. -ID tokens are more secure than the old `CI_JOB_JWT*` JSON web tokens which are exposed in every job, -and as a result these old JSON web tokens are deprecated: - -- `CI_JOB_JWT` -- `CI_JOB_JWT_V1` -- `CI_JOB_JWT_V2` - -To prepare for this change, configure your pipelines to use [ID tokens](https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens) -instead of the deprecated tokens. For OIDC compliance, the `iss` claim now uses -the fully qualified domain name, for example `https://example.com`, previously -introduced with the `CI_JOB_JWT_V2` token. - -In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication) -setting, which prevents the old tokens from being exposed to any jobs and enables -[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). - -In GitLab 16.0 and later: - -- This setting will be removed. -- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`, - and will not have any `CI_JOB_JWT*` tokens available. -- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*` - tokens available until GitLab 16.5. - -In GitLab 16.5, the deprecated tokens will be completely removed and will no longer -be available in CI/CD jobs. +Enabling the `ldap_settings_unlock_groups_by_owners` feature flag allowed non-LDAP synced users to be added to a locked LDAP group. This [feature](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) has always been disabled by default and behind a feature flag. We are removing this feature to keep continuity with our SAML integration, and because allowing non-synced group members defeats the "single source of truth" principle of using a directory service. Once this feature is removed, any LDAP group members that are not synced with LDAP will lose access to that group. </div> </div> @@ -1038,9 +1175,19 @@ However, enabling the bundled Grafana will no longer work from GitLab 16.3. - To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387561). </div> -**Update:** We previously announced we would remove the existing License Compliance CI template in GitLab 16.0. However, due to performance issues with the [license scanning of CycloneDX files](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/) we will do this change in 16.3 instead. +**Update:** We previously announced we would remove the existing License Compliance CI template in GitLab 16.0. However, due to performance issues with the [license scanning of CycloneDX files](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/) we will do this in 16.3 instead. + +The GitLab [**License Compliance**](https://docs.gitlab.com/ee/user/compliance/license_compliance/) CI/CD template is now deprecated and is scheduled for removal in the GitLab 16.3 release. + +To continue using GitLab for license compliance, remove the **License Compliance** template from your CI/CD pipeline and add the **Dependency Scanning** template. The **Dependency Scanning** template is now capable of gathering the required license information, so it is no longer necessary to run a separate license compliance job. + +Before you remove the **License Compliance** CI/CD template, verify that the instance has been upgraded to a version that supports the new method of license scanning. + +To begin using the Dependency Scanner quickly at scale, you may set up a scan execution policy at the group level to enforce the SBOM-based license scan for all projects in the group. Then, you may remove the inclusion of the `Jobs/License-Scanning.gitlab-ci.yml` template from your CI/CD configuration. + +If you wish to continue using the legacy license compliance feature, you can do so by setting the `LICENSE_MANAGEMENT_VERSION CI` variable to `4`. This variable can be set at the project, group, or instance level. This configuration change will allow you to continue using an existing version of license compliance without having to adopt the new approach. -The GitLab [License Compliance](https://docs.gitlab.com/ee/user/compliance/license_compliance/) CI template is now deprecated and is scheduled for removal in the GitLab 16.1 release. Users who wish to continue using GitLab for License Compliance should remove the License Compliance template from their CI pipeline and add the [Dependency Scanning template](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#configuration). The Dependency Scanning template is now capable of gathering the required license information so it is no longer necessary to run a separate License Compliance job. The License Compliance CI template should not be removed prior to verifying that the `license_scanning_sbom_scanner` and `package_metadata_synchronization` flags are enabled for the instance and that the instance has been upgraded to a version that supports [the new method of license scanning](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/). +Bugs and vulnerabilities in this legacy analyzer will no longer be fixed. | CI Pipeline Includes | GitLab <= 15.8 | 15.9 <= GitLab < 16.3 | GitLab >= 16.3 | | ------------- | ------------- | ------------- | ------------- | @@ -1052,6 +1199,22 @@ The GitLab [License Compliance](https://docs.gitlab.com/ee/user/compliance/licen <div class="deprecation breaking-change" data-milestone="16.3"> +### RSA key size limits + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.3</span> +- Removal in GitLab <span class="milestone">16.3</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/groups/gitlab-org/-/epics/11186). +</div> + +Go versions 1.20.7 and later add a `maxRSAKeySize` constant that limits RSA keys to a maximum of 8192 bits. As a result, RSA keys larger than 8192 bits will no longer work with GitLab. Any RSA keys larger than 8192 bits must be regenerated at a smaller size. + +You might notice this issue because your logs include an error like `tls: server sent certificate containing RSA key larger than 8192 bits`. To test the length of your key, use this command: `openssl rsa -in <your-key-file> -text -noout | grep "Key:"`. + +</div> + +<div class="deprecation breaking-change" data-milestone="16.3"> + ### Twitter OmniAuth login option is removed from GitLab.com <div class="deprecation-notes"> @@ -3376,7 +3539,7 @@ By default, all new applications expire access tokens after 2 hours. In GitLab 1 had no expiration. In GitLab 15.0, an expiry will be automatically generated for any existing token that does not already have one. -You should [opt in](https://docs.gitlab.com/ee/integration/oauth_provider.html#expiring-access-tokens) to expiring +You should [opt in](https://docs.gitlab.com/ee/integration/oauth_provider.html#access-token-expiration) to expiring tokens before GitLab 15.0 is released: 1. Edit the application. diff --git a/doc/update/index.md b/doc/update/index.md index 5a0fd0bdf31..b821be3deea 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -25,12 +25,9 @@ has additional information about upgrading, including: Depending on the installation method and your GitLab version, there are multiple official ways to upgrade GitLab: -- [Linux packages (Omnibus)](#linux-packages-omnibus) -- [Self-compiled installations](#self-compiled-installation) -- [Docker installations](#installation-using-docker) -- [Kubernetes (Helm) installations](#installation-using-helm) +::Tabs -### Linux packages (Omnibus) +:::TabTitle Linux packages (Omnibus) The [package upgrade guide](package/index.md) contains the steps needed to upgrade a package installed by official GitLab @@ -39,12 +36,27 @@ repositories. There are also instructions when you want to [upgrade to a specific version](package/index.md#upgrade-to-a-specific-version-using-the-official-repositories). -### Self-compiled installation +:::TabTitle Helm chart (Kubernetes) + +GitLab can be deployed into a Kubernetes cluster using Helm. +Instructions on how to upgrade a cloud-native deployment are in +[a separate document](https://docs.gitlab.com/charts/installation/upgrade.html). + +Use the [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html) +from the chart version to GitLab version to determine the [upgrade path](#upgrade-paths). + +:::TabTitle Docker + +GitLab provides official Docker images for both Community and Enterprise +editions, and they are based on the Omnibus package. See how to +[install GitLab using Docker](../install/docker.md). + +:::TabTitle Self-compiled (source) - [Upgrading Community Edition and Enterprise Edition from source](upgrading_from_source.md) - The guidelines for upgrading Community Edition and Enterprise Edition from source. - [Patch versions](patch_versions.md) guide includes the steps needed for a - patch version, such as 13.2.0 to 13.2.1, and apply to both Community and Enterprise + patch version, such as 15.2.0 to 15.2.1, and apply to both Community and Enterprise Editions. In the past we used separate documents for the upgrading instructions, but we @@ -54,20 +66,7 @@ can still be found in the Git repository: - [Old upgrading guidelines for Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/tree/11-8-stable/doc/update) - [Old upgrading guidelines for Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/tree/11-8-stable-ee/doc/update) -### Installation using Docker - -GitLab provides official Docker images for both Community and Enterprise -editions, and they are based on the Omnibus package. See how to -[install GitLab using Docker](../install/docker.md). - -### Installation using Helm - -GitLab can be deployed into a Kubernetes cluster using Helm. -Instructions on how to upgrade a cloud-native deployment are in -[a separate document](https://docs.gitlab.com/charts/installation/upgrade.html). - -Use the [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html) -from the chart version to GitLab version to determine the [upgrade path](#upgrade-paths). +::EndTabs ## Plan your upgrade @@ -188,9 +187,13 @@ When upgrading: 1. Find where your version sits in the upgrade path: - - GitLab 14: [`14.0.12`](#1400) > [`14.3.6`](#1430) > [`14.9.5`](#1490) > [`14.10.5`](#14100). - - GitLab 15: [`15.0.5`](#1500) > [`15.1.6`](#1510) (for GitLab instances with multiple web nodes) > [`15.4.6`](#1540) > [`15.11.13`](#15110). - - GitLab 16: [`16.0.x`](versions/gitlab_16_changes.md#1600) (only [instances with lots of users](versions/gitlab_16_changes.md#long-running-user-type-data-change)) > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases). + - GitLab 14: [`14.0.12`](versions/gitlab_14_changes.md#1400) > [`14.3.6`](versions/gitlab_14_changes.md#1430) > + [`14.9.5`](versions/gitlab_14_changes.md#1490) > [`14.10.5`](versions/gitlab_14_changes.md#14100). + - GitLab 15: [`15.0.5`](versions/gitlab_15_changes.md#1500) > [`15.1.6`](versions/gitlab_15_changes.md#1510) (for + GitLab instances with multiple web nodes) > [`15.4.6`](versions/gitlab_15_changes.md#1540) > + [`15.11.13`](versions/gitlab_15_changes.md#15110). + - GitLab 16: [`16.0.x`](versions/gitlab_16_changes.md#1600) (only + [instances with lots of users](versions/gitlab_16_changes.md#long-running-user-type-data-change)) > [`16.3`](versions/gitlab_16_changes.md#1630) > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases). 1. Check for [required upgrade stops](#required-upgrade-stops). 1. Consult the [version-specific upgrade instructions](#version-specific-upgrading-instructions). @@ -209,11 +212,9 @@ crucial database schema and migration patches may be included in the latest patc Required upgrade stops are versions of GitLab that you must upgrade to before upgrading to later versions. Required upgrade stops allow required background migrations to finish. -During GitLab 16.x, we are scheduling two or three required upgrade stops. We will give at least two milestones of -notice when we schedule a required upgrade stop. +During GitLab 16.x, we are scheduling required upgrade stops beforehand so users can better plan out appropriate upgrade stops and downtime when necessary. -The first planned required upgrade stop is scheduled for GitLab 16.3. If nothing is introduced requiring an upgrade stop, -GitLab 16.3 will be treated as a regular upgrade. +The first scheduled required upgrade stop has been announced for 16.3. When planning upgrades, please take this into account. ### Earlier GitLab versions @@ -287,1499 +288,15 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### GitLab 16 -Before upgrading, see [GitLab 16 changes](versions/gitlab_16_changes.md). - -### 15.11.1 - -- Many [project importers](../user/project/import/index.md) and [group importers](../user/group/import/index.md) now - require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation - for any importers you use. -- Geo: Some project imports do not initialize wiki repositories on project creation. Since the migration of project wikis to SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). This is not a result of an actual replication/verification failure but an invalid internal state for these missing repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for these wiki repositories. If you have not imported projects you are not impacted by this issue. - - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2. - - Versions containing fix: GitLab 16.1.3 and later. -- Geo: A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database to version 13. This leaves the secondary site in a broken state, and prevents upgrading the Geo installation to GitLab 16.x ([PostgreSQL 12 support has removed in 16.0](deprecations.md#postgresql-12-deprecated) and later releases). This occurs on secondary sites using the bundled PostgreSQL software, running both the secondary main Rails database and tracking database on the same node. There is a manual [workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround) for those impacted until a fix is backported to 15.11. - - Impacted versions: GitLab versions 15.2 - 15.11 - - Versions containing fix: 15.11.12 and later. - - Version 16.0 and later are not impacted. Note, 15.11 is a mandatory upgrade stop on the way to 16.0. - -### 15.11.0 - -- **Upgrade to patch release 15.11.3 or later**. This avoids [issue 408304](https://gitlab.com/gitlab-org/gitlab/-/issues/408304) when upgrading from 15.5.0 and earlier. -- Geo: Some project imports do not initialize wiki repositories on project creation. Since the migration of project wikis to SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). This is not a result of an actual replication/verification failure but an invalid internal state for these missing repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for these wiki repositories. If you have not imported projects you are not impacted by this issue. - - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2. - - Versions containing fix: GitLab 16.1.3 and later. -- Geo: A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database to version 13. This leaves the secondary site in a broken state, and prevents upgrading the Geo installation to GitLab 16.x ([PostgreSQL 12 support has removed in 16.0](deprecations.md#postgresql-12-deprecated) and later releases). This occurs on secondary sites using the bundled PostgreSQL software, running both the secondary main Rails database and tracking database on the same node. There is a manual [workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround) for those impacted until a fix is backported to 15.11. - - Impacted versions: GitLab versions 15.2 - 15.11.11. - - Versions containing fix: 15.11.12 and later. - - Version 16.0 and later are not impacted. Note, 15.11 is a mandatory upgrade stop on the way to 16.0. - -### 15.11.x - -- A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/411604) can cause new LDAP users signing in for the first time to be assigned a username based on their email address instead of their LDAP username attribute. A manual workaround is to set `gitlab_rails['omniauth_auto_link_ldap_user'] = true`, or upgrade to GitLab 16.1 or later where the bug has been fixed. - -### 15.10.5 - -- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. - - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. - - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. - - Elasticsearch does not need to be enabled for this to occur. - - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. -- Many [project importers](../user/project/import/index.md) and [group importers](../user/group/import/index.md) now - require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation - for any importers you use. - -### 15.10.0 - -- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. - - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. - - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. - - Elasticsearch does not need to be enabled for this to occur. - - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. -- Gitaly configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.10 while backwards compatibility is - maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#gitaly-omnibus-gitlab-configuration-structure-change). -- You might encounter the following error while upgrading to GitLab 15.10 or later: - - ```shell - STDOUT: rake aborted! - StandardError: An error has occurred, all later migrations canceled: - PG::CheckViolation: ERROR: check constraint "check_70f294ef54" is violated by some row - ``` - - This error is caused by a [batched background migration introduced in GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107701) - not being finalized before GitLab 15.10. To resolve this error: - - 1. Execute the following SQL statement using the database console (`sudo gitlab-psql` for Linux package installs): - - ```sql - UPDATE oauth_access_tokens SET expires_in = '7200' WHERE expires_in IS NULL; - ``` - - 1. [Re-run database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations). - -- You might also encounter the following error while upgrading to GitLab 15.10 or later: - - ```shell - "exception.class": "ActiveRecord::StatementInvalid", - "exception.message": "PG::SyntaxError: ERROR: zero-length delimited identifier at or near \"\"\"\"\nLINE 1: ...COALESCE(\"lock_version\", 0) + 1 WHERE \"ci_builds\".\"\" IN (SEL...\n - ``` - - This error is caused by a [batched background migration introduced in GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81410) - not being finalized before upgrading to GitLab 15.10 or later. To resolve this error, it is safe to [mark the migration as complete](background_migrations.md#mark-a-batched-migration-finished): - - ```ruby - # Start the rails console - - connection = Ci::ApplicationRecord.connection - - Gitlab::Database::SharedModel.using_connection(connection) do - migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( - Gitlab::Database.gitlab_schemas_for_connection(connection), 'NullifyOrphanRunnerIdOnCiBuilds', :ci_builds, :id, []) - - # mark all jobs completed - migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states[:succeeded].value) - migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) - end - ``` - -For more information, see [issue 415724](https://gitlab.com/gitlab-org/gitlab/-/issues/415724). - -### 15.9.0 - -- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. - - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. - - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. - - Elasticsearch does not need to be enabled for this to occur. - - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. -- **Upgrade to patch release 15.9.3 or later**. This provides fixes for two database migration bugs: - - Patch releases 15.9.0, 15.9.1, 15.9.2 have [a bug that can cause data loss](#user-profile-data-loss-bug-in-159x) from the user profile fields. - - The second [bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/394760) ensures it is possible to upgrade directly from 15.4.x. -- As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. -- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) is now enabled by default. - - The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its - metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk. - After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica. - - Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete - deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as - unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove - the repository using `praefect remove-repository` to remove the repository's remaining records. - - You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. [Read more about repository verification, and to see an example log entry](../administration/gitaly/praefect.md#repository-verification). -- Praefect configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.9 while backwards compatibility is - maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#praefect-omnibus-gitlab-configuration-structure-change). -- For **self-compiled (source) installations**, with the addition of `gitlab-sshd` the Kerberos headers are needed to build GitLab Shell. - - ```shell - sudo apt install libkrb5-dev - ``` - -### 15.8.2 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. - -### 15.8.1 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.8.0 - -- Git 2.38.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../install/installation.md#git). -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. - -### 15.7.6 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.5 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.4 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.3 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.2 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the upgrades. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.1 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.7.0 - -- This version validates a `NOT NULL DB` constraint on the `issues.work_item_type_id` column. - To upgrade to this version, no records with a `NULL` `work_item_type_id` should exist on the `issues` table. - There are multiple `BackfillWorkItemTypeIdForIssues` background migrations that will be finalized with - the `EnsureWorkItemTypeBackfillMigrationFinished` post-deploy migration. -- GitLab 15.4.0 introduced a [batched background migration](background_migrations.md#batched-background-migrations) to - [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This - migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration - has completed successfully before upgrading to 15.7.0. -- A database constraint is added, specifying that the `namespace_id` column on the issues - table has no `NULL` values. - - - If the `namespace_id` batched background migration from 15.4 failed (see above) then the 15.7 upgrade - fails with a database migration error. - - - On GitLab instances with large issues tables, validating this constraint causes the upgrade to take - longer than usual. All database changes need to complete within a one-hour limit: - - ```plaintext - FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] - [..] - Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: - ``` - - A workaround exists to [complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). -- The default Sidekiq `max_concurrency` has been changed to 20. This is now - consistent in our documentation and product defaults. - - For example, previously: - - - Linux package installation default (`sidekiq['max_concurrency']`): 50 - - Self-compiled installation default: 50 - - Helm chart default (`gitlab.sidekiq.concurrency`): 25 - - Reference architectures still use a default of 10 as this is set specifically - for those configurations. - - Sites that have configured `max_concurrency` will not be affected by this change. - [Read more about the Sidekiq concurrency setting](../administration/sidekiq/extra_sidekiq_processes.md#concurrency). -- GitLab Runner 15.7.0 introduced a breaking change that affects CI/CD jobs: [Correctly handle expansion of job file variables](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3613). - Previously, job-defined variables that referred to - [file type variables](../ci/variables/index.md#use-file-type-cicd-variables) - were expanded to the value of the file variable (its content). This behavior did not - respect the typical rules of shell variable expansion. There was also the potential - that secrets or sensitive information could leak if the file variable and its - contents printed. For example, if they were printed in an echo output. For more information, - see [Understanding the file type variable expansion change in GitLab 15.7](https://about.gitlab.com/blog/2023/02/13/impact-of-the-file-type-variable-change-15-7/). -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. - -### 15.6.7 - -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.6 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.5 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.4 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6, and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.3 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.2 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.1 - -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.6.0 - -- You should use one of the [officially supported PostgreSQL versions](../administration/package_information/postgresql_versions.md). Some database migrations can cause stability and performance issues with older PostgreSQL versions. -- Git 2.37.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../install/installation.md#git). -- A database change to modify the behavior of four indexes fails on instances - where these indexes do not exist: - - ```plaintext - Caused by: - PG::UndefinedTable: ERROR: relation "index_issues_on_title_trigram" does not exist - ``` - - The other three indexes are: `index_merge_requests_on_title_trigram`, `index_merge_requests_on_description_trigram`, - and `index_issues_on_description_trigram`. - - This issue was [fixed in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105375) and backported - to GitLab 15.6.2. The issue can also be worked around: - [read about how to create these indexes](https://gitlab.com/gitlab-org/gitlab/-/issues/378343#note_1199863087). -- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. -- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. - - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. - - Versions containing fix: GitLab 15.8.3 and later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.5 - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.4 - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.3 - -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['routing_rules'] = [['*', 'default']] - ``` - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.2 - -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['routing_rules'] = [['*', 'default']] - ``` - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.1 - -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['routing_rules'] = [['*', 'default']] - ``` - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.5.0 - -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['routing_rules'] = [['*', 'default']] - ``` - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.6 - -- Due to a [bug introduced in curl in GitLab 15.4.6](https://github.com/curl/curl/issues/10122), the [`no_proxy` environment variable may not work properly](../administration/geo/replication/troubleshooting.md#secondary-site-returns-received-http-code-403-from-proxy-after-connect). Either downgrade to GitLab 15.4.5, or upgrade to GitLab 15.5.7 or a later version. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.5 - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.4 - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.3 - -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.2 - -- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. - - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.1 - -- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. - - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. - -### 15.4.0 - -- GitLab 15.4.0 includes a [batched background migration](background_migrations.md#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318). - This migration might take hours or days to complete on larger GitLab instances. -- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../administration/gitaly/praefect.md#customize-time-server-setting). -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - - The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior. - - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['routing_rules'] = [['*', 'default']] - ``` - -- New Git repositories created in Gitaly cluster [no longer use the `@hashed` storage path](#change-to-praefect-generated-replica-paths-in-gitlab-153). Server - hooks for new repositories must be copied into a different location. -- The structure of `/etc/gitlab/gitlab-secrets.json` was modified in [GitLab 15.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6310), - and new configuration was added to `gitlab_pages`, `grafana`, and `mattermost` sections. - In a highly available or GitLab Geo environment, secrets need to be the same on all nodes. - If you're manually syncing the secrets file across nodes, or manually specifying secrets in - `/etc/gitlab/gitlab.rb`, make sure `/etc/gitlab/gitlab-secrets.json` is the same on all nodes. -- GitLab 15.4.0 introduced a [batched background migration](background_migrations.md#batched-background-migrations) to - [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This - migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration - has completed successfully before upgrading to 15.7.0 or later. -- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. -- A redesigned sign-in page is enabled by default in GitLab 15.4 and later, with improvements shipping in later releases. For more information, see [epic 8557](https://gitlab.com/groups/gitlab-org/-/epics/8557). - It can be disabled with a feature flag. Start [a Rails console](../administration/operations/rails_console.md) and run: - - ```ruby - Feature.disable(:restyle_login_page) - ``` - -### 15.3.4 - -A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - -- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. -- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.3.3 - -- In GitLab 15.3.3, [SAML Group Links](../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See -[the API documentation](../api/members.md). -- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - - - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. - - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.3.2 - -A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - -- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. -- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.3.1 - -A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - -- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. -- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.3.0 - -- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532). -- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details. -- New Git repositories created in Gitaly cluster [no longer use the `@hashed` storage path](#change-to-praefect-generated-replica-paths-in-gitlab-153). Server - hooks for new repositories must be copied into a different location. -- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - - - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. - - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.2.5 - -A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - -- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. -- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: - - 15.2.5 --> 15.3.5 - - 15.3.0 - 15.3.4 --> 15.3.5 - - 15.4.1 --> 15.4.3 - -### 15.2.0 - -- GitLab installations that have multiple web nodes should be - [upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a - configuration change in Rails that can result in inconsistent ETag key - generation. -- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../administration/sidekiq/sidekiq_job_migration.md#migrate-queued-and-future-jobs) before starting the upgrade to GitLab 15.2.0. -- Gitaly now executes its binaries in a [runtime location](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4670). By default on Omnibus GitLab, - this path is `/var/opt/gitlab/gitaly/run/`. If this location is mounted with `noexec`, merge requests generate the following error: - - ```plaintext - fork/exec /var/opt/gitlab/gitaly/run/gitaly-<nnnn>/gitaly-git2go-v15: permission denied - ``` - - To resolve this, remove the `noexec` option from the file system mount. An alternative is to change the Gitaly runtime directory: - - 1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set. - 1. Run `sudo gitlab-ctl reconfigure`. -- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532). -- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details. - -### 15.1.0 - -- If you run external PostgreSQL, particularly AWS RDS, - [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) - to avoid the database crashing. -- In GitLab 15.1.0, we are switching Rails `ActiveSupport::Digest` to use SHA256 instead of MD5. - This affects ETag key generation for resources such as raw Snippet file - downloads. To ensure consistent ETag key generation across multiple - web nodes when upgrading, all servers must first be upgraded to 15.1.Z before - upgrading to 15.2.0 or later: - - 1. Ensure all GitLab web nodes are running GitLab 15.1.Z. - 1. [Enable the `active_support_hash_digest_sha256` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to switch `ActiveSupport::Digest` to use SHA256: - - 1. [Start the rails console](../administration/operations/rails_console.md) - 1. Enable the feature flag: - - ```ruby - Feature.enable(:active_support_hash_digest_sha256) - ``` - - 1. Only then, continue to upgrade to later versions of GitLab. -- Unauthenticated requests to the [`ciConfig` GraphQL field](../api/graphql/reference/index.md#queryciconfig) are no longer supported. - Before you upgrade to GitLab 15.1, add an [access token](../api/rest/index.md#authentication) to your requests. - The user creating the token must have [permission](../user/permissions.md) to create pipelines in the project. -- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532). -- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details. - -### 15.0.0 - -- Elasticsearch 6.8 [is no longer supported](../integration/advanced_search/elasticsearch.md#version-requirements). Before you upgrade to GitLab 15.0, [update Elasticsearch to any 7.x version](../integration/advanced_search/elasticsearch.md#upgrade-to-a-new-elasticsearch-major-version). -- If you run external PostgreSQL, particularly AWS RDS, - [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) - to avoid the database crashing. -- The use of encrypted S3 buckets with storage-specific configuration is no longer supported after [removing support for using `background_upload`](deprecations.md#background-upload-for-object-storage). -- The [certificate-based Kubernetes integration (DEPRECATED)](../user/infrastructure/clusters/index.md#certificate-based-kubernetes-integration-deprecated) is disabled by default, but you can be re-enable it through the [`certificate_based_clusters` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) until GitLab 16.0. -- When you use the GitLab Helm Chart project with a custom `serviceAccount`, ensure it has `get` and `list` permissions for the `serviceAccount` and `secret` resources. -- The [`custom_hooks_dir`](../administration/server_hooks.md#create-global-server-hooks-for-all-repositories) setting for configuring global server hooks is now configured in - Gitaly. The previous implementation in GitLab Shell was removed in GitLab 15.0. With this change, global server hooks are stored only inside a subdirectory named after the - hook type. Global server hooks can no longer be a single hook file in the root of the custom hooks directory. For example, you must use `<custom_hooks_dir>/<hook_name>.d/*` rather - than `<custom_hooks_dir>/<hook_name>`. - - Use `gitaly['custom_hooks_dir']` in `gitlab.rb` ([introduced in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4208)) - for Omnibus GitLab. This replaces `gitlab_shell['custom_hooks_dir']`. -- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532). -- The `FF_GITLAB_REGISTRY_HELPER_IMAGE` [feature flag](../administration/feature_flags.md#enable-or-disable-the-feature) is removed and helper images are always pulled from GitLab Registry. -- The `AES256-GCM-SHA384` SSL cipher is no longer allowed by NGINX. - See how you can [add the cipher back](https://docs.gitlab.com/omnibus/update/gitlab_15_changes.html#aes256-gcm-sha384-ssl-cipher-no-longer-allowed-by-default-by-nginx) to the allow list. -- Support for more than one database has been added to GitLab. For **self-compiled (source) installations**, - `config/database.yml` must include a database name in the database configuration. - The `main: database` must be first. If an invalid or deprecated syntax is used, an error is generated - during application start: - - ```plaintext - ERROR: This installation of GitLab uses unsupported 'config/database.yml'. - The main: database needs to be defined as a first configuration item instead of primary. (RuntimeError) - ``` - - Previously, the `config/database.yml` file looked like the following: - - ```yaml - production: - adapter: postgresql - encoding: unicode - database: gitlabhq_production - ... - ``` - - Starting with GitLab 15.0, it must define a `main` database first: - - ```yaml - production: - main: - adapter: postgresql - encoding: unicode - database: gitlabhq_production - ... - ``` - -### 14.10.0 - -- Before upgrading to GitLab 14.10, you must already have the latest 14.9.Z installed on your instance. - The upgrade to GitLab 14.10 executes a [concurrent index drop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84308) of unneeded - entries from the `ci_job_artifacts` database table. This could potentially run for multiple minutes, especially if the table has a lot of - traffic and the migration is unable to acquire a lock. It is advised to let this process finish as restarting may result in data loss. - -- If you run external PostgreSQL, particularly AWS RDS, - [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) - to avoid the database crashing. - -- Upgrading to patch level 14.10.3 or later might encounter a one-hour timeout due to a long running database data change, - if it was not completed while running GitLab 14.9. - - ```plaintext - FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] - (gitlab::database_migrations line 51) had an error: - [..] - Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: - ``` - - A workaround exists to [complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). - -### 14.9.0 - -- Database changes made by the upgrade to GitLab 14.9 can take hours or days to complete on larger GitLab instances. - These [batched background migrations](background_migrations.md#batched-background-migrations) update whole database tables to ensure corresponding - records in `namespaces` table for each record in `projects` table. - - After you upgrade to 14.9.0 or a later 14.9 patch version, - [batched background migrations must finish](background_migrations.md#batched-background-migrations) - before you upgrade to a later version. - - If the migrations are not finished and you try to upgrade to a later version, - you see errors like: - - ```plaintext - Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - ``` - - Or - - ```plaintext - Error executing action `run` on resource 'bash[migrate gitlab-rails database]' - ================================================================================ - - Mixlib::ShellOut::ShellCommandFailed - ------------------------------------ - Command execution failed. STDOUT/STDERR suppressed for sensitive resource - ``` - -- GitLab 14.9.0 includes a - [background migration `ResetDuplicateCiRunnersTokenValuesOnProjects`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) - that may remain stuck permanently in a **pending** state. - - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments) - end - ``` - -- If you run external PostgreSQL, particularly AWS RDS, - [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) - to avoid the database crashing. - -### 14.8.0 - -- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. - Updating to 14.8.2 or later resets runner registration tokens for your groups and projects. -- The agent server for Kubernetes [is enabled by default](https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#the-agent-server-for-kubernetes-is-enabled-by-default) - on Omnibus installations. If you run GitLab at scale, - such as [the reference architectures](../administration/reference_architectures/index.md), - you must disable the agent on the following server types, **if the agent is not required**. - - - Praefect - - Gitaly - - Sidekiq - - Redis (if configured using `redis['enable'] = true` and not via `roles`) - - Container registry - - Any other server types based on `roles(['application_role'])`, such as the GitLab Rails nodes - - [The reference architectures](../administration/reference_architectures/index.md) have been updated - with this configuration change and a specific role for standalone Redis servers. - - Steps to disable the agent: - - 1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`. - 1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`. -- GitLab 14.8.0 includes a -[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) -that may remain stuck permanently in a **pending** state. - - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments) - end - ``` - -- If upgrading from a version earlier than 14.3.0, to avoid - [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade - to GitLab 14.7.x and make sure all batched migrations have finished. -- If upgrading from version 14.3.0 or later, you might notice a failed - [batched migration](background_migrations.md#batched-background-migrations) named - `BackfillNamespaceIdForNamespaceRoute`. You can [ignore](https://gitlab.com/gitlab-org/gitlab/-/issues/357822) - this. Retry it after you upgrade to version 14.9.x. -- If you run external PostgreSQL, particularly AWS RDS, - [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) - to avoid the database crashing. - -### 14.7.0 - -- See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472). -- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. - Updating to 14.7.4 or later resets runner registration tokens for your groups and projects. -- GitLab 14.7 introduced a change where Gitaly expects persistent files in the `/tmp` directory. - When using the `noatime` mount option on `/tmp` in a node running Gitaly, most Linux distributions - run into [an issue with Git server hooks getting deleted](https://gitlab.com/gitlab-org/gitaly/-/issues/4113). - These conditions are present in the default Amazon Linux configuration. - - If your Linux distribution manages files in `/tmp` with the `tmpfiles.d` service, you - can override the behavior of `tmpfiles.d` for the Gitaly files and avoid this issue: - - ```shell - sudo printf "x /tmp/gitaly-%s-*\n" hooks git-exec-path >/etc/tmpfiles.d/gitaly-workaround.conf - ``` - - This issue is fixed in GitLab 14.10 and later when using the [Gitaly runtime directory](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html#gitaly-runtime-directory) - to specify a location to store persistent files. - -### 14.6.0 - -- See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472). -- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. - Updating to 14.6.5 or later resets runner registration tokens for your groups and projects. - -### 14.5.0 - -- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you -are using a self-compiled installation, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units) -or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md). - -- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, - Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature) - feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301). - -- In 14.1 we introduced a background migration that changes how we store merge request diff commits, - to significantly reduce the amount of storage needed. - In 14.5 we introduce a set of migrations that wrap up this process by making sure - that all remaining jobs over the `merge_request_diff_commits` table are completed. - These jobs have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. - However, if there are remaining jobs or you haven't already upgraded to 14.1, - the deployment may take multiple hours to complete. - - All merge request diff commits automatically incorporate these changes, and there are no - additional requirements to perform the upgrade. - Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. - However, the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, - so the operation takes some time to complete and it blocks access to this table until the end of the process. - We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. - The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. - - For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823). - -- GitLab 14.5.0 includes a - [background migration `UpdateVulnerabilityOccurrencesLocation`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72788) - that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. - - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments) - end - ``` - -- Upgrading to 14.5 (or later) [might encounter a one hour timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/354211) - owing to a long running database data change. - - ```plaintext - FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] - (gitlab::database_migrations line 51) had an error: - [..] - Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: - ``` - - [There is a workaround to complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s) - -- As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default. - For **self-compiled (source) installations**, `config/cable.yml` is required to be present. - - Configure this by running: - - ```shell - cd /home/git/gitlab - sudo -u git -H cp config/cable.yml.example config/cable.yml - - # Change the Redis socket path if you are not using the default Debian / Ubuntu configuration - sudo -u git -H editor config/cable.yml - ``` - -### 14.4.4 - -- For [zero-downtime upgrades](zero_downtime.md) on a GitLab cluster with separate Web and API nodes, you must enable the `paginated_tree_graphql_query` [feature flag](../administration/feature_flags.md#enable-or-disable-the-feature) _before_ upgrading GitLab Web nodes to 14.4. - This is because we [enabled `paginated_tree_graphql_query` by default in 14.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70913/diffs), so if GitLab UI is on 14.4 and its API is on 14.3, the frontend has this feature enabled but the backend has it disabled. This results in the following error: - - ```shell - bundle.esm.js:63 Uncaught (in promise) Error: GraphQL error: Field 'paginatedTree' doesn't exist on type 'Repository' - ``` - -### 14.4.0 - -- Git 2.33.x and later is required. We recommend you use the - [Git version provided by Gitaly](../install/installation.md#git). -- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). -- After enabling database load balancing by default in 14.4.0, we found an issue where - [cron jobs would not work if the connection to PostgreSQL was severed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73716), - as Sidekiq would continue using a bad connection. Geo and other features that rely on - cron jobs running regularly do not work until Sidekiq is restarted. We recommend - upgrading to GitLab 14.4.3 and later if this issue affects you. -- After enabling database load balancing by default in 14.4.0, we found an issue where - [Database load balancing does not work with an AWS Aurora cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/220617). - We recommend moving your databases from Aurora to RDS for PostgreSQL before - upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../administration/postgresql/moving.md). -- GitLab 14.4.0 includes a -[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) -that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. - - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments) - end - ``` - -### 14.3.0 - -- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). -- Ensure [batched background migrations finish](background_migrations.md#batched-background-migrations) before upgrading - to 14.3.Z from earlier GitLab 14 releases. -- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) -for how to proceed. -- GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) - - [`ci_builds.stage_id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66688) - - [`ci_builds_metadata`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65692) - - [`taggings`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66625) - - [`events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64779) - - If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful: - - ```shell - # For Omnibus GitLab - sudo gitlab-rake db:migrate - - # For source installations - sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production - ``` - -- After upgrading to 14.3, ensure that all the `MigrateMergeRequestDiffCommitUsers` background - migration jobs have completed before continuing with upgrading to GitLab 14.5 or later. - This is especially important if your GitLab instance has a large - `merge_request_diff_commits` table. Any pending - `MigrateMergeRequestDiffCommitUsers` background migration jobs are - foregrounded in GitLab 14.5, and may take a long time to complete. - You can check the count of pending jobs for - `MigrateMergeRequestDiffCommitUsers` by using the PostgreSQL console (or `sudo - gitlab-psql`): - - ```sql - select status, count(*) from background_migration_jobs - where class_name = 'MigrateMergeRequestDiffCommitUsers' group by status; - ``` - - As jobs are completed, the database records change from `0` (pending) to `1`. If the number of - pending jobs doesn't decrease after a while, it's possible that the - `MigrateMergeRequestDiffCommitUsers` background migration jobs have failed. You - can check for errors in the Sidekiq logs: - - ```shell - sudo grep MigrateMergeRequestDiffCommitUsers /var/log/gitlab/sidekiq/current | grep -i error - ``` - - If needed, you can attempt to run the `MigrateMergeRequestDiffCommitUsers` background - migration jobs manually in the [GitLab Rails Console](../administration/operations/rails_console.md). - This can be done using Sidekiq asynchronously, or by using a Rails process directly: - - - Using Sidekiq to schedule jobs asynchronously: - - ```ruby - # For the first run, only attempt to execute 1 migration. If successful, increase - # the limit for subsequent runs - limit = 1 - - jobs = Gitlab::Database::BackgroundMigrationJob.for_migration_class('MigrateMergeRequestDiffCommitUsers').pending.to_a - - pp "#{jobs.length} jobs remaining" - - jobs.first(limit).each do |job| - BackgroundMigrationWorker.perform_in(5.minutes, 'MigrateMergeRequestDiffCommitUsers', job.arguments) - end - ``` - - NOTE: - The queued jobs can be monitored using the Sidekiq admin panel, which can be accessed at the `/admin/sidekiq` endpoint URI. - - - Using a Rails process to run jobs synchronously: - - ```ruby - def process(concurrency: 1) - queue = Queue.new - - Gitlab::Database::BackgroundMigrationJob - .where(class_name: 'MigrateMergeRequestDiffCommitUsers', status: 0) - .each { |job| queue << job } - - concurrency - .times - .map do - Thread.new do - Thread.abort_on_exception = true - - loop do - job = queue.pop(true) - time = Benchmark.measure do - Gitlab::BackgroundMigration::MigrateMergeRequestDiffCommitUsers - .new - .perform(*job.arguments) - end - - puts "#{job.id} finished in #{time.real.round(2)} seconds" - rescue ThreadError - break - end - end - end - .each(&:join) - end - - ActiveRecord::Base.logger.level = Logger::ERROR - process - ``` - - NOTE: - When using Rails to execute these background migrations synchronously, make sure that the machine running the process has sufficient resources to handle the task. If the process gets terminated, it's likely due to insufficient memory available. If your SSH session times out after a while, it might be necessary to run the previous code by using a terminal multiplexer like `screen` or `tmux`. - -- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). - -- You may see the following error when setting up two factor authentication (2FA) for accounts - that authenticate using an LDAP password: - - ```plaintext - You must provide a valid current password - ``` - - - The error occurs because verification is incorrectly performed against accounts' - randomly generated internal GitLab passwords, not the LDAP passwords. - - This is [fixed in GitLab 14.5.0 and backported to 14.4.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73538). - - Workarounds: - - Instead of upgrading to GitLab 14.3.x to comply with the supported upgrade path: - 1. Upgrade to 14.4.5. - 1. Make sure the [`MigrateMergeRequestDiffCommitUsers` background migration](#1430) has finished. - 1. Upgrade to GitLab 14.5 or later. - - Reset the random password for affected accounts, using [the Rake task](../security/reset_user_password.md#use-a-rake-task): - - ```plaintext - sudo gitlab-rake "gitlab:password:reset[user_handle]" - ``` - -- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [122978](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122978) as the `mr_iid`. - -### 14.2.0 - -- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). -- Ensure [batched background migrations finish](background_migrations.md#batched-background-migrations) before upgrading - to 14.2.Z from earlier GitLab 14 releases. -- GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) - - [`deployments`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67341) - - [`geo_job_artifact_deleted_events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66763) - - [`push_event_payloads`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67299) - - `ci_job_artifacts`: - - [Finalize `job_id` conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774) - - [Finalize `ci_job_artifacts` conversion to `bigint`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65601) - - If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful: - - ```shell - # For Omnibus GitLab - sudo gitlab-rake db:migrate - - # For source installations - sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production - ``` - -- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). -- GitLab 14.2.0 includes a - [background migration `BackfillDraftStatusOnMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67687) - that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. - - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "BackfillDraftStatusOnMergeRequests").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("BackfillDraftStatusOnMergeRequests", job.arguments) - end - ``` - -- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123476](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123476) as the `mr_iid`. - -### 14.1.0 - -- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) - but can upgrade to 14.1.Z. - - It is not required for instances already running 14.0.5 (or later) to stop at 14.1.Z. - 14.1 is included on the upgrade path for the broadest compatibility - with self-managed installations, and ensure 14.0.0-14.0.4 installations do not - encounter issues with [batched background migrations](background_migrations.md#batched-background-migrations). - -- Upgrading to GitLab [14.5](#1450) (or later) may take a lot longer if you do not upgrade to at least 14.1 - first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the - background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must - run the migration in the foreground and therefore take a lot longer to complete. - -- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). - -- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123475](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123475) as the `mr_iid`. - -### 14.0.0 - -Prerequisites: - -- The [GitLab 14.0 release post contains several important notes](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#upgrade) - about pre-requisites including [using Patroni instead of repmgr](../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni), - migrating [to hashed storage](../administration/raketasks/storage.md#migrate-to-hashed-storage), - and [to Puma](../administration/operations/puma.md). -- The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. - -Long running batched background database migrations: - -- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. - These [batched background migrations](background_migrations.md#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or later. -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). - - After you update to 14.0.5 or a later 14.0 patch version, - [batched background migrations must finish](background_migrations.md#batched-background-migrations) - before you upgrade to a later version. - - If the migrations are not finished and you try to upgrade to a later version, - you see an error like: - - ```plaintext - Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - ``` - - See how to [resolve this error](background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). - -Other issues: - -- In GitLab 13.3 some [pipeline processing methods were deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/218536) - and this code was completely removed in GitLab 14.0. If you plan to upgrade from - **GitLab 13.2 or older** directly to 14.0, this is [unsupported](#upgrading-to-a-new-major-version). - You should instead follow a [supported upgrade path](#upgrade-paths). -- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). - -#### Upgrading to later 14.Y releases - -- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, - because of [batched background migrations](background_migrations.md#batched-background-migrations). - 1. Upgrade first to either: - - 14.0.5 or a later 14.0.Z patch release. - - 14.1.0 or a later 14.1.Z patch release. - 1. [Batched background migrations must finish](background_migrations.md#batched-background-migrations) - before you upgrade to a later version [and may take longer than usual](#1400). - -### User profile data loss bug in 15.9.x - -There is a database migration bug in 15.9.0, 15.9.1, and 15.9.2 that can cause data loss from the user profile fields `linkedin`, `twitter`, `skype`, `website_url`, `location`, and `organization`. - -This bug is fixed in patch releases 15.9.3 and later. - -The following upgrade path also works around the bug: - -1. Upgrade to GitLab 15.6.x, 15.7.x, or 15.8.x. -1. [Ensure batched background migrations](background_migrations.md#batched-background-migrations) are complete. -1. Upgrade to an earlier GitLab 15.9 patch release that doesn't have the bug fix. - -It is not then required to upgrade to 15.9.3 or later for this issue. - -[Read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393216) for more information. - -### Gitaly: Omnibus GitLab configuration structure change - -Gitaly configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Gitaly configuration -structure used in self-compiled installations. - -As a result of this change, a single hash under `gitaly['configuration']` holds most Gitaly -configuration. Some `gitaly['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later: - -- `enable` -- `dir` -- `bin_path` -- `env_directory` -- `env` -- `open_files_ulimit` -- `consul_service_name` -- `consul_service_meta` - -Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.10. - -The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration: - -1. Replace the `...` with the value from the old key. -1. Skip any keys you haven't configured a value for previously. -1. Remove the old keys from the configuration once migrated. -1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. -1. When configuring `storage` to replace `git_data_dirs`, you must append `repositories` to the path as documented below. If you omit this step, your Git repositories are - inaccessible until the configuration is fixed. - - ```ruby -gitaly['configuration'] = { - # gitaly['socket_path'] - socket_path: ..., - # gitaly['runtime_dir'] - runtime_dir: ..., - # gitaly['listen_addr'] - listen_addr: ..., - # gitaly['prometheus_listen_addr'] - prometheus_listen_addr: ..., - # gitaly['tls_listen_addr'] - tls_listen_addr: ..., - tls: { - # gitaly['certificate_path'] - certificate_path: ..., - # gitaly['key_path'] - key_path: ..., - }, - # gitaly['graceful_restart_timeout'] - graceful_restart_timeout: ..., - logging: { - # gitaly['logging_level'] - level: ..., - # gitaly['logging_format'] - format: ..., - # gitaly['logging_sentry_dsn'] - sentry_dsn: ..., - # gitaly['logging_ruby_sentry_dsn'] - ruby_sentry_dsn: ..., - # gitaly['logging_sentry_environment'] - sentry_environment: ..., - # gitaly['log_directory'] - dir: ..., - }, - prometheus: { - # gitaly['prometheus_grpc_latency_buckets']. The old value was configured as a string - # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. - grpc_latency_buckets: ..., - }, - auth: { - # gitaly['auth_token'] - token: ..., - # gitaly['auth_transitioning'] - transitioning: ..., - }, - git: { - # gitaly['git_catfile_cache_size'] - catfile_cache_size: ..., - # gitaly['git_bin_path'] - bin_path: ..., - # gitaly['use_bundled_git'] - use_bundled_binaries: ..., - # gitaly['gpg_signing_key_path'] - signing_key: ..., - # gitaly['gitconfig']. This is still an array but the type of the elements have changed. - config: [ - { - # Previously the elements contained 'section', and 'subsection' in addition to 'key'. Now - # these all should be concatenated into just 'key', separated by dots. For example, - # {section: 'first', subsection: 'middle', key: 'last', value: 'value'}, should become - # {key: 'first.middle.last', value: 'value'}. - key: ..., - value: ..., - }, - ], - }, - # Storage could previously be configured through either gitaly['storage'] or 'git_data_dirs'. Migrate - # the relevant configuration according to the instructions below. - # For 'git_data_dirs', migrate only the 'path' to the gitaly['configuration'] and leave the rest of it untouched. - storage: [ - { - # gitaly['storage'][<index>]['name'] - # - # git_data_dirs[<name>]. The storage name was configured as a key in the map. - name: ..., - # gitaly['storage'][<index>]['path'] - # - # git_data_dirs[<name>]['path']. Use the value from git_data_dirs[<name>]['path'] and append '/repositories' to it. - # - # For example, if the path in 'git_data_dirs' was '/var/opt/gitlab/git-data', use - # '/var/opt/gitlab/git-data/repositories'. The '/repositories' extension was automatically - # appended to the path configured in `git_data_dirs`. - path: ..., - }, - ], - hooks: { - # gitaly['custom_hooks_dir'] - custom_hooks_dir: ..., - }, - daily_maintenance: { - # gitaly['daily_maintenance_disabled'] - disabled: ..., - # gitaly['daily_maintenance_start_hour'] - start_hour: ..., - # gitaly['daily_maintenance_start_minute'] - start_minute: ..., - # gitaly['daily_maintenance_duration'] - duration: ..., - # gitaly['daily_maintenance_storages'] - storages: ..., - }, - cgroups: { - # gitaly['cgroups_mountpoint'] - mountpoint: ..., - # gitaly['cgroups_hierarchy_root'] - hierarchy_root: ..., - # gitaly['cgroups_memory_bytes'] - memory_bytes: ..., - # gitaly['cgroups_cpu_shares'] - cpu_shares: ..., - repositories: { - # gitaly['cgroups_repositories_count'] - count: ..., - # gitaly['cgroups_repositories_memory_bytes'] - memory_bytes: ..., - # gitaly['cgroups_repositories_cpu_shares'] - cpu_shares: ..., - } - }, - # gitaly['concurrency']. While the structure is the same, the string keys in the array elements - # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. - concurrency: ..., - # gitaly['rate_limiting']. While the structure is the same, the string keys in the array elements - # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. - rate_limiting: ..., - pack_objects_cache: { - # gitaly['pack_objects_cache_enabled'] - enabled: ..., - # gitaly['pack_objects_cache_dir'] - dir: ..., - # gitaly['pack_objects_cache_max_age'] - max_age: ..., - } -} -``` - -### Praefect: Omnibus GitLab configuration structure change - -Praefect configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Praefect configuration -structure used in self-compiled installations. - -As a result of this change, a single hash under `praefect['configuration']` holds most Praefect -configuration. Some `praefect['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later: - -- `enable` -- `dir` -- `log_directory` -- `env_directory` -- `env` -- `wrapper_path` -- `auto_migrate` -- `consul_service_name` - -Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.9. - -The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration: - -1. Replace the `...` with the value from the old key. -1. Skip any keys you haven't configured a value for previously. -1. Remove the old keys from the configuration once migrated. -1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. - -```ruby -praefect['configuration'] = { - # praefect['listen_addr'] - listen_addr: ..., - # praefect['socket_path'] - socket_path: ..., - # praefect['prometheus_listen_addr'] - prometheus_listen_addr: ..., - # praefect['tls_listen_addr'] - tls_listen_addr: ..., - # praefect['separate_database_metrics'] - prometheus_exclude_database_from_default_metrics: ..., - auth: { - # praefect['auth_token'] - token: ..., - # praefect['auth_transitioning'] - transitioning: ..., - }, - logging: { - # praefect['logging_format'] - format: ..., - # praefect['logging_level'] - level: ..., - }, - failover: { - # praefect['failover_enabled'] - enabled: ..., - }, - background_verification: { - # praefect['background_verification_delete_invalid_records'] - delete_invalid_records: ..., - # praefect['background_verification_verification_interval'] - verification_interval: ..., - }, - reconciliation: { - # praefect['reconciliation_scheduling_interval'] - scheduling_interval: ..., - # praefect['reconciliation_histogram_buckets']. The old value was configured as a string - # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. - histogram_buckets: ..., - }, - tls: { - # praefect['certificate_path'] - certificate_path: ..., - # praefect['key_path'] - key_path: ..., - }, - database: { - # praefect['database_host'] - host: ..., - # praefect['database_port'] - port: ..., - # praefect['database_user'] - user: ..., - # praefect['database_password'] - password: ..., - # praefect['database_dbname'] - dbname: ..., - # praefect['database_sslmode'] - sslmode: ..., - # praefect['database_sslcert'] - sslcert: ..., - # praefect['database_sslkey'] - sslkey: ..., - # praefect['database_sslrootcert'] - sslrootcert: ..., - session_pooled: { - # praefect['database_direct_host'] - host: ..., - # praefect['database_direct_port'] - port: ..., - # praefect['database_direct_user'] - user: ..., - # praefect['database_direct_password'] - password: ..., - # praefect['database_direct_dbname'] - dbname: ..., - # praefect['database_direct_sslmode'] - sslmode: ..., - # praefect['database_direct_sslcert'] - sslcert: ..., - # praefect['database_direct_sslkey'] - sslkey: ..., - # praefect['database_direct_sslrootcert'] - sslrootcert: ..., - } - }, - sentry: { - # praefect['sentry_dsn'] - sentry_dsn: ..., - # praefect['sentry_environment'] - sentry_environment: ..., - }, - prometheus: { - # praefect['prometheus_grpc_latency_buckets']. The old value was configured as a string - # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. - grpc_latency_buckets: ..., - }, - # praefect['graceful_stop_timeout'] - graceful_stop_timeout: ..., - - # praefect['virtual_storages']. The old value was a hash map but the new value is an array. - virtual_storage: [ - { - # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]. The name was previously the key in - # the 'virtual_storages' hash. - name: ..., - # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. The old value was a hash map - # but the new value is an array. - node: [ - { - # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. Use NODE_NAME key as the - # storage. - storage: ..., - # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['address']. - address: ..., - # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['token']. - token: ..., - }, - ], - } - ] -} -``` - -### Change to Praefect-generated replica paths in GitLab 15.3 - -New Git repositories created in Gitaly cluster no longer use the `@hashed` storage path. - -Praefect now generates replica paths for use by Gitaly cluster. -This change is a pre-requisite for Gitaly cluster atomically creating, deleting, and -renaming Git repositories. - -To identify the replica path, [query the Praefect repository metadata](../administration/gitaly/troubleshooting.md#view-repository-metadata) -and pass the `@hashed` storage path to `-relative-path`. - -With this information, you can correctly install [server hooks](../administration/server_hooks.md). - -### Geo: LFS transfers redirect to primary from secondary site mid-session in GitLab 15.1.0 to 15.3.2 - -LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. - -This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later: - -- LFS is enabled. -- LFS objects are being replicated across Geo sites. -- Repositories are being pulled by using a Geo secondary site. - -### Geo: Incorrect object storage LFS file deletion on secondary sites in GitLab 15.0.0 to 15.3.2 - -[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) -can occur in GitLab 15.0.0 to 15.3.2 in the following situations: - -- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled. -- GitLab-managed replication to sync object storage is enabled and subsequently disabled. - -This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites -should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites. - -### PostgreSQL segmentation fault issue - -If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you upgrade PostgreSQL -to patch levels to a minimum of 12.7 or 13.3 before upgrading to GitLab 14.8 or later. - -[In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) -for GitLab Enterprise Edition and [in 15.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) -for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled. - -After it was enabled, we have had reports of unplanned PostgreSQL restarts caused -by a database engine bug that causes a segmentation fault. - -Read more [in the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). - -### LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2 - -When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects. - -[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3. +Before upgrading to GitLab 16, see [GitLab 16 changes](versions/gitlab_16_changes.md). -### Maintenance mode issue in GitLab 13.9 to 14.4 +### GitLab 15 -When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP. +Before upgrading to GitLab 15, see [GitLab 15 changes](versions/gitlab_15_changes.md). -Users who were signed in before Maintenance mode was enabled, continue to be signed in. If the administrator who enabled Maintenance mode loses their session, then they can't disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/index.md#disable-maintenance-mode). +### GitLab 14 -[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. +Before upgrading to GitLab 14, see [GitLab 14 changes](versions/gitlab_14_changes.md). ## Miscellaneous diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 148791dbc75..19d308bce6b 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -43,6 +43,14 @@ check the version your are upgrading to: - [GitLab 15](https://docs.gitlab.com/omnibus/update/gitlab_15_changes.html) - [GitLab 14](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html) +### Earlier GitLab versions + +For version-specific information for earlier GitLab versions, see the [documentation archives](https://archives.docs.gitlab.com). +The versions of the documentation in the archives contain version-specific information for even earlier versions of GitLab. + +For example, the [documentation for GitLab 15.11](https://archives.docs.gitlab.com/15.11/ee/update/package/#version-specific-changes) +contains information on versions back to GitLab 11. + ## Back up before upgrading The GitLab database is backed up before installing a newer GitLab version. You diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index 0bb760ec47e..25578dbef59 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -103,11 +103,7 @@ For the upgrade plan, start by creating an outline of a plan that best applies to your instance and then upgrade it for any relevant features you're using. - Generate an upgrade plan by reading and understanding the relevant documentation: - - upgrade based on the installation method: - - [Linux package (Omnibus)](index.md#linux-packages-omnibus) - - [Self-compiled](index.md#self-compiled-installation) - - [Docker](index.md#installation-using-docker) - - [Helm Charts](index.md#installation-using-helm) + - Upgrade based on the [installation method](index.md#upgrade-based-on-installation-method). - [Zero-downtime upgrades](zero_downtime.md) (if possible and desired) - [Convert from GitLab Community Edition to Enterprise Edition](package/convert_to_ee.md) - What version should you upgrade to: @@ -122,7 +118,7 @@ to your instance and then upgrade it for any relevant features you're using. [turning on maintenance mode](../administration/maintenance_mode/index.md) during the upgrade. - About PostgreSQL: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Look for the version of PostgreSQL you are using. If [a PostgreSQL upgrade is needed](../administration/package_information/postgresql_versions.md), diff --git a/doc/update/versions/gitlab_14_changes.md b/doc/update/versions/gitlab_14_changes.md new file mode 100644 index 00000000000..700baf4ef09 --- /dev/null +++ b/doc/update/versions/gitlab_14_changes.md @@ -0,0 +1,909 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab 14 changes **(FREE SELF)** + +This page contains upgrade information for minor and patch versions of GitLab 14. +Ensure you review these instructions for: + +- Your installation type. +- All versions between your current version and your target version. + +For more information about upgrading GitLab Helm Chart, see [the release notes for 5.0](https://docs.gitlab.com/charts/releases/5_0.html). + +## 14.10.0 + +- Before upgrading to GitLab 14.10, you must already have the latest 14.9.Z installed on your instance. + The upgrade to GitLab 14.10 executes a [concurrent index drop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84308) of unneeded + entries from the `ci_job_artifacts` database table. This could potentially run for multiple minutes, especially if the table has a lot of + traffic and the migration is unable to acquire a lock. It is advised to let this process finish as restarting may result in data loss. + +- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you + upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before + upgrading to GitLab 14.8 or later. + + [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) + for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) + for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled. + + After it was enabled, we have had reports of unplanned PostgreSQL restarts caused + by a database engine bug that causes a segmentation fault. + + For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). + +- Upgrading to patch level 14.10.3 or later might encounter a one-hour timeout due to a long running database data change, + if it was not completed while running GitLab 14.9. + + ```plaintext + FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] + (gitlab::database_migrations line 51) had an error: + [..] + Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: + ``` + + A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). + +### Linux package installations + +- In GitLab 14.10, Gitaly has introduced a new runtime directory. This directory is intended to hold all files and + directories Gitaly needs to create at runtime to operate correctly. This includes, for example, internal sockets, the + Git execution environment, or the temporary hooks directory. + + This new configuration can be set via `gitaly['runtime_dir']`. It replaces the old `gitaly['internal_socket_dir']` + configuration. If the internal socket directory is not explicitly configured, sockets will be created in the runtime directory. + + Support for `gitaly['internal_socket_dir']` will be removed in 15.0. + +## 14.9.0 + +- Database changes made by the upgrade to GitLab 14.9 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](../background_migrations.md#batched-background-migrations) update whole database tables to ensure corresponding + records in `namespaces` table for each record in `projects` table. + + After you upgrade to 14.9.0 or a later 14.9 patch version, + [batched background migrations must finish](../background_migrations.md#batched-background-migrations) + before you upgrade to a later version. + + If the migrations are not finished and you try to upgrade to a later version, + you see errors like: + + ```plaintext + Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': + ``` + + Or + + ```plaintext + Error executing action `run` on resource 'bash[migrate gitlab-rails database]' + ================================================================================ + + Mixlib::ShellOut::ShellCommandFailed + ------------------------------------ + Command execution failed. STDOUT/STDERR suppressed for sensitive resource + ``` + +- GitLab 14.9.0 includes a + [background migration `ResetDuplicateCiRunnersTokenValuesOnProjects`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) + that may remain stuck permanently in a **pending** state. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments) + end + ``` + +- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you + upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before + upgrading to GitLab 14.8 or later. + + [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) + for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) + for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled. + + After it was enabled, we have had reports of unplanned PostgreSQL restarts caused + by a database engine bug that causes a segmentation fault. + + For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). + +### Geo installations **(PREMIUM SELF)** + +- **Do not** upgrade to GitLab 14.9.0. Instead, use 14.9.1 or later. + + We've discovered an issue with Geo's CI verification feature that may + [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue was fixed + in [the GitLab 14.9.1 patch release](https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/). + + If you have already upgraded to GitLab 14.9.0, you can disable the feature causing the issue by + [disabling the `geo_job_artifact_replication` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags). + +## 14.8.0 + +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.8.2 or later resets runner registration tokens for your groups and projects. +- The agent server for Kubernetes [is enabled by default](https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#the-agent-server-for-kubernetes-is-enabled-by-default) + on Omnibus installations. If you run GitLab at scale, + such as [the reference architectures](../../administration/reference_architectures/index.md), + you must disable the agent on the following server types, **if the agent is not required**. + + - Praefect + - Gitaly + - Sidekiq + - Redis (if configured using `redis['enable'] = true` and not via `roles`) + - Container registry + - Any other server types based on `roles(['application_role'])`, such as the GitLab Rails nodes + + [The reference architectures](../../administration/reference_architectures/index.md) have been updated + with this configuration change and a specific role for standalone Redis servers. + + Steps to disable the agent: + + 1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`. + 1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`. +- GitLab 14.8.0 includes a +[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) +that may remain stuck permanently in a **pending** state. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments) + end + ``` + +- If upgrading from a version earlier than 14.3.0, to avoid + [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade + to GitLab 14.7.x and make sure all batched migrations have finished. +- If upgrading from version 14.3.0 or later, you might notice a failed + [batched migration](../background_migrations.md#batched-background-migrations) named + `BackfillNamespaceIdForNamespaceRoute`. You can [ignore](https://gitlab.com/gitlab-org/gitlab/-/issues/357822) + this. Retry it after you upgrade to version 14.9.x. +- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you + upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before + upgrading to GitLab 14.8 or later. + + [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) + for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) + for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled. + + After it was enabled, we have had reports of unplanned PostgreSQL restarts caused + by a database engine bug that causes a segmentation fault. + + For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). + +## 14.7.0 + +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.7.4 or later resets runner registration tokens for your groups and projects. +- GitLab 14.7 introduced a change where Gitaly expects persistent files in the `/tmp` directory. + When using the `noatime` mount option on `/tmp` in a node running Gitaly, most Linux distributions + run into [an issue with Git server hooks getting deleted](https://gitlab.com/gitlab-org/gitaly/-/issues/4113). + These conditions are present in the default Amazon Linux configuration. + + If your Linux distribution manages files in `/tmp` with the `tmpfiles.d` service, you + can override the behavior of `tmpfiles.d` for the Gitaly files and avoid this issue: + + ```shell + sudo printf "x /tmp/gitaly-%s-*\n" hooks git-exec-path >/etc/tmpfiles.d/gitaly-workaround.conf + ``` + + This issue is fixed in GitLab 14.10 and later when using the [Gitaly runtime directory](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html#gitaly-runtime-directory) + to specify a location to store persistent files. + +### Linux package installations + +- In GitLab 14.8, we are upgrading Redis from 6.0.16 to 6.2.6. This upgrade is expected to be fully backwards compatible. + + If your instance has Redis HA with Sentinel, follow the upgrade steps documented in + [Redis HA (using Sentinel)](../zero_downtime.md#redis-ha-using-sentinel). + +### Geo installations **(PREMIUM SELF)** + +- LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2. + When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects. + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3. +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +## 14.6.0 + +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.6.5 or later resets runner registration tokens for your groups and projects. + +### Geo installations **(PREMIUM SELF)** + +- LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2. + When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects. + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3. +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +## 14.5.0 + +- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, + Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../../administration/feature_flags.md#enable-or-disable-the-feature) + feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301). + +- In 14.1 we introduced a background migration that changes how we store merge request diff commits, + to significantly reduce the amount of storage needed. + In 14.5 we introduce a set of migrations that wrap up this process by making sure + that all remaining jobs over the `merge_request_diff_commits` table are completed. + These jobs have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. + However, if there are remaining jobs or you haven't already upgraded to 14.1, + the deployment may take multiple hours to complete. + + All merge request diff commits automatically incorporate these changes, and there are no + additional requirements to perform the upgrade. + Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. + However, the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, + so the operation takes some time to complete and it blocks access to this table until the end of the process. + We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. + The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. + + For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823). + +- GitLab 14.5.0 includes a + [background migration `UpdateVulnerabilityOccurrencesLocation`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72788) + that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments) + end + ``` + +- Upgrading to 14.5 (or later) [might encounter a one hour timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/354211) + owing to a long running database data change. + + ```plaintext + FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] + (gitlab::database_migrations line 51) had an error: + [..] + Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: + ``` + + [There is a workaround to complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s) + +- As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default. + For **self-compiled (source) installations**, `config/cable.yml` is required to be present. + + Configure this by running: + + ```shell + cd /home/git/gitlab + sudo -u git -H cp config/cable.yml.example config/cable.yml + + # Change the Redis socket path if you are not using the default Debian / Ubuntu configuration + sudo -u git -H editor config/cable.yml + ``` + +### Self-compiled installations + +- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you +are using a self-compiled installation, update paths to these binaries in your [systemd unit files](../upgrading_from_source.md#configure-systemd-units) +or [init scripts](../upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](../upgrading_from_source.md). + +### Geo installations **(PREMIUM SELF)** + +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +## 14.4.4 + +- For [zero-downtime upgrades](../zero_downtime.md) on a GitLab cluster with separate Web and API nodes, you must enable the `paginated_tree_graphql_query` [feature flag](../../administration/feature_flags.md#enable-or-disable-the-feature) _before_ upgrading GitLab Web nodes to 14.4. + This is because we [enabled `paginated_tree_graphql_query` by default in 14.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70913/diffs), so if GitLab UI is on 14.4 and its API is on 14.3, the frontend has this feature enabled but the backend has it disabled. This results in the following error: + + ```shell + bundle.esm.js:63 Uncaught (in promise) Error: GraphQL error: Field 'paginatedTree' doesn't exist on type 'Repository' + ``` + +## 14.4.1 and 14.4.2 + +### Geo installations **(PREMIUM SELF)** + +- There is [an issue in GitLab 14.4.0 through 14.4.2](#1440) that can affect + Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later. + +## 14.4.0 + +- Git 2.33.x and later is required. We recommend you use the + [Git version provided by Gitaly](../../install/installation.md#git). +- When [Maintenance mode](../../administration/maintenance_mode/index.md) is + enabled, users cannot sign in with SSO, SAML, or LDAP. + + Users who were signed in before Maintenance mode was enabled, continue to be + signed in. If the administrator who enabled Maintenance mode loses their + session, then they can't disable Maintenance mode via the UI. In that case, + you can + [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode). + + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in + GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. +- After enabling database load balancing by default in 14.4.0, we found an issue where + [cron jobs would not work if the connection to PostgreSQL was severed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73716), + as Sidekiq would continue using a bad connection. Geo and other features that rely on + cron jobs running regularly do not work until Sidekiq is restarted. We recommend + upgrading to GitLab 14.4.3 and later if this issue affects you. +- After enabling database load balancing by default in 14.4.0, we found an issue where + [Database load balancing does not work with an AWS Aurora cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/220617). + We recommend moving your databases from Aurora to RDS for PostgreSQL before + upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../../administration/postgresql/moving.md). +- GitLab 14.4.0 includes a +[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) +that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments) + end + ``` + +### Linux package installations + +- In GitLab 14.4, the provided Grafana version is 7.5, this is a downgrade from the Grafana 8.1 version introduced in + GitLab 14.3. This was reverted to an Apache-licensed Grafana release to allow time to consider the implications of the + newer AGPL-licensed releases. + + Users that have customized their Grafana install with plugins or library panels may experience errors in Grafana after + the downgrade. If the errors persist after a Grafana restart you may need to reset the Grafana db and re-add the + customizations. The Grafana database can be reset with `sudo gitlab-ctl reset-grafana`. + +### Geo installations **(PREMIUM SELF)** + +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +- There is [an issue in GitLab 14.4.0 through 14.4.2](#1440) that can affect + Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later. + +## 14.3.0 + +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](../background_migrations.md#batched-background-migrations) before upgrading + to 14.3.Z from earlier GitLab 14 releases. +- GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: + + - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) + - [`ci_builds.stage_id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66688) + - [`ci_builds_metadata`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65692) + - [`taggings`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66625) + - [`events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64779) + + If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful: + + ```shell + # For Omnibus GitLab + sudo gitlab-rake db:migrate + + # For source installations + sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + ``` + +- After upgrading to 14.3, ensure that all the `MigrateMergeRequestDiffCommitUsers` background + migration jobs have completed before continuing with upgrading to GitLab 14.5 or later. + This is especially important if your GitLab instance has a large + `merge_request_diff_commits` table. Any pending + `MigrateMergeRequestDiffCommitUsers` background migration jobs are + foregrounded in GitLab 14.5, and may take a long time to complete. + You can check the count of pending jobs for + `MigrateMergeRequestDiffCommitUsers` by using the PostgreSQL console (or `sudo + gitlab-psql`): + + ```sql + select status, count(*) from background_migration_jobs + where class_name = 'MigrateMergeRequestDiffCommitUsers' group by status; + ``` + + As jobs are completed, the database records change from `0` (pending) to `1`. If the number of + pending jobs doesn't decrease after a while, it's possible that the + `MigrateMergeRequestDiffCommitUsers` background migration jobs have failed. You + can check for errors in the Sidekiq logs: + + ```shell + sudo grep MigrateMergeRequestDiffCommitUsers /var/log/gitlab/sidekiq/current | grep -i error + ``` + + If needed, you can attempt to run the `MigrateMergeRequestDiffCommitUsers` background + migration jobs manually in the [GitLab Rails Console](../../administration/operations/rails_console.md). + This can be done using Sidekiq asynchronously, or by using a Rails process directly: + + - Using Sidekiq to schedule jobs asynchronously: + + ```ruby + # For the first run, only attempt to execute 1 migration. If successful, increase + # the limit for subsequent runs + limit = 1 + + jobs = Gitlab::Database::BackgroundMigrationJob.for_migration_class('MigrateMergeRequestDiffCommitUsers').pending.to_a + + pp "#{jobs.length} jobs remaining" + + jobs.first(limit).each do |job| + BackgroundMigrationWorker.perform_in(5.minutes, 'MigrateMergeRequestDiffCommitUsers', job.arguments) + end + ``` + + NOTE: + The queued jobs can be monitored using the Sidekiq admin panel, which can be accessed at the `/admin/sidekiq` endpoint URI. + + - Using a Rails process to run jobs synchronously: + + ```ruby + def process(concurrency: 1) + queue = Queue.new + + Gitlab::Database::BackgroundMigrationJob + .where(class_name: 'MigrateMergeRequestDiffCommitUsers', status: 0) + .each { |job| queue << job } + + concurrency + .times + .map do + Thread.new do + Thread.abort_on_exception = true + + loop do + job = queue.pop(true) + time = Benchmark.measure do + Gitlab::BackgroundMigration::MigrateMergeRequestDiffCommitUsers + .new + .perform(*job.arguments) + end + + puts "#{job.id} finished in #{time.real.round(2)} seconds" + rescue ThreadError + break + end + end + end + .each(&:join) + end + + ActiveRecord::Base.logger.level = Logger::ERROR + process + ``` + + NOTE: + When using Rails to execute these background migrations synchronously, make sure that the machine running the process has sufficient resources to handle the task. If the process gets terminated, it's likely due to insufficient memory available. If your SSH session times out after a while, it might be necessary to run the previous code by using a terminal multiplexer like `screen` or `tmux`. + +- When [Maintenance mode](../../administration/maintenance_mode/index.md) is + enabled, users cannot sign in with SSO, SAML, or LDAP. + + Users who were signed in before Maintenance mode was enabled, continue to be + signed in. If the administrator who enabled Maintenance mode loses their + session, then they can't disable Maintenance mode via the UI. In that case, + you can + [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode). + + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in + GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. + +- You may see the following error when setting up two factor authentication (2FA) for accounts + that authenticate using an LDAP password: + + ```plaintext + You must provide a valid current password + ``` + + - The error occurs because verification is incorrectly performed against accounts' + randomly generated internal GitLab passwords, not the LDAP passwords. + - This is [fixed in GitLab 14.5.0 and backported to 14.4.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73538). + - Workarounds: + - Instead of upgrading to GitLab 14.3.x to comply with the supported upgrade path: + 1. Upgrade to 14.4.5. + 1. Make sure the [`MigrateMergeRequestDiffCommitUsers` background migration](#1430) has finished. + 1. Upgrade to GitLab 14.5 or later. + - Reset the random password for affected accounts, using [the Rake task](../../security/reset_user_password.md#use-a-rake-task): + + ```plaintext + sudo gitlab-rake "gitlab:password:reset[user_handle]" + ``` + +- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [122978](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122978) as the `mr_iid`. + +### Self-compiled installations + +- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../../install/installation.md#2-ruby) + for how to proceed. + +### Geo installations **(PREMIUM SELF)** + +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication + wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture + (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. + + You can check if you are affected by running: + + ```shell + docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' + ``` + + Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site. + If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` + (there can be a `mediaType` entry at several levels, we only care about the top level entry), + then you don't need to do anything. + + Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following: + + ```ruby + list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' + + Geo::ContainerRepositoryRegistry.synced.each do |gcr| + cr = gcr.container_repository + primary = Geo::ContainerRepositorySync.new(cr) + cr.tags.each do |tag| + primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name)) + next unless primary_manifest['mediaType'].eql?(list_type) + + cr.delete_tag_by_name(tag.name) + end + primary.execute + end + ``` + + If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, + we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1. + +## 14.2.0 + +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](../background_migrations.md#batched-background-migrations) before upgrading + to 14.2.Z from earlier GitLab 14 releases. +- GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: + - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) + - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) + - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) + - [`deployments`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67341) + - [`geo_job_artifact_deleted_events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66763) + - [`push_event_payloads`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67299) + - `ci_job_artifacts`: + - [Finalize `job_id` conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774) + - [Finalize `ci_job_artifacts` conversion to `bigint`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65601) + + If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful: + + ```shell + # For Omnibus GitLab + sudo gitlab-rake db:migrate + + # For source installations + sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + ``` + +- When [Maintenance mode](../../administration/maintenance_mode/index.md) is + enabled, users cannot sign in with SSO, SAML, or LDAP. + + Users who were signed in before Maintenance mode was enabled, continue to be + signed in. If the administrator who enabled Maintenance mode loses their + session, then they can't disable Maintenance mode via the UI. In that case, + you can + [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode). + + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in + GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. +- GitLab 14.2.0 includes a + [background migration `BackfillDraftStatusOnMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67687) + that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "BackfillDraftStatusOnMergeRequests").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("BackfillDraftStatusOnMergeRequests", job.arguments) + end + ``` + +- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123476](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123476) as the `mr_iid`. + +### Geo installations **(PREMIUM SELF)** + +- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) + that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization. + + Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects. + + As verification is not yet implemented for files stored in object storage (see + [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this + results in a loop that consistently fails for all objects stored in object storage. + + For information on how to fix this, see + [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication). + +- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication + wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture + (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. + + You can check if you are affected by running: + + ```shell + docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' + ``` + + Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site. + If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` + (there can be a `mediaType` entry at several levels, we only care about the top level entry), + then you don't need to do anything. + + Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following: + + ```ruby + list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' + + Geo::ContainerRepositoryRegistry.synced.each do |gcr| + cr = gcr.container_repository + primary = Geo::ContainerRepositorySync.new(cr) + cr.tags.each do |tag| + primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name)) + next unless primary_manifest['mediaType'].eql?(list_type) + + cr.delete_tag_by_name(tag.name) + end + primary.execute + end + ``` + + If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, + we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1. + +## 14.1.0 + +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) + but can upgrade to 14.1.Z. + + It is not required for instances already running 14.0.5 (or later) to stop at 14.1.Z. + 14.1 is included on the upgrade path for the broadest compatibility + with self-managed installations, and ensure 14.0.0-14.0.4 installations do not + encounter issues with [batched background migrations](../background_migrations.md#batched-background-migrations). + +- Upgrading to GitLab [14.5](#1450) (or later) may take a lot longer if you do not upgrade to at least 14.1 + first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the + background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must + run the migration in the foreground and therefore take a lot longer to complete. + +- When [Maintenance mode](../../administration/maintenance_mode/index.md) is + enabled, users cannot sign in with SSO, SAML, or LDAP. + + Users who were signed in before Maintenance mode was enabled, continue to be + signed in. If the administrator who enabled Maintenance mode loses their + session, then they can't disable Maintenance mode via the UI. In that case, + you can + [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode). + + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in + GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. + +- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123475](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123475) as the `mr_iid`. + +### Geo installations **(PREMIUM SELF)** + +- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication + wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture + (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. + + You can check if you are affected by running: + + ```shell + docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType' + ``` + + Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site. + If the output matches `application/vnd.docker.distribution.manifest.list.v2+json` + (there can be a `mediaType` entry at several levels, we only care about the top level entry), + then you don't need to do anything. + + Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following: + + ```ruby + list_type = 'application/vnd.docker.distribution.manifest.list.v2+json' + + Geo::ContainerRepositoryRegistry.synced.each do |gcr| + cr = gcr.container_repository + primary = Geo::ContainerRepositorySync.new(cr) + cr.tags.each do |tag| + primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name)) + next unless primary_manifest['mediaType'].eql?(list_type) + + cr.delete_tag_by_name(tag.name) + end + primary.execute + end + ``` + + If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, + we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1. +- We found an issue where [Primary sites cannot be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). + + This bug only exists in the UI and does not block the removal of Primary sites using any other method. + + If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site + by using the [Geo Sites API](../../api/geo_nodes.md#delete-a-geo-node). + +## 14.0.0 + +Prerequisites: + +- The [GitLab 14.0 release post contains several important notes](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#upgrade) + about pre-requisites including [using Patroni instead of repmgr](../../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni), + migrating [to hashed storage](../../administration/raketasks/storage.md#migrate-to-hashed-storage), + and [to Puma](../../administration/operations/puma.md). +- The support of PostgreSQL 11 [has been dropped](../../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. + +Long running batched background database migrations: + +- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](../background_migrations.md#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or later. +- Due to an issue where `BatchedBackgroundMigrationWorkers` were + [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) + for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) + that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). + + After you update to 14.0.5 or a later 14.0 patch version, + [batched background migrations must finish](../background_migrations.md#batched-background-migrations) + before you upgrade to a later version. + + If the migrations are not finished and you try to upgrade to a later version, + you see an error like: + + ```plaintext + Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': + ``` + + See how to [resolve this error](../background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). + +Other issues: + +- In GitLab 13.3 some [pipeline processing methods were deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/218536) + and this code was completely removed in GitLab 14.0. If you plan to upgrade from + **GitLab 13.2 or older** directly to 14.0, this is [unsupported](../index.md#upgrading-to-a-new-major-version). + You should instead follow a [supported upgrade path](../index.md#upgrade-paths). +- When [Maintenance mode](../../administration/maintenance_mode/index.md) is + enabled, users cannot sign in with SSO, SAML, or LDAP. + + Users who were signed in before Maintenance mode was enabled, continue to be + signed in. If the administrator who enabled Maintenance mode loses their + session, then they can't disable Maintenance mode via the UI. In that case, + you can + [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode). + + [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in + GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. +- **In GitLab 13.12.2 and later**, users with expired passwords can no longer authenticate with API and Git using tokens because of + the [Insufficient Expired Password Validation](https://about.gitlab.com/releases/2021/06/01/security-release-gitlab-13-12-2-released/#insufficient-expired-password-validation) + security fix. If your users get authentication issues following the upgrade, check that their password is not expired: + + 1. [Connect to the PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-postgresql-database) and execute the + following query: + + ```sql + select id,username,password_expires_at from users where password_expires_at < now(); + ``` + + 1. If the user is in the returned list, reset the `password_expires_at` for that user: + + ```sql + update users set password_expires_at = null where username='<USERNAME>'; + ``` + +### Linux package installations + +- In GitLab 13.0, `sidekiq-cluster` was enabled by default and the `sidekiq` service ran `sidekiq-cluster` under the hood. + However, users could control this behavior using `sidekiq['cluster']` setting to run Sidekiq directly instead. Users + could also run `sidekiq-cluster` separately using the various `sidekiq_cluster[*]` settings available in `gitlab.rb`. + However these features were deprecated and are now being removed. + + Starting with GitLab 14.0, `sidekiq-cluster` becomes the only way to run Sidekiq in Linux package installations. As + part of this process, support for the following settings in `gitlab.rb` is being removed: + + - `sidekiq['cluster']` setting. Sidekiq can only be run using `sidekiq-cluster` now. + - `sidekiq_cluster[*]` settings. They should be set via respective `sidekiq[*]` counterparts. + - `sidekiq['concurrency']` setting. The limits should be controlled using the two settings `sidekiq['min_concurrency']` + and `sidekiq['max_concurrency']`. + +- In GitLab 13.0, Puma became the default web server for GitLab, but users were still able to continue using Unicorn if + needed. Starting with GitLab 14.0, Unicorn is no longer supported as a webserver for GitLab and is no longer shipped + with the Linux package. + + Users must migrate to Puma following [the documentation](../../administration/operations/puma.md) to upgrade to GitLab + 14.0. +- The Consul version has been updated from 1.6.10 to 1.9.6 for Geo and multi-node PostgreSQL installs. Its important + that Consul nodes be upgraded and restarted one at a time. + + For more information, see [Upgrade the Consul nodes](../../administration/consul.md#upgrade-the-consul-nodes). +- Starting with GitLab 14.0, GitLab automatically generates a password for initial administrator user (`root`) and stores + this value to `/etc/gitlab/initial_root_password`. + + For more information, see + [Set up the initial password](https://docs.gitlab.com/omnibus/installation/index.html#set-up-the-initial-password). +- The binaries for PostgreSQL 11 and repmgr have been removed. Prior to upgrading, administrators of Linux package + installations must: + 1. Ensure the installation is using [PostgreSQL 12](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). + 1. If using repmgr, [convert to using patroni](../../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). +- Two configuration options for Redis were deprecated in GitLab 13 and removed in GitLab 14: + + - `redis_slave_role` is replaced with `redis_replica_role`. + - `redis['client_output_buffer_limit_slave']` is replaced with `redis['client_output_buffer_limit_replica']`. + + Redis Cache nodes being upgraded from GitLab 13.12 to 14.0 that still refer to `redis_slave_role` in `gitlab.rb` will + encounter an error in the output of `gitlab-ctl reconfigure`: + + ```plaintext + There was an error running gitlab-ctl reconfigure: + + The following invalid roles have been set in 'roles': redis_slave_role + ``` + +### Geo installations **(PREMIUM SELF)** + +- We found an issue where [Primary sites cannot be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). + + This bug only exists in the UI and does not block the removal of Primary sites using any other method. + + If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site + by using the [Geo Sites API](../../api/geo_nodes.md#delete-a-geo-node). + +### Upgrading to later 14.Y releases + +- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, + because of [batched background migrations](../background_migrations.md#batched-background-migrations). + 1. Upgrade first to either: + - 14.0.5 or a later 14.0.Z patch release. + - 14.1.0 or a later 14.1.Z patch release. + 1. [Batched background migrations must finish](../background_migrations.md#batched-background-migrations) + before you upgrade to a later version [and may take longer than usual](#1400). diff --git a/doc/update/versions/gitlab_15_changes.md b/doc/update/versions/gitlab_15_changes.md new file mode 100644 index 00000000000..12cbf5c5ec1 --- /dev/null +++ b/doc/update/versions/gitlab_15_changes.md @@ -0,0 +1,921 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab 15 changes **(FREE SELF)** + +This page contains upgrade information for minor and patch versions of GitLab 15. +Ensure you review these instructions for: + +- Your installation type. +- All versions between your current version and your target version. + +For more information about upgrading GitLab Helm Chart, see [the release notes for 6.0](https://docs.gitlab.com/charts/releases/6_0.html). + +## 15.11.1 + +- Many [project importers](../../user/project/import/index.md) and [group importers](../../user/group/import/index.md) now + require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation + for any importers you use. + +## 15.11.0 + +- **Upgrade to patch release 15.11.3 or later**. This avoids [issue 408304](https://gitlab.com/gitlab-org/gitlab/-/issues/408304) when upgrading from 15.5.0 and earlier. + +### Linux package installations + +In GitLab 15.11, PostgreSQL will automatically be upgraded to 13.x except for the following cases: + +- You are running the database in high availability using Patroni. +- Your database nodes are part of a GitLab Geo configuration. +- You have specifically [opted out](https://docs.gitlab.com/omnibus/settings/database.html#opt-out-of-automatic-postgresql-upgrades) from automatically upgrading PostgreSQL. +- You have `postgresql['version'] = 12` in your `/etc/gitlab/gitlab.rb`. + +Fault-tolerant and Geo installations support manual upgrades to PostgreSQL 13, +see [Packaged PostgreSQL deployed in an HA/Geo Cluster](https://docs.gitlab.com/omnibus/settings/database.html#packaged-postgresql-deployed-in-an-hageo-cluster). + +### Geo installations **(PREMIUM SELF)** + +- Some project imports do not initialize wiki repositories on project creation. See + [the details and workaround](gitlab_16_changes.md#wiki-repositories-not-initialized-on-project-creation). +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). + +#### `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13 + +| Affected minor releases | Affected patch releases | Fixed in | +|-------------------------|-------------------------|----------| +| 15.2 - 15.10 | All | None | +| 15.11 | 15.11.0 - 15.11.11 | 15.11.12 and later | + +A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the +built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database +to version 13. This leaves the secondary site in a broken state, and prevents +upgrading the Geo installation to GitLab 16.x +([PostgreSQL 12 support has removed in 16.0](../deprecations.md#postgresql-12-deprecated) and later +releases). This occurs on secondary sites using the bundled PostgreSQL +software, running both the secondary main Rails database and tracking database +on the same node. There is a manual +[workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround) +if you can't upgrade to 15.11.12 and later. + +## 15.11.x + +- A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/411604) can cause new LDAP users signing in for the first time to be assigned a username based on their email address instead of their LDAP username attribute. A manual workaround is to set `gitlab_rails['omniauth_auto_link_ldap_user'] = true`, or upgrade to GitLab 16.1 or later where the bug has been fixed. + +## 15.10.5 + +- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. + - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. + - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. + - Elasticsearch does not need to be enabled for this to occur. + - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. +- Many [project importers](../../user/project/import/index.md) and [group importers](../../user/group/import/index.md) now + require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation + for any importers you use. + +## 15.10.0 + +- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. + - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. + - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. + - Elasticsearch does not need to be enabled for this to occur. + - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. +- A [bug with zero-downtime reindexing](https://gitlab.com/gitlab-org/gitlab/-/issues/422938) can cause a `Couldn't load task status` error when you reindex. You might also get a `sliceId must be greater than 0 but was [-1]` error on the Elasticsearch host. As a workaround, consider [reindexing from scratch](../../integration/advanced_search/elasticsearch_troubleshooting.md#last-resort-to-recreate-an-index) or upgrading to GitLab 16.3. +- Gitaly configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.10 while backwards compatibility is + maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](gitlab_16_changes.md#gitaly-configuration-structure-change). +- You might encounter the following error while upgrading to GitLab 15.10 or later: + + ```shell + STDOUT: rake aborted! + StandardError: An error has occurred, all later migrations canceled: + PG::CheckViolation: ERROR: check constraint "check_70f294ef54" is violated by some row + ``` + + This error is caused by a [batched background migration introduced in GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107701) + not being finalized before GitLab 15.10. To resolve this error: + + 1. Execute the following SQL statement using the database console (`sudo gitlab-psql` for Linux package installs): + + ```sql + UPDATE oauth_access_tokens SET expires_in = '7200' WHERE expires_in IS NULL; + ``` + + 1. [Re-run database migrations](../../administration/raketasks/maintenance.md#run-incomplete-database-migrations). + +- You might also encounter the following error while upgrading to GitLab 15.10 or later: + + ```shell + "exception.class": "ActiveRecord::StatementInvalid", + "exception.message": "PG::SyntaxError: ERROR: zero-length delimited identifier at or near \"\"\"\"\nLINE 1: ...COALESCE(\"lock_version\", 0) + 1 WHERE \"ci_builds\".\"\" IN (SEL...\n + ``` + + This error is caused by a [batched background migration introduced in GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81410) + not being finalized before upgrading to GitLab 15.10 or later. To resolve this error, it is safe to [mark the migration as complete](../background_migrations.md#mark-a-failed-migration-finished): + + ```ruby + # Start the rails console + + connection = Ci::ApplicationRecord.connection + + Gitlab::Database::SharedModel.using_connection(connection) do + migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( + Gitlab::Database.gitlab_schemas_for_connection(connection), 'NullifyOrphanRunnerIdOnCiBuilds', :ci_builds, :id, []) + + # mark all jobs completed + migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states[:succeeded].value) + migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) + end + ``` + + For more information, see [issue 415724](https://gitlab.com/gitlab-org/gitlab/-/issues/415724). + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). + +## 15.9.0 + +- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq. + - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur. + - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough. + - Elasticsearch does not need to be enabled for this to occur. + - To resolve this issue, upgrade to 15.11 or use the workaround in the issue. +- **Upgrade to patch release 15.9.3 or later**. This provides fixes for two database migration bugs: + - Patch releases 15.9.0, 15.9.1, 15.9.2 have a bug that can cause data loss + from the user profile fields `linkedin`, `twitter`, `skype`, `website_url`, + `location`, and `organization`. For more information, see + [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216). + - The second [bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/394760) ensures it is possible to upgrade directly from 15.4.x. +- As part of the [CI Partitioning effort](../../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. +- Praefect's metadata verifier's [invalid metadata deletion behavior](../../administration/gitaly/praefect.md#enable-deletions) is now enabled by default. + + The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its + metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk. + After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica. + + Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete + deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as + unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove + the repository using `praefect remove-repository` to remove the repository's remaining records. + + You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. [Read more about repository verification, and to see an example log entry](../../administration/gitaly/praefect.md#repository-verification). +- Praefect configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.9 while backwards compatibility is + maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](gitlab_16_changes.md#praefect-configuration-structure-change). + +### Self-compiled installations + +- For **self-compiled (source) installations**, with the addition of `gitlab-sshd` the Kerberos headers are needed to build GitLab Shell. + + ```shell + sudo apt install libkrb5-dev + ``` + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). + +## 15.8.2 + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.8.1 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.8.0 + +- Git 2.38.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git). +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.6 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.5 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.4 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.3 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.2 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the upgrades. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.1 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.7.0 + +- This version validates a `NOT NULL DB` constraint on the `issues.work_item_type_id` column. + To upgrade to this version, no records with a `NULL` `work_item_type_id` should exist on the `issues` table. + There are multiple `BackfillWorkItemTypeIdForIssues` background migrations that will be finalized with + the `EnsureWorkItemTypeBackfillMigrationFinished` post-deploy migration. +- GitLab 15.4.0 introduced a [batched background migration](../background_migrations.md#batched-background-migrations) to + [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This + migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration + has completed successfully before upgrading to 15.7.0. +- A database constraint is added, specifying that the `namespace_id` column on the issues + table has no `NULL` values. + + - If the `namespace_id` batched background migration from 15.4 failed (see above) then the 15.7 upgrade + fails with a database migration error. + + - On GitLab instances with large issues tables, validating this constraint causes the upgrade to take + longer than usual. All database changes need to complete within a one-hour limit: + + ```plaintext + FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] + [..] + Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: + ``` + + A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). +- The default Sidekiq `max_concurrency` has been changed to 20. This is now + consistent in our documentation and product defaults. + + For example, previously: + + - Linux package installation default (`sidekiq['max_concurrency']`): 50 + - Self-compiled installation default: 50 + - Helm chart default (`gitlab.sidekiq.concurrency`): 25 + + Reference architectures still use a default of 10 as this is set specifically + for those configurations. + + Sites that have configured `max_concurrency` will not be affected by this change. + [Read more about the Sidekiq concurrency setting](../../administration/sidekiq/extra_sidekiq_processes.md#concurrency). +- GitLab Runner 15.7.0 introduced a breaking change that affects CI/CD jobs: [Correctly handle expansion of job file variables](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3613). + Previously, job-defined variables that referred to + [file type variables](../../ci/variables/index.md#use-file-type-cicd-variables) + were expanded to the value of the file variable (its content). This behavior did not + respect the typical rules of shell variable expansion. There was also the potential + that secrets or sensitive information could leak if the file variable and its + contents printed. For example, if they were printed in an echo output. For more information, + see [Understanding the file type variable expansion change in GitLab 15.7](https://about.gitlab.com/blog/2023/02/13/impact-of-the-file-type-variable-change-15-7/). +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.6.7 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.6.6 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. + +## 15.6.5 + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.6.4 + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6, and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.6.3 + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.6.2 + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.6.1 + +### Geo installations **(PREMIUM SELF)** + +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.6.0 + +- You should use one of the [officially supported PostgreSQL versions](../../administration/package_information/postgresql_versions.md). Some database migrations can cause stability and performance issues with older PostgreSQL versions. +- Git 2.37.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git). +- A database change to modify the behavior of four indexes fails on instances + where these indexes do not exist: + + ```plaintext + Caused by: + PG::UndefinedTable: ERROR: relation "index_issues_on_title_trigram" does not exist + ``` + + The other three indexes are: `index_merge_requests_on_title_trigram`, `index_merge_requests_on_description_trigram`, + and `index_issues_on_description_trigram`. + + This issue was [fixed in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105375) and backported + to GitLab 15.6.2. The issue can also be worked around: + [read about how to create these indexes](https://gitlab.com/gitlab-org/gitlab/-/issues/378343#note_1199863087). + +### Linux package installations + +In GitLab 15.6, the [PostgreSQL versions shipped with `omnibus-gitlab` packages](../../administration/package_information/postgresql_versions.md) +have been upgraded to 12.12 and 13.8. Unless +[explicitly opted out](https://docs.gitlab.com/omnibus/settings/database.html#automatic-restart-when-the-postgresql-version-changes), +this can cause an automatic restart of the PostgreSQL service, and can +potentially cause downtime. + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). +- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover. +- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover. + - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2. + - Versions containing fix: GitLab 15.8.3 and later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.5 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.4 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.3 + +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. + - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. + - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['routing_rules'] = [['*', 'default']] + ``` + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.2 + +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. + - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. + - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['routing_rules'] = [['*', 'default']] + ``` + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.1 + +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. + - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. + - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['routing_rules'] = [['*', 'default']] + ``` + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.5.0 + +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. + - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. + - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['routing_rules'] = [['*', 'default']] + ``` + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). + +## 15.4.6 + +- Due to a [bug introduced in curl in GitLab 15.4.6](https://github.com/curl/curl/issues/10122), the [`no_proxy` environment variable may not work properly](../../administration/geo/replication/troubleshooting.md#secondary-site-returns-received-http-code-403-from-proxy-after-connect). Either downgrade to GitLab 15.4.5, or upgrade to GitLab 15.5.7 or a later version. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.5 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.4 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.3 + +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.2 + +- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. + - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.1 + +- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. + - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. + +## 15.4.0 + +- GitLab 15.4.0 includes a [batched background migration](../background_migrations.md#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318). + This migration might take hours or days to complete on larger GitLab instances. +- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../../administration/gitaly/praefect.md#customize-time-server-setting). +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. + - The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior. + - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['routing_rules'] = [['*', 'default']] + ``` + +- The structure of `/etc/gitlab/gitlab-secrets.json` was modified in [GitLab 15.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6310), + and new configuration was added to `gitlab_pages`, `grafana`, and `mattermost` sections. + In a highly available or GitLab Geo environment, secrets need to be the same on all nodes. + If you're manually syncing the secrets file across nodes, or manually specifying secrets in + `/etc/gitlab/gitlab.rb`, make sure `/etc/gitlab/gitlab-secrets.json` is the same on all nodes. +- GitLab 15.4.0 introduced a [batched background migration](../background_migrations.md#batched-background-migrations) to + [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This + migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration + has completed successfully before upgrading to 15.7.0 or later. +- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information. +- A redesigned sign-in page is enabled by default in GitLab 15.4 and later, with improvements shipping in later releases. For more information, see [epic 8557](https://gitlab.com/groups/gitlab-org/-/epics/8557). + It can be disabled with a feature flag. Start [a Rails console](../../administration/operations/rails_console.md) and run: + + ```ruby + Feature.disable(:restyle_login_page) + ``` + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). + +## 15.3.4 + +A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + +- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. +- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +## 15.3.3 + +- In GitLab 15.3.3, [SAML Group Links](../../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See +[the API documentation](../../api/members.md). +- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + + - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. + - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +## 15.3.2 + +A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + +- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. +- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +## 15.3.1 + +A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + +- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. +- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +## 15.3.0 + +- New Git repositories created in Gitaly cluster no longer use the `@hashed` + storage path. Server hooks for new repositories must be copied into a + different location. Praefect now generates replica paths for use by Gitaly + cluster. This change is a pre-requisite for Gitaly cluster atomically + creating, deleting, and renaming Git repositories. + + To identify the replica path, + [query the Praefect repository metadata](../../administration/gitaly/troubleshooting.md#view-repository-metadata) + and pass the `@hashed` storage path to `-relative-path`. + + With this information, you can correctly install + [server hooks](../../administration/server_hooks.md). + +- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + + - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. + - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). +- LFS transfers can redirect to the primary from secondary site mid-session. See + [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session). +- Incorrect object storage LFS files deletion on Geo secondary sites. See + [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites). + +#### LFS transfers redirect to primary from secondary site mid-session + +| Affected minor releases | Affected patch releases | Fixed in | +|-------------------------|-------------------------|----------| +| 15.1 | All | None | +| 15.2 | All | None | +| 15.3 | 15.3.0 - 15.3.2 | 15.3.3 and later | + +LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. + +This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later: + +- LFS is enabled. +- LFS objects are being replicated across Geo sites. +- Repositories are being pulled by using a Geo secondary site. + +#### Incorrect object storage LFS file deletion on secondary sites + +| Affected minor releases | Affected patch releases | Fixed in | +|-------------------------|-------------------------|----------| +| 15.0 | All | None | +| 15.1 | All | None | +| 15.2 | All | None | +| 15.3 | 15.3.0 - 15.3.2 | 15.3.3 and later | + +[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) +can occur in GitLab 15.0.0 to 15.3.2 in the following situations: + +- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled. +- GitLab-managed replication to sync object storage is enabled and subsequently disabled. + +This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites +should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites. + +## 15.2.5 + +A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: + +- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. +- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions: + - 15.2.5 --> 15.3.5 + - 15.3.0 - 15.3.4 --> 15.3.5 + - 15.4.1 --> 15.4.3 + +## 15.2.0 + +- GitLab installations that have multiple web nodes should be + [upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a + configuration change in Rails that can result in inconsistent ETag key + generation. +- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../../administration/sidekiq/sidekiq_job_migration.md#migrate-queued-and-future-jobs) before starting the upgrade to GitLab 15.2.0. +- Gitaly now executes its binaries in a [runtime location](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4670). By default on Omnibus GitLab, + this path is `/var/opt/gitlab/gitaly/run/`. If this location is mounted with `noexec`, merge requests generate the following error: + + ```plaintext + fork/exec /var/opt/gitlab/gitaly/run/gitaly-<nnnn>/gitaly-git2go-v15: permission denied + ``` + + To resolve this, remove the `noexec` option from the file system mount. An alternative is to change the Gitaly runtime directory: + + 1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set. + 1. Run `sudo gitlab-ctl reconfigure`. + +### Geo installations **(PREMIUM SELF)** + +- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See + [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13). +- LFS transfers can redirect to the primary from secondary site mid-session. See + [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session). +- Incorrect object storage LFS files deletion on Geo secondary sites. See + [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites). + +## 15.1.0 + +- In GitLab 15.1.0, we are switching Rails `ActiveSupport::Digest` to use SHA256 instead of MD5. + This affects ETag key generation for resources such as raw Snippet file + downloads. To ensure consistent ETag key generation across multiple + web nodes when upgrading, all servers must first be upgraded to 15.1.6 before + upgrading to 15.2.0 or later: + + 1. Ensure all GitLab web nodes are running GitLab 15.1.6. + 1. If you run [GitLab on Kubernetes](https://docs.gitlab.com/charts/installation/) by using the cloud native GitLab Helm chart, make sure that all + webservice pods are running GitLab 15.1.Z: + + ```shell + kubectl get pods -l app=webservice -o custom-columns=webservice-image:{.spec.containers[0].image},workhorse-image:{.spec.containers[1].image} + ``` + + 1. [Enable the `active_support_hash_digest_sha256` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to switch `ActiveSupport::Digest` to use SHA256: + + 1. [Start the rails console](../../administration/operations/rails_console.md) + 1. Enable the feature flag: + + ```ruby + Feature.enable(:active_support_hash_digest_sha256) + ``` + + 1. Only then, continue to upgrade to later versions of GitLab. +- Unauthenticated requests to the [`ciConfig` GraphQL field](../../api/graphql/reference/index.md#queryciconfig) are no longer supported. + Before you upgrade to GitLab 15.1, add an [access token](../../api/rest/index.md#authentication) to your requests. + The user creating the token must have [permission](../../user/permissions.md) to create pipelines in the project. + +### Geo installations **(PREMIUM SELF)** + +- [Geo proxying](../../administration/geo/secondary_proxy/index.md) was [enabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in 15.1. This may be a breaking change. If needed, you may [disable Geo proxying](../../administration/geo/secondary_proxy/index.md#disable-geo-proxying). If you are using SAML with different URLs, you must modify your SAML configuration and your Identity Provider configuration. For more information, see the [Geo with Single Sign-On (SSO) documentation](../../administration/geo/replication/single_sign_on.md). +- LFS transfers can redirect to the primary from secondary site mid-session. See + [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session). +- Incorrect object storage LFS files deletion on Geo secondary sites. See + [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites). + +## 15.0.0 + +- Elasticsearch 6.8 [is no longer supported](../../integration/advanced_search/elasticsearch.md#version-requirements). Before you upgrade to GitLab 15.0, [update Elasticsearch to any 7.x version](../../integration/advanced_search/elasticsearch.md#upgrade-to-a-new-elasticsearch-major-version). +- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you + upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before + upgrading to GitLab 14.8 or later. + + [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) + for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) + for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled. + + After it was enabled, we have had reports of unplanned PostgreSQL restarts caused + by a database engine bug that causes a segmentation fault. + + For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). + +- The use of encrypted S3 buckets with storage-specific configuration is no longer supported after [removing support for using `background_upload`](../deprecations.md#background-upload-for-object-storage). +- The [certificate-based Kubernetes integration (DEPRECATED)](../../user/infrastructure/clusters/index.md#certificate-based-kubernetes-integration-deprecated) is disabled by default, but you can be re-enable it through the [`certificate_based_clusters` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) until GitLab 16.0. +- When you use the GitLab Helm Chart project with a custom `serviceAccount`, ensure it has `get` and `list` permissions for the `serviceAccount` and `secret` resources. +- The `FF_GITLAB_REGISTRY_HELPER_IMAGE` [feature flag](../../administration/feature_flags.md#enable-or-disable-the-feature) is removed and helper images are always pulled from GitLab Registry. + +### Linux package installations + +- The [`custom_hooks_dir`](../../administration/server_hooks.md#create-global-server-hooks-for-all-repositories) setting for configuring global server hooks is now configured in + Gitaly. The previous implementation in GitLab Shell was removed in GitLab 15.0. With this change, global server hooks are stored only inside a subdirectory named after the + hook type. Global server hooks can no longer be a single hook file in the root of the custom hooks directory. For example, you must use `<custom_hooks_dir>/<hook_name>.d/*` rather + than `<custom_hooks_dir>/<hook_name>`. + - Use `gitaly['custom_hooks_dir']` in `gitlab.rb` ([introduced in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4208)) + for Omnibus GitLab. This replaces `gitlab_shell['custom_hooks_dir']`. +- PostgreSQL 13.6 is being shipped as the default version for fresh installs and + 12.10 for upgrades. You can manually upgrade to PostgreSQL 13.6 following the + [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). + Because of underlying structural changes, the running PostgreSQL process + **_must_** be restarted when it is upgraded before running database + migrations. If automatic restart is skipped, you must run the following + command before migrations are run: + + ```shell + # If using PostgreSQL + sudo gitlab-ctl restart postgresql + + # If using Patroni for Database replication + sudo gitlab-ctl restart patroni + ``` + + If PostgreSQL is not restarted, you might face + [errors related to loading libraries](https://docs.gitlab.com/omnibus/settings/database.html#could-not-load-library-plpgsqlso). + +- Starting with GitLab 15.0, `postgresql` and `geo-postgresql` services are + automatically restarted when the PostgreSQL version changes. Restarting + PostgreSQL services causes downtime due to the temporary unavailability of the + database for operations. While this restart is mandatory for proper functioning + of the Database services, you might want more control over when the PostgreSQL + is restarted. For that purpose, you can choose to skip the automatic restarts as + part of `gitlab-ctl reconfigure` and manually restart the services. + + To skip automatic restarts as part of GitLab 15.0 upgrade, perform the following + steps before the upgrade: + + 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + # For PostgreSQL/Patroni + postgresql['auto_restart_on_version_change'] = false + + # For Geo PostgreSQL + geo_postgresql['auto_restart_on_version_change'] = false + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + NOTE: + It is mandatory to restart PostgreSQL when underlying version changes, to avoid + errors like the [one related to loading necessary libraries](https://docs.gitlab.com/omnibus/settings/database.html#could-not-load-library-plpgsqlso) + that can cause downtime. So, if you skip the automatic restarts using the above + method, ensure that you restart the services manually before upgrading to GitLab + 15.0. + +- Starting with GitLab 15.0, the `AES256-GCM-SHA384` SSL cipher will not be allowed by + NGINX by default. If you require this cipher (for example, if you use + [AWS's Classic Load Balancer](https://docs.aws.amazon.com/en_en/elasticloadbalancing/latest/classic/elb-ssl-security-policy.html#ssl-ciphers)), + you can add the cipher back to the allow list by following the steps below: + + 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + nginx['ssl_ciphers'] = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +- Support for Gitaly's internal socket path is removed. + In GitLab 14.10, Gitaly introduced a new directory that holds all runtime + data Gitaly requires to operate correctly. This new directory replaces the + old internal socket directory, and consequentially the usage of + `gitaly['internal_socket_dir']` was deprecated in favor of + `gitaly['runtime_dir']`. + + The old `gitaly['internal_socket_dir']` configuration was removed in this release. + +- Background uploads settings for object storage are removed. + Object storage now preferentially uses direct uploads. + + The following keys are no longer supported in `/etc/gitlab/gitlab.rb`: + + - `gitlab_rails['artifacts_object_store_direct_upload']` + - `gitlab_rails['artifacts_object_store_background_upload']` + - `gitlab_rails['external_diffs_object_store_direct_upload']` + - `gitlab_rails['external_diffs_object_store_background_upload']` + - `gitlab_rails['lfs_object_store_direct_upload']` + - `gitlab_rails['lfs_object_store_background_upload']` + - `gitlab_rails['uploads_object_store_direct_upload']` + - `gitlab_rails['uploads_object_store_background_upload']` + - `gitlab_rails['packages_object_store_direct_upload']` + - `gitlab_rails['packages_object_store_background_upload']` + - `gitlab_rails['dependency_proxy_object_store_direct_upload']` + - `gitlab_rails['dependency_proxy_object_store_background_upload']` + +### Self-compiled installations + +- Support for more than one database has been added to GitLab. For **self-compiled (source) installations**, + `config/database.yml` must include a database name in the database configuration. + The `main: database` must be first. If an invalid or deprecated syntax is used, an error is generated + during application start: + + ```plaintext + ERROR: This installation of GitLab uses unsupported 'config/database.yml'. + The main: database needs to be defined as a first configuration item instead of primary. (RuntimeError) + ``` + + Previously, the `config/database.yml` file looked like the following: + + ```yaml + production: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + ... + ``` + + Starting with GitLab 15.0, it must define a `main` database first: + + ```yaml + production: + main: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + ... + ``` + +### Geo installations **(PREMIUM SELF)** + +- Incorrect object storage LFS files deletion on Geo secondary sites. See + [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites). diff --git a/doc/update/versions/gitlab_16_changes.md b/doc/update/versions/gitlab_16_changes.md index cd51ca2a518..04581a88a93 100644 --- a/doc/update/versions/gitlab_16_changes.md +++ b/doc/update/versions/gitlab_16_changes.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab 16 changes +# GitLab 16 changes **(FREE SELF)** This page contains upgrade information for minor and patch versions of GitLab 16. Ensure you review these instructions for: @@ -12,11 +12,76 @@ Ensure you review these instructions for: - Your installation type. - All versions between your current version and your target version. -Some GitLab installations must upgrade to GitLab 16.0 before upgrading to any other version. For more information, see -[Long-running user type data change](#long-running-user-type-data-change). - For more information about upgrading GitLab Helm Chart, see [the release notes for 7.0](https://docs.gitlab.com/charts/releases/7_0.html). +## Issues to be aware of when upgrading from 15.11 + +- Some GitLab installations must upgrade to GitLab 16.0 before upgrading to any other version. For more information, see + [Long-running user type data change](#long-running-user-type-data-change). +- Other installations can skip 16.0, 16.1, and 16.2 as the first required stop on the upgrade path is 16.3. Review the notes for those intermediate + versions. +- If your GitLab instance upgraded first to 15.11.0, 15.11.1, or 15.11.2 the database schema is incorrect. + Recommended: perform the workaround before upgrading to 16.x. + See [the details and workaround](#undefined-column-error-upgrading-to-162-or-later). +- Linux package installations must change Gitaly and Praefect configuration structure before upgrading to GitLab 16. + **To avoid data loss** reconfigure Praefect first, and as part of the new configuration, disable metadata verification. + Read more: + + - [Praefect configuration structure change](#praefect-configuration-structure-change). + - [Gitaly configuration structure change](#gitaly-configuration-structure-change). + +## 16.4.0 + +- Updating a group path [received a bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/419289) that uses a database index introduced in 16.3. + + If you upgrade to 16.4 from a version lower than 16.3, you must execute `ANALYZE packages_packages;` in the database before you use it. + +- You might encounter the following error while upgrading to GitLab 16.4 or later: + + ```plaintext + main: == 20230830084959 ValidatePushRulesConstraints: migrating ===================== + main: -- execute("SET statement_timeout TO 0") + main: -> 0.0002s + main: -- execute("ALTER TABLE push_rules VALIDATE CONSTRAINT force_push_regex_size_constraint;") + main: -> 0.0004s + main: -- execute("RESET statement_timeout") + main: -> 0.0003s + main: -- execute("ALTER TABLE push_rules VALIDATE CONSTRAINT delete_branch_regex_size_constraint;") + rails aborted! + StandardError: An error has occurred, all later migrations canceled: + + PG::CheckViolation: ERROR: check constraint "delete_branch_regex_size_constraint" of relation "push_rules" is violated by some row + ``` + + These constraints might return an error: + + - `author_email_regex_size_constraint` + - `branch_name_regex_size_constraint` + - `commit_message_negative_regex_size_constraint` + - `commit_message_regex_size_constraint` + - `delete_branch_regex_size_constraint` + - `file_name_regex_size_constraint` + - `force_push_regex_size_constraint` + + To fix the error, find the records in the `push_rules` table that exceed the 511 + character limit. + + ```sql + ;; replace `delete_branch_regex` with a name of the field used in constraint + SELECT id FROM push_rules WHERE LENGTH(delete_branch_regex) > 511; + ``` + + Reduce the value length of the regex field for affected push rules records, then + retry the migration. + +### Self-compiled installations + +- A new method of configuring paths for the GitLab secret and custom hooks is preferred in GitLab 16.4 and later: + 1. Update your configuration `[gitlab] secret_file` to [configure the path](../../administration/gitaly/reference.md#gitlab) to the GitLab secret token. + 1. If you have custom hooks, update your configuration `[hooks] custom_hooks_dir` to [configure the path](../../administration/gitaly/reference.md#custom-hooks) to + server-side custom hooks. + 1. Remove the `[gitlab-shell] dir` configuration. + ## 16.3.0 - For Go applications, [`crypto/tls`: verifying certificate chains containing large RSA keys is slow (CVE-2023-29409)](https://github.com/golang/go/issues/61460) @@ -54,6 +119,17 @@ Specific information applies to Linux package installations: - [An informal explanation](https://gitlab.com/gitlab-org/gitlab/-/issues/416714#note_1482388504). - `omnibus-gitlab` [merge request 7035](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/7035), which introduces the environment variable. +### Geo installations + +Specific information applies to installations using Geo: + +- Git pulls against a secondary Geo site are being proxied to the primary Geo site even when that secondary site is up to date. You are impacted if you are using Geo to accelerate remote users who make Git pull requests against a secondary Geo site. + + - Impacted versions: 16.3.0 - 16.3.2 + - Versions containing fix: 16.4.0 + + For more information, see [issue 425224](https://gitlab.com/gitlab-org/gitlab/-/issues/425224). + ## 16.2.0 - Legacy LDAP configuration settings may cause @@ -62,11 +138,15 @@ Specific information applies to Linux package installations: in the `tls_options` hash, or use the legacy `gitlab_rails['ldap_host']` option. See the [configuration workarounds](https://gitlab.com/gitlab-org/gitlab/-/issues/419485#workarounds) for more details. -- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4, - 16.2.3, 16.3.0, and later. - - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2. - - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) - to resync the affected job artifacts. +- If your GitLab database was created by or upgraded via versions 15.11.0 - 15.11.2 inclusive, upgrading to GitLab 16.2 fails with: + + ```plaintext + PG::UndefinedColumn: ERROR: column "id_convert_to_bigint" of relation "ci_build_needs" does not exist + LINE 1: ...db_config_name:main*/ UPDATE "ci_build_needs" SET "id_conver... + ``` + + See [the details and workaround](#undefined-column-error-upgrading-to-162-or-later). + - You might encounter the following error while upgrading to GitLab 16.2 or later: ```plaintext @@ -98,6 +178,17 @@ Specific information applies to Linux package installations: - Git 2.41.0 and later is required by Gitaly. You should use the [Git version provided by Gitaly](../../install/installation.md#git). +### Geo installations + +Specific information applies to installations using Geo: + +- New job artifacts are not replicated by Geo if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4, + 16.2.3, 16.3.0, and later. + - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2. + - While running an affected version, artifacts which appeared to become synced may actually be missing on the secondary site. + Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later. + You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed. + ## 16.1.0 - A `BackfillPreparedAtMergeRequests` background migration is finalized with @@ -106,11 +197,6 @@ Specific information applies to Linux package installations: [backfill `prepared_at` values on the `merge_requests` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111865). This migration may take multiple days to complete on larger GitLab instances. Make sure the migration has completed successfully before upgrading to 16.1.0. -- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4, - 16.2.3, 16.3.0, and later. - - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2. - - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) - to resync the affected job artifacts. ### Self-compiled installations @@ -118,23 +204,41 @@ Specific information applies to Linux package installations: [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118645). For more information, see the [`puma.rb.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/16-0-stable-ee/config/puma.rb.example) file. -### Geo installations +### Geo installations **(PREMIUM SELF)** Specific information applies to installations using Geo: -- Some project imports do not initialize wiki repositories on project creation. Because of the migration of project wikis to - SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). - This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing - repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for - these wiki repositories. If you have not imported projects you are not impacted by this issue. - - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2. - - Versions containing fix: GitLab 16.1.3 and later. +- Some project imports do not initialize wiki repositories on project creation. See + [the details and workaround](#wiki-repositories-not-initialized-on-project-creation). - Because of the migration of project designs to SSF, [missing design repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/414279). This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for these design repositories. You could be impacted by this issue even if you have not imported projects. - - Impacted versions: GitLab versions 16.1.x. - - Versions containing fix: GitLab 16.2.0 and later. + - Impacted versions: GitLab versions 16.1.0 - 16.1.2 + - Versions containing fix: GitLab 16.1.3 and later. +- New job artifacts are not replicated by Geo if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4, + 16.2.3, 16.3.0, and later. + - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2. + - While running an affected version, artifacts which appeared to become synced may actually be missing on the secondary site. + Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later. + You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed. + +#### Wiki repositories not initialized on project creation + +| Affected minor releases | Affected patch releases | Fixed in | +|-------------------------|-------------------------|----------| +| 15.11 | All | None | +| 16.0 | All | None | +| 16.1 | 16.1.0 - 16.1.2 | 16.1.3 and later | + +Some project imports do not initialize wiki repositories on project creation. +Since the migration of project wikis to SSF, +[missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). +This is not a result of an actual replication/verification failure but an +invalid internal state for these missing repositories inside Geo and results in +errors in the logs and the verification progress reporting a failed state for +these wiki repositories. If you have not imported projects you are not impacted +by this issue. ## 16.0.0 @@ -168,18 +272,371 @@ Specific information applies to Linux package installations: Workaround is to make use of a different key type, or upgrade the client OpenSSH to a version >= 8.7. -### Geo installations +- [Migrate your Praefect configuration to the new structure](#praefect-configuration-structure-change) + to ensure all your `praefect['..']` settings continue to work in GitLab 16.0 and later. + +- [Migrate your Gitaly configuration to the new structure](#gitaly-configuration-structure-change) + to ensure all your `gitaly['..']` settings continue to work in GitLab 16.0 and later. + +### Geo installations **(PREMIUM SELF)** Specific information applies to installations using Geo: -- Some project imports do not initialize wiki repositories on project creation. Because of the migration of project wikis to - SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). - This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing - repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for - these wiki repositories. If you have not imported projects you are not impacted by this issue. +- Some project imports do not initialize wiki repositories on project creation. See + [the details and workaround](#wiki-repositories-not-initialized-on-project-creation). + +### Gitaly configuration structure change + +The Gitaly configuration structure in the Linux package +[changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 +to be consistent with the Gitaly configuration structure used in +self-compiled installations. + +As a result of this change, a single hash under `gitaly['configuration']` holds most Gitaly +configuration. Some `gitaly['..']` configuration options continue to be used by GitLab 16.0 and later: + +- `enable` +- `dir` +- `bin_path` +- `env_directory` +- `env` +- `open_files_ulimit` +- `consul_service_name` +- `consul_service_meta` + +Migrate by moving your existing configuration under the new structure. The new structure is supported from GitLab 15.10. + +**Migrate to the new structure** + +WARNING: +If you are running Gitaly cluster, [migrate Praefect to the new configuration structure **first**](#praefect-configuration-structure-change). +Once this change is tested, proceed with your Gitaly nodes. +If Gitaly is misconfigured as part of the configuration structure change, [repository verification](../../administration/gitaly/praefect.md#repository-verification) +will [delete metadata required for Gitaly cluster to work](https://gitlab.com/gitlab-org/gitaly/-/issues/5529). +To protect against configuration mistakes, temporarily disable repository verification in Praefect. + +1. If you're running Gitaly Cluster, ensure repository verification is disabled on all Praefect nodes. + Configure `verification_interval: 0`, and apply with `gitlab-ctl reconfigure`. +1. When applying the new structure to your configuration + - Replace the `...` with the value from the old key. + - When configuring `storage` to replace `git_data_dirs`, **you must append `repositories` to the path** as documented below. + If you miss this out your Git repositories are inaccessible until the configuration is fixed. + This misconfiguration can cause metadata deletion, and is the reason for disabling repository verification. + - Skip any keys you haven't configured a value for previously. + - Recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. +1. Apply the change with `gitlab-ctl reconfigure`. +1. Test Git repository functionality in GitLab. +1. Remove the old keys from the configuration once migrated, and then re-run `gitlab-ctl reconfigure`. +1. Recommended, if you're running Gitaly Cluster. Reinstate Praefect [repository verification](../../administration/gitaly/praefect.md#repository-verification) + by removing `verification_interval: 0`. + +The new structure is documented below with the old keys described in a comment above the new keys. + +```ruby +gitaly['configuration'] = { + # gitaly['socket_path'] + socket_path: ..., + # gitaly['runtime_dir'] + runtime_dir: ..., + # gitaly['listen_addr'] + listen_addr: ..., + # gitaly['prometheus_listen_addr'] + prometheus_listen_addr: ..., + # gitaly['tls_listen_addr'] + tls_listen_addr: ..., + tls: { + # gitaly['certificate_path'] + certificate_path: ..., + # gitaly['key_path'] + key_path: ..., + }, + # gitaly['graceful_restart_timeout'] + graceful_restart_timeout: ..., + logging: { + # gitaly['logging_level'] + level: ..., + # gitaly['logging_format'] + format: ..., + # gitaly['logging_sentry_dsn'] + sentry_dsn: ..., + # gitaly['logging_ruby_sentry_dsn'] + ruby_sentry_dsn: ..., + # gitaly['logging_sentry_environment'] + sentry_environment: ..., + # gitaly['log_directory'] + dir: ..., + }, + prometheus: { + # gitaly['prometheus_grpc_latency_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + grpc_latency_buckets: ..., + }, + auth: { + # gitaly['auth_token'] + token: ..., + # gitaly['auth_transitioning'] + transitioning: ..., + }, + git: { + # gitaly['git_catfile_cache_size'] + catfile_cache_size: ..., + # gitaly['git_bin_path'] + bin_path: ..., + # gitaly['use_bundled_git'] + use_bundled_binaries: ..., + # gitaly['gpg_signing_key_path'] + signing_key: ..., + # gitaly['gitconfig']. This is still an array but the type of the elements have changed. + config: [ + { + # Previously the elements contained 'section', and 'subsection' in addition to 'key'. Now + # these all should be concatenated into just 'key', separated by dots. For example, + # {section: 'first', subsection: 'middle', key: 'last', value: 'value'}, should become + # {key: 'first.middle.last', value: 'value'}. + key: ..., + value: ..., + }, + ], + }, + # Storage could previously be configured through either gitaly['storage'] or 'git_data_dirs'. Migrate + # the relevant configuration according to the instructions below. + # For 'git_data_dirs', migrate only the 'path' to the gitaly['configuration'] and leave the rest of it untouched. + storage: [ + { + # gitaly['storage'][<index>]['name'] + # + # git_data_dirs[<name>]. The storage name was configured as a key in the map. + name: ..., + # gitaly['storage'][<index>]['path'] + # + # git_data_dirs[<name>]['path']. Use the value from git_data_dirs[<name>]['path'] and append '/repositories' to it. + # + # For example, if the path in 'git_data_dirs' was '/var/opt/gitlab/git-data', use + # '/var/opt/gitlab/git-data/repositories'. The '/repositories' extension was automatically + # appended to the path configured in `git_data_dirs`. + path: ..., + }, + ], + hooks: { + # gitaly['custom_hooks_dir'] + custom_hooks_dir: ..., + }, + daily_maintenance: { + # gitaly['daily_maintenance_disabled'] + disabled: ..., + # gitaly['daily_maintenance_start_hour'] + start_hour: ..., + # gitaly['daily_maintenance_start_minute'] + start_minute: ..., + # gitaly['daily_maintenance_duration'] + duration: ..., + # gitaly['daily_maintenance_storages'] + storages: ..., + }, + cgroups: { + # gitaly['cgroups_mountpoint'] + mountpoint: ..., + # gitaly['cgroups_hierarchy_root'] + hierarchy_root: ..., + # gitaly['cgroups_memory_bytes'] + memory_bytes: ..., + # gitaly['cgroups_cpu_shares'] + cpu_shares: ..., + repositories: { + # gitaly['cgroups_repositories_count'] + count: ..., + # gitaly['cgroups_repositories_memory_bytes'] + memory_bytes: ..., + # gitaly['cgroups_repositories_cpu_shares'] + cpu_shares: ..., + } + }, + # gitaly['concurrency']. While the structure is the same, the string keys in the array elements + # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. + concurrency: ..., + # gitaly['rate_limiting']. While the structure is the same, the string keys in the array elements + # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. + rate_limiting: ..., + pack_objects_cache: { + # gitaly['pack_objects_cache_enabled'] + enabled: ..., + # gitaly['pack_objects_cache_dir'] + dir: ..., + # gitaly['pack_objects_cache_max_age'] + max_age: ..., + } +} +``` - - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2. - - Versions containing fix: GitLab 16.1.3 and later. +### Praefect configuration structure change + +The Praefect configuration structure in the Linux package +[changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 +to be consistent with the Praefect configuration structure used in +self-compiled installations. + +As a result of this change, a single hash under `praefect['configuration']` holds most Praefect +configuration. Some `praefect['..']` configuration options continue to be used by GitLab 16.0 and later: + +- `enable` +- `dir` +- `log_directory` +- `env_directory` +- `env` +- `wrapper_path` +- `auto_migrate` +- `consul_service_name` + +Migrate by moving your existing configuration under the new structure. The new structure is supported from GitLab 15.9. + +**Migrate to the new structure** + +WARNING: +Migrate Praefect to the new configuration structure **first**. +Once this change is tested, [proceed with your Gitaly nodes](#gitaly-configuration-structure-change). +If Gitaly is misconfigured as part of the configuration structure change, [repository verification](../../administration/gitaly/praefect.md#repository-verification) +will [delete metadata required for Gitaly cluster to work](https://gitlab.com/gitlab-org/gitaly/-/issues/5529). +To protect against configuration mistakes, temporarily disable repository verification in Praefect. + +1. When applying the new structure to your configuration: + - Replace the `...` with the value from the old key. + - Disable repository verification using `verification_interval: 0`, as shown below. + - Skip any keys you haven't configured a value for previously. + - Recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. +1. Apply the change with `gitlab-ctl reconfigure`. +1. Test Git repository functionality in GitLab. +1. Remove the old keys from the configuration once migrated, and then re-run `gitlab-ctl reconfigure`. + +The new structure is documented below with the old keys described in a comment above the new keys. + +```ruby +praefect['configuration'] = { + # praefect['listen_addr'] + listen_addr: ..., + # praefect['socket_path'] + socket_path: ..., + # praefect['prometheus_listen_addr'] + prometheus_listen_addr: ..., + # praefect['tls_listen_addr'] + tls_listen_addr: ..., + # praefect['separate_database_metrics'] + prometheus_exclude_database_from_default_metrics: ..., + auth: { + # praefect['auth_token'] + token: ..., + # praefect['auth_transitioning'] + transitioning: ..., + }, + logging: { + # praefect['logging_format'] + format: ..., + # praefect['logging_level'] + level: ..., + }, + failover: { + # praefect['failover_enabled'] + enabled: ..., + }, + background_verification: { + # praefect['background_verification_delete_invalid_records'] + delete_invalid_records: ..., + # praefect['background_verification_verification_interval'] + # + # IMPORTANT: + # As part of reconfiguring Praefect, disable this feature. + # Read about this above. + # + verification_interval: 0, + }, + reconciliation: { + # praefect['reconciliation_scheduling_interval'] + scheduling_interval: ..., + # praefect['reconciliation_histogram_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + histogram_buckets: ..., + }, + tls: { + # praefect['certificate_path'] + certificate_path: ..., + # praefect['key_path'] + key_path: ..., + }, + database: { + # praefect['database_host'] + host: ..., + # praefect['database_port'] + port: ..., + # praefect['database_user'] + user: ..., + # praefect['database_password'] + password: ..., + # praefect['database_dbname'] + dbname: ..., + # praefect['database_sslmode'] + sslmode: ..., + # praefect['database_sslcert'] + sslcert: ..., + # praefect['database_sslkey'] + sslkey: ..., + # praefect['database_sslrootcert'] + sslrootcert: ..., + session_pooled: { + # praefect['database_direct_host'] + host: ..., + # praefect['database_direct_port'] + port: ..., + # praefect['database_direct_user'] + user: ..., + # praefect['database_direct_password'] + password: ..., + # praefect['database_direct_dbname'] + dbname: ..., + # praefect['database_direct_sslmode'] + sslmode: ..., + # praefect['database_direct_sslcert'] + sslcert: ..., + # praefect['database_direct_sslkey'] + sslkey: ..., + # praefect['database_direct_sslrootcert'] + sslrootcert: ..., + } + }, + sentry: { + # praefect['sentry_dsn'] + sentry_dsn: ..., + # praefect['sentry_environment'] + sentry_environment: ..., + }, + prometheus: { + # praefect['prometheus_grpc_latency_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + grpc_latency_buckets: ..., + }, + # praefect['graceful_stop_timeout'] + graceful_stop_timeout: ..., + # praefect['virtual_storages']. The old value was a hash map but the new value is an array. + virtual_storage: [ + { + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]. The name was previously the key in + # the 'virtual_storages' hash. + name: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. The old value was a hash map + # but the new value is an array. + node: [ + { + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. Use NODE_NAME key as the + # storage. + storage: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['address']. + address: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['token']. + token: ..., + }, + ], + } + ] +} +``` ## Long-running user type data change @@ -196,7 +653,7 @@ migration might take multiple days to complete on larger GitLab instances. Make has completed successfully before upgrading to 16.1.0 or later. GitLab 16.1 introduces the `FinalizeUserTypeMigration` migration which ensures the -16.0 `MigrateHumanUserType` background migration is completed, making the 16.0 changes synchronously +16.0 `MigrateHumanUserType` background migration is completed, executing the 16.0 change synchronously during the upgrade if it's not completed. GitLab 16.2 [implements a `NOT NULL` database constraint](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122454) @@ -219,3 +676,127 @@ an unusable state, generating `500` errors. The errors are caused by Sidekiq and application code that is incompatible with the database schema. At the end of the workaround process, Sidekiq and Puma are restarted to resolve that issue. + +## Undefined column error upgrading to 16.2 or later + +A bug in GitLab 15.11 incorrectly disabled a database change on self-managed instances. +For more information, see [issue 408835](https://gitlab.com/gitlab-org/gitlab/-/issues/408835). + +If your GitLab instance upgraded first to 15.11.0, 15.11.1, or 15.11.2 the database schema is +incorrect and upgrading to GitLab 16.2 or later fails with an error. A database change +requires the earlier modification to be in place: + +```plaintext +PG::UndefinedColumn: ERROR: column "id_convert_to_bigint" of relation "ci_build_needs" does not exist +LINE 1: ...db_config_name:main*/ UPDATE "ci_build_needs" SET "id_conver... +``` + +GitLab 15.11.3 shipped a fix for this bug, but it doesn't correct the problem on +instances already running the earlier 15.11 releases. + +If you're not sure if an instance is affected, check for the column on the +[database console](../../administration/troubleshooting/postgresql.md#start-a-database-console): + +```sql +select pg_typeof (id_convert_to_bigint) from public.ci_build_needs limit 1; +``` + +If you need the workaround, this query fails: + +```plaintext +ERROR: column "id_convert_to_bigintd" does not exist +LINE 1: select pg_typeof (id_convert_to_bigintd) from public.ci_buil... +``` + +Unaffected instances return: + +```plaintext + pg_typeof +----------- + bigint +``` + +The workaround for this issue differs if your GitLab instance's database schema +was recently created: + +| Installation version | Workaround | +| -------------------- | ---------- | +| 15.9 or earlier | [15.9](#workaround-instance-created-with-159-or-earlier) | +| 15.10 | [15.10](#workaround-instance-created-with-1510) | +| 15.11 | [15.11](#workaround-instance-created-with-1511) | + +Most instances should use the 15.9 procedure. Only very new instances require the +the 15.10 or 15.11 procedures. If you've migrated GitLab using backup and restore, +the database schema comes from the original instance. Select the workaround based +on the source instance. + +The commands in the following sections are for Linux package installations, and +differ for other installation types: + +::Tabs + +:::TabTitle Docker + +- Omit `sudo` +- Shell into the GitLab container and run the same commands: + + ```shell + docker exec -it <container-id> bash + ``` + +:::TabTitle Self-compiled (source) + +- Use `sudo -u git -H bundle exec rake` instead of `sudo gitlab-rake` +- Run the SQL on [your PostgreSQL database console](../../administration/troubleshooting/postgresql.md#start-a-database-console) + +:::TabTitle Helm chart (Kubernetes) + +- Omit `sudo`. +- Shell into the `toolbox` pod to run the Rake commands: `gitlab-rake` is in `/usr/local/bin` if not in the `PATH`. + - Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. +- Run the SQL on [your PostgreSQL database console](../../administration/troubleshooting/postgresql.md#start-a-database-console) + +::EndTabs + +### Workaround: instance created with 15.9 or earlier + +```shell +# Restore schema +sudo gitlab-psql -c "DELETE FROM schema_migrations WHERE version IN ('20230130175512', '20230130104819');" +sudo gitlab-rake db:migrate:up VERSION=20230130175512 +sudo gitlab-rake db:migrate:up VERSION=20230130104819 + +# Re-schedule background migrations +sudo gitlab-rake db:migrate:down VERSION=20230130202201 +sudo gitlab-rake db:migrate:down VERSION=20230130110855 +sudo gitlab-rake db:migrate:up VERSION=20230130202201 +sudo gitlab-rake db:migrate:up VERSION=20230130110855 +``` + +### Workaround: instance created with 15.10 + +```shell +# Restore schema for sent_notifications +sudo gitlab-psql -c "DELETE FROM schema_migrations WHERE version = '20230130175512';" +sudo gitlab-rake db:migrate:up VERSION=20230130175512 + +# Re-schedule background migration for sent_notifications +sudo gitlab-rake db:migrate:down VERSION=20230130202201 +sudo gitlab-rake db:migrate:up VERSION=20230130202201 + +# Restore schema for ci_build_needs +sudo gitlab-rake db:migrate:down VERSION=20230321163547 +sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230321163547');" +``` + +### Workaround: instance created with 15.11 + +```shell +# Restore schema for sent_notifications +sudo gitlab-rake db:migrate:down VERSION=20230411153310 +sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230411153310');" + +# Restore schema for ci_build_needs +sudo gitlab-rake db:migrate:down VERSION=20230321163547 +sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230321163547');" +``` diff --git a/doc/user/ai_features.md b/doc/user/ai_features.md index 9558e40d56f..feea06666dc 100644 --- a/doc/user/ai_features.md +++ b/doc/user/ai_features.md @@ -5,62 +5,49 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, reference --- -# AI/ML powered features +# GitLab Duo GitLab is creating AI-assisted features across our DevSecOps platform. These features aim to help increase velocity and solve key pain points across the software development lifecycle. -## Enable AI/ML features - -> Introduced in GitLab 16.0 and [actively being rolled out](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118222). - -Prerequisites: - -- You must have the Owner role for the group. - -To enable AI/ML features for a top-level group: - -- Enable [Experiment features](group/manage.md#enable-experiment-features). -- Enable [third-party AI features](group/manage.md#enable-third-party-ai-features) (enabled by default). - To disable AI features powered by third-party APIs, clear this setting. +| Feature | Purpose | Large Language Model | Current availability | Maturity | +|-|-|-|-|-| +| [Suggested Reviewers](project/merge_requests/reviews/index.md#gitlab-duo-suggested-reviewers) | Assists in creating faster and higher-quality reviews by automatically suggesting reviewers for your merge request. | GitLab creates a machine learning model for each project, which is used to generate reviewers <br><br> [View the issue](https://gitlab.com/gitlab-org/modelops/applied-ml/applied-ml-updates/-/issues/10) | SaaS only | [Generally Available (GA)](../policy/experiment-beta-support.md#generally-available-ga) | +| [Code Suggestions](project/repository/code_suggestions/index.md) | Helps you write code more efficiently by viewing code suggestions as you type. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS <br> Self-managed | [Beta](../policy/experiment-beta-support.md#beta) | +| [Vulnerability summary](application_security/vulnerabilities/index.md#explaining-a-vulnerability) | Helps you remediate vulnerabilities more efficiently, uplevel your skills, and write more secure code. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) <br><br> Anthropic's claude model if degraded performance | SaaS only <br><br> Ultimate tier | [Beta](../policy/experiment-beta-support.md#beta) | +| [Code explanation](#explain-code-in-the-web-ui-with-code-explanation) | Helps you understand code by explaining it in English language. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only <br><br> Ultimate tier | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Chat](#answer-questions-with-chat) | Process and generate text and code in a conversational manner. Helps you quickly identify useful information in large volumes of text in issues, epics, code, and GitLab documentation. | Anthropic's claude model <br><br> OpenAI Embeddings | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Value stream forecasting](#forecast-deployment-frequency-with-value-stream-forecasting) | Assists you with predicting productivity metrics and identifying anomalies across your software development lifecycle. | Statistical forecasting | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Discussion summary](#summarize-issue-discussions-with-discussion-summary) | Assists with quickly getting everyone up to speed on lengthy conversations to help ensure you are all on the same page. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Merge request summary](project/merge_requests/ai_in_merge_requests.md#summarize-merge-request-changes) | Efficiently communicate the impact of your merge request changes. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Code review summary](project/merge_requests/ai_in_merge_requests.md#summarize-my-merge-request-review) | Helps ease merge request handoff between authors and reviewers and help reviewers efficiently understand suggestions. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Merge request template population](project/merge_requests/ai_in_merge_requests.md#fill-in-merge-request-templates) | Generate a description for the merge request based on the contents of the template. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Test generation](project/merge_requests/ai_in_merge_requests.md#generate-suggested-tests-in-merge-requests) | Automates repetitive tasks and helps catch bugs early. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Git suggestions](https://gitlab.com/gitlab-org/gitlab/-/issues/409636) | Helps you discover or recall Git commands when and where you need them. | OpenAI | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| **Root cause analysis** | Assists you in determining the root cause for a pipeline failure and failed CI/CD build. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | +| [Issue description generation](#summarize-an-issue-with-issue-description-generation) | Generate issue descriptions. | [Google Vertex Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview) | SaaS only | [Experiment](../policy/experiment-beta-support.md#experiment) | -These settings work together so you can have a mix of both experimental and third-party AI features. - -## Generally Available AI features - -When a feature is [Generally Available](../policy/experiment-beta-support.md#generally-available-ga), -it does not require [Experiment features to be enabled](group/manage.md#enable-experiment-features). -Some of these features might require [third-party AI features to be enabled](group/manage.md#enable-third-party-ai-features). - -The following feature is Generally Available: - -- [Suggested Reviewers](project/merge_requests/reviews/index.md#suggested-reviewers) +## Enable AI/ML features -## Beta AI features +The [Generally Available](../policy/experiment-beta-support.md#generally-available-ga) features listed in the previous table do not need to be enabled. -[Beta features](../policy/experiment-beta-support.md#beta) do not require -[Experiment features to be enabled](group/manage.md#enable-experiment-features). +[Experiment features](../policy/experiment-beta-support.md#experiment) and [Beta features](../policy/experiment-beta-support.md#beta) (besides Code Suggestions) on SaaS must be enabled by a user who has the Owner role in the group. Their usage is subject to the [Testing Terms of Use](https://about.gitlab.com/handbook/legal/testing-agreement/). -The following features are in Beta: +In addition, all features built on large language models (LLM) from Google, Anthropic or OpenAI require that [third-party AI features are enabled](group/manage.md#enable-third-party-ai-features) (which they are by default). The table above shows which features are built on which LLM. To disable AI features powered by third-party APIs, clear this setting. -- [Code Suggestions](project/repository/code_suggestions.md) -- [Explain this vulnerability](application_security/vulnerabilities/index.md#explaining-a-vulnerability-beta) +Code Suggestions currently has its own settings: -## Experiment AI features +- View [how to enable for self-managed](project/repository/code_suggestions/saas.md#enable-code-suggestions). +- View [how to enable for SaaS](project/repository/code_suggestions/self_managed.md#enable-code-suggestions-on-self-managed-gitlab). -[Experiment](../policy/experiment-beta-support.md#experiment) AI features require -[Experiment features to be enabled](group/manage.md#enable-experiment-features) as well as [third-party AI services to be enabled](group/manage.md#enable-third-party-ai-features). +The use of Code Suggestions is also subject to the [Testing Terms of Use](https://about.gitlab.com/handbook/legal/testing-agreement/). -The following features are in Experiment: + -- [Fill in merge request templates](project/merge_requests/ai_in_merge_requests.md#fill-in-merge-request-templates) -- [Summarize merge request changes](project/merge_requests/ai_in_merge_requests.md#summarize-merge-request-changes) -- [Summarize my merge request review](project/merge_requests/ai_in_merge_requests.md#summarize-my-merge-request-review) -- [Suggested merge or squash commit message](project/merge_requests/ai_in_merge_requests.md#suggested-merge-or-squash-commit-message) -- [Generate suggested tests in merge requests](project/merge_requests/ai_in_merge_requests.md#generate-suggested-tests-in-merge-requests) +## Experimental AI features and how to use them -The rest of the features described on this page are also in the Experiment phase. +The following subsections describe the experimental AI features in more detail. -### Explain Selected Code in the Web UI **(ULTIMATE SAAS)** +### Explain code in the Web UI with Code explanation **(ULTIMATE SAAS EXPERIMENT)** > Introduced in GitLab 15.11 as an [Experiment](../policy/experiment-beta-support.md#experiment) on GitLab.com. @@ -75,7 +62,7 @@ By using a large language model, GitLab can explain the code in natural language Prerequisites: -Additional prerequisites [beyond the two above](#experiment-ai-features). +Additional prerequisites in addition to [the settings listed previously](#enable-aiml-features). - The project must be on GitLab.com. - You must have the GitLab Ultimate subscription tier. @@ -83,7 +70,7 @@ Additional prerequisites [beyond the two above](#experiment-ai-features). To explain your code: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select any file in your project that contains code. 1. On the file, select the lines that you want to have explained. 1. On the left side, select the question mark (**{question}**). You might have to scroll to the first line of your selection to view it. This sends the selected code, together with a prompt, to provide an explanation to the large language model. @@ -93,7 +80,7 @@ To explain your code: You can also have code explained in the context of a merge request. To explain code in a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Code > Merge requests**, then select your merge request. 1. On the secondary menu, select **Changes**. 1. On the file you would like explained, select the three dots (**{ellipsis_v}**) and select **View File @ $SHA**. @@ -109,7 +96,7 @@ code in a merge request: We cannot guarantee that the large language model produces results that are correct. Use the explanation with caution. -### GitLab Duo Chat **(ULTIMATE SAAS)** +### Answer questions with Chat **(ULTIMATE SAAS EXPERIMENT)** > Introduced in GitLab 16.0 as an [Experiment](../policy/experiment-beta-support.md#experiment). @@ -154,7 +141,7 @@ Or, you can add a comment in the [feedback issue](https://gitlab.com/gitlab-org/ NOTE: Only the last 50 messages are retained in the chat history. The chat history expires 3 days after last use. -### Summarize issue discussions **(ULTIMATE SAAS)** +### Summarize issue discussions with Discussion summary **(ULTIMATE SAAS EXPERIMENT)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10344) in GitLab 16.0 as an [Experiment](../policy/experiment-beta-support.md#experiment). @@ -174,7 +161,7 @@ Provide feedback on this experimental feature in [issue 407779](https://gitlab.c **Data usage**: When you use this feature, the text of public comments on the issue are sent to the large language model referenced above. -### Show deployment frequency forecast **(ULTIMATE SAAS)** +### Forecast deployment frequency with Value stream forecasting **(ULTIMATE ALL EXPERIMENT)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10228) in GitLab 16.2 as an [Experiment](../policy/experiment-beta-support.md#experiment). @@ -182,7 +169,7 @@ This feature is an [Experiment](../policy/experiment-beta-support.md) on GitLab. In CI/CD Analytics, you can view a forecast of deployment frequency: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. 1. Select the **Deployment frequency** tab. 1. Turn on the **Show forecast** toggle. @@ -191,9 +178,11 @@ In CI/CD Analytics, you can view a forecast of deployment frequency: The forecast is displayed as a dotted line on the chart. Data is forecasted for a duration that is half of the selected date range. For example, if you select a 30-day range, a forecast for the following 15 days is displayed. + + Provide feedback on this experimental feature in [issue 416833](https://gitlab.com/gitlab-org/gitlab/-/issues/416833). -### Generate issue descriptions **(ULTIMATE SAAS)** +### Summarize an issue with Issue description generation **(ULTIMATE SAAS EXPERIMENT)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10762) in GitLab 16.3 as an [Experiment](../policy/experiment-beta-support.md#experiment). diff --git a/doc/user/analytics/analytics_dashboards.md b/doc/user/analytics/analytics_dashboards.md index 49870e8c66c..68b9fef5cc7 100644 --- a/doc/user/analytics/analytics_dashboards.md +++ b/doc/user/analytics/analytics_dashboards.md @@ -4,7 +4,7 @@ group: Product Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Analytics dashboards (Experiment) **(ULTIMATE ALL)** +# Analytics dashboards **(ULTIMATE ALL EXPERIMENT)** > Introduced in GitLab 15.9 as an [Experiment](../../policy/experiment-beta-support.md#experiment) feature [with a flag](../../administration/feature_flags.md) named `combined_analytics_dashboards`. Disabled by default. @@ -29,15 +29,16 @@ The following data sources are configured for analytics dashboards: ## Built-in dashboards To help you get started with analytics, GitLab provides built-in dashboards with predefined visualizations. +These dashboards are labeled **By GitLab**, and you cannot edit them. +Instead, you can create a custom dashboard with a similar style. ### Product analytics +When [product analytics](../product_analytics/index.md) is enabled and onboarded, two built-in dashboard are added: + - **Audience** displays metrics related to traffic, such as the number of users and sessions. - **Behavior** displays metrics related to user activity, such as the number of page views and events. -These dashboards are labeled **By GitLab**, and you cannot edit them. -Instead, you can create a custom dashboard with a similar style. - ### Value Stream Management - **Value Streams Dashboard** displays metrics related to [DevOps performance, security exposure, and workstream optimization](../analytics/value_streams_dashboard.md#devsecops-metrics-comparison-panel). @@ -64,9 +65,6 @@ On self-managed GitLab, by default this feature is not available. To make it ava On GitLab.com, this feature is not available. This feature is not ready for production use. -NOTE: -This feature does not work in conjunction with the `product_analytics_snowplow_support` feature flag. - You can use the dashboard designer to: - Create custom dashboards. @@ -82,7 +80,7 @@ Prerequisite: To view a list of dashboards (both built-in and custom) for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Analytics dashboards**. 1. From the list of available dashboards, select the dashboard you want to view. @@ -97,9 +95,9 @@ This feature will be connected to group-level dashboards as part of [issue #4115 To change the location of a group's dashboards: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project you want to store your dashboard files in. +1. On the left sidebar, select **Search or go to** and find the project you want to store your dashboard files in. The project must belong to the group for which you create the dashboards. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Analytics**. 1. In the **Analytics Dashboards** section, select your dashboard files project. @@ -116,10 +114,10 @@ You can share dashboards only between projects that are located in the same grou To change the location of project dashboards: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project, +1. On the left sidebar, select **Search or go to** and find your project, or select **Create new...** (**{plus}**) and **New project/repository** to create the project to store your dashboard files. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) and find the analytics project. +1. On the left sidebar, select **Search or go to** and find the analytics project. 1. Select **Settings > General**. 1. Expand **Analytics**. 1. In the **Analytics Dashboards** section, select your dashboard files project. @@ -180,7 +178,7 @@ create a `line_chart.yaml` file with the following required fields: To create a custom dashboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Analytics dashboards**. 1. Select **New dashboard**. 1. In the **New dashboard** input, enter the name of the dashboard. @@ -194,7 +192,7 @@ You can edit your custom dashboard's title and add or resize visualizations in t To edit an existing custom dashboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Analytics dashboards**. 1. From the list of available dashboards, select a custom dashboard (one without the `By GitLab` label) you want to edit. 1. Select **Edit**. @@ -209,9 +207,13 @@ To edit an existing custom dashboard: If the dashboard displays a global error message that data could not be loaded, first try reloading the page. If the error persists: -- Check that your configurations match the [JSON schema](#define-a-dashboard) defined in `ee/app/validators/json_schemas/analytics_dashboard.json`. +- Check that your configurations match the [dashboard JSON schema](#define-a-dashboard) defined in `ee/app/validators/json_schemas/analytics_dashboard.json`. - For product analytics, check your [admin and project settings](../product_analytics/index.md#project-level-settings), and make sure they are set up correctly. +### `Invalid visualization configuration` + +If a dashboard panel displays a message that the visualization configuration is invalid, check that your visualization configurations match the [visualization JSON schema](#define-a-chart-visualization) defined in `ee/app/validators/json_schemas/analytics_visualization.json`. + ### Dashboard panel error If a dashboard panel displays an error message: diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 8dc69b2f664..61bc77e4469 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -32,7 +32,7 @@ View pipeline duration history: To view CI/CD analytics: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. ## View DORA deployment frequency chart **(ULTIMATE ALL)** @@ -50,7 +50,7 @@ The deployment frequency chart is available for groups and projects. To view the deployment frequency chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. 1. Select the **Deployment frequency** tab. @@ -72,7 +72,7 @@ merge requests to be deployed to a production environment. This chart is availab To view the lead time for changes chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. 1. Select the **Lead time** tab. @@ -88,7 +88,7 @@ Time to restore service is one of the four DORA metrics that DevOps teams use fo To view the time to restore service chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. 1. Select the **Time to restore service** tab. @@ -104,6 +104,6 @@ Change failure rate is one of the four DORA metrics that DevOps teams use for me To view the change failure rate chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > CI/CD analytics**. 1. Select the **Change failure rate** tab. diff --git a/doc/user/analytics/code_review_analytics.md b/doc/user/analytics/code_review_analytics.md index 27d3e45803e..e5b475d0e45 100644 --- a/doc/user/analytics/code_review_analytics.md +++ b/doc/user/analytics/code_review_analytics.md @@ -34,7 +34,7 @@ Prerequisite: To view code review analytics: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Code review analytics**. 1. Optional. Filter results: 1. Select the filter bar. diff --git a/doc/user/analytics/contributor_statistics.md b/doc/user/analytics/contributor_statistics.md index 514f1ca42ab..ee10522e80b 100644 --- a/doc/user/analytics/contributor_statistics.md +++ b/doc/user/analytics/contributor_statistics.md @@ -15,7 +15,7 @@ and line charts with the number of commits by each project member. To view contributor statistics for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Contributor statistics**. 1. From the **Branches** (**main**) dropdown list, select the branch you want to view commits for. 1. To view the number of commits made on a specific day, hover over the line chart. @@ -28,7 +28,7 @@ To view contributor statistics for a project: To view a list of commits made by project members per day: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Contributor statistics**. 1. Select **History**. 1. From the **Branches** (**main**) dropdown list, select the branch you want to view commits for. @@ -42,7 +42,7 @@ To view a list of commits made by project members per day: To view the list of commits to the project as an RSS feed in Atom format: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Contributor statistics**. 1. Select **History**. 1. In the upper-right corner, select the feed symbol (**{rss}**). diff --git a/doc/user/analytics/dora_metrics.md b/doc/user/analytics/dora_metrics.md index 0efe4a53427..391a1c7965f 100644 --- a/doc/user/analytics/dora_metrics.md +++ b/doc/user/analytics/dora_metrics.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # DevOps Research and Assessment (DORA) metrics **(ULTIMATE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. -> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. +> - Lead time for changes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) in GitLab 13.10. The [DevOps Research and Assessment (DORA)](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) team has identified four metrics that measure DevOps performance. @@ -33,7 +33,7 @@ This enables teams and managers to understand all aspects of productivity, quali ## Deployment frequency -> [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/394712) the frequency calculation formula for the `all` and `monthly` intervals in GitLab 16.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/394712) fix for the frequency calculation formula for `all` and `monthly` intervals in GitLab 16.0. Deployment frequency is the frequency of successful deployments to production over the given date range (hourly, daily, weekly, monthly, or yearly). @@ -144,6 +144,24 @@ The table below provides an overview of the DORA metrics' data aggregation in di | Time to restore service | Number of seconds an incident was open for | daily median per month | daily median | `day` (default) or `month` | | Change failure rate | percentage of deployments that cause an incident in production | daily median per month | percentage of failed deployments | `day` (default) or `month` | +## Configure DORA metrics calculation **(ULTIMATE ALL BETA)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96561) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `dora_configuration`. Disabled by default. This feature is in [Beta](../../policy/experiment-beta-support.md). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `dora_configuration`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + +You can configure the behavior of DORA metrics calculations. +To do this, in the Rails console run the following command: + +```ruby +Dora::Configuration.create!(project: my_project, ltfc_target_branches: \['master', 'main'\]) +``` + +This feature is in [Beta](../../policy/experiment-beta-support.md). + ## Retrieve DORA metrics data To retrieve DORA data, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. diff --git a/doc/user/analytics/img/dora_performers_score_panel_v16_3.png b/doc/user/analytics/img/dora_performers_score_panel_v16_3.png Binary files differdeleted file mode 100644 index 9f59667f9e9..00000000000 --- a/doc/user/analytics/img/dora_performers_score_panel_v16_3.png +++ /dev/null diff --git a/doc/user/analytics/img/issues_closed_analytics_v16_4.png b/doc/user/analytics/img/issues_closed_analytics_v16_4.png Binary files differnew file mode 100644 index 00000000000..5e1fe4eaa8c --- /dev/null +++ b/doc/user/analytics/img/issues_closed_analytics_v16_4.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index abeda983dad..f096d1e2882 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -6,84 +6,63 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Analyze GitLab usage **(FREE ALL)** -## Instance-level analytics +GitLab provides different types of analytics insights at the instance, group, and project level. +These insights appear on the left sidebar, under [**Analyze**](../project/settings/index.md#disable-project-analytics). -Instance-level analytics make it possible to aggregate analytics across -GitLab, so that users can view information across multiple projects and groups -in one place. +## Instance-level analytics -For more information, see [instance-level analytics](../admin_area/analytics/index.md). +Use [instance-level analytics](../../administration/analytics/index.md) to aggregate analytics across GitLab, +so that you can view information across multiple projects and groups in one place. ## Group-level analytics > Moved to GitLab Premium in 13.9. -GitLab provides several analytics features at the group level. Some of these features require you to use a higher tier than GitLab Free. +Use group-level analytics to get insights into your groups': -- [Application Security](../application_security/security_dashboard/index.md) -- [Contribution](../group/contribution_analytics/index.md) -- [DevOps Adoption](../group/devops_adoption/index.md) +- [Security Dashboards](../application_security/security_dashboard/index.md) +- [Contribution analytics](../group/contribution_analytics/index.md) +- [DevOps adoption](../group/devops_adoption/index.md) - [Insights](../group/insights/index.md) -- [Issue](../group/issues_analytics/index.md) -- [Productivity](productivity_analytics.md) -- [Repositories](../group/repositories_analytics/index.md) -- [Value Stream Management Analytics](../group/value_stream_analytics/index.md), and [Value Stream Management Dashboard](value_streams_dashboard.md) +- [Issue analytics](../group/issues_analytics/index.md) +- [Productivity analytics](productivity_analytics.md) +- [Repositories analytics](../group/repositories_analytics/index.md) +- [Value Stream Management Analytics](../group/value_stream_analytics/index.md) and [Value Stream Management Dashboard](value_streams_dashboard.md) ## Project-level analytics -You can use GitLab to review analytics at the project level. Some of these features require you to use a higher tier than GitLab Free. +Use project-level analytics to get insights into your projects': - [Analytics dashboards](analytics_dashboards.md), enabled with the `combined_analytics_dashboards_editor` [feature flag](../../development/feature_flags/index.md#enabling-a-feature-flag-locally-in-development) -- [Application Security](../application_security/security_dashboard/index.md) -- [CI/CD & DORA](ci_cd_analytics.md) -- [Code Review](code_review_analytics.md) +- [Security Dashboards](../application_security/security_dashboard/index.md) +- [CI/CD analytics and DORA metrics](ci_cd_analytics.md) +- [Code review analytics](code_review_analytics.md) - [Contributor statistics](../../user/analytics/contributor_statistics.md) - [Insights](../project/insights/index.md) -- [Issue](../../user/analytics/issue_analytics.md) -- [Merge Request](merge_request_analytics.md), enabled with the `project_merge_request_analytics` +- [Issue analytics](../../user/analytics/issue_analytics.md) +- [Merge request analytics](merge_request_analytics.md), enabled with the `project_merge_request_analytics` [feature flag](../../development/feature_flags/index.md#enabling-a-feature-flag-locally-in-development) -- [Repository](repository_analytics.md) -- [Value Stream Management Analytics](../group/value_stream_analytics/index.md), and [Value Stream Management Dashboard](value_streams_dashboard.md) - -### Remove project analytics from the left sidebar - -By default, analytics for a project are displayed under the **Analyze** item in the left sidebar. To remove this item: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Turn off the **Analytics** toggle. -1. Select **Save changes**. +- [Repository analytics](repository_analytics.md) +- [Value Stream Management Analytics](../group/value_stream_analytics/index.md) and [Value Stream Management Dashboard](value_streams_dashboard.md) ## User-configurable analytics -The following analytics features are available for users to create personalized views: - -- [Application Security](../application_security/security_dashboard/index.md#security-center) - -Be sure to review the documentation page for this feature for GitLab tier requirements. +View vulnerabilities of your selected projects in the [Security Center](../application_security/security_dashboard/index.md#security-center). ## Value streams management -You can use the following analytics features to analyze and visualize the performance of your projects and groups: +Analyze and visualize the performance of your projects and groups with: - [Value stream analytics for projects and groups](../group/value_stream_analytics/index.md) - [Value streams dashboard](value_streams_dashboard.md) ## Glossary -We use the following terms to describe GitLab analytics: - -- **Mean Time to Change (MTTC):** The average duration between idea and delivery. GitLab measures -MTTC from issue creation to the issue's latest related merge request's deployment to production. -- **Mean Time to Detect (MTTD):** The average duration that a bug goes undetected in production. -GitLab measures MTTD from deployment of bug to issue creation. -- **Mean Time To Merge (MTTM):** The average lifespan of a merge request. GitLab measures MTTM from -merge request creation to merge request merge (and closed/un-merged merge requests are excluded). -For more information, see [Merge Request Analytics](merge_request_analytics.md). -- **Mean Time to Recover/Repair/Resolution/Resolve/Restore (MTTR):** The average duration that a bug -is not fixed in production. GitLab measures MTTR from deployment of bug to deployment of fix. -- **Velocity:** The total issue burden completed in some period of time. The burden is usually measured -in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab -measures velocity as the total points or weight of issues closed in a given period of time. +| Metric | Definition | Measurement in GitLab | +| ------ | ---------- | --------------------- | +| Mean Time to Change (MTTC) | The average duration between idea and delivery. | From issue creation to the issue's latest related merge request's deployment to production. | +| Mean Time to Detect (MTTD) | The average duration that a bug goes undetected in production. | From deployment of bug to issue creation. | +| Mean Time To Merge (MTTM) | The average lifespan of a merge request. | From merge request creation to merge request merge (excluding closed and unmerged merge requests). For more information, see [Merge Request Analytics](merge_request_analytics.md). | +| Mean Time to Recover/Repair/Resolution/Resolve/Restore (MTTR) | The average duration that a bug is not fixed in production. | From deployment of bug to deployment of fix. | +| Velocity | The total issue burden completed in some period of time. The burden is usually measured in points or weight, often per sprint. | Total points or weight of issues closed in a given period of time. Expressed as, for example, "30 points per sprint". | diff --git a/doc/user/analytics/issue_analytics.md b/doc/user/analytics/issue_analytics.md index 7caee947318..8f29c008d75 100644 --- a/doc/user/analytics/issue_analytics.md +++ b/doc/user/analytics/issue_analytics.md @@ -15,9 +15,11 @@ prior. To access the chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Issue analytics**. +You can also access the chart from the [Value Streams Dashboard](value_streams_dashboard.md#dashboard-metrics-and-drill-down-reports) through the **New issues** drill-down report. + Hover over each bar to see the total number of issues. To narrow the scope of issues included in the graph, enter your criteria in the @@ -36,6 +38,20 @@ shows a total of 15 months for the chart in the GitLab.org group.  +## Enhanced issue analytics **(ULTIMATE ALL)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233905/) in GitLab 16.4 [with a flag](../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can +[enable the feature flag](../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. On GitLab.com, this feature is not +available. This feature is not ready for production use. + +Enhanced issue analytics display the additional metric "Issues closed", which represents the total number of resolved issues in your project over a selected period. +You can use this metric to improve the overall turn-around time and value delivered to your customers. + + + ## Drill into the information > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196547) in GitLab 13.1. diff --git a/doc/user/analytics/merge_request_analytics.md b/doc/user/analytics/merge_request_analytics.md index 998f56ac40a..0f8eb9ac211 100644 --- a/doc/user/analytics/merge_request_analytics.md +++ b/doc/user/analytics/merge_request_analytics.md @@ -29,7 +29,7 @@ Prerequisite: To view merge request analytics: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Merge request analytics**. ## View the number of merge requests in a date range @@ -39,7 +39,7 @@ To view merge request analytics: To view the number of merge requests merged during a specific date range: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Merge request analytics**. 1. Optional. Filter results: 1. Select the filter bar. @@ -73,6 +73,6 @@ created and when it's merged. Closed and not yet merged merge requests are not i To view **Mean time to merge**: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Merge request analytics**. The **Mean time to merge** number is displayed on the dashboard. diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index ea896f07204..a17eb08c53c 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -26,7 +26,7 @@ Prerequisite: - You must have at least the Reporter role for the group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Productivity analytics**. 1. Optional. Filter results: 1. Select a project from the dropdown list. @@ -44,7 +44,7 @@ Use the following charts in productivity analytics to view the velocity of your merge requests to merge after they were created. - **Trendline**: number of merge requests that were merged in a specific time period. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Productivity analytics**. To filter time metrics: @@ -56,7 +56,7 @@ To filter time metrics: To view commit statistics for your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Productivity analytics**. 1. Under the **Trendline** scatterplot, view the commit statistics: - The left histogram shows the number of hours between commits, comments, and merges. diff --git a/doc/user/analytics/repository_analytics.md b/doc/user/analytics/repository_analytics.md index 46fa36658c4..26bce8a20c2 100644 --- a/doc/user/analytics/repository_analytics.md +++ b/doc/user/analytics/repository_analytics.md @@ -20,7 +20,7 @@ Repository analytics is part of [GitLab Community Edition](https://gitlab.com/gi Repository analytics requires: - An initialized Git repository. -- At least one commit in the default branch (`master` by default). +- At least one commit in the default branch (`main` by default). NOTE: Without a Git commit in the default branch, the menu item isn't visible. @@ -30,7 +30,7 @@ Commits in a project's [wiki](../project/wiki/index.md#track-wiki-events) are no To review repository analytics for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Repository analytics**. ## How repository analytics chart data is updated diff --git a/doc/user/analytics/value_streams_dashboard.md b/doc/user/analytics/value_streams_dashboard.md index 2c6626ad315..ed637dd886f 100644 --- a/doc/user/analytics/value_streams_dashboard.md +++ b/doc/user/analytics/value_streams_dashboard.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Released in GitLab 15.11 as an Open [Beta](../../policy/experiment-beta-support.md#beta) feature [with a flag](../../administration/feature_flags.md) named `group_analytics_dashboards_page`. Enabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/392734) in GitLab 16.0. Feature flag `group_analytics_dashboards_page` removed. -You can leave feedback on dashboard bugs or functionality in [issue 419488](https://gitlab.com/gitlab-org/gitlab/-/issues/419488). +To help us improve the Value Streams Dashboard, please share feedback about your experience in this [survey](https://gitlab.fra1.qualtrics.com/jfe/form/SV_50guMGNU2HhLeT4). For more information, see also the [Value Stream Management category direction page](https://about.gitlab.com/direction/plan/value_stream_management/). The Value Streams Dashboard is a customizable dashboard you can use to identify trends, patterns, and opportunities for digital transformation improvements. @@ -71,22 +71,46 @@ For example, if a project has a high score for Deployment Frequency (Velocity), These scoring are based on Google's classifications in the [DORA 2022 Accelerate State of DevOps Report](https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out). +## Enable or disable overview background aggregation **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120610) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `modify_value_stream_dashboard_settings`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130704) in GitLab 16.4. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `modify_value_stream_dashboard_settings`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + +Prerequisite: + +- You must have administrator access to the instance. + +To enable or disable the overview count aggregation for the Value Streams Dashboard: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > General**. +1. Expand **Analytics**. +1. In **Value Streams Dashboard**, select or clear the **Enable overview background aggregation for Value Streams Dashboard** checkbox. + +To retrieve aggregated usage counts in the group, use the [GraphQL API](../../api/graphql/reference/index.md#groupvaluestreamdashboardusageoverview). + ## View the value streams dashboard Prerequisite: - You must have at least the Reporter role for the group. +- Overview background aggregation for Value Streams Dashboards must be enabled. To view the value streams dashboard: - From Analytics Dashboards: - 1. On the group left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the group left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Analytics Dashboards**. - From Value Stream Analytics: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. + 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. 1. Below the **Filter results** text box, in the **Lifecycle metrics** row, select **Value Streams Dashboard / DORA**. 1. Optional. To open the new page, append this path `/analytics/dashboards/value_streams_dashboard` to the group URL (for example, `https://gitlab.com/groups/gitlab-org/-/analytics/dashboards/value_streams_dashboard`). @@ -118,7 +142,7 @@ Prerequisite: - You must have at least the Maintainer role for the group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Analytics**. 1. Select the project where you would like to store your YAML configuration file. @@ -126,7 +150,7 @@ Prerequisite: After you have set up the project, set up the configuration file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. In the default branch, create the configuration file: `.gitlab/analytics/dashboards/value_streams/value_streams.yaml`. 1. In the `value_streams.yaml` configuration file, fill in the configuration options: diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index c365c7f6bab..1bac636ac3f 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -101,7 +101,7 @@ a YAML snippet that you can paste in your GitLab CI/CD configuration. To generate an API Fuzzing configuration snippet: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **API Fuzzing** row, select **Enable API Fuzzing**. 1. Complete the fields. For details see [Available CI/CD variables](#available-cicd-variables). diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 10e16a173d8..1e9163a4c26 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -1,5 +1,4 @@ --- -type: reference, howto stage: Secure group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments @@ -7,11 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Security configuration **(FREE ALL)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20711) in GitLab 12.6. -> - SAST configuration was [enabled](https://gitlab.com/groups/gitlab-org/-/epics/3659) in 13.3 and [improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in 13.4. -> - DAST Profiles feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40474) in 13.4. -> - A simplified version was made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/294076) in GitLab 13.10. -> - [Redesigned](https://gitlab.com/gitlab-org/gitlab/-/issues/326926) in 14.2. +> Security configuration page was [redesigned](https://gitlab.com/gitlab-org/gitlab/-/issues/326926) in GitLab 14.2. The **Security configuration** page lists the following for the security testing and compliance tools: @@ -40,7 +35,7 @@ all security features are configured by default. To view a project's security configuration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. Select **Configuration history** to see the `.gitlab-ci.yml` file's history. diff --git a/doc/user/application_security/continuous_vulnerability_scanning/index.md b/doc/user/application_security/continuous_vulnerability_scanning/index.md new file mode 100644 index 00000000000..4094a0add28 --- /dev/null +++ b/doc/user/application_security/continuous_vulnerability_scanning/index.md @@ -0,0 +1,59 @@ +--- +stage: Secure +group: Composition Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Continuous Vulnerability Scanning **(ULTIMATE EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371063) in GitLab 16.4 as an [Experiment](../../../policy/experiment-beta-support.md#experiment) with two [features flags](../../../administration/feature_flags.md) named `dependency_scanning_on_advisory_ingestion` and `package_metadata_advisory_sync`. Enabled by default. + +NOTE: +This feature is an [Experiment](../../../policy/experiment-beta-support.md#experiment) and subject to change without notice. +If you have any feedback, you can let us know in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/425072). + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flags](../../feature_flags.md) named `dependency_scanning_on_advisory_ingestion` and `package_metadata_advisory_sync`. +On GitLab.com, this feature is available. + +Continuous Vulnerability Scanning detects new vulnerabilities outside a pipeline. +Your projects are automatically scanned whenever advisories are added to the [`GitLab Advisory Database`](https://advisories.gitlab.com/). +Projects that depend on the affected components have new vulnerabilities automatically created. + +Continuous Vulnerability Scanning detects vulnerabilities in the latest CycloneDX SBOM reports for the default branch. +[Dependency Scanning](../dependency_scanning/index.md) is used to generate these reports. + +## Configuration + +To enable Continuous Vulnerability Scanning: + +- Enable the Continuous Vulnerability Scanning setting in the project's [security configuration](../configuration/index.md). +- Enable [Dependency Scanning](../dependency_scanning/index.md#configuration) and ensure that its prerequisites are met. + +On GitLab self-managed only, you can [choose package registry metadata to sync](../../../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync) in the Admin Area for the GitLab instance. + +### Requirements for offline environments + +For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, +some adjustments are required to successfully scan CycloneDX reports for vulnerabilities. +For more information, see the offline [quick start guide](../../../topics/offline/quick_start_guide.md#enabling-the-package-metadata-database). + +## Supported languages and package managers + +The supported files and versions are the ones supported by +[Dependency Scanning](../dependency_scanning/index.md#supported-languages-and-package-managers). + +Go pseudo versions are not supported. A project dependency that references a Go pseudo version is never considered as affected. This might result in false negatives. + +## Checking new vulnerabilities + +New vulnerabilities detected by Continuous Vulnerability Scanning are visible on the [Vulnerability Report](../vulnerability_report/index.md). +However, they are not listed on the [Dependency List](../dependency_list/index.md) or in the pipeline where the affected SBOM component was detected. + +After an advisory is added to the [`GitLab Advisory Database`](https://advisories.gitlab.com/), +it might take a few hours before the corresponding vulnerabilities are added to your projects. + +## Contributing to the vulnerability database + +To find a vulnerability, you can search the [`GitLab Advisory Database`](https://advisories.gitlab.com/). +You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 59bd2b1ffb3..599203dedcc 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -58,7 +58,7 @@ You can use the following fuzzing engines to test the specified languages. To confirm the status of coverage-guided fuzz testing: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Coverage Fuzzing** section the status is: - **Not configured** @@ -172,7 +172,7 @@ artifacts files you can download from the CI/CD pipeline. Also, a project member To view details of the corpus registry: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Coverage Fuzzing** section, select **Manage corpus**. @@ -200,7 +200,7 @@ provided by the `COVFUZZ_CORPUS_NAME` variable. The corpus is updated on every p To upload an existing corpus file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Coverage Fuzzing** section, select **Manage corpus**. 1. Select **New corpus**. diff --git a/doc/user/application_security/dast/authentication.md b/doc/user/application_security/dast/authentication.md index b13b41e4a37..aed4066bc52 100644 --- a/doc/user/application_security/dast/authentication.md +++ b/doc/user/application_security/dast/authentication.md @@ -412,9 +412,12 @@ Authentication failed because a home page should be displayed after login. Inste ### Configure the authentication report +WARNING: +The authentication report can contain sensitive information such as the credentials used to perform the login. + An authentication report can be saved as a CI/CD job artifact to assist with understanding the cause of an authentication failure. -The report contains steps during the login process, HTTP requests and responses, the Document Object Model (DOM) and screenshots. +The report contains steps performed during the login process, HTTP requests and responses, the Document Object Model (DOM) and screenshots.  diff --git a/doc/user/application_security/dast/checks/113.1.md b/doc/user/application_security/dast/checks/113.1.md new file mode 100644 index 00000000000..44c3be330f2 --- /dev/null +++ b/doc/user/application_security/dast/checks/113.1.md @@ -0,0 +1,27 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Improper Neutralization of CRLF Sequences in HTTP Headers + +## Description + +By inserting Carriage Return / Line Feed (CRLF) characters, malicious users could potentially inject arbitrary data into HTTP responses. By modifying HTTP responses, attackers could conduct cross-site scripting or cache poisoning attacks against other users of the system. + +## Remediation + +User input should never be used in constructing HTTP header responses without some form +of validation against newlines. This includes URLs supplied by the user for HTTP redirects. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 113.1 | false | 113 | Active | high | + +## Links + +- [OWASP](https://owasp.org/www-community/attacks/HTTP_Response_Splitting) +- [CWE](https://cwe.mitre.org/data/definitions/113.html) diff --git a/doc/user/application_security/dast/checks/16.7.md b/doc/user/application_security/dast/checks/16.7.md index edaace407ae..d407234d2c2 100644 --- a/doc/user/application_security/dast/checks/16.7.md +++ b/doc/user/application_security/dast/checks/16.7.md @@ -22,8 +22,8 @@ Only three directives are applicable for the `Strict-Transport-Security` header. 1. `includeSubDomains`: This optional, valueless directive signals that the policy applies to this host as well as any subdomains found under this host's domain. 1. `preload`: While not part of the specification, setting this optional value allows major browser organizations to add this site into the browser's preloaded set of HTTPS sites. This requires further action on behalf of the website operator to submit their domain to the browser's HSTS preload list. See [hstspreload.org](https://hstspreload.org/) for more information. -Invalid directives, or the `Strict-Transport-Security` header appearing more than once (if the -values are different) is considered invalid. +Note that invalid directives, or the `Strict-Transport-Security` header appearing more than once (if the values are +different) is considered invalid. Prior to adding to this security configuration to your website, it is recommended you review the hstspreload.org [Deployment Recommendations](https://hstspreload.org/#deployment-recommendations). diff --git a/doc/user/application_security/dast/checks/16.9.md b/doc/user/application_security/dast/checks/16.9.md index c63a620794e..b0ba502b578 100644 --- a/doc/user/application_security/dast/checks/16.9.md +++ b/doc/user/application_security/dast/checks/16.9.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description A `Content-Security-Policy-Report-Only` (CSPRO) was identified on the target site. CSP-Report-Only headers -aid in determining how to implement a `Content-Security-Policy` that does not disrupt use of the target +aid in determining how to implement a `Content-Security-Policy` that does not disrupt normal use of the target site. ## Remediation diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index 1b3ce45dc43..035f4a4b486 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -4,7 +4,7 @@ group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# DAST browser-based crawler vulnerability checks **(ULTIMATE ALL)** +# DAST browser-based crawler vulnerability checks **(ULTIMATE)** The [DAST browser-based crawler](../browser_based.md) provides a number of vulnerability checks that are used to scan for vulnerabilities in the site under test. @@ -167,4 +167,5 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | ID | Check | Severity | Type | |:---|:------|:---------|:-----| +| [113.1](113.1.md) | Improper Neutralization of CRLF Sequences in HTTP Headers | High | Active | | [22.1](22.1.md) | Improper limitation of a pathname to a restricted directory (Path traversal) | High | Active | diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 24aadd14dd1..4ec693593fb 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -81,8 +81,8 @@ analyzer-specific configuration instructions. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. -Detected vulnerabilities are shown in [Merge requests](../index.md#view-security-scan-information-in-merge-requests), the [Pipeline security tab](../index.md#view-security-scan-information-in-the-pipeline-security-tab), -and the [Vulnerability report](../index.md#view-security-scan-information-in-the-vulnerability-report). +Detected vulnerabilities appear in [merge requests](../index.md#merge-request), the [pipeline security tab](../index.md#pipeline-security-tab), +and the [vulnerability report](../index.md#vulnerability-report). 1. To see all vulnerabilities detected, either: - From your project, select **Security & Compliance**, then **Vulnerability report**. diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md index 7538bd38d9f..86af7d4c5da 100644 --- a/doc/user/application_security/dast/proxy-based.md +++ b/doc/user/application_security/dast/proxy-based.md @@ -89,7 +89,7 @@ In this method you manually edit the existing `.gitlab-ci.yml` file. Use this me To include the DAST template: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline editor**. 1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file. @@ -156,7 +156,7 @@ snippet is created that you paste into the `.gitlab-ci.yml` file. To configure DAST using the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or **Configure DAST**. @@ -467,6 +467,9 @@ The DAST job does not require the project's repository to be present when runnin > - Runner tags selection [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345430) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `on_demand_scans_runner_tags. Disabled by default. > - Runner tags selection [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111499) in GitLab 16.3. +WARNING: +On-demand scans are not available when GitLab is running in FIPS mode. + An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger the scan. You must either start it manually, or schedule it to run. For on-demand DAST scans, a [site profile](#site-profile) defines **what** is to be scanned, and a @@ -483,7 +486,7 @@ An on-demand scan can be run in active or passive mode: To view on-demand scans: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Secure > On-demand scans**. On-demand scans are grouped by their status. The scan library contains all available on-demand @@ -499,7 +502,7 @@ Prerequisites: To run an existing on-demand scan: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > On-demand scans**. 1. Select the **Scan library** tab. 1. In the scan's row, select **Run scan**. @@ -522,7 +525,7 @@ Create an on-demand scan to: To create an on-demand DAST scan: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Secure > On-demand scans**. 1. Select **New scan**. 1. Complete the **Scan name** and **Description** fields. @@ -550,7 +553,7 @@ The on-demand DAST scan runs as specified and the project's dashboard shows the To view details of an on-demand scan: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. @@ -559,7 +562,7 @@ To view details of an on-demand scan: To edit an on-demand scan: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. @@ -570,7 +573,7 @@ To edit an on-demand scan: To delete an on-demand scan: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > On-demand scans**. 1. Select the **Scan library** tab. 1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. @@ -632,7 +635,7 @@ equivalent in functionality, so use whichever is most suitable: To create a site profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select **New > Site profile**. @@ -654,7 +657,7 @@ When a validated site profile's file, header, or meta tag is edited, the site's To edit a site profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Site Profiles** tab. @@ -669,7 +672,7 @@ See [Scan execution policies](../policies/scan-execution-policies.md) for more i To delete a site profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Site Profiles** tab. @@ -687,7 +690,7 @@ Prerequisites: To validate a site profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Site Profiles** tab. @@ -724,7 +727,7 @@ page. To retry a site profile's failed validation: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Site Profiles** tab. @@ -738,7 +741,7 @@ have their validation status revoked. To revoke a site profile's validation status: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Beside the validated profile, select **Revoke validation**. @@ -809,7 +812,7 @@ A scanner profile contains: To create a scanner profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select **New > Scanner profile**. @@ -824,7 +827,7 @@ For more information, see [Scan execution policies](../policies/scan-execution-p To edit a scanner profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Scanner profiles** tab. @@ -840,7 +843,7 @@ page. For more information, see [Scan execution policies](../policies/scan-execu To delete a scanner profile: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. 1. Select the **Scanner profiles** tab. diff --git a/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png b/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png Binary files differdeleted file mode 100644 index 5b2bd985ce4..00000000000 --- a/doc/user/application_security/dependency_list/img/dependency_list_v13_11.png +++ /dev/null diff --git a/doc/user/application_security/dependency_list/img/dependency_list_v16_3.png b/doc/user/application_security/dependency_list/img/dependency_list_v16_3.png Binary files differindex f3a8f3d8d5d..7e4e80f9c00 100644 --- a/doc/user/application_security/dependency_list/img/dependency_list_v16_3.png +++ b/doc/user/application_security/dependency_list/img/dependency_list_v16_3.png diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md index a55f26f529d..d8726cbd456 100644 --- a/doc/user/application_security/dependency_list/index.md +++ b/doc/user/application_security/dependency_list/index.md @@ -9,17 +9,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - System dependencies [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6698) in GitLab 14.6. > - Group-level dependency list [introduced](https://gitlab.com/groups/gitlab-org/-/epics/8090) in GitLab 16.2 [with a flag](../../../administration/feature_flags.md) named `group_level_dependencies`. Disabled by default. +> - Group-level dependency list [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/411257) in GitLab 16.4. -Use the dependency list to review your project or group's dependencies and key -details about those dependencies, including their known vulnerabilities. It is a collection of dependencies in your project, including existing and new findings. +Use the dependency list to review your project or group's dependencies and key details about those +dependencies, including their known vulnerabilities. This list is a collection of dependencies in your +project, including existing and new findings. This information is sometimes referred to as a +Software Bill of Materials, SBOM, or BOM. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Project Dependency](https://www.youtube.com/watch?v=ckqkn9Tnbw4). -To see the dependency list, go to your project or group and select **Secure > Dependency list**. - -This information is sometimes referred to as a Software Bill of Materials, SBOM, or BOM. - ## Prerequisites To view your project's dependencies, ensure you meet the following requirements: @@ -34,21 +33,33 @@ To view your project's dependencies, ensure you meet the following requirements: You should not change the default behavior of allowing the [application security jobs](../../application_security/index.md#application-coverage) to fail. -## View a project's dependencies +## View project dependencies + +To view the dependencies of a project or all projects in a group: - +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. Select **Secure > Dependency list**. -GitLab displays dependencies with the following information: +Details of each dependency are listed, sorted by decreasing severity of vulnerabilities (if any). You can sort the list instead by component name or packager. | Field | Description | -|-----------|-------------| +|:----------|:-----------| | Component | The dependency's name and version. | | Packager | The packager used to install the dependency. | | Location | For system dependencies, this lists the image that was scanned. For application dependencies, this shows a link to the packager-specific lock file in your project that declared the dependency. It also shows the [dependency path](#dependency-paths) to a top-level dependency, if any, and if supported. | -| License | Links to dependency's software licenses. | +| License<sup>1</sup> | Links to dependency's software licenses. A warning badge that includes the number of vulnerabilities detected in the dependency. | +| Projects<sup>2</sup> | Links to the project with the dependency. If multiple projects have the same dependency, the total number of these projects is shown. To go to a project with this dependency, select the **Projects** number, then search for and select its name. The project search feature is supported only on groups that have up to 600 occurrences in their group hierarchy. | + +<html> +<small>Footnotes: + <ol> + <li>Project-level only.</li> + <li>Group-level only.</li> + </ol> +</small> +</html> -Displayed dependencies are initially sorted by the severity of their known vulnerabilities, if any. They -can also be sorted by name or by the packager that installed them. + ### Vulnerabilities @@ -60,7 +71,7 @@ select the vulnerability's description. The [vulnerability's details](../vulnera ### Dependency paths The dependency list shows the path between a dependency and a top-level dependency it's connected -to, if any. There are many possible paths connecting a transient dependency to top-level +to, if any. Multiple paths may connect a transient dependency to top-level dependencies, but the user interface shows only one of the shortest paths. NOTE: @@ -73,42 +84,20 @@ Dependency paths are supported for the following package managers: - [NuGet](https://www.nuget.org/) - [Yarn 1.x](https://classic.yarnpkg.com/lang/en/) - [sbt](https://www.scala-sbt.org) +- [Conan](https://conan.io) ### Licenses -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10536) in GitLab 12.3. - If the [Dependency Scanning](../../application_security/dependency_scanning/index.md) CI job is configured, -[discovered licenses](../../compliance/license_scanning_of_cyclonedx_files/index.md#enable-license-scanning) are displayed on this page. - -## View a group's dependencies - -FLAG: -On self-managed GitLab, and GitLab.com the feature is disabled by default. To show the feature, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `group_level_dependencies`. - - - -GitLab displays dependencies with the following information: - -| Field | Description | -|-----------|-------------| -| Component | The dependency's name and version. | -| Packager | The packager used to install the dependency. | -| Location | For operating system dependencies, this lists the image that was scanned. For application dependencies, this shows a link to the packager-specific lock file in your project that declared the dependency. It also shows the [dependency path](#dependency-paths) to a top-level dependency, if any, and if supported. If there are multiple locations, the total number of locations is displayed. | -| Projects | Links to the project related to the dependency. If there are multiple projects, the total number of projects is displayed. | - -Displayed dependencies are initially sorted by packager. They -can also be sorted by name. - -## Downloading the dependency list - -You can download the full list of dependencies and their details in -`JSON` format. +[discovered licenses](../../compliance/license_scanning_of_cyclonedx_files/index.md) are displayed on this page. -### In the UI +## Download the dependency list -You can download your group's or project's list of dependencies and their details in JSON format by selecting the **Export** button. The dependency list only shows the results of the last successful pipeline to run on the default branch. +You can download the full list of dependencies and their details in JSON format. The dependency +list shows only the results of the last successful pipeline that ran on the default branch. -### Using the API +To download the dependency list: -You can download your project's list of dependencies [using the API](../../../api/dependencies.md#list-project-dependencies). Note this only provides the dependencies identified by the [Gemnasium family of analyzers](../dependency_scanning/index.md#dependency-analyzers) and not any other of the GitLab dependency analyzers. +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. Select **Secure > Dependency list**. +1. Select **Export**. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 5a2394113cb..f8bd5b99cb6 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -18,6 +18,8 @@ Dependency Scanning can run in the development phase of your application's life pipeline runs, vulnerabilities are identified and compared between the source and target branches. Vulnerabilities and their severity are listed in the merge request, enabling you to proactively address the risk to your application, before the code change is committed. +Vulnerabilities can also be identified outside a pipeline by +[Continuous Vulnerability Scanning](../continuous_vulnerability_scanning/index.md). GitLab offers both Dependency Scanning and [Container Scanning](../container_scanning/index.md) to ensure coverage for all of these dependency types. To cover as much of your risk area as possible, @@ -327,7 +329,7 @@ GitLab analyzers obtain dependency information using one of the following two me The following package managers use lockfiles that GitLab analyzers are capable of parsing directly: -| Package Manager | Supported File Format Versions | Tested Versions | +| Package Manager | Supported File Format Versions | Tested Package Manager Versions | | ------ | ------ | ------ | | Bundler | Not applicable | [1.17.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/ruby-bundler/default/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | | Composer | Not applicable | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/php-composer/default/composer.lock) | @@ -335,7 +337,7 @@ The following package managers use lockfiles that GitLab analyzers are capable o | Go | Not applicable | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/go-modules/gosum/default/go.sum) <sup><strong><a href="#notes-regarding-parsing-lockfiles-1">1</a></strong></sup> | | NuGet | v1, v2<sup><b><a href="#notes-regarding-parsing-lockfiles-2">2</a></b></sup> | [4.9](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/csharp-nuget-dotnetcore/default/src/web.api/packages.lock.json#L2) | | npm | v1, v2, v3<sup><b><a href="#notes-regarding-parsing-lockfiles-3">3</a></b></sup> | [6.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-npm/default/package-lock.json#L4), [7.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-npm/lockfileVersion2/package-lock.json#L4), [9.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/scanner/parser/npm/fixtures/lockfile-v3/simple/package-lock.json#L4) | -| pnpm | v5.3, v5.4, v6 | [7.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-pnpm/default/pnpm-lock.yaml#L1), [8.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/scanner/parser/pnpm/fixtures/v6/simple/pnpm-lock.yaml#L1) | +| pnpm | v5, v6 | [7.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-pnpm/default/pnpm-lock.yaml#L1), [8.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/scanner/parser/pnpm/fixtures/v6/simple/pnpm-lock.yaml#L1) | | yarn | v1, v2<sup><b><a href="#notes-regarding-parsing-lockfiles-4">4</a></b></sup>, v3<sup><b><a href="#notes-regarding-parsing-lockfiles-4">4</a></b></sup> | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-yarn/classic/default/yarn.lock#L2), [2.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-yarn/berry/v2/default/yarn.lock), [3.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-yarn/berry/v3/default/yarn.lock) | | Poetry | v1 | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/python-poetry/default/poetry.lock) | @@ -559,7 +561,7 @@ always take the latest dependency scanning artifact available. To enable Dependency Scanning in a project, you can create a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Dependency Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable Dependency Scanning. @@ -1019,62 +1021,7 @@ variables: See explanations of the variables above in the [configuration section](#configuration). -### Specific settings for languages and package managers - -See the following sections for configuring specific languages and package managers. - -#### Python (pip) - -If you need to install Python packages before the analyzer runs, you should use `pip install --user` in the `before_script` of the scanning job. The `--user` flag causes project dependencies to be installed in the user directory. If you do not pass the `--user` option, packages are installed globally, and they are not scanned and don't show up when listing project dependencies. - -#### Python (setuptools) - -If you need to install Python packages before the analyzer runs, you should use `python setup.py install --user` in the `before_script` of the scanning job. The `--user` flag causes project dependencies to be installed in the user directory. If you do not pass the `--user` option, packages are installed globally, and they are not scanned and don't show up when listing project dependencies. - -When using self-signed certificates for your private PyPi repository, no extra job configuration (aside -from the template `.gitlab-ci.yml` above) is needed. However, you must update your `setup.py` to -ensure that it can reach your private repository. Here is an example configuration: - -1. Update `setup.py` to create a `dependency_links` attribute pointing at your private repository for each - dependency in the `install_requires` list: - - ```python - install_requires=['pyparsing>=2.0.3'], - dependency_links=['https://pypi.example.com/simple/pyparsing'], - ``` - -1. Fetch the certificate from your repository URL and add it to the project: - - ```shell - printf "\n" | openssl s_client -connect pypi.example.com:443 -servername pypi.example.com | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > internal.crt - ``` - -1. Point `setup.py` at the newly downloaded certificate: - - ```python - import setuptools.ssl_support - setuptools.ssl_support.cert_paths = ['internal.crt'] - ``` - -#### Python (Pipenv) - -If running in a limited network connectivity environment, you must configure the `PIPENV_PYPI_MIRROR` -variable to use a private PyPi mirror. This mirror must contain both default and development dependencies. - -```yaml -variables: - PIPENV_PYPI_MIRROR: https://pypi.example.com/simple -``` - -<!-- markdownlint-disable MD044 --> -Alternatively, if it's not possible to use a private registry, you can load the required packages -into the Pipenv virtual environment cache. For this option, the project must check in the -`Pipfile.lock` into the repository, and load both default and development packages into the cache. -See the example [python-pipenv](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/41cc017bd1ed302f6edebcfa3bc2922f428e07b6/.gitlab-ci.yml#L20-42) -project for an example of how this can be done. -<!-- markdownlint-enable MD044 --> - -## Hosting a copy of the `gemnasium_db` advisory database +### Hosting a copy of the `gemnasium_db` advisory database The [`gemnasium_db`](https://gitlab.com/gitlab-org/security-products/gemnasium-db) Git repository is used by `gemnasium`, `gemnasium-maven`, and `gemnasium-python` as the source of vulnerability data. @@ -1085,7 +1032,7 @@ one of the following: - [Host a copy of the advisory database](#host-a-copy-of-the-advisory-database) - [Use a local clone](#use-a-local-clone) -### Host a copy of the advisory database +#### Host a copy of the advisory database If [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is not reachable from within the environment, the user can host their own Git copy. Then the analyzer can be @@ -1098,7 +1045,7 @@ variables: ... ``` -### Use a local clone +#### Use a local clone If a hosted copy is not possible, then the user can clone [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) or create an archive before the scan and point the analyzer to the directory (using: @@ -1129,6 +1076,61 @@ variables: GRADLE_CLI_OPTS: "-Dhttps.proxyHost=squid-proxy -Dhttps.proxyPort=3128 -Dhttp.proxyHost=squid-proxy -Dhttp.proxyPort=3128 -Dhttp.nonProxyHosts=localhost" ``` +## Specific settings for languages and package managers + +See the following sections for configuring specific languages and package managers. + +### Python (pip) + +If you need to install Python packages before the analyzer runs, you should use `pip install --user` in the `before_script` of the scanning job. The `--user` flag causes project dependencies to be installed in the user directory. If you do not pass the `--user` option, packages are installed globally, and they are not scanned and don't show up when listing project dependencies. + +### Python (setuptools) + +If you need to install Python packages before the analyzer runs, you should use `python setup.py install --user` in the `before_script` of the scanning job. The `--user` flag causes project dependencies to be installed in the user directory. If you do not pass the `--user` option, packages are installed globally, and they are not scanned and don't show up when listing project dependencies. + +When using self-signed certificates for your private PyPi repository, no extra job configuration (aside +from the template `.gitlab-ci.yml` above) is needed. However, you must update your `setup.py` to +ensure that it can reach your private repository. Here is an example configuration: + +1. Update `setup.py` to create a `dependency_links` attribute pointing at your private repository for each + dependency in the `install_requires` list: + + ```python + install_requires=['pyparsing>=2.0.3'], + dependency_links=['https://pypi.example.com/simple/pyparsing'], + ``` + +1. Fetch the certificate from your repository URL and add it to the project: + + ```shell + printf "\n" | openssl s_client -connect pypi.example.com:443 -servername pypi.example.com | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > internal.crt + ``` + +1. Point `setup.py` at the newly downloaded certificate: + + ```python + import setuptools.ssl_support + setuptools.ssl_support.cert_paths = ['internal.crt'] + ``` + +### Python (Pipenv) + +If running in a limited network connectivity environment, you must configure the `PIPENV_PYPI_MIRROR` +variable to use a private PyPi mirror. This mirror must contain both default and development dependencies. + +```yaml +variables: + PIPENV_PYPI_MIRROR: https://pypi.example.com/simple +``` + +<!-- markdownlint-disable MD044 --> +Alternatively, if it's not possible to use a private registry, you can load the required packages +into the Pipenv virtual environment cache. For this option, the project must check in the +`Pipfile.lock` into the repository, and load both default and development packages into the cache. +See the example [python-pipenv](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/41cc017bd1ed302f6edebcfa3bc2922f428e07b6/.gitlab-ci.yml#L20-42) +project for an example of how this can be done. +<!-- markdownlint-enable MD044 --> + ## Warnings We recommend that you use the most recent version of all containers, and the most recent supported version of all package managers and languages. Using previous versions carries an increased security risk because unsupported versions may no longer benefit from active security reporting and backporting of security fixes. diff --git a/doc/user/application_security/gitlab_advisory_database/index.md b/doc/user/application_security/gitlab_advisory_database/index.md new file mode 100644 index 00000000000..e59eaccf8f8 --- /dev/null +++ b/doc/user/application_security/gitlab_advisory_database/index.md @@ -0,0 +1,95 @@ +--- +stage: Secure +group: Vulnerability Research +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab Advisory Database + +The [GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db) serves as a repository for security advisories related to software dependencies. + +The database is an essential component of both [Dependency Scanning](../dependency_scanning/index.md) and [Container Scanning](../container_scanning/index.md). + +A free and open-source version of the GitLab Advisory Database is also available as [GitLab Advisory Database (Open Source Edition)](https://gitlab.com/gitlab-org/advisories-community). However, there is a 30-day delay in updates. + +## Standardization + +In our advisories, we adopt standardized practices to effectively communicate vulnerabilities and their impact. + +- [CVE](../terminology/index.md#cve) +- [CVSS](../terminology/index.md#cvss) +- [CWE](../terminology/index.md#cwe) + +## Explore the database + +To view the database content, go to the [GitLab Advisory Database](https://advisories.gitlab.com) home page. On the home page you can: + +- Search the database, by identifier, package name, and description. +- View advisories that were added recently. +- View statistical information, including coverage and update frequency. + +### Search + +Each advisory has a page with the following details: + +- **Identifiers**: Public identifiers. For example, CVE ID, GHSA ID, or the GitLab internal ID (`GMS-<year>-<nr>`). +- **Package Slug**: Package type and package name separated by a slash. +- **Vulnerability**: A short description of the security flaw. +- **Description**: A detailed description of the security flaw and potential risks. +- **Affected Versions**: The affected versions. +- **Solution**: How to remediate the vulnerability. +- **Last Modified**: The date when the advisory was last modified. + +### Statistics + +The home page also offers a [statistic section](https://advisories.gitlab.com/stats/index.html) that provides valuable insights into advisory distribution, the origins of vulnerabilities, dependency scanning coverage, and timelines for vulnerability resolution. + +## Open Source Edition + +GitLab provides a free and open-source version of the database, the [GitLab Advisory Database (Open Source Edition)](https://gitlab.com/gitlab-org/advisories-community). + +The open-source version is a time-delayed clone of the GitLab Advisory Database, MIT-licensed and contains all advisories from the GitLab Advisory Database that are older than 30 days or with the `community-sync` flag. + +## Integrations + +- [Dependency Scanning](../dependency_scanning/index.md) +- [Container Scanning](../container_scanning/index.md) +- Third-party tools + +NOTE: +GitLab Advisory Database Terms prohibit the use of data contained in the GitLab Advisory Database by third-party tools. Third-party integrators can use the MIT-licensed, time-delayed [repository clone](https://gitlab.com/gitlab-org/advisories-community) instead. + +### How the database can be used + +As an example, we highlight the use of the database as a source for an Advisory Ingestion process as part of Continuous Vulnerability Scans. + +```mermaid +flowchart TB + subgraph Dependency Scanning + A[GitLab Advisory Database] + end + subgraph Container Scanning + C[GitLab Advisory Database \n Open Source Edition \n integrated into Trivy] + end + A --> B{Ingest} + C --> B + B --> |store| D{{"Cloud Storage \n (NDJSON format)"}} + F[\GitLab Instance/] --> |pulls data| D + F --> |stores| G[(Relational Database)] +``` + +## Maintenance + +The [Vulnerability Research](https://about.gitlab.com/handbook/engineering/development/sec/secure/vulnerability-research/) team is responsible for the maintenance and regular updates of the GitLab Advisory Database and the GitLab Advisory Database (Open Source Edition). + +Community contributions are accessible in [advisories-community](https://gitlab.com/gitlab-org/advisories-community) via the `community-sync` flag. + +## Contributing to the vulnerability database + +If you know about a vulnerability that is not listed, you can contribute to the GitLab Advisory Database by either opening an issue or submit the vulnerability. + +For more information, see [Contribution Guidelines](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/CONTRIBUTING.md). + +## License + +The GitLab Advisory Database is freely accessible in accordance with the [GitLab Advisory Database Terms](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/LICENSE.md#gitlab-advisory-database-term). diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index 334e8c28378..8cdeeb92e09 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -121,7 +121,7 @@ The **Configure with a merge request** button been temporarily disabled due to a To enable IaC Scanning in a project, you can create a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable IaC Scanning. @@ -130,7 +130,7 @@ Pipelines now include an IaC Scanning job. ## Customize rulesets **(ULTIMATE ALL)** -> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. +> Support for overriding rules [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) in GitLab 14.8. You can customize the default IaC Scanning rules provided with GitLab. diff --git a/doc/user/application_security/img/security_widget_v13_7.png b/doc/user/application_security/img/security_widget_v13_7.png Binary files differdeleted file mode 100644 index fb1eaf9a2be..00000000000 --- a/doc/user/application_security/img/security_widget_v13_7.png +++ /dev/null diff --git a/doc/user/application_security/img/security_widget_v16_4.png b/doc/user/application_security/img/security_widget_v16_4.png Binary files differnew file mode 100644 index 00000000000..dfed372ba31 --- /dev/null +++ b/doc/user/application_security/img/security_widget_v16_4.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 615ff7e3fda..62155e07fbc 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -103,7 +103,7 @@ The following vulnerability scanners and their databases are regularly updated: |:----------------------------------------------------------------|:---------------------------------| | [Container Scanning](container_scanning/index.md) | A job runs on a daily basis to build new images with the latest vulnerability database updates from the upstream scanner. GitLab monitors this job through an internal alert that tells the engineering team when the database becomes more than 48 hours old. For more information, see the [Vulnerabilities database update](container_scanning/index.md#vulnerabilities-database). | | [Dependency Scanning](dependency_scanning/index.md) | Relies on the [GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db). It is updated on a daily basis using [data from NVD, the `ruby-advisory-db` and the GitHub Advisory Database as data sources](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/SOURCES.md). See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | -| [Dynamic Application Security Testing (DAST)](dast/index.md) | The scanning engine is updated on a periodic basis. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/main/Dockerfile#L1). The scanning rules are downloaded at scan runtime. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) | [DAST proxy-based](dast/proxy-based.md) and [browser-based](dast/browser_based.md) engines are updated on a periodic basis. [DAST proxy-based](dast/proxy-based.md) analyzer downloads the scanning rules at scan runtime. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/main/Dockerfile#L27). [DAST browser-based](dast/browser_based.md) rules run [different vulnerability checks](dast/checks/index.md). | | [Secret Detection](secret_detection/index.md#detected-secrets) | GitLab maintains the [detection rules](secret_detection/index.md#detected-secrets) and [accepts community contributions](secret_detection/index.md#adding-new-patterns). The scanning engine is updated at least once per month if a relevant update is available. | | [Static Application Security Testing (SAST)](sast/index.md) | The source of scan rules depends on which [analyzer](sast/analyzers.md) is used for each [supported programming language](sast/index.md#supported-languages-and-frameworks). GitLab maintains a ruleset for the Semgrep-based analyzer and updates it regularly based on internal research and user feedback. For other analyzers, the ruleset is sourced from the upstream open-source scanner. Each analyzer is updated at least once per month if a relevant update is available. | @@ -223,20 +223,30 @@ We do not recommend changing the job [`allow_failure` setting](../../ci/yaml/ind The artifact generated by the secure analyzer contains all findings it discovers on the target branch, regardless of whether they were previously found, dismissed, or completely new (it puts in everything that it finds). -## View security scan information in merge requests **(FREE ALL)** +## View security scan information + +Security scan information appears in multiple locations and formats: + +- Merge request +- Pipeline security tab +- Security dashboard +- Vulnerability report +- GitLab Workflow extension for VS Code + +### Merge request **(FREE ALL)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4393) in GitLab Free 13.5. > - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/273205) in 13.6. > - Report download dropdown list [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249550) in GitLab 13.9. -### All tiers +#### All tiers Merge requests which have run security scans let you know that the generated reports are available to download. To download a report, select **Download results**, and select the desired report. - + Security scans produce at least one of these [CI `artifacts:reports` types](../../ci/yaml/artifacts_reports.md): @@ -248,7 +258,9 @@ Security scans produce at least one of these [CI `artifacts:reports` types](../. - `artifacts:reports:sast` - `artifacts:reports:secret_detection` -### Ultimate +In the Free tier, the reports above aren't parsed by GitLab. As a result, the widget does not change based on the results of the security scans. + +#### Ultimate A merge request contains a security widget which displays a summary of the _new_ results. New results are determined by comparing the findings of the merge request against the findings of the most recent completed pipeline (`success`, `failed`, `canceled` or `skipped`) for the commit when the feature branch was created from the target branch. @@ -263,26 +275,32 @@ findings, select **View full report** to go directly to the **Security** tab in  -## View security scan information in the pipeline Security tab +### Pipeline security tab A pipeline's security tab lists all findings in the current branch. It includes new findings introduced by this branch and existing vulnerabilities already present when you created the branch. These results likely do not match the findings displayed in the Merge Request security widget, as those do not include the existing vulnerabilities. Refer to [View vulnerabilities in a pipeline](vulnerability_report/pipeline.md) for more information. -## View security scan information in the Security Dashboard +### Security dashboard -The Security Dashboard show vulnerabilities present in a project's default branch. Data is updated every 24 hours. Vulnerability count updates resulting from any feature branches introducing new vulnerabilities that are merged to default are included after the daily data refresh. +The security dashboard shows the vulnerabilities on a project's default branch. Data is updated every 24 hours. Vulnerability count updates resulting from any feature branches introducing new vulnerabilities that are merged to default are included after the daily data refresh. For more details, see [Security Dashboard](security_dashboard/index.md). -## View security scan information in the Vulnerability Report +### Vulnerability report -The vulnerability report shows the results of the last completed pipeline on the default branch. It is updated on every pipeline completion. All detected vulnerabilities are shown as well as any previous ones that are no longer detected in the latest scan. Vulnerabilities that are no longer detected may have been remediated or otherwise removed and can be marked as `Resolved` after proper verification. Vulnerabilities that are no longer detected are denoted with an icon for filtering and review. +The vulnerability report shows the results of the last completed pipeline on the default branch. It is updated on every pipeline completion. All detected vulnerabilities are shown and any previous ones that are no longer detected in the latest scan. Vulnerabilities that are no longer detected may have been remediated or otherwise removed and can be marked as `Resolved` after proper verification. Vulnerabilities that are no longer detected are denoted with an icon for filtering and review. By default, the vulnerability report does not show vulnerabilities of `dismissed` or `resolved` status so you can focus on open vulnerabilities. You can change the Status filter to see these. [Read more about the Vulnerability report](vulnerability_report/index.md). +### GitLab Workflow extension for VS Code + +You can now see security findings directly in Visual Studio Code (VS Code) using [GitLab Workflow VS Code extension](../../editor_extensions/visual_studio_code/index.md), just as you would in a merge request. + +For more details, see [extension page](https://marketplace.visualstudio.com/items?itemName=gitlab.gitlab-workflow#security-findings). + ## Security approvals in merge requests > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9928) in GitLab 12.2. @@ -453,7 +471,7 @@ You can always find supported and deprecated schema versions in the [source code You can interact with the results of the security scanning tools in several locations: -- [Scan information in merge requests](#view-security-scan-information-in-merge-requests) +- [Scan information in merge requests](#merge-request) - [Project Security Dashboard](security_dashboard/index.md#view-vulnerabilities-over-time-for-a-project) - [Security pipeline tab](security_dashboard/index.md) - [Group Security Dashboard](security_dashboard/index.md#view-vulnerabilities-over-time-for-a-group) @@ -526,7 +544,7 @@ Additional details about the differences between the two solutions are outlined | **Flexibility** | Supports anything that can be done in a CI file. | Limited to only the items for which GitLab has explicitly added support. DAST, SAST, SAST IaC, Secret Detection, Dependency Scanning, and Container Scanning scans are supported. | | **Usability** | Requires knowledge of CI YAML. | Follows a `rules` and `actions`-based YAML structure. | | **Inclusion in CI pipeline** | The compliance pipeline is executed instead of the project's `.gitlab-ci.yml` file. To include the project's `.gitlab-ci.yml` file, use an `include` statement. Defined variables aren't allowed to be overwritten by the included project's YAML file. | Forced inclusion of a new job into the CI pipeline. DAST jobs that must be customized on a per-project basis can have project-level Site Profiles and Scan Profiles defined. To ensure separation of duties, these profiles are immutable when referenced in a scan execution policy. All jobs can be customized as part of the security policy itself with the same variables that are usually available to the CI job. | -| **Schedulable** | Can be scheduled through a scheduled pipeline on the group. | Can be scheduled natively through the policy configuration itself. | +| **Schedulable** | Has to be scheduled through a scheduled pipeline on each project. | Can be scheduled natively through the policy configuration itself. | | **Separation of Duties** | Only group owners can create compliance framework labels. Only project owners can apply compliance framework labels to projects. The ability to make or approve changes to the compliance pipeline definition is limited to individuals who are explicitly given access to the project that contains the compliance pipeline. | Only project owners can define a linked security policy project. The ability to make or approve changes to security policies is limited to individuals who are explicitly given access to the security policy project. | | **Ability to apply one standard to multiple projects** | The same compliance framework label can be applied to multiple projects inside a group. | The same security policy project can be used for multiple projects across GitLab with no requirement of being located in the same group. | @@ -721,8 +739,8 @@ Instructions are available in the [legacy template project](https://gitlab.com/g In these circumstances, that the job succeeds is the default behavior. The job's status indicates success or failure of the analyzer itself. Analyzer results are displayed in the [job logs](../../ci/jobs/index.md#expand-and-collapse-job-log-sections), -[Merge Request widget](#view-security-scan-information-in-merge-requests) or -[Security Dashboard](security_dashboard/index.md). +[merge request widget](#merge-request), or +[security dashboard](security_dashboard/index.md). ### Error: job `is used for configuration only, and its script should not be executed` diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index 25e2f523f08..84c28d4008c 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -58,7 +58,7 @@ to select, edit, and unlink a security policy project. As a project owner, take the following steps to create or edit an association between your current project and a project that you would like to designate as the security policy project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Policies**. 1. Select **Edit Policy Project**, and search for and select the project you would like to link from the dropdown list. @@ -82,7 +82,7 @@ policies for all available environments. You can check a policy's information (for example, description or enforcement status), and create and edit deployed policies: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Policies**.  @@ -93,7 +93,7 @@ status), and create and edit deployed policies: You can use the policy editor to create, edit, and delete policies: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Policies**. - To create a new policy, select **New policy** which is located in the **Policies** page's header. You can then select which type of policy to create. @@ -142,12 +142,13 @@ The workaround is to amend your group or instance push rules to allow branches f ### Troubleshooting common issues configuring security policies - Confirm that scanners are properly configured and producing results for the latest branch. Security Policies are designed to require approval when there are no results (no security report), as this ensures that no vulnerabilities are introduced. We cannot know if there are any vulnerabilities unless the scans enforced by the policy complete successfully and are evaluated. +- For scan result policies, we require artifacts for each scanner defined in the policy for both the source and target branch. To ensure scan result policies capture the necessary results, confirm your scan execution is properly implemented and enforced. If using scan execution policies, enforcing on `all branches` often address this need. - When running scan execution policies based on a SAST action, ensure target repositories contain proper code files. SAST runs different analyzers [based on the types of files in the repo](../sast/index.md#supported-languages-and-frameworks), and if no supported files are found it will not run any jobs. See the [SAST CI template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) for more details. -- Check for any branch configuration conflicts. If your policy is configured to enforce rules on `main` but some projects within the scope are using `master` as their default branch, the policy is not applied for the latter. Support for specifying the `default` branch in your policies is proposed in [epic 9468](https://gitlab.com/groups/gitlab-org/-/epics/9468). +- Check for any branch configuration conflicts. If your policy is configured to enforce rules on `main` but some projects within the scope are using `master` as their default branch, the policy is not applied for the latter. You can define policies to enforce rules generically on `default` branches regardless of the name used in the project or on `all protected branches` to address this issue. - Scan result policies created at the group or sub-group level can take some time to apply to all the merge requests in the group. - Scheduled scan execution policies run with a minimum 15 minute cadence. Learn more [about the schedule rule type](../policies/scan-execution-policies.md#schedule-rule-type). - When scheduling pipelines, keep in mind that CRON scheduling is based on UTC on GitLab SaaS and is based on your server time for self managed instances. When testing new policies, it may appear pipelines are not running properly when in fact they are scheduled in your server's timezone. -- When enforcing scan execution policies, security policies creates a bot in the target project that will trigger scheduled pipelines to ensure enforcement. If the bot is +- When enforcing scan execution policies, security policies creates a bot in the target project that will trigger scheduled pipelines to ensure enforcement. If the bot is deleted or missing, the target project's pipeline will not be executed. To recreate a security policy bot user unlink and link the security policy project again. - You should not link a security policy project to a development project and to the group or sub-group the development project belongs to at the same time. Linking this way will result in approval rules from the Scan Result Policy not being applied to merge requests in the development project. - When creating a Scan Result Policy, neither the array `severity_levels` nor the array `vulnerability_states` in the [scan_finding rule](../policies/scan-result-policies.md#scan_finding-rule-type) can be left empty; for a working rule, at least one entry must exist. diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 834a50f39ef..ac15dfc0a47 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -97,7 +97,12 @@ the following sections and tables provide an alternative. ## `pipeline` rule type > - The `branch_type` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/404774) in GitLab 16.1 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_type`. Disabled by default. -> - The `branch_type` field was [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/413062) in GitLab 16.2. +> - Generally available in GitLab 16.2. Feature flag `security_policies_branch_type` removed. +> - The `branch_exceptions` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418741) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. Enabled by default. + +FLAG: +On self-managed GitLab, by default the `branch_exceptions` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. +On GitLab.com, this feature is available. This rule enforces the defined actions whenever the pipeline runs for a selected branch. @@ -106,33 +111,35 @@ This rule enforces the defined actions whenever the pipeline runs for a selected | `type` | `string` | true | `pipeline` | The rule's type. | | `branches` <sup>1</sup> | `array` of `string` | true if `branch_type` field does not exist | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | | `branch_type` <sup>1</sup> | `string` | true if `branches` field does not exist | `default`, `protected` or `all` | The types of branches the given policy applies to. | +| `branch_exceptions` | `array` of `string` | false | Names of branches | Branches to exclude from this rule. | 1. You must specify only one of `branches` or `branch_type`. ## `schedule` rule type > - The `branch_type` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/404774) in GitLab 16.1 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_type`. Disabled by default. -> - The `branch_type` field was [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/413062) in GitLab 16.2. -> - The security policy bot users were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/394958) in GitLab 16.3 [with flags](../../../administration/feature_flags.md) named `scan_execution_group_bot_users` and `scan_execution_bot_users`. Enabled by default. +> - Generally available in GitLab 16.2. Feature flag `security_policies_branch_type` removed. +> - The `branch_exceptions` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418741) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. Enabled by default. FLAG: -On self-managed GitLab, security policy bot users are available. To hide the feature, an administrator can [disable the feature flags](../../../administration/feature_flags.md) named `scan_execution_group_bot_users` and `scan_execution_bot_users`. +On self-managed GitLab, by default the `branch_exceptions` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. On GitLab.com, this feature is available. -This rule enforces the defined actions and schedules a scan on the provided date/time. +This rule schedules a scan pipeline, enforcing the defined actions on the schedule defined in the `cadence` field. A scheduled pipeline does not run other jobs defined in the project's `.gitlab-ci.yml` file. When a project is linked to a security policy project, a security policy bot is created in the project and will become the author of any scheduled pipelines. | Field | Type | Required | Possible values | Description | |------------|------|----------|-----------------|-------------| | `type` | `string` | true | `schedule` | The rule's type. | | `branches` <sup>1</sup> | `array` of `string` | true if either `branch_type` or `agents` fields does not exist | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | | `branch_type` <sup>1</sup> | `string` | true if either `branches` or `agents` fields does not exist | `default`, `protected` or `all` | The types of branches the given policy applies to. | +| `branch_exceptions` | `array` of `string` | false | Names of branches | Branches to exclude from this rule. | | `cadence` | `string` | true | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. Minimum of 15 minute intervals when used together with the `branches` field. | | `timezone` | `string` | false | Time zone identifier (for example, `America/New_York`) | Time zone to apply to the cadence. Value must be an IANA Time Zone Database identifier. | | `agents` <sup>1</sup> | `object` | true if either `branch_type` or `branches` fields do not exists | | The name of the [GitLab agents](../../clusters/agent/index.md) where [Operational Container Scanning](../../clusters/agent/vulnerabilities.md) runs. The object key is the name of the Kubernetes agent configured for your project in GitLab. | 1. You must specify only one of `branches`, `branch_type`, or `agents`. -Scheduled scan pipelines are triggered by a security policy bot user that is a guest member of the project. Security policy bot users are automatically created when the security policy project is linked, and removed when the security policy project is unlinked. +Scheduled scan pipelines are triggered by a security policy bot user that is a guest member of the project with elevated permissions for users of type `security_policy_bot` so it may carry out this task. Security policy bot users are automatically created when the security policy project is linked, and removed when the security policy project is unlinked. If the project does not have a security policy bot user, the scheduled scan pipeline will not be triggered. To recreate a security policy bot user unlink and link the security policy project again. @@ -214,11 +221,12 @@ Note the following: is not scheduled successfully. - For a secret detection scan, only rules with the default ruleset are supported. [Custom rulesets](../secret_detection/index.md#custom-rulesets) are not supported. -- A secret detection scan runs in `normal` mode when executed as part of a pipeline, and in +- A secret detection scan runs in `default` mode when executed as part of a pipeline, and in [`historic`](../secret_detection/index.md#full-history-secret-detection) mode when executed as part of a scheduled scan. - A container scanning scan that is configured for the `pipeline` rule type ignores the agent defined in the `agents` object. The `agents` object is only considered for `schedule` rule types. An agent with a name provided in the `agents` object must be created and configured for the project. +- Variables defined in a Scan Execution Policy follow the standard [CI/CD variable precedence](../../../ci/variables/index.md#cicd-variable-precedence). ## Example security policies project diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index c5cfd63641b..a42b3f02c26 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -84,23 +84,36 @@ the following sections and tables provide an alternative. ## Scan result policy schema +> The `approval_settings` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418752) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `scan_result_policy_settings`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `scan_result_policy_settings`. +On GitLab.com, this feature is not available. + | Field | Type | Required |Possible values | Description | |-------|------|----------|----------------|-------------| | `name` | `string` | true | | Name of the policy. Maximum of 255 characters.| | `description` (optional) | `string` | true | | Description of the policy. | | `enabled` | `boolean` | true | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | | `rules` | `array` of rules | true | | List of rules that the policy applies. | -| `actions` | `array` of actions | true | | List of actions that the policy enforces. | +| `actions` | `array` of actions | false| | List of actions that the policy enforces. | +| `approval_settings` | `object` | false | `{prevent_approval_by_author: boolean, prevent_approval_by_commit_author: boolean, remove_approvals_with_new_commit: boolean, require_password_to_approve: boolean}` | Project settings that the policy overrides. | ## `scan_finding` rule type > - The scan result policy field `vulnerability_attributes` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123052) in GitLab 16.2 [with a flag](../../../administration/feature_flags.md) named `enforce_vulnerability_attributes_rules`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/418784) in GitLab 16.3. > - The scan result policy field `vulnerability_age` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123956) in GitLab 16.2. +> - The `branch_exceptions` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418741) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. Enabled by default. FLAG: On self-managed GitLab, by default the `vulnerability_attributes` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `enforce_vulnerability_attributes_rules`. On GitLab.com, this feature is available. + +FLAG: +On self-managed GitLab, by default the `branch_exceptions` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. +On GitLab.com, this feature is available. + This rule enforces the defined actions based on security scan findings. | Field | Type | Required | Possible values | Description | @@ -108,6 +121,7 @@ This rule enforces the defined actions based on security scan findings. | `type` | `string` | true | `scan_finding` | The rule's type. | | `branches` | `array` of `string` | true if `branch_type` field does not exist | `[]` or the branch's name | Applicable only to protected target branches. An empty array, `[]`, applies the rule to all protected target branches. Cannot be used with the `branch_type` field. | | `branch_type` | `string` | true if `branches` field does not exist | `default` or `protected` | The types of branches the given policy applies to. Cannot be used with the `branches` field. | +| `branch_exceptions` | `array` of `string` | false | Names of branches | Branches to exclude from this rule. | | `scanners` | `array` of `string` | true | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. `sast` includes results from both SAST and SAST IaC scanners. | | `vulnerabilities_allowed` | `integer` | true | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | | `severity_levels` | `array` of `string` | true | `info`, `unknown`, `low`, `medium`, `high`, `critical` | The severity levels for this rule to consider. | @@ -119,6 +133,11 @@ This rule enforces the defined actions based on security scan findings. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8092) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `license_scanning_policies`. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/397644) in GitLab 15.11. Feature flag `license_scanning_policies` removed. +> - The `branch_exceptions` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418741) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. Enabled by default. + +FLAG: +On self-managed GitLab, by default the `branch_exceptions` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. +On GitLab.com, this feature is available. This rule enforces the defined actions based on license findings. @@ -127,10 +146,30 @@ This rule enforces the defined actions based on license findings. | `type` | `string` | true | `license_finding` | The rule's type. | | `branches` | `array` of `string` | true if `branch_type` field does not exist | `[]` or the branch's name | Applicable only to protected target branches. An empty array, `[]`, applies the rule to all protected target branches. Cannot be used with the `branch_type` field. | | `branch_type` | `string` | true if `branches` field does not exist | `default` or `protected` | The types of branches the given policy applies to. Cannot be used with the `branches` field. | +| `branch_exceptions` | `array` of `string` | false | Names of branches | Branches to exclude from this rule. | | `match_on_inclusion` | `boolean` | true | `true`, `false` | Whether the rule matches inclusion or exclusion of licenses listed in `license_types`. | | `license_types` | `array` of `string` | true | license types | [SPDX license names](https://spdx.org/licenses) to match on, for example `Affero General Public License v1.0` or `MIT License`. | | `license_states` | `array` of `string` | true | `newly_detected`, `detected` | Whether to match newly detected and/or previously detected licenses. The `newly_detected` state triggers approval when either a new package is introduced or when a new license for an existing package is detected. | +## `any_merge_request` rule type + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418752) in GitLab 16.4. +> - The `branch_exceptions` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418741) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. Enabled by default. + +FLAG: +On self-managed GitLab, by default the `branch_exceptions` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `security_policies_branch_exceptions`. +On GitLab.com, this feature is available. + +This rule enforces the defined actions for any merge request based on the commits signature. + +| Field | Type | Required | Possible values | Description | +|---------------|---------------------|--------------------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `type` | `string` | true | `any_merge_request` | The rule's type. | +| `branches` | `array` of `string` | true if `branch_type` field does not exist | `[]` or the branch's name | Applicable only to protected target branches. An empty array, `[]`, applies the rule to all protected target branches. Cannot be used with the `branch_type` field. | +| `branch_type` | `string` | true if `branches` field does not exist | `default` or `protected` | The types of branches the given policy applies to. Cannot be used with the `branches` field. | +| `branch_exceptions` | `array` of `string` | false | Names of branches | Branches to exclude from this rule. | +| `commits` | `string` | true | `any`, `unsigned` | Whether the rule matches for any commits, or only if unsigned commits are detected in the merge request. | + ## `require_approval` action type This action sets an approval rule to be required when conditions are met for at least one rule in diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 832ad100701..f896616d537 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -64,7 +64,7 @@ content directly. Instead, it enhances the results with additional properties, i - CWEs. - Location tracking fields. -- A means of identifying false positives or insignificant findings. **(ULTIMATE)** +- A means of identifying false positives or insignificant findings. **(ULTIMATE ALL)** ## Transition to Semgrep-based scanning diff --git a/doc/user/application_security/sast/customize_rulesets.md b/doc/user/application_security/sast/customize_rulesets.md index 4ae8f1c4f8b..90731114303 100644 --- a/doc/user/application_security/sast/customize_rulesets.md +++ b/doc/user/application_security/sast/customize_rulesets.md @@ -7,10 +7,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Customize rulesets **(ULTIMATE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235382) in GitLab 13.5. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. -> - [Added](https://gitlab.com/gitlab-org/security-products/analyzers/ruleset/-/merge_requests/18) support for specifying ambiguous passthrough refs in GitLab 16.2. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. +> - [Enabled](https://gitlab.com/gitlab-org/security-products/analyzers/ruleset/-/merge_requests/18) support for specifying ambiguous passthrough refs in GitLab 16.2. You can customize the behavior of our SAST analyzers by [defining a ruleset configuration file](#create-the-configuration-file) in the repository being scanned. There are two kinds of customization: @@ -29,8 +29,8 @@ You can disable predefined rules for any SAST analyzer. When you disable a rule: - Most analyzers still scan for the vulnerability. The results are removed as a processing step after the scan completes, and they don't appear in the [`gl-sast-report.json` artifact](index.md#reports-json-format). -- Findings for the disabled rule no longer appear in the [Pipeline Security tab](../index.md#view-security-scan-information-in-the-pipeline-security-tab). -- Existing findings for the disabled rule on the default branch are marked ["No longer detected"](../vulnerability_report/index.md#activity-filter) in the [Vulnerability Report](../index.md#view-security-scan-information-in-the-vulnerability-report). +- Findings for the disabled rule no longer appear in the [pipeline security tab](../index.md#pipeline-security-tab). +- Existing findings for the disabled rule on the default branch are marked as [`No longer detected`](../vulnerability_report/index.md#activity-filter) in the [vulnerability report](../index.md#vulnerability-report). The Semgrep-based analyzer handles disabled rules differently: diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 717608274e5..acc7e9d9e84 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -55,21 +55,16 @@ For more information about our plans for language support in SAST, see the [cate | Language / framework | [Analyzer](analyzers.md) used for scanning | Minimum supported GitLab version | |------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| .NET Core<sup>3</sup> | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 11.0 | -| .NET Framework<sup>3</sup> | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 13.0 | | .NET (all versions, C# only) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 15.4 | | Apex (Salesforce) | [PMD](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | 12.1 | | C | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.2 | | C/C++ | [Flawfinder](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | 10.7 | | Elixir (Phoenix) | [Sobelow](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | 11.1 | -| Go<sup>2</sup> | [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | 10.7 | | Go | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.4 | | Groovy<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | | Helm Charts | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 13.1 | | Java (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.10 | -| Java<sup>1, 2</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | | Java (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | -| JavaScript<sup>2</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.8 | | JavaScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | | Kotlin (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | | Kotlin (General)<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 13.11 | @@ -77,24 +72,34 @@ For more information about our plans for language support in SAST, see the [cate | Node.js | [NodeJsScan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | 11.1 | | Objective-C (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | | PHP | [phpcs-security-audit](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | 10.8 | -| Python<sup>2</sup> | [bandit](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | 10.3 | | Python | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.9 | -| React<sup>2</sup> | [ESLint react plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 12.5 | | React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | | Ruby | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 13.9 | | Ruby on Rails | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 10.3 | | Scala (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 16.0 | | Scala<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | | Swift (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | -| TypeScript<sup>2</sup> | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | | TypeScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | 1. The SpotBugs-based analyzer supports [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), [Grails](https://grails.org/), and the [Maven wrapper](https://github.com/takari/maven-wrapper). However, SpotBugs has [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/350801) when used against [Ant](https://ant.apache.org/)-based projects. We recommend using the Semgrep-based analyzer for Ant-based Java or Scala projects. -1. These analyzers reached [End of Support](https://about.gitlab.com/handbook/product/gitlab-the-product/#end-of-support) status [in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554). -1. Security Code Scan reached [End of Support](https://about.gitlab.com/handbook/product/gitlab-the-product/#end-of-support) status [in GitLab 16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/390416). + +## End of supported analyzers + +GitLab has reached [End of Support](https://about.gitlab.com/handbook/product/gitlab-the-product/#end-of-support) for the below analyzers. These analyzers have been replaced by the Semgrep-based analyzer. + +| Language / framework | [Analyzer](analyzers.md) used for scanning | Minimum supported GitLab version | End Of Support GitLab version | +|------------------------------|--------------------------------------------------------------------------------------------------------------| --------------------------------- | ------------------------------------------------------------- | +| .NET Core | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 11.0 | [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/390416) | +| .NET Framework | [Security Code Scan](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) | 13.0 | [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/390416) | +| Go | [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) | 10.7 | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | +| Java | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (SBT) | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | +| Python | [bandit](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) | 10.3 | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | +| React | [ESLint react plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 12.5 | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | +| JavaScript | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.8 | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | +| TypeScript | [ESLint security plugin](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) | 11.9, with ESLint in 13.2 | [15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/352554) | ## Multi-project support @@ -151,8 +156,9 @@ Advanced vulnerability tracking is available in a subset of the [supported langu - C++, in the Flawfinder analyzer only - C#, in the Semgrep-based analyzer only - Go, in the Semgrep-based analyzer only -- Java, in the Semgrep-based and mobsf analyzers +- Java, in the mobsf, Semgrep-based and SpotBugs analyzers - JavaScript, in the Semgrep-based and NodeJS-Scan analyzers +- PHP, in the phpcs-security-audit analyzer - Python, in the Semgrep-based analyzer only - Ruby, in the Brakeman-based analyzer @@ -278,7 +284,7 @@ successfully, and an error may occur. To enable and configure SAST with customizations: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. If the project does not have a `.gitlab-ci.yml` file, select **Enable SAST** in the Static Application Security Testing (SAST) row, otherwise select **Configure SAST**. @@ -300,7 +306,7 @@ successfully, and an error may occur. To enable and configure SAST with default settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the SAST section, select **Configure with a merge request**. 1. Review and merge the merge request to enable SAST. @@ -619,7 +625,7 @@ For information, see [Download job artifacts](../../../ci/jobs/job_artifacts.md# For details of the report file's schema, see [SAST report file schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). -For an example SAST report file, see [`gl-secret-detection-report.json`](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/qa/expect/secrets/gl-secret-detection-report.json) example. +For an example SAST report file, see [`gl-sast-report.json`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/qa/expect/js/default/gl-sast-report.json) example. ## Running SAST in an offline environment diff --git a/doc/user/application_security/sast/rules.md b/doc/user/application_security/sast/rules.md new file mode 100644 index 00000000000..4e7a6387f9b --- /dev/null +++ b/doc/user/application_security/sast/rules.md @@ -0,0 +1,101 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# SAST rules **(FREE)** + +GitLab SAST uses a set of [analyzers](analyzers.md) to scan code for potential vulnerabilities. +Each analyzer processes the code then uses rules to find possible weaknesses in source code. +The rules determine what types of weaknesses the analyzer reports. + +## Source of rules + +### Semgrep-based analyzer + +GitLab creates, maintains, and supports the rules that are used in the Semgrep-based GitLab SAST analyzer. +This analyzer scans [many languages](index.md#supported-languages-and-frameworks) in a single CI/CD pipeline job. +It combines: + +- the Semgrep open-source engine. +- GitLab-managed detection rules. +- GitLab proprietary technology for [vulnerability tracking](index.md#advanced-vulnerability-tracking) and [false positive detection](index.md#false-positive-detection). + +### Other analyzers + +GitLab SAST uses other analyzers to scan the remaining [supported languages](index.md#supported-languages-and-frameworks). +The rules for these scans are defined in the upstream projects for each scanner. + +## How rule updates are released + +GitLab updates rules regularly based on customer feedback and internal research. +Rules are released as part of the container image for each analyzer. +You automatically receive updated analyzers and rules unless you [manually pin analyzers to a specific version](index.md#pinning-to-minor-image-version). + +Analyzers and their rules are updated [at least monthly](../index.md#vulnerability-scanner-maintenance) if relevant updates are available. + +The GitLab ruleset for the Semgrep-based analyzer is managed in [the GitLab-managed open-source `sast-rules` project](https://gitlab.com/gitlab-org/security-products/sast-rules). +When rules are updated, they're released as part of the [Semgrep-based analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep)'s container image. + +## Configure rules in your projects + +You should use the default SAST rules unless you have a specific reason to make a change. +The default ruleset is designed to be relevant to most projects. + +However, you can [customize which rules are used](#apply-local-rule-preferences) or [control how rule changes are rolled out](#coordinate-rule-rollouts) if needed. + +### Apply local rule preferences + +You may want to customize the rules used in SAST scans because: + +- Your organization has assigned priorities to specific vulnerability classes, such as choosing to address Cross-Site Scripting (XSS) or SQL Injection before other classes of vulnerabilities. +- You believe that a specific rule is a false positive result or isn't relevant in the context of your codebase. + +To change which rules are used to scan your projects, adjust their severity, or apply other preferences, see [Customize rulesets](customize_rulesets.md). +If your customization would benefit other users, consider [reporting a problem to GitLab](#report-a-problem-with-a-gitlab-sast-rule). + +### Coordinate rule rollouts + +To control the rollout of rule changes, you can [pin SAST analyzers to a specific version](index.md#pinning-to-minor-image-version). + +If you want to make these changes at the same time across multiple projects, consider setting the variables in: + +- [Group-level CI/CD variables](../../../ci/variables/index.md#for-a-group). +- Custom CI/CD variables in a [Scan Execution Policy](../policies/scan-execution-policies.md). + +## Report a problem with a GitLab SAST rule +<!-- This title is intended to match common search queries users might make. --> + +GitLab welcomes contributions to the rulesets used in SAST. +Contributions might address: + +- False positive results, where the potential vulnerability is incorrect. +- False negative results, where SAST did not report a potential vulnerability that truly exists. +- The name, severity rating, description, guidance, or other explanatory content for a rule. + +If you believe a detection rule could be improved for all users, consider: + +- Submitting a merge request to [the `sast-rules` repository](https://gitlab.com/gitlab-org/security-products/sast-rules). See the [contribution instructions](https://gitlab.com/gitlab-org/security-products/sast-rules#contributing) for details. +- Filing an issue in [the `gitlab-org/gitlab` issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/). + - Post a comment that says `@gitlab-bot label ~"group::static analysis" ~"Category:SAST"` so your issue lands in the correct triage workflow. + +## Important rule changes + +GitLab updates SAST rules [regularly](#how-rule-updates-are-released). +This section highlights the most important changes. +More details are available in release announcements and in the CHANGELOG links provided. + +### Rule changes in the Semgrep-based analyzer + +Key changes to the GitLab-managed ruleset for Semgrep-based scanning include: + +- Beginning in GitLab 16.3, the GitLab Static Analysis and Vulnerability Research teams are working to remove rules that tend to produce too many false positive results or not enough actionable true positive results. Existing findings from these removed rules are [automatically resolved](index.md#automatic-vulnerability-resolution); they no longer appear in the [Security Dashboard](../security_dashboard/index.md#view-vulnerabilities-over-time-for-a-project) or in the default view of the [Vulnerability Report](../vulnerability_report/index.md). This work is tracked in [epic 10907](https://gitlab.com/groups/gitlab-org/-/epics/10907). +- In GitLab 16.0 through 16.2, the GitLab Vulnerability Research team updated the guidance that's included in each result. +- In GitLab 15.10, the `detect-object-injection` rule was [removed by default](https://gitlab.com/gitlab-org/gitlab/-/issues/373920) and its findings were [automatically resolved](index.md#automatic-vulnerability-resolution). + +For more details, see the [CHANGELOG for `sast-rules`](https://gitlab.com/gitlab-org/security-products/sast-rules/-/blob/main/CHANGELOG.md). + +### Rule changes in other analyzers + +See the CHANGELOG file for each [analyzer](analyzers.md) for details of the changes, including new or updated rules, included in each version. diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 10e8356de16..a10c029bec6 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -27,7 +27,7 @@ You can run scans and view [Secret Detection JSON report artifacts](../../../ci/ With GitLab Ultimate, Secret Detection results are also processed so you can: -- See them in the [merge request widget](../index.md#view-security-scan-information-in-merge-requests), [pipeline security report](../vulnerability_report/pipeline.md), and [Vulnerability Report](../vulnerability_report/index.md). +- See them in the [merge request widget](../index.md#merge-request), [pipeline security report](../vulnerability_report/pipeline.md), and [vulnerability report](../vulnerability_report/index.md) UIs. - Use them in approval workflows. - Review them in the security dashboard. - [Automatically respond](automatic_response.md) to leaks in public repositories. @@ -155,7 +155,7 @@ To enable Secret Detection, either: This method requires you to manually edit the existing `.gitlab-ci.yml` file. Use this method if your GitLab CI/CD configuration file is complex. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline editor**. 1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file: @@ -188,7 +188,7 @@ error may occur. In that case, use the [manual](#edit-the-gitlab-ciyml-file-manu To enable Secret Detection: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. In the **Secret Detection** row, select **Configure with a merge request**. 1. Optional. Complete the fields. @@ -338,12 +338,14 @@ To enable full history Secret Detection, set the variable `SECRET_DETECTION_HIST ## Custom rulesets **(ULTIMATE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for passthrough chains. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for passthrough chains. > Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in > GitLab 14.8. -You can customize the default Secret Detection rules provided with GitLab. +You can customize which [secrets are reported in the GitLab UI](#secret-detection). +However, the `secret_detection` job logs always include the number +of secrets detected by the default Secret Detection rules. The following customization options can be used separately, or in combination: @@ -510,7 +512,7 @@ optional authentication, and optional Git SHA. The variable uses the following f ``` NOTE: -Loading local project configuration takes precedence over `SECRET_DETECTION_RULESET_GIT_REFERENCE` values. +A local `.gitlab/secret-detection-ruleset.toml` file in the project takes precedence over `SECRET_DETECTION_RULESET_GIT_REFERENCE`. The following example includes the Secret Detection template in a project to be scanned and specifies the `SECRET_DETECTION_RULESET_GIT_REFERENCE` variable for referencing a separate project configuration. diff --git a/doc/user/application_security/secure_your_application.md b/doc/user/application_security/secure_your_application.md index 0b028f6936d..f4fb8c49277 100644 --- a/doc/user/application_security/secure_your_application.md +++ b/doc/user/application_security/secure_your_application.md @@ -27,3 +27,4 @@ GitLab can check your applications for security vulnerabilities. - [Policies](policies/index.md) - [Security scanner integration](../../development/integrations/secure.md) - [Secure and Govern Terminology](terminology/index.md) +- [GitLab Advisory Database](gitlab_advisory_database/index.md) diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index b6d95e53227..53a6dfe6d0a 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -66,7 +66,7 @@ Project Security Dashboards show statistics for all vulnerabilities with a curre To view total number of vulnerabilities over time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security dashboard**. 1. Filter and search for what you need. - To filter the chart by severity, select the legend name. @@ -79,7 +79,7 @@ To view total number of vulnerabilities over time: To download an SVG image of the vulnerabilities chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security dashboard**. 1. Select **Save chart as an image** (**{download}**). @@ -90,7 +90,7 @@ branches of projects in a group and its subgroups. To view vulnerabilities over time for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Security > Security dashboard**. 1. Hover over the chart to get more details about vulnerabilities. - You can display the vulnerability trends over a 30, 60, or 90-day time frame (the default is 90 days). @@ -104,7 +104,7 @@ Use the group Security Dashboard to view the security status of projects. To view project security status for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Secure > Security dashboard**. Each project is assigned a letter [grade](#project-vulnerability-grades) according to the highest-severity open vulnerability. @@ -142,7 +142,7 @@ The Security Center includes: To view the Security Center: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Security > Security dashboard**. @@ -150,7 +150,7 @@ To view the Security Center: To add projects to the Security Center: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Expand **Security**. 1. Select **Settings**. diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index fa0027754ad..34c57292767 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -24,10 +24,10 @@ change its status to **Resolved**. This ensures that if it is accidentally reint merge, it is reported again as a new record. To change the status of multiple vulnerabilities, use the Vulnerability Report's [Activity filter](../vulnerability_report/index.md#activity-filter). -## Explaining a vulnerability (Beta) **(ULTIMATE SAAS)** +## Explaining a vulnerability **(ULTIMATE SAAS BETA)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10368) in GitLab 16.0 as an [Experiment](../../../policy/experiment-beta-support.md#experiment) on GitLab.com. -> - Promoted to [Beta](../../../policy/experiment-beta-support.md#beta) status in 16.2. +> - Promoted to [Beta](../../../policy/experiment-beta-support.md#beta) status in GitLab 16.2. GitLab can help you with a vulnerability by using a large language model to: @@ -37,8 +37,8 @@ GitLab can help you with a vulnerability by using a large language model to: ### Explain a vulnerability -Use the explain this vulnerability feature to better understand a vulnerability and its possible -mitigation. +Explain a vulnerability with GitLab Duo Vulnerability summary. Use the explanation to better +understand a vulnerability and its possible mitigation. Prerequisites: @@ -46,11 +46,11 @@ Prerequisites: - You must be a member of the project. - The vulnerability must be a SAST finding. -Learn more about [how to enable all AI features](../../ai_features.md#enable-aiml-features). +Learn more about [how to enable all GitLab Duo features](../../ai_features.md#enable-aiml-features). To explain the vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Security and Compliance > Vulnerability report**. 1. In the **Tool** dropdown list, select **SAST**. 1. Select the SAST vulnerability you want explained. @@ -108,7 +108,7 @@ When dismissing a vulnerability, one of the following reasons must be chosen to To change a vulnerability's status from its Vulnerability Page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Status** dropdown list select a status, then select **Change status**. @@ -130,15 +130,11 @@ You can create either: - [A GitLab issue](#create-a-gitlab-issue-for-a-vulnerability) (default). - [A Jira issue](#create-a-jira-issue-for-a-vulnerability). -Creating a Jira issue requires that -[Jira integration](../../../integration/jira/index.md) is enabled on the project. Note -that when Jira integration is enabled, the GitLab issue feature is not available. - ### Create a GitLab issue for a vulnerability To create a GitLab issue for a vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. Select **Create issue**. @@ -157,7 +153,7 @@ Prerequisites: To create a Jira issue for a vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. Select **Create Jira issue**. @@ -169,28 +165,21 @@ fields are pre-populated from the vulnerability's details. Unlike GitLab issues, the status of whether a Jira issue is open or closed does not display in the GitLab user interface. -## Linking a vulnerability to issues - -NOTE: -If Jira issue support is enabled, GitLab issues are disabled so this feature is not available. +## Linking a vulnerability to GitLab and Jira issues -You can link a vulnerability to one or more existing GitLab issues. Adding a link helps track -the issue that resolves or mitigates a vulnerability. +You can link a vulnerability to one or more existing [GitLab](#create-a-gitlab-issue-for-a-vulnerability) +or [Jira](#create-a-jira-issue-for-a-vulnerability) issues. Only one linking feature is available at the same time. +Adding a link helps track the issue that resolves or mitigates a vulnerability. -Issues linked to a vulnerability are shown in the Vulnerability Report and the vulnerability's page. +### Link a vulnerability to existing GitLab issues -Be aware of the following conditions between a vulnerability and a linked issue: +Prerequisite: -- The vulnerability page shows related issues, but the issue page doesn't show the vulnerability - it's related to. -- An issue can only be related to one vulnerability at a time. -- Issues can be linked across groups and projects. +- [Jira issue integration](../../../integration/jira/configure.md) must not be enabled. -## Link a vulnerability to existing issues +To link a vulnerability to existing GitLab issues: -To link a vulnerability to existing issues: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. In the **Linked issues** section, select the plus icon (**{plus}**). @@ -199,9 +188,43 @@ To link a vulnerability to existing issues: - Enter the issue's ID (prefixed with a hash `#`). 1. Select **Add**. -The selected issues are added to the **Linked issues** section, and the linked issues counter is +The selected GitLab issues are added to the **Linked items** section, and the linked issues counter is +updated. + +GitLab issues linked to a vulnerability are shown in the Vulnerability Report and the vulnerability's page. + +Be aware of the following conditions between a vulnerability and a linked GitLab issue: + +- The vulnerability page shows related issues, but the issue page doesn't show the vulnerability + it's related to. +- An issue can only be related to one vulnerability at a time. +- Issues can be linked across groups and projects. + +### Link a vulnerability to existing Jira issues + +Prerequisite: + +- [Jira issue integration](../../../integration/jira/configure.md) must be enabled, with option **Enable Jira issue creation from vulnerabilities** also enabled. + +To link a vulnerability to existing Jira issues, add the following line to the Jira issue's description: + +```plaintext +/-/security/vulnerabilities/<id> +``` + +`<id>` is any [vulnerability ID](../../../api/vulnerabilities.md#single-vulnerability). +You can add several lines with different IDs to one description. + +Jira issues with appropriate description are added to the **Related Jira issues** section, and the linked issues counter is updated. +Jira issues linked to a vulnerability are shown only on the vulnerability page. + +Be aware of the following conditions between a vulnerability and a linked Jira issue: + +- The vulnerability page and the issue page show the vulnerability they are related to. +- An issue can be related to one or more vulnerabilities at the same time. + ## Resolve a vulnerability For some vulnerabilities a solution is already known. In those instances, a vulnerability's page @@ -225,7 +248,7 @@ To resolve a vulnerability, you can either: To resolve the vulnerability with a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Resolve with merge request** dropdown list, select **Resolve with merge request**. @@ -237,7 +260,7 @@ Process the merge request according to your standard workflow. To manually apply the patch that GitLab generated for a vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Resolve with merge request** dropdown list, select **Download patch to resolve**. @@ -260,7 +283,7 @@ Security training helps your developers learn how to fix vulnerabilities. Develo To enable security training for vulnerabilities in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Security configuration**. 1. On the tab bar, select **Vulnerability Management**. 1. To enable a security training provider, turn on the toggle. @@ -280,7 +303,7 @@ Vulnerabilities with a CWE are most likely to return a training result. To view the security training for a vulnerability: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select the vulnerability for which you want to view security training. 1. Select **View training**. diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 46ce3173ee7..3ec1151b1d6 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -40,8 +40,6 @@ status of a Jira issue is not shown in the GitLab UI. ## Project-level Vulnerability Report -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6165) in GitLab 11.1. - At the project level, the Vulnerability Report also contains: - A time stamp showing when it was updated, including a link to the latest pipeline. @@ -55,7 +53,7 @@ this page displays the vulnerabilities that originate from the selected project. To view the project-level vulnerability report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. ## Vulnerability Report actions @@ -129,8 +127,6 @@ The content of the Project filter depends on the current level: ### Activity filter -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259255) in GitLab 13.9 - The Activity filter behaves differently from the other filters. The selected values form mutually exclusive sets to allow for precisely locating the desired vulnerability records. Additionally, not all options can be selected in combination. @@ -150,8 +146,6 @@ To view more details of a vulnerability, select the vulnerability's **Descriptio ## View vulnerable source location -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267509) in GitLab 13.10. - Some security scanners output the filename and line number of a potential vulnerability. When that information is available, the vulnerability's details include a link to the relevant file, in the default branch. @@ -160,8 +154,7 @@ To view the relevant file, select the filename in the vulnerability's details. ## Change status of vulnerabilities -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292636) in GitLab 13.10, all statuses became selectable. -> - Providing a comment and dismissal reason [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/408366) in GitLab 16.0. +> Providing a comment and dismissal reason [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/408366) in GitLab 16.0. From the Vulnerability Report you can change the status of one or more vulnerabilities. @@ -184,9 +177,6 @@ To sort vulnerabilities by the date each vulnerability was detected, select the ## Export vulnerability details -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213014) in the Security Center (previously known as the Instance Security Dashboard) and project-level Vulnerability Report (previously known as the Project Security Dashboard) in GitLab 13.0. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/213013) to the group-level Vulnerability Report in GitLab 13.1. - You can export details of the vulnerabilities listed in the Vulnerability Report. The export format is CSV (comma separated values). All vulnerabilities are included because filters do not apply to the export. @@ -229,8 +219,6 @@ thousands of vulnerabilities. Do not close the page until the download finishes. ## Dismiss a vulnerability -> The option of adding a dismissal reason was introduced in GitLab 12.0. - When you evaluate a vulnerability and decide it requires no more action, you can mark it as **Dismissed**. Dismissed vulnerabilities do not appear in the merge request security widget @@ -260,7 +248,7 @@ To undo this action, select a different status from the same menu. To add a new vulnerability finding from your project level Vulnerability Report page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Vulnerability report**. 1. Select **Submit vulnerability**. 1. Complete the fields and submit the form. diff --git a/doc/user/application_security/vulnerability_report/pipeline.md b/doc/user/application_security/vulnerability_report/pipeline.md index 7a414e9a4ae..ef66925b9c9 100644 --- a/doc/user/application_security/vulnerability_report/pipeline.md +++ b/doc/user/application_security/vulnerability_report/pipeline.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To view vulnerabilities in a pipeline: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipelines**. 1. From the list, select the pipeline you want to check for vulnerabilities. 1. Select the **Security** tab. @@ -25,9 +25,11 @@ For example, if a pipeline contains DAST and SAST jobs, but the DAST job fails b [exit code](../../../development/integrations/secure.md#exit-code), the report doesn't show DAST results. The pipeline vulnerability report only shows results contained in the security report artifacts. This report differs from -the [Vulnerability Report](index.md), which contains cumulative results of all successful jobs, and from the merge request -[security widget](../index.md#view-security-scan-information-in-merge-requests), which combines the branch results with -cumulative results. +the [vulnerability report](index.md), which contains cumulative results of all successful jobs, and from the merge request +[security widget](../index.md#merge-request), which contains new vulnerability findings that don't already exist on the default branch. + +NOTE: +If a new advisory is added to our advisory database and the last pipeline for the default branch is stale, the resulting vulnerability may appear in the MR widget as "New" when it is already in the default branch. This will be resolved by [Continuous Vulnerability Scans](https://gitlab.com/groups/gitlab-org/-/epics/7886). The pipeline vulnerability report only displays after the pipeline is complete. If the pipeline has a [blocking manual job](../../../ci/jobs/job_control.md#types-of-manual-jobs), the pipeline waits for the manual job and the vulnerabilities cannot be displayed if the blocking manual job did not run. diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index a7c58b4d2a8..26cccd7584e 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -47,7 +47,7 @@ To remove an emoji reaction, select the emoji again. > - [Introduced for GraphQL API](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 13.6 [with a flag](../administration/feature_flags.md) named `custom_emoji`. Disabled by default. > - Enabled on GitLab.com in GitLab 14.0. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/333095) UI to add emoji in GitLab 16.2. +> - UI to add emoji [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333095) in GitLab 16.2. Custom emoji show in the emoji picker everywhere you can react with emoji. To add an emoji reaction to a comment or description: diff --git a/doc/user/clusters/agent/ci_cd_workflow.md b/doc/user/clusters/agent/ci_cd_workflow.md index 260263632c5..a1281d3d75c 100644 --- a/doc/user/clusters/agent/ci_cd_workflow.md +++ b/doc/user/clusters/agent/ci_cd_workflow.md @@ -15,9 +15,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - The ability to switch between certificate-based clusters and agents was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335089) in GitLab 14.9. The certificate-based cluster context is always called `gitlab-deploy`. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80508) from _CI/CD tunnel_ to _CI/CD workflow_ in GitLab 14.9. -You can use a GitLab CI/CD workflow to safely deploy to and update your Kubernetes clusters. +You can use GitLab CI/CD to safely connect, deploy, and update your Kubernetes clusters. -To do so, you must first [install an agent in your cluster](install/index.md). When done, you have a Kubernetes context and can +To do so, [install an agent in your cluster](install/index.md). When done, you have a Kubernetes context and can run Kubernetes API commands in your GitLab CI/CD pipeline. To ensure access to your cluster is safe: @@ -25,11 +25,11 @@ To ensure access to your cluster is safe: - Each agent has a separate context (`kubecontext`). - Only the project where the agent is configured, and any additional projects you authorize, can access the agent in your cluster. -The CI/CD workflow requires runners to be registered with GitLab, but these runners do not have to be in the cluster where the agent is. +To use GitLab CI/CD to interact with your cluster, runners must be registered with GitLab. However, these runners do not have to be in the cluster where the agent is. -## GitLab CI/CD workflow steps +## Use GitLab CI/CD with your cluster -To update a Kubernetes cluster by using GitLab CI/CD, complete the following steps. +To update a Kubernetes cluster with GitLab CI/CD: 1. Ensure you have a working Kubernetes cluster and the manifests are in a GitLab project. 1. In the same GitLab project, [register and install the GitLab agent](install/index.md). @@ -64,7 +64,7 @@ Authorization configuration can take one or two minutes to propagate. To authorize the agent to access the GitLab project where you keep Kubernetes manifests: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). +1. On the left sidebar, select **Search or go to** and find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). 1. Edit the `config.yaml` file. Under the `ci_access` keyword, add the `projects` attribute. 1. For the `id`, add the path to the project. Do not wrap the path in quotation marks. @@ -89,7 +89,7 @@ Choose the context to run `kubectl` commands from your CI/CD scripts. To authorize the agent to access all of the GitLab projects in a group or subgroup: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). +1. On the left sidebar, select **Search or go to** and find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). 1. Edit the `config.yaml` file. Under the `ci_access` keyword, add the `groups` attribute. 1. For the `id`, add the path: diff --git a/doc/user/clusters/agent/gitops/flux_tutorial.md b/doc/user/clusters/agent/gitops/flux_tutorial.md index f2d65b900f0..27724a95291 100644 --- a/doc/user/clusters/agent/gitops/flux_tutorial.md +++ b/doc/user/clusters/agent/gitops/flux_tutorial.md @@ -83,7 +83,7 @@ You must register `agentk` before you install it in your cluster. To register `agentk`: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. If you have an [agent configuration file](../install/index.md#create-an-agent-configuration-file), it must be in this project. Your cluster manifest files should also be in this project. 1. Select **Operate > Kubernetes clusters**. @@ -123,9 +123,7 @@ To install `agentk`: name: gitlab-agent-token type: Opaque stringData: - values.yaml: |- - config: - token: "<your-token-here>" + token: "<your-token-here>" ``` 1. Apply `secret.yaml` to your cluster: @@ -173,12 +171,11 @@ To install `agentk`: values: config: kasAddress: "wss://kas.gitlab.com" - valuesFrom: - - kind: Secret - name: gitlab-agent-token - valuesKey: values.yaml + secretName: gitlab-agent-token ``` + The Helm release uses the secret from the previous step. + 1. To verify that `agentk` is installed and running in the cluster, run the following command: ```shell diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index 75571d1a4f5..d620a9f658c 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -43,7 +43,7 @@ To install the agent in your cluster: For configuration settings, the agent uses a YAML file in the GitLab project. You must create this file if: - You use [a GitOps workflow](../gitops/agent.md#gitops-workflow-steps). -- You use [a GitLab CI/CD workflow](../ci_cd_workflow.md#gitlab-cicd-workflow-steps) and want to authorize a different project to use the agent. +- You use [a GitLab CI/CD workflow](../ci_cd_workflow.md#use-gitlab-cicd-with-your-cluster) and want to authorize a different project to use the agent. - You [allow specific project or group members to access Kubernetes](../user_access.md). To create an agent configuration file: @@ -76,11 +76,11 @@ In GitLab 14.10, a [flag](../../../../administration/feature_flags.md) named `ce Prerequisites: - For a [GitLab CI/CD workflow](../ci_cd_workflow.md), ensure that - [GitLab CI/CD is enabled](../../../../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). + [GitLab CI/CD is not disabled](../../../../ci/enable_or_disable_ci.md#disable-cicd-in-a-project). You must register an agent before you can install the agent in your cluster. To register an agent: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. If you have an [agent configuration file](#create-an-agent-configuration-file), it must be in this project. Your cluster manifest files should also be in this project. 1. Select **Operate > Kubernetes clusters**. @@ -114,22 +114,26 @@ To connect to multiple clusters, you must configure, register, and install an ag #### Install the agent with Helm +WARNING: +For simplicity, the default Helm chart configuration sets up a service account for the agent with `cluster-admin` rights. You should not use this on production systems. To deploy to a production system, follow the instructions in [Customize the Helm installation](#customize-the-helm-installation) to create a service account with the minimum permissions required for your deployment and specify that during installation. + To install the agent on your cluster using Helm: 1. [Install Helm](https://helm.sh/docs/intro/install/). 1. In your computer, open a terminal and [connect to your cluster](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/). 1. Run the command you copied when you [registered your agent with GitLab](#register-the-agent-with-gitlab). - -Optionally, you can [customize the Helm installation](#customize-the-helm-installation). If you install the agent on a production system, you should customize the Helm installation to skip creating the service account. +1. Optional. [Customize the Helm installation](#customize-the-helm-installation). + If you install the agent on a production system, you should customize the Helm installation to restrict the permissions of the service account. See [How to deploy the GitLab Agent for Kubernetes with limited permissions](https://about.gitlab.com/blog/2021/09/10/setting-up-the-k-agent/). ##### Customize the Helm installation By default, the Helm installation command generated by GitLab: - Creates a namespace `gitlab-agent` for the deployment (`--namespace gitlab-agent`). You can skip creating the namespace by omitting the `--create-namespace` flag. -- Sets up a service account for the agent with `cluster-admin` rights. You can: +- Sets up a service account for the agent and assigns it the `cluster-admin` role. You can: - Skip creating the service account by adding `--set serviceAccount.create=false` to the `helm install` command. In this case, you must set `serviceAccount.name` to a pre-existing service account. - - Skip creating the RBAC permissions by adding `--set rbac.create=false` to the `helm install` command. In this case, you must bring your own RBAC permissions for the agent. Otherwise, it has no permissions at all. + - Customise the role assigned to the service account by adding `--set rbac.useExistingRole <your role name>` to the `helm install` command. In this case, you should have a pre-created role with restricted permissions that can be used by the service account. + - Skip role assignment altogether by adding `--set rbac.create=false` to your `helm install` command. In this case, you must create `ClusterRoleBinding` manually. - Creates a `Secret` resource for the agent's access token. To instead bring your own secret with a token, omit the token (`--set token=...`) and instead use `--set config.secretName=<your secret name>`. - Creates a `Deployment` resource for the `agentk` pod. diff --git a/doc/user/clusters/agent/user_access.md b/doc/user/clusters/agent/user_access.md index a5989d823f6..21dc249b1d1 100644 --- a/doc/user/clusters/agent/user_access.md +++ b/doc/user/clusters/agent/user_access.md @@ -4,7 +4,7 @@ group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Grant users Kubernetes access (Beta) **(FREE ALL)** +# Grant users Kubernetes access **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390769) in GitLab 16.1, with [flags](../../../administration/feature_flags.md) named `environment_settings_to_graphql`, `kas_user_access`, `kas_user_access_project`, and `expose_authorized_cluster_agents`. This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). > - Feature flag `environment_settings_to_graphql` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124177) in GitLab 16.2. @@ -141,6 +141,60 @@ subjects: kind: Group ``` +## Access a cluster with the Kubernetes API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131144) in GitLab 16.4. + +You can configure an agent to allow GitLab users to access a cluster with the Kubernetes API. + +Prerequisite: + +- You have an agent configured with the `user_access` entry. + +To grant Kubernetes API access: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Operate > Kubernetes clusters** and retrieve the numerical ID of the agent you want to access. You need the ID to construct the full API token. +1. Create a [personal access token](../../profile/personal_access_tokens.md) with the `k8s_proxy` scope. You need the access token to construct the full API token. +1. Construct `kube config` entries to access the cluster: + 1. Make sure that the proper `kube config` is selected. + For example, you can set the `KUBECONFIG` environment variable. + 1. Add the GitLab KAS proxy cluster to the `kube config`: + + ```shell + kubectl config set-cluster <cluster_name> --server "https://kas.gitlab.com/k8s-proxy" + ``` + + The `server` argument points to the KAS address of your GitLab instance. + On GitLab.com, this is `https://kas.gitlab.com/k8s-proxy`. + You can get the KAS address of your instance when you register an agent. + + 1. Use your numerical agent ID and personal access token to construct an API token: + + ```shell + kubectl config set-credentials <gitlab_user> --token "pat:<agent-id>:<token>" + ``` + + 1. Add the context to combine the cluster and the user: + + ```shell + kubectl config set-context <gitlab_agent> --cluster <cluster_name> --user <gitlab_user> + ``` + + 1. Activate the new context: + + ```shell + kubectl config use-context <gitlab_agent> + ``` + +1. Check that the configuration works: + + ```shell + kubectl get nodes + ``` + +The configured user can access your cluster with the Kubernetes API. + ## Related topics - [Architectural blueprint](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kubernetes_user_access.md) diff --git a/doc/user/clusters/agent/vulnerabilities.md b/doc/user/clusters/agent/vulnerabilities.md index a967ec9ea24..a2dc50e43d7 100644 --- a/doc/user/clusters/agent/vulnerabilities.md +++ b/doc/user/clusters/agent/vulnerabilities.md @@ -9,9 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6346) in GitLab 14.8. > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/368828) the starboard directive in GitLab 15.4. The starboard directive is scheduled for removal in GitLab 16.0. -To view cluster vulnerabilities, you can view the [vulnerability report](../../application_security/vulnerabilities/index.md). -You can also configure your agent so the vulnerabilities are displayed with other agent information in GitLab. - ## Enable operational container scanning You can use operational container scanning to scan container images in your cluster for security vulnerabilities. You @@ -24,7 +21,7 @@ If both `agent config` and `scan execution policies` are configured, the configu ### Enable via agent configuration To enable scanning of all images within your Kubernetes cluster via the agent configuration, add a `container_scanning` configuration block to your agent -configuration with a `cadence` field containing a [CRON expression](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) for when the scans are run. +configuration with a `cadence` field containing a [CRON expression](https://en.wikipedia.org/wiki/Cron) for when the scans are run. ```yaml container_scanning: @@ -129,7 +126,7 @@ Resource requirements can only be set up using the agent configuration. If you e To view vulnerability information in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains the agent configuration file. +1. On the left sidebar, select **Search or go to** and find the project that contains the agent configuration file. 1. Select **Operate > Kubernetes clusters**. 1. Select the **Agent** tab. 1. Select an agent to view the cluster vulnerabilities. @@ -140,3 +137,9 @@ This information can also be found under [operational vulnerabilities](../../../ NOTE: You must have at least the Developer role. + +## Scanning private images + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415451) in GitLab 16.4. + +To scan private images, the scanner relies on the image pull secrets (direct references and from the service account) to pull the image. diff --git a/doc/user/clusters/agent/work_with_agent.md b/doc/user/clusters/agent/work_with_agent.md index 08e1021f886..70abbebaaad 100644 --- a/doc/user/clusters/agent/work_with_agent.md +++ b/doc/user/clusters/agent/work_with_agent.md @@ -4,9 +4,9 @@ group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Working with the agent for Kubernetes **(FREE ALL)** +# Managing the agent for Kubernetes instances **(FREE ALL)** -Use the following tasks when working with the agent for Kubernetes. +Use the following tasks when you work with the agent for Kubernetes. ## View your agents @@ -18,7 +18,7 @@ Prerequisite: To view the list of agents: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains your agent configuration file. +1. On the left sidebar, select **Search or go to** and find the project that contains your agent configuration file. You cannot view registered agents from a project that does not contain the agent configuration file. 1. Select **Operate > Kubernetes clusters**. 1. Select **Agent** tab to view clusters connected to GitLab through the agent. @@ -40,7 +40,7 @@ is shared with a project, it automatically appears in the project agent tab. To view the list of shared agents: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Kubernetes clusters**. 1. Select the **Agent** tab. @@ -54,7 +54,7 @@ The activity logs help you to identify problems and get the information you need for troubleshooting. You can see events from a week before the current date. To view an agent's activity: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains your agent configuration file. +1. On the left sidebar, select **Search or go to** and find the project that contains your agent configuration file. 1. Select **Operate > Kubernetes clusters**. 1. Select the agent you want to see activity for. @@ -115,7 +115,7 @@ An agent can have only two active tokens at one time. To reset the agent token without downtime: 1. Create a new token: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Kubernetes clusters**. 1. Select the agent you want to create a token for. 1. On the **Access tokens** tab, select **Create token**. @@ -137,7 +137,7 @@ clean up those resources manually. To remove an agent from the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project that contains the agent configuration file. +1. On the left sidebar, select **Search or go to** and find the project that contains the agent configuration file. 1. Select **Operate > Kubernetes clusters**. 1. In the table, in the row for your agent, in the **Options** column, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Delete agent**. diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md deleted file mode 100644 index a155dcf4a3c..00000000000 --- a/doc/user/clusters/cost_management.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Deploy -group: Environments -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Cluster cost management (removed) **(ULTIMATE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md deleted file mode 100644 index 0f389b94a33..00000000000 --- a/doc/user/clusters/integrations.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Deploy -group: Environments -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Cluster integrations (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index 3341074f54d..3c5c90abdae 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -62,7 +62,7 @@ To associate a cluster management project with your cluster: - [Group-level cluster](../group/clusters/index.md), go to your group's **Kubernetes** page. - [Instance-level cluster](../instance/clusters/index.md): - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Kubernetes**. 1. Expand **Advanced settings**. diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index f4f42f4dc27..a40fc5a262e 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -86,9 +86,7 @@ cluster applications with [Helm v3](https://helm.sh/). This file has a list of paths to other Helm files for each app. They're all commented out by default, so you must uncomment the paths for the apps that you would like to use in your cluster. -By default, each `helmfile.yaml` in these sub-paths has the attribute `installed: true`. This means that every time -the pipeline runs, Helmfile tries to either install or update your apps according to the current state of your -cluster and Helm releases. If you change this attribute to `installed: false`, Helmfile tries try to uninstall this app +By default, each `helmfile.yaml` in these sub-paths has the attribute `installed: true`. This means that, depending on the state of your cluster and Helm releases, Helmfile attempts to install or update apps every time the pipeline runs. If you change this attribute to `installed: false`, Helmfile tries to uninstall this app from your cluster. [Read more](https://helmfile.readthedocs.io/en/latest/) about how Helmfile works. ### Built-in applications @@ -101,7 +99,6 @@ The [built-in supported applications](https://gitlab.com/gitlab-org/project-temp - [Cert-manager](../infrastructure/clusters/manage/management_project_applications/certmanager.md) - [GitLab Runner](../infrastructure/clusters/manage/management_project_applications/runner.md) - [Ingress](../infrastructure/clusters/manage/management_project_applications/ingress.md) -- [Prometheus](../../operations/metrics/index.md) - [Vault](../infrastructure/clusters/manage/management_project_applications/vault.md) Each application has an `applications/{app}/values.yaml` file. diff --git a/doc/user/compliance/compliance_center/index.md b/doc/user/compliance/compliance_center/index.md index bd3409abe01..2510b5e73a7 100644 --- a/doc/user/compliance/compliance_center/index.md +++ b/doc/user/compliance/compliance_center/index.md @@ -32,7 +32,7 @@ Prerequisites: To view the standards adherence dashboard for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. ### GitLab standard @@ -75,13 +75,13 @@ information, see [Merge request approval rules](../../project/merge_requests/app > - GraphQL API [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7222) in GitLab 14.9. > - [Generally available](https://gitlab.com/groups/gitlab-org/-/epics/5237) in GitLab 14.10. [Feature flag `compliance_violations_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/346266) removed. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112111) to compliance violations report in GitLab 15.9. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/394950) ability to create/edit compliance frameworks in GitLab 16.0. +> - Ability to create and edit compliance frameworks [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/394950) in GitLab 16.0. With compliance violations report, you can see a high-level view of merge request activity for all projects in the group. When you select a row in the compliance report, a drawer appears that provides: -- The project name and [compliance framework label](../../project/settings/index.md#add-a-compliance-framework-to-a-project), +- The project name and [compliance framework label](../../project/working_with_projects.md#add-a-compliance-framework-to-a-project), if the project has one assigned. - A link to the merge request that introduced the violation. - The merge request's branch path in the format `[source] into [target]`. @@ -100,7 +100,7 @@ Prerequisites: To view the compliance violations report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. You can sort the compliance report on: @@ -207,7 +207,7 @@ If the commit has a related merge commit, then the following are also included: To generate the Chain of Custody report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. Select **List of all merge commits**. @@ -223,7 +223,7 @@ details for the provided commit SHA. To generate a commit-specific Chain of Custody report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. At the top of the compliance report, to the right of **List of all commits**, select the down arrow (**{chevron-lg-down}**). @@ -254,7 +254,7 @@ Prerequisites: To view the compliance frameworks report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. @@ -271,7 +271,7 @@ Prerequisites: To apply a compliance framework to one project in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. Next to the project you want to add the compliance framework to, select **{plus}** **Add framework**. @@ -279,7 +279,7 @@ To apply a compliance framework to one project in a group: To apply a compliance framework to multiple projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. Select multiple projects. @@ -300,20 +300,46 @@ Prerequisites: To remove a compliance framework from one project in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. Next to the compliance framework to remove from the project, select **{close}** on the framework label. To remove a compliance framework from multiple projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. Select multiple projects. 1. From the **Choose one bulk action** dropdown list, select **Remove framework from selected projects**. 1. Select **Remove**. +### Export a report of merge request compliance violations on projects in a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356791) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `compliance_violation_csv_export`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named +`compliance_violation_csv_export`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Export a report of merge request compliance violations on merge requests belonging to projects in a group. Reports: + +- Do not use filters on the violations report. +- Are truncated at 15 MB so the email attachment is not too large. + +Prerequisites: + +- You must be an administrator or have the Owner role for the group. + +To export a report of merge request compliance violations for projects in a group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. On the left sidebar, select **Secure > Compliance center**. +1. On the page, select the **Violations** tab. +1. On the Violations tab, select the **Export full report as CSV** action in the top right corner + +A report is compiled and delivered to your email inbox as an attachment. + ### Export a report of compliance frameworks on projects in a group > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387912) in GitLab 16.0. @@ -329,7 +355,7 @@ Prerequisites: To export a report of compliance frameworks on projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. On the Frameworks tab, select the **Export as CSV** action in the top right corner @@ -342,7 +368,7 @@ A report is compiled and delivered to your email inbox as an attachment. To filter the list of compliance frameworks: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Secure > Compliance center**. 1. On the page, select the **Frameworks** tab. 1. In the search field: diff --git a/doc/user/compliance/license_approval_policies.md b/doc/user/compliance/license_approval_policies.md index e3350b1ae10..cf5f340c0f6 100644 --- a/doc/user/compliance/license_approval_policies.md +++ b/doc/user/compliance/license_approval_policies.md @@ -41,7 +41,7 @@ Create a license approval policy to enforce license compliance. To create a license approval policy: 1. [Link a security policy project](../application_security/policies/index.md#managing-the-linked-security-policy-project) to your development group, subgroup, or project (the Owner role is required). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Secure > Policies**. 1. Create a new [Scan Result Policy](../application_security/policies/scan-result-policies.md). 1. In your policy rule, select **License scanning**. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md deleted file mode 100644 index 00578219016..00000000000 --- a/doc/user/compliance/license_compliance/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -type: reference, howto -stage: Secure -group: Composition Analysis -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../license_approval_policies.md' ---- - -# License Compliance (removed) **(ULTIMATE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387561) in GitLab 15.9 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/421363) in GitLab 16.3. -Use [License Approval Policies](https://gitlab.com/groups/gitlab-org/-/epics/8092) instead. diff --git a/doc/user/compliance/license_list.md b/doc/user/compliance/license_list.md index 96ea43e5ce2..f315f319b71 100644 --- a/doc/user/compliance/license_list.md +++ b/doc/user/compliance/license_list.md @@ -20,7 +20,7 @@ requirements must be met: Alternatively, licenses will also appear under the license list when using our deprecated [`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/License-Scanning.gitlab-ci.yml) as long as the following requirements are met: -1. The Dependency Scanning CI/CD job must be [enabled](license_scanning_of_cyclonedx_files/index.md#enable-license-scanning) for your project. +1. The Dependency Scanning CI/CD job must be [enabled](license_scanning_of_cyclonedx_files/index.md#configuration) for your project. 1. Your project must use at least one of the [supported languages and package managers](license_scanning_of_cyclonedx_files/index.md#supported-languages-and-package-managers). diff --git a/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md b/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md index 97f9f277776..98404ecd2ed 100644 --- a/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md +++ b/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md @@ -7,13 +7,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # License scanning of CycloneDX files **(ULTIMATE ALL)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384932) in GitLab 15.9 for GitLab SaaS [with two flags](../../../administration/feature_flags.md) named `license_scanning_sbom_scanner` and `package_metadata_synchronization`. Both flags are disabled by default and both flags must be enabled for this feature to work. -> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/385173) in GitLab 15.10 for GitLab SaaS. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385173) in GitLab 15.10 for self-managed GitLab [with two flags](../../../administration/feature_flags.md) named `license_scanning_sbom_scanner` and `package_metadata_synchronization`, both of which must be enabled for this feature to work. The flags are disabled by default due to the initial performance load when the license database is first imported. Work to improve performance is being tracked in [issue 397670](https://gitlab.com/gitlab-org/gitlab/-/issues/397670). -> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/385173) in GitLab 15.11 for self-managed GitLab. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384932) in GitLab 15.9 for GitLab SaaS [with two flags](../../../administration/feature_flags.md) named `license_scanning_sbom_scanner` and `package_metadata_synchronization`. Both flags disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/385176) in GitLab 16.4. Feature flags `license_scanning_sbom_scanner` and `package_metadata_synchronization` removed. -FLAG: -The legacy License Compliance analyzer was deprecated in GitLab 15.9 and removed in GitLab 16.3. To continue using GitLab for License Compliance, remove the License Compliance template from your CI/CD pipeline and add the [Dependency Scanning template](../../application_security/dependency_scanning/index.md#configuration). The Dependency Scanning template is now capable of gathering the required license information so it is no longer necessary to run a separate License Compliance job. The License Compliance CI/CD template should not be removed prior to verifying that the `license_scanning_sbom_scanner` and `package_metadata_synchronization` flags are enabled for the instance and that the instance has been upgraded to a version that supports the new method of license scanning. To begin using the Dependency Scanner quickly at scale, you may set up a [scan execution policy](../../application_security/policies/scan-execution-policies.md) at the group level to enforce the SBOM-based license scan for all projects in the group. Then, you may remove the inclusion of the `Jobs/License-Scanning.gitlab-ci.yml` template from your CI/CD configuration. If you wish to continue using the legacy License Compliance feature, you can do so by setting the `LICENSE_MANAGEMENT_VERSION CI` variable to `4`. This variable can be set at the [project](../../../ci/variables/index.md#for-a-project), [group](../../../ci/variables/index.md#for-a-group) or [instance](../../../ci/variables/index.md#for-an-instance) level. This configuration change will allow you to continue using an existing version of the License Compliance without having to adopt the new approach. **Bugs and vulnerabilities in this legacy analyzer will no longer be fixed.** +NOTE: +The legacy License Compliance analyzer was deprecated in GitLab 15.9 and removed in GitLab 16.3. To continue using GitLab for License Compliance, remove the License Compliance template from your CI/CD pipeline and add the [Dependency Scanning template](../../application_security/dependency_scanning/index.md#configuration). The Dependency Scanning template is now capable of gathering the required license information so it is no longer necessary to run a separate License Compliance job. The License Compliance CI/CD template should not be removed prior to verifying that the instance has been upgraded to a version that supports the new method of license scanning. To begin using the Dependency Scanner quickly at scale, you may set up a [scan execution policy](../../application_security/policies/scan-execution-policies.md) at the group level to enforce the SBOM-based license scan for all projects in the group. Then, you may remove the inclusion of the `Jobs/License-Scanning.gitlab-ci.yml` template from your CI/CD configuration. If you wish to continue using the legacy License Compliance feature, you can do so by setting the `LICENSE_MANAGEMENT_VERSION CI` variable to `4`. This variable can be set at the [project](../../../ci/variables/index.md#for-a-project), [group](../../../ci/variables/index.md#for-a-group) or [instance](../../../ci/variables/index.md#for-an-instance) level. This configuration change will allow you to continue using the existing version of License Compliance to generate [license scanning report](../../../ci/yaml/artifacts_reports.md#artifactsreportslicense_scanning) artifacts in your pipelines. However, since legacy license scanning support is being removed from our codebase, switching back to this legacy analyzer prevents other License Compliance features from working as expected, so this approach is not recommended. In addition to this, **bugs and vulnerabilities in this legacy analyzer will no longer be fixed.** To detect the licenses in use, License Compliance relies on running the [Dependency Scanning CI Jobs](../../application_security/dependency_scanning/index.md), @@ -22,17 +20,16 @@ Other 3rd party scanners may also be used as long as they produce a CycloneDX fi This method of scanning is also capable of parsing and identifying over 500 different types of licenses, as defined in [the SPDX list](https://spdx.org/licenses/). Licenses not in the SPDX list are reported as "Unknown". License information can also be extracted from packages that are dual-licensed, or have multiple different licenses that apply. -## Enable license scanning +## Configuration -Prerequisites: - -- On GitLab self-managed only, enable [Synchronization with the GitLab License Database](../../../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync) in Admin Area for the GitLab instance. On GitLab SaaS this step has already been completed. -- Enable [Dependency Scanning](../../application_security/dependency_scanning/index.md#configuration) - and ensure that its prerequisites are met. +Enable [Dependency Scanning](../../application_security/dependency_scanning/index.md#configuration) +and ensure that its prerequisites are met. From the `.gitlab-ci.yml` file, remove the deprecated line `Jobs/License-Scanning.gitlab-ci.yml`, if it's present. +On GitLab self-managed only, you can [choose package registry metadata to sync](../../../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync) in the Admin Area for the GitLab instance. + ## Supported languages and package managers License scanning is supported for the following languages and package managers: @@ -164,3 +161,29 @@ gemnasium-dependency_scanning: - apk update && apk add jq - jq '.components |= unique' gl-sbom-gem-bundler.cdx.json > tmp.json && mv tmp.json gl-sbom-gem-bundler.cdx.json ``` + +### Remove unused license data + +License scanning changes (released in GitLab 15.9) required a significant amount of additional disk space to be available on the instances. This issue was resolved in GitLab 16.3 by the [Reduce package metadata table on-disk footprint](https://gitlab.com/groups/gitlab-org/-/epics/10415) epic. But if your instance was running license scanning between GitLab 15.9 and 16.3, you may want to remove the unneeded data. + +To remove the unneeded data: + +1. Check if the [package_metadata_synchronization](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#new-license-compliance-scanner) feature flag is currently, or was previously enabled, and if so, disable it. Use [Rails console](../../../administration/operations/rails_console.md) to execute the following commands. + + ```ruby + Feature.enabled?(:package_metadata_synchronization) && Feature.disable(:package_metadata_synchronization) + ``` + +1. Check if there is deprecated data in the database: + + ```ruby + PackageMetadata::PackageVersionLicense.count + PackageMetadata::PackageVersion.count + ``` + +1. If there is deprecated data in the database, remove it by running the following commands in order: + + ```ruby + PackageMetadata::PackageVersionLicense.delete_all + PackageMetadata::PackageVersion.delete_all + ``` diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index 53add75ee19..46e395af5bc 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -36,7 +36,7 @@ you must enable CRM features for the subgroup. To enable customer relations management in a group or subgroup: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group or subgroup. +1. On the left sidebar, select **Search or go to** and find your group or subgroup. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Select **Customer relations is enabled**. @@ -52,7 +52,7 @@ Prerequisites: To view a group's contacts: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer contacts**.  @@ -65,7 +65,7 @@ Prerequisites: To create a contact: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer contacts**. 1. Select **New contact**. 1. Complete all required fields. @@ -82,7 +82,7 @@ Prerequisites: To edit an existing contact: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer contacts**. 1. Next to the contact you wish to edit, select **Edit** (**{pencil}**). 1. Edit the required fields. @@ -100,7 +100,7 @@ Each contact can be in one of two states: To change the state of a contact: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer contacts**. 1. Next to the contact you wish to edit, select **Edit** (**{pencil}**). 1. Select or clear the **Active** checkbox. @@ -116,7 +116,7 @@ Prerequisites: To view a group's organizations: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer organizations**.  @@ -129,7 +129,7 @@ Prerequisites: To create an organization: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer organizations**. 1. Select **New organization**. 1. Complete all required fields. @@ -146,7 +146,7 @@ Prerequisites: To edit an existing organization: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer organizations**. 1. Next to the organization you wish to edit, select **Edit** (**{pencil}**). 1. Edit the required fields. @@ -168,7 +168,7 @@ Prerequisites: To view a contact's issues, select a contact from the issue sidebar, or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer contacts**. 1. Next to the contact whose issues you wish to view, select **View issues** (**{issues}**). @@ -180,7 +180,7 @@ Prerequisites: To view an organization's issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Customer organizations**. 1. Next to the organization whose issues you wish to view, select **View issues** (**{issues}**). diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 0bdbfe79775..9d4a151cc53 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -13,7 +13,7 @@ type: reference, howto > - Paginated merge request discussions [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/370075) in GitLab 15.8. Feature flag `paginated_mr_discussions` removed. GitLab encourages communication through comments, threads, and -[code suggestions](../project/merge_requests/reviews/suggestions.md). +[Code Suggestions](../project/merge_requests/reviews/suggestions.md). Two types of comments are available: @@ -275,3 +275,29 @@ To create a thread: A threaded comment is created.  + +## Resolve a thread + +> - Resolvable threads for issues [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31114) in GitLab 16.3 [with a flag](../../administration/feature_flags.md) named `resolvable_issue_threads`. Disabled by default. +> - Resolvable threads for issues [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/31114) in GitLab 16.4. + +FLAG: +On self-managed GitLab, resolvable threads _for issues_ are available by default. +To hide the feature, an administrator can +[disable the feature flag](../../administration/feature_flags.md) named `resolvable_issue_threads`. +On GitLab.com, this feature is available. + +You can resolve a thread when you want to finish a conversation. + +Prerequisites: + +- You must be in an issue or merge request. +- You must have at least the Developer role or be the author of the issue or merge request. + +To resolve a thread: + +1. Go to the thread. +1. Do one of the following: + - In the upper-right corner of the original comment, select **Resolve thread** (**{check-circle}**). + - Below the last reply, in the **Reply** field, select **Resolve thread**. + - Below the last reply, in the **Reply** field, enter text, select the **Resolve thread** checkbox, and select **Add comment now**. diff --git a/doc/user/enterprise_user/index.md b/doc/user/enterprise_user/index.md index e7be5bfb140..04683620ba9 100644 --- a/doc/user/enterprise_user/index.md +++ b/doc/user/enterprise_user/index.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference @@ -79,7 +79,7 @@ For more information on group-level domain verification, see [epic 5299](https:/ The custom domain must match the email domain exactly. For example, if your email is `username@example.com`, verify the `example.com` domain. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your top-level group. +1. On the left sidebar, select **Search or go to** and find your top-level group. 1. Select **Settings > Domain Verification**. 1. In the upper-right corner, select **Add Domain**. 1. In **Domain**, enter the domain name. @@ -105,7 +105,7 @@ and paste them in your domain's control panel as a `TXT` record. After you have added all the DNS records: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Domain Verification**. 1. On the domain table row, Select **Retry verification** (**{retry}**). @@ -125,7 +125,7 @@ For GitLab instances with domain verification enabled, if the domain cannot be v To view all configured domains in your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your top-level group. +1. On the left sidebar, select **Search or go to** and find your top-level group. 1. Select **Settings > Domain Verification**. You then see: @@ -138,7 +138,7 @@ You then see: To edit or remove a domain: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your top-level group. +1. On the left sidebar, select **Search or go to** and find your top-level group. 1. Select **Settings > Domain Verification**. 1. When viewing **Domain Verification**, select the project listed next to the relevant domain. 1. Edit or remove a domain following the relevant [GitLab Pages custom domains](../project/pages/custom_domains_ssl_tls_certification/index.md) instructions. @@ -159,7 +159,7 @@ Top-level group Owners can disable two-factor authentication (2FA) for enterpris To disable 2FA: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Find a user with the **Enterprise** and **2FA** badges. 1. Select **More actions** (**{ellipsis_v}**) and select **Disable two-factor authentication**. diff --git a/doc/user/free_user_limit.md b/doc/user/free_user_limit.md index 0af5c76ade8..c978105c13b 100644 --- a/doc/user/free_user_limit.md +++ b/doc/user/free_user_limit.md @@ -25,7 +25,7 @@ Prerequisite: - You must have the Owner role for the group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Settings > Usage Quotas**. 1. To view all members, select the **Seats** tab. 1. To remove a member, select **Remove user**. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 47f6e0ac1d9..09e05538fd7 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -18,7 +18,7 @@ GitLab.com has the: - [`email_confirmation_setting`](../../administration/settings/sign_up_restrictions.md#confirm-user-email) setting set to **Hard**. - [`unconfirmed_users_delete_after_days`](../../administration/moderate_users.md#automatically-delete-unconfirmed-users) - setting set to one day. + setting set to three days. ## Password requirements @@ -82,7 +82,7 @@ The IP addresses for `mg.gitlab.com` are subject to change at any time. On GitLab.com, there's a mailbox configured for Service Desk with the email address: `contact-project+%{key}@incoming.gitlab.com`. To use this mailbox, configure the -[custom suffix](../project/service_desk/index.md#configure-a-suffix-for-service-desk-alias-email) in project +[custom suffix](../project/service_desk/configure.md#configure-a-suffix-for-service-desk-alias-email) in project settings. ## Backups @@ -217,7 +217,7 @@ the default value [is the same as for self-managed instances](../../administrati | Maximum remote file size for imports from external object storages | 10 GB | | Maximum download file size when importing from source GitLab instances by direct transfer | 5 GB | | Maximum attachment size | 100 MB | -| [Maximum decompressed file size for imported archives](../../administration/settings/account_and_limit_settings.md#maximum-decompressed-file-size-for-imported-archives) | 25 GB | +| [Maximum decompressed file size for imported archives](../../administration/settings/import_and_export_settings.md#maximum-decompressed-file-size-for-imported-archives) | 25 GB | If you are near or over the repository size limit, you can either [reduce your repository size with Git](../project/repository/reducing_the_repo_size_using_git.md) @@ -225,7 +225,7 @@ or [purchase additional storage](https://about.gitlab.com/pricing/licensing-faq/ NOTE: `git push` and GitLab project imports are limited to 5 GB per request through -Cloudflare. Git LFS and imports other than a file upload are not affected by +Cloudflare. Imports other than a file upload are not affected by this limit. Repository limits apply to both public and private projects. ## Default import sources @@ -236,7 +236,7 @@ The import sources that are available by default depend on which GitLab you use: - GitLab.com: all available import sources are enabled by default. - GitLab self-managed: no import sources are enabled by default and must be - [enabled](../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources). + [enabled](../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). | Import source | GitLab.com default | GitLab self-managed default | |:----------------------------------------------------------------------------------------------------|:-----------------------|:----------------------------| diff --git a/doc/user/group/access_and_permissions.md b/doc/user/group/access_and_permissions.md index 0ccd4512039..428c87143f6 100644 --- a/doc/user/group/access_and_permissions.md +++ b/doc/user/group/access_and_permissions.md @@ -46,7 +46,7 @@ configured by an administrator. To change the permitted Git access protocols for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Choose the permitted protocols from **Enabled Git access protocols**. @@ -71,7 +71,7 @@ Administrators can combine restricted access by IP address with To restrict group access by IP address: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. In the **Restrict access by IP address** text box, enter a list of IPv4 or IPv6 @@ -102,6 +102,15 @@ Keep in mind that restricting group access by IP address has the following impli IP access restrictions applied to self-managed instances are possible with [`gitlab-sshd`](../../administration/operations/gitlab_sshd.md) with [PROXY protocol](../../administration/operations/gitlab_sshd.md#proxy-protocol-support) enabled. - IP restriction is not applicable to shared resources belonging to a group. Any shared resource is accessible to a user even if that user is not able to access the group. +- While IP restrictions apply to public projects, they aren't a complete firewall and cached files for a project may still be accessible to users not in the IP block + +### GitLab.com access restrictions + +On GitLab.com shared runners are added to the [global allowlist](../../administration/settings/visibility_and_access_controls.md#configure-globally-allowed-ip-address-ranges), so that they are available regardless of IP restrictions. + +Artifact and Registry downloading from runners is sourced from any Google or, in the case of MacOS runners, Amazon IP address in that region. +The download is therefore not added to the global allowlist. +To allow runner downloading, add the [outbound runner CIDR ranges](../gitlab_com/index.md#ip-range) to your group allowlist. ## Restrict group access by domain **(PREMIUM ALL)** @@ -113,7 +122,7 @@ You can prevent users with email addresses in specific domains from being added To restrict group access by domain: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. In the **Restrict membership by email** field, enter the domain names. @@ -157,7 +166,7 @@ If you prevent group sharing outside the hierarchy for the **Animals** group: To prevent sharing outside of the group's hierarchy: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Select **Members cannot invite groups outside of `<group_name>` and its subgroups**. @@ -173,7 +182,7 @@ which can be confusing and difficult to control. To restrict the permission to invite project members to a single source, prevent a project from being shared with other groups: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Select **Projects in `<group_name>` cannot be shared with other groups**. @@ -187,7 +196,7 @@ added to a project lose access when the setting is enabled. As a group Owner, you can prevent non-members from requesting access to your group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Clear the **Allow users to request access** checkbox. @@ -207,7 +216,7 @@ If even one is set to `true`, then the group does not allow outside forks. To prevent projects from being forked outside the group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Check **Prevent project forking outside current group**. @@ -232,7 +241,7 @@ The setting does not cascade. Projects in subgroups observe the subgroup configu To prevent members from being added to projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Under **Membership**, select **Users cannot be added to projects in this group**. @@ -254,7 +263,7 @@ For more information on the administration of LDAP and group sync, refer to the NOTE: When you add LDAP synchronization, if an LDAP user is a group member and they are not part of the LDAP group, they are removed from the group. -You can use a workaround to [manage project access through LDAP groups](../project/settings/index.md#manage-project-access-through-ldap-groups). +You can use a workaround to [manage project access through LDAP groups](../project/working_with_projects.md#manage-project-access-through-ldap-groups). ### Create group links via CN **(PREMIUM SELF)** @@ -284,7 +293,7 @@ To create group links via filter: LDAP user permissions can be manually overridden by an administrator. To override a user's permissions: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Manage > Members**. If LDAP synchronization has granted a user a role with: - More permissions than the parent group membership, that user is displayed as having diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index e41991f365c..5c0844dff92 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -21,7 +21,7 @@ your group, enabling you to use the same cluster across multiple projects. To view your group-level Kubernetes clusters: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Operate > Kubernetes**. ## Cluster management project @@ -89,7 +89,7 @@ your cluster, which can cause deployment jobs to fail. To clear the cache: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Operate > Kubernetes**. 1. Select your cluster. 1. Expand **Advanced settings**. diff --git a/doc/user/group/compliance_frameworks.md b/doc/user/group/compliance_frameworks.md index 35e8ad27bc3..7258791a983 100644 --- a/doc/user/group/compliance_frameworks.md +++ b/doc/user/group/compliance_frameworks.md @@ -15,7 +15,7 @@ requirements or needs additional oversight. The label can optionally enforce Compliance frameworks are created on top-level groups. Group owners can create, edit, and delete compliance frameworks: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings** > **General**. 1. Expand the **Compliance frameworks** section. 1. Create, edit, or delete compliance frameworks. @@ -31,7 +31,7 @@ Prerequisite: To assign a compliance framework to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings** > **General**. 1. Expand **Compliance frameworks**. 1. Select a compliance framework. @@ -66,7 +66,7 @@ A compliance framework that is set to default has a **default** label. Group owners can set a compliance framework as default (or remove the setting): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Compliance frameworks** section and locate the compliance framework to set (or remove) as default. 1. Select the vertical ellipsis (**{ellipsis_v}**) for the compliance frame and then select **Set default** (or @@ -155,7 +155,7 @@ Therefore, communicate with project users about compliance pipeline configuratio To configure a compliance pipeline: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings** > **General**. 1. Expand the **Compliance frameworks** section. 1. In **Compliance pipeline configuration (optional)**, add the path to the compliance framework configuration. Use the @@ -165,7 +165,7 @@ To configure a compliance pipeline: - `.compliance-ci.yaml@gitlab-org/gitlab`. This configuration is inherited by projects where the compliance framework label is -[applied](../project/settings/index.md#add-a-compliance-framework-to-a-project). In projects with the applied compliance +[applied](../project/working_with_projects.md#add-a-compliance-framework-to-a-project). In projects with the applied compliance framework label, the compliance pipeline configuration is run instead of the labeled project's own pipeline configuration. The user running the pipeline in the labeled project must at least have the Reporter role on the compliance project. diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 9716143f5e2..96cc9ce974c 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -20,7 +20,7 @@ Use contribution analytics data visualizations to: To view contribution analytics: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Contribution analytics**. Three bar charts and a table illustrate the number of contributions made by each group member: diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 1a481641111..87c1c548abd 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -40,7 +40,7 @@ Projects in nested subgroups are not included in the template list. ## Which projects are available as templates - Public and internal projects can be selected by any authenticated user as a template for a new project, - if all [project features](../project/settings/index.md#configure-project-visibility-features-and-permissions) + if all [project features](../project/settings/index.md#configure-project-features-and-permissions) except for **GitLab Pages** and **Security and Compliance** are set to **Everyone With Access**. - Private projects can be selected only by users who are members of the projects. @@ -69,6 +69,35 @@ gitlab.com/myorganization/ ... ``` +## What is copied from the templates + +The entire custom instance-level project templates repository is copied, including: + +- Branches +- Commits +- Tags + +If the user: + +- Has the Owner role on the custom instance-level project templates project or is a GitLab administrator, + all project settings are copied over to the new project. +- Doesn't have the Owner role or is not a GitLab administrator, + project deploy keys and project webhooks aren't copied over because they contain sensitive data. + +To learn more about what is migrated, see +[Items that are exported](../project/settings/import_export.md#items-that-are-exported). + +## User assignments in templates + +When you use a template created by another user, any items that were assigned +to a user in the template are reassigned to you. It's important to understand +this reassignment when you configure security features like protected branches +and tags. For example, if the template contains a protected branch: + +- In the template, the branch allows the _template owner_ to merge into the default branch. +- In the project created from the template, the branch allows _you_ to merge into + the default branch. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 852d26f3816..5359d7b52a0 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -36,7 +36,7 @@ Prerequisite: To view DevOps Adoption: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > DevOps adoption** ## DevOps Adoption categories @@ -80,7 +80,7 @@ twelve months. The chart only shows data from when you enabled DevOps Adoption f To view feature adoption over time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > DevOps adoption**. 1. Select the **Overview** tab. diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index 41231db5964..69b64861bd5 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -25,7 +25,7 @@ On the top of each list, you can see the number of epics in the list (**{epic}** To view an epic board: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Epic boards**.  @@ -38,7 +38,7 @@ Prerequisites: To create a new epic board: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Epic boards**. 1. In the upper-left corner, select the dropdown list with the current board name. 1. Select **Create new board**. @@ -87,7 +87,7 @@ Prerequisites: To create a new list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Epic boards**. 1. In the upper-right corner, select **Create list**. 1. In the **New list** column expand the **Select a label** dropdown list and select the label to use as diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index e6116151012..b79177e1571 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -209,7 +209,7 @@ Prerequisites: To view epics in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Epics**. ### Who can view an epic @@ -254,7 +254,7 @@ You can filter the list of epics by: To filter: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Epics**. 1. Select the field **Search or filter results**. 1. From the dropdown list, select the scope or enter plain text to search by epic title or description. diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index b645ea04038..a049b4afcc1 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -30,7 +30,7 @@ If you migrate from GitLab.com to self-managed GitLab, an administrator can crea > - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. On self-managed GitLab, by default [migrating group items](#migrated-group-items) is not available. To show the -feature, an administrator can [enable it in application settings](../../../administration/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer). +feature, an administrator can [enable it in application settings](../../../administration/settings/import_and_export_settings.md#enable-migration-of-groups-and-projects-by-direct-transfer). Migrating groups by direct transfer copies the groups from one place to another. You can: @@ -47,7 +47,7 @@ Migrating groups by direct transfer copies the groups from one place to another. Not all group and project resources are copied. See list of copied resources below: - [Migrated group items](#migrated-group-items). -- [Migrated project items](#migrated-project-items-beta). +- [Migrated project items](#migrated-project-items). WARNING: Importing groups with projects is in [Beta](../../../policy/experiment-beta-support.md#beta). This feature is not @@ -157,16 +157,20 @@ To migrate groups by direct transfer: - The network connection between instances or GitLab.com must support HTTPS. - Any firewalls must not block the connection between the source and destination GitLab instances. - Both GitLab instances must have group migration by direct transfer - [enabled in application settings](../../../administration/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer) + [enabled in application settings](../../../administration/settings/import_and_export_settings.md#enable-migration-of-groups-and-projects-by-direct-transfer) by an instance administrator. - The source GitLab instance must be running GitLab 14.0 or later. -- You must have a [personal access token](../../../user/profile/personal_access_tokens.md) for the source GitLab - instance: - - For GitLab 15.1 and later source instances, the personal access token must have the `api` scope. - - For GitLab 15.0 and earlier source instances, the personal access token must have both the `api` and - `read_repository` scopes. +- You must have a + [personal access token](../../../user/profile/personal_access_tokens.md) for + the source GitLab instance: + - For GitLab 15.1 and later source instances, the personal access token must + have the `api` scope. + - For GitLab 15.0 and earlier source instances, the personal access token must + have both the `api` and `read_repository` scopes. - You must have the Owner role on the source group to migrate from. -- You must have at least the Maintainer role on the destination group to migrate to. +- Your must have a role on the destination namespace the enables you to + [create a subgroup](../../group/subgroups/index.md#create-a-subgroup) in that + namespace. ### Prepare user accounts @@ -287,7 +291,7 @@ Some group items are excluded from migration because they either: - May contain sensitive information: CI/CD variables, webhooks, and deploy tokens. - Are not supported: push rules. -### Migrated project items (Beta) +### Migrated project items **(BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. @@ -460,7 +464,7 @@ To solve this, you must change the source group path to include a non-numerical - The GitLab UI: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. Under **Change group URL**, change the group URL to include non-numeric characters. @@ -593,7 +597,7 @@ Prerequisite: To enable import and export for a group: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. @@ -607,7 +611,7 @@ Prerequisites: To export the contents of a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. In the **Advanced** section, select **Export group**. 1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 13fba43f8ef..484fd8c533b 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -49,14 +49,14 @@ the immediate parent group. To explore all public groups: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. 1. At the top right, select **Explore groups**. To view groups where you have a direct or indirect membership: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. This page shows groups that you are a member of: @@ -67,7 +67,7 @@ This page shows groups that you are a member of: To view the activity of a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Activity**. 1. Optional. To filter activity by contribution type, select a tab: @@ -106,7 +106,7 @@ For details about groups, watch [GitLab Namespaces (users, groups and subgroups) To remove a group and its contents: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Advanced** section. 1. In the **Remove group** section, select **Remove group**. @@ -115,8 +115,8 @@ To remove a group and its contents: A group can also be removed from the groups dashboard: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. 1. Select (**{ellipsis_v}**) for the group you want to delete. 1. Select **Delete**. 1. In the Remove group section, select **Remove group**. @@ -143,7 +143,7 @@ Prerequisites: To immediately remove a group marked for deletion: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the "Permanently remove group" section, select **Remove group**. @@ -158,7 +158,7 @@ are deleted. To restore a group that is marked for deletion: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. In the Restore group section, select **Restore group**. @@ -167,10 +167,12 @@ To restore a group that is marked for deletion: As a user, you can request to be a member of a group, if an administrator allows it. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your groups**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my groups**. 1. At the top right side, select **Explore groups**. -1. Under the group name, select **Request Access**. +1. Search for the group by name. +1. In the search results, select the name of the group. +1. On the group page, under the group name, select **Request Access**. As many as ten of the most-recently-active group owners receive an email with your request. Any group owner can approve or decline the request. @@ -182,7 +184,7 @@ If you change your mind before your request is approved, select To view a group's members: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. A table displays the member's: @@ -219,7 +221,7 @@ In lists of group members, entries can display the following badges: - **SAML**, to indicate the member has a [SAML account](saml_sso/index.md) connected to them. - **Enterprise**, to indicate that the member is an [enterprise user](../enterprise_user/index.md). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Above the list of members, in the **Filter members** box, enter filter criteria. - To view members in the group only, select **Membership = Direct**. @@ -231,7 +233,7 @@ In lists of group members, entries can display the following badges: You can search for members by name, username, or [public email](../profile/index.md#set-your-public-email). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Above the list of members, in the **Filter members** box, enter search criteria. 1. To the right of the **Filter members** box, select the magnifying glass (**{search}**). @@ -240,7 +242,7 @@ You can search for members by name, username, or [public email](../profile/index You can sort members by **Account**, **Access granted**, **Max role**, or **Last sign-in**. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Above the list of members, in the upper-right corner, from the **Account** list, select the criteria to filter by. @@ -257,7 +259,7 @@ Prerequisite: - You must have the Owner role. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Select **Invite members**. 1. Fill in the fields. @@ -287,7 +289,7 @@ Prerequisites: To remove a member from a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Next to the member you want to remove, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Remove member**. @@ -325,7 +327,7 @@ By default, users with: To change the role that can create projects under a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Select the desired option in the **Roles allowed to create projects** dropdown list. diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 5cb982a85e4..2ded1b08de2 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -23,7 +23,7 @@ Prerequisites: To access your group's insights: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Insights**. ## Interact with insights charts @@ -65,7 +65,7 @@ To configure group insights: 1. Create a new file [`.gitlab/insights.yml`](../../project/insights/index.md#configure-project-insights) in a project that belongs to your group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Analytics** and find the **Insights** section. 1. Select the project that contains your `.gitlab/insights.yml` configuration file. diff --git a/doc/user/group/issues_analytics/img/issues_closed_analytics_v16_4.png b/doc/user/group/issues_analytics/img/issues_closed_analytics_v16_4.png Binary files differnew file mode 100644 index 00000000000..5e1fe4eaa8c --- /dev/null +++ b/doc/user/group/issues_analytics/img/issues_closed_analytics_v16_4.png diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md index 46ba1747b06..4f1c7b4be7a 100644 --- a/doc/user/group/issues_analytics/index.md +++ b/doc/user/group/issues_analytics/index.md @@ -1,7 +1,7 @@ --- type: reference stage: Plan -group: Project Management +group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -15,9 +15,11 @@ prior. To access the chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Analyze > Issue analytics**. +You can also access the chart from the [Value Streams Dashboard](../../analytics/value_streams_dashboard.md) through the **New issues** drill-down report. + Hover over each bar to see the total number of issues. To narrow the scope of issues included in the graph, enter your criteria in the @@ -36,6 +38,20 @@ shows a total of 15 months for the chart in the GitLab.org group.  +## Enhanced issue analytics **(ULTIMATE ALL)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233905/) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can +[enable the feature flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. On GitLab.com, this feature is not +available. This feature is not ready for production use. + +Enhanced issue analytics display the additional metric "Issues closed", which represents the total number of resolved issues in your group over a selected period. +You can use this metric to improve the overall turn-around time and value delivered to your customers. + + + ## Drill into the information > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196547) in GitLab 13.1. diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index 91e69f3a02d..8c0838cf33c 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -50,7 +50,7 @@ Prerequisites: To create an iteration cadence: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. 1. Select **New iteration cadence**. 1. Enter the title and description of the iteration cadence. @@ -70,7 +70,7 @@ If you want to manually manage the created cadence, read [Manual Iteration Manag ### View the iterations list -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. To view all the iterations in a cadence, ordered by descending date, select that iteration cadence. @@ -78,7 +78,7 @@ From there you can create a new iteration or select an iteration to get a more d NOTE: If a project has issue tracking -[turned off](../../project/settings/index.md#configure-project-visibility-features-and-permissions), +[turned off](../../project/settings/index.md#configure-project-features-and-permissions), to view the iterations list, enter its URL. To do so, add: `/-/cadences` to your project or group URL. For example `https://gitlab.com/gitlab-org/sample-data-templates/sample-gitlab-project/-/cadences`. This is tracked in [issue 339009](https://gitlab.com/gitlab-org/gitlab/-/issues/339009). @@ -91,7 +91,7 @@ Prerequisites: To edit an iteration cadence: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. 1. Select **Edit iteration cadence**. @@ -105,7 +105,7 @@ doesn't delete the eight existing upcoming iterations. #### Turn on and off automatic scheduling for an iteration cadence -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. 1. Next to the cadence for which you want to turn on or off automatic scheduling, select the three-dot menu (**{ellipsis_v}**) **> Edit cadence**. @@ -157,7 +157,7 @@ Deleting an iteration cadence also deletes all iterations within that cadence. To delete an iteration cadence: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. 1. Select the three-dot menu (**{ellipsis_v}**) > **Delete cadence** for the cadence you want to delete. 1. Select **Delete cadence**. @@ -178,7 +178,7 @@ Prerequisites: To create an iteration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Plan > Iterations** and select an iteration cadence. 1. Select **New iteration**. 1. Enter the title, a description (optional), a start date, and a due date. @@ -277,7 +277,7 @@ and get a more accurate understanding of scope attributable to each label. To group issues by label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Iterations**. 1. In the **Group by** dropdown list, select **Label**. 1. Select the **Filter by label** dropdown list. diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index 743aabd8f4b..65190847b05 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -29,7 +29,7 @@ Prerequisite: To add a group README: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. In the **Group README** section, select **Add README**. This action creates a new project `gitlab-profile` that contains the `README.md` file. 1. On the prompt for creating a README, select **Create and add README**. You're redirected to the Web IDE, where a README file is created. @@ -41,12 +41,12 @@ You can change the owner of a group. Each group must always have at least one member with the Owner role. - As an administrator: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Give a different member the **Owner** role. 1. Refresh the page. You can now remove the **Owner** role from the original owner. - As the current group's owner: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Give a different member the **Owner** role. 1. Have the new owner sign in and remove the **Owner** role from you. @@ -72,7 +72,7 @@ create a new group and transfer projects to it instead. To change your group path (group URL): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. Under **Change group URL**, enter a new name. @@ -130,14 +130,15 @@ After sharing the `Frontend` group with the `Engineering` group: - The **Groups** tab lists the `Engineering` group. - The **Groups** tab lists a group regardless of whether it is a public or private group. -- All direct members of the `Engineering` group have access to the `Frontend` group. Direct members of `Engineering` that gain access to the `Frontend` group keep their same access level as in `Engineering`, but up to the maximum access level selected when sharing the group. Inherited members of the `Engineering` group do not gain access to the `Frontend` group. +- All direct members of the `Engineering` group have access to the `Frontend` group. Direct members of `Engineering` that gain access to the `Frontend` group keep their same access level as in `Engineering`, but up to the maximum access level selected when sharing the group. +- Inherited members of the `Engineering` group do not gain access to the `Frontend` group. - Direct members of the `Engineering` group who have the **Group Invite** badge next to their profile on the group's usage quota page count towards the billable members of the `Frontend` group. ## Remove a shared group To unshare a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. 1. Select the **Groups** tab. 1. To the right of the account you want to remove, select **Remove group** (**{remove}**). @@ -174,7 +175,7 @@ When transferring groups, note: To transfer a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. In the **Remove group** section, select **Transfer group**. @@ -189,7 +190,7 @@ You can disable all email notifications related to the group, which includes its To disable email notifications: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Select **Email notifications are disabled**. @@ -209,7 +210,7 @@ This is particularly helpful for groups with a large number of users. To disable group mentions: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Permissions and group features** section. 1. Select **Group mentions are disabled**. @@ -222,7 +223,7 @@ To disable group mentions: You can export a list of members in a group or subgroup as a CSV. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group or subgroup. +1. On the left sidebar, select **Search or go to** and find your group or subgroup. 1. On the left sidebar, **Manage > Members**. 1. Select **Export as CSV**. 1. After the CSV file has been generated, it is emailed as an attachment to the user that requested it. @@ -252,7 +253,7 @@ Prerequisite: To specify a user cap: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. You can set a cap on the top-level group only. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. @@ -274,7 +275,7 @@ Prerequisite: To remove the user cap: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. In the **User cap** box, delete the value. @@ -295,7 +296,7 @@ Prerequisite: To approve members that are pending because they've exceeded the user cap: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Settings > Usage Quotas**. 1. On the **Seats** tab, under the alert, select **View pending approvals**. 1. For each member you want to approve, select **Approve**. @@ -336,7 +337,7 @@ For more information, see [group-level project templates](custom_project_templat To enable group file templates: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Templates** section. 1. Choose a project to act as the template repository. @@ -369,7 +370,7 @@ Prerequisites: To enable this setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Merge requests**. 1. Under **Merge checks**, select **Pipelines must succeed**. @@ -388,7 +389,7 @@ Prerequisite: To change this behavior: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Merge requests**. 1. Under **Merge checks**: @@ -407,7 +408,7 @@ Prerequisite: To enable this setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Merge requests**. 1. Under **Merge checks**, select **All threads must be resolved**. @@ -425,7 +426,7 @@ that belong to the group. To view the merge request approval settings for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand the **Merge request approvals** section. 1. Select the settings you want. @@ -443,19 +444,19 @@ for the ability to set merge request approval rules for groups is tracked in WARNING: This feature is in [Beta](../../policy/experiment-beta-support.md#beta). -Beta users should read about the [known limitations](../project/repository/code_suggestions.md#known-limitations). -We look forward to hearing your [feedback](../project/repository/code_suggestions.md#feedback). +Beta users should read about the [known limitations](../project/repository/code_suggestions/index.md#known-limitations). +We look forward to hearing your [feedback](../project/repository/code_suggestions/index.md#feedback). -You can give all users in a group and its subgroups access to [Code Suggestions](../project/repository/code_suggestions.md). +You can give all users in a group and its subgroups access to [Code Suggestions](../project/repository/code_suggestions/index.md). - This setting [cascades to all projects](../project/merge_requests/approvals/settings.md#settings-cascading) in the group. - Each user can - [enable or disable Code Suggestions for themselves](../project/repository/code_suggestions.md#enable-code-suggestions-for-an-individual-user). + [Enable Code Suggestions](../../user/profile/preferences.md#enable-code-suggestions). To enable Code Suggestions for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Under **Code Suggestions**, select the **Projects in this group can use Code Suggestions** checkbox. @@ -476,7 +477,7 @@ that belong to the group. To enable Experiment features for a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Under **Experiment features**, select the **Use Experiment features** checkbox. @@ -496,7 +497,7 @@ that belong to the group. To disable third-party AI features for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Under **Third-party AI services**, uncheck the **Use third-party AI services** checkbox. @@ -521,7 +522,7 @@ Changes to [group wikis](../project/wiki/group.md) do not appear in group activi You can view the most recent actions taken in a group, either in your browser or in an RSS feed: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Manage > Activity**. To view the activity feed in Atom format, select the diff --git a/doc/user/group/moderate_users.md b/doc/user/group/moderate_users.md index 7e67bb6ddb5..6859125b323 100644 --- a/doc/user/group/moderate_users.md +++ b/doc/user/group/moderate_users.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/group/reporting/git_abuse_rate_limit.md b/doc/user/group/reporting/git_abuse_rate_limit.md index abb967ad8b1..1b14edb04d9 100644 --- a/doc/user/group/reporting/git_abuse_rate_limit.md +++ b/doc/user/group/reporting/git_abuse_rate_limit.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 3e7bacbb817..6f02b27a214 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -37,7 +37,7 @@ The **Analyze > Repository analytics** group page displays the average test cove To see the latest code coverage for each project in your group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Repository analytics**. 1. In the **Latest test coverage results** section, from the **Select projects** dropdown list, choose the projects you want to check. @@ -52,7 +52,7 @@ You can get a CSV of the code coverage data for all of the projects in your grou To get the report: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Repository analytics**. 1. Select **Download historic test coverage data (.csv)**. 1. Select the projects and date range you want to include in the report. diff --git a/doc/user/group/saml_sso/example_saml_config.md b/doc/user/group/saml_sso/example_saml_config.md index c5f84510c57..70e01e1d9a9 100644 --- a/doc/user/group/saml_sso/example_saml_config.md +++ b/doc/user/group/saml_sso/example_saml_config.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference diff --git a/doc/user/group/saml_sso/group_sync.md b/doc/user/group/saml_sso/group_sync.md index c3edc01fe74..335c989c85f 100644 --- a/doc/user/group/saml_sso/group_sync.md +++ b/doc/user/group/saml_sso/group_sync.md @@ -1,6 +1,6 @@ --- type: reference, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -155,10 +155,11 @@ To integrate Microsoft Azure AD, you: To configure for a GitLab.com group: -1. Configure [SAML SSO for the group](../../../user/group/saml_sso/index.md). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your top-level group. +1. On the left sidebar, select **Search or go to** and find your top-level group. 1. Select **Settings > SAML SSO**. +1. Configure [SAML SSO for the group](../../../user/group/saml_sso/index.md). 1. In the **Microsoft Azure integration** section, select the **Enable Microsoft Azure integration for this group** checkbox. + This section will only be visible if SAML SSO is configured and enabled for the group. 1. Enter the **Tenant ID**, **Client ID**, and **Client secret** obtained earlier when configuring Azure Active Directory in the Azure Portal. 1. Optional. If using Azure AD for US Government or Azure AD China, enter the appropriate **Login API endpoint** and **Graph API endpoint**. The default values work for most organizations. 1. Select **Save changes**. @@ -166,7 +167,7 @@ To configure for a GitLab.com group: To configure for self-managed: 1. Configure [SAML SSO for the instance](../../../integration/saml.md). -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. In the **Microsoft Azure integration** section, select the **Enable Microsoft Azure integration for this group** checkbox. @@ -197,7 +198,7 @@ When global group memberships lock is enabled: To enable global group memberships lock: 1. [Configure SAML](../../../integration/saml.md) for your self-managed GitLab instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 41c2090f4bc..734679cf331 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -38,7 +38,7 @@ If you are having issues setting up your identity provider, see the To set up SSO with Azure as your identity provider: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Note the information on this page. 1. Go to Azure and [follow the instructions for configuring SSO for an application](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/add-application-portal-setup-sso). The following GitLab settings correspond to the Azure fields. @@ -71,7 +71,7 @@ For more information, see an [example configuration page](example_saml_config.md To set up Google Workspace as your identity provider: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Note the information on this page. 1. Follow the instructions for [setting up SSO with Google as your identity provider](https://support.google.com/a/answer/6087519?hl=en). The following GitLab settings correspond to the Google Workspace fields. @@ -115,7 +115,7 @@ View a demo of [how to configure SAML with Google Workspaces and set up Group Sy To set up SSO with Okta as your identity provider: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Note the information on this page. 1. Follow the instructions for [setting up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/main/). @@ -152,7 +152,7 @@ OneLogin supports its own [GitLab (SaaS) application](https://onelogin.service-n To set up OneLogin as your identity provider: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Note the information on this page. 1. If you use the OneLogin generic @@ -179,7 +179,7 @@ To set up OneLogin as your identity provider: To configure some identity providers, you need a GitLab metadata URL. To find this URL: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Copy the provided **GitLab metadata URL**. 1. Follow your identity provider's documentation and paste the metadata URL when it's requested. @@ -217,7 +217,7 @@ If the **NameID** is configured with the email address, [change the **NameID** f After you set up your identity provider to work with GitLab, you must configure GitLab to use it for authentication: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Complete the fields: - In the **Identity provider single sign-on URL** field, enter the SSO URL from your identity provider. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 9096824cc2c..a9b9bf26444 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -10,12 +10,14 @@ You can use the open standard System for Cross-domain Identity Management (SCIM) - Create users. - Remove users (deactivate SCIM identity). +- Re-add users (reactivate SCIM identity). GitLab SAML SSO SCIM doesn't support updating users. When SCIM is enabled for a GitLab group, membership of that group is synchronized between GitLab and an identity provider. The [internal GitLab group SCIM API](../../../development/internal_api/index.md#group-scim-api) implements part of [the RFC7644 protocol](https://www.rfc-editor.org/rfc/rfc7644). +Identity providers can use the [internal GitLab group SCIM API](../../../development/internal_api/index.md#group-scim-api) to develop a SCIM app. ## Configure GitLab @@ -25,7 +27,7 @@ Prerequisites: To configure GitLab SAML SSO SCIM: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > SAML SSO**. 1. Select **Generate a SCIM token**. 1. For configuration of your identity provider, save the: @@ -40,7 +42,7 @@ You can configure one of the following as an identity provider: - [Okta](#configure-okta). NOTE: -Other providers can work with GitLab but they have not been tested and are not supported. +Other providers can work with GitLab but they have not been tested and are not supported. You should contact the provider for support. GitLab support can assist by reviewing related log entries. ### Configure Azure Active Directory @@ -165,7 +167,8 @@ To configure Okta for SCIM: During the synchronization process, all new users: - Receive GitLab accounts. -- Are welcomed to their groups with an invitation email. You may want to warn your employees to expect this email. +- Are welcomed to their groups with an invitation email. + You can [bypass email confirmation with a verified domain](index.md#bypass-user-email-confirmation-with-verified-domains). The following diagram describes what happens when you add users to your SCIM app: @@ -216,10 +219,11 @@ Remove or deactivate a user on the identity provider to remove their access to: - The top-level group. - All subgroups and projects. -After the identity provider performs a sync based on its configured schedule, the user's membership is revoked and they -lose access. +After the identity provider performs a sync based on its configured schedule, +the user's membership is revoked and they lose access. -When you enable SCIM, this does not automatically remove existing users who do not have a SAML identity. +When you enable SCIM, this does not automatically remove existing users who do +not have a SAML identity. NOTE: Deprovisioning does not delete the GitLab user account. @@ -230,3 +234,14 @@ graph TD B -->|No| C[Nothing to do] B -->|Yes| D[GitLab removes user from GitLab group] ``` + +### Reactivate access + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/379149) in GitLab 16.0 [with a flag](../../feature_flags.md) named `skip_saml_identity_destroy_during_scim_deprovision`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121226) in GitLab 16.4. Feature flag `skip_saml_identity_destroy_during_scim_deprovision` removed. + +After a user is removed or deactivated through SCIM, you can reactivate that user by +adding them to the SCIM identity provider. + +After the identity provider performs a sync based on its configured schedule, +the user's SCIM identity is reactivated and their group memberships are restored. diff --git a/doc/user/group/saml_sso/troubleshooting.md b/doc/user/group/saml_sso/troubleshooting.md index 82703893834..cf9b9f5d4eb 100644 --- a/doc/user/group/saml_sso/troubleshooting.md +++ b/doc/user/group/saml_sso/troubleshooting.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -217,6 +217,18 @@ to [reset their password](https://gitlab.com/users/password/new) if both: - The account was provisioned by SCIM. - They are signing in with username and password for the first time. +### Message: "SAML Name ID and email address do not match your user account" **(PREMIUM SAAS)** + +Users might get an error that states "SAML Name ID and email address do not match your user account. Contact an administrator." +This means: + +- The NameID value sent by SAML does not match the existing SAML identity `extern_uid` value. +- Either the SAML response did not include an email address or the email address did not match the user's GitLab email address. + +The workaround is that a GitLab group Owner uses the [SAML API](../../../api/saml.md) to update the user's SAML `extern_uid`. +The `extern_uid` value must match the Name ID value sent by the SAML identity provider (IdP). Depending on the IdP configuration +this may be a generated unique ID, an email address, or other value. + ## Other user sign in issues ### Verify `NameID` @@ -254,12 +266,24 @@ If all users are receiving a `404` after signing in to the identity provider (Id - In the GitLab configuration by [matching it to the HTTPS endpoint of GitLab](../../../integration/saml.md#configure-saml-support-in-gitlab). - As the `Assertion Consumer Service URL` or equivalent when setting up the SAML app on your IdP. -- Verify if the `404` is related to [the user having too many groups assigned to them in their Azure IdP](group_sync.md#user-that-belongs-to-many-saml-groups-automatically-removed-from-gitlab-group) by checking: +- Verify if the `404` is related to [the user having too many groups assigned to them in their Azure IdP](group_sync.md#user-that-belongs-to-many-saml-groups-automatically-removed-from-gitlab-group). + +If a subset of users are receiving a `404` after signing in to the IdP, first verify audit events if the user gets added to the group and then immediately removed. Alternatively, if the user can successfully sign in, but they do not show as [a member of the top-level group](../index.md#search-a-group): - - If the user has group links configured. - - Audit events if the user gets added to the group and then immediately removed. +- Ensure the user has been [added to the SAML identity provider](index.md#user-access-and-management), and [SCIM](scim_setup.md) if configured. +- Ensure the user's SCIM identity's `active` attribute is `true` using the [SCIM API](../../../api/scim.md). + If the `active` attribute is `false`, you can do one of the following to possibly resolve the issue: -For configuration examples for some of the common providers, see the [example group SAML and SCIM configurations](example_saml_config.md). + - Trigger a sync for the user in the SCIM identity provider. For example, Azure has a "Provision on demand" option. + - Remove and re-add the user in the SCIM identity provider. + - Have the user [unlink their account](index.md#unlink-accounts) if possible, then [link their account](index.md#link-saml-to-your-existing-gitlabcom-account). + - Use the [internal SCIM API](../../../development/internal_api/index.md#update-a-single-scim-provisioned-user) to update the user's SCIM identity using your group's SCIM token. + If you do not know your group's SCIM token, reset the token and update the SCIM identity provider app with the new token. + Example request: + + ```plaintext + curl --request PATCH "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" --header "Authorization: Bearer <SCIM_TOKEN>" --data '{ "Operations": [{"op":"Replace","path":"active","value":"true"}] }' + ``` ### 500 error after login **(FREE SELF)** @@ -283,6 +307,21 @@ Make sure the ACS URL points to `https://gitlab.example.com/users/auth/saml/call If the ACS URL is correct, and you still have errors, review the other Troubleshooting sections. +#### 422 error with non-allowed email + +You might get an 422 error that states "Email is not allowed for sign-up. Please use your regular email address." + +This message might indicate that you must add or remove a domain from your domain allowlist or denylist settings. + +To implement this workaround: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings** > **General**. +1. Expand **Sign-up restrictions**. +1. Add or remove a domain as appropriate to **Allowed domains for sign-ups** and **Denied domains for sign-ups**. +1. Select **Save changes**. + ### User is blocked when signing in through SAML **(FREE SELF)** The following are the most likely reasons that a user is blocked when signing in through SAML: @@ -298,18 +337,6 @@ Pay particular attention to the following 403 errors: - `app_not_configured` - `app_not_configured_for_user` -## SAML Name ID and email address do not match your user account **(PREMIUM SAAS)** - -If users encounter the error `SAML Name ID and email address do not match your user account. Contact an administrator.` -this means: - -- The NameID value sent by SAML does not match the existing SAML identity `extern_uid` value. -- Either the SAML response did not include an email address or the email address did not match the user's GitLab email address. - -A GitLab group Owner can use the [SAML API](../../../api/saml.md) to update the user's SAML `extern_uid`. -The `extern_uid` value must match the Name ID value sent by the SAML identity provider (IdP). Depending on the IdP configuration -this may be a generated unique ID, an email address, or other value. - ## Message: "The member's email address is not linked to a SAML account" **(PREMIUM SAAS)** This error appears when you try to invite a user to a GitLab.com group (or subgroup or project within a group) that has [SAML SSO enforcement](index.md#sso-enforcement) enabled. @@ -318,3 +345,6 @@ If you see this message after trying to invite a user to a group: 1. Ensure the user has been [added to the SAML identity provider](index.md#user-access-and-management). 1. Ask the user to [link SAML to their existing GitLab.com account](index.md#link-saml-to-your-existing-gitlabcom-account), if they have one. Otherwise, ask the user to create a GitLab.com account by [accessing GitLab.com through the identity provider's dashboard](index.md#user-access-and-management), or by [signing up manually](https://gitlab.com/users/sign_up) and linking SAML to their new account. +1. Ensure the user is a [member of the top-level group](../index.md#search-a-group). + +Additionally, see [troubleshooting users receiving a 404 after sign in](#users-receive-a-404). diff --git a/doc/user/group/saml_sso/troubleshooting_scim.md b/doc/user/group/saml_sso/troubleshooting_scim.md index 7d2aa8faa99..63d10f3e932 100644 --- a/doc/user/group/saml_sso/troubleshooting_scim.md +++ b/doc/user/group/saml_sso/troubleshooting_scim.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -13,7 +13,18 @@ This section contains possible solutions for problems you might encounter. When you remove a user, they are removed from the group but their account is not deleted (see [remove access](scim_setup.md#remove-access)). -When the user is added back to the SCIM app, GitLab cannot create a new user because the user already exists. +When the user is added back to the SCIM app, GitLab does not create a new user because the user already exists. + +From August 11, 2023, the `skip_saml_identity_destroy_during_scim_deprovision` feature flag is enabled. + +For a user de-provisioned by SCIM from that date, their SAML identity is not removed. + +When that user is added back to the SCIM app: + +- Their SCIM identity `active` attribute is set to `true`. +- They can sign in using SSO. + +For users de-provisioned by SCIM before that date, their SAML identity is destroyed. To solve this problem: diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index 76aa77d026b..795967e9f91 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -1,11 +1,11 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, howto --- -# Group access tokens +# Group access tokens **(FREE)** With group access tokens, you can use a single token to: @@ -57,7 +57,7 @@ all projects that have visibility level set to [Internal](../../public_access.md To create a group access token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Access Tokens**. 1. Enter a name. The token name is visible to any user with permissions to view the group. 1. Enter an expiry date for the token: @@ -119,7 +119,7 @@ or API. However, administrators can use a workaround: To revoke a group access token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Access Tokens**. 1. Next to the group access token to revoke, select **Revoke**. @@ -138,6 +138,8 @@ token.revoke! ## Scopes for a group access token +> `k8s_proxy` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422408) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `k8s_proxy_pat`. Enabled by default. + The scope determines the actions you can perform when you authenticate with a group access token. | Scope | Description | @@ -149,12 +151,14 @@ The scope determines the actions you can perform when you authenticate with a gr | `read_repository` | Grants read access (pull) to all repositories within a group. | | `write_repository` | Grants read and write access (pull and push) to all repositories within a group. | | `create_runner` | Grants permission to create runners in a group. | +| `ai_features` | Grants permission to perform API actions for GitLab Duo. | +| `k8s_proxy` | Grants permission to perform Kubernetes API calls using the agent for Kubernetes in a group. | ## Enable or disable group access token creation To enable or disable group access token creation for all subgroups in a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Under **Permissions**, turn on or off **Users can create project access tokens and group access tokens in this group**. diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 4fb8b293334..e9064ff72a6 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -56,7 +56,7 @@ the private subgroup. To view the subgroups of a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select the **Subgroups and projects** tab. 1. To view a nested subgroup, expand a subgroup in the hierarchy list. @@ -77,14 +77,14 @@ Prerequisites: - At least the Maintainer role for a group to create subgroups for it. - The [role determined by a setting](#change-who-can-create-subgroups). These users can create subgroups even if group creation is - [disabled by an Administrator](../../../administration/admin_area.md#prevent-a-user-from-creating-groups) in the user's settings. + [disabled by an Administrator](../../../administration/admin_area.md#prevent-a-user-from-creating-top-level-groups) in the user's settings. NOTE: You cannot host a GitLab Pages subgroup website with a top-level domain name. For example, `subgroupname.example.io`. To create a subgroup: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find a parent group for the subgroup. +1. On the left sidebar, select **Search or go to** and find a parent group for the subgroup. 1. On the parent group's overview page, in the upper-right corner, select **New subgroup**. 1. Select **Create group**. 1. Fill in the fields. View a list of [reserved names](../../reserved_names.md) that cannot be used as group names. @@ -99,13 +99,13 @@ Prerequisite: To change who can create subgroups on a group: - As a user with the Owner role on the group: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Select a role from **Roles allowed to create subgroups**. 1. Select **Save changes**. - As an administrator: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Overview > Groups**. 1. In the group's row select **Edit**. @@ -161,7 +161,7 @@ Group permissions for a member can be changed only by: To see if a member has inherited the permissions from a parent group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Members**. Members list for an example subgroup _Four_: diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 1e042ab1924..0d91416dfa5 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -212,7 +212,7 @@ Prerequisites: To view value stream analytics for your group or project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. 1. To view metrics for a particular stage, select a stage below the **Filter results** text box. 1. Optional. Filter the results: @@ -302,7 +302,7 @@ Prerequisite: To view lifecycle metrics: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. Lifecycle metrics display below the **Filter results** text box. 1. Optional. Filter the results: @@ -316,7 +316,7 @@ To view lifecycle metrics: To view the [Value Streams Dashboard](../../analytics/value_streams_dashboard.md) and [DORA metrics](../../analytics/dora_metrics.md): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. 1. Below the **Filter results** text box, in the **Lifecycle metrics** row, select **Value Streams Dashboard / DORA**. 1. Optional. To open the new page, append this path `/analytics/dashboards/value_streams_dashboard` to the group URL @@ -331,7 +331,7 @@ Value stream analytics shows the median time spent by issues or merge requests i To view the median time spent in each stage by a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. 1. Optional. Filter the results: 1. Select the **Filter results** text box. @@ -355,7 +355,7 @@ group and time frame. To view tasks by type: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Analyze > Value stream analytics**. 1. Below the **Filter results** text box, select **Overview**. The **Tasks by type** chart displays below the **Total time** chart. 1. To switch between the task type, select the **Settings** (**{settings}**) dropdown list @@ -372,7 +372,7 @@ To view tasks by type: When you create a value stream, you can use GitLab default stages and hide or re-order them. You can also create custom stages in addition to those provided in the default template. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value Stream analytics**. 1. Select **Create new Value Stream**. 1. Enter a name for the value stream. @@ -396,7 +396,7 @@ If you have recently upgraded to GitLab Premium, it can take up to 30 minutes fo When you create a value stream, you can create and add custom stages that align with your own development workflows. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value Stream analytics**. 1. Select **Create value stream**. 1. For each stage: @@ -430,7 +430,7 @@ The first value stream uses standard timestamp-based events for defining the sta After you create a value stream, you can customize it to suit your purposes. To edit a value stream: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value Stream analytics**. 1. In the upper-right corner, select the dropdown list, then select a value stream. 1. Next to the value stream dropdown list, select **Edit**. @@ -449,7 +449,7 @@ After you create a value stream, you can customize it to suit your purposes. To To delete a custom value stream: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. In the upper-right corner, select the dropdown list, then select the value stream you would like to delete. 1. Select **Delete (name of value stream)**. 1. To confirm, select **Delete**. @@ -464,7 +464,7 @@ To delete a custom value stream: The **Total time chart** shows the average number of days it takes for development cycles to complete. The chart shows data for the last 500 workflow items. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Value stream analytics**. 1. Above the **Filter results** box, select a stage: - To view a summary of the cycle time for all stages, select **Overview**. diff --git a/doc/user/img/enable_AI_ML_features.png b/doc/user/img/enable_AI_ML_features.png Binary files differnew file mode 100644 index 00000000000..577fb367e4d --- /dev/null +++ b/doc/user/img/enable_AI_ML_features.png diff --git a/doc/user/img/forecast_deployment_frequency.png b/doc/user/img/forecast_deployment_frequency.png Binary files differnew file mode 100644 index 00000000000..4c8c6f47a08 --- /dev/null +++ b/doc/user/img/forecast_deployment_frequency.png diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index 5010c2e92a3..961914aac9c 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -34,17 +34,17 @@ your cluster's level. **Project-level clusters:** -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Kubernetes clusters**. **Group-level clusters:** -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Operate > Kubernetes clusters**. **Instance-level clusters:** -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Kubernetes**. diff --git a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md index 91429e203f3..de35b21c8b6 100644 --- a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md @@ -35,8 +35,8 @@ Start by [importing the example project by URL](../../../project/import/repo_by_ To import the project: -1. In GitLab, on the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**.. +1. In GitLab, on the left sidebar, select **Search or go to**. +1. Select **View all my projects**.. 1. On the right of the page, select **New project**. 1. Select **Import project**. 1. Select **Repository by URL**. diff --git a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md index ef51db097d9..4ec28a7c3c6 100644 --- a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md @@ -34,8 +34,8 @@ Start by [importing the example project by URL](../../../project/import/repo_by_ To import the project: -1. In GitLab, on the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. In GitLab, on the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Import project**. 1. Select **Repository by URL**. diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 3d717e0f473..96819860a2f 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -41,8 +41,8 @@ Start by [importing the example project by URL](../../../project/import/repo_by_ To import the project: -1. In GitLab, on the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. In GitLab, on the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select **Import project**. 1. Select **Repository by URL**. diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md index ad587154021..cace9c66fcf 100644 --- a/doc/user/infrastructure/clusters/index.md +++ b/doc/user/infrastructure/clusters/index.md @@ -13,9 +13,9 @@ To connect clusters to GitLab, use the [GitLab agent](../../clusters/agent/index WARNING: In GitLab 14.5, the certificate-based method to connect Kubernetes clusters to GitLab was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8), -as well as its related [features](#deprecated-features). In self-managed GitLab 15.0 and later, +as well as its related [features](#deprecated-features). In self-managed GitLab 17.0 and later, this feature is disabled by default. For GitLab SaaS users, this feature is available until -GitLab 15.6 for users who have at least one certificate-based cluster enabled in their namespace hierarchy. +GitLab 15.9 for users who have at least one certificate-based cluster enabled in their namespace hierarchy. For GitLab SaaS users that never used this feature previously, it is no longer available. The certificate-based Kubernetes integration with GitLab is deprecated. @@ -54,12 +54,9 @@ This feature flag re-enables the certificate-based Kubernetes integration. - [GitLab-managed clusters](../../project/clusters/gitlab_managed_clusters.md) - [Deploy applications through certificate-based connection](../../project/clusters/deploy_to_cluster.md) - [Cluster Management Project](../../clusters/management_project.md) -- [Cluster integrations](../../clusters/integrations.md) -- [Cluster cost management](../../clusters/cost_management.md) - [Cluster environments](../../clusters/environments.md) - [Show Canary Ingress deployments on deploy boards](../../project/canary_deployments.md#show-canary-ingress-deployments-on-deploy-boards-deprecated) - [Deploy Boards](../../project/deploy_boards.md) -- [Clusters health](manage/clusters_health.md) - [Web terminals](../../../administration/integration/terminal.md) ### Cluster levels diff --git a/doc/user/infrastructure/clusters/manage/clusters_health.md b/doc/user/infrastructure/clusters/manage/clusters_health.md deleted file mode 100644 index cf1b5585a0c..00000000000 --- a/doc/user/infrastructure/clusters/manage/clusters_health.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Deploy -group: Environments -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../index.md' ---- - -# Clusters health (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md index 193e3b70fba..ac61c0c11b4 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md @@ -90,7 +90,7 @@ After you have successfully installed Vault, you must and obtain the initial root token. You need access to your Kubernetes cluster that Vault has been deployed into to do this. To initialize the Vault, get a shell to one of the Vault pods running inside Kubernetes (typically this is done -by using the `kubectl` command line tool). After you have a shell into the pod, +by using the `kubectl` command-line tool). After you have a shell into the pod, run the `vault operator init` command: ```shell diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index 26b4b66ac63..25605cdac62 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -64,7 +64,7 @@ In each GitLab major release (for example, 15.0), the latest templates replace t To use a Terraform template: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project you want to integrate with Terraform. +1. On the left sidebar, select **Search or go to** and find your project you want to integrate with Terraform. 1. Select **Code > Repository**. 1. Edit your `.gitlab-ci.yml` file, use the `include` attribute to fetch the Terraform template: diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index 4a8f2f11e58..081e20b158e 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -116,7 +116,7 @@ inconsistent. Instead, use a remote storage resource. [initialized for CI/CD](#initialize-a-terraform-state-as-a-backend-by-using-gitlab-cicd). 1. Copy a pre-populated Terraform `init` command: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Terraform states**. 1. Next to the environment you want to use, select **Actions** (**{ellipsis_v}**) and select **Copy Terraform init command**. @@ -294,7 +294,7 @@ To read the Terraform state in the target project, you need at least the Develop To view Terraform state files: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Terraform states**. [An epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4563) to track improvements to this UI. diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 15a3703adbf..5787051a2c2 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -21,7 +21,7 @@ projects. To view the instance level Kubernetes clusters: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Kubernetes**. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index c724ae465b8..c996b29c9a2 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -691,11 +691,6 @@ To update the rendered references if the assignee, milestone, or health status c edit the comment or description and save it. For more information, see issue [420807](https://gitlab.com/gitlab-org/gitlab/-/issues/420807). -### Embedding metrics - -Metric charts can be embedded in GitLab Flavored Markdown. Read -[Embedding Metrics in GitLab flavored Markdown](../operations/metrics/embed.md) for more details. - ### Embedding Observability dashboards You can embed GitLab Observability UI dashboards descriptions and comments, for example in epics, issues, and MRs. diff --git a/doc/user/okrs.md b/doc/user/okrs.md index f9c242e7c2a..bb58dcf516b 100644 --- a/doc/user/okrs.md +++ b/doc/user/okrs.md @@ -60,7 +60,7 @@ Prerequisites: To create an objective: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Plan > Issues**. 1. In the upper-right corner, next to **New issue**, select the down arrow **{chevron-lg-down}** and then select **New objective**. 1. Select **New objective** again. @@ -77,7 +77,7 @@ Prerequisites: To view an objective: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Plan > Issues**. 1. [Filter the list of issues](project/issues/managing_issues.md#filter-the-list-of-issues) for `Type = objective`. @@ -91,7 +91,7 @@ Prerequisites: To view a key result: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Plan > Issues**. 1. [Filter the list of issues](project/issues/managing_issues.md#filter-the-list-of-issues) for `Type = key_result`. @@ -246,7 +246,7 @@ To refer to an objective or key result elsewhere in GitLab, you can use its full To copy the objective or key result reference to your clipboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your objective or key result to view it. 1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. @@ -266,7 +266,7 @@ For more information about creating comments by sending an email and the necessa To copy the objective's or key result's email address: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy objective email address** or **Copy key result email address**. diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 3f86e945492..3364d927a9b 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -11,7 +11,7 @@ including pipeline and alert status. To access the dashboard: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Operations**. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 0fcc44e8780..3c80e739465 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Composer packages in the Package Registry **(FREE ALL)** +# Composer packages in the Package Registry **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in GitLab 13.2. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. @@ -280,21 +280,53 @@ To install a package: composer req <package-name>:<package-version> ``` - If successful, you should see output indicating that the package installed successfully. - - You can also install from source (by pulling the Git repository directly) using the - `--prefer-source` option: - - ```shell - composer update --prefer-source - ``` - WARNING: Never commit the `auth.json` file to your repository. To install packages from a CI/CD job, consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages.md#satis) tool with your access token stored in a [GitLab CI/CD variable](../../../ci/variables/index.md) or in [HashiCorp Vault](../../../ci/secrets/index.md). +### Install from source + +You can install from source by pulling the Git repository directly. To do so, either: + +- Use the `--prefer-source` option: + + ```shell + composer update --prefer-source + ``` + +- In the `composer.json`, use the [`preferred-install` field under the `config` key](https://getcomposer.org/doc/06-config.md#preferred-install): + + ```json + { + ... + "config": { + "preferred-install": { + "<package name>": "source" + } + } + ... + } + ``` + +#### SSH access + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119739) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `composer_use_ssh_source_urls`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can +[enable the feature flag](../../../administration/feature_flags.md) named `composer_use_ssh_source_urls`. + +When you install from source, the `composer` configures an +access to the project's Git repository. +Depending on the project visibility, the access type is different: + +- On public projects, the `https` Git URL is used. Make sure you can [clone the repository with HTTPS](../../../gitlab-basics/start-using-git.md#clone-with-https). +- On internal or private projects, the `ssh` Git URL is used. Make sure you can [clone the repository with SSH](../../../gitlab-basics/start-using-git.md#clone-with-ssh). + +You can access the `ssh` Git URL from a CI/CD job using [SSH keys with GitLab CI/CD](../../../ci/ssh_keys/index.md). + ### Working with Deploy Tokens Although Composer packages are accessed at the group level, a group or project deploy token can be diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 72c5a1980b5..74152515198 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Conan packages in the Package Registry **(FREE ALL)** +# Conan packages in the Package Registry **(FREE ALL EXPERIMENT)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in GitLab 12.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. diff --git a/doc/user/packages/container_registry/delete_container_registry_images.md b/doc/user/packages/container_registry/delete_container_registry_images.md index 88a318d0770..0b91a9453a7 100644 --- a/doc/user/packages/container_registry/delete_container_registry_images.md +++ b/doc/user/packages/container_registry/delete_container_registry_images.md @@ -32,7 +32,7 @@ The online garbage collector is an instance-wide feature, and applies to all nam To delete container images using the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. For: - A group, select **Operate > Container Registry**. - A project, select **Deploy > Container Registry**. diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 3ec5ddb235e..1f95d2f9403 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -21,7 +21,7 @@ rate limits and speed up your pipelines. For more information about the Docker R You can view the Container Registry for a project or group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. For: - A group, select **Operate > Container Registry**. - A project, select **Deploy > Container Registry**. @@ -38,7 +38,7 @@ If a project is public, the Container Registry is also public. You can use the Container Registry **Tag Details** page to view a list of tags associated with a given container image: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. For: - A group, select **Operate > Container Registry**. - A project, select **Deploy > Container Registry**. @@ -54,7 +54,7 @@ tags on this page. You can share a filtered view by copying the URL from your br To download and run a container image hosted in the Container Registry: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. For: - A group, select **Operate > Container Registry**. - A project, select **Deploy > Container Registry**. @@ -115,7 +115,7 @@ The Container Registry is enabled by default. You can, however, remove the Container Registry for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand the **Visibility, project features, permissions** section and disable **Container Registry**. @@ -133,7 +133,7 @@ You can, however, change the visibility of the Container Registry for a project. For more information about the permissions that this setting grants to users, see [Container Registry visibility permissions](#container-registry-visibility-permissions). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand the section **Visibility, project features, permissions**. 1. Under **Container Registry**, select an option from the dropdown list: diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index fd781847855..2af16dcc85a 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -229,7 +229,7 @@ For self-managed instances, those settings can be updated in the [Rails console] They are also available in the [administrator area](../../admin_area/index.md): -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > CI/CD** 1. Expand **Container Registry**. diff --git a/doc/user/packages/container_registry/troubleshoot_container_registry.md b/doc/user/packages/container_registry/troubleshoot_container_registry.md index 34c86a90d49..13e14dfdeb4 100644 --- a/doc/user/packages/container_registry/troubleshoot_container_registry.md +++ b/doc/user/packages/container_registry/troubleshoot_container_registry.md @@ -91,7 +91,7 @@ The following procedure uses these sample project names: There may be a delay while the images are queued and deleted. 1. Change the path or transfer the project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand the **Advanced** section. 1. In the **Change path** text box, edit the path. @@ -127,3 +127,12 @@ time is set to 15 minutes. If you are using self-managed GitLab, an administrator can [increase the token duration](../../../administration/packages/container_registry.md#increase-token-duration). + +## Slow uploads when using `kaniko` to push large images + +When you push large images with `kaniko`, you might experience uncharacteristically long delays. + +This is typically a result of [a performance issue with `kaniko` and HTTP/2](https://github.com/GoogleContainerTools/kaniko/issues/2751). +The current workaround is to use HTTP/1.1 when pushing with `kaniko`. + +To use HTTP/1.1, set the `GODEBUG` environment variable to `"http2client=0"`. diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index b7fbeb96202..8e555204f80 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Debian packages in the Package Registry **(FREE ALL)** +# Debian packages in the Package Registry **(FREE ALL EXPERIMENT)** > - Debian API [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42670) in GitLab 13.5. > - Debian group API [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66188) in GitLab 14.2. @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The Debian package registry for GitLab is under development and isn't ready for production use. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6057) details the remaining -work and timelines to make it production ready. +work and timelines to make it production ready. Support for [Debian packages is an experiment](../package_registry/supported_package_managers.md), and has known security vulnerabilities. Publish Debian packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -27,8 +27,11 @@ Prerequisites: - The `dpkg-deb` binary must be installed on the GitLab instance. This binary is usually provided by the [`dpkg` package](https://wiki.debian.org/Teams/Dpkg/Downstream), installed by default on Debian and derivatives. +- Support for compression algorithm ZStandard requires version `dpkg >= + 1.21.18` from Debian 12 Bookworm or `dpkg >= 1.19.0.5ubuntu2` from Ubuntu + 18.04 Bionic Beaver. -## Enable the Debian API **(FREE SELF)** +## Enable the Debian API Debian repository support is still a work in progress. It's gated behind a feature flag that's **disabled by default**. @@ -50,7 +53,7 @@ To disable it: Feature.disable(:debian_packages) ``` -## Enable the Debian group API **(FREE SELF)** +## Enable the Debian group API The Debian group repository is also behind a second feature flag that is disabled by default. diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index 11b67e5cda3..02810bcb922 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -39,7 +39,7 @@ For a list of planned additions, view the To enable or turn off the Dependency Proxy for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Packages and registries**. 1. Expand the **Dependency Proxy** section. 1. To enable the proxy, turn on **Enable Proxy**. To turn it off, turn the toggle off. @@ -52,7 +52,7 @@ for the entire GitLab instance. To view the Dependency Proxy: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Operate > Dependency Proxy**. The Dependency Proxy is not available for projects. @@ -179,7 +179,7 @@ You can also use [custom CI/CD variables](../../../ci/variables/index.md#for-a-p To store a Docker image in Dependency Proxy storage: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Operate > Dependency Proxy**. 1. Copy the **Dependency Proxy image prefix**. 1. Use one of these commands. In these examples, the image is `alpine:latest`. @@ -367,7 +367,12 @@ see [issue 354826](https://gitlab.com/gitlab-org/gitlab/-/issues/354826). ### `exec format error` when running images from the dependency proxy -This error occurs if you try to use the dependency proxy on an ARM-based Docker install. +NOTE: +This issue was [resolved](https://gitlab.com/gitlab-org/gitlab/-/issues/325669) in GitLab 16.3. +For self managed instances that are 16.2 or earlier, you can update your instance to 16.3 +or use the workaround documented below. + +This error occurs if you try to use the dependency proxy on an ARM-based Docker install in GitLab 16.2 or earlier. The dependency proxy only supports the x86_64 architecture when pulling an image with a specific tag. As a workaround, you can specify the SHA256 of the image to force the dependency proxy @@ -378,5 +383,3 @@ docker pull ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/library/docker:20.10.3@sha ``` In this example, `bc9dcf5c8e5908845acc6d34ab8824bca496d6d47d1b08af3baf4b3adb1bd8fe` is the SHA256 of the ARM based image. - -For more information about the work to add support for other architectures, see [issue 325669](https://gitlab.com/gitlab-org/gitlab/-/issues/325669). diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index e309ab002c4..938093f2a27 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -127,7 +127,7 @@ or the UI. In the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Packages and registries**. 1. In the **Generic** row of the **Duplicate packages** table, turn off the **Allow duplicates** toggle. 1. Optional. In the **Exceptions** text box, enter a regular expression that matches the names and versions of packages to allow. diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 52f98ecf4dc..6dffd7371b6 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Go proxy for GitLab **(FREE ALL)** +# Go proxy for GitLab **(FREE ALL EXPERIMENT)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27376) in GitLab 13.1. > - It's deployed behind a feature flag, disabled by default. diff --git a/doc/user/packages/harbor_container_registry/index.md b/doc/user/packages/harbor_container_registry/index.md index 0cdaaf8ece0..d45b6ea7026 100644 --- a/doc/user/packages/harbor_container_registry/index.md +++ b/doc/user/packages/harbor_container_registry/index.md @@ -12,7 +12,7 @@ You can integrate the [Harbor container registry](../../../user/project/integrat You can view the Harbor Registry for a project or group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Operate > Harbor Registry**. You can search, sort, and filter images on this page. You can share a filtered view by copying the URL from your browser. @@ -29,7 +29,7 @@ Default settings for the Harbor integration at the project level are inherited f To download and run a Harbor image hosted in the GitLab Harbor Registry: 1. Copy the link to your container image: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. + 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Operate > Harbor Registry** and find the image you want. 1. Select the **Copy** icon next to the image name. @@ -39,7 +39,7 @@ To download and run a Harbor image hosted in the GitLab Harbor Registry: To view the list of tags associated with a specific artifact: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Go to **Operate > Harbor Registry**. 1. Select the image name to view its artifacts. 1. Select the artifact you want. @@ -57,7 +57,7 @@ To build and push to the Harbor Registry: To view these commands: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Operate > Harbor Registry**. 1. Select **CLI Commands**. @@ -65,7 +65,7 @@ To view these commands: To remove the Harbor Registry for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Integrations**. 1. Select **Harbor** under **Active integrations**. 1. Under **Enable integration**, clear the **Active** checkbox. diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index 57aee4fb4ca..b7888fe2d18 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Helm charts in the Package Registry **(FREE ALL)** +# Helm charts in the Package Registry **(FREE ALL BETA)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18997) in GitLab 14.1. diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 51755eda7bb..13d84fa3d99 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -480,7 +480,7 @@ To prevent users from publishing duplicate Maven packages, you can use the [Grap In the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Packages and registries**. 1. In the **Maven** row of the **Duplicate packages** table, turn off the **Allow duplicates** toggle. 1. Optional. In the **Exceptions** text box, enter a regular expression that matches the names and versions of packages to allow. diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 74daf9fd891..340df4a3c5f 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -432,6 +432,21 @@ choco push MyPackage.1.0.0.nupkg --source "https://gitlab.example.com/api/v4/pro When you publish a package with the same name or version as an existing package, the existing package is overwritten. +### Do not allow duplicate NuGet packages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293748) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `nuget_duplicates_option`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `nuget_duplicates_option`. +The feature is not ready for production use. + +To prevent users from publishing duplicate NuGet packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings). + +WARNING: +If the .nuspec file isn't located in the root of the package, the package might +not be recognized as a duplicate. + ## Install packages If multiple packages have the same name and version, when you install diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 4d9ad03afde..59184b811d4 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -45,7 +45,7 @@ For information on how to create and upload a package, view the GitLab documenta <!--- start_remove The following content will be removed on remove_date: '2023-11-22' --> WARNING: -[External authorization](../../admin_area/settings/external_authorization.md) will be enabled by default in GitLab 16.0. External authorization prevents personal access tokens and deploy tokens from accessing container and package registries and affects all users who use these tokens to access the registries. You can disable external authorization if you want to use personal access tokens and deploy tokens with the container or package registries. +In GitLab 16.0 and later, [external authorization](../../admin_area/settings/external_authorization.md) prevents personal access tokens and deploy tokens from accessing container and package registries and affects all users who use these tokens to access the registries. You can disable external authorization if you want to use personal access tokens and deploy tokens with the container or package registries. <!--- end_remove --> Authentication depends on the package manager being used. For more information, see the docs on the @@ -104,14 +104,9 @@ You can view which pipeline published the package, and the commit and user who t ### To import packages If you already have packages built in a different registry, you can import them -into your GitLab package registry with the [Packages Importer](https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer). +into your GitLab package registry with the [package importer](https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer). -The Packages Importer runs a CI/CD pipeline that [can import these package types](https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer#formats-supported): - -- Maven -- NPM -- NuGet -- PyPI +For a list of supported packages, see [Importing packages from other repositories](supported_functionality.md#importing-packages-from-other-repositories). ## Reduce storage usage @@ -163,7 +158,7 @@ Registry disables all Package Registry operations. To allow anyone to pull from the Package Registry, regardless of project visibility: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your private or internal project. +1. On the left sidebar, select **Search or go to** and find your private or internal project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Turn on the **Allow anyone to pull from Package Registry** toggle. diff --git a/doc/user/packages/package_registry/supported_functionality.md b/doc/user/packages/package_registry/supported_functionality.md index d3aa522f780..3e8852da808 100644 --- a/doc/user/packages/package_registry/supported_functionality.md +++ b/doc/user/packages/package_registry/supported_functionality.md @@ -39,7 +39,7 @@ Packages can be pulled from your project, group, or instance. | [Maven (with `mvn`)](../maven_repository/index.md) | Y | Y | Y | | [Maven (with `gradle`)](../maven_repository/index.md) | Y | Y | Y | | [Maven (with `sbt`)](../maven_repository/index.md) | Y | Y | Y | -| [npm](../npm_registry/index.md) | Y | N | Y | +| [npm](../npm_registry/index.md) | Y | Y | Y | | [NuGet](../nuget_repository/index.md) | Y | Y | N | | [PyPI](../pypi_repository/index.md) | Y | Y | N | | [Generic packages](../generic_packages/index.md) | Y | N | N | @@ -72,7 +72,7 @@ Requests for packages not found in your GitLab project are forwarded to the publ | [Go](../go_proxy/index.md) | N | | [Ruby gems](../rubygems_registry/index.md) | N | -### Deleting packages +## Deleting packages When package requests are forwarded to a public registry, deleting packages can be a [dependency confusion vulnerability](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610). @@ -89,6 +89,27 @@ To reduce the associated security risks, before deleting a package you can: - Instance administrators can disable forwarding in the [**Continuous Integration** section](../../../administration/settings/continuous_integration.md#package-registry-configuration) of the Admin Area. - Group owners can disable forwarding in the **Packages and Registries** section of the group settings. +## Importing packages from other repositories + +You can use GitLab pipelines to import packages from other repositories, such as Maven Central or Artifactory with the [package importer tool](https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer). + +| Package type | Importer available? | +|-------------------------------------------------------|---------------------| +| [Maven (with `mvn`)](../maven_repository/index.md) | Y | +| [Maven (with `gradle`)](../maven_repository/index.md) | Y | +| [Maven (with `sbt`)](../maven_repository/index.md) | Y | +| [npm](../npm_registry/index.md) | Y | +| [NuGet](../nuget_repository/index.md) | Y | +| [PyPI](../pypi_repository/index.md) | Y | +| [Generic packages](../generic_packages/index.md) | N | +| [Terraform](../terraform_module_registry/index.md) | N | +| [Composer](../composer_repository/index.md) | N | +| [Conan](../conan_repository/index.md) | N | +| [Helm](../helm_repository/index.md) | N | +| [Debian](../debian_repository/index.md) | N | +| [Go](../go_proxy/index.md) | N | +| [Ruby gems](../rubygems_registry/index.md) | N | + ## Allow or prevent duplicates **(FREE ALL)** By default, the GitLab package registry either allows or prevents duplicates based on the default of that specific package manager format. diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index 1c898ee6d92..361114e6f9e 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Ruby gems in the Package Registry **(FREE ALL)** +# Ruby gems in the Package Registry **(FREE ALL EXPERIMENT)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/803) in GitLab 13.10. diff --git a/doc/user/packages/yarn_repository/index.md b/doc/user/packages/yarn_repository/index.md index 262e15fe240..52accfc4fae 100644 --- a/doc/user/packages/yarn_repository/index.md +++ b/doc/user/packages/yarn_repository/index.md @@ -82,7 +82,7 @@ authentication token or use a private runner. To create an authentication token for your project or group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. On the left sidebar, select **Settings > Repository > Deploy Tokens**. 1. Create a deployment token with `read_package_registry` and `write_package_registry` scopes and copy the generated token. 1. On the left sidebar, select **Settings > CI/CD > Variables**. @@ -301,7 +301,7 @@ Then you can use `yarn add` to install your packages. ## Related topics - [npm documentation](../npm_registry/index.md#helpful-hints) -- [Yarn Migration Guide](https://yarnpkg.com/getting-started/migration) +- [Yarn Migration Guide](https://yarnpkg.com/migration/guide/) ## Troubleshooting diff --git a/doc/user/permissions.md b/doc/user/permissions.md index d19f98b98ed..dadfb75ed4e 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -66,10 +66,10 @@ The following table lists project permissions available for each role: | [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/proxy-based.md#on-demand-scans) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | +| [Application security](application_security/index.md):<br>Create, edit, delete [individual security policies](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | | [GitLab Agent for Kubernetes](clusters/agent/index.md):<br>View agents | | | ✓ | ✓ | ✓ | | [GitLab Agent for Kubernetes](clusters/agent/index.md):<br>Manage agents | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete [cleanup policies](packages/container_registry/delete_container_registry_images.md#use-a-cleanup-policy) | | | | ✓ | ✓ | @@ -132,9 +132,6 @@ The following table lists project permissions available for each role: | [Merge requests](project/merge_requests/index.md):<br>[Resolve a thread](project/merge_requests/index.md#resolve-a-thread) | | | ✓ | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Manage [merge approval rules](project/merge_requests/approvals/settings.md) (project settings) | | | | ✓ | ✓ | | [Merge requests](project/merge_requests/index.md):<br>Delete | | | | | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Manage user-starred metrics dashboards (6) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | | [Package registry](packages/index.md):<br>Pull a package | ✓ (1) | ✓ | ✓ | ✓ | ✓ | | [Package registry](packages/index.md):<br>Publish a package | | | ✓ | ✓ | ✓ | | [Package registry](packages/index.md):<br>Delete a package | | | | ✓ | ✓ | @@ -173,7 +170,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (7) | ✓ (7) | | [Projects](project/index.md):<br>View 2FA status of members | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Assign project to a [compliance framework](project/settings/index.md#add-a-compliance-framework-to-a-project) | | | | | ✓ | +| [Projects](project/index.md):<br>Assign project to a [compliance framework](project/working_with_projects.md#add-a-compliance-framework-to-a-project) | | | | | ✓ | | [Projects](project/index.md):<br>Archive project | | | | | ✓ | | [Projects](project/index.md):<br>Change project visibility level | | | | | ✓ | | [Projects](project/index.md):<br>Delete project | | | | | ✓ | @@ -384,7 +381,7 @@ The following table lists group permissions available for each role: | Delete [group wiki](project/wiki/group.md) pages | | | ✓ | ✓ | ✓ | | Edit [epic](group/epics/index.md) comments (posted by any user) | | | | ✓ | ✓ | | List group deploy tokens | | | | ✓ | ✓ | -| Manage [group push rules](group/access_and_permissions.md#group-push-rules) | | | | ✓ | ✓ | +| Manage [group push rules](group/access_and_permissions.md#group-push-rules) | | | | ✓ | ✓ | | View/manage group-level Kubernetes cluster | | | | ✓ | ✓ | | Create and manage compliance frameworks | | | | | ✓ | | Create/Delete group deploy tokens | | | | | ✓ | @@ -401,6 +398,7 @@ The following table lists group permissions available for each role: | View 2FA status of members | | | | | ✓ | | View [Billing](../subscriptions/gitlab_com/index.md#view-your-gitlab-saas-subscription) | | | | | ✓ (3) | | View group [Usage Quotas](usage_quotas.md) page | | | | | ✓ (3) | +| View group runners | | | | ✓ | ✓ | | Manage group runners | | | | | ✓ | | [Migrate groups](group/import/index.md) | | | | | ✓ | | Manage [subscriptions, and purchase storage and compute minutes](../subscriptions/gitlab_com/index.md) | | | | | ✓ | @@ -473,10 +471,10 @@ To work around the issue, give these users the Guest role or higher to any proje > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `customizable_roles`. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110810) in GitLab 15.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114524) in GitLab 15.10. -> - The ability for a custom role to view a vulnerability report [introduced](https://gitlab.com/groups/gitlab-org/-/epics/10160) in GitLab 16.1. - -FLAG: -On self-managed GitLab, by default the ability for a custom role to view a vulnerability report is not available. To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `elevated_guests`. On GitLab.com, this feature is available. +> - The ability for a custom role to view a vulnerability report [introduced](https://gitlab.com/groups/gitlab-org/-/epics/10160) in GitLab 16.1 [with a flag](../administration/feature_flags.md) named `custom_roles_vulnerability`. +> - Ability to view a vulnerability report [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123835) in GitLab 16.1. +> - [Feature flag `custom_roles_vulnerability` removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124049) in GitLab 16.2. +> - Ability to create and remove a custom role with the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393235) in GitLab 16.4. Custom roles allow group members who are assigned the Owner role to create roles specific to the needs of their organization. @@ -488,24 +486,65 @@ The following custom roles are available: - The Guest+1 role, which allows users with the Guest role to view code. - In GitLab 16.1 and later, you can create a custom role that can view vulnerability reports and change the status of the vulnerabilities. +- In GitLab 16.3 and later, you can create a custom role that can view the dependency list. +- In GitLab 16.4 and later, you can create a custom role that can approve merge requests. You can discuss individual custom role and permission requests in [issue 391760](https://gitlab.com/gitlab-org/gitlab/-/issues/391760). -When you enable the view vulnerability custom role for a user with the Guest role, that user has access to elevated permissions, and therefore: +When you enable a custom role for a user with the Guest role, that user has +access to elevated permissions, and therefore: - Is considered a [billable user](../subscriptions/self_managed/index.md#billable-users) on self-managed GitLab. - [Uses a seat](../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined) on GitLab.com. -This does not apply to the Guest+1 custom role because the `view_code` ability is excluded from this behavior. +This does not apply to Guest+1, a Guest custom role that only enables the `read_code` +permission. Users with that specific custom role are not considered billable users +and do not use a seat. ### Create a custom role -To enable custom roles for your group, a group member with the Owner role: +Prerequisites: + +- You must be an administrator for the self-managed instance, or have the Owner + role in the group you are creating the custom role in. +- The group must be in the Ultimate tier. +- You must have: + - At least one private project so that you can see the effect of giving a + user with the Guest role a custom role. The project can be in the group itself + or one of that group's subgroups. + - A [personal access token with the API scope](profile/personal_access_tokens.md#create-a-personal-access-token). + +#### GitLab SaaS + +Prerequisite: + +- You must have the Owner role in the group you are creating the custom role in. + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > Roles and Permissions**. +1. Select **Add new role**. +1. In **Base role to use as template**, select **Guest**. +1. In **Role name**, enter the custom role's title. +1. Select the **Permissions** for the new custom role. +1. Select **Create new role**. + +#### Self Managed GitLab Instances + +Prerequisite: + +- You must be an administrator for the self-managed instance you are creating the custom role in. + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. Select **Settings > Roles and Permissions**. +1. From the top dropdown list, select the group you want to create a custom role in. +1. Select **Add new role**. +1. In **Base role to use as template**, select **Guest**. +1. In **Role name**, enter the custom role's title. +1. Select the **Permissions** for the new custom role. +1. Select **Create new role**. -1. Makes sure that there is at least one private project in this group or one of - its subgroups, so that you can see the effect of giving a Guest a custom role. -1. Creates a personal access token with the API scope. -1. Uses [the API](../api/member_roles.md#add-a-member-role-to-a-group) to create a custom role for the root group. +To create a custom role, you can also [use the API](../api/member_roles.md#add-a-member-role-to-a-group). #### Custom role requirements @@ -518,7 +557,9 @@ You can see the required minimal access levels and abilities requirements in the | Ability | Minimal access level | Required ability | | -- | -- | -- | | `read_code` | Guest | - | +| `read_dependency` | Guest | - | | `read_vulnerability` | Guest | - | +| `admin_merge_request` | Guest | - | | `admin_vulnerability` | Guest | `read_vulnerability` | ### Associate a custom role with an existing group member @@ -529,53 +570,77 @@ the Owner role: 1. Invites a user as a direct member to the root group or any subgroup or project in the root group's hierarchy as a Guest. At this point, this Guest user cannot see any code on the projects in the group or subgroup. -1. Optional. If the Owner does not know the `ID` of the Guest user receiving a custom - role, finds that `ID` by making an [API request](../api/member_roles.md#list-all-member-roles-of-a-group). +1. Optional. If the Owner does not know the `id` of the Guest user receiving a custom + role, finds that `id` by making an [API request](../api/member_roles.md#list-all-member-roles-of-a-group). 1. Associates the member with the Guest+1 role using the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project) ```shell # to update a project membership - curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": '$MEMBER_ROLE_ID', "access_level": 10}' "https://example.gitlab.com/api/v4/projects/$ID/members/$GUEST_USER_ID" + curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" --data '{"member_role_id": '<member_role_id>', "access_level": 10}' "https://gitlab.example.com/api/v4/projects/<project_id>/members/<user_id>" # to update a group membership - curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": '$MEMBER_ROLE_ID', "access_level": 10}' "https://example.gitlab.com/api/v4/groups/$ID/members/$GUEST_USER_ID" + curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" --data '{"member_role_id": '<member_role_id>', "access_level": 10}' "https://gitlab.example.com/api/v4/groups/<group_id>/members/<user_id>" ``` Where: - - `$ID`: The `ID` or [URL-encoded path of the project or group](../api/rest/index.md#namespaced-path-encoding) associated with the membership receiving the custom role. - - `$MEMBER_ROLE_ID`: The `ID` of the member role created in the previous section. - - `$GUEST_USER_ID`: The `ID` of the Guest user receiving a custom role. + - `<project_id` and `<group_id>`: The `id` or [URL-encoded path of the project or group](../api/rest/index.md#namespaced-path-encoding) associated with the membership receiving the custom role. + - `<member_role_id>`: The `id` of the member role created in the previous section. + - `<user_id>`: The `id` of the user receiving a custom role. Now the Guest+1 user can view code on all projects associated with this membership. -### Remove a custom role from a group member +### Remove a custom role + +Prerequisite: + +- You must be an administrator or have the Owner role in the group you are removing the custom role from. + +You can remove a custom role from a group only if no group members have that role. + +To do this, you can either remove the custom role from all group members with that custom role, or remove those members from the group. + +#### Remove a custom role from a group member To remove a custom role from a group member, use the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project) and pass an empty `member_role_id` value. ```shell -curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": "", "access_level": 10}' "https://example.gitlab.com/api/v4/groups/$GROUP_PATH/members/$GUEST_USER_ID" +# to update a project membership +curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" --data '{"member_role_id": "", "access_level": 10}' "https://gitlab.example.com/api/v4/projects/<project_id>/members/<user_id>" + +# to update a group membership +curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer <your_access_token>" --data '{"member_role_id": "", "access_level": 10}' "https://gitlab.example.com/api/v4/groups/<group_id>/members/<user_id>" ``` -Now the user is a regular Guest. +#### Remove a group member with a custom role from the group -### Remove a custom role +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Manage > Members**. +1. On the member row you want to remove, select the vertical ellipsis + (**{ellipsis_v}**) and select **Remove member**. +1. In the **Remove member** confirmation dialog, do not select any checkboxes. +1. Select **Remove member**. -Removing a custom role also removes all members with that custom role from -the group. If you decide to delete a custom role, you must re-add any users with that custom -role to the group. +#### Delete the custom role -To remove a custom role from a group, a group member with -the Owner role: +After you have made sure no group members have that custom role, delete the +custom role. + +1. On the left sidebar, select **Search or go to**. +1. GitLab.com only. Select **Admin Area**. +1. Select **Settings > Roles and Permissions**. +1. Select **Custom Roles**. +1. In the **Actions** column, select **Delete role** (**{remove}**) and confirm. -1. Optional. If the Owner does not know the `ID` of a custom - role, finds that `ID` by making an [API request](../api/member_roles.md#list-all-member-roles-of-a-group). -1. Uses [the API](../api/member_roles.md#remove-member-role-of-a-group) to delete the custom role. +To delete a custom role, you can also [use the API](../api/member_roles.md#remove-member-role-of-a-group). +To use the API, you must know the `id` of the custom role. If you do not know this +`id`, find it by making an [API request](../api/member_roles.md#list-all-member-roles-of-a-group). ### Known issues -- Additional permissions can only be applied to users with the Guest role. -- If a user with a custom role is shared with a group or project, their custom role is not transferred over with them. The user has the regular Guest role in the new group or project. +- If a user with a custom role is shared with a group or project, their custom + role is not transferred over with them. The user has the regular Guest role in + the new group or project. - You cannot use an [Auditor user](../administration/auditor_users.md) as a template for a custom role. diff --git a/doc/user/product_analytics/index.md b/doc/user/product_analytics/index.md index 06a90af55c7..d90b2d5c882 100644 --- a/doc/user/product_analytics/index.md +++ b/doc/user/product_analytics/index.md @@ -4,26 +4,23 @@ group: Product Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Product analytics (Experiment) **(ULTIMATE ALL)** +# Product analytics **(ULTIMATE ALL EXPERIMENT)** > - Introduced in GitLab 15.4 as an [Experiment](../../policy/experiment-beta-support.md#experiment) feature [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. > - `cube_api_proxy` revised to only reference the [Product Analytics API](../../api/product_analytics.md) in GitLab 15.6. > - `cube_api_proxy` removed and replaced with `product_analytics_internal_preview` in GitLab 15.10. > - `product_analytics_internal_preview` replaced with `product_analytics_dashboards` in GitLab 15.11. -> - Snowplow integration introduced in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `product_analytics_snowplow_support`. Disabled by default. +> - Snowplow integration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398253) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `product_analytics_snowplow_support`. Disabled by default. +> - Snowplow integration feature flag `product_analytics_snowplow_support` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130228) in GitLab 16.4. FLAG: On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_dashboards`. On GitLab.com, this feature is not available. This feature is not ready for production use. -FLAG: -On self-managed GitLab, by default the Snowplow integration is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_snowplow_support`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. - This page is a work in progress, and we're updating the information as we add more features. For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/product-analytics/). +To leave feedback about Product Analytics bugs or functionality, please comment in [issue 391970](https://gitlab.com/gitlab-org/gitlab/-/issues/391970) or open a new issue with the label `group::product analytics`. ## How product analytics works @@ -66,7 +63,7 @@ flowchart TB > - `product_analytics_internal_preview` replaced with `product_analytics_dashboards` in GitLab 15.11. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flags](../../administration/feature_flags.md) named `product_analytics_dashboards` and `product_analytics_admin_settings`. +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flags](../../administration/feature_flags.md) named `product_analytics_dashboards`, `product_analytics_admin_settings`, and `combined_analytics_dashboards`. On GitLab.com, this feature is not available. This feature is not ready for production use. @@ -77,7 +74,7 @@ Prerequisite: - You must be an administrator of a self-managed GitLab instance. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand the **Analytics** tab and find the **Product analytics** section. @@ -93,8 +90,9 @@ Prerequisites: - Product analytics must be enabled at the instance-level. - You must have at least the Maintainer role for the project or group the project belongs to. +- The project must be in a group namespace. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Analytics**. 1. Expand **Configure** and enter the configuration values. 1. Select **Save changes**. @@ -105,6 +103,9 @@ To instrument code to collect data, use one or more of the existing SDKs: - [Browser SDK](https://gitlab.com/gitlab-org/analytics-section/product-analytics/gl-application-sdk-browser) - [Ruby SDK](https://gitlab.com/gitlab-org/analytics-section/product-analytics/gl-application-sdk-rb) +- [Python SDK](https://gitlab.com/gitlab-org/analytics-section/product-analytics/gl-application-sdk-python) +- [Node SDK](https://gitlab.com/gitlab-org/analytics-section/product-analytics/gl-application-sdk-node) +- [.NET SDK](https://gitlab.com/gitlab-org/analytics-section/product-analytics/gl-application-sdk-dotnet) ## Product analytics dashboards diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index f2f1921f0bb..c3612b787ac 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -1,6 +1,6 @@ --- type: reference -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -33,7 +33,7 @@ Prerequisite: To create a user manually: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select **New user**. diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 2694ed5f213..cf6ee61660f 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -1,6 +1,6 @@ --- type: howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -36,7 +36,7 @@ On GitLab.com, there is a seven day delay between a user deleting their own acco As an administrator, to delete a user account: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Overview > Users**. 1. Select a user. diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 83bdf883510..c7633b2c664 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -62,6 +62,8 @@ In GitLab 14.3 and later, your account email must be confirmed to enable 2FA. To enable 2FA with a one-time password: +<!-- vale gitlab.Substitutions = NO --> + 1. **In GitLab:** 1. Access your [**User settings**](../index.md#access-your-user-settings). 1. Select **Account**. @@ -70,7 +72,7 @@ To enable 2FA with a one-time password: 1. Install a compatible application. For example: - Cloud-based (recommended because you can restore access if you lose the hardware device): - [Authy](https://authy.com/). - - [Duo](https://duo.com/). + - [Cisco Duo](https://duo.com/). - Other (proprietary): - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en). - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app). @@ -85,6 +87,8 @@ To enable 2FA with a one-time password: 1. Enter your current password. 1. Select **Submit**. +<!-- vale gitlab.Substitutions = YES --> + If you entered the correct pin, GitLab displays a list of [recovery codes](#recovery-codes). Download them and keep them in a safe place. @@ -152,27 +156,33 @@ Configure FortiAuthenticator in GitLab. On your GitLab server: (Linux package installations) or [restart](../../../administration/restart_gitlab.md#self-compiled-installations) (self-compiled installations). -### Enable one-time password using Duo +<!-- vale gitlab.Substitutions = NO --> + +### Enable one-time password using Cisco Duo > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15760) in GitLab 15.10. FLAG: On self-managed GitLab, by default this feature is available. On GitLab.com this feature is not available. -You can use Duo as an OTP provider in GitLab. +You can use Cisco Duo as an OTP provider in GitLab. + +DUO® is a registered trademark of Cisco Systems, Inc., and/or its affiliates in the United States and certain other countries. #### Prerequisites -To use Duo as an OTP provider: +To use Cisco Duo as an OTP provider: + +- Your account must exist in both Cisco Duo and GitLab, with the same username in both applications. +- You must have [configured Cisco Duo](https://admin.duosecurity.com/) and have an integration key, secret key, and API hostname. -- Your account must exist in both Duo and GitLab, with the same username in both applications. -- You must have [configured Duo](https://admin.duosecurity.com/) and have an integration key, secret key, and API hostname. +For more information, see the [Cisco Duo API documentation](https://duo.com/docs/authapi). -For more information, see the [Duo API documentation](https://duo.com/docs/authapi). +GitLab 15.10 has been tested with Cisco Duo version D261.14 -GitLab 15.10 has been tested with Duo version D261.14 +#### Configure Cisco Duo in GitLab -#### Configure Duo in GitLab +<!-- vale gitlab.Substitutions = YES --> On your GitLab server: diff --git a/doc/user/profile/achievements.md b/doc/user/profile/achievements.md index 080ab41083b..1b875e984a1 100644 --- a/doc/user/profile/achievements.md +++ b/doc/user/profile/achievements.md @@ -4,7 +4,7 @@ group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Achievements (Experiment) **(FREE ALL)** +# Achievements **(FREE ALL EXPERIMENT)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113156) in GitLab 15.10 [with a flag](../../administration/feature_flags.md) named `achievements`. Disabled by default. diff --git a/doc/user/profile/img/profile-preferences-syntax-themes_v15_11.png b/doc/user/profile/img/profile-preferences-syntax-themes_v15_11.png Binary files differdeleted file mode 100644 index 39cd35f3b5b..00000000000 --- a/doc/user/profile/img/profile-preferences-syntax-themes_v15_11.png +++ /dev/null diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index a25260c3db9..cff18654292 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -1,6 +1,6 @@ --- type: index, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -353,7 +353,7 @@ A list of **Most Recent Activity** contributions is displayed. To view your activity: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Activity**. 1. Optional. To filter your activity by contribution type, in the **Your Activity** tab, select a tab: @@ -397,15 +397,15 @@ When you sign in, three cookies are set: - A session cookie called `_gitlab_session`. This cookie has no set expiration date. However, it expires based on its `session_expire_delay`. -- A session cookie called `about_gitlab_active_user`. - This cookie is used by the [marketing site](https://about.gitlab.com/) to determine if a user has an active GitLab session. No user information is passed to the cookie and it expires with the session. +- A session cookie called `gitlab_user`. + This cookie is used by the [marketing site](https://about.gitlab.com/) to determine if a user has an active GitLab session. No user information is passed to the cookie and it expires two weeks from login. - A persistent cookie called `remember_user_token`, which is set only if you selected **Remember me** on the sign-in page. -When you close your browser, the `_gitlab_session` and `about_gitlab_active_user` cookies are usually cleared client-side. +When you close your browser, the `_gitlab_session` and `gitlab_user` cookies are usually cleared client-side. When it expires or isn't available, GitLab: - Uses the `remember_user_token`cookie to get you a new `_gitlab_session` cookie and keep you signed in, even if you close your browser. -- Sets the `about_gitlab_active_user` to `true`. +- Sets the `gitlab_user` to `true`. When both the `remember_user_token` and `_gitlab_session` cookies are gone or expired, you must sign in again. diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index f1310bbb323..706065d4693 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -107,7 +107,7 @@ To select a notification level for a group, use either of these methods: Or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select the notification dropdown list, next to the bell icon (**{notifications}**). 1. Select the desired [notification level](#notification-levels). @@ -138,7 +138,7 @@ To select a notification level for a project, use either of these methods: Or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select the notification dropdown list, next to the bell icon (**{notifications}**). 1. Select the desired [notification level](#notification-levels). @@ -275,6 +275,7 @@ epics: | Merge Request | New note | Participants, Watchers, Subscribers, and Custom notification level with this event selected. Also anyone mentioned by username in the comment, with notification level "Mention" or higher. | | Merge Request | Pushed | Participants and Custom notification level with this event selected. | | Merge Request | Reassigned | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old assignee. | +| Merge Request | Review requested | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old reviewer. | | Merge Request | Reopened | Subscribers and participants. | | Merge Request | Title or description changed | Any new mentions by username. | | Pipeline | Failed | The author of the pipeline. | @@ -367,20 +368,21 @@ a merge request or an issue. The following table lists all GitLab-specific email headers: -| Header | Description | -| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | -| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | -| `X-GitLab-ConfidentialIssue` | The boolean value indicating issue confidentiality for notifications. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222908) in GitLab 16.0. | -| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | -| `X-GitLab-Group-Id` | The group's ID. Only present on notification emails for [epics](../group/epics/index.md). | -| `X-GitLab-Group-Path` | The group's path. Only present on notification emails for [epics](../group/epics/index.md) | -| `X-GitLab-NotificationReason` | The reason for the notification. [See possible values.](#x-gitlab-notificationreason). | -| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | -| `X-GitLab-Project-Id` | The project's ID. | -| `X-GitLab-Project-Path` | The project's path. | -| `X-GitLab-Project` | The name of the project the notification belongs to. | -| `X-GitLab-Reply-Key` | A unique token to support reply by email. | +| Header | Description | +| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | +| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | +| `X-GitLab-(Resource)-State` | The state of the resource the notification is for. The resource can be, for example, `Issue` or `MergeRequest`. The value can be `opened`, `closed`, `merged`, or `locked`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130967) in GitLab 16.4. | +| `X-GitLab-ConfidentialIssue` | The boolean value indicating issue confidentiality for notifications. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222908) in GitLab 16.0. | +| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | +| `X-GitLab-Group-Id` | The group's ID. Only present on notification emails for [epics](../group/epics/index.md). | +| `X-GitLab-Group-Path` | The group's path. Only present on notification emails for [epics](../group/epics/index.md) | +| `X-GitLab-NotificationReason` | The reason for the notification. [See possible values.](#x-gitlab-notificationreason). | +| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | +| `X-GitLab-Project-Id` | The project's ID. | +| `X-GitLab-Project-Path` | The project's path. | +| `X-GitLab-Project` | The name of the project the notification belongs to. | +| `X-GitLab-Reply-Key` | A unique token to support reply by email. | ### X-GitLab-NotificationReason diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 9161f5d4cf6..c3361040a00 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -1,6 +1,6 @@ --- type: concepts, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -104,7 +104,8 @@ To view the last time a token was used: ## Personal access token scopes -> Personal access tokens no longer being able to access container or package registries [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387721) in GitLab 16.0. +> - Personal access tokens no longer being able to access container or package registries [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387721) in GitLab 16.0. +> - `k8s_proxy` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422408) in GitLab 16.4 [with a flag](../../administration/feature_flags.md) named `k8s_proxy_pat`. Enabled by default. A personal access token can perform actions based on the assigned scopes. @@ -120,13 +121,15 @@ A personal access token can perform actions based on the assigned scopes. | `sudo` | Grants permission to perform API actions as any user in the system, when authenticated as an administrator. | | `admin_mode` | Grants permission to perform API actions as an administrator, when Admin Mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107875) in GitLab 15.8.) | | `create_runner` | Grants permission to create runners. | +| `ai_features` | Grants permission to perform API actions for GitLab Duo. | +| `k8s_proxy` | Grants permission to perform Kubernetes API calls using the agent for Kubernetes. | WARNING: If you enabled [external authorization](../admin_area/settings/external_authorization.md), personal access tokens cannot access container or package registries. If you use personal access tokens to access these registries, this measure breaks this use of these tokens. Disable external authorization to use personal access tokens with container or package registries. ## When personal access tokens expire -Personal access tokens expire on the date you define, at midnight UTC. +Personal access tokens expire on the date you define, at midnight, 00:00 AM UTC. - GitLab runs a check at 01:00 AM UTC every day to identify personal access tokens that expire in the next seven days. The owners of these tokens are notified by email. - GitLab runs a check at 02:00 AM UTC every day to identify personal access tokens that expire on the current date. The owners of these tokens are notified by email. diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 2df2674d539..f057e62694b 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -13,7 +13,7 @@ You can update your preferences to change the look and feel of GitLab. You can change the color theme of the GitLab UI. These colors are displayed on the left sidebar. Using individual color themes might help you differentiate between your different -GitLab instances. +GitLab instances. To change the color theme: @@ -25,7 +25,7 @@ To change the color theme: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28252) in GitLab 13.1 as an [Experiment](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28252). -Dark mode makes elements on the GitLab UI stand out on a dark background. +Dark mode makes elements on the GitLab UI stand out on a dark background. - To turn on Dark mode, Select **Preferences > Color theme > Dark Mode**. @@ -44,139 +44,304 @@ To change the syntax highlighting theme: 1. In the **Syntax highlighting theme** section, select a theme. 1. Select **Save changes**. -To view the updated syntax highlighting theme, refresh your project's page. +To view the updated syntax highlighting theme, refresh your project's page. To customize the syntax highlighting theme, you can also [use the Application settings API](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). Use `default_syntax_highlighting_theme` to change the syntax highlighting colors on a more granular level. -If these steps do not work, your programming language might not be supported by the syntax highlighters. +If these steps do not work, your programming language might not be supported by the syntax highlighters. For more information, view [Rouge Ruby Library](https://github.com/rouge-ruby/rouge) for guidance on code files and Snippets. View [Moncaco Editor](https://microsoft.github.io/monaco-editor/) and [Monarch](https://microsoft.github.io/monaco-editor/monarch.html) for guidance on the Web IDE. ## Change the diff colors -Diffs use two different background colors to show changes between versions of code. By default, the original file in red and the changes made in green. +Diffs use two different background colors to show changes between versions of code. By default, the original file is in red, and the changes are in green. To change the diff colors: 1. On the left sidebar, select your avatar. 1. Select **Preferences**. -1. Go to the **Diff colors** section. -1. Complete the fields. +1. Go to the **Diff colors** section. +1. Select a color or enter a color code. 1. Select **Save changes**. -1. Optional. Type a color code in the fields. + +To change back to the default colors, clear the **Color for removed lines** and **Color for added lines** text boxes and select **Save changes**. ## Behavior -The following settings allow you to customize the behavior of the GitLab layout -and default views of your dashboard and the projects' landing pages. +Use the **Behavior** section to customize the behavior and layout of your GitLab self-managed instance. You can change your layout width and choose the default content for your homepage, group and project overview pages. You have options to customize appearance and function, like whitespace rendering, file display, and text automation. -### Layout width +### Change the layout width on the UI -GitLab can be set up to use different widths depending on your liking. Choose -between the fixed (max. `1280px`) and the fluid (`100%`) application layout. +You can stretch content on the GitLab UI to fill the entire page. By default, page content is fixed at 1280 pixels wide. -NOTE: -While `1280px` is the standard max width when using fixed layout, some pages still use 100% width, depending on the content. +To change the layout width of your UI: -### Homepage +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Under **Layout width**, choose **Fixed** or **Fluid**. +1. Select **Save changes**. -This setting changes the behavior of the tanuki icon in the upper-left corner of GitLab. +### Choose your homepage -### Group overview content +Control what page you view when you select the GitLab logo (**{tanuki}**). You can set your homepage to be Projects (default), Your Groups, Your Activity, and other content. -The **Group overview content** dropdown list allows you to choose what information is -displayed on a group's home page. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. For **Homepage**, select a default. +1. Select **Save changes**. -You can choose between 2 options: +### Customize default content on your group overview page -- Details (default) -- [Security dashboard](../application_security/security_dashboard/index.md) +You can change the main content on your group overview page. Your group overview page is the page that shows when you select **Groups** on the left sidebar. You can customize the default content for your group overview page to the: -### Project overview content +- Details Dashboard (default), which includes an overview of group activities and projects. +- Security Dashboard, which might include group security policies and other security topics. -The **Project overview content** setting allows you to choose what content you want to -see on a project's home page. +For more information, view [Groups](../../user/group/index.md). -If **Files and Readme** is selected, you can show or hide the shortcut buttons above the file list on the project overview with the **Show shortcut buttons above files on project overview** setting. +To change the default content on your group overview page: -### Tab width +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. For **Group overivew content**, select an option. +1. Select **Save changes**. -You can set the displayed width of tab characters across various parts of -GitLab, for example, blobs, diffs, and snippets. +### Customize default content on your project overview page -NOTE: -Some parts of GitLab do not respect this setting, including the WebIDE, file -editor and Markdown editor. +Your project overview page is the page you view when you select **Project overview** on the left sidebar. You can set your main project overview page to the Activity page, the Readme file, and other content. + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. For **Project overivew content**, select an option. +1. Select **Save changes**. + +### Hide shortcut buttons + +Shortcut buttons precede the list of files on a project's overview page. These buttons provide links to parts of a project, such as the README file or license agreements. + +To hide shortcut buttons on the project overview page: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Clear the **Show shortcut buttons above files on project overview** checkbox. +1. Select **Save changes**. + +### Show whitespace characters in the Web IDE + +Whitespace characters are any blank characters in a text, such as spaces and indentations. You might use whitespace to structure content in code. If your programming language is sensitive to whitespaces, the Web IDE can detect changes to them. + +To render whitespace in the Web IDE: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Select the **Render whitespace characters in the Web IDE** checkbox. +1. Select **Save changes**. + +You can view changes to whitespace in diffs. + +To view diffs on the Web IDE, follow these steps: + +1. On the left sidebar, select **Source Control** (**{branch}**). +1. Under the **Changes** tab, select your file. + +### Show whitespace changes in diffs + +View changes to whitespace in diff files. For more information on whitespaces, view the previous task. + +To view changes to whitespace in diffs: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Select the **Show whitespace changes in diffs** checkbox. +1. Select **Save changes**. + +For more information on diffs, view [Change the diff colors](#change-the-diff-colors). + +### Show one file per page in a merge request + +The **Changes** tab lets you view all file changes in a merge request on one page. +Instead, you can choose to view one file at a time. + +To show one file per page on the **Changes** tab: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Select the **Show one file at a time on merge request's Changes tab** checkbox. +1. Select **Save changes**. + +Then, to move between files on the **Changes** tab, below each file, select the **Previous** and **Next** buttons. + +### Auto-enclose characters + +Automatically add the corresponding closing character to text when you type the opening character. For example, you can automatically insert a closing bracket when you type an opening bracket. This setting works only in description and comment boxes and for the following characters: `**"`, `'`, ```, `(`, `[`, `{`, `<`, `*`, `_**`. + +To auto-enclose characters in description and comment boxes: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Select the **Surround text selection when typing quotes or brackets** checkbox. +1. Select **Save changes**. + +In a description or comment box, you can now type a word, highlight it, then type an +opening character. Instead of replacing the text, the closing character is added to the end. + +### Automate new list items + +Create a new list item when you press <kbd>Enter</kbd> in a list in description and comment boxes. + +To add a new list item when you press the <kbd>Enter</kbd> key: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. Select the **Automatically add new list items** checkbox. +1. Select **Save changes**. + +### Change the tab width + +Change the default size of tabs in diffs, blobs, and snippets. The WebIDE, file editor, and Markdown editor do not support this feature. + +To adjust the default tab width: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Behavior** section. +1. For **Tab width**, enter a value. +1. Select **Save changes**. ## Localization -### Language +Change localization settings such as your language, calendar start day, and time preferences. -Select your preferred language from a list of supported languages. +### Change your display language on the GitLab UI -*This feature is experimental and translations are not complete yet.* +GitLab supports multiple languages on the UI. To help improve translations or request support for an unlisted language, view [Translating GitLab](../../development/i18n/translation.md). -### First day of the week +To choose a language for the GitLab UI: -The first day of the week can be customized for calendar views and date pickers. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Localization** section. +1. Under **Language**, select an option. +1. Select **Save changes**. + +You might need to refresh your page to view the updated language. -You can choose one of the following options as the first day of the week: +### Customize your contribution calendar start day -- Saturday -- Sunday -- Monday +Choose which day of the week the contribution calendar starts with. The contribution calendar shows project contributions over the past year. You can view this calendar on each user profile. To access your user profile: -If you select **System Default**, the first day of the week is set to the -[instance default](../../administration/settings/index.md#change-the-default-first-day-of-the-week). +- On the left sidebar, select your avatar > select your name or username. + +To change your contribution calendar start day: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Localization** section. +1. Under **First day of the week**, select an option. +1. Select **Save changes**. -## Time preferences +After you change your calendar start day, refresh your user profile page. -### Use relative times +### Show exact times instead of relative times > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65570) in GitLab 14.1. -You can select your preferred time format for the GitLab user interface: +Customize the format used to display times of activities on your group and project overview pages and user profiles. You can display times in a: + +- Relative format, for example `30 minutes ago`. +- Absolute format, for example `September 3, 2022, 3:57 PM`. + +To use relative times on the GitLab UI: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Go to the **Time preferences** section. +1. Clear the **Use relative times** checkbox. +1. Select **Save changes**. + +## User identities in CI job JSON web tokens + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387537) in GitLab 16.0. + +CI/CD jobs generate JSON web tokens, which can include a list of your external identities. +Instead of making separate API calls to get individual accounts, you can find your user identities in a single authentication token. + +External identities are not included by default. +To enable including external identities, see [Token payload](../../ci/secrets/id_token_authentication.md#token-payload). + +## Control follower engagement + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325558) in GitLab 16.0. + +Turn off the ability to follow or be followed by other GitLab users. By default, your user profile, including your name and profile photo, is public in the **Following** tabs of other users. When you deactivate this setting: + +- GitLab deletes all of your followers and followed connections. +- GitLab automatically removes your user profile from the pages of each connection. + +To remove the ability to be followed by and follow other users: + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Clear the **Enable follow users** checkbox. +1. Select **Save changes**. + +To access your **Followers** and **Following** tabs: -- Relative times, for example, `30 minutes ago`. -- Absolute times, for example, `May 18, 2021, 3:57 PM`. +- On the left sidebar, select your avatar > select your name or username. +- Select **Followers** or **Following**. -The times are formatted depending on your chosen language and browser locale. +## Enable Code Suggestions -To set your time preference: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../policy/experiment-beta-support.md#beta). -1. On the **Preferences** page, go to **Time preferences**. -1. Select the **Use relative times** checkbox to use relative times, - or clear the checkbox to use absolute times. +To enable [Code Suggestions](../../user/project/repository/code_suggestions/index.md): + +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Select the **Enable Code Suggestions** checkbox. 1. Select **Save changes**. NOTE: -This feature is experimental, and choosing absolute times might break certain layouts. -Open an issue if you notice that using absolute times breaks a layout. +If Code Suggestions are turned off [for the group](../../user/group/manage.md#enable-code-suggestions), then you cannot enable them for yourself. (Your setting has no effect.) -## User identities in CI job JSON web tokens +## Integrate your GitLab instance with third-party services + +Give third-party services access to your GitLab account. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387537) in GitLab 16.0. False by default. +### Integrate your GitLab instance with Gitpod -You can select to include the list of your external identities in the JSON Web Token information that is generated for a CI job. -For more information and examples, see [Token Payload](../../ci/secrets/id_token_authentication.md#token-payload). +Configure your GitLab instance with Gitpod when you want to launch and manage code directly from your GitLab browser. Gitpod automatically prepares and builds development environments for your projects. -## Integrations +To integrate with Gitpod: -Configure your preferences with third-party services which provide enhancements to your GitLab experience. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Find the **Integrations** section. +1. Select the **Enable Gitpod integration** checkbox. +1. Select **Save changes**. -### Sourcegraph +### Integrate your GitLab instance with Sourcegraph -NOTE: -This setting is only visible if Sourcegraph has been enabled by a GitLab administrator. +GitLab supports Sourcegraph integration for all public projects on GitLab. -Manage the availability of integrated code intelligence features powered by -Sourcegraph. View [the Sourcegraph feature documentation](../../integration/sourcegraph.md#enable-sourcegraph-in-user-preferences) -for more information. +To integrate with Sourcegraph: -### Gitpod +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. Find the **Integrations** section. +1. Select the **Enable integrated code intelligence on code views** checkbox. +1. Select **Save changes**. -Enable and disable the [GitLab-Gitpod integration](../../integration/gitpod.md). This is only -visible after the integration is configured by a GitLab administrator. View -[the Gitpod feature documentation](../../integration/gitpod.md) for more information. +You must be the administrator of the GitLab instance to configure GitLab with Sourcegraph. <!-- ## Troubleshooting diff --git a/doc/user/profile/service_accounts.md b/doc/user/profile/service_accounts.md index e593378ce4a..8845ee55e14 100644 --- a/doc/user/profile/service_accounts.md +++ b/doc/user/profile/service_accounts.md @@ -1,6 +1,6 @@ --- type: index, howto -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -22,6 +22,11 @@ A service account: You should use service accounts in pipelines or integrations where credentials must be set up and maintained without being impacted by changes in human user membership. +You can authenticate as a service account with a [personal access token](personal_access_tokens.md). +Service account users with a personal access token have the same abilities as a standard user. +This includes interacting with [registries](../packages/index.md) and using the personal access +token for [Git operations](personal_access_tokens.md#clone-repository-using-personal-access-token). + ## Create a service account The number of service accounts you can create is restricted by the number of service @@ -50,7 +55,8 @@ Prerequisite: The response includes the personal access token value. -1. Use the returned personal access token value to authenticate with the GitLab API as the service account user. +1. Make this service account a group or project member by [manually adding the service account user to the group or project](#add-a-service-account-to-subgroup-or-project). +1. Use the returned personal access token value to authenticate as the service account user. ### Self-managed GitLab @@ -63,14 +69,16 @@ Prerequisite: This service account is associated with the entire instance, not a specific group or project in the instance. -1. [Create a personal access token](../../api/users.md#create-service-account-user) +1. [Create a personal access token](../../api/users.md#create-a-personal-access-token) for the service account user. You define the scopes for the service account by [setting the scopes for the personal access token](personal_access_tokens.md#personal-access-token-scopes). The response includes the personal access token value. -1. Use the returned personal access token value to authenticate with the GitLab API as the service account user. +1. Make this service account a group or project member by + [manually adding the service account user to the group or project](#add-a-service-account-to-subgroup-or-project). +1. Use the returned personal access token value to authenticate as the service account user. ## Add a service account to subgroup or project @@ -88,27 +96,13 @@ A service account: - Can have different roles across multiple subgroups and projects of the same top level group. - On GitLab.com, only belongs to one top-level group. -### Add to a subgroup - -You can add the service account to a subgroup [through the UI](../group/index.md#add-users-to-a-group) -or API. - -To add the service account through the API, call the following endpoint: - -```shell -curl --request POST --header "PRIVATE-TOKEN: <ACCESS TOKEN>" --data "user_id=<service_account_user_id>&access_level=30" "https://gitlab.example.com/api/v4/groups/<subgroup_id>/members" -``` - -### Add to a project - -You can add the service account to a project [through the UI](../project/members/index.md#add-users-to-a-project) -or API. +### Add to a subgroup or project -To add the service account through the API, call the following endpoint: +You can add the service account to a subgroup or project through the: -```shell -curl --request POST --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" --data "user_id=<service_account_user_id>&access_level=30" "https://gitlab.example.com/api/v4/projects/<project_id>/members" -``` +- [API](../../api/members.md#add-a-member-to-a-group-or-project). +- [Group members UI](../group/index.md#add-users-to-a-group). +- [Project members UI](../project/members/index.md#add-users-to-a-project). ### Change a service account role in a subgroup or project diff --git a/doc/user/profile/user_passwords.md b/doc/user/profile/user_passwords.md index d8604bc712e..7882502a588 100644 --- a/doc/user/profile/user_passwords.md +++ b/doc/user/profile/user_passwords.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index c275d2b39db..39c1d228d63 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -130,7 +130,7 @@ If you find that you have to add the same badges to several projects, you may wa To add a new badge to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Badges**. 1. Select **Add badge**. @@ -152,7 +152,7 @@ A common project badge presents the GitLab CI pipeline status. To add this badge to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Badges**. 1. Under **Name**, enter _Pipeline Status_. @@ -181,7 +181,7 @@ If you need individual badges for each project, either: To add a new badge to a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Badges**. 1. Under "Link", enter the URL that the badges should point to and under @@ -203,7 +203,7 @@ Badges associated with a group can be edited or deleted only at the [group level You can view the exact link for your badges. Then you can use the link to embed the badge in your HTML or Markdown pages. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **General pipelines**. 1. In the **Pipeline status**, **Coverage report**, or **Latest release** sections, view the URLs for the images. @@ -270,7 +270,7 @@ https://gitlab.example.com/<project_path>/-/raw/<default_branch>/my-image.svg To add a new badge with a custom image to a group or project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > General**. 1. Expand **Badges**. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index 45d3834542b..d8b1b6fd413 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -56,7 +56,7 @@ cluster certificates: 1. Go to your: - Project's **Operate > Kubernetes clusters** page, for a project-level cluster. - Group's **Kubernetes** page, for a group-level cluster. - - **Main menu > Admin > Kubernetes**, for an instance-level cluster. + - The Admin Area's **Kubernetes** page, for an instance-level cluster. 1. Select **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, select **Amazon EKS** to display an `Account ID` and `External ID` needed for later steps. @@ -248,7 +248,7 @@ For example, the following policy document allows assuming a role whose name sta To configure Amazon authentication in GitLab, generate an access key for the IAM user in the Amazon AWS console, and follow these steps: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Amazon EKS**. diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 5127248baed..7e2c429c6bb 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -68,7 +68,7 @@ To add a Kubernetes cluster to your project, group, or instance: 1. Navigate to your: 1. Project's **{cloud-gear}** **Operate > Kubernetes clusters** page, for a project-level cluster. 1. Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - 1. **Main menu > Admin > Kubernetes** page, for an instance-level cluster. + 1. The Admin Area's **Kubernetes** page, for an instance-level cluster. 1. On the **Kubernetes clusters** page, select the **Connect with a certificate** option from the **Actions** dropdown list. 1. On the **Connect a cluster** page, fill in the details: 1. **Kubernetes cluster name** (required) - The name you wish to give the cluster. diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index a3a7cb35346..a087460d89e 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -63,7 +63,7 @@ cluster certificates: - Project's **{cloud-gear}** **Operate > Kubernetes clusters** page, for a project-level cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - - **Main menu > Admin > Kubernetes** page, for an instance-level cluster. + - The Admin Area's **Kubernetes** page, for an instance-level cluster. 1. Select **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, select **Google GKE**. 1. Connect your Google account if you haven't done already by selecting the diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 16ad95bb95c..a3263109bb1 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -19,7 +19,7 @@ When you successfully connect an existing cluster using cluster certificates, th 1. Go to your: - Project's **{cloud-gear}** **Operate > Kubernetes clusters** page, for a project-level cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - - **Main menu > Admin > Kubernetes** page, for an instance-level cluster. + - The Admin Area's **Kubernetes** page, for an instance-level cluster. 1. Select the name of the cluster you want to disable. 1. Toggle **GitLab Integration** off (in gray). 1. Select **Save changes**. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 58553fbb1c3..1a69b45b651 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -24,5 +24,5 @@ to a single project. To view project-level Kubernetes clusters: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Operate > Kubernetes clusters**. diff --git a/doc/user/project/codeowners/index.md b/doc/user/project/codeowners/index.md index 74974958910..d783471f0da 100644 --- a/doc/user/project/codeowners/index.md +++ b/doc/user/project/codeowners/index.md @@ -6,17 +6,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Code Owners **(PREMIUM ALL)** -> Moved to GitLab Premium in 13.9. - Use the Code Owners feature to define who has expertise for specific parts of your project's codebase. Define the owners of files and directories in a repository to: - **Require owners to approve changes.** Combine protected branches with Code Owners to require experts to approve merge requests before they merge into a protected branch. - **Identify owners.** Code Owner names are displayed on the files and directories they own: +  -Use Code Owners in combination with merge request +Combine Code Owners with merge request [approval rules](../merge_requests/approvals/rules.md) (either optional or required) to build a flexible approval workflow: @@ -42,13 +41,11 @@ For example: <iframe src="https://www.youtube-nocookie.com/embed/RoyBySTUSB0" frameborder="0" allowfullscreen> </iframe> </figure> -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> - ## View Code Owners of a file or directory To view the Code Owners of a file or directory: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. Go to the file or directory you want to see the Code Owners for. 1. Optional. Select a branch or tag. @@ -57,14 +54,14 @@ GitLab shows the Code Owners at the top of the page. ## Set up Code Owners -1. Create a `CODEOWNERS` file in your [preferred location](#code-owners-file). +1. Create a `CODEOWNERS` file in your [preferred location](#codeowners-file). 1. Define some rules in the file following the [Code Owners syntax reference](reference.md). Some suggestions: - Configure [All eligible approvers](../merge_requests/approvals/rules.md#code-owners-as-eligible-approvers) approval rule. - [Require Code Owner approval](../protected_branches.md#require-code-owner-approval-on-a-protected-branch) on a protected branch. 1. Commit your changes, and push them up to GitLab. -### Code Owners file +### `CODEOWNERS` file A `CODEOWNERS` file (with no extension) specifies the users or [shared groups](../members/share_project_with_groups.md) responsible for @@ -78,9 +75,17 @@ all others are ignored: 1. In the `docs` directory: `./docs/CODEOWNERS`. 1. In the `.gitlab` directory: `./.gitlab/CODEOWNERS`. -### Add a group as a Code Owner +For more information, see [Code Owners syntax and error handling](reference.md). + +#### When user or group change names + +When a user or group change their names, the `CODEOWNERS` isn't automatically updated with the new names. +To enter the new names, you must edit the file. -> Group and subgroup hierarchy support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32432) in GitLab 13.0. +Organizations using SAML SSO can [set usernames](../../../integration/saml.md#set-a-username) to +prevent users from being able to change their usernames. + +### Add a group as a Code Owner To set the members of a group or subgroup as a Code Owner: @@ -99,8 +104,6 @@ file.md @group-x @group-x/subgroup-y #### Group inheritance and eligibility -> Group and subgroup hierarchy support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32432) in GitLab 13.0. - ```mermaid graph TD A[Parent group X] -->|owns| B[Project A] @@ -188,9 +191,6 @@ Only one CODEOWNERS pattern per section is matched to a file path. ### Organize Code Owners by putting them into sections -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12137) in GitLab 13.2 [with a flag](../../../administration/feature_flags.md) named `sectional_codeowners`. Disabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42389) in GitLab 13.4. Feature flag `sectional_codeowners` removed. - You can organize Code Owners by putting them into named sections. You can use sections for shared directories, so that multiple @@ -215,9 +215,10 @@ The following image shows a **Groups** and **Documentation** section: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371711) in GitLab 15.11 [with a flag](../../../administration/feature_flags.md) named `codeowners_default_owners`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115888) in GitLab 15.11. Feature flag `codeowners_default_owners` removed. -If multiple file paths inside a section share the same ownership, define a default -Code Owner for the section. All paths in that section inherit this default, unless -you override the section default on a specific line. +If multiple file paths inside a section share the same ownership, define default +Code Owners for the section. +All paths in that section inherit this default, unless you override the section +default on a specific line. Default owners are applied when specific owners are not specified for file paths. Specific owners defined beside the file path override default owners: @@ -227,7 +228,7 @@ Specific owners defined beside the file path override default owners: docs/ README.md -[Database] @database-team +[Database] @database-team @agarcia model/db/ config/db/database-setup.md @docs-team ``` @@ -235,9 +236,14 @@ config/db/database-setup.md @docs-team In this example: - `@docs-team` owns all items in the `Documentation` section. -- `@database-team` owns all items in the `Database` section except +- `@database-team` and `@agarcia` own all items in the `Database` section except `config/db/database-setup.md`, which has an override assigning it to `@docs-team`. +Compare this behavior to when you use [regular entries and sections together](#use-regular-entries-and-sections-together), +when entries in sections don't override entries without sections. + +#### Use default owners and optional sections together + To combine the syntax for default owners with [optional sections](#make-a-code-owners-section-optional) and required approvals, place default owners at the end: @@ -251,6 +257,38 @@ model/db/ config/db/database-setup.md @docs-team ``` +#### Use regular entries and sections together + +If you set a default Code Owner for a path outside a section, their approval is always required, and +the entry isn't overridden. +Entries without sections are treated as if they were another, unnamed section: + +```plaintext +# Required for all files +* @general-approvers + +[Documentation] @docs-team +docs/ +README.md +*.txt + +[Database] @database-team +model/db/ +config/db/database-setup.md @docs-team +``` + +In this example: + +- `@general-approvers` owns all items everywhere, without overrides. +- `@docs-team` owns all items in the `Documentation` section. +- `@database-team` owns all items in the `Database` section except + `config/db/database-setup.md`, which has an override assigning it to `@docs-team`. +- A merge request that modifies `model/db/CHANGELOG.txt` would require three approvals: one from each + of the `@general-approvers`,`@docs-team`, and `@database-team` groups. + +Compare this behavior to when you use only [default owners for sections](#set-default-owner-for-a-section), +when specific entries within a section override the section default. + #### Sections with duplicate names If multiple sections have the same name, they are combined. @@ -275,8 +313,6 @@ entries under **Database**. The entries defined under the sections **Documentati #### Make a Code Owners section optional -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232995) in GitLab 13.8. - You can designate optional sections in your Code Owners file. Prepend the section name with the caret `^` character to treat the entire section as optional. Optional sections enable you to designate responsible parties for various parts @@ -314,14 +350,14 @@ section is marked as optional. You can require multiple approvals for the Code Owners sections under the Approval Rules area in merge requests. Append the section name with a number `n` in brackets. This requires `n` approvals from the Code Owners in this section. -Please note valid entries for `n` are integers `≥ 1`. `[1]` is optional as it is the default. Invalid values for `n` are treated as `1`. +Valid entries for `n` are integers `≥ 1`. `[1]` is optional because it is the default. Invalid values for `n` are treated as `1`. WARNING: [Issue #384881](https://gitlab.com/gitlab-org/gitlab/-/issues/385881) proposes changes to the behavior of this setting. Do not intentionally set invalid values. They may become valid in the future, and cause unexpected behavior. -Please confirm you enabled `Require approval from code owners` in `Settings > Repository > Protected branches`, otherwise the Code Owner approvals will be optional. +Make sure you enabled `Require approval from code owners` in `Settings > Repository > Protected branches`, otherwise the Code Owner approvals are optional. In this example, the `[Documentation]` section requires 2 approvals: @@ -337,7 +373,7 @@ The `Documentation` Code Owners section under the **Approval Rules** area displa  -### Allowed to Push +### Allowed to push Users who are **Allowed to push** can choose to create a merge request for their changes, or push the changes directly to a branch. If the user @@ -350,12 +386,15 @@ and release tooling. All changes from users _without_ the **Allowed to push** permission must be routed through a merge request. -## Technical Resources +## Related topics -[Code Owners development guidelines](../../../development/code_owners/index.md) +- [Syntax reference](reference.md) +- [Development guidelines](../../../development/code_owners/index.md) ## Troubleshooting +When working with Code Owners, you might encounter the following issues. + For more information about how the Code Owners feature handles errors, see the [Code Owners reference](reference.md). @@ -363,7 +402,8 @@ For more information about how the Code Owners feature handles errors, see the A Code Owner approval rule is optional if any of these conditions are true: -- The user or group are not a member of the project. Code Owners [cannot inherit from parent groups](https://gitlab.com/gitlab-org/gitlab/-/issues/288851/). +- The user or group is not a member of the project. + Code Owners [cannot inherit members from parent groups](https://gitlab.com/gitlab-org/gitlab/-/issues/288851/). - [Code Owner approval on a protected branch](../protected_branches.md#require-code-owner-approval-on-a-protected-branch) has not been set up. - The section is [marked as optional](#make-a-code-owners-section-optional). @@ -383,7 +423,15 @@ if any of these conditions are true: member of the Code Owner group. - Current user is an external user who does not have permission to the internal Code Owner group. -### Approval rule is invalid. GitLab has approved this rule automatically to unblock the merge request +### Approval rule is invalid + +You might get an error that states: + +```plaintext +Approval rule is invalid. +GitLab has approved this rule automatically to unblock the merge request. +``` + +This issue occurs when an approval rule uses a Code Owner that is not a direct member of the project. -This message may appear if an approval rule uses a Code Owner that is not a direct member of the project. -Check that the group or user has been invited to the project. +The workaround is to check that the group or user has been invited to the project. diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index 395873d3107..91a3d71642d 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -21,7 +21,7 @@ type: howto, reference WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. [An epic exists](https://gitlab.com/groups/gitlab-org/-/epics/2493) -to add this functionality to the [agent](../index.md). +to add this functionality to the [agent](../clusters/agent/index.md). FLAG: On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `certificate_based_clusters`. diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index f0d84616c24..c0a50dade31 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -64,7 +64,7 @@ Deploy keys work even if the user who created them is removed from the group or To view the deploy keys available to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Deploy keys**. @@ -82,7 +82,7 @@ Prerequisites: - [Generate an SSH key pair](../../ssh.md#generate-an-ssh-key-pair). Put the private SSH key on the host that requires access to the repository. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Deploy keys**. 1. Select **Add new key**. @@ -104,7 +104,7 @@ Prerequisites: To create a public deploy key: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Deploy Keys**. 1. Select **New deploy key**. @@ -122,7 +122,7 @@ Prerequisites: To grant a public deploy key access to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Deploy keys**. 1. Select **Publicly accessible deploy keys**. @@ -139,7 +139,7 @@ Prerequisites: To edit the project access permissions of a deploy key: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Deploy keys**. 1. In the key's row, select **Edit** (**{pencil}**). @@ -156,7 +156,7 @@ Prerequisites: To disable a deploy key: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Deploy keys**. 1. Select **Disable** (**{cancel}**). @@ -192,7 +192,7 @@ If you need to find the keys that belong to a non-member or blocked user, you can use [the Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) to identify unusable deploy keys using a script similar to the following: ```ruby -ghost_user_id = User.ghost.id +ghost_user_id = Users::Internal.ghost.id DeployKeysProject.with_write_access.find_each do |deploy_key_mapping| project = deploy_key_mapping.project diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 41fb0018be9..8b7e185508b 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Deploy tokens **(FREE ALL)** -> [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) package registry scopes in GitLab 13.0. - You can use a deploy token to enable authentication of deployment tasks, independent of a user account. In most cases you use a deploy token from an external host, like a build server or CI/CD server. @@ -92,7 +90,7 @@ Prerequisites: - You must have at least the Maintainer role for the project or group. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Repository**. 1. Expand **Deploy tokens**. 1. Select **Add token**. @@ -112,7 +110,7 @@ Prerequisites: To revoke a deploy token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Repository**. 1. Expand **Deploy tokens**. 1. In the **Active Deploy Tokens** section, by the token you want to revoke, select **Revoke**. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 4da49c4fb12..97b350c8692 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -16,7 +16,7 @@ You might want to use these templates: - For different stages of your workflow, for example, feature proposal, feature improvement, or a bug report. - For every issue or merge request for a specific project, so the layout is consistent. -- For a [Service Desk email template](service_desk/index.md#use-a-custom-template-for-service-desk-tickets). +- For a [Service Desk email template](service_desk/configure.md#use-a-custom-template-for-service-desk-tickets). For description templates to work, they must be: @@ -32,7 +32,7 @@ directory in your repository. To create an issue description template: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. Next to the default branch, select **{plus}**. 1. Select **New file**. @@ -52,7 +52,7 @@ that depend on the contents of commit messages and branch names. To create a merge request description template for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. Next to the default branch, select **{plus}**. 1. Select **New file**. @@ -110,7 +110,7 @@ You can set a description template at the **instance level** for issues and merge requests by using an [instance template repository](../admin_area/settings/instance_template_repository.md). You can also use the instance template repository for file templates. -You might also be interested [project templates](../admin_area/custom_project_templates.md) +You might also be interested in [project templates](../admin_area/custom_project_templates.md) that you can use when creating a new project in the instance. ### Set group-level description templates **(PREMIUM ALL)** @@ -118,13 +118,18 @@ that you can use when creating a new project in the instance. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321247) in GitLab 14.0. -With **group-level** description templates, you can select a repository within the group to store -your templates. Then, you can access these templates from other projects in the group. +With **group-level** description templates, you can select a project within the group to store +your templates. Then, you can access these templates in other projects in the group. As a result, you can use the same templates in issues and merge requests in all the group's projects. +Prerequisites: + +- You must have the Owner role for the group. +- The project must be a direct child of the group. + To re-use templates [you've created](../project/description_templates.md#create-an-issue-template): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Templates**. 1. From the dropdown list, select your template project as the template repository at group level. @@ -155,7 +160,7 @@ To set a default description template for merge requests, either: This [doesn't overwrite](#priority-of-default-description-templates) the default template if one has been set in the project settings. - Users on GitLab Premium and Ultimate: set the default template in project settings: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Default description template for merge requests** section, fill in the text area. 1. Select **Save changes**. @@ -167,7 +172,7 @@ To set a default description template for issues, either: This [doesn't overwrite](#priority-of-default-description-templates) the default template if one has been set in the project settings. - Users on GitLab Premium and Ultimate: set the default template in project settings: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Default description template for issues**. 1. Fill in the text area. diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 88aa9446787..783f5f3ee7c 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -151,7 +151,7 @@ You can push files to GitLab whether they're locked or unlocked. NOTE: Although multi-branch file locks can be created and managed through the Git LFS -command line interface, file locks can be created for any file. +command-line interface, file locks can be created for any file. ### View exclusively-locked files @@ -221,7 +221,7 @@ similar functionality for locked files is discussed in To view and remove file locks: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Locked files**. This list shows all the files locked either through LFS or GitLab UI. diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 0f612d5b222..bafcbbb9cbd 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -36,7 +36,7 @@ When importing: - [Bitbucket Cloud integration](../../../integration/bitbucket.md) must be enabled. If that integration is not enabled, ask your GitLab administrator to enable it. The Bitbucket Cloud integration is enabled by default on GitLab.com. -- [Bitbucket Cloud import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your +- [Bitbucket Cloud import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The Bitbucket Cloud import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. @@ -117,5 +117,5 @@ current Bitbucket public name, and reconnect if there's a mismatch: 1. Following reconnection, the user should use the API again to verify that their `extern_uid` in the GitLab database now matches their current Bitbucket public name. -The importer must then [delete the imported project](../../project/working_with_projects.md#delete-a-project) +The importer must then [delete the imported project](../../project/settings/index.md#delete-a-project) and import again. diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index a80ce95c163..4afe23a29fa 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -32,7 +32,7 @@ You can import Bitbucket repositories to GitLab. > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. -- [Bitbucket Server import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [Bitbucket Server import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The Bitbucket Server import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. @@ -87,6 +87,9 @@ original creator. The importer creates any new namespaces (groups) if they don't exist. If the namespace is taken, the repository imports under the namespace of the user who started the import process. +The importer attempts to find reviewers by their email address in the GitLab user database. If they don't exist in GitLab, they cannot be added as reviewers to a +merge request. + ### User assignment by username > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218609) in GitLab 13.4 [with a flag](../../../administration/feature_flags.md) named `bitbucket_server_user_mapping_by_username`. Disabled by default. diff --git a/doc/user/project/import/cvs.md b/doc/user/project/import/cvs.md index f5f924ef603..90ed6cfdb2c 100644 --- a/doc/user/project/import/cvs.md +++ b/doc/user/project/import/cvs.md @@ -39,10 +39,10 @@ The following list illustrates the main differences between CVS and Git: your state in version control, then you merge the other developer's changes. You can also ask the other developer to do the merge and resolve any conflicts themselves. -- **Signed commits.** Git supports signing your commits with GPG for additional +- **Signed commits.** Git supports + [signing your commits](../repository/signed_commits/index.md) for additional security and verification that the commit indeed came from its original author. - GitLab can [integrate with GPG](../repository/gpg_signed_commits/index.md) - and show whether a signed commit is correctly verified. + GitLab shows whether a signed commit is correctly verified. _Some of the items above were taken from this great [Stack Overflow post](https://stackoverflow.com/a/824241/974710). For a more diff --git a/doc/user/project/import/fogbugz.md b/doc/user/project/import/fogbugz.md index 85c24886687..2bce6708556 100644 --- a/doc/user/project/import/fogbugz.md +++ b/doc/user/project/import/fogbugz.md @@ -19,7 +19,7 @@ users. > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. -- [FogBugz import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [FogBugz import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The FogBugz import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. diff --git a/doc/user/project/import/gitea.md b/doc/user/project/import/gitea.md index 98457b96dde..9629ec03443 100644 --- a/doc/user/project/import/gitea.md +++ b/doc/user/project/import/gitea.md @@ -32,7 +32,7 @@ on the issue about the original Gitea author. > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. - Gitea version 1.0.0 or later. -- [Gitea import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [Gitea import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The Gitea import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 2079c61da31..75d25e29bd9 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -28,6 +28,9 @@ When importing projects: - If a user referenced in the project is not found in the GitLab database, the project creator is set as the author and assignee. The project creator is usually the user that initiated the import process. A note on the issue mentioning the original GitHub author is added. +- Reviewers assigned to GitHub pull requests that do not exist in GitLab are not imported. In this case, the import + creates comments describing that non-existent users were added as reviewers and approvers. However, the actual + reviewer status and approval are not applied to the merge request in GitLab. - You can change the target namespace and target repository name before you import. - The importer also imports branches on forks of projects related to open pull requests. These branches are imported with a naming scheme similar to `GH-SHA-username/pull-request-number/fork-name/branch`. This may lead to @@ -43,7 +46,7 @@ For an overview of the import process, see [How to migrate from GitHub to GitLab To import projects from GitHub: -- [GitHub import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [GitHub import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The GitHub import source is enabled by default on GitLab.com. - You must have at least the Maintainer role on the destination group to import to. @@ -55,12 +58,8 @@ To import projects from GitHub: When issues and pull requests are being imported, the importer attempts to find their GitHub authors and assignees in the database of the GitLab instance. Pull requests are called _merge requests_ in GitLab. For the importer to succeed, matching email addresses are required. -- GitHub accounts must have a GitHub public-facing email address is populated. This means all comments and contributions - are properly mapped to the same user in GitLab. GitHub Enterprise does not require this field to be populated so you - may have to add it on existing accounts. - -Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383047), if you are using GitHub as an OmniAuth provider, ensure that the URL -perimeter is specified in the [OmniAuth configuration](../../../integration/github.md#enable-github-oauth-in-gitlab). +- GitHub accounts must have a GitHub public-facing email address so that all comments and contributions can be properly mapped to + the same user in GitLab. GitHub Enterprise does not require this field to be populated so you might have to add it on existing accounts. ### Importing from GitHub Enterprise to self-managed GitLab @@ -68,7 +67,7 @@ If you are importing from GitHub Enterprise to a self-managed GitLab instance: - You must first enable the [GitHub integration](../../../integration/github.md). - GitHub must be enabled as an import source in the - [Admin Area](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources). + [Admin Area](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). - For GitLab 15.10 and earlier, you must add `github.com` and `api.github.com` entries in the [allowlist for local requests](../../../security/webhooks.md#allow-outbound-requests-to-certain-ip-addresses-and-domains). @@ -78,7 +77,17 @@ If you are importing from GitHub.com to a self-managed GitLab instance: - You don't need to enable the [GitHub integration](../../../integration/github.md). - GitHub must be enabled as an import source in the - [Admin Area](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources). + [Admin Area](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). + +### Known issues + +- GitHub pull request comments (known as diff notes in GitLab) created before 2017 are imported in separate threads. + This occurs because of a limitation of the GitHub API that doesn't include `in_reply_to_id` for comments before 2017. +- Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383047), if you are using GitHub as an + OmniAuth provider, ensure that the URL perimeter is specified in the + [OmniAuth configuration](../../../integration/github.md#enable-github-oauth-in-gitlab). +- Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424400), Markdown attachments from + repositories on GitHub Enterprise Server instances aren't imported. ## Import your GitHub repository into GitLab @@ -288,10 +297,10 @@ When they are imported, supported GitHub branch protection rules are mapped to e | :---------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------ | | **Require conversation resolution before merging** for the project's default branch | **All threads must be resolved** [project setting](../merge_requests/index.md#prevent-merge-unless-all-threads-are-resolved) | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/371110) | | **Require a pull request before merging** | **No one** option in the **Allowed to push and merge** list of [branch protection settings](../protected_branches.md#add-protection-to-existing-branches) | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/370951) | -| **Require signed commits** for the project's default branch | **Reject unsigned commits** GitLab [push rule](../repository/push_rules.md#prevent-unintended-consequences) **(PREMIUM)** | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) | +| **Require signed commits** for the project's default branch | **Reject unsigned commits** GitLab [push rule](../repository/push_rules.md#prevent-unintended-consequences) **(PREMIUM ALL)** | [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/370949) | | **Allow force pushes - Everyone** | **Allowed to force push** [branch protection setting](../protected_branches.md#allow-force-push-on-a-protected-branch) | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/370943) | -| **Require a pull request before merging - Require review from Code Owners** | **Require approval from code owners** [branch protection setting](../protected_branches.md#require-code-owner-approval-on-a-protected-branch) **(PREMIUM)** | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/376683) | -| **Require a pull request before merging - Allow specified actors to bypass required pull requests** | List of users in the **Allowed to push and merge** list of [branch protection settings](../protected_branches.md#add-protection-to-existing-branches) **(PREMIUM)**. Without a **Premium** subscription, the list of users that are allowed to push and merge is limited to roles. | [GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/384939) | +| **Require a pull request before merging - Require review from Code Owners** | **Require approval from code owners** [branch protection setting](../protected_branches.md#require-code-owner-approval-on-a-protected-branch) **(PREMIUM ALL)** | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/376683) | +| **Require a pull request before merging - Allow specified actors to bypass required pull requests** | List of users in the **Allowed to push and merge** list of [branch protection settings](../protected_branches.md#add-protection-to-existing-branches) **(PREMIUM ALL)**. Without a **Premium** subscription, the list of users that are allowed to push and merge is limited to roles. | [GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/issues/384939) | Mapping GitHub rule **Require status checks to pass before merging** to [external status checks](../merge_requests/status_checks.md) was considered in issue diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index c1d6f5dbcbe..df831c2f3eb 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -41,7 +41,7 @@ with a malicious `.gitlab-ci.yml` file could allow an attacker to exfiltrate gro GitLab self-managed administrators can reduce their attack surface by disabling import sources they don't need: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Visibility and access controls**. diff --git a/doc/user/project/import/manifest.md b/doc/user/project/import/manifest.md index 433496ba6c9..aaf19439c24 100644 --- a/doc/user/project/import/manifest.md +++ b/doc/user/project/import/manifest.md @@ -19,7 +19,7 @@ repositories like the Android Open Source Project (AOSP). > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. -- [Manifest import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [Manifest import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The Manifest import source is enabled by default on GitLab.com. - GitLab must use PostgreSQL for its database, because [subgroups](../../group/subgroups/index.md) are needed for the manifest import diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md deleted file mode 100644 index 7b9615b883c..00000000000 --- a/doc/user/project/import/phabricator.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -stage: Manage -group: Import and Integrate -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' ---- - -# Import Phabricator tasks into a GitLab project (removed) **(FREE SELF)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106369) in GitLab 15.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117649) in 16.0. diff --git a/doc/user/project/import/repo_by_url.md b/doc/user/project/import/repo_by_url.md index 5d23ee885bb..375cb718538 100644 --- a/doc/user/project/import/repo_by_url.md +++ b/doc/user/project/import/repo_by_url.md @@ -12,15 +12,15 @@ You can import your existing repositories by providing the Git URL. > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. -- [Repository by URL import source](../../../administration/settings/visibility_and_access_controls.md#configure-allowed-import-sources) +- [Repository by URL import source](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your GitLab administrator to enable it. The Repository by URL import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. ## Import project by URL -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. On the right of the page, select **New project**. 1. Select the **Import project** tab. 1. Select **Repository by URL**. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 478e9c6bf1b..b60d87adbd3 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -134,7 +134,7 @@ Prerequisites: [added to your GitLab account](../ssh.md#add-an-ssh-key-to-your-gitlab-account). - You must have permission to add new projects to a namespace. To check if you have permission: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. + 1. On the left sidebar, select **Search or go to** and find your group. 1. In the upper-right corner, confirm that **New project** is visible. Contact your GitLab administrator if you require permission. diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 60b23540ac3..d8ad7f9a29c 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -24,7 +24,7 @@ Prerequisites: To view project insights: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Insights**. 1. To view a report, select the **Select report** dropdown list. @@ -52,7 +52,7 @@ To configure project insights, either: - Create a `.gitlab/insights.yml` file locally in the root directory of your project, and push your changes. - Create a `.gitlab/insights.yml` file in the UI: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Above the file list, select the branch you want to commit to, select the plus icon, then select **New file**. 1. In the **File name** text box, enter `.gitlab/insights.yml`. 1. In the large text box, update the file contents. diff --git a/doc/user/project/integrations/apple_app_store.md b/doc/user/project/integrations/apple_app_store.md index 6005bc48d56..763f478eb99 100644 --- a/doc/user/project/integrations/apple_app_store.md +++ b/doc/user/project/integrations/apple_app_store.md @@ -29,7 +29,7 @@ An Apple ID enrolled in the [Apple Developer Program](https://developer.apple.co GitLab supports enabling the Apple App Store integration at the project level. Complete these steps in GitLab: 1. In the Apple App Store Connect portal, generate a new private key for your project by following [these instructions](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Apple App Store Connect**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md index c73563ba162..442b8695136 100644 --- a/doc/user/project/integrations/asana.md +++ b/doc/user/project/integrations/asana.md @@ -32,7 +32,7 @@ In Asana, create a Personal Access Token. Complete these steps in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Asana**. 1. Ensure that the **Active** toggle is enabled. diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index 9abbe75cc4c..242d5e6974c 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -36,7 +36,7 @@ integration in GitLab. ## Configure GitLab -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Atlassian Bamboo**. 1. Ensure the **Active** checkbox is selected. diff --git a/doc/user/project/integrations/bugzilla.md b/doc/user/project/integrations/bugzilla.md index 048673cd4a2..febbee66c33 100644 --- a/doc/user/project/integrations/bugzilla.md +++ b/doc/user/project/integrations/bugzilla.md @@ -14,7 +14,7 @@ You can configure Bugzilla as an To enable the Bugzilla integration in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Bugzilla**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -39,7 +39,7 @@ project pages. This link takes you to the appropriate Bugzilla project. You can also disable [GitLab internal issue tracking](../issues/index.md) in this project. For more information about the steps and consequences of disabling GitLab issues, see -[Configure project visibility, features, and permissions](../settings/index.md#configure-project-visibility-features-and-permissions). +Configure project [visibility](../../../user/public_access.md#change-project-visibility), [features, and permissions](../settings/index.md#configure-project-features-and-permissions). ## Reference Bugzilla issues in GitLab diff --git a/doc/user/project/integrations/clickup.md b/doc/user/project/integrations/clickup.md index 7075aca28c2..dc93ff8ed06 100644 --- a/doc/user/project/integrations/clickup.md +++ b/doc/user/project/integrations/clickup.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can use [ClickUp](https://clickup.com/) as an external issue tracker. To enable the ClickUp integration in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **ClickUp**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -34,7 +34,7 @@ For example, this is a configuration for a project named `gitlab-ci`: You can also disable [GitLab internal issue tracking](../issues/index.md) in this project. For more information about the steps and consequences of disabling GitLab issues, see -[Configure project visibility, features, and permissions](../settings/index.md#configure-project-visibility-features-and-permissions). +Configure project [visibility](../../../user/public_access.md#change-project-visibility), [features, and permissions](../settings/index.md#configure-project-features-and-permissions). ## Reference ClickUp issues in GitLab diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md index 54d092d2fa9..0a31e66fe96 100644 --- a/doc/user/project/integrations/custom_issue_tracker.md +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -20,7 +20,7 @@ on the left sidebar in your project. To enable a custom issue tracker in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Custom issue tracker**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/user/project/integrations/discord_notifications.md b/doc/user/project/integrations/discord_notifications.md index 67f76d48744..29d92b16e32 100644 --- a/doc/user/project/integrations/discord_notifications.md +++ b/doc/user/project/integrations/discord_notifications.md @@ -28,7 +28,7 @@ and configure it in GitLab. With the webhook URL created in the Discord channel, you can set up the Discord Notifications integration in GitLab. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Discord Notifications**. 1. Ensure that the **Active** toggle is enabled. diff --git a/doc/user/project/integrations/emails_on_push.md b/doc/user/project/integrations/emails_on_push.md index 105061efba7..5c1130c7893 100644 --- a/doc/user/project/integrations/emails_on_push.md +++ b/doc/user/project/integrations/emails_on_push.md @@ -11,7 +11,7 @@ that is pushed to your project. To enable emails on push: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Emails on push**. 1. In the **Recipients** section, provide a list of emails separated by spaces or newlines. diff --git a/doc/user/project/integrations/ewm.md b/doc/user/project/integrations/ewm.md index 7ee0306e7c0..831078e4e4b 100644 --- a/doc/user/project/integrations/ewm.md +++ b/doc/user/project/integrations/ewm.md @@ -14,7 +14,7 @@ This IBM product was [formerly named Rational Team Concert (RTC)](https://jazz.n To enable the EWM integration, in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **EWM**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 3f4b110468b..c368a9da2f6 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -29,7 +29,7 @@ Complete these steps on GitHub: Complete these steps in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **GitHub**. 1. Ensure the **Active** checkbox is selected. diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index a082d9aa5be..d762c71242d 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -30,7 +30,7 @@ Although functionality has not changed, you should [reinstall the app](#update-t To install the GitLab for Slack app from project integration settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **GitLab for Slack app**. 1. Select **Install GitLab for Slack app**. @@ -55,7 +55,7 @@ When GitLab releases new features for the GitLab for Slack app, you might have t To update your GitLab for Slack app: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find a project for +1. On the left sidebar, select **Search or go to** and find a project for which the GitLab for Slack app is configured. 1. Select **Settings > Integrations**. 1. Select **GitLab for Slack app**. @@ -105,7 +105,7 @@ The command returns an error if no matching action is found. By default, slash commands expect a project full path. To create a shorter project alias in the GitLab for Slack app: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **GitLab for Slack app**. 1. The current **Project Alias**, if any, is displayed. To edit this value, @@ -122,7 +122,7 @@ With Slack notifications, GitLab can send messages to Slack workspace channels f To configure Slack notifications: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find a project for +1. On the left sidebar, select **Search or go to** and find a project for which the GitLab for Slack app is [installed](#install-the-gitlab-for-slack-app). 1. Select **Settings > Integrations**. 1. Select **GitLab for Slack app**. @@ -168,10 +168,19 @@ The following events are available for Slack notifications: | **Wiki page** | A wiki page is created or updated. | | **Deployment** | A deployment starts or finishes. | | **Alert** | A new, unique alert is recorded. | -| **Group mention in public** | A group is mentioned in a public context. | -| **Group mention in private** | A group is mentioned in a confidential context. | +| **[Group mention](#trigger-notifications-for-group-mentions) in public** | A group is mentioned in a public context. | +| **[Group mention](#trigger-notifications-for-group-mentions) in private** | A group is mentioned in a confidential context. | | [**Vulnerability**](../../application_security/vulnerabilities/index.md) | A new, unique vulnerability is recorded. | +### Trigger notifications for group mentions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/417751) in GitLab 16.4. + +To trigger a [notification event](#notification-events) for a group mention, use `@<group_name>` in: + +- Issue and merge request descriptions +- Comments on issues, merge requests, and commits + ## Troubleshooting ### GitLab for Slack app does not appear in the list of integrations diff --git a/doc/user/project/integrations/google_play.md b/doc/user/project/integrations/google_play.md index 696bb6acf99..1affa7abfb4 100644 --- a/doc/user/project/integrations/google_play.md +++ b/doc/user/project/integrations/google_play.md @@ -29,7 +29,7 @@ Prerequisites: To enable the Google Play integration in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Google Play**. 1. In **Enable integration**, select the **Active** checkbox. diff --git a/doc/user/project/integrations/harbor.md b/doc/user/project/integrations/harbor.md index 803984f82bf..9d6be5da810 100644 --- a/doc/user/project/integrations/harbor.md +++ b/doc/user/project/integrations/harbor.md @@ -25,7 +25,7 @@ In the Harbor instance, ensure that: GitLab supports integrating Harbor projects at the group or project level. Complete these steps in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Harbor**. 1. Under **Enable integration**, select the **Active** checkbox. diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index 00ae1a0478c..59b5043b8f7 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -18,7 +18,7 @@ Prerequisites: To view the available integrations for your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. You can also view and manage integration settings across [all projects in an instance or group](../../admin_area/settings/project_integration_management.md). diff --git a/doc/user/project/integrations/irker.md b/doc/user/project/integrations/irker.md index f5084392841..146123f97cd 100644 --- a/doc/user/project/integrations/irker.md +++ b/doc/user/project/integrations/irker.md @@ -39,7 +39,7 @@ network. For more details, read ## Complete these steps in GitLab -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **irker (IRC gateway)**. 1. Ensure that the **Active** toggle is enabled. diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 9e1de5e3aab..9588fbff922 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -41,7 +41,7 @@ Display name override is not enabled by default, you need to ask your administra After the Mattermost instance has an incoming webhook set up, you can set up GitLab to send the notifications: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Mattermost notifications**. 1. Select the GitLab events to generate notifications for. For each event you select, input the Mattermost channel diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 4898d5bd337..715e744988c 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -31,7 +31,7 @@ you must have Mattermost [3.4 or later](https://mattermost.com/blog/category/pla If Mattermost is installed on the same server as GitLab, you can automatically configure Mattermost slash commands: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Mattermost slash commands**. 1. Under **Enable integration**, ensure the **Active** checkbox is selected. @@ -67,7 +67,7 @@ To get configuration values from GitLab: 1. In a different browser tab, sign in to GitLab as a user with administrator access. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select **Mattermost slash commands**. GitLab displays potential values for Mattermost settings. diff --git a/doc/user/project/integrations/microsoft_teams.md b/doc/user/project/integrations/microsoft_teams.md index b359696c72b..dea34709dff 100644 --- a/doc/user/project/integrations/microsoft_teams.md +++ b/doc/user/project/integrations/microsoft_teams.md @@ -35,7 +35,7 @@ After you configure Microsoft Teams to receive notifications, you must configure GitLab to send the notifications: 1. Sign in to GitLab as an administrator. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Microsoft Teams notifications**. 1. To enable the integration, select **Active**. diff --git a/doc/user/project/integrations/pivotal_tracker.md b/doc/user/project/integrations/pivotal_tracker.md index 3697e660deb..cd0a1819afd 100644 --- a/doc/user/project/integrations/pivotal_tracker.md +++ b/doc/user/project/integrations/pivotal_tracker.md @@ -37,7 +37,7 @@ In Pivotal Tracker, [create an API token](https://www.pivotaltracker.com/help/ar Complete these steps in GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Pivotal Tracker**. 1. Ensure that the **Active** toggle is enabled. diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md deleted file mode 100644 index c0b2591d824..00000000000 --- a/doc/user/project/integrations/prometheus.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: 'index.md' ---- - -# Prometheus (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md deleted file mode 100644 index e31f3b26e66..00000000000 --- a/doc/user/project/integrations/prometheus_library/cloudwatch.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring AWS resources (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md deleted file mode 100644 index 4a0d324932b..00000000000 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring HAProxy (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md deleted file mode 100644 index 49345f41514..00000000000 --- a/doc/user/project/integrations/prometheus_library/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Prometheus Metrics library (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md deleted file mode 100644 index f5bbaffa58e..00000000000 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring Kubernetes (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md deleted file mode 100644 index e38f26710fc..00000000000 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring NGINX (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md deleted file mode 100644 index 189cf59a514..00000000000 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring NGINX Ingress Controller (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md deleted file mode 100644 index e4edd63314f..00000000000 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Monitor -group: Respond -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-08-22' -redirect_to: '../../../../operations/index.md' ---- - -# Monitoring NGINX Ingress Controller with VTS metrics (removed) **(FREE ALL)** - -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7 -and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/399231) in 16.0. diff --git a/doc/user/project/integrations/pumble.md b/doc/user/project/integrations/pumble.md index 218a036f0a2..3f1907d04cb 100644 --- a/doc/user/project/integrations/pumble.md +++ b/doc/user/project/integrations/pumble.md @@ -26,7 +26,7 @@ notifications: 1. To enable the integration for your group or project: 1. In your group or project, on the left sidebar, select **Settings > Integrations**. 1. To enable the integration for your instance: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. On the left sidebar, select **Settings > Integrations**. 1. Select the **Pumble** integration. diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index af2c7265f5d..a66aa9cd00a 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can use [Redmine](https://www.redmine.org/) as an external issue tracker. To enable the Redmine integration in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Redmine**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -38,7 +38,7 @@ For example, this is a configuration for a project named `gitlab-ci`: You can also disable [GitLab internal issue tracking](../issues/index.md) in this project. For more information about the steps and consequences of disabling GitLab issues, see -[Configure project visibility, features, and permissions](../settings/index.md#configure-project-visibility-features-and-permissions). +Configure project [visibility](../../../user/public_access.md#change-project-visibility), [features, and permissions](../settings/index.md#configure-project-features-and-permissions). ## Reference Redmine issues in GitLab diff --git a/doc/user/project/integrations/shimo.md b/doc/user/project/integrations/shimo.md index b2c85c8cb4c..aeddee29109 100644 --- a/doc/user/project/integrations/shimo.md +++ b/doc/user/project/integrations/shimo.md @@ -23,7 +23,7 @@ This change is a breaking change. To enable the Shimo integration for your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Shimo**. 1. Under **Enable integration**, ensure the **Active** checkbox is selected. @@ -36,7 +36,7 @@ On the left sidebar, **Shimo** now appears instead of **Wiki**. To view the Shimo Workspace from your group or project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Shimo**. 1. On the **Shimo Workspace** page, select **Go to Shimo Workspace**. diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index 87d7476e42e..d48fd929d54 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -32,7 +32,7 @@ to control GitLab from Slack. Slash commands are configured separately. > [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106760) in GitLab 15.9 to limit Slack channels to 10 per event. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Slack notifications**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -79,10 +79,19 @@ The following triggers are available for Slack notifications: | **Wiki page** | A wiki page is created or updated. | | **Deployment** | A deployment starts or finishes. | | **Alert** | A new, unique alert is recorded. | -| **Group mention in public** | A group is mentioned in a public context. | -| **Group mention in private** | A group is mentioned in a confidential context. | +| **[Group mention](#trigger-notifications-for-group-mentions) in public** | A group is mentioned in a public context. | +| **[Group mention](#trigger-notifications-for-group-mentions) in private** | A group is mentioned in a confidential context. | | [**Vulnerability**](../../application_security/vulnerabilities/index.md) | A new, unique vulnerability is recorded. | +## Trigger notifications for group mentions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/417751) in GitLab 16.4. + +To trigger a [notification event](#triggers-for-slack-notifications) for a group mention, use `@<group_name>` in: + +- Issue and merge request descriptions +- Comments on issues, merge requests, and commits + ## Troubleshooting If your Slack integration is not working, start troubleshooting by @@ -147,7 +156,7 @@ Commands that change data can cause damage if not run correctly or under the rig ```ruby # Grab all projects that have the Slack notifications enabled -p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN integrations s ON p.id = s.project_id WHERE s.type_new = 'Slack' AND s.active = true") +p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN integrations s ON p.id = s.project_id WHERE s.type_new = 'Integrations::Slack' AND s.active = true") # Disable the integration on each of the projects that were found. p.each do |project| diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index 78c8180cb4c..67894e158c8 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -23,7 +23,7 @@ For a list of available slash commands, see [Slash commands](gitlab_slack_applic Slack slash commands are scoped to a project. To configure Slack slash commands: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Slack slash commands** and leave this browser tab open. 1. In a new browser tab, sign in to Slack and [add a new slash command](https://my.slack.com/services/new/slash-commands). diff --git a/doc/user/project/integrations/squash_tm.md b/doc/user/project/integrations/squash_tm.md index ff633783e83..75849d61551 100644 --- a/doc/user/project/integrations/squash_tm.md +++ b/doc/user/project/integrations/squash_tm.md @@ -26,7 +26,7 @@ are synchronized as requirements in Squash TM and test progress is reported in G ## Configure GitLab -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Squash TM**. 1. Ensure that the **Active** toggle is enabled. diff --git a/doc/user/project/integrations/telegram.md b/doc/user/project/integrations/telegram.md index d68222d2f94..4e94ef76f58 100644 --- a/doc/user/project/integrations/telegram.md +++ b/doc/user/project/integrations/telegram.md @@ -40,10 +40,10 @@ After you invite the bot to a Telegram channel, you can configure GitLab to send 1. To enable the integration: - **For your group or project:** - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. + 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Integrations**. - **For your instance:** - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select **Telegram**. diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index d59bfde5944..34f8cf262cd 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -15,7 +15,7 @@ copy its URL. In GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Unify Circuit**. 1. Turn on the **Active** toggle. diff --git a/doc/user/project/integrations/webex_teams.md b/doc/user/project/integrations/webex_teams.md index 370584303f8..b675ffa7a36 100644 --- a/doc/user/project/integrations/webex_teams.md +++ b/doc/user/project/integrations/webex_teams.md @@ -26,10 +26,10 @@ notifications: 1. To enable integration: - At the project or group level: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. + 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Settings > Integrations**. - At the instance level: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Integrations**. 1. Select the **Webex Teams** integration. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index c4dd8dcf812..269fdf304a8 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -988,7 +988,7 @@ Payload example: "draft": { "previous": true, "current": false - } + }, "updated_at": { "previous": "2017-09-15 16:50:55 UTC", "current":"2017-09-15 16:52:00 UTC" @@ -1026,7 +1026,7 @@ Payload example: "last_edited_by_id": { "previous": null, "current": 3278533 - }, + } }, "assignees": [ { @@ -1521,7 +1521,8 @@ Deployment events are triggered when a deployment: - Fails - Is cancelled -The `deployable_id` in the payload is the ID of the CI/CD job. +The `deployable_id` and `deployable_url` in the payload represent a CI/CD job that executed the deployment. +When the deployment event occurs by [API](../../../ci/environments/external_deployment_tools.md) or [`trigger` jobs](../../../ci/pipelines/downstream_pipelines.md), `deployable_url` is `null`. Request header: @@ -1879,12 +1880,13 @@ Payload example: ## Emoji events -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123952) in GitLab 16.2 [with a flag](../../../administration/feature_flags.md) named `emoji_webhooks`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123952) in GitLab 16.2 [with a flag](../../../administration/feature_flags.md) named `emoji_webhooks`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/417288) in GitLab 16.3. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/417288) in GitLab 16.4. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `emoji_webhooks`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can +[disable the feature flag](../../../administration/feature_flags.md) named `emoji_webhooks`. On GitLab.com, this feature is available. NOTE: To have the `emoji_webhooks` flag enabled on GitLab.com, see [issue 417288](https://gitlab.com/gitlab-org/gitlab/-/issues/417288). diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index fe3d5e015a6..8e7f8bcd67d 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -40,7 +40,7 @@ including: ## Group webhooks **(PREMIUM ALL)** You can configure a group webhook, which is triggered by events -that occur across all projects in the group. If you configure identical webhooks +that occur across all projects in the group and its subgroups. If you configure identical webhooks in a group and a project, they are both triggered by an event in the project. diff --git a/doc/user/project/integrations/youtrack.md b/doc/user/project/integrations/youtrack.md index 651c1a15b7a..c05083a1d83 100644 --- a/doc/user/project/integrations/youtrack.md +++ b/doc/user/project/integrations/youtrack.md @@ -14,7 +14,7 @@ You can configure YouTrack as an To enable the YouTrack integration in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **YouTrack**. 1. Under **Enable integration**, select the **Active** checkbox. @@ -30,7 +30,7 @@ project pages. This link takes you to the appropriate YouTrack project. You can also disable [GitLab internal issue tracking](../issues/index.md) in this project. For more information about the steps and consequences of disabling GitLab issues, see -[Configure project visibility, features, and permissions](../settings/index.md#configure-project-visibility-features-and-permissions). +Configure project [visibility](../../../user/public_access.md#change-project-visibility), [features, and permissions](../settings/index.md#configure-project-features-and-permissions). ## Reference YouTrack issues in GitLab diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index 0b8f6cc33fc..60b50251555 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -32,7 +32,7 @@ When you create a confidential issue in a project, the project becomes listed in To create a confidential issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, at the top, select **Create new** (**{plus}**). 1. From the dropdown list, select **New issue**. 1. Complete the [fields](create_issues.md#fields-in-the-new-issue-form). @@ -43,7 +43,7 @@ To create a confidential issue: To change the confidentiality of an existing issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. Select the title of your issue to view it. 1. On the right sidebar, next to **Confidentiality**, select **Edit**. diff --git a/doc/user/project/issues/create_issues.md b/doc/user/project/issues/create_issues.md index 3014b088fd6..cd4ef43d333 100644 --- a/doc/user/project/issues/create_issues.md +++ b/doc/user/project/issues/create_issues.md @@ -28,7 +28,7 @@ Prerequisites: To create an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Either: - On the left sidebar, select **Plan > Issues**, and then, in the upper-right corner, select **New issue**. @@ -51,7 +51,7 @@ Prerequisites: To create an issue from a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Issues**. 1. In the upper-right corner, select **Select project to create issue**. 1. Select the project you'd like to create an issue for. The button now reflects the selected @@ -98,7 +98,7 @@ Prerequisites: To create an issue from a project issue board: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issue boards**. 1. At the top of a board list, select **Create new issue** (**{plus-square}**). 1. Enter the issue's title. @@ -106,7 +106,7 @@ To create an issue from a project issue board: To create an issue from a group issue board: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Issue boards**. 1. At the top of a board list, select **Create new issue** (**{plus-square}**). 1. Enter the issue's title. @@ -130,7 +130,7 @@ Prerequisites: To email an issue to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. At the bottom of the page, select **Email a new issue to this project**. 1. To copy the email address, select **Copy** (**{copy-to-clipboard}**). diff --git a/doc/user/project/issues/crosslinking_issues.md b/doc/user/project/issues/crosslinking_issues.md index b3fe7da4280..882e4d97938 100644 --- a/doc/user/project/issues/crosslinking_issues.md +++ b/doc/user/project/issues/crosslinking_issues.md @@ -76,3 +76,10 @@ you can also [set an issue to close automatically](managing_issues.md#closing-is as soon as the merge request is merged.  + +## From branch names + +When you create a branch in the same project as an issue and start the branch name with the issue +number, followed by a hyphen, the issue and MR you create are linked. +For more information, see +[Prefix branch names with issue numbers](../repository/branches/index.md#prefix-branch-names-with-issue-numbers). diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md index 86370cab963..34a9b69038b 100644 --- a/doc/user/project/issues/csv_export.md +++ b/doc/user/project/issues/csv_export.md @@ -1,97 +1,102 @@ --- -stage: none -group: unassigned +stage: Plan +group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Export issues to CSV **(FREE ALL)** -> Moved to GitLab Free in 12.10. - -You can export issues as CSV files from GitLab, which are sent to your default -notification email address as an attachment. - -**Export Issues to CSV** enables you and your team to export all the data -collected from issues into a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) -file, which stores tabular data in plain text. +You can export issues from GitLab to a plain-text CSV +([comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)) +file. The CSV file is attached to an email, and sent to your default +notification email address. <!-- vale gitlab.Spelling = NO --> -CSV files can be used with any plotter or spreadsheet-based program, such as -Microsoft Excel, Open Office Calc, or Google Sheets. +CSV files can be used with any plotter or spreadsheet-based program, like +Microsoft Excel, OpenOffice Calc, or Google Sheets. Use a CSV list of issues to: <!-- vale gitlab.Spelling = YES --> -Here are some of the uses of exporting issues as CSV files: - -- Make a snapshot of issues for offline analysis or to communicate with other - teams who may not be in GitLab. +- Create a snapshot of issues for offline analysis, or to share with other + teams who might not be in GitLab. - Create diagrams, graphs, and charts from the CSV data. -- Present the data in any other format for auditing or sharing reasons. -- Import the issues elsewhere to a system outside of GitLab. -- Long-term issues' data analysis with multiple snapshots created along the - time. +- Convert the data to other formats for auditing or sharing. +- Import the issues to a system outside of GitLab. +- Analyze long-term trends with multiple snapshots created over time. - Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics. -## Choosing which issues to include - -After selecting a project, from the issues page you can narrow down which -issues to export using the search bar, along with the All/Open/Closed tabs. All -issues returned are exported, including those not shown on the first page. +## Select issues to export - +You can export issues from individual projects, but not groups. -GitLab asks you to confirm the number of issues and email address for the -export, after which the email is prepared. +Prerequisites: - +- You must have at least the Reporter role. -## Sorting +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Plan > Issues**. +1. Above the list of issues, select **Search or filter results...**. +1. In the dropdown list that appears, select the attributes to filter by. + For more information about filter options, see + [Filter the list of issues](managing_issues.md#filter-the-list-of-issues). +1. In the upper right, select **Actions** (**{ellipsis_v}**) **> Export as CSV**. +1. In the dialog, verify that the email address is correct, then select **Export issues**. -Exported issues are always sorted by `Title`. +All matching issues are exported, including those not shown on the first page. +The exported CSV does not contain attachments from issues. ## Format -Data is encoded with a comma as the column delimiter, with `"` used to quote -fields if needed, and newlines to separate rows. The first row contains the -headers, which are listed in the following table along with a description of -the values: - -| Column | Description | -|------------------------------------------|-----------------------------------------------------------| -| Title | Issue `title` | -| Description | Issue `description` | -| Issue ID | Issue `iid` | -| URL | A link to the issue on GitLab | -| State | `Open` or `Closed` | -| Author | Full name of the issue author | -| Author Username | Username of the author, with the `@` symbol omitted | -| Assignee | Full name of the issue assignee | -| Assignee Username | Username of the author, with the `@` symbol omitted | -| Confidential | `Yes` or `No` | -| Locked | `Yes` or `No` | -| Due Date | Formatted as `YYYY-MM-DD` | -| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Milestone | Title of the issue milestone | -| Weight | Issue weight | -| Labels | Title of any labels joined with a `,` | -| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | -| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | -| [Epic](../../group/epics/index.md) ID | ID of the parent epic, introduced in 12.7 | -| [Epic](../../group/epics/index.md) Title | Title of the parent epic, introduced in 12.7 | - -In GitLab 14.7 and earlier, the first two columns were `Issue ID` and `URL`, -which [caused an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/34769) -when importing back into GitLab. - -## Limitations - -- Export Issues to CSV is not available at the Group's Issues List. -- Issues are sent as an email attachment, with a 15 MB export limit to ensure - successful delivery across a range of email providers. If you reach the limit, - we suggest narrowing the search before export, perhaps by exporting open and - closed issues separately. -- CSV files are plain text files. This means that the exported CSV file doesn't - contain any issue attachments. +The CSV file has this format: + +- Sort is by title. +- Columns are delimited with commas. +- Fields are quoted with double quotes (`"`) if needed. +- Newline characters separate rows. + +## Columns + +The following columns are included in the CSV file. + +| Column | Description | +|-------------------|-------------| +| Title | Issue `title` | +| Description | Issue `description` | +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| State | `Open` or `Closed` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | +| Assignee Username | Username of the author, with the `@` symbol omitted | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formatted as `YYYY-MM-DD` | +| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Labels, separated by commas | +| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | +| Epic ID | ID of the parent epic | +| Epic Title | Title of the parent epic | + +## Troubleshooting + +When working with exported issues, you might encounter the following issues. + +### Column order + +In GitLab 14.7 and earlier, the first two columns in exported files were `Issue ID` and `URL`, +which caused problems importing data back into GitLab. For more information, see +[issue 34769](https://gitlab.com/gitlab-org/gitlab/-/issues/34769). + +### Size of export + +Issues are sent as an email attachment, with a 15 MB export limit to ensure +successful delivery across a range of email providers. If you reach the limit, +narrow your search before export. For example, consider exporting open and +closed issues separately. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 4068083cd1e..0ea49ff387f 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -6,12 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Design management **(FREE ALL)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab 12.2. -> - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. -> - Design Management section in issues [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223193) in GitLab 13.2, with a feature flag named `design_management_moved`. In earlier versions, designs were displayed in a separate tab. -> - Design Management section in issues [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/223197) for new displays in GitLab 13.4. - With Design Management you can upload design assets (including wireframes and mockups) to GitLab issues and keep them stored in a single place. Product designers, product managers, and engineers can collaborate on designs with a single source of truth. @@ -29,7 +23,7 @@ For a video overview, see [Design Management (GitLab 12.2)](https://www.youtube. - On self-managed instances, a GitLab administrator must [enable LFS globally](../../../administration/lfs/index.md). - On both GitLab.com and self-managed instances, LFS must be - [enabled for the project itself](../settings/index.md#configure-project-visibility-features-and-permissions). + [enabled for the project itself](../settings/index.md#configure-project-features-and-permissions). If enabled globally, LFS is enabled by default for all projects. If you have disabled it for your project, you must enable it again. @@ -58,7 +52,6 @@ You can upload files of the following types as designs: - JPEG - JPG - PNG -- SVG - TIFF - WEBP @@ -69,7 +62,7 @@ Support for PDF files is tracked in [issue 32811](https://gitlab.com/gitlab-org/ - Design Management data isn't deleted when: - [A project is destroyed](https://gitlab.com/gitlab-org/gitlab/-/issues/13429). - [An issue is deleted](https://gitlab.com/gitlab-org/gitlab/-/issues/13427). -- In GitLab 12.7 and later, Design Management data [can be replicated](../../../administration/geo/replication/datatypes.md#limitations-on-replicationverification) +- Design Management data [can be replicated](../../../administration/geo/replication/datatypes.md#limitations-on-replicationverification) and in GitLab 16.1 and later it can be [verified by Geo as well](https://gitlab.com/gitlab-org/gitlab/-/issues/355660). ## View a design @@ -106,9 +99,6 @@ a blue icon (**{file-modified-solid}**) is displayed. ### Zoom in on a design -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab 12.7. -> - Ability to drag a zoomed image to move it [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. - You can explore a design in more detail by zooming in and out of the image: - To control the amount of zoom, select plus (`+`) and minus (`-`) @@ -124,7 +114,7 @@ To move around the image while zoomed in, drag the image. Prerequisites: - You must have at least the Developer role for the project. -- In GitLab 13.1 and later, the names of the uploaded files must be no longer than 255 characters. +- The names of the uploaded files must be no longer than 255 characters. To add a design to an issue: @@ -163,6 +153,7 @@ Prerequisites: To do so, [add a design](#add-a-design-to-an-issue) with the same filename. To browse all the design versions, use the dropdown list at the top of the **Designs** section. +It's shown as either **Showing latest version** or **Showing version #N**. ### Skipped designs @@ -172,14 +163,19 @@ When designs are skipped, a warning message is displayed. ## Archive a design -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in GitLab 12.4. -> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/220964) the button from "Delete" to "Archive" in GitLab 13.3. - You can archive individual designs or select a few of them to archive at once. +Archived designs are not permanently lost. +You can browse [previous versions](#add-a-new-version-of-a-design). + +When you archive a design, its URL changes. +If the design isn't available in the latest version, you can link to it only with the version in the +URL. + Prerequisites: - You must have at least the Developer role for the project. +- You can archive only the latest version of a design. To archive a single design: @@ -192,15 +188,10 @@ To archive multiple designs at once: 1. Select the checkboxes on the designs you want to archive. 1. Select **Archive selected**. -NOTE: -Only the latest version of the designs can be archived. -Archived designs are not permanently lost. You can browse -[previous versions](#add-a-new-version-of-a-design). +## Markdown and rich text editors for descriptions <!-- When content_editor_on_issues flag is removed, move version notes - to "Add a design to an issue", update that topic, and delete the one below. --> - -## Markdown and rich text editors for descriptions + to "Add a design to an issue", update that topic, and delete this one. --> > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388449) in GitLab 16.1 [with a flag](../../../administration/feature_flags.md) named `content_editor_on_issues`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/375172) in GitLab 16.2. @@ -214,30 +205,28 @@ It's the same editor you use for comments across GitLab. ## Reorder designs -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34382) in GitLab 13.3. - You can change the order of designs by dragging them to a new position. ## Add a comment to a design -> Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.8. - You can start [discussions](../../discussions/index.md) on uploaded designs. To do so: -<!-- vale gitlab.SubstitutionWarning = NO --> 1. Go to an issue. 1. Select the design. +<!-- vale gitlab.SubstitutionWarning = NO --> +<!-- Disable Vale so it doesn't catch "click" --> 1. Click or tap the image. A pin is created in that spot, identifying the discussion's location. +<!-- vale gitlab.SubstitutionWarning = YES --> 1. Enter your message. 1. Select **Comment**. -<!-- vale gitlab.SubstitutionWarning = YES --> -You can adjust a pin's position by dragging it around the image. You can use this when your design's -layout has changed, or when you want to move a pin to add a new one in its place. +You can adjust a pin's position by dragging it around the image. +Use this when your design's layout has changed, or to move a pin so you can add a new one in +its place. New discussion threads get different pin numbers, which you can use to refer to them. -In GitLab 12.5 and later, new discussions are output to the issue activity, +New discussions are output to the issue activity, so that everyone involved can participate in the discussion. ## Delete a comment from a design @@ -255,8 +244,6 @@ To delete a comment from a design: ## Resolve a discussion thread on a design -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13049) in GitLab 13.1. - When you're done discussing part of a design, you can resolve the discussion thread. To mark a thread as resolved or unresolved, either: @@ -272,16 +259,10 @@ To revisit a resolved discussion, expand **Resolved Comments** below the visible ## Add a to-do item for a design -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198439) in GitLab 13.4. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245074) in GitLab 13.5. - To add a [to-do item](../../todos.md) for a design, select **Add a to do** on the design sidebar. ## Refer to a design in Markdown -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217160) in GitLab 13.1. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258662) in GitLab 13.5. - To refer to a design in a [Markdown](../../markdown.md) text box in GitLab, for example, in a comment or description, paste its URL. It's then displayed as a short reference. @@ -297,9 +278,6 @@ It's rendered as: ## Design activity records -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33051) in GitLab 13.1. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/225205) in GitLab 13.2. - User activity events on designs (creation, deletion, and updates) are tracked by GitLab and displayed on the [user profile](../../profile/index.md#access-your-user-profile), [group](../../group/manage.md#view-group-activity), @@ -307,8 +285,6 @@ and [project](../working_with_projects.md#view-project-activity) activity pages. ## GitLab-Figma plugin -> [Introduced](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/issues/2) in GitLab 13.2. - You can use the GitLab-Figma plugin to upload your designs from Figma directly to your issues in GitLab. @@ -316,3 +292,26 @@ To use the plugin in Figma, install it from the [Figma Directory](https://www.fi and connect to GitLab through a personal access token. For more information, see the [plugin documentation](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/wikis/home). + +## Troubleshooting + +When working with Design Management, you might encounter the following issues. + +### Could not find design + +You might get an error that states `Could not find design`. + +This issue occurs when a design has been [archived](#archive-a-design), +so it's not available in the latest version, and the link you've followed doesn't specify a version. + +When you archive a design, its URL changes. +If the design isn't available in the latest version, it can be linked to only with the version in the URL. + +For example, `https://gitlab.example.com/mygroup/myproject/-/issues/123456/designs/menu.png?version=503554`. +You can no longer access `menu.png` with `https://gitlab.example.com/mygroup/myproject/-/issues/123456/designs/menu.png`. + +The workaround is to select one of the previous versions from the dropdown list at the top of the +**Designs** section. +It's shown as either **Showing latest version** or **Showing version #N**. + +Issue [392540](https://gitlab.com/gitlab-org/gitlab/-/issues/392540) tracks improving this behavior. diff --git a/doc/user/project/issues/img/csv_export_button_v12_9.png b/doc/user/project/issues/img/csv_export_button_v12_9.png Binary files differdeleted file mode 100644 index 702b6439d7c..00000000000 --- a/doc/user/project/issues/img/csv_export_button_v12_9.png +++ /dev/null diff --git a/doc/user/project/issues/img/csv_export_modal.png b/doc/user/project/issues/img/csv_export_modal.png Binary files differdeleted file mode 100644 index f988deec966..00000000000 --- a/doc/user/project/issues/img/csv_export_modal.png +++ /dev/null diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 02cefaa47ef..cb3cbf5fc36 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -18,7 +18,7 @@ Prerequisites: To edit an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select the title of your issue to view it. 1. To the right of the title, select **Edit title and description** (**{pencil}**). 1. Edit the available fields. @@ -54,7 +54,7 @@ Prerequisites: To edit multiple issues at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. Select **Bulk edit**. A sidebar on the right of your screen appears. 1. Select the checkboxes next to each issue you want to edit. @@ -88,7 +88,7 @@ Prerequisites: To edit multiple issues at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Issues**. 1. Select **Bulk edit**. A sidebar on the right of your screen appears. 1. Select the checkboxes next to each issue you want to edit. @@ -117,7 +117,7 @@ Prerequisites: To move an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, select **Move issue**. 1. Search for a project to move the issue to. @@ -138,7 +138,7 @@ Prerequisite: To move multiple issues at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. Select **Bulk edit**. A sidebar on the right of your screen appears. 1. Select the checkboxes next to each issue you want to move. @@ -212,7 +212,7 @@ To close an issue, you can either: - In an [issue board](../issue_board.md), drag an issue card from its list into the **Closed** list. - From any other page in the GitLab UI: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. At the top of the issue, select **Close issue**. @@ -309,7 +309,7 @@ Prerequisites: To disable automatic issue closing: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Branch defaults**. 1. Clear the **Auto-close referenced issues on default branch** checkbox. @@ -357,7 +357,7 @@ Prerequisites: To change issue type: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. To the right of the title, select **Edit title and description** (**{pencil}**). 1. Edit the issue and select an issue type from the **Issue type** dropdown list: @@ -377,14 +377,14 @@ Prerequisites: To delete an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the top right corner, select **Issue actions** (**{ellipsis_v}**). 1. Select **Delete issue**. Alternatively: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select the title of your issue to view it. 1. Select **Edit title and description** (**{pencil}**). 1. Select **Delete issue**. @@ -399,7 +399,7 @@ You can promote an issue to an [epic](../../group/epics/index.md) in the immedia To promote an issue to an epic: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the top right corner, select **Issue actions** (**{ellipsis_v}**). 1. Select **Promote to epic**. @@ -422,7 +422,7 @@ You can use the `/promote_to_incident` [quick action](../quick_actions.md) to pr To add an issue to an [iteration](../../group/iterations/index.md): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, in the **Iteration** section, select **Edit**. 1. From the dropdown list, select the iteration to associate this issue with. @@ -434,7 +434,7 @@ Alternatively, you can use the `/iteration` [quick action](../quick_actions.md#i To view all issues assigned to you: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**). +1. On the left sidebar, select **Search or go to**. 1. From the dropdown list, select **Issues assigned to me**. Or: @@ -453,7 +453,7 @@ Or: To filter the list of issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. Above the list of issues, select **Search or filter results...**. 1. In the dropdown list that appears, select the attribute you want to filter by. @@ -491,7 +491,7 @@ when you [filter the list of issues](#filter-the-list-of-issues) by: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39908) in GitLab 12.1. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. In the **Search** box, type the issue ID. For example, enter filter `#10` to return only issue 10. @@ -504,7 +504,7 @@ To refer to an issue elsewhere in GitLab, you can use its full URL or a short re To copy the issue reference to your clipboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, next to **Reference**, select **Copy Reference** (**{copy-to-clipboard}**). @@ -528,7 +528,7 @@ For more information about creating comments by sending an email and the necessa To copy the issue's email address: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, next to **Issue email**, select **Copy Reference** (**{copy-to-clipboard}**). @@ -545,7 +545,7 @@ themselves or another project member assigns them. To change the assignee on an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, in the **Assignee** section, select **Edit**. 1. From the dropdown list, select the user to add as an assignee. @@ -586,7 +586,7 @@ Prerequisites: To edit health status of an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. On the right sidebar, in the **Health status** section, select **Edit**. 1. From the dropdown list, select the status to add to this issue: diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 4daaaf72683..86d21d07950 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -69,7 +69,7 @@ You can also assign and unassign labels with [quick actions](quick_actions.md): To view the **project's labels**: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. Or: @@ -86,7 +86,7 @@ project or group path where it was created. To view the **group's labels**: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Labels**. Or: @@ -108,7 +108,7 @@ Prerequisites: To create a project label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Select **New label**. 1. In the **Title** field, enter a short, descriptive name for the label. You @@ -142,7 +142,7 @@ To do so: To create a group label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Labels**. 1. Select **New label**. 1. In the **Title** field, enter a short, descriptive name for the label. You @@ -182,7 +182,7 @@ Prerequisites: To edit a **project** label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Next to the label you want to edit, select **Edit** (**{pencil}**). @@ -190,7 +190,7 @@ To edit a **project** label: To edit a **group** label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Labels**. 1. Next to the label you want to edit, select **Edit** (**{pencil}**). @@ -208,7 +208,7 @@ Prerequisites: To delete a **project** label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Either: @@ -221,7 +221,7 @@ To delete a **project** label: To delete a **group** label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Manage > Labels**. 1. Either: @@ -251,7 +251,7 @@ Prerequisites: To promote a project label to a group label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Next to the **Subscribe** button, select the three dots (**{ellipsis_v}**) and select **Promote to group label**. @@ -302,7 +302,7 @@ Prerequisites: To add the default labels to the project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Select **Generate a default set of labels**. @@ -441,7 +441,7 @@ Prerequisites: To prioritize a label: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Labels**. 1. Next to a label you want to prioritize, select the star (**{star-o}**). diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 1e9bd307333..901a8fe9850 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -119,7 +119,7 @@ Prerequisite: To add a user to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. Select **Invite members**. 1. If the user: @@ -180,7 +180,7 @@ Prerequisites: To add a group to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. Select **Invite a group**. 1. Select a group. @@ -211,7 +211,7 @@ If the importing member's role in the target project is: To import users: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. Select **Import from a project**. 1. Select the project. You can view only the projects for which you're a maintainer. @@ -236,7 +236,7 @@ Prerequisites: To remove a member from a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. Next to the project member you want to remove, select **Remove member**. 1. Optional. On the confirmation dialog, select the @@ -273,7 +273,7 @@ You can filter and sort members in a project. ### Display inherited members -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. In the **Filter members** box, select `Membership` `=` `Inherited`. 1. Press <kbd>Enter</kbd>. @@ -282,7 +282,7 @@ You can filter and sort members in a project. ### Display direct members -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. In the **Filter members** box, select `Membership` `=` `Direct`. 1. Press <kbd>Enter</kbd>. @@ -305,7 +305,7 @@ You can sort members by **Account**, **Access granted**, **Max role**, or **Last GitLab users can request to become a member of a project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the project you want to be a member of. +1. On the left sidebar, select **Search or go to** and find the project you want to be a member of. 1. By the project's name, select **Request Access**.  @@ -329,7 +329,7 @@ Prerequisite: - You must be the project owner. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Under **Project visibility**, select **Users can request access**. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 3780018bf11..deefe9040fa 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w When you want a group to have access to your project, you can invite [a group](../../group/index.md) to the project. -The group's members get access to the project, which becomes a *shared project*. +The group's direct and inherited members get access to the project, which becomes a *shared project*. ## Example @@ -53,11 +53,12 @@ must be at least as restrictive as that of the project. For example, you can inv > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. -You can share a project with a group by inviting that group to the project. +Similar to how you [share a group with another group](../../group/manage.md#share-a-group-with-another-group), +you can share a project with a group by inviting that group to the project. To invite a group to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. Select **Invite a group**. 1. **Select a group** you want to add to the project. @@ -99,7 +100,7 @@ For example, if a group member has the role of Developer, and the group is invit To view the maximum role assigned to a member: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Members**. 1. In the **Max role** column, view the user's maximum assigned role. @@ -109,7 +110,7 @@ In a group, a shared project is a project to which the group members gained acce To view a group's shared projects: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the group page, select the **Shared projects** tab. A list of shared projects is displayed. diff --git a/doc/user/project/merge_requests/ai_in_merge_requests.md b/doc/user/project/merge_requests/ai_in_merge_requests.md index c1aa729d8c5..304717cf9fc 100644 --- a/doc/user/project/merge_requests/ai_in_merge_requests.md +++ b/doc/user/project/merge_requests/ai_in_merge_requests.md @@ -34,8 +34,7 @@ Provide feedback on this experimental feature in [issue 416537](https://gitlab.c - Title of the merge request - Contents of the description -- Source branch name -- Target branch name +- Diff of changes between the source branch's head and the target branch ## Summarize merge request changes **(ULTIMATE SAAS)** diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index cbbeac17355..87ff6376ebd 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -89,17 +89,15 @@ To edit a merge request approval rule: ## Add multiple approval rules -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1979) in GitLab 11.10. - In GitLab Premium and Ultimate tiers, you can enforce multiple approval rules on a merge request, and multiple default approval rules for a project. If your tier supports multiple default rules: - When [adding](#add-an-approval-rule) or [editing](#edit-an-approval-rule) an approval rule - for a project, GitLab displays the **Add approval rule** button even after a rule is defined. + for a project, GitLab displays **Add approval rule**, even after a rule is defined. - When editing or overriding multiple approval rules [on a merge request](#edit-or-override-merge-request-approval-rules), GitLab - displays the **Add approval rule** button even after a rule is defined. + displays **Add approval rule**, even after a rule is defined. When an [eligible approver](#eligible-approvers) approves a merge request, it reduces the number of approvals left (the **Approvals** column) for all rules that the approver belongs to: @@ -111,8 +109,6 @@ For an overview, see [Multiple Approvers](https://www.youtube.com/watch?v=8JQJ58 ## Eligible approvers -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10294) in GitLab 13.3, when an eligible approver comments on a merge request, it appears in the **Commented by** column of the Approvals widget. - To be eligible as an approver for a project, a user must be a member of one or more of these: @@ -140,15 +136,20 @@ users were not explicitly listed in the approval rules. ### Group approvers -You can add a group of users as approvers, but those users count as approvers only if -they have **direct membership** to the group. Inherited members do not count. Group approvers are -restricted to only groups [with share access to the project](../../members/share_project_with_groups.md). +You can add a group of users as approvers. All **direct members** of this group +can approve the rule. **Inherited members** cannot approve the rule. + +Typically the group is a subgroup in your top-level namespace, unless you are +collaborating with an external group. If you are collaborating with another group, +you must [share access to the project](../../members/share_project_with_groups.md) +before assigning the group as a group approver. A user's membership in an approvers group affects their individual ability to approve in these ways: -- A user already part of a group approver who is later added as an individual approver - counts as one approver, and not two. +- Inherited members are not considered approvers. Only direct members can approve merge requests. +- A user from a group approver group who is later _also_ added as an individual approver + counts as one approver, not two. - Merge request authors do not count as eligible approvers on their own merge requests by default. To change this behavior, disable the [**Prevent author approval**](settings.md#prevent-approval-by-author) @@ -157,6 +158,12 @@ approve in these ways: [**Prevent committers approval**](settings.md#prevent-approvals-by-users-who-add-commits) project setting. +NOTE: +Creating multiple top-level groups to manage groups of users is not necessary, and is +discouraged because GitLab Free [is limited to 5 group members](https://about.gitlab.com/pricing/faq-efficient-free-tier/). +Managing groups of users using subgroups in your top-level namespace enables you +to use a single license. + ### Code owners as eligible approvers > Moved to GitLab Premium in 13.9. @@ -261,12 +268,9 @@ For more information, see [Coverage check approval rule](../../../../ci/testing/ ## Security Approvals **(ULTIMATE ALL)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357021) in GitLab 15.0. +> - Security approvals moved to merge request approvals settings [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357021) in GitLab 15.0. > - Bot comment for approvals [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/411656) in GitLab 16.2 [with a flag](../../../../administration/feature_flags.md) named `security_policy_approval_notification`. Enabled by default. - -FLAG: -On self-managed GitLab, by default the bot comment for approvals is available. To hide the feature, an administrator can [disable the feature flag](../../../../administration/feature_flags.md) named `security_policy_approval_notification`. -On GitLab.com, the bot comment for approvals is available. +> - Bot comment for approvals [generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130827) in GitLab 16.3. Feature flag `security_policy_approval_notification` removed. You can use [scan result policies](../../../application_security/policies/scan-result-policies.md#scan-result-policy-editor) to define security approvals based on the status of vulnerabilities in the merge request and the default branch. Details for each security policy is shown in the Security Approvals section of your Merge Request configuration. diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index b520ee41493..ae16eb2a790 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -65,6 +65,7 @@ this setting, unless you configure one of these options: > - Moved to GitLab Premium in 13.9. > - [Feature flag `keep_merge_commits_for_approvals`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127744) added in GitLab 16.3 to also include merge commits in this check. +> - [Feature flag `keep_merge_commits_for_approvals`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131778) removed in GitLab 16.5. This check now includes merge commits. By default, users who commit to a merge request can still approve it. At both the project level or [instance level](../../../admin_area/merge_requests_approvals.md), diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index ab882af7eda..18f19197f4c 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -158,7 +158,7 @@ rebases and file changes. To add a comment to a merge request file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. Select **Changes**. 1. In the header for the file you want to comment on, select **Comment** (**{comment}**). diff --git a/doc/user/project/merge_requests/cherry_pick_changes.md b/doc/user/project/merge_requests/cherry_pick_changes.md index d9b2ecf0806..ef1554f3b86 100644 --- a/doc/user/project/merge_requests/cherry_pick_changes.md +++ b/doc/user/project/merge_requests/cherry_pick_changes.md @@ -52,7 +52,7 @@ Commit `G` is added after the cherry-pick. After a merge request is merged, you can cherry-pick all changes introduced by the merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**, and find your merge request. 1. Scroll to the merge request reports section, and find the **Merged by** report. 1. In the upper-right corner, select **Cherry-pick**: @@ -70,7 +70,7 @@ You can cherry-pick a single commit from multiple locations in your GitLab proje To cherry-pick a commit from the list of all commits for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Commits**. 1. Select the [title](https://git-scm.com/docs/git-commit#_discussion) of the commit you want to cherry-pick. 1. In the upper-right corner, select **Options > Cherry-pick** to show the cherry-pick modal. @@ -84,7 +84,7 @@ You can cherry-pick commits from any merge request in your project, regardless o whether the merge request is open or closed. To cherry-pick a commit from the list of commits included in a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**, and find your merge request. 1. In the merge request's secondary menu, select **Commits** to display the commit details page. 1. Select the [title](https://git-scm.com/docs/git-commit#_discussion) of the commit you want to cherry-pick. @@ -98,7 +98,7 @@ list of commits included in a merge request: You can cherry-pick from the list of previous commits affecting an individual file when you view that file in your project's Git repository: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository** and go to the file changed by the commit. 1. Select **History**, then select the [title](https://git-scm.com/docs/git-commit#_discussion) diff --git a/doc/user/project/merge_requests/commit_templates.md b/doc/user/project/merge_requests/commit_templates.md index 1fc7188ffea..5ca097da29f 100644 --- a/doc/user/project/merge_requests/commit_templates.md +++ b/doc/user/project/merge_requests/commit_templates.md @@ -8,7 +8,7 @@ type: reference, howto # Commit message templates **(FREE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) squash commit templates in GitLab 14.6. +> - Squash commit templates [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6. GitLab uses commit templates to create default messages for specific types of commits. These templates encourage commit messages to follow a particular format, @@ -29,7 +29,7 @@ Prerequisite: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Depending on the type of template you want to create, scroll to either [**Merge commit message template**](#default-template-for-merge-commits) or @@ -66,13 +66,13 @@ GitLab creates a squash commit message with this template: ## Supported variables in commit templates > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/346805) `first_commit` and `first_multiline_commit` variables in GitLab 14.6. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75639) `url`, `approved_by`, and `merged_by` variables in GitLab 14.7. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/20421) `co_authored_by` variable in GitLab 14.7. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/26303) `all_commits` variable in GitLab 14.9. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/378352) `reviewed_by` variable in GitLab 15.7. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/199823) `local_reference` variable in GitLab 16.1. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128553) `source_project_id` variables in GitLab 16.3. +> - `first_commit` and `first_multiline_commit` variables [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346805) in GitLab 14.6. +> - `url`, `approved_by`, and `merged_by` variables [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75639) in GitLab 14.7. +> - `co_authored_by` variable [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20421) in GitLab 14.7. +> - `all_commits` variable [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26303) in GitLab 14.9. +> - `reviewed_by` variable [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378352) in GitLab 15.7. +> - `local_reference` variable [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/199823) in GitLab 16.1. +> - `source_project_id` variables [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128553) in GitLab 16.3. Commit message templates support these variables: diff --git a/doc/user/project/merge_requests/commits.md b/doc/user/project/merge_requests/commits.md index ddd2a0ddcb5..502dfd7acab 100644 --- a/doc/user/project/merge_requests/commits.md +++ b/doc/user/project/merge_requests/commits.md @@ -18,7 +18,7 @@ From this tab, you can review commit messages and copy a commit's SHA when you n To see the commits included in a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**, then select your merge request. 1. To show a list of the commits in the merge request, newest first, select **Commits** . To read more about the commit, select **Toggle commit description** (**{ellipsis_h}**) @@ -48,7 +48,7 @@ if another merge request: To add previously merged commits to a merge request for more context: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**, then select your merge request. 1. Select **Commits**. 1. Scroll to the end of the list of commits, and select **Add previously merged commits**. @@ -63,7 +63,7 @@ force push. To add discussion to a specific commit: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Commits**. 1. Below the commits, in the **Comment** field, enter a comment. 1. Save your comment as either a standalone comment, or a thread: @@ -74,7 +74,7 @@ To add discussion to a specific commit: To view the changes between previously merged commits: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**, then select your merge request. 1. Select **Changes**. 1. By **Compare** (**{file-tree}**), select the commits to compare: diff --git a/doc/user/project/merge_requests/conflicts.md b/doc/user/project/merge_requests/conflicts.md index fefc6aabddf..e132ddfb729 100644 --- a/doc/user/project/merge_requests/conflicts.md +++ b/doc/user/project/merge_requests/conflicts.md @@ -59,7 +59,7 @@ in the user interface, and you can also resolve conflicts locally through the co To resolve less-complex conflicts from the GitLab user interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find the merge request. 1. Select **Overview**, and scroll to the merge request reports section. 1. Find the merge conflicts message, and select **Resolve conflicts**. @@ -84,7 +84,7 @@ Some merge conflicts are more complex, requiring you to manually modify lines to resolve their conflicts. Use the merge conflict resolution editor to resolve complex conflicts in the GitLab interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find the merge request. 1. Select **Overview**, and scroll to the merge request reports section. 1. Find the merge conflicts message, and select **Resolve conflicts**. diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index e31460c1997..ec269bec84d 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -19,7 +19,7 @@ to streamline merge request creation. You can create a merge request from the list of merge requests. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. In the upper-right corner, select **New merge request**. 1. Select a source and target branch and then **Compare branches and continue**. @@ -95,7 +95,7 @@ You can create a merge request when you add, edit, or upload a file to a reposit You can create a merge request when you create a branch. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. Type a branch name and select **New branch**. 1. Above the file list, on the right side, select **Create merge request**. @@ -142,7 +142,7 @@ to reduce the need for editing merge requests manually through the UI. You can create a merge request from your fork to contribute back to the main project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select your fork of the repository. 1. On the left sidebar, select **Code > Merge requests**, and select **New merge request**. 1. In the **Source branch** dropdown list box, select the branch in your forked repository as the source branch. @@ -171,7 +171,7 @@ Prerequisites: To create a merge request by sending an email: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. In the upper-right corner, select **Email a new merge request to this project**. An email address is displayed. Copy this address. @@ -216,7 +216,7 @@ scenarios when you create a new merge request: To have merge requests from a fork by default target your own fork (instead of the upstream project), you can change the default. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Target project** section, select the option you want to use for your default target project. diff --git a/doc/user/project/merge_requests/csv_export.md b/doc/user/project/merge_requests/csv_export.md index 15b2e53d7d9..cc7fa050766 100644 --- a/doc/user/project/merge_requests/csv_export.md +++ b/doc/user/project/merge_requests/csv_export.md @@ -12,7 +12,7 @@ Export all the data collected from a project's merge requests into a comma-separ To export merge requests to a CSV file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Add any searches or filters. This can help you keep the size of the CSV file under the 15 MB limit. The limit ensures the file can be emailed to a variety of email providers. diff --git a/doc/user/project/merge_requests/dependencies.md b/doc/user/project/merge_requests/dependencies.md index b80698f99c3..e208785fd63 100644 --- a/doc/user/project/merge_requests/dependencies.md +++ b/doc/user/project/merge_requests/dependencies.md @@ -57,7 +57,7 @@ information about the dependency: To view dependency information on a merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and identify your merge request. 1. Scroll to the merge request reports area. Dependent merge requests display information about the total number of dependencies set, such as @@ -105,7 +105,7 @@ Prerequisite: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and identify your merge request. 1. Select **Edit**. 1. In **Merge request dependencies**, paste either the reference or the full URL @@ -120,7 +120,7 @@ Prerequisite: - You must have a role in the project that allows you to edit merge requests. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and identify your merge request. 1. Select **Edit**. 1. Scroll to **Merge request dependencies** and select **Remove** next to the reference diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 3ad07d0377e..ed762979ff1 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -46,7 +46,7 @@ You can view merge requests for your project, group, or yourself. To view all merge requests for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. Or, to use a [keyboard shortcut](../../shortcuts.md), press <kbd>g</kbd> + <kbd>m</kbd>. @@ -55,7 +55,7 @@ Or, to use a [keyboard shortcut](../../shortcuts.md), press <kbd>g</kbd> + <kbd> To view merge requests for all projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Code > Merge requests**. If your group contains subgroups, this view also displays merge requests from the subgroup projects. @@ -64,7 +64,7 @@ If your group contains subgroups, this view also displays merge requests from th To view all merge requests assigned to you: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**). +1. On the left sidebar, select **Search or go to**. 1. From the dropdown list, select **Merge requests assigned to me**. or: @@ -85,16 +85,16 @@ or: To filter the list of merge requests: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Above the list of merge requests, select **Search or filter results...**. 1. From the dropdown list, select the attribute you wish to filter by. Some examples: - [**By environment or deployment date**](#by-environment-or-deployment-date). - **ID**: Enter filter `#30` to return only merge request 30. - User filters: Type (or select from the dropdown list) any of these filters to display a list of users: - - **Approved-By**, for merge requests already approved by a user. **(PREMIUM)**. + - **Approved-By**, for merge requests already approved by a user. **(PREMIUM ALL)**. - **Approver**, for merge requests that this user is eligible to approve. - (For more information, read about [Code owners](../codeowners/index.md)). **(PREMIUM)** + (For more information, read about [Code owners](../codeowners/index.md)). **(PREMIUM ALL)** - **Reviewer**, for merge requests reviewed by this user. 1. Select or type the operator to use for filtering the attribute. The following operators are available: @@ -156,7 +156,7 @@ To assign the merge request to a user, use the `/assign @user` [quick action](../quick_actions.md#issues-merge-requests-and-epics) in a text area in a merge request, or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. On the right sidebar, expand the right sidebar and locate the **Assignees** section. 1. Select **Edit**. @@ -176,7 +176,7 @@ accountable for it: To assign multiple assignees to a merge request, use the `/assign @user` [quick action](../quick_actions.md#issues-merge-requests-and-epics) in a text area, or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. On the right sidebar, expand the right sidebar and locate the **Assignees** section. 1. Select **Edit** and, from the dropdown list, select all users you want @@ -312,7 +312,7 @@ On GitLab.com, this feature is enabled for GitLab team members only. To understand the history of a merge request, filter its activity feed to show you only the items that are relevant to you. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Select a merge request. 1. Scroll to **Activity**. @@ -341,22 +341,7 @@ sort order by clicking the sort button on the right. > Resolving comments individually was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/28750) in GitLab 13.6. -In a merge request, you can resolve a thread when you want to finish a conversation. - -Prerequisites: - -- You must have at least the Developer role - or be the author of the change being reviewed. -- Resolvable threads can be added only to merge requests. It doesn't work - for comments in issues, commits, or snippets. - -To resolve a thread: - -1. Go to the thread. -1. Do one of the following: - - In the upper-right corner of the original comment, select **Resolve thread** (**{check-circle}**). - - Below the last reply, in the **Reply** field, select **Resolve thread**. - - Below the last reply, in the **Reply** field, enter text, select the **Resolve thread** checkbox, and select **Add comment now**. +In a merge request, you can [resolve a thread](../../discussions/index.md#resolve-a-thread) when you want to finish a conversation. At the top of the page, the number of unresolved threads is updated: @@ -390,7 +375,7 @@ You can prevent merge requests from being merged until all threads are resolved. When this setting is enabled, the **Unresolved threads** counter in a merge request is shown in orange when at least one thread remains unresolved. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge checks** section, select the **All threads must be resolved** checkbox. 1. Select **Save changes**. @@ -400,7 +385,7 @@ is shown in orange when at least one thread remains unresolved. You can set merge requests to automatically resolve threads when lines are modified with a new push. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Merge options** section, select **Automatically resolve merge request diff threads when they become outdated**. diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index 16482d92d7e..77dcb269071 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -39,7 +39,7 @@ To do this when pushing from the command line, use the `merge_request.merge_when To do this from the GitLab user interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Select the merge request to edit. 1. Scroll to the merge request reports section. @@ -63,7 +63,7 @@ Prerequisites: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Select the merge request to edit. 1. Scroll to the merge request reports section. @@ -79,7 +79,7 @@ merge. This configuration works for both: - GitLab CI/CD pipelines. - Pipelines run from an [external CI integration](../integrations/index.md#available-integrations). -As a result, [disabling GitLab CI/CD pipelines](../../../ci/enable_or_disable_ci.md) +As a result, [disabling GitLab CI/CD pipelines](../../../ci/enable_or_disable_ci.md#disable-cicd-in-a-project) does not disable this feature, but you can use pipelines from external CI providers with it. @@ -90,7 +90,7 @@ Prerequisites: To enable this setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Scroll to **Merge checks**, and select **Pipelines must succeed**. This setting also prevents merge requests from being merged if there is no pipeline, @@ -111,7 +111,7 @@ Prerequisite: To change this behavior: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Under **Merge checks**: - Select **Pipelines must succeed**. diff --git a/doc/user/project/merge_requests/methods/index.md b/doc/user/project/merge_requests/methods/index.md index 7eac0e8839a..959ada4928e 100644 --- a/doc/user/project/merge_requests/methods/index.md +++ b/doc/user/project/merge_requests/methods/index.md @@ -26,7 +26,7 @@ gitGraph ## Configure a project's merge method -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Select your desired **Merge method** from these options: - Merge commit diff --git a/doc/user/project/merge_requests/revert_changes.md b/doc/user/project/merge_requests/revert_changes.md index 14e2ff84eae..7e6bf606f10 100644 --- a/doc/user/project/merge_requests/revert_changes.md +++ b/doc/user/project/merge_requests/revert_changes.md @@ -30,7 +30,7 @@ Prerequisites: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and identify your merge request. 1. Scroll to the merge request reports area, and find the report showing when the merge request was merged. @@ -55,7 +55,7 @@ Prerequisites: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. If you know the merge request that contains the commit: 1. Select **Code > Merge requests**, then identify and select your merge request. 1. Select **Commits**, then select the title of the commit you want to revert. This displays the commit in the **Changes** tab of your merge request. diff --git a/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v15_9.png b/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v15_9.png Binary files differdeleted file mode 100644 index 70db0d79eaf..00000000000 --- a/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v15_9.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v16_3.png b/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v16_3.png Binary files differnew file mode 100644 index 00000000000..9c3ecebd395 --- /dev/null +++ b/doc/user/project/merge_requests/reviews/img/suggested_reviewers_v16_3.png diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index e4882134654..c09071e856c 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -10,7 +10,7 @@ type: index, reference [Merge requests](../index.md) are the primary method of making changes to files in a GitLab project. [Create and submit a merge request](../creating_merge_requests.md) to propose changes. Your team leaves [comments](../../../discussions/index.md) on -your merge request, and makes [code suggestions](suggestions.md) you can accept +your merge request, and makes [Code Suggestions](suggestions.md) you can accept from the user interface. When your work is reviewed, your team members can choose to accept or reject it. @@ -21,17 +21,24 @@ review merge requests in Visual Studio Code. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Merge request review](https://www.youtube.com/watch?v=2MayfXKpU08&list=PLFGfElNsQthYDx0A_FaNNfUm9NHsK6zED&index=183). -## Suggested reviewers **(ULTIMATE SAAS)** +## GitLab Duo Suggested Reviewers **(ULTIMATE SAAS)** > - [Introduced](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/review-recommender/-/epics/3) in GitLab 15.4 as a [Beta](../../../../policy/experiment-beta-support.md#beta) feature [with a flag](../../../../administration/feature_flags.md) named `suggested_reviewers_control`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/368356) in GitLab 15.6. > - Beta designation [removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113058) in GitLab 15.10. -GitLab can suggest reviewers. Using the changes in a merge request and a project's contribution graph, machine learning suggestions appear in the reviewer section of the right sidebar. +GitLab uses machine learning to suggest reviewers for your merge request. - +To suggest reviewers, GitLab uses: -For more information, see [Data usage in Suggested Reviewers](data_usage.md). +- The changes in the merge request +- The project's contribution graph + +GitLab Duo Suggested Reviewers also integrates with Code Owners, profile status, and merge request rules, helping you make a more informed decision when choosing reviewers that can meet your review criteria. + + + +For more information, see [Data usage in GitLab Duo Suggested Reviewers](data_usage.md). ### Enable suggested reviewers @@ -73,11 +80,40 @@ If you [approve a merge request](../approvals/index.md#approve-a-merge-request) are shown in the reviewer list, a green check mark **{check-circle-filled}** displays next to your name. +### Request a review + +To assign a reviewer to a merge request, in a text area in +the merge request, use the `/assign_reviewer @user` +[quick action](../../quick_actions.md#issues-merge-requests-and-epics). Alternatively: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Code > Merge requests** and find your merge request. +1. On the right sidebar, in the **Reviewers** section, select **Edit**. +1. Search for the user you want to assign, and select the user. + +The merge request is added to the user's review requests. + +#### From multiple users **(PREMIUM ALL)** + +> Moved to GitLab Premium in 13.9. + +To assign multiple reviewers to a merge request, in a text area in +the merge request, use the `/assign_reviewer @user` +[quick action](../../quick_actions.md#issues-merge-requests-and-epics). Alternatively: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Code > Merge requests** and find your merge request. +1. On the right sidebar, in the **Reviewers** section, select **Edit**. +1. From the dropdown list, select all the users you want + to assign to the merge request. + +To remove a reviewer, clear the user from the same dropdown list. + ### Download merge request changes as a diff To download the changes included in a merge request as a diff: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Select your merge request. 1. In the upper-right corner, select **Code > Plain diff**. @@ -100,7 +136,7 @@ curl "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/000000.diff" | git a To download the changes included in a merge request as a patch file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. 1. Select your merge request. 1. In the upper-right corner, select **Code > Patches**. diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 4857d58933a..2b046399c4e 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -14,7 +14,7 @@ merge request, authored by the user who suggested the changes. ## Create suggestions -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. On the secondary menu, select **Changes**. 1. Find the lines of code you want to change. @@ -92,7 +92,7 @@ Prerequisites: To apply suggested changes directly from the merge request: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. Find the comment containing the suggestion you want to apply. - To apply suggestions individually, select **Apply suggestion**. @@ -140,7 +140,7 @@ Merge requests created from forks use the template defined in the target project To meet your project's needs, you can customize these messages and include other placeholder variables: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Scroll to **Merge suggestions**, and alter the text to meet your needs. See [Supported variables](#supported-variables) for a list of placeholders @@ -172,7 +172,7 @@ For example, to customize the commit message to output To reduce the number of commits added to your branch, you can apply multiple suggestions in a single commit. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. For each suggestion you want to apply, and select **Add suggestion to batch**. 1. Optional. To remove a suggestion, select **Remove from batch**. diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index ea999fe039a..a6d55ee44e2 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -71,7 +71,7 @@ Prerequisites: To configure the default squashing behavior for all merge requests in your project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. In the **Squash commits when merging** section, select your desired behavior: - **Do not allow**: Squashing is never performed, and the option is not displayed. diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index e0e65c99433..f87d379b974 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -36,7 +36,7 @@ see [epic 3869](https://gitlab.com/groups/gitlab-org/-/epics/3869). By default, merge requests in projects can be merged even if external status checks fail. To block the merging of merge requests when external checks fail: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Select the **Status checks must succeed** checkbox. 1. Select **Save changes**. diff --git a/doc/user/project/milestones/burndown_and_burnup_charts.md b/doc/user/project/milestones/burndown_and_burnup_charts.md index 7c9a5a37292..65cc9477c49 100644 --- a/doc/user/project/milestones/burndown_and_burnup_charts.md +++ b/doc/user/project/milestones/burndown_and_burnup_charts.md @@ -13,7 +13,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Burndown charts -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6495) to GitLab 11.2 for group milestones. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6903) [fixed burndown charts](#fixed-burndown-charts) in GitLab 13.6. > - Moved to GitLab Premium in 13.9. @@ -32,13 +31,13 @@ For an overview, check the video demonstration on [Mapping work versus time with To view a project's burndown chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Milestones**. 1. Select a milestone from the list. To view a group's burndown chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Milestones**. 1. Select a milestone from the list. @@ -112,13 +111,13 @@ Burnup charts show the assigned and completed work for a milestone. To view a project's burnup chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Milestones**. 1. Select a milestone from the list. To view a group's burnup chart: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Plan > Milestones**. 1. Select a milestone from the list. diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 4bb751aedc5..328e56e2bd8 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -37,7 +37,7 @@ For information about project and group milestones API, see: To view the milestone list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. In a project, GitLab displays milestones that belong to the project. @@ -46,7 +46,7 @@ In a group, GitLab displays milestones that belong to the group and all projects ### View milestones in a project with issues turned off If a project has issue tracking -[turned off](../settings/index.md#configure-project-visibility-features-and-permissions), +[turned off](../settings/index.md#configure-project-features-and-permissions), to get to the milestones page, enter its URL. To do so: @@ -66,7 +66,7 @@ You might not see some milestones because they're in projects or groups you're n To do so: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. On the left sidebar, select **Milestones**. @@ -121,7 +121,7 @@ Prerequisites: To create a milestone: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. 1. Select **New milestone**. 1. Enter the title. @@ -140,7 +140,7 @@ Prerequisites: To edit a milestone: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. 1. Select a milestone's title. 1. In the top right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Edit**. @@ -157,7 +157,7 @@ Prerequisites: To edit a milestone: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. 1. Select a milestone's title. 1. In the top right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Delete**. @@ -183,7 +183,7 @@ Prerequisites: To promote a project milestone: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Milestones**. 1. Either: - Select **Promote to Group Milestone** (**{level-up}**) next to the milestone you want to promote. diff --git a/doc/user/project/ml/experiment_tracking/index.md b/doc/user/project/ml/experiment_tracking/index.md index d4516746afc..0aad9a69743 100644 --- a/doc/user/project/ml/experiment_tracking/index.md +++ b/doc/user/project/ml/experiment_tracking/index.md @@ -70,7 +70,7 @@ on how to use GitLab as a backend for the MLflow Client. To list the current active experiments, either go to `https/-/ml/experiments` or: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Model experiments**. 1. To display all candidates that have been logged, along with their metrics, parameters, and metadata, select an experiment. 1. To display details for a candidate, select **Details**. diff --git a/doc/user/project/ml/experiment_tracking/mlflow_client.md b/doc/user/project/ml/experiment_tracking/mlflow_client.md index 3e78cc5b7f2..9cedb5780ed 100644 --- a/doc/user/project/ml/experiment_tracking/mlflow_client.md +++ b/doc/user/project/ml/experiment_tracking/mlflow_client.md @@ -24,7 +24,7 @@ Prerequisites: - A [personal](../../../../user/profile/personal_access_tokens.md), [project](../../../../user/project/settings/project_access_tokens.md), or [group](../../../../user/group/settings/group_access_tokens.md) access token with at least the Developer role and the `api` permission. - The project ID. To find the project ID: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. To use MLflow client compatibility from a local environment: @@ -83,6 +83,7 @@ tested. More information can be found in the [MLflow Documentation](https://www. | `set_experiment` | Yes | 15.11 | | | `get_run` | Yes | 15.11 | | | `start_run` | Yes | 15.11 | (16.3) If a name is not provided, the candidate receives a random nickname. | +| `search_runs` | Yes | 15.11 | (16.4) `experiment_ids` supports only a single experiment ID with order by column or metric. | | `log_artifact` | Yes with caveat | 15.11 | (15.11) `artifact_path` must be empty. Does not support directories. | | `log_artifacts` | Yes with caveat | 15.11 | (15.11) `artifact_path` must be empty. Does not support directories. | | `log_batch` | Yes | 15.11 | | diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 8bf4875ba1e..d8e4fce41ef 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -43,7 +43,7 @@ this document for an [overview on DNS records](dns_concepts.md). To add your custom domain to GitLab Pages: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. In the upper-right corner, select **New Domain**. 1. In **Domain**, enter the domain name. @@ -113,7 +113,7 @@ Subdomains (`subdomain.example.com`) require: Whether it's a user or a project website, the DNS record should point to your Pages domain (`namespace.gitlab.io`), -without any `/project-name`. +without any path.  @@ -154,7 +154,7 @@ If you're using Cloudflare, check After you have added all the DNS records: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. 1. In **Verification status**, select **Retry verification** (**{retry}**). @@ -246,7 +246,7 @@ meet these requirements. - To add the certificate at the time you add a new domain: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. In the upper-right corner, select **New Domain**. 1. In **Domain**, enter the domain name. @@ -255,7 +255,7 @@ meet these requirements. - To add the certificate to a domain previously added: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. 1. In **Certificate**, turn off the **Automatic certificate management using Let's Encrypt** toggle to add an [SSL/TLS certificate](#adding-an-ssltls-certificate-to-pages). @@ -283,7 +283,7 @@ domain (as long as you've set a valid certificate for it). To enable this setting: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Select the **Force HTTPS (requires valid certificates)** checkbox. 1. Select **Save changes**. @@ -303,7 +303,7 @@ You can edit a custom domain to: To edit a custom domain: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. @@ -313,7 +313,7 @@ After a custom domain is deleted, the domain is no longer verified in GitLab and To delete and remove a custom domain: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Deploy > Pages**. 1. Next to the domain name, select **Remove**. 1. When prompted, select **Remove domain**. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index 5c1fc41d947..3d9fe4b04e9 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -42,7 +42,7 @@ For **self-managed** GitLab instances, make sure your administrator has Once you've met the requirements, enable Let's Encrypt integration: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. 1. Turn on the **Automatic certificate management using Let's Encrypt** toggle. @@ -69,7 +69,7 @@ associated Pages domain. GitLab also renews it automatically. If you get an error **Something went wrong while obtaining the Let's Encrypt certificate**, first, make sure that your pages site is set to "Everyone" in your project's **Settings > General > Visibility**. This allows the Let's Encrypt Servers reach your pages site. Once this is confirmed, you can try obtaining the certificate again by following these steps: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. 1. In **Verification status**, select **Retry verification** (**{retry}**). @@ -84,7 +84,7 @@ If you get an error **Something went wrong while obtaining the Let's Encrypt cer If you've enabled Let's Encrypt integration, but a certificate is absent after an hour and you see the message, "GitLab is obtaining a Let's Encrypt SSL certificate for this domain. This process can take some time. Please try again later.", try to remove and add the domain for GitLab Pages again by following these steps: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Next to the domain name, select **Remove**. 1. [Add the domain again, and verify it](index.md#1-add-a-custom-domain). diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index f8ee3f10d1e..90fc1a72eb3 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -16,7 +16,7 @@ Use a `.gitlab-ci.yml` template when you have an existing project that you want Your GitLab repository should contain files specific to an SSG, or plain HTML. After you complete these steps, you may have to do additional configuration for the Pages site to generate properly. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the **Add** (**{plus}**) dropdown list, select **New file**. 1. From the **Select a template type** dropdown list, select `.gitlab-ci.yml`. 1. From the **Apply a template** dropdown list, in the **Pages** section, select the name of your SSG. diff --git a/doc/user/project/pages/getting_started/pages_ui.md b/doc/user/project/pages/getting_started/pages_ui.md index ab3c12427b3..8fa927bd61c 100644 --- a/doc/user/project/pages/getting_started/pages_ui.md +++ b/doc/user/project/pages/getting_started/pages_ui.md @@ -33,7 +33,7 @@ a pipeline deploys your Pages website. To complete the setup and generate a GitLab Pages deployment: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. A **Get Started with Pages** form appears. If this form is not available, diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index e44a7bf347f..ec511ee0a5f 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -25,13 +25,13 @@ For GitLab self-managed instances, replace `example.io` with your instance's Pages domain. For GitLab.com, Pages domains are `*.gitlab.io`. -| Type of GitLab Pages | The name of the project created in GitLab | Website URL | +| Type of GitLab Pages | Example path of a project in GitLab | Website URL | | -------------------- | ------------ | ----------- | -| User pages | `username.example.io` | `http(s)://username.example.io` | -| Group pages | `groupname.example.io` | `http(s)://groupname.example.io` | -| Project pages owned by a user | `projectname` | `http(s)://username.example.io/projectname` | -| Project pages owned by a group | `projectname` | `http(s)://groupname.example.io/projectname`| -| Project pages owned by a subgroup | `subgroup/projectname` | `http(s)://groupname.example.io/subgroup/projectname`| +| User pages | `username/username.example.io` | `http(s)://username.example.io` | +| Group pages | `acmecorp/acmecorp.example.io` | `http(s)://acmecorp.example.io` | +| Project pages owned by a user | `username/my-website` | `http(s)://username.example.io/my-website` | +| Project pages owned by a group | `acmecorp/webshop` | `http(s)://acmecorp.example.io/webshop`| +| Project pages owned by a subgroup | `acmecorp/documentation/product-manual` | `http(s)://acmecorp.example.io/documentation/product-manual`| WARNING: There are some known [limitations](introduction.md#subdomains-of-subdomains) @@ -72,7 +72,7 @@ To understand Pages domains clearly, read the examples below. **General example:** - On GitLab.com, a project site is always available under - `https://namespace.gitlab.io/project-name` + `https://namespace.gitlab.io/project-slug` - On GitLab.com, a user or group website is available under `https://namespace.gitlab.io/` - On your GitLab instance, replace `gitlab.io` above with your @@ -86,7 +86,7 @@ The `baseurl` option might be named differently in some static site generators. Every Static Site Generator (SSG) default configuration expects to find your website under a (sub)domain (`example.com`), not in a subdirectory of that domain (`example.com/subdir`). Therefore, -whenever you publish a project website (`namespace.gitlab.io/project-name`), +whenever you publish a project website (for example, `namespace.gitlab.io/project-slug`), you must look for this configuration (base URL) on your static site generator's documentation and set it up to reflect this pattern. diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 4585ee2cecd..664c80e0351 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -47,9 +47,9 @@ depending on your static generator configuration. If the case of `404.html`, there are different scenarios. For example: -- If you use project Pages (served under `/projectname/`) and try to access - `/projectname/non/existing_file`, GitLab Pages tries to serve first - `/projectname/404.html`, and then `/404.html`. +- If you use project Pages (served under `/project-slug/`) and try to access + `/project-slug/non/existing_file`, GitLab Pages tries to serve first + `/project-slug/404.html`, and then `/404.html`. - If you use user or group Pages (served under `/`) and try to access `/non/existing_file` GitLab Pages tries to serve `/404.html`. - If you use a custom domain and try to access `/non/existing_file`, GitLab @@ -64,7 +64,7 @@ You can configure redirects for your site using a `_redirects` file. For more in To remove your pages: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Select **Remove pages**. @@ -87,7 +87,7 @@ For [group websites](../../project/pages/getting_started_part_one.md#user-and-gr the group must be at the top level and not a subgroup. For [project websites](../../project/pages/getting_started_part_one.md#project-website-examples), -you can create your project first and access it under `http(s)://namespace.example.io/projectname`. +you can create your project first and access it under `http(s)://namespace.example.io/project-path`. ## Enable unique domains @@ -99,7 +99,7 @@ By default, every project in a group shares the same domain, for example, `group To ensure your project uses a unique Pages domain, enable the unique domains feature for the project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Pages**. 1. Select the **Use unique domain** checkbox. 1. Select **Save changes**. @@ -308,7 +308,7 @@ For a list of known issues, see the GitLab [public issue tracker](https://gitlab This problem most likely results from a missing `index.html` file in the public directory. If after deploying a Pages site a 404 is encountered, confirm that the public directory contains an `index.html` file. If the file contains a different name -such as `test.html`, the Pages site can still be accessed, but the full path would be needed. For example: `https//group-name.pages.example.com/project-name/test.html`. +such as `test.html`, the Pages site can still be accessed, but the full path would be needed. For example: `https//group-name.pages.example.com/project-slug/test.html`. The contents of the public directory can be confirmed by [browsing the artifacts](../../../ci/jobs/job_artifacts.md#download-job-artifacts) from the latest pipeline. diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index 5f1f5bc59ae..ff1f138e5b7 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -47,14 +47,14 @@ To create redirects, create a configuration file named `_redirects` in the configured at the instance level. Only the first matching rules within the configured maximum are processed. The default file size limit is 64 KB, and the default maximum number of rules is 1,000. - If your GitLab Pages site uses the default domain name (such as - `namespace.gitlab.io/projectname`) you must prefix every rule with the project name: + `namespace.gitlab.io/project-slug`) you must prefix every rule with the path: ```plaintext - /projectname/wardrobe.html /projectname/narnia.html 302 + /project-slug/wardrobe.html /project-slug/narnia.html 302 ``` - If your GitLab Pages site uses [custom domains](custom_domains_ssl_tls_certification/index.md), - no project name prefix is needed. For example, if your custom domain is `example.com`, + no project path prefix is needed. For example, if your custom domain is `example.com`, your `_redirects` file would look like: ```plaintext @@ -69,7 +69,7 @@ the file instead of your redirect. For example, if the files `hello.html` and is ignored because `hello.html` exists: ```plaintext -/projectname/hello.html /projectname/world.html 302 +/project-slug/hello.html /project-slug/world.html 302 ``` GitLab does not support Netlify @@ -210,8 +210,7 @@ would **not** match a request URL like `/old/file`: ## Debug redirect rules If a redirect isn't working as expected, or you want to check your redirect syntax, visit -`https://[namespace.gitlab.io]/projectname/_redirects`, replacing `[namespace.gitlab.io]` with -your domain name. The `_redirects` file isn't served directly, but your browser +`[your pages url]/_redirects`. The `_redirects` file isn't served directly, but your browser displays a numbered list of your redirect rules, and whether the rule is valid or invalid: ```plaintext diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 505f1a530cd..fac07a1313a 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -95,7 +95,7 @@ Prerequisites: To protect a branch: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -133,7 +133,7 @@ Prerequisite: To protect a branch for all the projects in a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -157,7 +157,7 @@ Prerequisite: To protect multiple branches at the same time: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -193,7 +193,7 @@ from the command line or from a Git client application. To create a new branch through the user interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. Select **New branch**. 1. Fill in the branch name and select an existing branch, tag, or commit to @@ -205,7 +205,7 @@ To create a new branch through the user interface: You can force everyone to submit a merge request, rather than allowing them to check in directly to a protected branch: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -218,7 +218,7 @@ check in directly to a protected branch: You can allow everyone with write access to push to the protected branch. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -247,7 +247,7 @@ Prerequisites: To allow a deploy key to push to a protected branch: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -267,7 +267,7 @@ protected branches. To protect a new branch and enable force push: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -280,7 +280,7 @@ To protect a new branch and enable force push: To enable force pushes on branches that are already protected: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -322,7 +322,7 @@ the applicable rules have **Required approval from code owners** enabled. To protect a new branch and enable Code Owner's approval: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -333,7 +333,7 @@ To protect a new branch and enable Code Owner's approval: To enable Code Owner's approval on branches that are already protected: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected branches**. 1. Select **Add protected branch**. @@ -366,7 +366,7 @@ for details about the pipelines security model. Users with at least the Maintainer role can manually delete protected branches by using the GitLab web interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. Next to the branch you want to delete, select **Delete** (**{remove}**). 1. On the confirmation dialog, enter the branch name and select **Yes, delete protected branch**. diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 8686d02271c..b312acd49f4 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -31,7 +31,7 @@ Prerequisites: - You must have at least the Maintainer role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected tags**. 1. Select **Add new**. @@ -77,7 +77,9 @@ all matching tags: A tag and a branch with identical names can contain different commits. If your tags and branches use the same names, users running `git checkout` commands might check out the _tag_ `qa` when they instead meant to check out -the _branch_ `qa`. +the _branch_ `qa`. As an added security measure, avoid creating tags with the +same name as branches. Confusing the two could lead to potential +security or operational issues. To prevent this problem: @@ -111,7 +113,7 @@ Prerequisites: To allow a deploy key to create a protected tag: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Protected tags**. 1. From the **Tag** dropdown list, select the tag you want to protect. @@ -129,7 +131,7 @@ Prerequisite: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Tags**. 1. Next to the tag you want to delete, select **Delete** (**{remove}**). 1. On the confirmation dialog, enter the tag name and select **Yes, delete protected tag**. diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index d0a938e054d..e8451e3049d 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -6,119 +6,97 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Push options **(FREE ALL)** -GitLab supports using client-side [Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) -to perform various actions at the same time as pushing changes. Additionally, [Push Rules](repository/push_rules.md) offer server-side control and enforcement options. +When you push changes to a branch, you can use client-side +[Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt). +In Git 2.10 and later, use Git push options to: -Currently, there are push options available for: +- [Skip CI jobs](#push-options-for-gitlab-cicd) +- [Push to merge requests](#push-options-for-merge-requests) -- [Skipping CI jobs](#push-options-for-gitlab-cicd) -- [Merge requests](#push-options-for-merge-requests) - -NOTE: -Git push options are only available with Git 2.10 or newer. - -For Git versions 2.10 to 2.17 use `--push-option`: +In Git 2.18 and later, you can use either the long format (`--push-option`) or the shorter `-o`: ```shell -git push --push-option=<push_option> +git push -o <push_option> ``` -For version 2.18 and later, you can use the above format, or the shorter `-o`: +In Git 2.10 to 2.17, you must use the long format: ```shell -git push -o <push_option> +git push --push-option=<push_option> ``` +For server-side controls and enforcement of best practices, see +[push rules](repository/push_rules.md) and [server hooks](../../administration/server_hooks.md). + ## Push options for GitLab CI/CD You can use push options to skip a CI/CD pipeline, or pass CI/CD variables. -| Push option | Description | Introduced in version | -| ------------------------------ | ------------------------------------------------------------------------------------------- |---------------------- | -| `ci.skip` | Do not create a CI pipeline for the latest push. Only skips branch pipelines and not [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). This does not skip pipelines for CI integrations, such as Jenkins. | [11.7](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15643) | -| `ci.variable="<name>=<value>"` | Provide [CI/CD variables](../../ci/variables/index.md) to be used in a CI pipeline, if one is created due to the push. Only passes variables to branch pipelines and not [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). | [12.6](https://gitlab.com/gitlab-org/gitlab/-/issues/27983) | -| `integrations.skip_ci` | Skip push events for CI integrations, such as Atlassian Bamboo, Buildkite, Drone, Jenkins, and JetBrains TeamCity. | [16.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123837) | +| Push option | Description | Example | +|--------------------------------|-------------|---------| +| `ci.skip` | Do not create a CI/CD pipeline for the latest push. Skips only branch pipelines and not [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). This does not skip pipelines for CI/CD integrations, such as Jenkins. | `git push -o ci.skip` | +| `ci.variable="<name>=<value>"` | Provide [CI/CD variables](../../ci/variables/index.md) to the CI/CD pipeline, if one is created due to the push. Passes variables only to branch pipelines and not [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). | `git push -o ci.variable="MAX_RETRIES=10" -o ci.variable="MAX_TIME=600"` | +| `integrations.skip_ci` | Skip push events for CI/CD integrations, such as Atlassian Bamboo, Buildkite, Drone, Jenkins, and JetBrains TeamCity. Introduced in [GitLab 16.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123837). | `git push -o integrations.skip_ci` | -An example of using `ci.skip`: - -```shell -git push -o ci.skip -``` - -An example of passing some CI/CD variables for a pipeline: - -```shell -git push -o ci.variable="MAX_RETRIES=10" -o ci.variable="MAX_TIME=600" -``` +## Push options for merge requests -An example of using `integrations.skip_ci`: +Git push options can perform actions for merge requests while pushing changes: -```shell -git push -o integrations.skip_ci -``` +| Push option | Description | +|----------------------------------------------|-------------| +| `merge_request.create` | Create a new merge request for the pushed branch. | +| `merge_request.target=<branch_name>` | Set the target of the merge request to a particular branch or upstream project, such as: `git push -o merge_request.target=project_path/branch`. | +| `merge_request.merge_when_pipeline_succeeds` | Set the merge request to [merge when its pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md). | +| `merge_request.remove_source_branch` | Set the merge request to remove the source branch when it's merged. | +| `merge_request.title="<title>"` | Set the title of the merge request. For example: `git push -o merge_request.title="The title I want"`. | +| `merge_request.description="<description>"` | Set the description of the merge request. For example: `git push -o merge_request.description="The description I want"`. | +| `merge_request.draft` | Mark the merge request as a draft. For example: `git push -o merge_request.draft`. Introduced in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/296673). | +| `merge_request.milestone="<milestone>"` | Set the milestone of the merge request. For example: `git push -o merge_request.milestone="3.0"`. Introduced in [GitLab 14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63960). | +| `merge_request.label="<label>"` | Add labels to the merge request. If the label does not exist, it is created. For example, for two labels: `git push -o merge_request.label="label1" -o merge_request.label="label2"`. | +| `merge_request.unlabel="<label>"` | Remove labels from the merge request. For example, for two labels: `git push -o merge_request.unlabel="label1" -o merge_request.unlabel="label2"`. | +| `merge_request.assign="<user>"` | Assign users to the merge request. Accepts username or user ID. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. Support for usernames added in [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/344276). | +| `merge_request.unassign="<user>"` | Remove assigned users from the merge request. Accepts username or user ID. For example, for two users: `git push -o merge_request.unassign="user1" -o merge_request.unassign="user2"`. Support for usernames added in [GitLab 15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/344276). | -## Push options for merge requests +## Formats for push options -You can use Git push options to perform certain actions for merge requests at the same -time as pushing changes: - -| Push option | Description | Introduced in version | -| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------- | -| `merge_request.create` | Create a new merge request for the pushed branch. | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | -| `merge_request.target=<branch_name>` | Set the target of the merge request to a particular branch or upstream project, such as: `git push -o merge_request.target=project_path/branch` | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | -| `merge_request.merge_when_pipeline_succeeds` | Set the merge request to [merge when its pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md). | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | -| `merge_request.remove_source_branch` | Set the merge request to remove the source branch when it's merged. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.title="<title>"` | Set the title of the merge request. For example: `git push -o merge_request.title="The title I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.description="<description>"` | Set the description of the merge request. For example: `git push -o merge_request.description="The description I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | -| `merge_request.draft` | Mark the merge request as a draft. For example: `git push -o merge_request.draft`. | [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/296673) | -| `merge_request.milestone="<milestone>"` | Set the milestone of the merge request. For example: `git push -o merge_request.milestone="3.0"`. | [14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63960) | -| `merge_request.label="<label>"` | Add labels to the merge request. If the label does not exist, it is created. For example, for two labels: `git push -o merge_request.label="label1" -o merge_request.label="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | -| `merge_request.unlabel="<label>"` | Remove labels from the merge request. For example, for two labels: `git push -o merge_request.unlabel="label1" -o merge_request.unlabel="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | -| `merge_request.assign="<user>"` | Assign users to the merge request. Accepts username or user ID. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904), support for usernames added in [15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/344276) | -| `merge_request.unassign="<user>"` | Remove assigned users from the merge request. Accepts username or user ID.For example, for two users: `git push -o merge_request.unassign="user1" -o merge_request.unassign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904), support for usernames added in [15.5](https://gitlab.com/gitlab-org/gitlab/-/issues/344276) | - -If you use a push option that requires text with spaces in it, you need to enclose it -in quotes (`"`). You can omit the quotes if there are no spaces. Some examples: +If your push option requires text containing spaces, enclose the text in +double quotes (`"`). You can omit the quotes if there are no spaces. Some examples: ```shell git push -o merge_request.label="Label with spaces" git push -o merge_request.label=Label-with-no-spaces ``` -You can combine push options to accomplish multiple tasks at once, by using -multiple `-o` (or `--push-option`) flags. For example, if you want to create a new -merge request, and target a branch named `my-target-branch`: +To combine push options to accomplish multiple tasks at once, use +multiple `-o` (or `--push-option`) flags. This command creates a +new merge request, targets a branch (`my-target-branch`), and sets auto-merge: ```shell -git push -o merge_request.create -o merge_request.target=my-target-branch +git push -o merge_request.create -o merge_request.target=my-target-branch -o merge_request.merge_when_pipeline_succeeds ``` -Additionally if you want the merge request to merge as soon as the pipeline succeeds you can do: +## Create Git aliases for common commands -```shell -git push -o merge_request.create -o merge_request.target=my-target-branch -o merge_request.merge_when_pipeline_succeeds -``` +Adding push options to Git commands can create very long commands. If +you use the same push options frequently, create Git aliases for them. +Git aliases are command-line shortcuts for longer Git commands. -## Useful Git aliases +To create and use a Git alias for the +[merge when pipeline succeeds Git push option](#push-options-for-merge-requests): -As shown above, Git push options can cause Git commands to grow very long. If -you use the same push options frequently, it's useful to create -[Git aliases](https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases). Git aliases -are command line shortcuts for Git which can significantly simplify the use of -long Git commands. +1. In your terminal window, run this command: -### Merge when pipeline succeeds alias + ```shell + git config --global alias.mwps "push -o merge_request.create -o merge_request.target=main -o merge_request.merge_when_pipeline_succeeds" + ``` -To set up a Git alias for the -[merge when pipeline succeeds Git push option](#push-options-for-merge-requests): +1. To use the alias to push a local branch that targets the default branch (`main`) + and auto-merges, run this command: -```shell -git config --global alias.mwps "push -o merge_request.create -o merge_request.target=master -o merge_request.merge_when_pipeline_succeeds" -``` + ```shell + git mwps origin <local-branch-name> + ``` -Then to quickly push a local branch that targets the default branch and merges when the -pipeline succeeds: +## Related topics -```shell -git mwps origin <local-branch-name> -``` +- [Git aliases](https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases) in the Git documentation diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index aee052631fc..097c726d163 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -146,10 +146,13 @@ To auto-format this table, use the VS Code Markdown Table formatter: `https://do |:--------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-------| | `/assign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Assign one or more users. | | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Assign yourself. | +| `/award :emoji:` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Toggle an emoji reaction. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412275) in GitLab 16.5 | | `/cc @user` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mention a user. In GitLab 15.0 and later, this command performs no action. You can instead type `CC @user` or only `@user`. [In GitLab 14.9 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/31200), mentioning a user at the start of a line creates a specific type of to-do item notification. | +| `/checkin_reminder` | **{dotted-circle}** No| **{check-circle}** Yes | **{dotted-circle}** No | Set checkin reminder cadence. Options are `weekly`, `twice-monthly`, `monthly`, `never`. This action is behind a feature flag. | | `/clear_health_status` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Clear [health status](issues/managing_issues.md#health-status). | | `/clear_weight` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Clear weight. | | `/close` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Close. | +| `/confidential` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mark work item as confidential. [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412276) in GitLab 16.4. | | `/done` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mark to-do item as done. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412277) in GitLab 16.2. | | `/due <date>` | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | | `/health_status <value>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Set [health status](issues/managing_issues.md#health-status). Valid options for `<value>` are `on_track`, `needs_attention`, or `at_risk`. | @@ -160,6 +163,7 @@ To auto-format this table, use the VS Code Markdown Table formatter: `https://do | `/remove_due_date` | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes | Remove due date. | | `/reopen` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Reopen. | | `/shrug <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `¯\＿(ツ)＿/¯`. | +| `/subscribe` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Subscribe to notifications. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/420796) in GitLab 16.4 | | `/tableflip <comment>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Append the comment with `(╯°□°)╯︵ ┻━┻`. | | `/title <new title>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Change title. | | `/todo` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add a to-do item. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412277) in GitLab 16.2. | @@ -168,6 +172,7 @@ To auto-format this table, use the VS Code Markdown Table formatter: `https://do | `/unassign` | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | Remove all assignees. | | `/unlabel ~label1 ~label2` or `/remove_label ~label1 ~label2` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Remove specified labels. | | `/unlabel` or `/remove_label` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Remove all labels. | +| `/unsubscribe` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe to notifications. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/420796) in GitLab 16.4 | | `/weight <value>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set weight. Valid options for `<value>` include `0`, `1`, and `2`. | ## Commit messages diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 8e65514cd1d..17ea0e3f694 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -74,7 +74,7 @@ Prerequisites: To create a release in the Releases page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Releases** and select **New release**. 1. From the [**Tag name**](release_fields.md#tag-name) dropdown list, either: - Select an existing Git tag. Selecting an existing tag that is already associated with a release @@ -215,7 +215,7 @@ To delete a release, use either the In the UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Deploy > Releases**. 1. In the upper-right corner of the release you want to delete, select **Edit this release** (**{pencil}**). @@ -287,22 +287,40 @@ are defined as [crontab](https://crontab.guru/) entries. If the job that's executing is in a freeze period, GitLab CI/CD creates an environment variable named `$CI_DEPLOY_FREEZE`. -To prevent the deployment job from executing, create a `rules` entry in your -`.gitlab-ci.yml`, for example: +To prevent the deployment job from executing in multiple projects in a group, +define the `.freezedeployment` job in a file shared across the group. +Use the [`includes`](../../../ci/yaml/includes.md) keyword to incorporate the +template in your project's `.gitlab-ci.yml` file: ```yaml -deploy_to_production: +.freezedeployment: stage: deploy - script: deploy_to_prod.sh + before_script: + - '[[ ! -z "$CI_DEPLOY_FREEZE" ]] && echo "INFRASTRUCTURE OUTAGE WINDOW" && exit 1; ' rules: - - if: $CI_DEPLOY_FREEZE == null + - if: '$CI_DEPLOY_FREEZE' + when: manual + allow_failure: true + - when: on_success +``` + +To prevent the deployment job from executing, use the [`extends`](../../../ci/yaml/index.md#extends) keyword in the `deploy_to_production` job of your `.gitlab-ci.yml` file to inherit the configuration from the `.freezedeployment` template job: + +```yaml +deploy_to_production: + extends: .freezedeployment + script: deploy_to_prod.sh environment: production ``` +This configuration blocks deployment jobs conditionally and maintains pipeline continuity. When a freeze period is defined, the job fails and the pipeline can proceed without deployment. Manual deployment is possible after the freeze period. + +This approach offers deployment control during critical maintenance, and ensures the uninterrupted flow of the CI/CD pipeline. + To set a deploy freeze window in the UI, complete these steps: 1. Sign in to GitLab as a user with the Maintainer role. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Scroll to **Deploy freezes**. 1. Select **Expand** to see the deploy freeze table. @@ -342,7 +360,7 @@ Releases can be made accessible to non-project members while keeping repository- projects that use releases as a way to give access to new versions of software but do not want the source code to be public. -To make releases available publicly, set the following [project settings](../settings/index.md#project-feature-settings): +To make releases available publicly, set the following [project settings](../settings/index.md#configure-project-features-and-permissions): - Repository is enabled and set to **Only Project Members** - Releases is enabled and set to **Everyone With Access** diff --git a/doc/user/project/releases/release_cli.md b/doc/user/project/releases/release_cli.md index e1de9cbdc1c..9daa6705ad8 100644 --- a/doc/user/project/releases/release_cli.md +++ b/doc/user/project/releases/release_cli.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Release CLI tool -The [GitLab Release CLI (`release-cli`)](https://gitlab.com/gitlab-org/release-cli) tool +The [GitLab Release CLI (`release-cli`)](https://gitlab.com/gitlab-org/release-cli) is a command-line tool for managing releases from the command line or from a CI/CD pipeline. You can use the release CLI to create, update, modify, and delete releases. diff --git a/doc/user/project/remote_development/connect_machine.md b/doc/user/project/remote_development/connect_machine.md index ec3d63462f3..6bbc0b8123a 100644 --- a/doc/user/project/remote_development/connect_machine.md +++ b/doc/user/project/remote_development/connect_machine.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Tutorial: Connect a remote machine to the Web IDE (Beta) **(FREE ALL)** +# Tutorial: Connect a remote machine to the Web IDE **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/371084) in GitLab 15.7. diff --git a/doc/user/project/remote_development/index.md b/doc/user/project/remote_development/index.md index 16110af984a..8ca505be500 100644 --- a/doc/user/project/remote_development/index.md +++ b/doc/user/project/remote_development/index.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Remote development (Beta) **(FREE ALL)** +# Remote development **(FREE ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/371084) in GitLab 15.7. diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index e123debb724..fdc822ca49f 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -42,7 +42,7 @@ Prerequisites: To update the default branch for an individual [project](../../index.md): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Branch defaults**. For **Default branch**, select a new default branch. 1. Optional. Select the **Auto-close referenced issues on default branch** checkbox to close @@ -68,7 +68,7 @@ GitLab [administrators](../../../permissions.md) of self-managed instances can customize the initial branch for projects hosted on that instance. Individual groups and subgroups can override this instance-wide setting for their projects. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand **Default branch**. @@ -85,7 +85,7 @@ overrides it. Users with the Owner role of groups and subgroups can configure the default branch name for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Repository**. 1. Expand **Default branch**. 1. For **Initial default branch name**, select a new default branch. @@ -128,7 +128,7 @@ you must either: Administrators of self-managed instances can customize the initial default branch protection for projects hosted on that instance. Individual groups and subgroups can override this instance-wide setting for their projects. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand **Default branch**. @@ -146,7 +146,7 @@ can be overridden on a per-group basis by the group's owner. In [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/), GitLab administrators can disable this privilege for group owners, enforcing the instance-level protection rule: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand the **Default branch** section. @@ -167,7 +167,7 @@ can be overridden on a per-group basis by the group's owner. In [enforce protection of initial default branches](#prevent-overrides-of-default-branch-protection) which locks this setting for group owners. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Repository**. 1. Expand **Default branch**. 1. Select [**Initial default branch protection**](#protect-initial-default-branches). diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index f7445ffe950..f33d443cd7f 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -11,7 +11,7 @@ Branches are versions of a project's working tree. When you create a new cannot be deleted) for your repository. Default branch settings can be configured at the project, subgroup, group, or instance level. -As your project grows, your team [creates](../web_editor.md#create-a-branch) more +As your project grows, your team creates more branches, preferably by following [branch naming patterns](#prefix-branch-names-with-issue-numbers). Each branch represents a set of changes, which allows development work to be done in parallel. Development work in one branch does not affect another branch. @@ -27,9 +27,13 @@ Branches are the foundation of development in a project: ## Create branch +Prerequisites: + +- You must have at least the Developer role for the project. + To create a new branch from the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. On the top right, select **New branch**. 1. Enter a **Branch name**. @@ -44,15 +48,15 @@ you can add one. Prerequisites: -- You must have at least the Developer role in the project. -- Unless you have the Maintainer or Owner roles, the +- You must have at least the Developer role for the project. +- If you don't have the Maintainer or Owner role, the [default branch protection](../../../group/manage.md#change-the-default-branch-protection-of-a-group) must be set to `Partially protected` or `Not protected` for you to push a commit to the default branch. To add a [default branch](default.md) to an empty project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Scroll to **The repository for this project is empty** and select the type of file you want to add. 1. In the Web IDE, make any desired changes to this file, then select **Create commit**. @@ -64,7 +68,7 @@ GitLab creates a default branch and adds your file to it. Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. When viewing an issue, you can create an associated branch directly from that page. Branches created this way use the @@ -73,11 +77,11 @@ including variables. Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. To create a branch from an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues** and find your issue. 1. Below the issue description, find the **Create merge request** dropdown list, and select **{chevron-down}** to display the dropdown list. @@ -89,7 +93,7 @@ To create a branch from an issue: ## Manage and protect branches -GitLab provides you multiple methods to protect individual branches. These methods +GitLab provides multiple methods to protect individual branches. These methods ensure your branches receive oversight and quality checks from their creation to their deletion: - The [default branch](default.md) in your project receives extra protection. @@ -104,19 +108,19 @@ ensure your branches receive oversight and quality checks from their creation to You can manage your branches: - With the GitLab user interface. -- With the [command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). +- With Git on the [command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). - With the [Branches API](../../../../api/branches.md). ### View all branches To view and manage your branches in the GitLab user interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. On this page, you can: -- See all branches, active branches, or stale branches. +- See all branches, or filter to see only active or stale branches. - Create new branches. - [Compare branches](#compare-branches). - Delete merged branches. @@ -145,19 +149,19 @@ and their protection methods: Prerequisites: -- You must have at least the Maintainer role in the project. +- You must have at least the Maintainer role for the project. To view the **Branch rules overview** list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. -1. Expand **Branch Rules** to view all branches with protections. +1. Expand **Branch rules** to view all branches with protections. - To add protections to a new branch: 1. Select **Add branch rule**. 1. Select **Create protected branch**. - To view more information about protections on an existing branch: 1. Identify the branch you want more information about. - 1. Select **Details** to see information about its: + 1. Select **View details** to see information about its: - [Branch protections](../../protected_branches.md). - [Approval rules](../../merge_requests/approvals/rules.md). - [Status checks](../../merge_requests/status_checks.md). @@ -173,12 +177,17 @@ GitLab enforces these additional rules on all branches: - No spaces are allowed in branch names. - Branch names with 40 hexadecimal characters are prohibited, because they are similar to Git commit hashes. -Common software packages, like Docker, may enforce +Common software packages, like Docker, can enforce [additional branch naming restrictions](../../../../administration/packages/container_registry.md#docker-connection-error). -For best compatibility with other software packages, use only numbers, hyphens (`-`), -underscores (`_`), and lower-case letters from the ASCII standard table. You -can use forward slashes (`/`) and emoji in branch names, but compatibility with other +For the best compatibility with other software packages, use only: + +- Numbers +- Hyphens (`-`) +- Underscores (`_`) +- Lowercase letters from the ASCII standard table + +You can use forward slashes (`/`) and emoji in branch names, but compatibility with other software packages cannot be guaranteed. Branch names with specific formatting offer extra benefits: @@ -200,7 +209,7 @@ Prerequisites: To change the default pattern for branches created from issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Branch defaults**. 1. Scroll to **Branch name template** and enter a value. The field supports these variables: @@ -210,8 +219,13 @@ To change the default pattern for branches created from issues: ### Prefix branch names with issue numbers -To streamline the creation of merge requests, start your branch name with an -issue number. GitLab uses the issue number to import data into the merge request: +To streamline the creation of merge requests, start your Git branch name with the +issue number, followed by a hyphen. +For example, to link a branch to issue `#123`, start the branch name with `123-`. + +The issue and the branch must be in the same project. + +GitLab uses the issue number to import data into the merge request: - The issue is marked as related to the merge request. The issue and merge request display links to each other. @@ -224,21 +238,9 @@ issue number. GitLab uses the issue number to import data into the merge request ## Compare branches -> - Repository filter search box [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52967) in GitLab 13.10. -> - Revision swapping [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60491) in GitLab 13.12. - -The default compare mode uses the `git diff from...to` method, instead of the -`git diff from to` method, to compare branches. The `git diff from...to` method -provides a more human-readable diff, because it does not include unrelated changes -made to the target branch after the source branch was created. - -NOTE: -The `git diff from...to` method shows all changes from a cherry-picked commit as -new changes. It uses the merge base, not the actual commit content, to compare branches. - To compare branches in a repository: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Compare revisions**. 1. Select the **Source** branch to search for your desired branch. Exact matches are shown first. You can refine your search with operators: @@ -247,8 +249,23 @@ To compare branches in a repository: - `*` matches using a wildcard: `branch*cache*` matches `fix/branch-search-cache-expiration`. - You can combine operators: `^chore/*migration$` matches `chore/user-data-migration`. 1. Select the **Target** repository and branch. Exact matches are shown first. -1. Select **Compare** to show the list of commits, and changed files. To reverse - the **Source** and **Target**, select **Swap revisions**. +1. Below **Show changes**, select the method to compare branches: + <!-- vale gitlab.SubstitutionWarning = NO --> + <!-- Disable Vale so it doesn't flag "since" --> + - **Only incoming changes from source** (default) shows differences from the source branch since + the latest common commit on both branches. + It doesn't include unrelated changes made to the target branch after the source branch was created. + This method uses the `git diff <from>...<to>` + [Git command](https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-emgitdiffemltoptionsgtltcommitgtltcommitgt--ltpathgt82308203-1). + To compare branches, this method uses the merge base instead of the actual commit, so + changes from cherry-picked commits are shown as new changes. + - **Include changes to target since source was created** shows all the differences between the two + branches. + This method uses the `git diff <from> <to>` + [Git command](https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-emgitdiffemltoptionsgt--merge-baseltcommitgtltcommitgt--ltpathgt82308203). + <!-- vale gitlab.SubstitutionWarning = YES --> +1. Select **Compare** to show the list of commits, and changed files. +1. Optional. To reverse the **Source** and **Target**, select **Swap revisions** (**{substitute}**). ## Delete merged branches @@ -259,15 +276,15 @@ Merged branches can be deleted in bulk if they meet all of these criteria: Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. On the upper right corner of the page, select **More** **{ellipsis_v}**. 1. Select **Delete merged branches**. -1. In the modal window, enter the word `delete` to confirm, then select **Delete merged branches**. +1. In the dialog, enter the word `delete` to confirm, then select **Delete merged branches**. ## Related topics @@ -323,7 +340,7 @@ Error: Could not set the default branch. Do you have a branch named 'HEAD' in yo To fix this problem: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. Search for a branch named `HEAD`. 1. Make sure the branch has no uncommitted changes. diff --git a/doc/user/project/repository/code_suggestions.md b/doc/user/project/repository/code_suggestions.md deleted file mode 100644 index 6f18aabaa46..00000000000 --- a/doc/user/project/repository/code_suggestions.md +++ /dev/null @@ -1,375 +0,0 @@ ---- -stage: AI-powered -group: AI Model Validation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: index, reference ---- - -# Code Suggestions (Beta) **(FREE ALL)** - -> - [Introduced](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#code-suggestions-available-in-closed-beta) in GitLab 15.9 as [Beta](../../../policy/experiment-beta-support.md#beta) for early access Ultimate customers on GitLab.com. -> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/408104) as opt-in with GitLab 15.11 as [Beta](../../../policy/experiment-beta-support.md#beta). -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1. -> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta) on self-managed GitLab. -> - Code suggestions in the GitLab WebIDE enabled for all GitLab-hosted customers. -> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. - -WARNING: -This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). -Beta users should read about the [known limitations](#known-limitations). We look forward to hearing your [feedback](#feedback). - -Write code more efficiently by using generative AI to suggest code while you're developing. - -Code Suggestions are available: - -- To users of GitLab SaaS (by default) and self-managed GitLab Enterprise Edition (when requested). Code Suggestions are not available for GitLab Community Edition. -- In VS Code, Microsoft Visual Studio, JetBrains IDEs, and Neovim. You must have the corresponding GitLab extension installed. -- In the GitLab WebIDE. - -<div class="video-fallback"> - <a href="https://www.youtube.com/watch?v=WnxBYxN2-p4">View an end-to-end demo of Code Suggestions in VS Code</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/WnxBYxN2-p4" frameborder="0" allowfullscreen> </iframe> -</figure> - -Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). -Learn about [data usage when using Code Suggestions](#code-suggestions-data-usage). - -## Supported languages - -The best results from Code Suggestions are expected [for languages the Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_coding_languages) directly support: - -- C++ -- C# -- Go -- Google SQL -- Java -- JavaScript -- Kotlin -- PHP -- Python -- Ruby -- Rust -- Scala -- Swift -- TypeScript - -### Supported code infrastructure interfaces - -Code Suggestions includes [Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_code_infrastructure_interfaces) support for the following infrastructure as code interfaces: - -- Google Cloud CLI -- Kubernetes Resource Model (KRM) -- Terraform - -Suggestion quality for other languages and using natural language code comments to request completions may not yet result in high-quality suggestions. - -### Supported languages in IDEs - -Editor support for languages is documented in the following table. - -| Language | VS Code | JetBrains IDEs | Visual Studio | Neovim | -|---------------------------------|--------------------------------------------------------------|------------------------------|---------------|--------| -| C++ | ✓ | | ✓ | | -| C# | ✓ | ✓ | ✓ | | -| Go | ✓ | ✓ (IDEA Ultimate / GoLand) | | | -| Google SQL | | | | | -| Java | ✓ | ✓ | | | -| JavaScript | ✓ | ✓ | | | -| Kotlin | ✓ | ✓ | | | -| PHP | ✓ | ✓ (IDEA Ultimate) | | | -| Python | ✓ | ✓ | | ✓ | -| Ruby | ✓ | ✓ (IDEA Ultimate / RubyMine) | | ✓ | -| Rust | ✓ | ✓ | | | -| Scala | ✓ | ✓ | | | -| Swift | ✓ | ✓ | | | -| TypeScript | ✓ | ✓ | | | -| Google Cloud CLI | | | | | -| Kubernetes Resource Model (KRM) | | | | | -| Terraform | ✓ (Requires 3rd party extension providing Terraform support) | | | | - -## Supported editor extensions - -Code Suggestions supports a variety of popular editors including: - -- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). -- [GitLab WebIDE (VS Code in the Cloud)](../../project/web_ide/index.md), with no additional configuration. -- Microsoft Visual Studio, using the [Visual Studio GitLab extension](https://marketplace.visualstudio.com/items?itemName=GitLab.GitLabExtensionForVisualStudio). -- JetBrains IDEs, using the [GitLab plugin](https://plugins.jetbrains.com/plugin/22325-gitlab). -- Neovim, using the [`gitlab.vim` plugin](https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim). - -A [GitLab Language Server for Code Suggestions](https://gitlab.com/gitlab-org/editor-extensions/gitlab-language-server-for-code-suggestions) -is also in process. -This improvement should result in: - -- Faster iteration and standardization of the IDE extensions. -- The ability to use Code Suggestions even when an official editor extension isn't available. - -## Enable Code Suggestions on GitLab SaaS **(FREE SAAS)** - -Code Suggestions can be enabled [for all members of a group](../../group/manage.md#enable-code-suggestions). - -Each individual user must also choose to enable Code Suggestions. - -### Enable Code Suggestions for an individual user - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta). - -Each user can enable Code Suggestions for themselves: - -1. On the left sidebar, select your avatar. -1. Select **Preferences**. -1. In the **Code Suggestions** section, select the **Enable Code Suggestions** checkbox. -1. Select **Save changes**. - -If Code Suggestions is enabled for the group, the group setting overrides the user setting. - -NOTE: -This setting controls Code Suggestions for all IDEs. Support for [more granular control per IDE](https://gitlab.com/groups/gitlab-org/-/epics/10624) is proposed. - -## Enable Code Suggestions on self-managed GitLab **(FREE SELF)** - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta). - -To enable Code Suggestions on a self-managed GitLab EE instance, you must: - -- Be an administrator. -- Have a [GitLab SaaS account](https://gitlab.com/users/sign_up). -You do not need to have a GitLab SaaS subscription. - -Then, you will: - -1. Enable Code Suggestions for your SaaS account. -1. Enable Code Suggestions for the instance. -1. [Request early access](#request-access-to-code-suggestions) to the Code Suggestions Beta. - -### Enable Code Suggestions for your SaaS account - -To enable Code Suggestions for your GitLab SaaS account: - -1. Create a [personal access token](../../profile/personal_access_tokens.md#create-a-personal-access-token) - with the `api` scope. -1. On the left sidebar, select your avatar. -1. Select **Preferences**. -1. In the **Code Suggestions** section, select **Enable Code Suggestions**. -1. Select **Save changes**. - -### Enable Code Suggestions for the instance - -You must enable Code Suggestions for the instance. When you do this, you: - -- Agree to the [GitLab testing agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). -- Acknowledge that GitLab: - - Sends data from the instance, including personal data, to GitLab.com infrastructure. - -To enable Code Suggestions for your self-managed GitLab instance: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Settings > General**. -1. Expand **Code Suggestions** and: - - Select **Turn on Code Suggestions for this instance**. - - In **Personal access token**, enter your GitLab SaaS personal access token. -1. Select **Save changes**. - -This setting is visible only in self-managed GitLab instances. - -WARNING: -If you clear the **Turn on code suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. - -### Request access to Code Suggestions - -GitLab provisions access on a customer-by-customer basis for Code Suggestions -on self-managed instances. To request access: - -1. Sign into your GitLab SaaS account. -1. Comment on [issue 415393](https://gitlab.com/gitlab-org/gitlab/-/issues/415393) - and tag your customer success manager. - -After GitLab has provisioned access to Code Suggestions for your instance, -the users in your instance can now enable Code Suggestions. - -## Use Code Suggestions - -Prerequisites: - -- For self-managed GitLab, Code Suggestions must be enabled [for the instance](#enable-code-suggestions-on-self-managed-gitlab). -- For GitLab SaaS, Code Suggestions must be enabled [for the top-level group](../../group/manage.md#enable-code-suggestions) and [for your user account](#enable-code-suggestions-for-an-individual-user). -- Install and configure a [supported IDE editor extension](#supported-editor-extensions). -To use Code Suggestions: - -1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: - - - Provides entire code snippets, like generating functions. - - Completes the current line. - -1. To accept a suggestion, press <kbd>Tab</kbd>. - -Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. - -GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. - -This feature is currently in [Beta](../../../policy/experiment-beta-support.md#beta). -Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to -mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. - -## Code Suggestions data usage - -Code Suggestions is a generative artificial intelligence (AI) model. - -Your personal access token enables a secure API connection to GitLab.com. -This API connection securely transmits a context window from your IDE/editor to the Code Suggestions GitLab hosted service which calls Google Vertex AI Codey APIs, -and the generated suggestion is transmitted back to your IDE/editor. - -GitLab currently leverages [Google Cloud's Vertex AI Codey API models](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview). - -### Data privacy - -No new additional data is collected to enable this feature. Private non-public GitLab customer data is -not used as training data. - -Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) - -### Inference window context - -Code Suggestions currently inferences against the currently opened file and has a context window of 2,048 tokens and 8,192 character limits. This limit includes content before and after the cursor, the file name, and the extension type. -Learn more about Google Vertex AI [code-gecko](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). - -The maximum number of tokens that is generated in the response is default 64. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words. -Learn more about Google Vertex AI [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion). - -### Self-managed instance data privacy - -A self-managed GitLab instance does not generate the code suggestion. After successful -authentication to the self-managed instance, a token is generated. - -The IDE/editor then uses this token to securely transmit data directly to -GitLab.com's Code Suggestions service for processing. - -The Code Suggestion service then securely returns an AI-generated code suggestion. - -Neither GitLab nor Google Vertex AI Codey APIs have any visibility into a self-managed customer's code other than -what is sent to generate the code suggestion. - -### Training data - -Code suggestions are routed through Google Vertex AI Codey APIs. Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) and [Responsible AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/responsible-ai). - -Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data. - -Google has [shared the following](https://ai.google/discover/foundation-models/) about the data Codey models are trained on: - -> Codey is our family of foundational coding models built on PaLM 2. Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources - -## Progressive enhancement - -This feature is designed as a progressive enhancement to developer's IDEs. -Code Suggestions offer a completion if the machine learning engine can generate a recommendation. -In the event of a connection issue or model inference failure, the feature gracefully degrades. -Code Suggestions do not prevent you from writing code in your IDE. - -### Internet connectivity - -Code Suggestions does not work with offline environments. - -To use Code Suggestions: - -- On GitLab.com, you must have an internet connection and be able to access GitLab. -- In GitLab 16.1 and later, on self-managed GitLab, you must have an internet connection. - -### Model accuracy and quality - -Code Suggestions can generate low-quality, incomplete, and possibly insecure code. -We strongly encourage all beta users to leverage GitLab native -[Code Quality Scanning](../../../ci/testing/code_quality.md) and -[Security Scanning](../../application_security/index.md) capabilities. - -GitLab currently does not retrain Google Vertex AI Codey APIs. GitLab makes no claims -to the accuracy or quality of code suggestions generated by Google Vertex AI Codey API. -Read more about [Google Vertex AI foundation model capabilities](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). - -## Known limitations - -While in Beta, we are working on improving the accuracy of overall generated content. -However, Code Suggestions may generate suggestions that are: - -- Low-quality -- Incomplete -- Produce failed pipelines -- Insecure code -- Offensive or insensitive - -We are also aware of specific situations that can produce unexpected or incoherent results including: - -- Suggestions written in the middle of existing functions, or "fill in the middle." -- Suggestions based on natural language code comments. -- Suggestions that mixed programming languages in unexpected ways. - -## Feedback - -Report issues in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152). - -## Troubleshooting - -### Code Suggestions aren't displayed - -If Code Suggestions are not displayed, try the following troubleshooting steps. - -In GitLab, ensure Code Suggestions is enabled: - -- [For your user account](#enable-code-suggestions-for-an-individual-user). -- [For *all* top-level groups your account belongs to](../../group/manage.md#enable-code-suggestions). If you don't have a role that lets you view the top-level group's settings, contact a group owner. - -To confirm that your account is enabled, go to [https://gitlab.com/api/v4/ml/ai-assist](https://gitlab.com/api/v4/ml/ai-assist). A response of `user_is_allowed` should return `true`. - -#### Code Suggestions not displayed in VS Code or GitLab WebIDE - -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. - -If you are a self-managed user, ensure that Code Suggestions for the [GitLab WebIDE](../../project/web_ide/index.md) are enabled. The same settings apply to VS Code as local IDE. - -1. On the left sidebar, select **Extensions > GitLab Workflow**. -1. Select **Settings** (**{settings}**), and then select **Extension Settings**. -1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion (Beta)** - checkbox. - -If the settings are enabled, but Code Suggestions are still not displayed, try the following steps: - -1. Enable the `Debug` checkbox in the GitLab Workflow **Extension Settings**. -1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Workflow** as the log filter. The command palette command is `GitLab: Show Extension Logs`. -1. Disable and re-enable the **Enable code completion (Beta)** checkbox. -1. Verify that the debug log contains similar output: - -```shell -2023-07-14T17:29:00:763 [debug]: Disabling code completion -2023-07-14T17:29:01:802 [debug]: Enabling code completion -2023-07-14T17:29:01:802 [debug]: AI Assist: Using server: https://codesuggestions.gitlab.com/v2/completions -``` - -#### Code Suggestions not displayed in Microsoft Visual Studio - -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. - -1. Ensure you have properly [set up the extension](https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension#setup). -1. From the **Tools > Options** menu, find the **GitLab** option. Ensure **Log Level** is set to **Debug**. -1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Extension** as the log filter. -1. Verify that the debug log contains similar output: - -```shell -14:48:21:344 GitlabProposalSource.GetCodeSuggestionAsync -14:48:21:344 LsClient.SendTextDocumentCompletionAsync("GitLab.Extension.Test\TestData.cs", 34, 0) -14:48:21:346 LS(55096): time="2023-07-17T14:48:21-05:00" level=info msg="update context" -``` - -### Authentication troubleshooting - -If the above steps do not solve your issue, the problem may be related to the recent changes in authentication, -specifically the token system. To resolve the issue: - -1. Remove the existing personal access token from your GitLab account settings. -1. Reauthorize your GitLab account in VS Code using OAuth. -1. Test the code suggestions feature with different file extensions to verify if the issue is resolved. diff --git a/doc/user/project/repository/code_suggestions/index.md b/doc/user/project/repository/code_suggestions/index.md new file mode 100644 index 00000000000..4f0c0b2c9a6 --- /dev/null +++ b/doc/user/project/repository/code_suggestions/index.md @@ -0,0 +1,192 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions **(FREE ALL BETA)** + +> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. +> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. + +WARNING: +This feature is in [Beta](../../../../policy/experiment-beta-support.md#beta). +Beta users should read about the [known limitations](#known-limitations). We look forward to hearing your [feedback](#feedback). + +Write code more efficiently by using generative AI to suggest code while you're developing. + +GitLab Duo Code Suggestions are available: + +- On [self-managed](self_managed.md) and [SaaS](saas.md). +- In VS Code, Microsoft Visual Studio, JetBrains IDEs, and Neovim. You must have the corresponding GitLab extension installed. +- In the GitLab WebIDE. + +<div class="video-fallback"> + <a href="https://www.youtube.com/watch?v=WnxBYxN2-p4">View an end-to-end demo of Code Suggestions in VS Code</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/WnxBYxN2-p4" frameborder="0" allowfullscreen> </iframe> +</figure> + +Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](#code-suggestions-data-usage). + +## Supported languages + +The best results from Code Suggestions are expected [for languages the Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_coding_languages) directly support: + +- C++ +- C# +- Go +- Google SQL +- Java +- JavaScript +- Kotlin +- PHP +- Python +- Ruby +- Rust +- Scala +- Swift +- TypeScript + +### Supported code infrastructure interfaces + +Code Suggestions includes [Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_code_infrastructure_interfaces) support for the following infrastructure as code interfaces: + +- Google Cloud CLI +- Kubernetes Resource Model (KRM) +- Terraform + +Suggestion quality for other languages and using natural language code comments to request completions may not yet result in high-quality suggestions. + +### Supported languages in IDEs + +Editor support for languages is documented in the following table. + +| Language | VS Code | JetBrains IDEs | Visual Studio | Neovim | +|------------------|------------------------|------------------------|------------------------|--------| +| C++ | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | +| C# | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Go | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate / GoLand) | **{check-circle}** Yes | **{check-circle}** Yes | +| Google SQL | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | +| Java | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| JavaScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Kotlin | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| PHP | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate) | **{check-circle}** Yes | **{check-circle}** Yes | +| Python | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Ruby | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate / RubyMine) | **{check-circle}** Yes | **{check-circle}** Yes | +| Rust | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Scala | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Swift | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| TypeScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Google Cloud | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | +| Kubernetes Resource Model (KRM) | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | +| Terraform | **{check-circle}** Yes (Requires third-party extension providing Terraform support) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes (Requires third-party extension providing the `terraform` file type) | + +## Supported editor extensions + +Code Suggestions supports a variety of popular editors including: + +- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). +- [GitLab WebIDE (VS Code in the Cloud)](../../../project/web_ide/index.md), with no additional configuration. +- Microsoft Visual Studio, using the [Visual Studio GitLab extension](https://marketplace.visualstudio.com/items?itemName=GitLab.GitLabExtensionForVisualStudio). +- JetBrains IDEs, using the [GitLab plugin](https://plugins.jetbrains.com/plugin/22325-gitlab). +- Neovim, using the [`gitlab.vim` plugin](https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim). + +A [GitLab Language Server for Code Suggestions](https://gitlab.com/gitlab-org/editor-extensions/gitlab-language-server-for-code-suggestions) +is also in process. +This improvement should result in: + +- Faster iteration and standardization of the IDE extensions. +- The ability to use Code Suggestions even when an official editor extension isn't available. + +## Code Suggestions data usage + +Code Suggestions is a generative artificial intelligence (AI) model. + +Your personal access token enables a secure API connection to GitLab.com. +This API connection securely transmits a context window from your IDE/editor to the Code Suggestions GitLab hosted service which calls Google Vertex AI Codey APIs, +and the generated suggestion is transmitted back to your IDE/editor. + +GitLab currently leverages [Google Cloud's Vertex AI Codey API models](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview). Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance). + +### Telemetry + +For self-managed instances that have enabled Code Suggestions and SaaS accounts, we collect aggregated or de-identified first-party usage data through our [Snowplow collector](https://about.gitlab.com/handbook/business-technology/data-team/platform/snowplow/). This usage data includes the following metrics: + +- Language the code suggestion was in (for example, Python) +- Editor being used (for example, VS Code) +- Number of suggestions shown, accepted, rejected, or that had errors +- Duration of time that a suggestion was shown +- Prompt and suffix lengths +- Model used +- Number of unique users +- Number of unique instances + +### Inference window context + +Code Suggestions currently inferences against the currently opened file and has a context window of 2,048 tokens and 8,192 character limits. This limit includes content before and after the cursor, the file name, and the extension type. +Learn more about Google Vertex AI [code-gecko](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). + +The maximum number of tokens that is generated in the response is default 64. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words. +Learn more about Google Vertex AI [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion). + +### Training data + +Code Suggestions are routed through Google Vertex AI Codey APIs. Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) and [Responsible AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/responsible-ai). + +Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data. + +Google has [shared the following](https://ai.google/discover/foundation-models/) about the data Codey models are trained on: + +> Codey is our family of foundational coding models built on PaLM 2. Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources + +## Progressive enhancement + +This feature is designed as a progressive enhancement to developer's IDEs. +Code Suggestions offer a completion if the machine learning engine can generate a recommendation. +In the event of a connection issue or model inference failure, the feature gracefully degrades. +Code Suggestions do not prevent you from writing code in your IDE. + +### Internet connectivity + +Code Suggestions does not work with offline environments. + +To use Code Suggestions: + +- On GitLab.com, you must have an internet connection and be able to access GitLab. +- In GitLab 16.1 and later, on self-managed GitLab, you must have an internet connection. + +### Model accuracy and quality + +Code Suggestions can generate low-quality, incomplete, and possibly insecure code. +We strongly encourage all beta users to leverage GitLab native +[Code Quality Scanning](../../../../ci/testing/code_quality.md) and +[Security Scanning](../../../application_security/index.md) capabilities. + +GitLab currently does not retrain Google Vertex AI Codey APIs. GitLab makes no claims +to the accuracy or quality of Code Suggestions generated by Google Vertex AI Codey API. +Read more about [Google Vertex AI foundation model capabilities](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). + +## Known limitations + +While in Beta, we are working on improving the accuracy of overall generated content. +However, Code Suggestions may generate suggestions that are: + +- Low-quality +- Incomplete +- Produce failed pipelines +- Insecure code +- Offensive or insensitive + +We are also aware of specific situations that can produce unexpected or incoherent results including: + +- Suggestions written in the middle of existing functions, or "fill in the middle." +- Suggestions based on natural language code comments. +- Suggestions that mixed programming languages in unexpected ways. + +## Feedback + +Report issues in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152). diff --git a/doc/user/project/repository/code_suggestions/saas.md b/doc/user/project/repository/code_suggestions/saas.md new file mode 100644 index 00000000000..174c227b6fe --- /dev/null +++ b/doc/user/project/repository/code_suggestions/saas.md @@ -0,0 +1,54 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions on GitLab SaaS **(FREE SAAS BETA)** + +> - [Introduced](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#code-suggestions-available-in-closed-beta) in GitLab 15.9 as [Beta](../../../../policy/experiment-beta-support.md#beta) for early access Ultimate customers on GitLab.com. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/408104) as opt-in with GitLab 15.11 as [Beta](../../../../policy/experiment-beta-support.md#beta). +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1. + +Write code more efficiently by using generative AI to suggest code while you're developing. + +Usage of GitLab Duo Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](index.md#code-suggestions-data-usage). + +## Enable Code Suggestions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). + +You can enable Code Suggestions for an individual or group: + +- [Enable Code Suggestions for all group members](../../../group/manage.md#enable-code-suggestions). (You must be a group owner). +- [Enable Code Suggestions for your own account](../../../profile/preferences.md#enable-code-suggestions). + +The group setting takes precedence over the user setting. + +## Use Code Suggestions + +Prerequisites: + +- Ensure Code Suggestions is enabled for your user and group. +- You must have installed and configured a [supported IDE editor extension](index.md#supported-editor-extensions). + +To use Code Suggestions: + +1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: + + - Provides entire code snippets, like generating functions. + - Completes the current line. + +1. To accept a suggestion, press <kbd>Tab</kbd>. + +Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. + +GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. + +This feature is currently in [Beta](../../../../policy/experiment-beta-support.md#beta). +Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to +mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. diff --git a/doc/user/project/repository/code_suggestions/self_managed.md b/doc/user/project/repository/code_suggestions/self_managed.md new file mode 100644 index 00000000000..3c149604086 --- /dev/null +++ b/doc/user/project/repository/code_suggestions/self_managed.md @@ -0,0 +1,187 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions on self-managed GitLab **(PREMIUM SELF BETA)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta) on self-managed GitLab. +> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. +> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. +> - Code Suggestions in the GitLab WebIDE enabled for all GitLab-hosted customers. + +Write code more efficiently by using generative AI to suggest code while you're developing. + +GitLab Duo Code Suggestions are available on GitLab Enterprise Edition. +Cloud licensing is required for Premium and Ultimate subscription tiers. + +Code Suggestions are not available for GitLab Community Edition. + +WARNING: +In GitLab 16.3 and later, only Premium and Ultimate customers can participate in the free trial of Code Suggestions on self-managed GitLab. + +Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](index.md#code-suggestions-data-usage). + +## Enable Code Suggestions on self-managed GitLab **(FREE SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). + +When you enable Code Suggestions for your self-managed instance, you: + +- Agree to the [GitLab testing agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +- Acknowledge that GitLab sends data from the instance, including personal data, to GitLab.com infrastructure. + +How you enable Code Suggestions differs depending on your version of GitLab. + +### GitLab 16.3 and later + +Prerequisites: + +- You are a new Code Suggestions customer as of GitLab 16.3. +- All of the users in your instance have the latest version of their IDE extension. +- You are an administrator. + +To enable Code Suggestions for your self-managed GitLab instance: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Code Suggestions** and select **Turn on Code Suggestions for this instance**. + In GitLab 16.3, you do not need to enter anything into the **Personal access token** field. + In GitLab 16.4 and later, there is no **Personal access token** field. +1. Select **Save changes**. + +This setting is visible only in self-managed GitLab instances. + +WARNING: +In GitLab 16.2 and earlier, if you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. + +To make sure Code Suggestions works immediately, you must [manually synchronize your subscription](#manually-synchronize-your-subscription). + +The users in your instance can now use Code Suggestions. + +### GitLab 16.2 and earlier + +Prerequisites: + +- You are an administrator. +- You have a [customer success manager](https://about.gitlab.com/handbook/customer-success/csm/]). +- You have a [GitLab SaaS account](https://gitlab.com/users/sign_up). You do not need to have a GitLab SaaS subscription. + +NOTE: +If you do not have a customer success manager, you cannot participate in the free trial of Code Suggestions on self-managed GitLab. Upgrade to GitLab 16.3 to [perform self-service onboarding](#gitlab-163-and-later). + +Then, you will: + +1. Enable Code Suggestions for your SaaS account. +1. Enable Code Suggestions for the instance. +1. [Request early access](#request-access-to-code-suggestions) to the Code Suggestions Beta. + +#### Enable Code Suggestions for your SaaS account + +To enable Code Suggestions for your GitLab SaaS account: + +1. Create a [personal access token](../../../profile/personal_access_tokens.md#create-a-personal-access-token) + with the `api` scope. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. In the **Code Suggestions** section, select **Enable Code Suggestions**. +1. Select **Save changes**. + +#### Enable Code Suggestions for the instance + +To enable Code Suggestions for your self-managed GitLab instance: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Code Suggestions** and: + - Select **Turn on Code Suggestions for this instance**. + - In **Personal access token**, enter your GitLab SaaS personal access token. +1. Select **Save changes**. + +This setting is visible only in self-managed GitLab instances. + +WARNING: +If you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. + +#### Request access to Code Suggestions + +GitLab provisions access on a customer-by-customer basis for Code Suggestions +on self-managed instances. To request access, contact your customer success manager. + +Your customer success manager then provisions access by commenting on [issue 415393](https://gitlab.com/gitlab-org/gitlab/-/issues/415393) (internal access only). + +After GitLab has provisioned access to Code Suggestions for your instance, +the users in your instance can now enable Code Suggestions. + +### Configure network and proxy settings + +Configure any firewalls to allow outbound connections to `https://codesuggestions.gitlab.com/`. + +If your GitLab instance uses an HTTP proxy server to access the internet, ensure +the server is configured to allow outbound connections, including the +[`gitlab_workhorse` environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html). + +### Update GitLab + +In GitLab 16.3 and later, GitLab is enforcing the cloud licensing requirement for Code Suggestions: + +- The Premium and Ultimate subscription tiers support cloud Licensing. +- GitLab Free does not have cloud licensing support. + +If you have a GitLab Free subscription and update to GitLab 16.3 or later, +to continue having early access to Code Suggestions, you must: + +1. Have a [subscription that supports cloud licensing](https://about.gitlab.com/pricing/). +1. Make sure you have the latest version of your [IDE extension](index.md#supported-editor-extensions). +1. [Manually synchronize your subscription](#manually-synchronize-your-subscription). + +#### Manually synchronize your subscription + +You must [manually synchronize your subscription](../../../../subscriptions/self_managed/index.md#manually-synchronize-your-subscription-details) if either: + +- You have already updated to GitLab 16.3 and have just bought a Premium or Ultimate tier subscription. +- You already have a Premium or Ultimate tier subscription and have just updated to GitLab 16.3. + +Without the manual synchronization, it might take up to 24 hours to active Code Suggestions on your instance. + +## Use Code Suggestions + +Prerequisites: + +- Code Suggestions must be enabled [for the instance](#enable-code-suggestions-on-self-managed-gitlab). +- You must have installed and configured a [supported IDE editor extension](index.md#supported-editor-extensions). + +To use Code Suggestions: + +1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: + + - Provides entire code snippets, like generating functions. + - Completes the current line. + +1. To accept a suggestion, press <kbd>Tab</kbd>. + +Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. + +GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. + +This feature is currently in [Beta](../../../../policy/experiment-beta-support.md#beta). +Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to +mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. + +### Data privacy + +A self-managed GitLab instance does not generate the code suggestion. After successful +authentication to the self-managed instance, a token is generated. + +The IDE/editor then uses this token to securely transmit data directly to +GitLab.com's Code Suggestions service for processing. + +The Code Suggestions service then securely returns an AI-generated code suggestion. + +Neither GitLab nor Google Vertex AI Codey APIs have any visibility into a self-managed customer's code other than +what is sent to generate the code suggestion. diff --git a/doc/user/project/repository/code_suggestions/troubleshooting.md b/doc/user/project/repository/code_suggestions/troubleshooting.md new file mode 100644 index 00000000000..c0cdb3cc32d --- /dev/null +++ b/doc/user/project/repository/code_suggestions/troubleshooting.md @@ -0,0 +1,69 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Troubleshooting Code Suggestions **(FREE ALL BETA)** + +When working with GitLab Duo Code Suggestions, you might encounter the following issues. + +## Code Suggestions aren't displayed + +If Code Suggestions are not displayed, try the following troubleshooting steps. + +In GitLab, ensure Code Suggestions is enabled: + +- [For your user account](../../../profile/preferences.md#enable-code-suggestions). +- [For *all* top-level groups your account belongs to](../../../group/manage.md#enable-code-suggestions). If you don't have a role that lets you view the top-level group's settings, contact a group owner. + +To confirm that your account is enabled, go to [https://gitlab.com/api/v4/ml/ai-assist](https://gitlab.com/api/v4/ml/ai-assist). A response of `user_is_allowed` should return `true`. + +### Code Suggestions not displayed in VS Code or GitLab WebIDE + +Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. + +If you are a self-managed user, ensure that Code Suggestions for the [GitLab WebIDE](../../../project/web_ide/index.md) are enabled. The same settings apply to VS Code as local IDE. + +1. On the left sidebar, select **Extensions > GitLab Workflow**. +1. Select **Settings** (**{settings}**), and then select **Extension Settings**. +1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion (Beta)** + checkbox. + +If the settings are enabled, but Code Suggestions are still not displayed, try the following steps: + +1. Enable the `Debug` checkbox in the GitLab Workflow **Extension Settings**. +1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Workflow** as the log filter. The command palette command is `GitLab: Show Extension Logs`. +1. Disable and re-enable the **Enable code completion (Beta)** checkbox. +1. Verify that the debug log contains similar output: + +```shell +2023-07-14T17:29:00:763 [debug]: Disabling code completion +2023-07-14T17:29:01:802 [debug]: Enabling code completion +2023-07-14T17:29:01:802 [debug]: AI Assist: Using server: https://codesuggestions.gitlab.com/v2/completions +``` + +### Code Suggestions not displayed in Microsoft Visual Studio + +Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. + +1. Ensure you have properly [set up the extension](https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension#setup). +1. From the **Tools > Options** menu, find the **GitLab** option. Ensure **Log Level** is set to **Debug**. +1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Extension** as the log filter. +1. Verify that the debug log contains similar output: + +```shell +14:48:21:344 GitlabProposalSource.GetCodeSuggestionAsync +14:48:21:344 LsClient.SendTextDocumentCompletionAsync("GitLab.Extension.Test\TestData.cs", 34, 0) +14:48:21:346 LS(55096): time="2023-07-17T14:48:21-05:00" level=info msg="update context" +``` + +## Authentication troubleshooting + +If the above steps do not solve your issue, the problem may be related to the recent changes in authentication, +specifically the token system. To resolve the issue: + +1. Remove the existing personal access token from your GitLab account settings. +1. Reauthorize your GitLab account in VS Code using OAuth. +1. Test the Code Suggestions feature with different file extensions to verify if the issue is resolved. diff --git a/doc/user/project/repository/file_finder.md b/doc/user/project/repository/file_finder.md index ee2be6dee7c..5dd8ad39889 100644 --- a/doc/user/project/repository/file_finder.md +++ b/doc/user/project/repository/file_finder.md @@ -14,7 +14,7 @@ File finder is powered by the [`fuzzaldrin-plus`](https://github.com/jeancroy/fu To search for a file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. In the upper right, select **Find file**. 1. In the search box, start typing the filename. diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 4c37b92b388..a304a8d108d 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -59,8 +59,8 @@ or the command line. GitLab Premium and Ultimate tiers can also automate updates To update your fork from the GitLab UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. Select the fork you want to update. 1. Below the dropdown list for branch name, find the **Forked from** (**{fork}**) information box to determine if your fork is ahead, behind, or both. In this example, @@ -181,13 +181,13 @@ To restore the fork relationship, [use the API](../../../api/projects.md#create- To remove a fork relationship: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the **Remove fork relationship** section, select **Remove fork relationship**. 1. To confirm, enter the project path and select **Confirm**. -When you unlink a fork that uses a [hashed storage pool](../../../administration/repository_storage_types.md#hashed-object-pools) +When you unlink a fork that uses a [hashed storage pool](../../../administration/repository_storage_paths.md#hashed-object-pools) to share objects with another repository: - All objects are copied from the pool into your fork. diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 8bb6d868270..592041ef4e2 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -1,330 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../signed_commits/gpg.md' +remove_date: '2023-12-01' --- -# Sign commits with GPG **(FREE ALL)** +This document was moved to [another location](../signed_commits/gpg.md). -You can sign the commits you make in a GitLab repository with a -GPG ([GNU Privacy Guard](https://gnupg.org/)) key. When you add a cryptographic -signature to your commit, you provide extra assurance that a commit -originated from you, rather than an impersonator. If GitLab can verify a commit -author's identity with a public GPG key, the commit is marked **Verified** in the -GitLab UI. You can then configure [push rules](../push_rules.md) -for your project to reject individual commits not signed with GPG, or reject all -commits from unverified users. - -NOTE: -GitLab uses the term GPG for all OpenPGP, PGP, and GPG-related material and -implementations. - -For GitLab to consider a commit verified: - -- The committer must have a GPG public/private key pair. -- The committer's public key must be uploaded to their GitLab account. -- One of the email addresses in the GPG public key must match a **verified** email address - used by the committer in GitLab. To keep this address private, use the automatically generated - [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) - GitLab provides in your profile. -- The committer's email address must match the verified email address from the - GPG key. - -GitLab uses its own keyring to verify the GPG signature. It does not access any -public key server. - -GPG verified tags are not supported. - -For more details about GPG, refer to the [related topics list](#related-topics). - -## View a user's public GPG key - -To view a user's public GPG key, you can either: - -- Go to `https://gitlab.example.com/<USERNAME>.gpg`. GitLab displays the GPG key, - if the user has configured one, or a blank page for users without a configured GPG key. -- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper-right corner - of the user's profile, select **View public GPG keys** (**{key}**). - -## Configure commit signing - -To sign commits, you must configure both your local machine and your GitLab account: - -1. [Create a GPG key](#create-a-gpg-key). -1. [Add a GPG key to your account](#add-a-gpg-key-to-your-account). -1. [Associate your GPG key with Git](#associate-your-gpg-key-with-git). -1. [Sign your Git commits](#sign-your-git-commits). - -### Create a GPG key - -If you don't already have a GPG key, create one: - -1. [Install GPG](https://www.gnupg.org/download/) for your operating system. - If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in - the commands on this page. -1. To generate your key pair, run the command appropriate for your version of `gpg`: - - ```shell - # Use this command for the default version of GPG, including - # Gpg4win on Windows, and most macOS versions: - gpg --gen-key - - # Use this command for versions of GPG later than 2.1.17: - gpg --full-gen-key - ``` - -1. Select the algorithm your key should use, or press <kbd>Enter</kbd> to select - the default option, `RSA and RSA`. -1. Select the key length, in bits. GitLab recommends 4096-bit keys. -1. Specify the validity period of your key. This value is subjective, and the - default value is no expiration. -1. To confirm your answers, enter `y`. -1. Enter your name. -1. Enter your email address. It must match a - [verified email address](../../../profile/index.md#change-the-email-displayed-on-your-commits) - in your GitLab account. -1. Optional. Enter a comment to display in parentheses after your name. -1. GPG displays the information you've entered so far. Edit the information or press - <kbd>O</kbd> (for `Okay`) to continue. -1. Enter a strong password, then enter it again to confirm it. -1. To list your private GPG key, run this command, replacing - `<EMAIL>` with the email address you used when you generated the key: - - ```shell - gpg --list-secret-keys --keyid-format LONG <EMAIL> - ``` - -1. In the output, identify the `sec` line, and copy the GPG key ID. It begins after - the `/` character. In this example, the key ID is `30F2B65B9246B6CA`: - - ```plaintext - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` - -1. To show the associated public key, run this command, replacing `<ID>` with the - GPG key ID from the previous step: - - ```shell - gpg --armor --export <ID> - ``` - -1. Copy the public key, including the `BEGIN PGP PUBLIC KEY BLOCK` and - `END PGP PUBLIC KEY BLOCK` lines. You need this key in the next step. - -### Add a GPG key to your account - -To add a GPG key to your user settings: - -1. Sign in to GitLab. -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Add new key**. -1. In **Key**, paste your _public_ key. -1. To add the key to your account, select **Add key**. GitLab shows the key's - fingerprint, email address, and creation date: - -  - -After you add a key, you cannot edit it. Instead, remove the offending key and re-add it. - -### Associate your GPG key with Git - -After you [create your GPG key](#create-a-gpg-key) and -[add it to your account](#add-a-gpg-key-to-your-account), you must configure Git -to use this key: - -1. Run this command to list the private GPG key you just created, - replacing `<EMAIL>` with the email address for your key: - - ```shell - gpg --list-secret-keys --keyid-format LONG <EMAIL> - ``` - -1. Copy the GPG private key ID that starts with `sec`. In this example, the private key ID is - `30F2B65B9246B6CA`: - - ```plaintext - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` - -1. Run this command to configure Git to sign your commits with your key, - replacing `<KEY ID>` with your GPG key ID: - - ```shell - git config --global user.signingkey <KEY ID> - ``` - -### Sign your Git commits - -After you [add your public key to your account](#add-a-gpg-key-to-your-account), -you can sign individual commits manually, or configure Git to default to signed commits: - -- Sign individual Git commits manually: - 1. Add `-S` flag to any commit you want to sign: - - ```shell - git commit -S -m "My commit message" - ``` - - 1. Enter the passphrase of your GPG key when asked. - 1. Push to GitLab and check that your commits [are verified](#verify-commits). -- Sign all Git commits by default by running this command: - - ```shell - git config --global commit.gpgsign true - ``` - -#### Set signing key conditionally - -If you maintain signing keys for separate purposes, such as work and personal -use, use an `IncludeIf` statement in your `.gitconfig` file to set which key -you sign commits with. - -Prerequisites: - -- Requires Git version 2.13 or later. - -1. In the same directory as your main `~/.gitconfig` file, create a second file, - such as `.gitconfig-gitlab`. -1. In your main `~/.gitconfig` file, add your Git settings for work in non-GitLab projects. -1. Append this information to the end of your main `~/.gitconfig` file: - - ```ini - # The contents of this file are included only for GitLab.com URLs - [includeIf "hasconfig:remote.*.url:https://gitlab.com/**"] - - # Edit this line to point to your alternate configuration file - path = ~/.gitconfig-gitlab - ``` - -1. In your alternate `.gitconfig-gitlab` file, add the configuration overrides to - use when you're committing to a GitLab repository. All settings from your - main `~/.gitconfig` file are retained unless you explicitly override them. - In this example, - - ```ini - # Alternate ~/.gitconfig-gitlab file - # These values are used for repositories matching the string 'gitlab.com', - # and override their corresponding values in ~/.gitconfig - - [user] - email = you@example.com - signingkey = <KEY ID> - - [commit] - gpgsign = true - ``` - -## Verify commits - -You can review commits for a merge request, or for an entire project: - -1. To review commits for a project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Commits**. -1. To review commits for a merge request: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Merge requests**, then select your merge request. - 1. Select **Commits**. -1. Identify the commit you want to review. Signed commits show either a **Verified** - or **Unverified** badge, depending on the verification status of the GPG - signature. Unsigned commits do not display a badge: - -  - -1. To display the signature details for a commit, select the GPG badge: - -  - -  - -## Revoke a GPG key - -If a GPG key becomes compromised, revoke it. Revoking a key changes both future and past commits: - -- Past commits signed by this key are marked as unverified. -- Future commits signed by this key are marked as unverified. - -To revoke a GPG key: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Revoke** next to the GPG key you want to delete. - -## Remove a GPG key - -When you remove a GPG key from your GitLab account: - -- Previous commits signed with this key remain verified. -- Future commits (including any commits created but not yet pushed) that attempt - to use this key are unverified. - -To remove a GPG key from your account: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Remove** (**{remove}**) next to the GPG key you want to delete. - -If you must unverify both future and past commits, -[revoke the associated GPG key](#revoke-a-gpg-key) instead. - -## Related topics - -- [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) -- [Sign commits with SSH keys](../ssh_signed_commits/index.md) -- [Commits API](../../../../api/commits.md) -- GPG resources: - - [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) - - [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) - - [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) - - [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) - - [Review existing GPG keys in your instance](../../../../administration/credentials_inventory.md#review-existing-gpg-keys) - -## Troubleshooting - -### Fix verification problems with signed commits - -Commits can be signed with [X.509 certificates](../x509_signed_commits/index.md) -or a GPG key. The verification process for both methods can fail for multiple reasons: - -| Value | Description | Possible Fixes | -|-----------------------------|-------------|----------------| -| `UNVERIFIED` | The commit signature is not valid. | Sign the commit with a valid signature. | -| `SAME_USER_DIFFERENT_EMAIL` | The GPG key used to sign the commit does not contain the committer email, but does contain a different valid email for the committer. | Amend the commit to use an email address that matches the GPG key, or update the GPG key [to include the email address](https://security.stackexchange.com/a/261468). | -| `OTHER_USER` | The signature and GPG key are valid, but the key belongs to a different user than the committer. | Amend the commit to use the correct email address, or amend the commit to use a GPG key associated with your user. | -| `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | -| `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](#add-a-gpg-key-to-your-account) to your GitLab profile. | -| `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | - -### Secret key not available - -If you receive the errors `secret key not available` -or `gpg: signing failed: secret key not available`, try using `gpg2` instead of `gpg`: - -```shell -git config --global gpg.program gpg2 -``` - -If your GPG key is password protected and the password entry prompt does not appear, -add `export GPG_TTY=$(tty)` to your shell's `rc` file (commonly `~/.bashrc` or `~/.zshrc`) - -### GPG failed to sign the data - -If your GPG key is password protected and you receive the error: - -```shell -error: gpg failed to sign the data -fatal: failed to write commit object -``` - -If the password entry prompt does not appear, add `export GPG_TTY=$(tty)` to your shell's `rc` file -(commonly `~/.bashrc` or `~/.zshrc`) and restart your terminal. +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 3d00ceafc05..e97892458b4 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -55,7 +55,7 @@ to a branch in the repository. When you use the command line, you can commit mul [Revert a commit](../merge_requests/revert_changes.md#revert-a-commit) from the UI to a selected branch. - **Sign a commit:** - Use GPG to [sign your commits](gpg_signed_commits/index.md). + Add extra security by [signing your commits](signed_commits/index.md). ## Clone a repository @@ -73,7 +73,7 @@ into Xcode on macOS. 1. Select **Xcode**. The project is cloned onto your computer and you are -prompted to open XCode. +prompted to open Xcode. ### Clone and open in Visual Studio Code @@ -216,6 +216,11 @@ To render an OpenAPI file: 1. Go to the OpenAPI file in your repository. 1. Between the **Display source** and **Edit** buttons, select **Display OpenAPI**. When an OpenAPI file is found, it replaces the **Display rendered file** button. +1. To display the `operationId` in the operations list, add `displayOperationId=true` to the query string. + +NOTE: +When `displayOperationId` is present in the query string and has _any_ value, it +evaluates to `true`. This behavior matches the default behavior of Swagger. ## Repository size diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index 70ee841a991..2a1fc0cddf8 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -37,7 +37,7 @@ When commits include changes to Jupyter Notebook files, GitLab: - Enables switching between raw and rendered diffs on the Commit and Compare pages. (Not available on merge request pages.) - Renders images on the diffs. -Code suggestions are not available on diffs and merge requests for `.ipynb` files. +Code Suggestions are not available on diffs and merge requests for `.ipynb` files. Cleaner notebook diffs are not generated when the notebook is too large. diff --git a/doc/user/project/repository/managing_large_repositories.md b/doc/user/project/repository/managing_large_repositories.md index e5c1e4a6614..1d5127b5e08 100644 --- a/doc/user/project/repository/managing_large_repositories.md +++ b/doc/user/project/repository/managing_large_repositories.md @@ -1,53 +1,411 @@ --- stage: Systems -group: Distribution +group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: "Documentation on large repositories." --- -# Managing large repositories **(FREE SELF)** +# Managing monorepos -GitLab, like any Git based system, is subject to similar performance restraints when it comes to large -repositories that size into the gigabytes. +Monorepos have become a regular part of development team workflows. While they have many advantages, monorepos can present performance challenges +when using them in GitLab. Therefore, you should know: -In the following sections, we detail several best practices for improving performance with these large repositories on GitLab. +- What repository characteristics can impact performance. +- Some tools and steps to optimize monorepos. -## Large File System (LFS) +## Impact on performance -It's *strongly* recommended in any Git system that binary or blob files (for example, packages, audio, video, or graphics) are stored as Large File Storage (LFS) objects. With LFS, the objects are stored externally, such as in Object Storage, which reduces the number and size of objects in the repository. Storing objects in external Object Storage can improve performance. +Because GitLab is a Git-based system, it is subject to similar performance +constraints as Git when it comes to large repositories that are gigabytes in +size. -To analyze if a repository has large objects, you can use a tool like [`git-sizer`](https://github.com/github/git-sizer) for detailed analysis. This tool shows details about what makes up the repository, and highlights any areas of concern. If any large objects are found, you can then remove them with a tool such as [`git filter-repo`](reducing_the_repo_size_using_git.md). +Monorepos can be large for [many reasons](https://about.gitlab.com/blog/2022/09/06/speed-up-your-monorepo-workflow-in-git/#characteristics-of-monorepos). + +Large repositories pose a performance risk performance when used in GitLab, especially if a large monorepo receives many clones or pushes a day, which is common for them. + +Git itself has performance limitations when it comes to handling +monorepos. + +[Gitaly](https://gitlab.com/gitlab-org/gitaly) is our Git storage service built +on top of [Git](https://git-scm.com/). This means that any limitations of +Git are experienced in Gitaly, and in turn by end users of GitLab. + +## Profiling repositories + +Large repositories generally experience performance issues in Git. Knowing why +your repository is large can help you develop mitigation strategies to avoid +performance problems. + +You can use [`git-sizer`](https://github.com/github/git-sizer) to get a snapshot +of repository characteristics and discover problem aspects of your monorepo. + +For example: + +```shell +Processing blobs: 1652370 +Processing trees: 3396199 +Processing commits: 722647 +Matching commits to trees: 722647 +Processing annotated tags: 534 +Processing references: 539 +| Name | Value | Level of concern | +| ---------------------------- | --------- | ------------------------------ | +| Overall repository size | | | +| * Commits | | | +| * Count | 723 k | * | +| * Total size | 525 MiB | ** | +| * Trees | | | +| * Count | 3.40 M | ** | +| * Total size | 9.00 GiB | **** | +| * Total tree entries | 264 M | ***** | +| * Blobs | | | +| * Count | 1.65 M | * | +| * Total size | 55.8 GiB | ***** | +| * Annotated tags | | | +| * Count | 534 | | +| * References | | | +| * Count | 539 | | +| | | | +| Biggest objects | | | +| * Commits | | | +| * Maximum size [1] | 72.7 KiB | * | +| * Maximum parents [2] | 66 | ****** | +| * Trees | | | +| * Maximum entries [3] | 1.68 k | * | +| * Blobs | | | +| * Maximum size [4] | 13.5 MiB | * | +| | | | +| History structure | | | +| * Maximum history depth | 136 k | | +| * Maximum tag depth [5] | 1 | | +| | | | +| Biggest checkouts | | | +| * Number of directories [6] | 4.38 k | ** | +| * Maximum path depth [7] | 13 | * | +| * Maximum path length [8] | 134 B | * | +| * Number of files [9] | 62.3 k | * | +| * Total size of files [9] | 747 MiB | | +| * Number of symlinks [10] | 40 | | +| * Number of submodules | 0 | | +``` + +In this example, a few items are raised with a high level of concern. See the +following sections for information on solving: + +- A high number of references. +- Large blobs. + +### Large number of references + +A reference in Git (a branch or tag) is used to refer to a commit. Each +reference is stored as an individual file. If you are curious, you can go +to any `.git` directory and look under the `refs` directory. + +A large number of references can cause performance problems because, with more references, +object walks that Git does are larger for various operations such as clones, pushes, and +housekeeping tasks. + +#### Mitigation strategies + +To mitigate the effects of a large number of references in a monorepo: + +- Create an automated process for cleaning up old branches. +- If certain references don't need to be visible to the client, hide them using the + [`transfer.hideRefs`](https://git-scm.com/docs/git-config#Documentation/git-config.txt-transferhideRefs) + configuration setting. Because Gitaly ignores any on-server Git configuration, you must change the Gitaly configuration + itself in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['configuration'] = { + # ... + git: { + # ... + config: [ + # ... + { key: "transfer.hideRefs", value: "refs/namespace_to_hide" }, + ], + }, + } + ``` + +In Git 2.42.0 and later, different Git operations can skip over hidden references +when doing an object graph walk. + +### Using LFS for large blobs + +Because Git is built to handle text data, it doesn't handle large +binary files efficiently. + +Therefore, you should store binary or blob files (for example, packages, audio, video, or graphics) +as Large File Storage (LFS) objects. With LFS, the objects are stored externally, such as in Object +Storage, which reduces the number and size of objects in the repository. Storing +objects in external Object Storage can improve performance. + +To analyze if a repository has large objects, you can use a tool like +[`git-sizer`](https://github.com/github/git-sizer) for detailed analysis. This +tool shows details about what makes up the repository, and highlights any areas +of concern. If any large objects are found, you can then remove them with a tool +such as [`git filter-repo`](reducing_the_repo_size_using_git.md). For more information, refer to the [Git LFS documentation](../../../topics/git/lfs/index.md). -## Gitaly Pack Objects Cache +## Optimizing large repositories for GitLab + +Other than modifying your workflow and the actual repository, you can take other +steps to maximize performance of monorepos with GitLab. + +### Gitaly pack-objects cache + +For very active repositories with a large number of references and files, consider using the +[Gitaly pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). +The pack-objects cache: + +- Benefits all repositories on your GitLab server. +- Automatically works for forks. + +You should always: + +- Fetch incrementally. Do not clone in a way that recreates all of the worktree. +- Use shallow clones to reduce data transfer. Be aware that this puts more burden on GitLab instance because of higher CPU impact. + +Control the clone directory if you heavily use a fork-based workflow. Optimize +`git clean` flags to ensure that you remove or keep data that might affect or +speed-up your build. + +For more information, see [Pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). + +### Reduce concurrent clones in CI/CD + +Large repositories tend to be monorepos. This usually means that these +repositories get a lot of traffic not only from users, but from CI/CD. + +CI/CD loads tend to be concurrent because pipelines are scheduled during set times. +As a result, the Git requests against the repositories can spike notably during +these times and lead to reduced performance for both CI/CD and users alike. + +You should reduce CI/CD pipeline concurrency by staggering them to run at different times. For example, a set running at one time and another set running several +minutes later. + +#### Shallow cloning + +GitLab and GitLab Runner perform a [shallow clone](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) +by default. + +Ideally, you should always use `GIT_DEPTH` with a small number +like 10. This instructs GitLab Runner to perform shallow clones. +Shallow clones make Git request only the latest set of changes for a given branch, +up to desired number of commits as defined by the `GIT_DEPTH` variable. + +This significantly speeds up fetching of changes from Git repositories, +especially if the repository has a very long backlog consisting of a number +of big files because we effectively reduce amount of data transfer. +The following pipeline configuration example makes the runner shallow clone to fetch only a given branch. +The runner does not fetch any other branches nor tags. + +```yaml +variables: + GIT_DEPTH: 10 + +test: + script: + - ls -al +``` + +#### Git strategy + +By default, GitLab is configured to use the [`fetch` Git strategy](../../../ci/runners/configure_runners.md#git-strategy), +which is recommended for large repositories. +This strategy reduces the amount of data to transfer and +does not really impact the operations that you might do on a repository from CI/CD. + +#### Git clone path + +[`GIT_CLONE_PATH`](../../../ci/runners/configure_runners.md#custom-build-directories) allows you to +control where you clone your repositories. This can have implications if you +heavily use big repositories with a fork-based workflow. + +A fork, from the perspective of GitLab Runner, is stored as a separate repository +with a separate worktree. That means that GitLab Runner cannot optimize the usage +of worktrees and you might have to instruct GitLab Runner to use that. + +In such cases, ideally you want to make the GitLab Runner executor be used only +for the given project and not shared across different projects to make this +process more efficient. + +The [`GIT_CLONE_PATH`](../../../ci/runners/configure_runners.md#custom-build-directories) must be +in the directory set in `$CI_BUILDS_DIR`. You can't pick any path from disk. + +#### Git clean flags + +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags) allows you to control +whether or not you require the `git clean` command to be executed for each CI/CD +job. By default, GitLab ensures that: + +- You have your worktree on the given SHA. +- Your repository is clean. + +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags) is disabled when set +to `none`. On very big repositories, this might be desired because `git +clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx +-e .build/` (for example) allows you to control and disable removal of some +directories in the worktree between subsequent runs, which can speed-up +the incremental builds. This has the biggest effect if you re-use existing +machines and have an existing worktree that you can re-use for builds. + +For exact parameters accepted by +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags), see the documentation +for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters +are dependent on the Git version. + +#### Git fetch extra flags + +[`GIT_FETCH_EXTRA_FLAGS`](../../../ci/runners/configure_runners.md#git-fetch-extra-flags) allows you +to modify `git fetch` behavior by passing extra flags. + +For example, if your project contains a large number of tags that your CI/CD jobs don't rely on, +you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) +to the extra flags to make your fetches faster and more compact. + +Also in the case where you repository does _not_ contain a lot of +tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). +If your CI/CD builds do not depend on Git tags, setting `--no-tags` is worth trying. + +For more information, see the [`GIT_FETCH_EXTRA_FLAGS` documentation](../../../ci/runners/configure_runners.md#git-fetch-extra-flags). + +#### Fork-based workflow + +Following the guidelines above, let's imagine that we want to: + +- Optimize for a big project (more than 50k files in directory). +- Use forks-based workflow for contributing. +- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. +- Runner assigned only to project and all forks. + +Let's consider the following two examples, one using `shell` executor and +other using `docker` executor. + +##### `shell` executor example + +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "shell" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.custom_build_dir] + enabled = true +``` + +This `config.toml`: + +- Uses the `shell` executor, +- Specifies a custom `/builds` directory where all clones are stored. +- Enables the ability to specify `GIT_CLONE_PATH`, +- Runs at most 4 jobs at once. + +##### `docker` executor example + +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This `config.toml`: + +- Uses the `docker` executor, +- Specifies a custom `/builds` directory on disk where all clones are stored. + We host mount the `/builds` directory to make it reusable between subsequent runs + and be allowed to override the cloning strategy. +- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. +- Runs at most 4 jobs at once. + +##### Our `.gitlab-ci.yml` + +Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. + +Our pipeline is most performant if we use the following `.gitlab-ci.yml`: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME + +build: + script: ls -al +``` + +This YAML setting configures a custom clone path. This path makes it possible to re-use worktrees +between the parent project and forks because we use the same clone path for all forks. + +Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting +between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor. +When we use it to construct the path, this directory does not conflict +with other concurrent jobs running. + +### Store custom clone options in `config.toml` + +Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. +However, sometimes it is desirable to make these schemes part of the runner's configuration. -Gitaly, the service that provides storage for Git repositories, can be configured to cache a short rolling window of Git fetch responses. This is recommended for large repositories as it can notably reduce server load when your server receives lots of fetch traffic. +In the above example of forks, making this configuration discoverable for users may be preferred, +but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. +In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it +a configuration of the runner. -Refer to the [Gitaly Pack Objects Cache for more information](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). +We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +with the following specification that is used by the runner if `.gitlab-ci.yml` does not override it: -## Reference Architectures +```toml +concurrent = 4 -Large repositories tend to be found in larger organisations with many users. The GitLab Quality and Support teams provide several [Reference Architectures](../../../administration/reference_architectures/index.md) that are the recommended way to deploy GitLab at scale. +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" -In these types of setups it's recommended that the GitLab environment used matches a Reference Architecture to improve performance. + environment = [ + "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" + ] -## Gitaly Cluster + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` -Gitaly Cluster can notably improve large repository performance as it holds multiple replicas of the repository across several nodes. As a result, Gitaly Cluster can load balance read requests against those repositories and is also fault-tolerant. +This makes the cloning configuration to be part of the given runner +and does not require us to update each `.gitlab-ci.yml`. -It's recommended for large repositories, however, Gitaly Cluster is a large solution with additional complexity of setup, and management. Refer to the [Gitaly Cluster documentation for more information](../../../administration/gitaly/index.md), specifically the [Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. +### Reference architectures -## Keep GitLab up to date +Large repositories tend to be found in larger organisations with many users. The GitLab Quality and Support teams provide several [reference architectures](../../../administration/reference_architectures/index.md) that are the recommended way to deploy GitLab at scale. -Performance improvements and fixes are added continuously in GitLab. As such, it's recommended you keep GitLab updated to the latest version where possible to benefit from these. +In these types of setups, the GitLab environment used should match a reference architecture to improve performance. -## Reduce concurrent clones in CI/CD +### Gitaly Cluster -Large repositories tend to be monorepos. This in turn typically means that these repositories get a lot of traffic not only from users, but from CI/CD. +Gitaly Cluster can notably improve large repository performance because it holds multiple replicas of the repository across several nodes. +As a result, Gitaly Cluster can load balance read requests against those replicas and is fault-tolerant. -CI/CD loads tend to be concurrent as pipelines are scheduled during set times. As a result, the Git requests against the repositories can spike notably during these times and lead to reduced performance for both CI and users alike. +Though Gitaly Cluster is recommended for large repositories, it is a large solution with additional complexity of setup and management. Refer to the +[Gitaly Cluster documentation for more information](../../../administration/gitaly/index.md), specifically the +[Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. -When designing CI/CD pipelines, it's advisable to reduce their concurrency by staggering them to run at different times, for example, a set running at one time, and another set running several minutes later. +### Keep GitLab up to date -There's several other actions that can be explored to improve CI/CD performance with large repositories. Refer to the [Runner documentation for more information](../../../ci/large_repositories/index.md). +You should keep GitLab updated to the latest version where possible to benefit from performance improvements and fixes are added continuously to GitLab. diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md index fade9e1b63c..9d97f53cb43 100644 --- a/doc/user/project/repository/mirror/bidirectional.md +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -44,7 +44,7 @@ and [pull](pull.md#pull-from-a-remote-repository) mirrors in the upstream GitLab To create the webhook in the downstream instance: 1. Create a [personal access token](../../../profile/personal_access_tokens.md) with `API` scope. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > Webhooks**. 1. Add the webhook **URL**, which (in this case) uses the [Pull Mirror API](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project) diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index 7ade27e556d..b16197e45b2 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -40,7 +40,7 @@ Prerequisites: - If your mirror connects with `ssh://`, the host key must be detectable on the server, or you must have a local copy of the key. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Select **Add new**. @@ -111,7 +111,7 @@ Prerequisite: - You must have at least the Maintainer role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Scroll to **Mirrored repositories** and identify the mirror to update. @@ -124,9 +124,8 @@ When you create a mirror, you must configure the authentication method for it. GitLab supports these authentication methods: - [SSH authentication](#ssh-authentication). -- Password. +- Username and password. -When using password authentication, ensure you specify the username. For a [project access token](../../settings/project_access_tokens.md) or [group access token](../../../group/settings/group_access_tokens.md), use the username (not token name) and the token as the password. @@ -154,7 +153,7 @@ When you mirror a repository and select the **SSH public key** as your authentication method, GitLab generates a public key for you. The non-GitLab server needs this key to establish trust with your GitLab repository. To copy your SSH public key: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Scroll to **Mirrored repositories**. diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md index ba54c18f8ee..5b3c76d982a 100644 --- a/doc/user/project/repository/mirror/pull.md +++ b/doc/user/project/repository/mirror/pull.md @@ -64,14 +64,13 @@ Prerequisites: token serves as your GitHub password. - [GitLab Silent Mode](../../../../administration/silent_mode/index.md) is not enabled. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. -1. Enter the **Git repository URL**. Include the username - in the URL, if required: `https://MYUSERNAME@gitlab.com/GROUPNAME/PROJECTNAME.git` +1. Enter the **Git repository URL**. NOTE: - To mirror the `gitlab` repository, use `git@gitlab.com:gitlab-org/gitlab.git` + To mirror the `gitlab` repository, use `gitlab.com:gitlab-org/gitlab.git` or `https://gitlab.com/gitlab-org/gitlab.git`. 1. In **Mirror direction**, select **Pull**. diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md index cd4fe68b01b..e18e3631d7f 100644 --- a/doc/user/project/repository/mirror/push.md +++ b/doc/user/project/repository/mirror/push.md @@ -37,7 +37,7 @@ and pulling from, remote mirrors. To set up push mirroring for an existing project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Enter a repository URL. @@ -89,12 +89,12 @@ To configure a mirror from GitLab to GitHub: 1. Enter a **Git repository URL** with this format, changing the variables as needed: ```plaintext - https://USERNAME@github.com/GROUP/PROJECT.git + https://github.com/GROUP/PROJECT.git ``` - - `USERNAME`: The username of the owner of the personal access token. - `GROUP`: The group on GitHub. - `PROJECT`: The project on GitHub. +1. For **Username**, enter the username of the owner of the personal access token. 1. For **Password**, enter your GitHub personal access token. 1. Select **Mirror repository**. @@ -164,19 +164,17 @@ To set up a mirror from GitLab to AWS CodeCommit: 1. Open your new repository, and then select **Clone URL > Clone HTTPS** (not **Clone HTTPS (GRC)**). 1. In GitLab, open the repository to be push-mirrored. 1. Select **Settings > Repository**, and then expand **Mirroring repositories**. -1. Fill in the **Git repository URL** field using this format: +1. Fill in the **Git repository URL** field using this format, replacing + `<aws-region>` with your AWS region, and + `<your_codecommit_repo>` with the name of your repository in CodeCommit: ```plaintext - https://<your_aws_git_userid>@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> + https://git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> ``` - Replace `<your_aws_git_userid>` with the AWS **special HTTPS Git user ID** - from the IAM Git credentials created earlier. Replace `<your_codecommit_repo>` - with the name of your repository in CodeCommit. - -1. For **Mirror direction**, select **Push**. -1. For **Authentication method**, select **Password**. Fill in the **Password** box - with the special IAM Git clone user ID **password** created earlier in AWS. +1. For **Authentication method**, select **Username and Password**. +1. For **Username**, enter the AWS **special HTTPS Git user ID**. +1. For **Password**, enter the special IAM Git clone user ID password created earlier in AWS. 1. Leave the option **Only mirror protected branches** for CodeCommit. It pushes more frequently (from every five minutes to every minute). @@ -202,7 +200,8 @@ If it isn't working correctly, a red `error` tag appears, and shows the error me [personal access token](../../../profile/personal_access_tokens.md) with `write_repository` scope. 1. On the source GitLab instance: 1. Enter the **Git repository URL** using this format: - `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + `https://<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + 1. Enter the **Username** `oauth2`. 1. Enter the **Password**. Use the GitLab personal access token created on the destination GitLab instance. 1. Select **Mirror repository**. diff --git a/doc/user/project/repository/mirror/troubleshooting.md b/doc/user/project/repository/mirror/troubleshooting.md index 5817aab5fc7..a83231ceb63 100644 --- a/doc/user/project/repository/mirror/troubleshooting.md +++ b/doc/user/project/repository/mirror/troubleshooting.md @@ -67,7 +67,7 @@ If you receive this error after creating a new project using Check if the repository owner is specified in the URL of your mirrored repository: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. If no repository owner is specified, delete and add the URL again in this format, @@ -168,7 +168,7 @@ Prerequisites: To resolve the issue: 1. [Verify the host key](index.md#verify-a-host-key). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. To refresh the keys, either: diff --git a/doc/user/project/repository/push_rules.md b/doc/user/project/repository/push_rules.md index 2756149b5bd..d61d09301a5 100644 --- a/doc/user/project/repository/push_rules.md +++ b/doc/user/project/repository/push_rules.md @@ -39,7 +39,7 @@ Prerequisite: To create global push rules: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Push Rules**. 1. Expand **Push rules**. @@ -52,7 +52,7 @@ The push rule of an individual project overrides the global push rule. To override global push rules for a specific project, or to update the rules for an existing project to match new global push rules: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Push rules**. 1. Set the rule you want. @@ -126,7 +126,7 @@ Some validation examples: Use these rules to prevent unintended consequences. -- **Reject unsigned commits**: Commit must be signed through [GPG](gpg_signed_commits/index.md). This rule +- **Reject unsigned commits**: Commit must be signed through [GPG](signed_commits/gpg.md). This rule can block some legitimate commits [created in the Web IDE](#reject-unsigned-commits-push-rule-disables-web-ide), and allow [unsigned commits created in the GitLab UI](#unsigned-commits-created-in-the-gitlab-ui). - **Do not allow users to remove Git tags with `git push`**: Users cannot use `git push` to remove Git tags. @@ -274,7 +274,7 @@ to use them as standard characters in a match condition. ## Related topics - [Git server hooks](../../../administration/server_hooks.md) (previously called server hooks), to create complex custom push rules -- [Signing commits with GPG](gpg_signed_commits/index.md) +- [Signing commits with GPG](signed_commits/gpg.md) - [Protected branches](../protected_branches.md) ## Troubleshooting diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 590323bfadd..bfe8964a876 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -26,7 +26,7 @@ you begin. The best way to back up a repository is to The size of a repository is determined by computing the accumulated size of all files in the repository. It is similar to executing `du --summarize --bytes` on your repository's -[hashed storage path](../../../administration/repository_storage_types.md). +[hashed storage path](../../../administration/repository_storage_paths.md). ## Purge files from repository history @@ -53,7 +53,7 @@ visible even after they have been removed from the repository. To purge files from a GitLab repository: -1. Install either [`git filter-repo`](https://github.com/newren/git-filter-repo/blob/main/INSTALL.md) or +1. Install [`git filter-repo`](https://github.com/newren/git-filter-repo/blob/main/INSTALL.md) and optionally [`git-sizer`](https://github.com/github/git-sizer#getting-started) using a supported package manager or from source. diff --git a/doc/user/project/repository/signed_commits/gpg.md b/doc/user/project/repository/signed_commits/gpg.md new file mode 100644 index 00000000000..88f6917d4b6 --- /dev/null +++ b/doc/user/project/repository/signed_commits/gpg.md @@ -0,0 +1,284 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sign commits with GPG **(FREE ALL)** + +You can sign the commits you make in a GitLab repository with a +GPG ([GNU Privacy Guard](https://gnupg.org/)) key. + +NOTE: +GitLab uses the term GPG for all OpenPGP, PGP, and GPG-related material and +implementations. + +For GitLab to consider a commit verified: + +- The committer must have a GPG public/private key pair. +- The committer's public key must be uploaded to their GitLab account. +- One of the email addresses in the GPG public key must match a **verified** email address + used by the committer in GitLab. To keep this address private, use the automatically generated + [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) + GitLab provides in your profile. +- The committer's email address must match the verified email address from the + GPG key. + +GitLab uses its own keyring to verify the GPG signature. It does not access any +public key server. + +GPG verified tags are not supported. + +For more details about GPG, refer to the [related topics list](#related-topics). + +## View a user's public GPG key + +To view a user's public GPG key, you can either: + +- Go to `https://gitlab.example.com/<USERNAME>.gpg`. GitLab displays the GPG key, + if the user has configured one, or a blank page for users without a configured GPG key. +- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper-right corner + of the user's profile, select **View public GPG keys** (**{key}**). + +## Configure commit signing + +To sign commits, you must configure both your local machine and your GitLab account: + +1. [Create a GPG key](#create-a-gpg-key). +1. [Add a GPG key to your account](#add-a-gpg-key-to-your-account). +1. [Associate your GPG key with Git](#associate-your-gpg-key-with-git). +1. [Sign your Git commits](#sign-your-git-commits). + +### Create a GPG key + +If you don't already have a GPG key, create one: + +1. [Install GPG](https://www.gnupg.org/download/) for your operating system. + If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in + the commands on this page. +1. To generate your key pair, run the command appropriate for your version of `gpg`: + + ```shell + # Use this command for the default version of GPG, including + # Gpg4win on Windows, and most macOS versions: + gpg --gen-key + + # Use this command for versions of GPG later than 2.1.17: + gpg --full-gen-key + ``` + +1. Select the algorithm your key should use, or press <kbd>Enter</kbd> to select + the default option, `RSA and RSA`. +1. Select the key length, in bits. GitLab recommends 4096-bit keys. +1. Specify the validity period of your key. This value is subjective, and the + default value is no expiration. +1. To confirm your answers, enter `y`. +1. Enter your name. +1. Enter your email address. It must match a + [verified email address](../../../profile/index.md#change-the-email-displayed-on-your-commits) + in your GitLab account. +1. Optional. Enter a comment to display in parentheses after your name. +1. GPG displays the information you've entered so far. Edit the information or press + <kbd>O</kbd> (for `Okay`) to continue. +1. Enter a strong password, then enter it again to confirm it. +1. To list your private GPG key, run this command, replacing + `<EMAIL>` with the email address you used when you generated the key: + + ```shell + gpg --list-secret-keys --keyid-format LONG <EMAIL> + ``` + +1. In the output, identify the `sec` line, and copy the GPG key ID. It begins after + the `/` character. In this example, the key ID is `30F2B65B9246B6CA`: + + ```plaintext + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` + +1. To show the associated public key, run this command, replacing `<ID>` with the + GPG key ID from the previous step: + + ```shell + gpg --armor --export <ID> + ``` + +1. Copy the public key, including the `BEGIN PGP PUBLIC KEY BLOCK` and + `END PGP PUBLIC KEY BLOCK` lines. You need this key in the next step. + +### Add a GPG key to your account + +To add a GPG key to your user settings: + +1. Sign in to GitLab. +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Add new key**. +1. In **Key**, paste your _public_ key. +1. To add the key to your account, select **Add key**. GitLab shows the key's + fingerprint, email address, and creation date: + +  + +After you add a key, you cannot edit it. Instead, remove the offending key and re-add it. + +### Associate your GPG key with Git + +After you [create your GPG key](#create-a-gpg-key) and +[add it to your account](#add-a-gpg-key-to-your-account), you must configure Git +to use this key: + +1. Run this command to list the private GPG key you just created, + replacing `<EMAIL>` with the email address for your key: + + ```shell + gpg --list-secret-keys --keyid-format LONG <EMAIL> + ``` + +1. Copy the GPG private key ID that starts with `sec`. In this example, the private key ID is + `30F2B65B9246B6CA`: + + ```plaintext + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` + +1. Run this command to configure Git to sign your commits with your key, + replacing `<KEY ID>` with your GPG key ID: + + ```shell + git config --global user.signingkey <KEY ID> + ``` + +### Sign your Git commits + +After you [add your public key to your account](#add-a-gpg-key-to-your-account), +you can sign individual commits manually, or configure Git to default to signed commits: + +- Sign individual Git commits manually: + 1. Add `-S` flag to any commit you want to sign: + + ```shell + git commit -S -m "My commit message" + ``` + + 1. Enter the passphrase of your GPG key when asked. + 1. Push to GitLab and check that your commits [are verified](../signed_commits/index.md#verify-commits). +- Sign all Git commits by default by running this command: + + ```shell + git config --global commit.gpgsign true + ``` + +#### Set signing key conditionally + +If you maintain signing keys for separate purposes, such as work and personal +use, use an `IncludeIf` statement in your `.gitconfig` file to set which key +you sign commits with. + +Prerequisites: + +- Requires Git version 2.13 or later. + +1. In the same directory as your main `~/.gitconfig` file, create a second file, + such as `.gitconfig-gitlab`. +1. In your main `~/.gitconfig` file, add your Git settings for work in non-GitLab projects. +1. Append this information to the end of your main `~/.gitconfig` file: + + ```ini + # The contents of this file are included only for GitLab.com URLs + [includeIf "hasconfig:remote.*.url:https://gitlab.com/**"] + + # Edit this line to point to your alternate configuration file + path = ~/.gitconfig-gitlab + ``` + +1. In your alternate `.gitconfig-gitlab` file, add the configuration overrides to + use when you're committing to a GitLab repository. All settings from your + main `~/.gitconfig` file are retained unless you explicitly override them. + In this example, + + ```ini + # Alternate ~/.gitconfig-gitlab file + # These values are used for repositories matching the string 'gitlab.com', + # and override their corresponding values in ~/.gitconfig + + [user] + email = you@example.com + signingkey = <KEY ID> + + [commit] + gpgsign = true + ``` + +## Revoke a GPG key + +If a GPG key becomes compromised, revoke it. Revoking a key changes both future and past commits: + +- Past commits signed by this key are marked as unverified. +- Future commits signed by this key are marked as unverified. + +To revoke a GPG key: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Revoke** next to the GPG key you want to delete. + +## Remove a GPG key + +When you remove a GPG key from your GitLab account: + +- Previous commits signed with this key remain verified. +- Future commits (including any commits created but not yet pushed) that attempt + to use this key are unverified. + +To remove a GPG key from your account: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Remove** (**{remove}**) next to the GPG key you want to delete. + +If you must unverify both future and past commits, +[revoke the associated GPG key](#revoke-a-gpg-key) instead. + +## Related topics + +- GPG resources: + - [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) + - [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) + - [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) + - [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) + - [Review existing GPG keys in your instance](../../../../administration/credentials_inventory.md#review-existing-gpg-keys) + +## Troubleshooting + +### Secret key not available + +If you receive the errors `secret key not available` +or `gpg: signing failed: secret key not available`, try using `gpg2` instead of `gpg`: + +```shell +git config --global gpg.program gpg2 +``` + +If your GPG key is password protected and the password entry prompt does not appear, +add `export GPG_TTY=$(tty)` to your shell's `rc` file (commonly `~/.bashrc` or `~/.zshrc`) + +### GPG failed to sign the data + +If your GPG key is password protected and you receive the error: + +```shell +error: gpg failed to sign the data +fatal: failed to write commit object +``` + +If the password entry prompt does not appear, add `export GPG_TTY=$(tty)` to your shell's `rc` file +(commonly `~/.bashrc` or `~/.zshrc`) and restart your terminal. diff --git a/doc/user/project/repository/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png b/doc/user/project/repository/signed_commits/img/profile_settings_gpg_keys_single_key.png Binary files differindex ae0a8696c6c..ae0a8696c6c 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png +++ b/doc/user/project/repository/signed_commits/img/profile_settings_gpg_keys_single_key.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_and_unsigned_commits.png b/doc/user/project/repository/signed_commits/img/project_signed_and_unsigned_commits.png Binary files differindex e1d44f15f3f..e1d44f15f3f 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_and_unsigned_commits.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_and_unsigned_commits.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_unverified_signature.png b/doc/user/project/repository/signed_commits/img/project_signed_commit_unverified_signature.png Binary files differindex 763a677f94a..763a677f94a 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_unverified_signature.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_commit_unverified_signature.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_verified_signature.png b/doc/user/project/repository/signed_commits/img/project_signed_commit_verified_signature.png Binary files differindex 1b6fa3fc2e2..1b6fa3fc2e2 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_verified_signature.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_commit_verified_signature.png diff --git a/doc/user/project/repository/signed_commits/index.md b/doc/user/project/repository/signed_commits/index.md new file mode 100644 index 00000000000..e6632ecb81a --- /dev/null +++ b/doc/user/project/repository/signed_commits/index.md @@ -0,0 +1,64 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Signed commits **(FREE ALL)** + +When you add a cryptographic signature to your commit, you provide extra assurance that a commit +originated from you, rather than an impersonator. If GitLab can verify a commit +author's identity with a public GPG key, the commit is marked **Verified** in the +GitLab UI. You can then configure [push rules](../push_rules.md) +for your project to reject individual commits not signed with GPG, or reject all +commits from unverified users. + +Sign commits with your: + +- [SSH key](ssh.md). +- [GPG key](gpg.md). +- [Personal x.509 certificate](x509.md). + +## Verify commits + +You can review commits for a merge request, or for an entire project, to confirm +they are signed: + +1. To review commits for a project: + 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. Select **Code > Commits**. +1. To review commits for a merge request: + 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Merge requests**, then select your merge request. + 1. Select **Commits**. +1. Identify the commit you want to review. Signed commits show either a **Verified** + or **Unverified** badge, depending on the verification status of the signature. + Unsigned commits do not display a badge: + +  + +1. To display the signature details for a commit, select **Verified** to see + the fingerprint or key ID: + +  + +  + +You can also [use the Commits API](../../../../api/commits.md#get-gpg-signature-of-a-commit) +to check a commit's signature. + +## Troubleshooting + +### Fix verification problems with signed commits + +The verification process for commits signed with GPG keys or X.509 certificates +can fail for multiple reasons: + +| Value | Description | Possible Fixes | +|-----------------------------|-------------|----------------| +| `UNVERIFIED` | The commit signature is not valid. | Sign the commit with a valid signature. | +| `SAME_USER_DIFFERENT_EMAIL` | The GPG key used to sign the commit does not contain the committer email, but does contain a different valid email for the committer. | Amend the commit to use an email address that matches the GPG key, or update the GPG key [to include the email address](https://security.stackexchange.com/a/261468). | +| `OTHER_USER` | The signature and GPG key are valid, but the key belongs to a different user than the committer. | Amend the commit to use the correct email address, or amend the commit to use a GPG key associated with your user. | +| `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | +| `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](gpg.md#add-a-gpg-key-to-your-account) to your GitLab profile. | +| `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | diff --git a/doc/user/project/repository/signed_commits/ssh.md b/doc/user/project/repository/signed_commits/ssh.md new file mode 100644 index 00000000000..3572e56da84 --- /dev/null +++ b/doc/user/project/repository/signed_commits/ssh.md @@ -0,0 +1,167 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sign commits with SSH keys **(FREE ALL)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343879) in GitLab 15.7 [with a flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. Enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/384202) in GitLab 15.8. Feature flag `ssh_commit_signatures` removed. + +When you sign commits with SSH keys, GitLab uses the SSH public keys associated +with your GitLab account to cryptographically verify the commit signature. +If successful, GitLab displays a **Verified** label on the commit. + +You may use the same SSH keys for `git+ssh` authentication to GitLab +and signing commit signatures as long as their usage type is **Authentication & Signing**. +It can be verified on the page for [adding an SSH key to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account). + +For more information about managing the SSH keys associated with your GitLab account, see +[Use SSH keys to communicate with GitLab](../../../ssh.md). + +## Configure Git to sign commits with your SSH key + +After you [create an SSH key](../../../ssh.md#generate-an-ssh-key-pair) and +[add it to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) +or [generate it using a password manager](../../../ssh.md#generate-an-ssh-key-pair-with-a-password-manager), +configure Git to begin using the key. + +Prerequisites: + +- Git 2.34.0 or newer. +- OpenSSH 8.0 or newer. + + NOTE: + OpenSSH 8.7 has broken signing functionality. If you are on OpenSSH 8.7, upgrade to OpenSSH 8.8. + +- A SSH key with the usage type of either **Authentication & Signing** or **Signing**. + The SSH key must be one of these types: + - [ED25519](../../../ssh.md#ed25519-ssh-keys) (recommended) + - [RSA](../../../ssh.md#rsa-ssh-keys) + +To configure Git to use your key: + +1. Configure Git to use SSH for commit signing: + + ```shell + git config --global gpg.format ssh + ``` + +1. Specify which SSH key should be used as the signing key, changing the filename + (here, `~/.ssh/examplekey`) to the location of your key. The filename may + differ, depending on how you generated your key: + + ```shell + git config --global user.signingkey ~/.ssh/examplekey + ``` + +## Sign commits with your SSH key + +Prerequisites: + +- You've [created an SSH key](../../../ssh.md#generate-an-ssh-key-pair). +- You've [added the key](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) to your GitLab account. +- You've [configured Git to sign commits](#configure-git-to-sign-commits-with-your-ssh-key) with your SSH key. + +To sign a commit: + +1. Use the `-S` flag when signing your commits: + + ```shell + git commit -S -m "My commit msg" + ``` + +1. Optional. If you don't want to type the `-S` flag every time you commit, tell + Git to sign your commits automatically: + + ```shell + git config --global commit.gpgsign true + ``` + +1. If your SSH key is protected, Git prompts you to enter your passphrase. +1. Push to GitLab. +1. Check that your commits [are verified](#verify-commits). + Signature verification uses the `allowed_signers` file to associate emails and SSH keys. + For help configuring this file, read [Verify commits locally](#verify-commits-locally). + +## Verify commits + +You can verify all types of signed commits +[in the GitLab UI](../signed_commits/index.md#verify-commits). Commits signed +with an SSH key can also be verified locally. + +### Verify commits locally + +To verify commits locally, create an +[allowed signers file](https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#ALLOWED_SIGNERS) +for Git to associate SSH public keys with users: + +1. Create an allowed signers file: + + ```shell + touch allowed_signers + ``` + +1. Configure the `allowed_signers` file in Git: + + ```shell + git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers" + ``` + +1. Add your entry to the allowed signers file. Use this command to add your + email address and public SSH key to the `allowed_signers` file. Replace `<MY_KEY>` + with the name of your key, and `~/.ssh/allowed_signers` + with the location of your project's `allowed_signers` file: + + ```shell + # Modify this line to meet your needs. + # Declaring the `git` namespace helps prevent cross-protocol attacks. + echo "$(git config --get user.email) namespaces=\"git\" $(cat ~/.ssh/<MY_KEY>.pub)" >> ~/.ssh/allowed_signers + ``` + + The resulting entry in the `allowed_signers` file contains your email address, key type, + and key contents, like this: + + ```plaintext + example@gitlab.com namespaces="git" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAmaTS47vRmsKyLyK1jlIFJn/i8wdGQ3J49LYyIYJ2hv + ``` + +1. Repeat the previous step for each user who you want to verify signatures for. + Consider checking this file in to your Git repository if you want to locally + verify signatures for many different contributors. + +1. Use `git log --show-signature` to view the signature status for the commits: + + ```shell + $ git log --show-signature + + commit e2406b6cd8ebe146835ceab67ff4a5a116e09154 (HEAD -> main, origin/main, origin/HEAD) + Good "git" signature for johndoe@example.com with ED25519 key SHA256:Ar44iySGgxic+U6Dph4Z9Rp+KDaix5SFGFawovZLAcc + Author: John Doe <johndoe@example.com> + Date: Tue Nov 29 06:54:15 2022 -0600 + + SSH signed commit + ``` + +## Revoke an SSH key for signing commits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108344) in GitLab 15.9. + +If an SSH key becomes compromised, revoke it. Revoking a key changes both future and past commits: + +- Past commits signed by this key are marked as unverified. +- Future commits signed by this key are marked as unverified. + +To revoke an SSH key: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. On the left sidebar, select (**{key}**) **SSH Keys**. +1. Select **Revoke** next to the SSH key you want to delete. + +## Related topics + +- [Sign commits and tags with X.509 certificates](../signed_commits/x509.md) +- [Sign commits with GPG](gpg.md) +- [Commits API](../../../../api/commits.md) diff --git a/doc/user/project/repository/signed_commits/x509.md b/doc/user/project/repository/signed_commits/x509.md new file mode 100644 index 00000000000..17767cbd8f4 --- /dev/null +++ b/doc/user/project/repository/signed_commits/x509.md @@ -0,0 +1,362 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sign commits and tags with X.509 certificates **(FREE ALL)** + +[X.509](https://en.wikipedia.org/wiki/X.509) is a standard format for public key +certificates issued by a public or private Public Key Infrastructure (PKI). +Personal X.509 certificates are used for authentication or signing purposes +such as S/MIME (Secure/Multipurpose Internet Mail Extensions). +However, Git also supports signing of commits and tags with X.509 certificates in a +similar way as with [GPG (GnuPG, or GNU Privacy Guard)](gpg.md). +The main difference is the way GitLab determines whether or not the developer's signature is trusted: + +- For X.509, a root certificate authority is added to the GitLab trust store. + (A trust store is a repository of trusted security certificates.) Combined with + any required intermediate certificates in the signature, the developer's certificate + can be chained back to a trusted root certificate. +- For GPG, developers [add their GPG key](gpg.md#add-a-gpg-key-to-your-account) + to their account. + +GitLab uses its own certificate store and therefore defines the +[trust chain](https://www.ssl.com/faqs/what-is-a-certificate-authority/). +For a commit or tag to be *verified* by GitLab: + +- The signing certificate email must match a verified email address in GitLab. +- The GitLab instance must be able to establish a full trust chain + from the certificate in the signature to a trusted certificate in the GitLab certificate store. + This chain may include intermediate certificates supplied in the signature. You may + need to add certificates, such as Certificate Authority root certificates, + [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). +- The signing time must be in the time range of the + [certificate validity](https://www.rfc-editor.org/rfc/rfc5280.html#section-4.1.2.5), + which is usually up to three years. +- The signing time is equal to, or later than, the commit time. + +If a commit's status has already been determined and stored in the database, +use the Rake task [to re-check the status](../../../../raketasks/x509_signatures.md). +Refer to the [Troubleshooting section](#troubleshooting). +GitLab checks certificate revocation lists on a daily basis with a background worker. + +## Limitations + +- Certificates without `authorityKeyIdentifier`, + `subjectKeyIdentifier`, and `crlDistributionPoints` display as **Unverified**. We + recommend using certificates from a PKI that are in line with + [RFC 5280](https://www.rfc-editor.org/rfc/rfc5280). +- If you have more than one email in the Subject Alternative Name list in + your signing certificate, + [only the first one is used to verify commits](https://gitlab.com/gitlab-org/gitlab/-/issues/336677). +- The `X509v3 Subject Key Identifier` (SKI) in the issuer certificate and the + signing certificate + [must be 40 characters long](https://gitlab.com/gitlab-org/gitlab/-/issues/332503). + If your SKI is shorter, commits don't show as verified in GitLab, and + short subject key identifiers may also + [cause errors when accessing the project](https://gitlab.com/gitlab-org/gitlab/-/issues/332464), + such as 'An error occurred while loading commit signatures' and + `HTTP 422 Unprocessable Entity` errors. + +## Configure for signed commits + +To sign your commits, tags, or both, you must: + +1. [Obtain an X.509 key pair](#obtain-an-x509-key-pair). +1. [Associate your X.509 certificate with Git](#associate-your-x509-certificate-with-git). +1. [Sign and verify commits](#sign-and-verify-commits). +1. [Sign and verify tags](#sign-and-verify-tags). + +### Obtain an X.509 key pair + +If your organization has Public Key Infrastructure (PKI), that PKI provides +an S/MIME key. If you do not have an S/MIME key pair from a PKI, you can either +create your own self-signed pair, or purchase a pair. + +### Associate your X.509 certificate with Git + +To take advantage of X.509 signing, you need Git 2.19.0 or later. You can +check your Git version with the command `git --version`. + +If you have the correct version, you can proceed to configure Git. + +### Linux + +Configure Git to use your key for signing: + +```shell +signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) +git config --global user.signingkey $signingkey +git config --global gpg.format x509 +``` + +#### Windows and macOS + +To configure Windows or macOS: + +1. Install [S/MIME Sign](https://github.com/github/smimesign) by either: + - Downloading the installer. + - Running `brew install smimesign` on macOS. +1. Get the ID of your certificate by running `smimesign --list-keys`. +1. Set your signing key by running `git config --global user.signingkey <ID>`, replacing `<ID>` with the certificate ID. +1. Configure X.509 with this command: + + ```shell + git config --global gpg.x509.program smimesign + git config --global gpg.format x509 + ``` + +### Sign and verify commits + +After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you +can sign your commits: + +1. When you create a Git commit, add the `-S` flag: + + ```shell + git commit -S -m "feat: x509 signed commits" + ``` + +1. Push to GitLab, and check that your commits are verified with the `--show-signature` flag: + + ```shell + git log --show-signature + ``` + +1. *If you don't want to type the `-S` flag every time you commit,* run this command + for Git to sign your commits every time: + + ```shell + git config --global commit.gpgsign true + ``` + +### Sign and verify tags + +After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you +can start signing your tags: + +1. When you create a Git tag, add the `-s` flag: + + ```shell + git tag -s v1.1.1 -m "My signed tag" + ``` + +1. Push to GitLab and verify your tags are signed with this command: + + ```shell + git tag --verify v1.1.1 + ``` + +1. *If you don't want to type the `-s` flag every time you tag,* run this command + for Git to sign your tags each time: + + ```shell + git config --global tag.gpgsign true + ``` + +## Related topics + +- [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) + +## Troubleshooting + +For committers without administrator access, review the list of +[verification problems with signed commits](../signed_commits/index.md#fix-verification-problems-with-signed-commits) +for possible fixes. The other troubleshooting suggestions on this page require +administrator access. + +### Re-verify commits + +GitLab stores the status of any checked commits in the database. You can use a +Rake task to [check the status of any previously checked commits](../../../../raketasks/x509_signatures.md). + +After you make any changes, run this command: + +```shell +sudo gitlab-rake gitlab:x509:update_signatures +``` + +### Main verification checks + +The code performs +[these key checks](https://gitlab.com/gitlab-org/gitlab/-/blob/v14.1.0-ee/lib/gitlab/x509/signature.rb#L33), +which all must return `verified`: + +- `x509_certificate.nil?` should be false. +- `x509_certificate.revoked?` should be false. +- `verified_signature` should be true. +- `user.nil?` should be false. +- `user.verified_emails.include?(@email)` should be true. +- `certificate_email == @email` should be true. + +To investigate why a commit shows as `Unverified`: + +1. [Start a Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session): + + ```shell + sudo gitlab-rails console + ``` + +1. Identify the project (either by path or ID) and full commit SHA that you're investigating. + Use this information to create `signature` to run other checks against: + + ```ruby + project = Project.find_by_full_path('group/subgroup/project') + project = Project.find_by_id('121') + commit = project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653') + signedcommit=Gitlab::X509::Commit.new(commit) + signature=Gitlab::X509::Signature.new(signedcommit.signature_text, signedcommit.signed_text, commit.committer_email, commit.created_at) + ``` + + If you make changes to address issues identified running through the checks, restart the + Rails console and run though the checks again from the start. + +1. Check the certificate on the commit: + + ```ruby + signature.x509_certificate.nil? + signature.x509_certificate.revoked? + ``` + + Both checks should return `false`: + + ```ruby + > signature.x509_certificate.nil? + => false + > signature.x509_certificate.revoked? + => false + ``` + + A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332503) causes + these checks to fail with `Validation failed: Subject key identifier is invalid`. + +1. Run a cryptographic check on the signature. The code must return `true`: + + ```ruby + signature.verified_signature + ``` + + If it returns `false` then [investigate this check further](#cryptographic-verification-checks). + +1. Confirm the email addresses match on the commit and the signature: + + - The Rails console displays the email addresses being compared. + - The final command must return `true`: + + ```ruby + sigemail=signature.__send__:certificate_email + commitemail=commit.committer_email + sigemail == commitemail + ``` + + A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336677) exists: + only the first email in the `Subject Alternative Name` list is compared. To + display the `Subject Alternative Name` list, run: + + ```ruby + signature.__send__ :get_certificate_extension,'subjectAltName' + ``` + + If the developer's email address is not the first one in the list, this check + fails, and the commit is marked `unverified`. + +1. The email address on the commit must be associated with an account in GitLab. + This check should return `false`: + + ```ruby + signature.user.nil? + ``` + +1. Check the email address is associated with a user in GitLab. This check should + return a user, such as `#<User id:1234 @user_handle>`: + + ```ruby + User.find_by_any_email(commit.committer_email) + ``` + + If it returns `nil`, the email address is not associated with a user, and the check fails. + +1. Confirm the developer's email address is verified. This check must return true: + + ```ruby + signature.user.verified_emails.include?(commit.committer_email) + ``` + + If the previous check returned `nil`, this command displays an error: + + ```plaintext + NoMethodError (undefined method `verified_emails' for nil:NilClass) + ``` + +1. The verification status is stored in the database. To display the database record: + + ```ruby + pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil + ``` + + If all the previous checks returned the correct values: + + - `verification_status: "unverified"` indicates the database record needs + updating. [Use the Rake task](#re-verify-commits). + + - `[]` indicates the database doesn't have a record yet. Locate the commit + in GitLab to check the signature and store the result. + +#### Cryptographic verification checks + +If GitLab determines that `verified_signature` is `false`, investigate the reason +in the Rails console. These checks require `signature` to exist. Refer to the `signature` +step of the previous [main verification checks](#main-verification-checks). + +1. Check the signature, without checking the issuer, returns `true`: + + ```ruby + signature.__send__ :valid_signature? + ``` + +1. Check the signing time and date. This check must return `true`: + + ```ruby + signature.__send__ :valid_signing_time? + ``` + + - The code allows for code signing certificates to expire. + - A commit must be signed during the validity period of the certificate, + and at or after the commit's datestamp. Display the commit time and + certificate details including `not_before`, `not_after` with: + + ```ruby + commit.created_at + pp signature.__send__ :cert; nil + ``` + +1. Check the signature, including that TLS trust can be established. This check must return `true`: + + ```ruby + signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text)) + ``` + + 1. If this fails, add the missing certificates required to establish trust + [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). + + 1. After adding more certificates, (if these troubleshooting steps then pass) + run the Rake task to [re-verify commits](#re-verify-commits). + + 1. Display the certificates, including in the signature: + + ```ruby + pp signature.__send__(:p7).certificates ; nil + ``` + +Ensure any additional intermediate certificates and the root certificate are added +to the certificate store. For consistency with how certificate chains are built on +web servers: + +- Git clients that are signing commits should include the certificate + and all intermediate certificates in the signature. +- The GitLab certificate store should only contain the root. + +If you remove a root certificate from the GitLab +trust store, such as when it expires, commit signatures which chain back to that +root display as `unverified`. diff --git a/doc/user/project/repository/ssh_signed_commits/index.md b/doc/user/project/repository/ssh_signed_commits/index.md index d8d798ac651..89e3d811dba 100644 --- a/doc/user/project/repository/ssh_signed_commits/index.md +++ b/doc/user/project/repository/ssh_signed_commits/index.md @@ -1,181 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../signed_commits/ssh.md' +remove_date: '2023-12-01' --- -# Sign commits with SSH keys **(FREE ALL)** +This document was moved to [another location](../signed_commits/ssh.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343879) in GitLab 15.7 [with a flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. Enabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/384202) in GitLab 15.8. Feature flag `ssh_commit_signatures` removed. - -Use SSH keys to sign Git commits in the same manner as -[GPG signed commits](../gpg_signed_commits/index.md). When you sign commits -with SSH keys, GitLab uses the SSH public keys associated with your -GitLab account to cryptographically verify the commit signature. -If successful, GitLab displays a **Verified** label on the commit. - -You may use the same SSH keys for `git+ssh` authentication to GitLab -and signing commit signatures as long as their usage type is **Authentication & Signing**. -It can be verified on the page for [adding an SSH key to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account). - -For more information about managing the SSH keys associated with your GitLab account, see -[Use SSH keys to communicate with GitLab](../../../ssh.md). - -## Configure Git to sign commits with your SSH key - -After you [create an SSH key](../../../ssh.md#generate-an-ssh-key-pair) and -[add it to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) -or [generate it using a password manager](../../../ssh.md#generate-an-ssh-key-pair-with-a-password-manager), -configure Git to begin using the key. - -Prerequisites: - -- Git 2.34.0 or newer. -- OpenSSH 8.0 or newer. - - NOTE: - OpenSSH 8.7 has broken signing functionality. If you are on OpenSSH 8.7, upgrade to OpenSSH 8.8. - -- A SSH key with the usage type of either **Authentication & Signing** or **Signing**. - The SSH key must be one of these types: - - [ED25519](../../../ssh.md#ed25519-ssh-keys) (recommended) - - [RSA](../../../ssh.md#rsa-ssh-keys) - -To configure Git to use your key: - -1. Configure Git to use SSH for commit signing: - - ```shell - git config --global gpg.format ssh - ``` - -1. Specify which SSH key should be used as the signing key, changing the filename - (here, `~/.ssh/examplekey`) to the location of your key. The filename may - differ, depending on how you generated your key: - - ```shell - git config --global user.signingkey ~/.ssh/examplekey - ``` - -## Sign commits with your SSH key - -Prerequisites: - -- You've [created an SSH key](../../../ssh.md#generate-an-ssh-key-pair). -- You've [added the key](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) to your GitLab account. -- You've [configured Git to sign commits](#configure-git-to-sign-commits-with-your-ssh-key) with your SSH key. - -To sign a commit: - -1. Use the `-S` flag when signing your commits: - - ```shell - git commit -S -m "My commit msg" - ``` - -1. Optional. If you don't want to type the `-S` flag every time you commit, tell - Git to sign your commits automatically: - - ```shell - git config --global commit.gpgsign true - ``` - -1. If your SSH key is protected, Git prompts you to enter your passphrase. -1. Push to GitLab. -1. Check that your commits [are verified](#verify-commits). - Signature verification uses the `allowed_signers` file to associate emails and SSH keys. - For help configuring this file, read [Verify commits locally](#verify-commits-locally). - -## Verify commits - -You can review commits for a merge request, or for an entire project, to confirm -they are signed: - -1. To review commits for a project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Commits**. -1. To review commits for a merge request: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. On the left sidebar, select **Merge requests**, then select your merge request. - 1. Select **Commits**. -1. Identify the commit you want to review. Signed commits show either a **Verified** - or **Unverified** badge, depending on the verification status of the signature. - Unsigned commits do not display a badge. -1. To display the signature details for a commit, select **Verified**. GitLab shows - the SSH key's fingerprint. - -## Verify commits locally - -To verify commits locally, create an -[allowed signers file](https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#ALLOWED_SIGNERS) -for Git to associate SSH public keys with users: - -1. Create an allowed signers file: - - ```shell - touch allowed_signers - ``` - -1. Configure the `allowed_signers` file in Git: - - ```shell - git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers" - ``` - -1. Add your entry to the allowed signers file. Use this command to add your - email address and public SSH key to the `allowed_signers` file. Replace `<MY_KEY>` - with the name of your key, and `~/.ssh/allowed_signers` - with the location of your project's `allowed_signers` file: - - ```shell - # Modify this line to meet your needs. - # Declaring the `git` namespace helps prevent cross-protocol attacks. - echo "$(git config --get user.email) namespaces=\"git\" $(cat ~/.ssh/<MY_KEY>.pub)" >> ~/.ssh/allowed_signers - ``` - - The resulting entry in the `allowed_signers` file contains your email address, key type, - and key contents, like this: - - ```plaintext - example@gitlab.com namespaces="git" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAmaTS47vRmsKyLyK1jlIFJn/i8wdGQ3J49LYyIYJ2hv - ``` - -1. Repeat the previous step for each user who you want to verify signatures for. - Consider checking this file in to your Git repository if you want to locally - verify signatures for many different contributors. - -1. Use `git log --show-signature` to view the signature status for the commits: - - ```shell - $ git log --show-signature - - commit e2406b6cd8ebe146835ceab67ff4a5a116e09154 (HEAD -> main, origin/main, origin/HEAD) - Good "git" signature for johndoe@example.com with ED25519 key SHA256:Ar44iySGgxic+U6Dph4Z9Rp+KDaix5SFGFawovZLAcc - Author: John Doe <johndoe@example.com> - Date: Tue Nov 29 06:54:15 2022 -0600 - - SSH signed commit - ``` - -## Revoke an SSH key for signing commits - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108344) in GitLab 15.9. - -If an SSH key becomes compromised, revoke it. Revoking a key changes both future and past commits: - -- Past commits signed by this key are marked as unverified. -- Future commits signed by this key are marked as unverified. - -To revoke an SSH key: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. On the left sidebar, select (**{key}**) **SSH Keys**. -1. Select **Revoke** next to the SSH key you want to delete. - -## Related topics - -- [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) -- [Sign commits with GPG](../gpg_signed_commits/index.md) -- [Commits API](../../../../api/commits.md) +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/tags/index.md b/doc/user/project/repository/tags/index.md index 5a01d6f2085..765f5539244 100644 --- a/doc/user/project/repository/tags/index.md +++ b/doc/user/project/repository/tags/index.md @@ -44,14 +44,14 @@ In the GitLab UI, each tag displays: To view all existing tags for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Tags**. ## View tagged commits in the commits list > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18795) in GitLab 15.10. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Commits**. 1. Commits with a tag are labeled with a tag icon (**{tag}**) and the name of the tag. This example shows a commit tagged `v1.26.0`: @@ -88,7 +88,7 @@ To create either a lightweight or annotated tag from the command line, and push To create a tag from the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Tags**. 1. Select **New tag**. 1. Provide a **Tag name**. diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 121a7b41f54..3acc62186a3 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -25,7 +25,7 @@ for any change you commit through the Web Editor. To create a text file in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New file**. @@ -35,7 +35,7 @@ To create a text file in the Web Editor: ### Create a file from a template -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. Next to the project name, select the plus icon (**{plus}**) to display a dropdown list, then select **New file** from the list. @@ -56,7 +56,7 @@ To create a text file in the Web Editor: To edit a text file in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Go to your file. 1. In the upper right, select **Edit > Edit single file**. @@ -105,7 +105,7 @@ To upload a binary file in the Web Editor: <!-- This list is duplicated at doc/gitlab-basics/add-file.md#from-the-ui --> <!-- For why we duplicated the info, see https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111072#note_1267429478 --> -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **Upload file**. 1. Complete the fields. To create a merge request with the uploaded file, ensure the **Start a new merge request with these changes** toggle is turned on. @@ -115,7 +115,7 @@ To upload a binary file in the Web Editor: To create a directory in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New directory**. 1. Complete the fields. To create a merge request with the new directory, ensure the **Start a new merge request with these changes** toggle is turned on. @@ -125,7 +125,7 @@ To create a directory in the Web Editor: To create a [branch](branches/index.md) in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New branch**. 1. Complete the fields. @@ -136,7 +136,7 @@ To create a [branch](branches/index.md) in the Web Editor: You can create [tags](tags/index.md) to mark milestones such as production releases and release candidates. To create a tag in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New tag**. 1. Complete the fields. diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index 20860718b43..ae418581820 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -1,366 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../signed_commits/x509.md' +remove_date: '2023-12-01' --- -# Sign commits and tags with X.509 certificates **(FREE ALL)** +This document was moved to [another location](../signed_commits/x509.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17773) in GitLab 12.8. - -[X.509](https://en.wikipedia.org/wiki/X.509) is a standard format for public key -certificates issued by a public or private Public Key Infrastructure (PKI). -Personal X.509 certificates are used for authentication or signing purposes -such as S/MIME (Secure/Multipurpose Internet Mail Extensions). -However, Git also supports signing of commits and tags with X.509 certificates in a -similar way as with [GPG (GnuPG, or GNU Privacy Guard)](../gpg_signed_commits/index.md). -The main difference is the way GitLab determines whether or not the developer's signature is trusted: - -- For X.509, a root certificate authority is added to the GitLab trust store. - (A trust store is a repository of trusted security certificates.) Combined with - any required intermediate certificates in the signature, the developer's certificate - can be chained back to a trusted root certificate. -- For GPG, developers [add their GPG key](../gpg_signed_commits/index.md#add-a-gpg-key-to-your-account) - to their account. - -GitLab uses its own certificate store and therefore defines the -[trust chain](https://www.ssl.com/faqs/what-is-a-certificate-authority/). -For a commit or tag to be *verified* by GitLab: - -- The signing certificate email must match a verified email address in GitLab. -- The GitLab instance must be able to establish a full trust chain - from the certificate in the signature to a trusted certificate in the GitLab certificate store. - This chain may include intermediate certificates supplied in the signature. You may - need to add certificates, such as Certificate Authority root certificates, - [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). -- The signing time must be in the time range of the - [certificate validity](https://www.rfc-editor.org/rfc/rfc5280.html#section-4.1.2.5), - which is usually up to three years. -- The signing time is equal to, or later than, the commit time. - -If a commit's status has already been determined and stored in the database, -use the Rake task [to re-check the status](../../../../raketasks/x509_signatures.md). -Refer to the [Troubleshooting section](#troubleshooting). -GitLab checks certificate revocation lists on a daily basis with a background worker. - -## Limitations - -- Self-signed certificates without `authorityKeyIdentifier`, - `subjectKeyIdentifier`, and `crlDistributionPoints` are not supported. We - recommend using certificates from a PKI that are in line with - [RFC 5280](https://www.rfc-editor.org/rfc/rfc5280). -- If you have more than one email in the Subject Alternative Name list in - your signing certificate, - [only the first one is used to verify commits](https://gitlab.com/gitlab-org/gitlab/-/issues/336677). -- The `X509v3 Subject Key Identifier` (SKI) in the issuer certificate and the - signing certificate - [must be 40 characters long](https://gitlab.com/gitlab-org/gitlab/-/issues/332503). - If your SKI is shorter, commits don't show as verified in GitLab, and - short subject key identifiers may also - [cause errors when accessing the project](https://gitlab.com/gitlab-org/gitlab/-/issues/332464), - such as 'An error occurred while loading commit signatures' and - `HTTP 422 Unprocessable Entity` errors. - -## Configure for signed commits - -To sign your commits, tags, or both, you must: - -1. [Obtain an X.509 key pair](#obtain-an-x509-key-pair). -1. [Associate your X.509 certificate with Git](#associate-your-x509-certificate-with-git). -1. [Sign and verify commits](#sign-and-verify-commits). -1. [Sign and verify tags](#sign-and-verify-tags). - -### Obtain an X.509 key pair - -If your organization has Public Key Infrastructure (PKI), that PKI provides -an S/MIME key. If you do not have an S/MIME key pair from a PKI, you can either -create your own self-signed pair, or purchase a pair. - -### Associate your X.509 certificate with Git - -To take advantage of X.509 signing, you need Git 2.19.0 or later. You can -check your Git version with the command `git --version`. - -If you have the correct version, you can proceed to configure Git. - -### Linux - -Configure Git to use your key for signing: - -```shell -signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) -git config --global user.signingkey $signingkey -git config --global gpg.format x509 -``` - -#### Windows and macOS - -To configure Windows or macOS: - -1. Install [S/MIME Sign](https://github.com/github/smimesign) by either: - - Downloading the installer. - - Running `brew install smimesign` on macOS. -1. Get the ID of your certificate by running `smimesign --list-keys`. -1. Set your signing key by running `git config --global user.signingkey <ID>`, replacing `<ID>` with the certificate ID. -1. Configure X.509 with this command: - - ```shell - git config --global gpg.x509.program smimesign - git config --global gpg.format x509 - ``` - -### Sign and verify commits - -After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you -can sign your commits: - -1. When you create a Git commit, add the `-S` flag: - - ```shell - git commit -S -m "feat: x509 signed commits" - ``` - -1. Push to GitLab, and check that your commits are verified with the `--show-signature` flag: - - ```shell - git log --show-signature - ``` - -1. *If you don't want to type the `-S` flag every time you commit,* run this command - for Git to sign your commits every time: - - ```shell - git config --global commit.gpgsign true - ``` - -### Sign and verify tags - -After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you -can start signing your tags: - -1. When you create a Git tag, add the `-s` flag: - - ```shell - git tag -s v1.1.1 -m "My signed tag" - ``` - -1. Push to GitLab and verify your tags are signed with this command: - - ```shell - git tag --verify v1.1.1 - ``` - -1. *If you don't want to type the `-s` flag every time you tag,* run this command - for Git to sign your tags each time: - - ```shell - git config --global tag.gpgsign true - ``` - -## Related topics - -- [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) -- [Sign commits with GPG](../gpg_signed_commits/index.md) -- [Sign commits with SSH keys](../ssh_signed_commits/index.md) - -## Troubleshooting - -For committers without administrator access, review the list of -[verification problems with signed commits](../gpg_signed_commits/index.md#fix-verification-problems-with-signed-commits) -for possible fixes. The other troubleshooting suggestions on this page require -administrator access. - -### Re-verify commits - -GitLab stores the status of any checked commits in the database. You can use a -Rake task to [check the status of any previously checked commits](../../../../raketasks/x509_signatures.md). - -After you make any changes, run this command: - -```shell -sudo gitlab-rake gitlab:x509:update_signatures -``` - -### Main verification checks - -The code performs -[these key checks](https://gitlab.com/gitlab-org/gitlab/-/blob/v14.1.0-ee/lib/gitlab/x509/signature.rb#L33), -which all must return `verified`: - -- `x509_certificate.nil?` should be false. -- `x509_certificate.revoked?` should be false. -- `verified_signature` should be true. -- `user.nil?` should be false. -- `user.verified_emails.include?(@email)` should be true. -- `certificate_email == @email` should be true. - -To investigate why a commit shows as `Unverified`: - -1. [Start a Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session): - - ```shell - sudo gitlab-rails console - ``` - -1. Identify the project (either by path or ID) and full commit SHA that you're investigating. - Use this information to create `signature` to run other checks against: - - ```ruby - project = Project.find_by_full_path('group/subgroup/project') - project = Project.find_by_id('121') - commit = project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653') - signedcommit=Gitlab::X509::Commit.new(commit) - signature=Gitlab::X509::Signature.new(signedcommit.signature_text, signedcommit.signed_text, commit.committer_email, commit.created_at) - ``` - - If you make changes to address issues identified running through the checks, restart the - Rails console and run though the checks again from the start. - -1. Check the certificate on the commit: - - ```ruby - signature.x509_certificate.nil? - signature.x509_certificate.revoked? - ``` - - Both checks should return `false`: - - ```ruby - > signature.x509_certificate.nil? - => false - > signature.x509_certificate.revoked? - => false - ``` - - A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332503) causes - these checks to fail with `Validation failed: Subject key identifier is invalid`. - -1. Run a cryptographic check on the signature. The code must return `true`: - - ```ruby - signature.verified_signature - ``` - - If it returns `false` then [investigate this check further](#cryptographic-verification-checks). - -1. Confirm the email addresses match on the commit and the signature: - - - The Rails console displays the email addresses being compared. - - The final command must return `true`: - - ```ruby - sigemail=signature.__send__:certificate_email - commitemail=commit.committer_email - sigemail == commitemail - ``` - - A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336677) exists: - only the first email in the `Subject Alternative Name` list is compared. To - display the `Subject Alternative Name` list, run: - - ```ruby - signature.__send__ :get_certificate_extension,'subjectAltName' - ``` - - If the developer's email address is not the first one in the list, this check - fails, and the commit is marked `unverified`. - -1. The email address on the commit must be associated with an account in GitLab. - This check should return `false`: - - ```ruby - signature.user.nil? - ``` - -1. Check the email address is associated with a user in GitLab. This check should - return a user, such as `#<User id:1234 @user_handle>`: - - ```ruby - User.find_by_any_email(commit.committer_email) - ``` - - If it returns `nil`, the email address is not associated with a user, and the check fails. - -1. Confirm the developer's email address is verified. This check must return true: - - ```ruby - signature.user.verified_emails.include?(commit.committer_email) - ``` - - If the previous check returned `nil`, this command displays an error: - - ```plaintext - NoMethodError (undefined method `verified_emails' for nil:NilClass) - ``` - -1. The verification status is stored in the database. To display the database record: - - ```ruby - pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil - ``` - - If all the previous checks returned the correct values: - - - `verification_status: "unverified"` indicates the database record needs - updating. [Use the Rake task](#re-verify-commits). - - - `[]` indicates the database doesn't have a record yet. Locate the commit - in GitLab to check the signature and store the result. - -#### Cryptographic verification checks - -If GitLab determines that `verified_signature` is `false`, investigate the reason -in the Rails console. These checks require `signature` to exist. Refer to the `signature` -step of the previous [main verification checks](#main-verification-checks). - -1. Check the signature, without checking the issuer, returns `true`: - - ```ruby - signature.__send__ :valid_signature? - ``` - -1. Check the signing time and date. This check must return `true`: - - ```ruby - signature.__send__ :valid_signing_time? - ``` - - - The code allows for code signing certificates to expire. - - A commit must be signed during the validity period of the certificate, - and at or after the commit's datestamp. Display the commit time and - certificate details including `not_before`, `not_after` with: - - ```ruby - commit.created_at - pp signature.__send__ :cert; nil - ``` - -1. Check the signature, including that TLS trust can be established. This check must return `true`: - - ```ruby - signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text)) - ``` - - 1. If this fails, add the missing certificates required to establish trust - [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). - - 1. After adding more certificates, (if these troubleshooting steps then pass) - run the Rake task to [re-verify commits](#re-verify-commits). - - 1. Display the certificates, including in the signature: - - ```ruby - pp signature.__send__(:p7).certificates ; nil - ``` - -Ensure any additional intermediate certificates and the root certificate are added -to the certificate store. For consistency with how certificate chains are built on -web servers: - -- Git clients that are signing commits should include the certificate - and all intermediate certificates in the signature. -- The GitLab certificate store should only contain the root. - -If you remove a root certificate from the GitLab -trust store, such as when it expires, commit signatures which chain back to that -root display as `unverified`. +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 1f79982bb1f..d99729e3c1b 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -137,7 +137,7 @@ You can also sort the requirements list by: ## Allow requirements to be satisfied from a CI job > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/215514) ability to specify individual requirements and their statuses in GitLab 13.2. +> - Ability to specify individual requirements and their statuses [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215514) in GitLab 13.2. GitLab supports [requirements test reports](../../../ci/yaml/artifacts_reports.md#artifactsreportsrequirements) now. You can add a job to your CI pipeline that, when triggered, marks all existing diff --git a/doc/user/project/service_desk/configure.md b/doc/user/project/service_desk/configure.md new file mode 100644 index 00000000000..f8f4ab44e5a --- /dev/null +++ b/doc/user/project/service_desk/configure.md @@ -0,0 +1,873 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Configure Service Desk **(FREE ALL)** + +By default, Service Desk is active in new projects. +If it's not active, you can do it in the project's settings. + +Prerequisites: + +- You must have at least the Maintainer role for the project. +- On GitLab self-managed, you must [set up incoming email](../../../administration/incoming_email.md#set-it-up) + for the GitLab instance. You should use + [email sub-addressing](../../../administration/incoming_email.md#email-sub-addressing), + but you can also use [catch-all mailboxes](../../../administration/incoming_email.md#catch-all-mailbox). + To do this, you must have administrator access. +- You must have enabled [issue](../settings/index.md#configure-project-features-and-permissions) + tracker for the project. + +To enable Service Desk in your project: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. Turn on the **Activate Service Desk** toggle. +1. Optional. Complete the fields. + - [Add a suffix](#configure-a-suffix-for-service-desk-alias-email) to your Service Desk email address. + - If the list below **Template to append to all Service Desk issues** is empty, create a + [description template](../description_templates.md) in your repository. +1. Select **Save changes**. + +Service Desk is now enabled for this project. +If anyone sends an email to the address available below **Email address to use for Service Desk**, +GitLab creates a confidential issue with the email's content. + +## Improve your project's security + +To improve your Service Desk project's security, you should: + +- Put the Service Desk email address behind an alias on your email system so you can change it later. +- [Enable Akismet](../../../integration/akismet.md) on your GitLab instance to add spam checking to this service. + Unblocked email spam can result in many spam issues being created. + +## Customize emails sent to the requester + +> - Moved from GitLab Premium to GitLab Free in 13.2. +> - `UNSUBSCRIBE_URL`, `SYSTEM_HEADER`, `SYSTEM_FOOTER`, and `ADDITIONAL_TEXT` placeholders [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285512) in GitLab 15.9. +> - `%{ISSUE_DESCRIPTION}` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223751) in GitLab 16.0. +> - `%{ISSUE_URL}` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/408793) in GitLab 16.1. + +An email is sent to the requester when: + +- A requester submits a new ticket by emailing Service Desk. +- A new public comment is added on a Service Desk ticket. + - Editing a comment does not trigger a new email to be sent. + +You can customize the body of these email messages with Service Desk email templates. The templates +can include [GitLab Flavored Markdown](../../markdown.md) and [some HTML tags](../../markdown.md#inline-html). +For example, you can format the emails to include a header and footer in accordance with your +organization's brand guidelines. You can also include the following placeholders to display dynamic +content specific to the Service Desk ticket or your GitLab instance. + +| Placeholder | `thank_you.md` | `new_note.md` | Description +| ---------------------- | ---------------------- | ---------------------- | ----------- +| `%{ISSUE_ID}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket IID. +| `%{ISSUE_PATH}` | **{check-circle}** Yes | **{check-circle}** Yes | Project path appended with the ticket IID. +| `%{ISSUE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | URL of the ticket. External participants can only view the ticket if the project is public and ticket is not confidential (Service Desk tickets are confidential by default). +| `%{ISSUE_DESCRIPTION}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket description. If a user has edited the description, it may contain sensitive information that is not intended to be delivered to external participants. Use this placeholder with care and ideally only if you never modify descriptions or your team is aware of the template design. +| `%{UNSUBSCRIBE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe URL. +| `%{NOTE_TEXT}` | **{dotted-circle}** No | **{check-circle}** Yes | The new comment added to the ticket by a user. Take care to include this placeholder in `new_note.md`. Otherwise, the requesters may never see the updates on their Service Desk ticket. + +### Thank you email + +When a requester submits an issue through Service Desk, GitLab sends a **thank you email**. +Without additional configuration, GitLab sends the default thank you email. + +To create a custom thank you email template: + +1. In the `.gitlab/service_desk_templates/` directory of your repository, create a file named `thank_you.md`. +1. Populate the Markdown file with text, [GitLab Flavored Markdown](../../markdown.md), + [some selected HTML tags](../../markdown.md#inline-html), and placeholders to customize the reply + to Service Desk requesters. + +### New note email + +When a Service Desk ticket has a new public comment, GitLab sends a **new note email**. +Without additional configuration, GitLab sends the content of the comment. + +To keep your emails on brand, you can create a custom new note email template. To do so: + +1. In the `.gitlab/service_desk_templates/` directory in your repository, create a file named `new_note.md`. +1. Populate the Markdown file with text, [GitLab Flavored Markdown](../../markdown.md), + [some selected HTML tags](../../markdown.md#inline-html), and placeholders to customize the new note + email. Be sure to include the `%{NOTE_TEXT}` in the template to make sure the email recipient can + read the contents of the comment. + +### Instance-level email header, footer, and additional text **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344819) in GitLab 15.9. + +Instance administrators can add a header, footer or additional text to the GitLab instance and apply +them to all emails sent from GitLab. If you're using a custom `thank_you.md` or `new_note.md`, to include +this content, add `%{SYSTEM_HEADER}`, `%{SYSTEM_FOOTER}`, or `%{ADDITIONAL_TEXT}` to your templates. + +For more information, see [System header and footer messages](../../../administration/appearance.md#system-header-and-footer-messages) and [custom additional text](../../../administration/settings/email.md#custom-additional-text). + +## Use a custom template for Service Desk tickets + +You can select one [description template](../description_templates.md#create-an-issue-template) +**per project** to be appended to every new Service Desk ticket's description. + +You can set description templates at various levels: + +- The entire [instance](../description_templates.md#set-instance-level-description-templates). +- A specific [group or subgroup](../description_templates.md#set-group-level-description-templates). +- A specific [project](../description_templates.md#set-a-default-template-for-merge-requests-and-issues). + +The templates are inherited. For example, in a project, you can also access templates set for the instance, or the project's parent groups. + +Prerequisite: + +- You must have [created a description template](../description_templates.md#create-an-issue-template). + +To use a custom description template with Service Desk: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. From the dropdown list **Template to append to all Service Desk issues**, search or select your template. + +## Support Bot user + +Behind the scenes, Service Desk works by the special Support Bot user creating issues. +This user isn't a [billable user](../../../subscriptions/self_managed/index.md#billable-users), +so it does not count toward the license limit count. + +In GitLab 16.0 and earlier, comments generated from Service Desk emails show `GitLab Support Bot` +as the author. In [GitLab 16.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/226995), +these comments show the email of the user who sent the email. +This feature only applies to comments made in GitLab 16.1 and later. + +### Change the Support Bot's display name + +You can change the display name of the Support Bot user. Emails sent from Service Desk have +this name in the `From` header. The default display name is `GitLab Support Bot`. + +To edit the custom email display name: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. Below **Email display name**, enter a new name. +1. Select **Save changes**. + +## Custom email address **(BETA)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329990) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `service_desk_custom_email`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/387003) in GitLab 16.4. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per project or for +your entire instance, an administrator can [disable the feature flag](../../../administration/feature_flags.md) +named `service_desk_custom_email`. On GitLab.com, this feature is available. + +Configure a custom email address to show as the sender of your support communication. +Maintain brand identity and instill confidence among support requesters with a domain they recognize. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [a short showcase video](https://youtu.be/_moD5U3xcQs). + +This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). +A Beta feature is not production-ready, but is unlikely to change drastically +before it's released. We encourage users to try Beta features and provide feedback +in [the feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/416637). + +### Prerequisites + +You can use one custom email address for Service Desk per project and it must be unique across the instance. + +The custom email address you want to use must meet all of the following requirements: + +- You can set up email forwarding. +- Forwarded emails preserve the original `From` header. +- Your service provider must support sub-addressing. An email address consists of a local part (everything before `@`) and a + domain part. + + With email sub-addressing you can create unique variations of an email address by adding a `+` symbol followed + by any text to the local part. Given the email address `support@example.com`, check whether sub-addressing is supported by + sending an email to `support+1@example.com`. This email should appear in your mailbox. +- You have SMTP credentials (ideally, you should use an app password). +- You must have at least the Maintainer role for the project. +- Service Desk must be configured for the project. + +### Configure a custom email address + +Configure and verify a custom email address when you want to send Service Desk emails using your own email address. + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk** and find the **Custom email** settings. +1. Note the presented Service Desk address of this project, and with your email provider + (for example, Gmail), set up email forwarding from the custom email address to the + Service Desk address. +1. Back in GitLab, complete the fields. **SMTP host** must be resolvable from the network of your GitLab instance (on GitLab self-managed) + or the public internet (on GitLab SaaS). +1. Select **Save & test settings**. + +The configuration has been saved and the verification of the custom email address is triggered. + +#### Verification + +1. After completing the configuration, all project owners and the administrator that saved the custom email configuration receive a notification email. +1. A verification email is sent using the provided SMTP credentials to the custom email address (with a sub-addressing part). + The email contains a verification token. When email forwarding is set up correctly and all prerequisites are met, + the email is forwarded to your Service Desk address and ingested by GitLab. GitLab checks the following conditions: + 1. GitLab can send an email using the SMTP credentials. + 1. Sub-addressing is supported (with the `+verify` sub-addressing part). + 1. `From` header is preserved after forwarding. + 1. Verification token is correct. + 1. Email is received in 30 minutes. + +Typically the process takes only a few minutes. + +To cancel verification at any time or if it fails, select **Reset custom email**. +The settings page updates accordingly and reflects the current state of the verification. +The SMTP credentials are deleted and you can start the configuration again. + +On failure and success all project owners and the user who triggered the verification process receive a +notification email with the verification result. +If the verification failed, the email also contains details of the reason. + +If the verification was successful, the custom email address is ready to be used. +You can now enable sending Service Desk emails via the custom email address. + +### Enable or disable the custom email address + +After the custom email address has been verified, administrators can enable or disable sending Service Desk emails via the custom email address. + +To **enable** the custom email address: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. Turn on the **Enable custom email** toggle. + Service Desk emails to external participants are sent using the SMTP credentials. + +To **disable** the custom email address: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. Turn off the **Enable custom email** toggle. + Because you set up email forwarding, emails to your custom email address continue to be processed and + appear as Service Desk Tickets in your project. + + Service Desk emails to external participants are now sent using the GitLab instance's default outgoing + email configuration. + +### Change or remove custom email configuration + +To change the custom email configuration you must reset and remove it and configure custom email again. + +To reset the configuration at any step in the process, select **Reset custom email**. +The credentials are then removed from the database. + +### Custom email reply address + +External participants can [reply by email](../../../administration/reply_by_email.md) to Service Desk tickets. +GitLab uses an email reply address with a 32-character reply key that corresponds to the ticket. +When a custom email is configured, GitLab generates the reply address from that email. + +### Known issues + +- Some service providers don't allow SMTP connections any more. + Often you can enable them on a per user basis and create an app password. +- Microsoft Exchange doesn't preserve the `From` header, so you cannot use a custom email from the same tenant. + As a workaround: + - On GitLab SaaS, use a transport rule to forward emails from the custom email address to the Service Desk email + from GitLab SaaS. Forwarding to an email address outside the current tenant preserves the original `From` header. + - On GitLab self-managed, use a subdomain or a different domain from another service provider for the + custom email address or the GitLab instance `incoming_email` or `service_desk_email`. + +## Use an additional Service Desk alias email **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2201) in GitLab 13.0. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) in GitLab 13.8. + +You can use an additional alias email address for Service Desk on an instance level. + +To do this, you must configure +a [`service_desk_email`](#configure-service-desk-alias-email) in the instance configuration. You can also configure a +[custom suffix](#configure-a-suffix-for-service-desk-alias-email) that replaces the default `-issue-` portion on the sub-addressing part. + +### Configure Service Desk alias email + +NOTE: +On GitLab.com a custom mailbox is already configured with `contact-project+%{key}@incoming.gitlab.com` as the email address. You can still configure the +[custom suffix](#configure-a-suffix-for-service-desk-alias-email) in project settings. + +Service Desk uses the [incoming email](../../../administration/incoming_email.md) +configuration by default. However, to have a separate email address for Service Desk, +configure `service_desk_email` with a [custom suffix](#configure-a-suffix-for-service-desk-alias-email) +in project settings. + +Prerequisites: + +- The `address` must include the `+%{key}` placeholder in the `user` portion of the address, + before the `@`. The placeholder is used to identify the project where the issue should be created. +- The `service_desk_email` and `incoming_email` configurations must always use separate mailboxes + to make sure Service Desk emails are processed correctly. + +To configure a custom mailbox for Service Desk with IMAP, add the following snippets to your configuration file in full: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +NOTE: +In GitLab 15.3 and later, Service Desk uses `webhook` (internal API call) by default instead of enqueuing a Sidekiq job. +To use `webhook` on a Linux package installation running GitLab 15.3, you must generate a secret file. +For more information, see [merge request 5927](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5927). +In GitLab 15.4, reconfiguring a Linux package installation generates this secret file automatically, so no +secret file configuration setting is needed. +For more information, see [issue 1462](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1462). + +```ruby +gitlab_rails['service_desk_email_enabled'] = true +gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@gmail.com" +gitlab_rails['service_desk_email_email'] = "project_contact@gmail.com" +gitlab_rails['service_desk_email_password'] = "[REDACTED]" +gitlab_rails['service_desk_email_mailbox_name'] = "inbox" +gitlab_rails['service_desk_email_idle_timeout'] = 60 +gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" +gitlab_rails['service_desk_email_host'] = "imap.gmail.com" +gitlab_rails['service_desk_email_port'] = 993 +gitlab_rails['service_desk_email_ssl'] = true +gitlab_rails['service_desk_email_start_tls'] = false +``` + +:::TabTitle Self-compiled (source) + +```yaml +service_desk_email: + enabled: true + address: "project_contact+%{key}@example.com" + user: "project_contact@example.com" + password: "[REDACTED]" + host: "imap.gmail.com" + delivery_method: webhook + secret_file: .gitlab-mailroom-secret + port: 993 + ssl: true + start_tls: false + log_path: "log/mailroom.log" + mailbox: "inbox" + idle_timeout: 60 + expunge_deleted: true +``` + +::EndTabs + +The configuration options are the same as for configuring +[incoming email](../../../administration/incoming_email.md#set-it-up). + +#### Use encrypted credentials + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108279) in GitLab 15.9. + +Instead of having the Service Desk email credentials stored in plaintext in the configuration files, you can optionally +use an encrypted file for the incoming email credentials. + +Prerequisites: + +- To use encrypted credentials, you must first enable the + [encrypted configuration](../../../administration/encrypted_configuration.md). + +The supported configuration items for the encrypted file are: + +- `user` +- `password` + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. If initially your Service Desk configuration in `/etc/gitlab/gitlab.rb` looked like: + + ```ruby + gitlab_rails['service_desk_email_email'] = "service-desk-email@mail.example.com" + gitlab_rails['service_desk_email_password'] = "examplepassword" + ``` + +1. Edit the encrypted secret: + + ```shell + sudo gitlab-rake gitlab:service_desk_email:secret:edit EDITOR=vim + ``` + +1. Enter the unencrypted contents of the Service Desk email secret: + + ```yaml + user: 'service-desk-email@mail.example.com' + password: 'examplepassword' + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and remove the `service_desk` settings for `email` and `password`. +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +Use a Kubernetes secret to store the Service Desk email password. For more information, +read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-service-desk-emails). + +:::TabTitle Docker + +1. If initially your Service Desk configuration in `docker-compose.yml` looked like: + + ```yaml + version: "3.6" + services: + gitlab: + image: 'gitlab/gitlab-ee:latest' + restart: always + hostname: 'gitlab.example.com' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['service_desk_email_email'] = "service-desk-email@mail.example.com" + gitlab_rails['service_desk_email_password'] = "examplepassword" + ``` + +1. Get inside the container, and edit the encrypted secret: + + ```shell + sudo docker exec -t <container_name> bash + gitlab-rake gitlab:service_desk_email:secret:edit EDITOR=editor + ``` + +1. Enter the unencrypted contents of the Service Desk secret: + + ```yaml + user: 'service-desk-email@mail.example.com' + password: 'examplepassword' + ``` + +1. Edit `docker-compose.yml` and remove the `service_desk` settings for `email` and `password`. +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. If initially your Service Desk configuration in `/home/git/gitlab/config/gitlab.yml` looked like: + + ```yaml + production: + service_desk_email: + user: 'service-desk-email@mail.example.com' + password: 'examplepassword' + ``` + +1. Edit the encrypted secret: + + ```shell + bundle exec rake gitlab:service_desk_email:secret:edit EDITOR=vim RAILS_ENVIRONMENT=production + ``` + +1. Enter the unencrypted contents of the Service Desk secret: + + ```yaml + user: 'service-desk-email@mail.example.com' + password: 'examplepassword' + ``` + +1. Edit `/home/git/gitlab/config/gitlab.yml` and remove the `service_desk_email:` settings for `user` and `password`. +1. Save the file and restart GitLab and Mailroom + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs + +#### Microsoft Graph + +> - Alternative Azure deployments [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5978) in GitLab 14.9. +> - [Introduced for self-compiled (source) installs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116494) in GitLab 15.11. + +`service_desk_email` can be configured to read Microsoft Exchange Online mailboxes with the Microsoft +Graph API instead of IMAP. Set up an OAuth 2.0 application for Microsoft Graph +[the same way as for incoming email](../../../administration/incoming_email.md#microsoft-graph). + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting + the values you want: + + ```ruby + gitlab_rails['service_desk_email_enabled'] = true + gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" + gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" + gitlab_rails['service_desk_email_mailbox_name'] = "inbox" + gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" + gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' + gitlab_rails['service_desk_email_inbox_options'] = { + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } + ``` + + For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), + configure the `azure_ad_endpoint` and `graph_endpoint` settings. For example: + + ```ruby + gitlab_rails['service_desk_email_inbox_options'] = { + 'azure_ad_endpoint': 'https://login.microsoftonline.us', + 'graph_endpoint': 'https://graph.microsoft.us', + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Create the [Kubernetes Secret containing the OAuth 2.0 application client secret](https://docs.gitlab.com/charts/installation/secrets.html#microsoft-graph-client-secret-for-service-desk-emails): + + ```shell + kubectl create secret generic service-desk-email-client-secret --from-literal=secret=<YOUR-CLIENT_SECRET> + ``` + +1. Create the [Kubernetes Secret for the GitLab Service Desk email auth token](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-service-desk-email-auth-token). + Replace `<name>` with the name of the [Helm release name](https://helm.sh/docs/intro/using_helm/) for the GitLab installation: + + ```shell + kubectl create secret generic <name>-service-desk-email-auth-token --from-literal=authToken=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64) + ``` + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + appConfig: + serviceDeskEmail: + enabled: true + address: "project_contact+%{key}@example.onmicrosoft.com" + user: "project_contact@example.onmicrosoft.com" + mailbox: inbox + inboxMethod: microsoft_graph + azureAdEndpoint: https://login.microsoftonline.com + graphEndpoint: https://graph.microsoft.com + tenantId: "YOUR-TENANT-ID" + clientId: "YOUR-CLIENT-ID" + clientSecret: + secret: service-desk-email-client-secret + key: secret + deliveryMethod: webhook + authToken: + secret: <name>-service-desk-email-auth-token + key: authToken + ``` + + For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), +configure the `azureAdEndpoint` and `graphEndpoint` settings. These fields are case-sensitive: + + ```yaml + global: + appConfig: + serviceDeskEmail: + [..] + azureAdEndpoint: https://login.microsoftonline.us + graphEndpoint: https://graph.microsoft.us + [..] + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['service_desk_email_enabled'] = true + gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" + gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" + gitlab_rails['service_desk_email_mailbox_name'] = "inbox" + gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" + gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' + gitlab_rails['service_desk_email_inbox_options'] = { + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), +configure the `azure_ad_endpoint` and `graph_endpoint` settings: + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['service_desk_email_enabled'] = true + gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" + gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" + gitlab_rails['service_desk_email_mailbox_name'] = "inbox" + gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" + gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' + gitlab_rails['service_desk_email_inbox_options'] = { + 'azure_ad_endpoint': 'https://login.microsoftonline.us', + 'graph_endpoint': 'https://graph.microsoft.us', + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } + ``` + +1. Save the file and restart GitLab: + + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + service_desk_email: + enabled: true + address: "project_contact+%{key}@example.onmicrosoft.com" + user: "project_contact@example.onmicrosoft.com" + mailbox: "inbox" + delivery_method: webhook + log_path: "log/mailroom.log" + secret_file: .gitlab-mailroom-secret + inbox_method: "microsoft_graph" + inbox_options: + tenant_id: "<YOUR-TENANT-ID>" + client_id: "<YOUR-CLIENT-ID>" + client_secret: "<YOUR-CLIENT-SECRET>" + poll_interval: 60 # Optional + ``` + + For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), + configure the `azure_ad_endpoint` and `graph_endpoint` settings. For example: + + ```yaml + service_desk_email: + enabled: true + address: "project_contact+%{key}@example.onmicrosoft.com" + user: "project_contact@example.onmicrosoft.com" + mailbox: "inbox" + delivery_method: webhook + log_path: "log/mailroom.log" + secret_file: .gitlab-mailroom-secret + inbox_method: "microsoft_graph" + inbox_options: + azure_ad_endpoint: "https://login.microsoftonline.us" + graph_endpoint: "https://graph.microsoft.us" + tenant_id: "<YOUR-TENANT-ID>" + client_id: "<YOUR-CLIENT-ID>" + client_secret: "<YOUR-CLIENT-SECRET>" + poll_interval: 60 # Optional + ``` + +::EndTabs + +### Configure a suffix for Service Desk alias email + +You can set a custom suffix in your project's Service Desk settings. + +A suffix can contain only lowercase letters (`a-z`), numbers (`0-9`), or underscores (`_`). + +When configured, the custom suffix creates a new Service Desk email address, consisting of the +`service_desk_email_address` setting and a key of the format: `<project_full_path>-<custom_suffix>` + +Prerequisites: + +- You must have configured a [Service Desk alias email](#configure-service-desk-alias-email). + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Service Desk**. +1. Below **Email address suffix**, enter the suffix to use. +1. Select **Save changes**. + +For example, suppose the `mygroup/myproject` project Service Desk settings has the following configured: + +- Email address suffix is set to `support`. +- Service Desk email address is configured to `contact+%{key}@example.com`. + +The Service Desk email address for this project is: `contact+mygroup-myproject-support@example.com`. +The [incoming email](../../../administration/incoming_email.md) address still works. + +If you don't configure a custom suffix, the default project identification is used for identifying +the project. + +## Configure email ingestion in multi-node environments + +A multi-node environment is a setup where GitLab is run across multiple servers +for scalability, fault tolerance, and performance reasons. + +GitLab uses a separate process called `mail_room` to ingest new unread emails +from the `incoming_email` and `service_desk_email` mailboxes. + +### Helm chart (Kubernetes) + +The [GitLab Helm chart](https://docs.gitlab.com/charts/) is made up of multiple subcharts, and one of them is +the [Mailroom subchart](https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html). Configure the +[common settings for `incoming_email`](https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration) +and the [common settings for `service_desk_email`](https://docs.gitlab.com/charts/installation/command-line-options.html#service-desk-email-configuration). + +### Linux package (Omnibus) + +In multi-node Linux package installation environments, run `mail_room` only on one node. Run it either on a single +`rails` node (for example, `application_role`) +or completely separately. + +#### Set up all nodes + +1. Add basic configuration for `incoming_email` and `service_desk_email` on every node + to render email addresses in the web UI and in generated emails. + + Find the `incoming_email` or `service_desk_email` section in `/etc/gitlab/gitlab.rb`: + + ::Tabs + + :::TabTitle `incoming_email` + + ```ruby + gitlab_rails['incoming_email_enabled'] = true + gitlab_rails['incoming_email_address'] = "incoming+%{key}@example.com" + ``` + + :::TabTitle `service_desk_email` + + ```ruby + gitlab_rails['service_desk_email_enabled'] = true + gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.com" + ``` + + ::EndTabs + +1. GitLab offers two methods to transport emails from `mail_room` to the GitLab +application. You can configure the `delivery_method` for each email setting individually: + 1. Recommended: `webhook` (default in GitLab 15.3 and later) sends the email payload via an API POST request to your GitLab + application. It uses a shared token to authenticate. If you choose this method, + make sure the `mail_room` process can access the API endpoint and distribute the shared + token across all application nodes. + + ::Tabs + + :::TabTitle `incoming_email` + + ```ruby + gitlab_rails['incoming_email_delivery_method'] = "webhook" + + # The URL that mail_room can contact. You can also use an internal URL or IP, + # just make sure mail_room can access the GitLab API via that address. + # Do not end with "/". + gitlab_rails['incoming_email_gitlab_url'] = "https://gitlab.example.com" + + # The shared secret file that should contain a random token. Make sure it's the same on every node. + gitlab_rails['incoming_email_secret_file'] = ".gitlab_mailroom_secret" + ``` + + :::TabTitle `service_desk_email` + + ```ruby + gitlab_rails['service_desk_email_delivery_method'] = "webhook" + + # The URL that mail_room can contact. You can also use an internal URL or IP, + # just make sure mail_room can access the GitLab API via that address. + # Do not end with "/". + + gitlab_rails['service_desk_email_gitlab_url'] = "https://gitlab.example.com" + + # The shared secret file that should contain a random token. Make sure it's the same on every node. + gitlab_rails['service_desk_email_secret_file'] = ".gitlab_mailroom_secret" + ``` + + ::EndTabs + + 1. [Deprecated in GitLab 16.0 and planned for removal in 17.0)](../../../update/deprecations.md#sidekiq-delivery-method-for-incoming_email-and-service_desk_email-is-deprecated): + If you experience issues with the `webhook` setup, use `sidekiq` to deliver the email payload directly to GitLab Sidekiq using Redis. + + ::Tabs + + :::TabTitle `incoming_email` + + ```ruby + # It uses the Redis configuration to directly add Sidekiq jobs + gitlab_rails['incoming_email_delivery_method'] = "sidekiq" + ``` + + :::TabTitle `service_desk_email` + + ```ruby + # It uses the Redis configuration to directly add Sidekiq jobs + gitlab_rails['service_desk_email_delivery_method'] = "sidekiq" + ``` + + ::EndTabs + +1. Disable `mail_room` on all nodes that should not run email ingestion. For example, in `/etc/gitlab/gitlab.rb`: + + ```ruby + mailroom['enabled'] = false + ``` + +1. [Reconfigure GitLab](../../../administration/restart_gitlab.md) for the changes to take effect. + +#### Set up a single email ingestion node + +After setting up all nodes and disabling the `mail_room` process, enable `mail_room` on a single node. +This node polls the mailboxes for `incoming_email` and `service_desk_email` on a regular basis and +move new unread emails to GitLab. + +1. Choose an existing node that additionally handles email ingestion. +1. Add [full configuration and credentials](../../../administration/incoming_email.md#configuration-examples) + for `incoming_email` and `service_desk_email`. +1. Enable `mail_room` on this node. For example, in `/etc/gitlab/gitlab.rb`: + + ```ruby + mailroom['enabled'] = true + ``` + +1. [Reconfigure GitLab](../../../administration/restart_gitlab.md) on this node for the changes to take effect. diff --git a/doc/user/project/service_desk/index.md b/doc/user/project/service_desk/index.md index 208e798bd21..ab807553436 100644 --- a/doc/user/project/service_desk/index.md +++ b/doc/user/project/service_desk/index.md @@ -39,870 +39,21 @@ Meanwhile: - Your team saves time by not having to leave GitLab (or set up integrations) to follow up with your customer. -## Configure Service Desk - -By default, Service Desk is active in new projects. -If it's not active, you can do it in the project's settings. - -Prerequisites: - -- You must have at least the Maintainer role for the project. -- On GitLab self-managed, you must [set up incoming email](../../../administration/incoming_email.md#set-it-up) - for the GitLab instance. You should use - [email sub-addressing](../../../administration/incoming_email.md#email-sub-addressing), - but you can also use [catch-all mailboxes](../../../administration/incoming_email.md#catch-all-mailbox). - To do this, you must have administrator access. -- You must have enabled [issue](../settings/index.md#configure-project-visibility-features-and-permissions) - tracker for the project. - -To enable Service Desk in your project: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Service Desk**. -1. Turn on the **Activate Service Desk** toggle. -1. Optional. Complete the fields. - - [Add a suffix](#configure-a-suffix-for-service-desk-alias-email) to your Service Desk email address. - - If the list below **Template to append to all Service Desk issues** is empty, create a - [description template](../description_templates.md) in your repository. -1. Select **Save changes**. - -Service Desk is now enabled for this project. -If anyone sends an email to the address available below **Email address to use for Service Desk**, -GitLab creates a confidential issue with the email's content. - -### Improve your project's security - -To improve your Service Desk project's security, you should: - -- Put the Service Desk email address behind an alias on your email system so you can change it later. -- [Enable Akismet](../../../integration/akismet.md) on your GitLab instance to add spam checking to this service. - Unblocked email spam can result in many spam issues being created. - -### Customize emails sent to the requester - -> - Moved from GitLab Premium to GitLab Free in 13.2. -> - `UNSUBSCRIBE_URL`, `SYSTEM_HEADER`, `SYSTEM_FOOTER`, and `ADDITIONAL_TEXT` placeholders [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285512) in GitLab 15.9. -> - `%{ISSUE_DESCRIPTION}` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223751) in GitLab 16.0. -> - `%{ISSUE_URL}` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/408793) in GitLab 16.1. - -An email is sent to the requester when: - -- A requester submits a new ticket by emailing Service Desk. -- A new public comment is added on a Service Desk ticket. - - Editing a comment does not trigger a new email to be sent. - -You can customize the body of these email messages with Service Desk email templates. The templates -can include [GitLab Flavored Markdown](../../markdown.md) and [some HTML tags](../../markdown.md#inline-html). -For example, you can format the emails to include a header and footer in accordance with your -organization's brand guidelines. You can also include the following placeholders to display dynamic -content specific to the Service Desk ticket or your GitLab instance. - -| Placeholder | `thank_you.md` | `new_note.md` | Description -| ---------------------- | ---------------------- | ---------------------- | ----------- -| `%{ISSUE_ID}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket IID. -| `%{ISSUE_PATH}` | **{check-circle}** Yes | **{check-circle}** Yes | Project path appended with the ticket IID. -| `%{ISSUE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | URL of the ticket. External participants can only view the ticket if the project is public and ticket is not confidential (Service Desk tickets are confidential by default). -| `%{ISSUE_DESCRIPTION}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket description. If a user has edited the description, it may contain sensitive information that is not intended to be delivered to external participants. Use this placeholder with care and ideally only if you never modify descriptions or your team is aware of the template design. -| `%{UNSUBSCRIBE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe URL. -| `%{NOTE_TEXT}` | **{dotted-circle}** No | **{check-circle}** Yes | The new comment added to the ticket by a user. Take care to include this placeholder in `new_note.md`. Otherwise, the requesters may never see the updates on their Service Desk ticket. - -#### Thank you email - -When a requester submits an issue through Service Desk, GitLab sends a **thank you email**. -Without additional configuration, GitLab sends the default thank you email. - -To create a custom thank you email template: - -1. In the `.gitlab/service_desk_templates/` directory of your repository, create a file named `thank_you.md`. -1. Populate the Markdown file with text, [GitLab Flavored Markdown](../../markdown.md), - [some selected HTML tags](../../markdown.md#inline-html), and placeholders to customize the reply - to Service Desk requesters. - -#### New note email - -When a Service Desk ticket has a new public comment, GitLab sends a **new note email**. -Without additional configuration, GitLab sends the content of the comment. - -To keep your emails on brand, you can create a custom new note email template. To do so: - -1. In the `.gitlab/service_desk_templates/` directory in your repository, create a file named `new_note.md`. -1. Populate the Markdown file with text, [GitLab Flavored Markdown](../../markdown.md), - [some selected HTML tags](../../markdown.md#inline-html), and placeholders to customize the new note - email. Be sure to include the `%{NOTE_TEXT}` in the template to make sure the email recipient can - read the contents of the comment. - -#### Instance-level email header, footer, and additional text **(FREE SELF)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344819) in GitLab 15.9. - -Instance administrators can add a header, footer or additional text to the GitLab instance and apply -them to all emails sent from GitLab. If you're using a custom `thank_you.md` or `new_note.md`, to include -this content, add `%{SYSTEM_HEADER}`, `%{SYSTEM_FOOTER}`, or `%{ADDITIONAL_TEXT}` to your templates. - -For more information, see [System header and footer messages](../../../administration/appearance.md#system-header-and-footer-messages) and [custom additional text](../../../administration/settings/email.md#custom-additional-text). - -### Use a custom template for Service Desk tickets - -You can select one [description template](../description_templates.md#create-an-issue-template) -**per project** to be appended to every new Service Desk ticket's description. - -You can set description templates at various levels: - -- The entire [instance](../description_templates.md#set-instance-level-description-templates). -- A specific [group or subgroup](../description_templates.md#set-group-level-description-templates). -- A specific [project](../description_templates.md#set-a-default-template-for-merge-requests-and-issues). - -The templates are inherited. For example, in a project, you can also access templates set for the instance, or the project's parent groups. - -Prerequisite: - -- You must have [created a description template](../description_templates.md#create-an-issue-template). - -To use a custom description template with Service Desk: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Service Desk**. -1. From the dropdown list **Template to append to all Service Desk issues**, search or select your template. - -### Support Bot user - -Behind the scenes, Service Desk works by the special Support Bot user creating issues. -This user isn't a [billable user](../../../subscriptions/self_managed/index.md#billable-users), -so it does not count toward the license limit count. - -In GitLab 16.0 and earlier, comments generated from Service Desk emails show `GitLab Support Bot` -as the author. In [GitLab 16.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/226995), -these comments show the email of the user who sent the email. -This feature only applies to comments made in GitLab 16.1 and later. - -#### Change the Support Bot's display name - -You can change the display name of the Support Bot user. Emails sent from Service Desk have -this name in the `From` header. The default display name is `GitLab Support Bot`. - -To edit the custom email display name: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Service Desk**. -1. Below **Email display name**, enter a new name. -1. Select **Save changes**. - -### Use an additional Service Desk alias email **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2201) in GitLab 13.0. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) in GitLab 13.8. - -You can use an additional alias email address for Service Desk on an instance level. - -To do this, you must configure -a [`service_desk_email`](#configure-service-desk-alias-email) in the instance configuration. You can also configure a -[custom suffix](#configure-a-suffix-for-service-desk-alias-email) that replaces the default `-issue-` portion on the sub-addressing part. - -#### Configure Service Desk alias email - -NOTE: -On GitLab.com a custom mailbox is already configured with `contact-project+%{key}@incoming.gitlab.com` as the email address. You can still configure the -[custom suffix](#configure-a-suffix-for-service-desk-alias-email) in project settings. - -Service Desk uses the [incoming email](../../../administration/incoming_email.md) -configuration by default. However, to have a separate email address for Service Desk, -configure `service_desk_email` with a [custom suffix](#configure-a-suffix-for-service-desk-alias-email) -in project settings. - -Prerequisites: - -- The `address` must include the `+%{key}` placeholder in the `user` portion of the address, - before the `@`. The placeholder is used to identify the project where the issue should be created. -- The `service_desk_email` and `incoming_email` configurations must always use separate mailboxes - to make sure Service Desk emails are processed correctly. - -To configure a custom mailbox for Service Desk with IMAP, add the following snippets to your configuration file in full: - -::Tabs - -:::TabTitle Linux package (Omnibus) - -NOTE: -In GitLab 15.3 and later, Service Desk uses `webhook` (internal API call) by default instead of enqueuing a Sidekiq job. -To use `webhook` on a Linux package installation running GitLab 15.3, you must generate a secret file. -For more information, see [merge request 5927](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5927). -In GitLab 15.4, reconfiguring a Linux package installation generates this secret file automatically, so no -secret file configuration setting is needed. -For more information, see [issue 1462](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1462). - -```ruby -gitlab_rails['service_desk_email_enabled'] = true -gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@gmail.com" -gitlab_rails['service_desk_email_email'] = "project_contact@gmail.com" -gitlab_rails['service_desk_email_password'] = "[REDACTED]" -gitlab_rails['service_desk_email_mailbox_name'] = "inbox" -gitlab_rails['service_desk_email_idle_timeout'] = 60 -gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" -gitlab_rails['service_desk_email_host'] = "imap.gmail.com" -gitlab_rails['service_desk_email_port'] = 993 -gitlab_rails['service_desk_email_ssl'] = true -gitlab_rails['service_desk_email_start_tls'] = false -``` - -:::TabTitle Self-compiled (source) - -```yaml -service_desk_email: - enabled: true - address: "project_contact+%{key}@example.com" - user: "project_contact@example.com" - password: "[REDACTED]" - host: "imap.gmail.com" - delivery_method: webhook - secret_file: .gitlab-mailroom-secret - port: 993 - ssl: true - start_tls: false - log_path: "log/mailroom.log" - mailbox: "inbox" - idle_timeout: 60 - expunge_deleted: true -``` - -::EndTabs - -The configuration options are the same as for configuring -[incoming email](../../../administration/incoming_email.md#set-it-up). - -##### Use encrypted credentials - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108279) in GitLab 15.9. - -Instead of having the Service Desk email credentials stored in plaintext in the configuration files, you can optionally -use an encrypted file for the incoming email credentials. - -Prerequisites: - -- To use encrypted credentials, you must first enable the - [encrypted configuration](../../../administration/encrypted_configuration.md). - -The supported configuration items for the encrypted file are: - -- `user` -- `password` - -::Tabs - -:::TabTitle Linux package (Omnibus) - -1. If initially your Service Desk configuration in `/etc/gitlab/gitlab.rb` looked like: - - ```ruby - gitlab_rails['service_desk_email_email'] = "service-desk-email@mail.example.com" - gitlab_rails['service_desk_email_password'] = "examplepassword" - ``` - -1. Edit the encrypted secret: - - ```shell - sudo gitlab-rake gitlab:service_desk_email:secret:edit EDITOR=vim - ``` - -1. Enter the unencrypted contents of the Service Desk email secret: - - ```yaml - user: 'service-desk-email@mail.example.com' - password: 'examplepassword' - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and remove the `service_desk` settings for `email` and `password`. -1. Save the file and reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -:::TabTitle Helm chart (Kubernetes) - -Use a Kubernetes secret to store the Service Desk email password. For more information, -read about [Helm IMAP secrets](https://docs.gitlab.com/charts/installation/secrets.html#imap-password-for-service-desk-emails). - -:::TabTitle Docker - -1. If initially your Service Desk configuration in `docker-compose.yml` looked like: - - ```yaml - version: "3.6" - services: - gitlab: - image: 'gitlab/gitlab-ee:latest' - restart: always - hostname: 'gitlab.example.com' - environment: - GITLAB_OMNIBUS_CONFIG: | - gitlab_rails['service_desk_email_email'] = "service-desk-email@mail.example.com" - gitlab_rails['service_desk_email_password'] = "examplepassword" - ``` - -1. Get inside the container, and edit the encrypted secret: - - ```shell - sudo docker exec -t <container_name> bash - gitlab-rake gitlab:service_desk_email:secret:edit EDITOR=editor - ``` - -1. Enter the unencrypted contents of the Service Desk secret: - - ```yaml - user: 'service-desk-email@mail.example.com' - password: 'examplepassword' - ``` - -1. Edit `docker-compose.yml` and remove the `service_desk` settings for `email` and `password`. -1. Save the file and restart GitLab: - - ```shell - docker compose up -d - ``` - -:::TabTitle Self-compiled (source) - -1. If initially your Service Desk configuration in `/home/git/gitlab/config/gitlab.yml` looked like: - - ```yaml - production: - service_desk_email: - user: 'service-desk-email@mail.example.com' - password: 'examplepassword' - ``` - -1. Edit the encrypted secret: - - ```shell - bundle exec rake gitlab:service_desk_email:secret:edit EDITOR=vim RAILS_ENVIRONMENT=production - ``` - -1. Enter the unencrypted contents of the Service Desk secret: - - ```yaml - user: 'service-desk-email@mail.example.com' - password: 'examplepassword' - ``` - -1. Edit `/home/git/gitlab/config/gitlab.yml` and remove the `service_desk_email:` settings for `user` and `password`. -1. Save the file and restart GitLab and Mailroom - - ```shell - # For systems running systemd - sudo systemctl restart gitlab.target - - # For systems running SysV init - sudo service gitlab restart - ``` - -::EndTabs - -##### Microsoft Graph - -> - Alternative Azure deployments [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5978) in GitLab 14.9. -> - [Introduced for self-compiled (source) installs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116494) in GitLab 15.11. - -`service_desk_email` can be configured to read Microsoft Exchange Online mailboxes with the Microsoft -Graph API instead of IMAP. Set up an OAuth 2.0 application for Microsoft Graph -[the same way as for incoming email](../../../administration/incoming_email.md#microsoft-graph). - -::Tabs - -:::TabTitle Linux package (Omnibus) - -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting - the values you want: - - ```ruby - gitlab_rails['service_desk_email_enabled'] = true - gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" - gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" - gitlab_rails['service_desk_email_mailbox_name'] = "inbox" - gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" - gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' - gitlab_rails['service_desk_email_inbox_options'] = { - 'tenant_id': '<YOUR-TENANT-ID>', - 'client_id': '<YOUR-CLIENT-ID>', - 'client_secret': '<YOUR-CLIENT-SECRET>', - 'poll_interval': 60 # Optional - } - ``` - - For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), - configure the `azure_ad_endpoint` and `graph_endpoint` settings. For example: - - ```ruby - gitlab_rails['service_desk_email_inbox_options'] = { - 'azure_ad_endpoint': 'https://login.microsoftonline.us', - 'graph_endpoint': 'https://graph.microsoft.us', - 'tenant_id': '<YOUR-TENANT-ID>', - 'client_id': '<YOUR-CLIENT-ID>', - 'client_secret': '<YOUR-CLIENT-SECRET>', - 'poll_interval': 60 # Optional - } - ``` - -:::TabTitle Helm chart (Kubernetes) - -1. Create the [Kubernetes Secret containing the OAuth 2.0 application client secret](https://docs.gitlab.com/charts/installation/secrets.html#microsoft-graph-client-secret-for-service-desk-emails): - - ```shell - kubectl create secret generic service-desk-email-client-secret --from-literal=secret=<YOUR-CLIENT_SECRET> - ``` - -1. Create the [Kubernetes Secret for the GitLab Service Desk email auth token](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-service-desk-email-auth-token). - Replace `<name>` with the name of the [Helm release name](https://helm.sh/docs/intro/using_helm/) for the GitLab installation: - - ```shell - kubectl create secret generic <name>-service-desk-email-auth-token --from-literal=authToken=$(head -c 512 /dev/urandom | LC_CTYPE=C tr -cd 'a-zA-Z0-9' | head -c 32 | base64) - ``` - -1. Export the Helm values: - - ```shell - helm get values gitlab > gitlab_values.yaml - ``` - -1. Edit `gitlab_values.yaml`: - - ```yaml - global: - appConfig: - serviceDeskEmail: - enabled: true - address: "project_contact+%{key}@example.onmicrosoft.com" - user: "project_contact@example.onmicrosoft.com" - mailbox: inbox - inboxMethod: microsoft_graph - azureAdEndpoint: https://login.microsoftonline.com - graphEndpoint: https://graph.microsoft.com - tenantId: "YOUR-TENANT-ID" - clientId: "YOUR-CLIENT-ID" - clientSecret: - secret: service-desk-email-client-secret - key: secret - deliveryMethod: webhook - authToken: - secret: <name>-service-desk-email-auth-token - key: authToken - ``` - - For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), -configure the `azureAdEndpoint` and `graphEndpoint` settings. These fields are case-sensitive: - - ```yaml - global: - appConfig: - serviceDeskEmail: - [..] - azureAdEndpoint: https://login.microsoftonline.us - graphEndpoint: https://graph.microsoft.us - [..] - ``` - -1. Save the file and apply the new values: - - ```shell - helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab - ``` - -:::TabTitle Docker - -1. Edit `docker-compose.yml`: - - ```yaml - version: "3.6" - services: - gitlab: - environment: - GITLAB_OMNIBUS_CONFIG: | - gitlab_rails['service_desk_email_enabled'] = true - gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" - gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" - gitlab_rails['service_desk_email_mailbox_name'] = "inbox" - gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" - gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' - gitlab_rails['service_desk_email_inbox_options'] = { - 'tenant_id': '<YOUR-TENANT-ID>', - 'client_id': '<YOUR-CLIENT-ID>', - 'client_secret': '<YOUR-CLIENT-SECRET>', - 'poll_interval': 60 # Optional - } - ``` - -1. Save the file and restart GitLab: - - ```shell - docker compose up -d - ``` - -For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), -configure the `azure_ad_endpoint` and `graph_endpoint` settings: - -1. Edit `docker-compose.yml`: - - ```yaml - version: "3.6" - services: - gitlab: - environment: - GITLAB_OMNIBUS_CONFIG: | - gitlab_rails['service_desk_email_enabled'] = true - gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.onmicrosoft.com" - gitlab_rails['service_desk_email_email'] = "project_contact@example.onmicrosoft.com" - gitlab_rails['service_desk_email_mailbox_name'] = "inbox" - gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log" - gitlab_rails['service_desk_email_inbox_method'] = 'microsoft_graph' - gitlab_rails['service_desk_email_inbox_options'] = { - 'azure_ad_endpoint': 'https://login.microsoftonline.us', - 'graph_endpoint': 'https://graph.microsoft.us', - 'tenant_id': '<YOUR-TENANT-ID>', - 'client_id': '<YOUR-CLIENT-ID>', - 'client_secret': '<YOUR-CLIENT-SECRET>', - 'poll_interval': 60 # Optional - } - ``` - -1. Save the file and restart GitLab: - - ```shell - docker compose up -d - ``` - -:::TabTitle Self-compiled (source) - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - service_desk_email: - enabled: true - address: "project_contact+%{key}@example.onmicrosoft.com" - user: "project_contact@example.onmicrosoft.com" - mailbox: "inbox" - delivery_method: webhook - log_path: "log/mailroom.log" - secret_file: .gitlab-mailroom-secret - inbox_method: "microsoft_graph" - inbox_options: - tenant_id: "<YOUR-TENANT-ID>" - client_id: "<YOUR-CLIENT-ID>" - client_secret: "<YOUR-CLIENT-SECRET>" - poll_interval: 60 # Optional - ``` - - For Microsoft Cloud for US Government or [other Azure deployments](https://learn.microsoft.com/en-us/graph/deployments), - configure the `azure_ad_endpoint` and `graph_endpoint` settings. For example: - - ```yaml - service_desk_email: - enabled: true - address: "project_contact+%{key}@example.onmicrosoft.com" - user: "project_contact@example.onmicrosoft.com" - mailbox: "inbox" - delivery_method: webhook - log_path: "log/mailroom.log" - secret_file: .gitlab-mailroom-secret - inbox_method: "microsoft_graph" - inbox_options: - azure_ad_endpoint: "https://login.microsoftonline.us" - graph_endpoint: "https://graph.microsoft.us" - tenant_id: "<YOUR-TENANT-ID>" - client_id: "<YOUR-CLIENT-ID>" - client_secret: "<YOUR-CLIENT-SECRET>" - poll_interval: 60 # Optional - ``` - -::EndTabs - -#### Configure a suffix for Service Desk alias email - -You can set a custom suffix in your project's Service Desk settings. - -A suffix can contain only lowercase letters (`a-z`), numbers (`0-9`), or underscores (`_`). - -When configured, the custom suffix creates a new Service Desk email address, consisting of the -`service_desk_email_address` setting and a key of the format: `<project_full_path>-<custom_suffix>` - -Prerequisites: - -- You must have configured a [Service Desk alias email](#configure-service-desk-alias-email). - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Service Desk**. -1. Below **Email address suffix**, enter the suffix to use. -1. Select **Save changes**. - -For example, suppose the `mygroup/myproject` project Service Desk settings has the following configured: - -- Email address suffix is set to `support`. -- Service Desk email address is configured to `contact+%{key}@example.com`. - -The Service Desk email address for this project is: `contact+mygroup-myproject-support@example.com`. -The [incoming email](../../../administration/incoming_email.md) address still works. - -If you don't configure a custom suffix, the default project identification is used for identifying -the project. - -### Configure email ingestion in multi-node environments - -A multi-node environment is a setup where GitLab is run across multiple servers -for scalability, fault tolerance, and performance reasons. - -GitLab uses a separate process called `mail_room` to ingest new unread emails -from the `incoming_email` and `service_desk_email` mailboxes. - -#### Helm chart (Kubernetes) - -The [GitLab Helm chart](https://docs.gitlab.com/charts/) is made up of multiple subcharts, and one of them is -the [Mailroom subchart](https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html). Configure the -[common settings for `incoming_email`](https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration) -and the [common settings for `service_desk_email`](https://docs.gitlab.com/charts/installation/command-line-options.html#service-desk-email-configuration). - -#### Linux package (Omnibus) - -In multi-node Linux package installation environments, run `mail_room` only on one node. Run it either on a single -`rails` node (for example, `application_role`) -or completely separately. - -##### Set up all nodes - -1. Add basic configuration for `incoming_email` and `service_desk_email` on every node - to render email addresses in the web UI and in generated emails. - - Find the `incoming_email` or `service_desk_email` section in `/etc/gitlab/gitlab.rb`: - - ::Tabs - - :::TabTitle `incoming_email` - - ```ruby - gitlab_rails['incoming_email_enabled'] = true - gitlab_rails['incoming_email_address'] = "incoming+%{key}@example.com" - ``` - - :::TabTitle `service_desk_email` - - ```ruby - gitlab_rails['service_desk_email_enabled'] = true - gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@example.com" - ``` - - ::EndTabs - -1. GitLab offers two methods to transport emails from `mail_room` to the GitLab -application. You can configure the `delivery_method` for each email setting individually: - 1. Recommended: `webhook` (default in GitLab 15.3 and later) sends the email payload via an API POST request to your GitLab - application. It uses a shared token to authenticate. If you choose this method, - make sure the `mail_room` process can access the API endpoint and distribute the shared - token across all application nodes. - - ::Tabs - - :::TabTitle `incoming_email` - - ```ruby - gitlab_rails['incoming_email_delivery_method'] = "webhook" - - # The URL that mail_room can contact. You can also use an internal URL or IP, - # just make sure mail_room can access the GitLab API via that address. - # Do not end with "/". - gitlab_rails['incoming_email_gitlab_url'] = "https://gitlab.example.com" - - # The shared secret file that should contain a random token. Make sure it's the same on every node. - gitlab_rails['incoming_email_secret_file'] = ".gitlab_mailroom_secret" - ``` - - :::TabTitle `service_desk_email` - - ```ruby - gitlab_rails['service_desk_email_delivery_method'] = "webhook" - - # The URL that mail_room can contact. You can also use an internal URL or IP, - # just make sure mail_room can access the GitLab API via that address. - # Do not end with "/". - - gitlab_rails['service_desk_email_gitlab_url'] = "https://gitlab.example.com" - - # The shared secret file that should contain a random token. Make sure it's the same on every node. - gitlab_rails['service_desk_email_secret_file'] = ".gitlab_mailroom_secret" - ``` - - ::EndTabs - - 1. [Deprecated in GitLab 16.0 and planned for removal in 17.0)](../../../update/deprecations.md#sidekiq-delivery-method-for-incoming_email-and-service_desk_email-is-deprecated): - If you experience issues with the `webhook` setup, use `sidekiq` to deliver the email payload directly to GitLab Sidekiq using Redis. - - ::Tabs - - :::TabTitle `incoming_email` - - ```ruby - # It uses the Redis configuration to directly add Sidekiq jobs - gitlab_rails['incoming_email_delivery_method'] = "sidekiq" - ``` - - :::TabTitle `service_desk_email` - - ```ruby - # It uses the Redis configuration to directly add Sidekiq jobs - gitlab_rails['service_desk_email_delivery_method'] = "sidekiq" - ``` - - ::EndTabs - -1. Disable `mail_room` on all nodes that should not run email ingestion. For example, in `/etc/gitlab/gitlab.rb`: - - ```ruby - mailroom['enabled'] = false - ``` - -1. [Reconfigure GitLab](../../../administration/restart_gitlab.md) for the changes to take effect. - -##### Set up a single email ingestion node - -After setting up all nodes and disabling the `mail_room` process, enable `mail_room` on a single node. -This node polls the mailboxes for `incoming_email` and `service_desk_email` on a regular basis and -move new unread emails to GitLab. - -1. Choose an existing node that additionally handles email ingestion. -1. Add [full configuration and credentials](../../../administration/incoming_email.md#configuration-examples) - for `incoming_email` and `service_desk_email`. -1. Enable `mail_room` on this node. For example, in `/etc/gitlab/gitlab.rb`: - - ```ruby - mailroom['enabled'] = true - ``` - -1. [Reconfigure GitLab](../../../administration/restart_gitlab.md) on this node for the changes to take effect. - -## Use Service Desk - -You can use Service Desk to [create an issue](#as-an-end-user-issue-creator) or [respond to one](#as-a-responder-to-the-issue). -In these issues, you can also see our friendly neighborhood [Support Bot](#support-bot-user). - -### View Service Desk email address - -To check what the Service Desk email address is for your project: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Monitor > Service Desk**. - -The email address is available at the top of the issue list. - -### As an end user (issue creator) - -> Support for additional email headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346600) in GitLab 14.6. In earlier versions, the Service Desk email address had to be in the "To" field. - -To create a Service Desk issue, an end user does not need to know anything about -the GitLab instance. They just send an email to the address they are given, and -receive an email back confirming receipt: - - - -This also gives the end user an option to unsubscribe. - -If they don't choose to unsubscribe, then any new comments added to the issue -are sent as emails: - - - -Any responses they send via email are displayed in the issue itself. - -For information about headers used for treating email, see -[the incoming email documentation](../../../administration/incoming_email.md#accepted-headers). - -### As a responder to the issue - -For responders to the issue, everything works just like other GitLab issues. -GitLab displays a familiar-looking issue tracker where responders can see -issues created through customer support requests, and filter or interact with them. - - - -Messages from the end user are shown as coming from the special -[Support Bot user](../../../subscriptions/self_managed/index.md#billable-users). -You can read and write comments as you usually do in GitLab: - - - -- The project's visibility (private, internal, public) does not affect Service Desk. -- The path to the project, including its group or namespace, is shown in emails. - -#### View Service Desk issues - -Prerequisites: - -- You must have at least the Reporter role for the project. - -To view Service Desk issues: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Monitor > Service Desk**. - -### Email contents and formatting - -#### Special HTML formatting in HTML emails - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109811) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `service_desk_html_to_text_email_handler`. Disabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116809) in GitLab 15.11. Feature flag `service_desk_html_to_text_email_handler` removed. - -HTML emails show HTML formatting, such as: - -- Tables -- Blockquotes -- Images -- Collapsible sections - -#### Files attached to comments - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11733) in GitLab 15.8 [with a flag](../../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/386860) in GitLab 15.10. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. -On GitLab.com, this feature is available. - -If a comment contains any attachments and their total size is less than or equal to 10 MB, these -attachments are sent as part of the email. In other cases, the email contains links to the attachments. - -In GitLab 15.9 and earlier, uploads to a comment are sent as links in the email. - -## Privacy considerations - -> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108901) the minimum required role to view the creator's and participant's email in GitLab 15.9. - -Service Desk issues are [confidential](../issues/confidential_issues.md), so they are -only visible to project members. The project owner can -[make an issue public](../issues/confidential_issues.md#in-an-existing-issue). -When a Service Desk issue becomes public, the issue creator's and participants' email addresses are -visible to signed-in users with at least the Reporter role for the project. - -In GitLab 15.8 and earlier, when a Service Desk issue becomes public, the issue creator's email -address is disclosed to everyone who can view the project. - -Anyone in your project can use the Service Desk email address to create an issue in this project, **regardless -of their role** in the project. - -The unique internal email address is visible to project members at least -the Reporter role in your GitLab instance. -An external user (issue creator) cannot see the internal email address -displayed in the information note. - -### Moving a Service Desk issue - -> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/372246) in GitLab 15.7: customers continue receiving notifications when a Service Desk issue is moved. - -You can move a Service Desk issue the same way you -[move a regular issue](../issues/managing_issues.md#move-an-issue) in GitLab. - -If a Service Desk issue is moved to a different project with Service Desk enabled, -the customer who created the issue continues to receive email notifications. -Because a moved issue is first closed, then copied, the customer is considered to be a participant -in both issues. They continue to receive any notifications in the old issue and the new one. +## Related topics + +- [Configure Service Desk](configure.md) + - [Improve your project's security](configure.md#improve-your-projects-security) + - [Customize emails sent to the requester](configure.md#customize-emails-sent-to-the-requester) + - [Use a custom template for Service Desk tickets](configure.md#use-a-custom-template-for-service-desk-tickets) + - [Support Bot user](configure.md#support-bot-user) + - [Custom email address (Beta)](configure.md#custom-email-address) + - [Use an additional Service Desk alias email](configure.md#use-an-additional-service-desk-alias-email) + - [Configure email ingestion in multi-node environments](configure.md#configure-email-ingestion-in-multi-node-environments) +- [Use Service Desk](using_service_desk.md#use-service-desk) + - [As an end user (issue creator)](using_service_desk.md#as-an-end-user-issue-creator) + - [As a responder to the issue](using_service_desk.md#as-a-responder-to-the-issue) + - [Email contents and formatting](using_service_desk.md#email-contents-and-formatting) + - [Privacy considerations](using_service_desk.md#privacy-considerations) ## Troubleshooting Service Desk diff --git a/doc/user/project/service_desk/using_service_desk.md b/doc/user/project/service_desk/using_service_desk.md new file mode 100644 index 00000000000..73d85d418d9 --- /dev/null +++ b/doc/user/project/service_desk/using_service_desk.md @@ -0,0 +1,182 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Use Service Desk **(FREE ALL)** + +You can use Service Desk to [create an issue](#as-an-end-user-issue-creator) or [respond to one](#as-a-responder-to-the-issue). +In these issues, you can also see our friendly neighborhood [Support Bot](configure.md#support-bot-user). + +## View Service Desk email address + +To check what the Service Desk email address is for your project: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Service Desk**. + +The email address is available at the top of the issue list. + +## As an end user (issue creator) + +> Support for additional email headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346600) in GitLab 14.6. In earlier versions, the Service Desk email address had to be in the "To" field. + +To create a Service Desk issue, an end user does not need to know anything about +the GitLab instance. They just send an email to the address they are given, and +receive an email back confirming receipt: + + + +This also gives the end user an option to unsubscribe. + +If they don't choose to unsubscribe, then any new comments added to the issue +are sent as emails: + + + +Any responses they send via email are displayed in the issue itself. + +For information about headers used for treating email, see +[the incoming email documentation](../../../administration/incoming_email.md#accepted-headers). + +## As a responder to the issue + +For responders to the issue, everything works just like other GitLab issues. +GitLab displays a familiar-looking issue tracker where responders can see +issues created through customer support requests, and filter or interact with them. + + + +Messages from the end user are shown as coming from the special +[Support Bot user](../../../subscriptions/self_managed/index.md#billable-users). +You can read and write comments as you usually do in GitLab: + + + +- The project's visibility (private, internal, public) does not affect Service Desk. +- The path to the project, including its group or namespace, is shown in emails. + +### View Service Desk issues + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To view Service Desk issues: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Service Desk**. + +#### Redesigned issue list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/413092) in GitLab 16.1 [with a flag](../../../administration/feature_flags.md) named `service_desk_vue_list`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `service_desk_vue_list`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +When this feature is enabled, the Service Desk issue list more closely matches the regular issue list. +Available features include: + +- The same sorting and ordering options [as on the issue list](../issues/sorting_issue_lists.md). +- The same filters, including [the OR operator](#filter-with-the-or-operator) and [filtering by issue ID](#filter-issues-by-id). + +There is no longer an option to create a new issue from the Service Desk issue list. +This decision better reflects the nature of Service Desk, where new issues are created by emailing +a dedicated email address. + +##### Filter the list of issues + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Service Desk**. +1. Above the list of issues, select **Search or filter results...**. +1. In the dropdown list that appears, select the attribute you want to filter by. +1. Select or type the operator to use for filtering the attribute. The following operators are + available: + - `=`: Is + - `!=`: Is not one of +1. Enter the text to filter the attribute by. + You can filter some attributes by **None** or **Any**. +1. Repeat this process to filter by multiple attributes. Multiple attributes are joined by a logical + `AND`. + +##### Filter with the OR operator + +When [filtering with the OR operator](../issues/managing_issues.md#filter-with-the-or-operator) is enabled, +you can use **is one of: `||`** +when you [filter the list of issues](#filter-the-list-of-issues) by: + +- Assignees +- Labels + +`is one of` represents an inclusive OR. For example, if you filter by `Assignee is one of Sidney Jones` and +`Assignee is one of Zhang Wei`, GitLab shows issues where either `Sidney`, `Zhang`, or both of them are assignees. + +##### Filter issues by ID + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Service Desk**. +1. In the **Search** box, type the issue ID. For example, enter filter `#10` to return only issue 10. + +## Email contents and formatting + +### Special HTML formatting in HTML emails + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109811) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `service_desk_html_to_text_email_handler`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116809) in GitLab 15.11. Feature flag `service_desk_html_to_text_email_handler` removed. + +HTML emails show HTML formatting, such as: + +- Tables +- Blockquotes +- Images +- Collapsible sections + +### Files attached to comments + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11733) in GitLab 15.8 [with a flag](../../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/386860) in GitLab 15.10. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. +On GitLab.com, this feature is available. + +If a comment contains any attachments and their total size is less than or equal to 10 MB, these +attachments are sent as part of the email. In other cases, the email contains links to the attachments. + +In GitLab 15.9 and earlier, uploads to a comment are sent as links in the email. + +## Privacy considerations + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108901) the minimum required role to view the creator's and participant's email in GitLab 15.9. + +Service Desk issues are [confidential](../issues/confidential_issues.md), so they are +only visible to project members. The project owner can +[make an issue public](../issues/confidential_issues.md#in-an-existing-issue). +When a Service Desk issue becomes public, the issue creator's and participants' email addresses are +visible to signed-in users with at least the Reporter role for the project. + +In GitLab 15.8 and earlier, when a Service Desk issue becomes public, the issue creator's email +address is disclosed to everyone who can view the project. + +Anyone in your project can use the Service Desk email address to create an issue in this project, **regardless +of their role** in the project. + +The unique internal email address is visible to project members at least +the Reporter role in your GitLab instance. +An external user (issue creator) cannot see the internal email address +displayed in the information note. + +### Moving a Service Desk issue + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/372246) in GitLab 15.7: customers continue receiving notifications when a Service Desk issue is moved. + +You can move a Service Desk issue the same way you +[move a regular issue](../issues/managing_issues.md#move-an-issue) in GitLab. + +If a Service Desk issue is moved to a different project with Service Desk enabled, +the customer who created the issue continues to receive email notifications. +Because a moved issue is first closed, then copied, the customer is considered to be a participant +in both issues. They continue to receive any notifications in the old issue and the new one. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 087330431ad..22eb7c54d12 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -79,14 +79,14 @@ For example: Before you can migrate projects on a self-managed GitLab instance using file exports, GitLab administrators must: -1. [Enable file exports](../../../administration/settings/visibility_and_access_controls.md#enable-project-export) on the source +1. [Enable file exports](../../../administration/settings/import_and_export_settings.md#enable-project-export) on the source instance. 1. Enable file exports as an import source for the destination instance. On GitLab.com, file exports are already enabled as an import source. To enable file exports as an import source for the destination instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Visibility and access controls**. @@ -113,7 +113,7 @@ Prerequisites: To export a project and its data, follow these steps: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. Select **Export project**. @@ -245,7 +245,7 @@ If you have a larger project, consider [using a Rake task](../../../administrati Administrators can set the maximum import file size one of two ways: - With the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings). -- In the [Admin Area UI](../../../administration/settings/account_and_limit_settings.md#max-import-size). +- In the [Admin Area UI](../../../administration/settings/import_and_export_settings.md#max-import-size). The default is `0` (unlimited). diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 6c2140595d9..d9c114c0a59 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -11,9 +11,11 @@ Use the **Settings** page to manage the configuration options in your [project]( ## View project settings -You must have at least the Maintainer role to view project settings. +Prerequisite: + +- You must have at least the Maintainer role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. To display all settings in a section, select **Expand**. 1. Optional. Use the search box to find a setting. @@ -22,8 +24,11 @@ You must have at least the Maintainer role to view project settings. Use the project general settings to edit your project details. -1. Sign in to GitLab with at least the Maintainer role. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +Prerequisite: + +- You must have at least the Maintainer role for the project. + +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. In the **Project name** text box, enter your project name. 1. In the **Project description** text box, enter your project description. @@ -31,11 +36,11 @@ Use the project general settings to edit your project details. ## Assign topics to a project -Use topics to categorize projects and find similar new projects. +Use [topics](../working_with_projects.md#organizing-projects-with-topics) to categorize projects and find similar new projects. To assign topics to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings** > **General**. 1. In the **Topics** text box, enter the project topics. Popular topics are suggested as you type. 1. Select **Save changes**. @@ -46,50 +51,38 @@ If you're an instance administrator, you can administer all project topics from NOTE: The assigned topics are visible only to users with access to the project, but everyone can see which topics exist on the GitLab instance. Do not include sensitive information in the name of a topic. -## Add a compliance framework to a project **(PREMIUM ALL)** +## Rename a repository + +A project's repository name defines its URL and its place on the file disk +where GitLab is installed. + +Prerequisite: -You can -[add compliance frameworks to projects](../../group/compliance_frameworks.md#add-a-compliance-framework-to-a-project) -in a group that has a compliance framework. +- You must be an administrator or have the Maintainer or Owner role for the project. + +NOTE: +When you change the repository path, users may experience issues if they push to, or pull from, the old URL. For more information, see +[redirects when renaming repositories](../repository/index.md#what-happens-when-a-repository-path-changes). + +To rename a repository: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Advanced**. +1. In the **Change path** text box, edit the path. +1. Select **Change path**. -## Configure project visibility, features, and permissions +## Configure project features and permissions -To configure visibility, features, and permissions for a project: +To configure features and permissions for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. -1. To change the project visibility, select the dropdown list. If you select to **Public**, you limit access to some features to **Only Project Members**. 1. To allow users to request access to the project, select the **Users can request access** checkbox. -1. Use the [toggles](#project-feature-settings) to enable or disable features in the project. +1. To enable or disable features in the project, use the feature toggles. 1. Select **Save changes**. -### Project feature settings - -Use the toggles to enable or disable features in the project. - -| Option | More access limit options | Description -| :------------------------------- | :------------------------ | :---------- | -| **Issues** | **{check-circle}** Yes | Activates the GitLab issues tracker. -| **Repository** | **{check-circle}** Yes | Enables [repository](../repository/index.md) functionality. -| **Merge requests** | **{check-circle}** Yes | Enables [merge request](../merge_requests/index.md) functionality; also see [Merge request settings](#configure-merge-request-settings-for-a-project). -| **Forks** | **{check-circle}** Yes | Enables [forking](../repository/forking_workflow.md) functionality. -| **Git Large File Storage (LFS)** | **{dotted-circle}** No | Enables the use of [large files](../../../topics/git/lfs/index.md). -| **Packages** | **{dotted-circle}** No | Supports configuration of a [package registry](../../../administration/packages/index.md#gitlab-package-registry-administration) functionality. -| **CI/CD** | **{check-circle}** Yes | Enables [CI/CD](../../../ci/index.md) functionality. -| **Container Registry** | **{dotted-circle}** No | Activates a [registry](../../packages/container_registry/index.md) for your Docker images. -| **Analytics** | **{check-circle}** Yes | Enables [analytics](../../analytics/index.md). -| **Requirements** | **{check-circle}** Yes | Control access to [Requirements Management](../requirements/index.md). -| **Security and Compliance** | **{check-circle}** Yes | Control access to [security features](../../application_security/index.md). -| **Wiki** | **{check-circle}** Yes | Enables a separate system for [documentation](../wiki/index.md). -| **Snippets** | **{check-circle}** Yes | Enables [sharing of code and text](../../snippets.md). -| **Pages** | **{check-circle}** Yes | Allows you to [publish static websites](../pages/index.md). -| **Releases** | **{check-circle}** Yes | Control access to [Releases](../releases/index.md). -| **Environments** | **{check-circle}** Yes | Control access to [Environments and Deployments](../../../ci/environments/index.md). -| **Feature flags** | **{check-circle}** Yes | Control access to [Feature flags](../../../operations/feature_flags.md). -| **Monitor** | **{check-circle}** Yes | Control access to [Monitor](../../../operations/index.md) features. -| **Infrastructure** | **{check-circle}** Yes | Control access to [Infrastructure](../../infrastructure/index.md) features. - When you disable a feature, the following additional features are also disabled: - If you disable the **Issues** feature, project users cannot use: @@ -111,164 +104,66 @@ When you disable a feature, the following additional features are also disabled: - **Git Large File Storage** - **Packages** -- Metrics dashboard access requires reading project environments and deployments. +- The metrics dashboard requires read access to project environments and deployments. Users with access to the metrics dashboard can also access environments and deployments. -### Manage project access through LDAP groups - -You can [use LDAP to manage group membership](../../group/access_and_permissions.md#manage-group-memberships-via-ldap). - -You cannot use LDAP groups to manage project access, but you can use the following -workaround. - -Prerequisites: - -- You must [integrate LDAP with GitLab](../../../administration/auth/ldap/index.md). -- You must be an administrator. - -1. [Create a group](../../group/index.md#create-a-group) to track membership of your project. -1. [Set up LDAP synchronization](../../../administration/auth/ldap/ldap_synchronization.md) for that group. -1. To use LDAP groups to manage access to a project, [add the LDAP-synchronized group as a member](../members/index.md#add-groups-to-a-project) - to the project. - -## Disable CVE identifier request in issues **(FREE SAAS)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41203) in GitLab 13.4, only for public projects on GitLab.com. - -In some environments, users can submit a [CVE identifier request](../../application_security/cve_id_request.md) in an issue. - -To disable the CVE identifier request option in issues in your project: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Under **Issues**, turn off the **CVE ID requests in the issue sidebar** toggle. -1. Select **Save changes**. - -## Disable project email notifications - -Prerequisites: - -- You must be an Owner of the project to disable email notifications related to the project. - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Clear the **Disable email notifications** checkbox. - ## Configure merge request settings for a project Configure your project's merge request settings: - Set up the [merge request method](../merge_requests/methods/index.md) (merge commit, fast-forward merge). - Add merge request [description templates](../description_templates.md). -- Enable [merge request approvals](../merge_requests/approvals/index.md). -- Enable [status checks](../merge_requests/status_checks.md). -- Enable [merge only if pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). -- Enable [merge only when all threads are resolved](../merge_requests/index.md#prevent-merge-unless-all-threads-are-resolved). -- Enable [require an associated issue from Jira](../../../integration/jira/issues.md#require-associated-jira-issue-for-merge-requests-to-be-merged). -- Enable [**Delete source branch when merge request is accepted** option by default](#delete-the-source-branch-on-merge-by-default). -- Configure [suggested changes commit messages](../merge_requests/reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). -- Configure [merge and squash commit message templates](../merge_requests/commit_templates.md). -- Configure [the default target project](../merge_requests/creating_merge_requests.md#set-the-default-target-project) for merge requests coming from forks. -- Enable [Suggested Reviewers](../merge_requests/reviews/index.md#suggested-reviewers). - -## Service Desk - -Enable [Service Desk](../service_desk/index.md) for your project to offer customer support. - -## Export project - -Learn how to [export a project](import_export.md#import-a-project-and-its-data) in GitLab. - -## Advanced project settings - -Use the advanced settings to archive, rename, transfer, -[remove a fork relationship](../repository/forking_workflow.md#unlink-a-fork), or delete a project. - -### Archive a project - -When you archive a project, the repository, packages, issues, merge requests, and all -other features are read-only. Archived projects are also hidden from project listings. - -To archive a project: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Advanced**. -1. In the **Archive project** section, select **Archive project**. -1. To confirm, select **OK**. - -### Unarchive a project - -When you unarchive a project, you remove the read-only restriction and make it -available in project lists. - -Prerequisites: - -- To unarchive a project, you must be an administrator or a project Owner. - -1. Find the archived project. - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). - 1. Select **View all your projects**. - 1. Select **Explore projects**. - 1. In the **Sort projects** dropdown list, select **Show archived projects**. - 1. In the **Filter by name** field, enter the project name. - 1. Select the project link. -1. On the left sidebar, select **Settings > General**. -1. Under **Advanced**, select **Expand**. -1. In the **Unarchive project** section, select **Unarchive project**. -1. To confirm, select **OK**. - -### Rename a repository - -A project's repository name defines its URL and its place on the file disk -where GitLab is installed. - -Prerequisites: - -You must be a project maintainer, owner, or administrator to rename a repository. - -NOTE: -When you change the repository path, users may experience issues if they push to, or pull from, the old URL. For more information, see -[redirects when renaming repositories](../repository/index.md#what-happens-when-a-repository-path-changes). - -To rename a repository: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand **Advanced**. -1. In the **Change path** text box, edit the path. -1. Select **Change path**. - -## Delete the source branch on merge by default +- Enable: + - [Merge request approvals](../merge_requests/approvals/index.md). + - [Status checks](../merge_requests/status_checks.md). + - [Merge only if pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). + - [Merge only when all threads are resolved](../merge_requests/index.md#prevent-merge-unless-all-threads-are-resolved). + - [Required associated issue from Jira](../../../integration/jira/issues.md#require-associated-jira-issue-for-merge-requests-to-be-merged). + - [GitLab Duo Suggested Reviewers](../merge_requests/reviews/index.md#gitlab-duo-suggested-reviewers) + - [**Delete source branch when merge request is accepted** option by default](#delete-the-source-branch-on-merge-by-default). +- Configure: + - [Suggested changes commit messages](../merge_requests/reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). + - [Merge and squash commit message templates](../merge_requests/commit_templates.md). + - [Default target project](../merge_requests/creating_merge_requests.md#set-the-default-target-project) for merge requests coming from forks. + +### Delete the source branch on merge by default In merge requests, you can change the default behavior so that the **Delete the source branch** checkbox is always selected. To set this default: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. 1. Select **Enable "Delete source branch" option by default**. 1. Select **Save changes**. +## Export project + +You can [export a project and its data](import_export.md#export-a-project-and-its-data). + ## Transfer a project to another namespace When you transfer a project to another namespace, you move the project to a different group. Prerequisites: -- You must have at least the Maintainer role for the [group](../../group/index.md#create-a-group) to which you are transferring. +- You must have at least the Maintainer role for the [group](../../group/index.md#create-a-group) you are transferring to. - You must be the Owner of the project you transfer. - The group must allow creation of new projects. - The project must not contain any [container images](../../packages/container_registry/index.md#move-or-rename-container-registry-repositories). -- Remove any npm packages. If you transfer a project to a different root namespace, the project must not contain any npm packages. When you update the path of a user or group, or transfer a subgroup or project, you must remove any npm packages first. You cannot update the root namespace of a project with npm packages. Make sure you update your .npmrc files to follow the naming convention and run npm publish if necessary. -- If a security policy is assigned to the project, it is automatically unassigned during the transfer. +- The project must not have a security policy. + If a security policy is assigned to the project, it is automatically unassigned during the transfer. +- If the root namespace changes, you must remove npm packages that follow the [naming convention](../../../user/packages/npm_registry/index.md#naming-convention) from the project. + After you transfer the project you can either: + + - Update the package scope with the new root namespace path, and publish it again to the project. + - Republish the package to the project without updating the root namespace path, which causes the package to no longer follow the naming convention. + If you republish the package without updating the root namespace path, it will not be available at the [instance level endpoint](../../../user/packages/npm_registry/index.md#install-from-the-instance-level). To transfer a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. Under **Transfer project**, choose the namespace to transfer the project to. @@ -283,12 +178,46 @@ to move any project to any namespace. ### Transferring a GitLab SaaS project to a different subscription tier -When you transfer a project from a namespace licensed for GitLab SaaS Premium or Ultimate to GitLab Free, the following paid feature data is deleted: +When you transfer a project from a namespace licensed for GitLab SaaS Premium or Ultimate to GitLab Free: -- [Project access tokens](../../../user/project/settings/project_access_tokens.md) are revoked +- [Project access tokens](../../../user/project/settings/project_access_tokens.md) are revoked. - [Pipeline subscriptions](../../../ci/pipelines/index.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt) and [test cases](../../../ci/test_cases/index.md) are deleted. +## Archive a project + +When you archive a project, the repository, packages, issues, merge requests, and all +other features become read-only. Archived projects are also hidden from project lists. + +To archive a project: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Advanced**. +1. In the **Archive project** section, select **Archive project**. +1. To confirm, select **OK**. + +## Unarchive a project + +When you unarchive a project, the read-only restriction is removed, +and the project becomes available in project lists. + +Prerequisite: + +- You must be an administrator or have the Owner role for the project. + +1. Find the archived project. + 1. On the left sidebar, select **Search or go to**. + 1. Select **View all my projects**. + 1. Select **Explore projects**. + 1. In the **Sort projects** dropdown list, select **Show archived projects**. + 1. In the **Filter by name** field, enter the project name. + 1. Select the project link. +1. On the left sidebar, select **Settings > General**. +1. Under **Advanced**, select **Expand**. +1. In the **Unarchive project** section, select **Unarchive project**. +1. To confirm, select **OK**. + ## Delete a project > - Default deletion behavior for projects changed to [delayed project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6. @@ -297,6 +226,10 @@ When you transfer a project from a namespace licensed for GitLab SaaS Premium or > - Default deletion behavior changed to delayed deletion on the Premium and Ultimate tier [on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/393622) and [on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119606) in GitLab 16.0. You can mark a project to be deleted. +After you delete a project: + +- Projects in personal namespaces are deleted immediately. +- Projects in groups are deleted after a retention period. Prerequisite: @@ -304,7 +237,7 @@ Prerequisite: To delete a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the **Delete this project** section, select **Delete project**. @@ -312,6 +245,8 @@ To delete a project: This action deletes the project and all associated resources (such as issues and merge requests). +You can also [delete projects using the Rails console](../working_with_projects.md#delete-a-project-using-console). + ### Delayed project deletion **(PREMIUM ALL)** > - [Enabled for projects in personal namespaces](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89466) in GitLab 15.1. @@ -323,6 +258,10 @@ Projects in a group (not a personal namespace) can be deleted after a delay peri On self-managed instances, group administrators can define a deletion delay period of between 1 and 90 days. On SaaS, there is a non-adjustable default retention period of seven days. +You can [view projects that are pending deletion](../working_with_projects.md#view-projects-pending-deletion), +and use the Rails console to +[find projects that are pending deletion](../working_with_projects.md#find-projects-that-are-pending-deletion). + ### Delete a project immediately > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191367) in GitLab 14.1. @@ -340,7 +279,7 @@ Prerequisites: To immediately delete a project marked for deletion: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the **Delete this project** section, select **Delete project**. @@ -352,39 +291,55 @@ To immediately delete a project marked for deletion: To restore a project marked for deletion: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the Restore project section, select **Restore project**. -## Monitor settings +## Disable project analytics -### Alerts +By default, [analytics for a project](../../analytics/index.md#project-level-analytics) are displayed under the **Analyze** item in the left sidebar. +To disable this feature and remove the **Analyze** item from the left sidebar: -Configure [alert integrations](../../../operations/incident_management/integrations.md#configuration) to triage and manage critical problems in your application as [alerts](../../../operations/incident_management/alerts.md). +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Turn off the **Analytics** toggle. +1. Select **Save changes**. -### Incidents +## Disable CVE identifier request in issues **(FREE SAAS)** -#### Alert integration +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41203) in GitLab 13.4, only for public projects on GitLab.com. -Automatically [create](../../../operations/incident_management/alerts.md#trigger-actions-from-alerts), [notify on](../../../operations/incident_management/paging.md#email-notifications-for-alerts), and [resolve](../../../operations/incident_management/manage_incidents.md#automatically-close-incidents-via-recovery-alerts) incidents based on GitLab alerts. +In some environments, users can submit a [CVE identifier request](../../application_security/cve_id_request.md) in an issue. -#### PagerDuty integration +To disable the CVE identifier request option in issues in your project: -[Create incidents in GitLab for each PagerDuty incident](../../../operations/incident_management/manage_incidents.md#using-the-pagerduty-webhook). +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Under **Issues**, turn off the **CVE ID requests in the issue sidebar** toggle. +1. Select **Save changes**. -#### Incident settings +## Disable project email notifications -[Manage Service Level Agreements for incidents](../../../operations/incident_management/incidents.md#service-level-agreement-countdown-timer) with an SLA countdown timer. +Prerequisite: -### Error Tracking +- You must have the Owner role for the project. -Configure Error Tracking to discover and view [Sentry errors within GitLab](../../../operations/error_tracking.md). +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Clear the **Disable email notifications** checkbox. -### Status Page **(ULTIMATE ALL)** +## Related topics -[Add Storage credentials](../../../operations/incident_management/status_page.md#sync-incidents-to-the-status-page) -to enable the syncing of public Issues to a [deployed status page](../../../operations/incident_management/status_page.md#create-a-status-page-project). +- [Alert integrations](../../../operations/incident_management/integrations.md#configuration) +- [PagerDuty incident management](../../../operations/incident_management/manage_incidents.md#using-the-pagerduty-webhook) +- [SLA countdown timer](../../../operations/incident_management/incidents.md#service-level-agreement-countdown-timer) +- [Error tracking](../../../operations/error_tracking.md) +- [Incidents sync](../../../operations/incident_management/status_page.md#sync-incidents-to-the-status-page) +- [Service Desk](../service_desk/index.md) ## Troubleshooting diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index 78620eb2ab3..29d57328532 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, howto @@ -55,7 +55,7 @@ all projects that have visibility level set to [Internal](../../public_access.md To create a project access token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Access Tokens**. 1. Enter a name. The token name is visible to any user with permissions to view the project. 1. Enter an expiry date for the token. @@ -73,12 +73,14 @@ A project access token is displayed. Save the project access token somewhere saf To revoke a project access token: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Access Tokens**. 1. Next to the project access token to revoke, select **Revoke**. ## Scopes for a project access token +> `k8s_proxy` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422408) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `k8s_proxy_pat`. Enabled by default. + The scope determines the actions you can perform when you authenticate with a project access token. NOTE: @@ -93,6 +95,8 @@ See the warning in [create a project access token](#create-a-project-access-toke | `read_repository` | Grants read access (pull) to the repository. | | `write_repository` | Grants read and write access (pull and push) to the repository. | | `create_runner` | Grants permission to create runners in the project. | +| `ai_features` | Grants permission to perform API actions for GitLab Duo. | +| `k8s_proxy` | Grants permission to perform Kubernetes API calls using the agent for Kubernetes in the project. | ## Enable or disable project access token creation @@ -100,7 +104,7 @@ See the warning in [create a project access token](#create-a-project-access-toke To enable or disable project access token creation for all projects in a top-level group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Under **Permissions**, turn on or off **Allow project and group access token creation**. diff --git a/doc/user/project/system_notes.md b/doc/user/project/system_notes.md index 84ff9efbd14..661f10290c6 100644 --- a/doc/user/project/system_notes.md +++ b/doc/user/project/system_notes.md @@ -31,7 +31,7 @@ The filtering options are: ### On an epic -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Epics**. 1. Identify your desired epic, and select its title. 1. Go to the **Activity** section. @@ -39,14 +39,14 @@ The filtering options are: ### On an issue -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues** and find your issue. 1. Go to **Activity**. 1. For **Sort or filter**, select **Show all activity**. ### On a merge request -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests** and find your merge request. 1. Go to **Activity**. 1. For **Sort or filter**, select **Show all activity**. diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index ec8d886c68e..f9a11a51c98 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -185,7 +185,7 @@ The following time units are available: In GitLab self-managed instances, you can limit the display of time units to hours. To do so: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Settings > Preferences**. 1. Expand **Localization**. 1. Under **Time tracking**, select the **Limit display of time tracking units to hours** checkbox. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 994ed244832..fb2bd707c56 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -23,7 +23,7 @@ To pair the Web IDE with a remote development environment, see [remote developme To open the Web IDE from the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Use the <kbd>.</kbd> keyboard shortcut. You can also open the Web IDE from: @@ -81,6 +81,20 @@ To view a list of files you changed in the Web IDE: Your `CHANGES`, `STAGED CHANGES`, and `MERGE CHANGES` are displayed. For more information, see the [VS Code documentation](https://code.visualstudio.com/docs/sourcecontrol/overview#_commit). +## Restore uncommitted changes + +You don't have to manually save any file you modify in the Web IDE. +Modified files are automatically staged and can be [committed](#commit-changes). +Uncommitted changes are saved in your browser's local storage and persist +even if you close the browser tab or refresh the Web IDE. + +If your uncommitted changes are not available, you can restore the changes from local history. +To restore uncommitted changes in the Web IDE: + +1. Press <kbd>Shift</kbd>+<kbd>Command</kbd>+<kbd>P</kbd>. +1. In the search box, enter `Local History: Find Entry to Restore`. +1. Select the file that contains the uncommitted changes. + ## Upload a new file To upload a new file in the Web IDE: @@ -203,7 +217,7 @@ To protect your privacy and data: - Carefully review the permissions requested by an extension before you install the extension. - Keep your extensions up to date to ensure that any security or privacy vulnerabilities are addressed promptly. --> -## Interactive web terminals for the Web IDE (Beta) +## Interactive web terminals for the Web IDE **(BETA)** WARNING: This feature is in [Beta](../../../policy/experiment-beta-support.md#beta) and subject to change without notice. diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 916c8abf066..ae182f1f183 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -26,7 +26,7 @@ can edit group wikis. Group wiki repositories can be moved using the To access a group wiki: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. To display the wiki, either: - On the left sidebar, select **Plan > Wiki**. - On any page in the group, use the <kbd>g</kbd> + <kbd>w</kbd> @@ -66,7 +66,7 @@ can enable or disable a group wiki through the group settings. To open group settings: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. 1. Expand **Permissions and group features**. 1. Scroll to **Wiki** and select one of these options: diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 71fc4dd20a3..4f3aa0d0d49 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -31,7 +31,7 @@ with sibling pages listed in alphabetical order. To view a list of all pages, se To access a project wiki: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. To display the wiki, either: - On the left sidebar, select **Plan > Wiki**. - On any page in the project, use the <kbd>g</kbd> + <kbd>w</kbd> @@ -61,7 +61,7 @@ When a wiki is created, it is empty. On your first visit, you can create the home page users see when viewing the wiki. This page requires a specific title to be used as your wiki's home page. To create it: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Select **Create your first page**. 1. GitLab requires this first page be titled `home`. The page with this @@ -77,7 +77,7 @@ to be used as your wiki's home page. To create it: Users with at least the Developer role can create new wiki pages: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Select **New page** on this page, or any other wiki page. 1. Select a content format. @@ -138,7 +138,7 @@ may not be able to check out the wiki locally afterward. You need at least the Developer role to edit a wiki page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Go to the page you want to edit, and either: - Use the <kbd>e</kbd> wiki [keyboard shortcut](../../shortcuts.md#wiki-pages). @@ -157,7 +157,7 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). You need at least the Developer role to delete a wiki page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Go to the page you want to delete. 1. Select the edit icon (**{pencil}**). @@ -168,7 +168,7 @@ You need at least the Developer role to delete a wiki page: You need at least the Developer role to move a wiki page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Go to the page you want to move. 1. Select the edit icon (**{pencil}**). @@ -192,7 +192,7 @@ The history page shows: To view the changes for a wiki page: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Go to the page you want to view history for. 1. Select **Page history**. @@ -203,7 +203,7 @@ To view the changes for a wiki page: You can see the changes made in a version of a wiki page, similar to versioned diff file views: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. Go to the wiki page you're interested in. 1. Select **Page history** to see all page versions. @@ -234,7 +234,7 @@ You need at least the Developer role to customize the wiki navigation sidebar. This process creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. 1. In the upper-right corner of the page, select **Edit sidebar**. 1. When complete, select **Save changes**. @@ -257,7 +257,7 @@ A `_sidebar` example, formatted with Markdown: Wikis are enabled by default in GitLab. Project [administrators](../../permissions.md) can enable or disable a project wiki by following the instructions in -[Sharing and permissions](../settings/index.md#configure-project-visibility-features-and-permissions). +[Sharing and permissions](../settings/index.md#configure-project-features-and-permissions). Administrators for self-managed GitLab installs can [configure additional wiki settings](../../../administration/wikis/index.md). @@ -268,7 +268,7 @@ You can disable group wikis from the [group settings](group.md#configure-group-w To add a link to an external wiki from a project's left sidebar: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **External wiki**. 1. Add the URL to your external wiki. @@ -284,7 +284,7 @@ To hide the internal wiki from the sidebar, [disable the project's wiki](#disabl To hide the link to an external wiki: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **External wiki**. 1. Under **Enable integration**, clear the **Active** checkbox. @@ -294,7 +294,7 @@ To hide the link to an external wiki: To disable a project's internal wiki: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Scroll down to find **Wiki** and toggle it off (in gray). diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 8c050caed17..2a4f0b99246 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -13,7 +13,7 @@ code are saved in projects, and most features are in the scope of projects. To view all projects for the GitLab instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Explore**. On the left sidebar, **Projects** is selected. On the right, the list shows @@ -25,7 +25,7 @@ If you are not authenticated, then the list shows public projects only. To view projects you are a member of: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. On the left sidebar, **Projects** is selected. On the list, on the **Yours** tab, @@ -68,7 +68,7 @@ Do not include sensitive information in the name of a topic. To explore project topics: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Explore**. 1. On the left sidebar, select **Topics**. 1. To view projects associated with a topic, select a topic. @@ -110,7 +110,7 @@ To subscribe to a topic: - From a project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. In the **Project overview** page, from the **Topics** list select the topic you want to subscribe to. 1. In the upper-right corner, select **Subscribe to the new projects feed** (**{rss}**). @@ -132,25 +132,9 @@ You can add a star to projects you use frequently to make them easier to find. To add a star to a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. In the upper-right corner of the page, select **Star**. -## Delete a project - -After you delete a project: - -- Projects in personal namespaces are deleted immediately. -- Projects in groups are [deleted after a retention period](../project/settings/index.md#delayed-project-deletion). - -To delete a project: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > General**. -1. Expand the **Advanced** section. -1. Scroll down to the **Delete project** section. -1. Select **Delete project**. -1. Confirm this action by completing the field. - ## View projects pending deletion **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3 for Administrators. @@ -160,8 +144,8 @@ To delete a project: To view a list of all projects that are pending deletion: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. Based on your GitLab version: - GitLab 14.6 and later: select the **Pending deletion** tab. - GitLab 14.5 and earlier: select the **Deleted projects** tab. @@ -176,7 +160,7 @@ Each project in the list shows: To view the activity of a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Manage > Activity**. 1. Optional. To filter activity by contribution type, select a tab: @@ -190,8 +174,8 @@ To view the activity of a project: ## Search in projects -To search through your projects, on the left sidebar, at the top, select **Search GitLab** -(**{search}**). GitLab filters as you type. +To search through your projects, on the left sidebar, select **Search or go to**. +GitLab filters as you type. You can also look for the projects you [starred](#star-a-project) (**Starred projects**). @@ -214,7 +198,7 @@ You can also choose to hide or show archived projects. You can filter projects by the programming language they use. To do this: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select either: - **View all your projects**, to filter your projects. - **Explore**, to filter all projects you can access. @@ -230,7 +214,7 @@ Prerequisite: - You must have the Owner role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Use the toggle by each feature you want to turn on or off, or change access for. @@ -274,7 +258,7 @@ When you leave a project: To leave a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Leave project**. The **Leave project** option only displays on the project dashboard when a project is part of a group under a [group namespace](../namespace/index.md). @@ -408,18 +392,44 @@ To access the Geo secondary server with HTTP: The `go get` request generates HTTP traffic to the primary Geo server. When the module download starts, the `insteadOf` configuration sends the traffic to the secondary Geo server. +## Add a compliance framework to a project **(PREMIUM)** + +You can add compliance frameworks to projects in a group that has a [compliance framework](../group/compliance_frameworks.md). + +## Manage project access through LDAP groups + +You can [use LDAP to manage group membership](../group/access_and_permissions.md#manage-group-memberships-via-ldap). + +You cannot use LDAP groups to manage project access, but you can use the following workaround. + +Prerequisites: + +- You must [integrate LDAP with GitLab](../../administration/auth/ldap/index.md). +- You must be an administrator. + +1. [Create a group](../group/index.md#create-a-group) to track membership of your project. +1. [Set up LDAP synchronization](../../administration/auth/ldap/ldap_synchronization.md) for that group. +1. To use LDAP groups to manage access to a project, +[add the LDAP-synchronized group as a member](../group/manage.md) to the project. + ## Related topics - [Import a project](../../user/project/import/index.md). - [Connect an external repository to GitLab CI/CD](../../ci/ci_cd_for_external_repos/index.md). - [Fork a project](repository/forking_workflow.md#create-a-fork). -- [Adjust project visibility and access levels](settings/index.md#configure-project-visibility-features-and-permissions). +- Adjust [project visibility](../../user/public_access.md#change-project-visibility) and [permissions](settings/index.md#configure-project-features-and-permissions). - [Limitations on project and group names](../../user/reserved_names.md#limitations-on-project-and-group-names) ## Troubleshooting When working with projects, you might encounter the following issues, or require alternate methods to complete specific tasks. +### `An error occurred while fetching commit data` + +When you visit a project, the message `An error occurred while fetching commit data` might be displayed +if you use an ad blocker in your browser. The solution is to disable your ad blocker +for the GitLab instance you are trying to access. + ### Find projects using an SQL query While in [a Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session), you can find and store an array of projects based on a SQL query: diff --git a/doc/user/public_access.md b/doc/user/public_access.md index 5f43f177e36..37b79c8f1a8 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -67,7 +67,7 @@ Prerequisite: - You must have the Owner role for a project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Change **Project visibility** to either **Private**, **Internal**, or **Public**. @@ -86,7 +86,7 @@ Prerequisites: restrictive as the new setting of the parent group. For example, you cannot set a group to private if a subgroup or project in that group is public. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Naming, visibility**. 1. Under **Visibility level** select either **Private**, **Internal**, or **Public**. diff --git a/doc/user/report_abuse.md b/doc/user/report_abuse.md index bad62825ba5..45113562e87 100644 --- a/doc/user/report_abuse.md +++ b/doc/user/report_abuse.md @@ -1,5 +1,5 @@ --- -stage: Anti-Abuse +stage: Govern group: Anti-Abuse info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -24,10 +24,19 @@ You can report a user through their: ## Report abuse from the user's profile page +> - Report abuse from overflow menu [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/414773) in GitLab 16.4 [with a flag](../administration/feature_flags.md) named `user_profile_overflow_menu_vue`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/414773) in GitLab 16.4. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `user_profile_overflow_menu_vue`. +On GitLab.com, this feature is available. + To report abuse from a user's profile page: 1. Anywhere in GitLab, select the name of the user. -1. In the upper-right corner of the user's profile, select **Report abuse to administrator** (**{information-o}**). +1. In the upper-right corner of the user's profile, if the `user_profile_overflow_menu_vue` feature flag is: + - Enabled, select the vertical ellipsis (**{ellipsis_v}**), then **Report abuse to administrator**. + - Disabled, select **Report abuse to administrator** (**{information-o}**). 1. Select a reason for reporting the user. 1. Complete an abuse report. 1. Select **Send report**. diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index c12d7889fb8..c1da3e9e2ba 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -39,8 +39,6 @@ You can use advanced search in: Advanced search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). The syntax supports both exact and fuzzy search queries. -<!-- markdownlint-disable --> - | Syntax | Description | Example | |--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `"` | Exact search | [`"gem sidekiq"`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=%22gem+sidekiq%22) | @@ -53,11 +51,12 @@ Advanced search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elas | `#` | Issue ID | [`#23456`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964) | | `!` | Merge request ID | [`!23456`](https://gitlab.com/search?snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964) | -### Refining user search +### User search -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388409) in GitLab 15.10. +> Ability to refine user search [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388409) in GitLab 15.10. -In user search, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/query-dsl-fuzzy-query.html) is used by default. You can refine your search with [Elasticsearch syntax](#syntax). +When you search for a user, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/query-dsl-fuzzy-query.html) is used by default. +You can refine user search with [Elasticsearch syntax](#syntax). ### Code search @@ -68,11 +67,13 @@ In user search, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/re | `extension:` | File extension without `.` <sup>2</sup> | [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | | `blob:` | Git object ID <sup>2</sup> | [`blob:998707*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707*&group_id=9970) | -1. `path:` returns matches for full paths or subpaths. +1. `path:` returns matches for full or partial paths. 1. `extension:` and `blob:` return exact matches only. ### Examples +<!-- markdownlint-disable MD044 --> + | Query | Description | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------| | [`rails -filename:gemfile.lock`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=rails+-filename%3Agemfile.lock&snippets=) | Returns `rails` in all files except the `gemfile.lock` file. | @@ -81,7 +82,6 @@ In user search, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/re | [<code>helper -extension:yml -extension:js</code>](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=helper+-extension%3Ayml+-extension%3Ajs&snippets=) | Returns `helper` in all files except files with a `.yml` or `.js` extension. | | [<code>helper path:lib/git</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=helper+path%3Alib%2Fgit) | Returns `helper` in all files with a `lib/git*` path (for example, `spec/lib/gitlab`). | - <!-- markdownlint-enable --> ## Known issues @@ -89,8 +89,8 @@ In user search, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/re - You can only search files smaller than 1 MB. For more information, see [issue 195764](https://gitlab.com/gitlab-org/gitlab/-/issues/195764). For self-managed GitLab instances, an administrator can - [change this limit](../../integration/advanced_search/elasticsearch.md#advanced-search-configuration). -- You can only use advanced search on the default branch of a project. + [configure this setting](../../integration/advanced_search/elasticsearch.md#advanced-search-configuration). +- You can use advanced search on the default branch of a project only. For more information, see [issue 229966](https://gitlab.com/gitlab-org/gitlab/-/issues/229966). - The search query must not contain any of the following characters: diff --git a/doc/user/search/command_palette.md b/doc/user/search/command_palette.md index 671afe13ace..0f42e727eda 100644 --- a/doc/user/search/command_palette.md +++ b/doc/user/search/command_palette.md @@ -7,21 +7,17 @@ type: reference # Command palette **(FREE ALL)** -> Introduced in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `command_palette`. Enabled by default. +> - Introduced in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `command_palette`. Enabled by default. +> - Feature flag `command_palette` removed in GitLab 16.4. You can use command palette to narrow down the scope of your search or to find an object more quickly. -FLAG: -On self-managed GitLab, by default this feature is available. -To hide the feature, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `command_palette`. -On GitLab.com, this feature is available. - ## Open the command palette To open the command palette: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) or use the <kbd>/</kbd> key to enable. +1. On the left sidebar, at the top, select **Search or go to** or use the <kbd>/</kbd> key to enable. 1. Type one of the special characters: - <kbd>></kbd> - Create a new object or find a menu item. diff --git a/doc/user/search/exact_code_search.md b/doc/user/search/exact_code_search.md index 8a64dc9e70f..48445ccfc3c 100644 --- a/doc/user/search/exact_code_search.md +++ b/doc/user/search/exact_code_search.md @@ -28,5 +28,19 @@ searches. ## Syntax -To understand the possible filtering options, see the -[Zoekt query syntax](https://github.com/sourcegraph/zoekt/blob/main/doc/query_syntax.md). +This table shows some example queries for exact code search. + +| Query | Description | +| -------------------- |-------------------------------------------------------------------------------------- | +| `foo` | Returns files that contain `foo` | +| `"class foo"` | Returns files that contain the exact string `class foo` | +| `class foo` | Returns files that contain both `class` and `foo` | +| `foo or bar` | Returns files that contain either `foo` or `bar` | +| `class Foo` | Returns files that contain `class` (case insensitive) and `Foo` (case sensitive) | +| `class Foo case:yes` | Returns files that contain `class` and `Foo` (both case sensitive) | +| `foo -bar` | Returns files that contain `foo` but not `bar` | +| `foo file:js` | Searches for `foo` in files with names that contain `js` | +| `foo -file:test` | Searches for `foo` in files with names that do not contain `test` | +| `foo lang:ruby` | Searches for `foo` in Ruby source code | +| `foo f:\.js$` | Searches for `foo` in files with names that end with `.js` | +| `foo.*bar` | Searches for strings that match the regular expression `foo.*bar` | diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 5600f18c61c..8c7db5ca29e 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -17,23 +17,26 @@ Both types of search are the same, except when you are searching through code. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68640) in GitLab 14.3. -To improve the performance of your instance's global search, a GitLab administrator -can limit the search scope by disabling the following [`ops` feature flags](../../development/feature_flags/index.md#ops-type). - -| Scope | Feature flag | Description | -|--|--|--| -| Code | `global_search_code_tab` | When enabled, global search includes code. | -| Commits | `global_search_commits_tab` | When enabled, global search includes commits. | -| Issues | `global_search_issues_tab` | When enabled, global search includes issues. | -| Merge requests | `global_search_merge_requests_tab` | When enabled, global search includes merge requests. | -| Users | `global_search_users_tab` | When enabled, global search includes users. | -| Wiki | `global_search_wiki_tab` | When enabled, global search includes project and [group wikis](../project/wiki/group.md). | +To improve the performance of your instance's global search, an administrator can limit the search scope +by disabling one or more [`ops` feature flags](../../development/feature_flags/index.md#ops-type). + +| Scope | Feature flag | Description | +|----------------|------------------------------------|-------------------------------------------------------------------------------------------| +| Code | `global_search_code_tab` | When enabled, global search includes code. | +| Commits | `global_search_commits_tab` | When enabled, global search includes commits. | +| Issues | `global_search_issues_tab` | When enabled, global search includes issues. | +| Merge requests | `global_search_merge_requests_tab` | When enabled, global search includes merge requests. | +| Users | `global_search_users_tab` | When enabled, global search includes users. | +| Wiki | `global_search_wiki_tab` | When enabled, global search includes project and [group wikis](../project/wiki/group.md). | All global search scopes are enabled by default on self-managed instances. ## Global search validation -Global search ignores and logs as abusive any search with: +> - Support for partial matches in issue search [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71913) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `issues_full_text_search`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124703) in GitLab 16.2. Feature flag `issues_full_text_search` removed. + +Global search ignores and logs as abusive any search that includes: - Fewer than two characters - A term longer than 100 characters (URL search terms must not exceed 200 characters) @@ -47,11 +50,28 @@ Global search only flags with an error any search that includes more than: - 4096 characters - 64 terms +Partial matches are not supported in issue search. +For example, when you search issues for `play`, the query does not return issues that contain `display`. +However, the query matches all possible variations of the string (for example, `plays`). + +## Autocomplete suggestions + +As you type in the search box, autocomplete suggestions are displayed for: + +- [Projects](#search-for-a-project-by-full-path) and groups +- Users +- Help pages +- Project features (for example, milestones) +- Settings (for example, user settings) +- Recently viewed merge requests +- Recently viewed issues and epics +- [GitLab Flavored Markdown references](../markdown.md#gitlab-specific-references) for issues in a project + ## Search in all GitLab To search in all GitLab: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**). +1. On the left sidebar, at the top, select **Search or go to**. 1. Type your search query. You must type at least two characters. 1. Press <kbd>Enter</kbd> to search, or select from the list. @@ -61,18 +81,56 @@ The results are displayed. To filter the results, on the left sidebar, select a To search in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Search GitLab** (**{search}**) again and type the string you want to search for. +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Search or go to** again and type the string you want to search for. 1. Press <kbd>Enter</kbd> to search, or select from the list. The results are displayed. To filter the results, on the left sidebar, select a filter. +## Search for a project by full path + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108906) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `full_path_project_search`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114932) in GitLab 15.11. Feature flag `full_path_project_search` removed. + +You can search for a project by entering its full path (including the namespace it belongs to) in the search box. +As you type the project path, [autocomplete suggestions](#autocomplete-suggestions) are displayed. + +For example: + +- `gitlab-org/gitlab` searches for the `gitlab` project in the `gitlab-org` namespace. +- `gitlab-org/` displays autocomplete suggestions for projects that belong to the `gitlab-org` namespace. + +## Include archived projects in search results + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121981) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `search_projects_hide_archived`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/413821) in GitLab 16.3. Feature flag `search_projects_hide_archived` removed. + +By default, archived projects are excluded from search results. +To include archived projects: + +1. On the project search page, on the left sidebar, select the **Include archived** checkbox. +1. On the left sidebar, select **Apply**. + +## Exclude issues in archived projects from search results + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124846) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `search_issues_hide_archived_projects`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +an administrator can [enable the feature flag](../../administration/feature_flags.md) named `search_issues_hide_archived_projects`. On GitLab.com, this feature is not available. + +By default, issues in archived projects are included in search results. +To exclude issues in archived projects, ensure the `search_issues_hide_archived_projects` flag is enabled. + +To include issues in archived projects with `search_issues_hide_archived_projects` enabled, +you must add the parameter `include_archived=true` to the URL. + ## Search for code To search for code in a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Search GitLab** (**{search}**) again and type the code you want to search for. +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Search or go to** again and type the code you want to search for. 1. Press <kbd>Enter</kbd> to search, or select from the list. Code search shows only the first result in the file. @@ -97,110 +155,26 @@ To filter code search results by one or more languages: 1. On the code search page, on the left sidebar, select one or more languages. 1. On the left sidebar, select **Apply**. -## Exclude search results - -### From archived projects - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121981) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `search_projects_hide_archived`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -an administrator can [enable the feature flag](../../administration/feature_flags.md) named `search_projects_hide_archived`. On GitLab.com, this feature is not available. - -Archived projects are included in search results by default. To exclude archived projects, ensure the `search_projects_hide_archived` flag is enabled. - -To include archived projects with `search_projects_hide_archived` enabled, you must add the parameter `include_archived=true` to the URL. - -### From issues in archived projects - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124846) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `search_issues_hide_archived_projects`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -an administrator can [enable the feature flag](../../administration/feature_flags.md) named `search_issues_hide_archived_projects`. On GitLab.com, this feature is not available. - -Issues in archived projects are included in search results by default. To exclude issues in archived projects, ensure the `search_issues_hide_archived_projects` flag is enabled. - -To include issues in archived projects with `search_issues_hide_archived_projects` enabled, you must add the parameter `include_archived=true` to the URL. - -## Search for a project by full path - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108906) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `full_path_project_search`. Disabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114932) in GitLab 15.11. Feature flag `full_path_project_search` removed. - -You can search for a project by entering its full path (including the namespace it belongs to) in the search box. -As you type the project path, [autocomplete suggestions](#autocomplete-suggestions) are displayed. - -For example: - -- `gitlab-org/gitlab` searches for the `gitlab` project in the `gitlab-org` namespace. -- `gitlab-org/` displays autocomplete suggestions for projects that belong to the `gitlab-org` namespace. - ## Search for a commit SHA To search for a commit SHA: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Search GitLab** (**{search}**) again and type the commit SHA you want to search for. +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Search or go to** again and type the commit SHA you want to search for. 1. Press <kbd>Enter</kbd> to search, or select from the list. If a single result is returned, GitLab redirects to the commit result and gives you the option to return to the search results page. -## Search for specific terms - -> - [Support for partial matches in issue search](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71913) removed in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `issues_full_text_search`. Disabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124703) in GitLab 16.2. Feature flag `issues_full_text_search` removed. - -You can filter issues and merge requests by specific terms included in titles or descriptions. - -- Syntax - - Searches look for all the words in a query, in any order. For example: searching - issues for `display bug` returns all issues matching both those words, in any order. - - To find the exact term, use double quotes: `"display bug"`. -- Limitation - - For performance reasons, terms shorter than three characters are ignored. For example: searching - issues for `included in titles` is same as `included titles` - - Search is limited to 4096 characters and 64 terms per query. - - When searching issues, partial matches are not allowed. For example: searching for `play` will - not return issues that have the word `display`. But variations of words match, so searching - for `displays` also returns issues that have the word `display`. - ## Run a search from history You can run a search from history for issues and merge requests. Search history is stored locally in your browser. To run a search from history: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. To view recent searches: - For issues, on the left sidebar, select **Plan > Issues**. Above the list, to the left of the search box, select (**{history}**). - For merge requests, on the left sidebar, select **Code > Merge requests**. Above the list, to the left of the search box, select **Recent searches**. 1. From the dropdown list, select a search. - -## Remove search filters - -Individual filters can be removed by selecting the filter's (x) button or backspacing. The entire search filter can be cleared by selecting the search box's (x) button or via <kbd>⌘</kbd> (Mac) + <kbd>⌫</kbd>. - -To delete filter tokens one at a time, the <kbd>⌥</kbd> (Mac) / <kbd>Control</kbd> + <kbd>⌫</kbd> keyboard combination can be used. - -## Autocomplete suggestions - -In the search box, you can view autocomplete suggestions for: - -- [Projects](#search-for-a-project-by-full-path) and groups -- Users -- Various help pages (try and type **API help**) -- Project feature pages (try and type **milestones**) -- Various settings pages (try and type **user settings**) -- Recently viewed issues (try and type some word from the title of a recently viewed issue) -- Recently viewed merge requests (try and type some word from the title of a recently viewed merge request) -- Recently viewed epics (try and type some word from the title of a recently viewed epic) -- [GitLab Flavored Markdown](../markdown.md#gitlab-specific-references) (GLFM) for issues in a project (try and type a GLFM reference for an issue) - -## Search settings - -You can search inside a Project, Group, Administrator, or User's settings by entering -a search term in the search box located at the top of the page. The search results -appear highlighted in the sections that match the search term. diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index d8b4d147d24..b0ef1fcc99a 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -109,18 +109,19 @@ These shortcuts are available when viewing issues: These shortcuts are available when viewing [merge requests](project/merge_requests/index.md): -| macOS shortcut | Windows shortcut | Description | -|-----------------------------------|---------------------|-------------| -| <kbd>]</kbd> or <kbd>j</kbd> | | Move to next file. | -| <kbd>[</kbd> or <kbd>k</kbd> | | Move to previous file. | +| macOS shortcut | Windows shortcut | Description | +|-----------------------------------|-----------------------------------|-------------| +| <kbd>]</kbd> or <kbd>j</kbd> | | Move to next file. | +| <kbd>[</kbd> or <kbd>k</kbd> | | Move to previous file. | | <kbd>Command</kbd> + <kbd>p</kbd> | <kbd>Control</kbd> + <kbd>p</kbd> | Search for, and then jump to a file for review. | -| <kbd>n</kbd> | | Move to next unresolved discussion. | -| <kbd>p</kbd> | | Move to previous unresolved discussion. | -| <kbd>b</kbd> | | Copy source branch name. | -| <kbd>c</kbd> + <kbd>r</kbd> | | Copy merge request reference. | -| <kbd>r</kbd> | | Start writing a comment. Pre-selected text is quoted in the comment. Can't be used to reply in a thread. | -| <kbd>c</kbd> | | Move to next commit. | -| <kbd>x</kbd> | | Move to previous commit. | +| <kbd>n</kbd> | | Move to next unresolved discussion. | +| <kbd>p</kbd> | | Move to previous unresolved discussion. | +| <kbd>b</kbd> | | Copy source branch name. | +| <kbd>c</kbd> + <kbd>r</kbd> | | Copy merge request reference. | +| <kbd>r</kbd> | | Start writing a comment. Pre-selected text is quoted in the comment. Can't be used to reply in a thread. | +| <kbd>c</kbd> | | Move to next commit. | +| <kbd>x</kbd> | | Move to previous commit. | +| <kbd>f</kbd> | | Toggle file browser. | ### Project files @@ -324,20 +325,25 @@ These shortcuts are available when viewing [epics](group/epics/index.md): ## Disable keyboard shortcuts -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22113) in GitLab 12.8. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/202494) from the shortcuts page to user preferences in GitLab 16.4. To disable keyboard shortcuts: -1. While viewing a page that supports keyboard shortcuts, and outside a text box, -press <kbd>?</kbd> to display the list of shortcuts. -1. Select **Toggle shortcuts**. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. In the **Behavior** section, clear the **Enable keyboard shortcuts** checkbox. +1. Select **Save changes**. ## Enable keyboard shortcuts +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/202494) from the shortcuts page to user preferences in GitLab 16.4. + To enable keyboard shortcuts: -1. On the left sidebar, at the bottom, select **Help** (**{question}**), then **Keyboard shortcuts**. -1. Select **Toggle shortcuts**. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. In the **Behavior** section, select the **Enable keyboard shortcuts** checkbox. +1. Select **Save changes**. ## Troubleshooting diff --git a/doc/user/snippets.md b/doc/user/snippets.md index a5a547727d3..dbcc90c26df 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -63,17 +63,17 @@ In GitLab versions 13.0 and later, snippets are [versioned by default](#versione To discover all snippets visible to you in GitLab, you can: - **View a project's snippets**: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Snippets**. - **View all the snippets you created**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Snippets**. On GitLab.com, you can also visit your [snippets directly](https://gitlab.com/dashboard/snippets). - **Explore all public snippets**: - 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. On the left sidebar, select **Search or go to**. 1. Select **Explore**. 1. Select **Snippets**. @@ -110,11 +110,11 @@ content was saved as the initial commit to the snippets' repository. ## Filenames Snippets support syntax highlighting based on the filename and -extension provided for them. While you can submit a snippet -without a filename and extension, it needs a valid name so the -content can be created as a file in the snippet's repository. +extension provided for them. You can submit a snippet +without a filename and extension, but a valid name is required for +creating content as a file in the repository. -If you don't give a snippet a filename and extension, +If no filename and extension are provided for the snippet, GitLab adds a filename in the format `snippetfile<x>.txt` where `<x>` represents a number added to the file, starting with 1. This number increments if you add more unnamed snippets. @@ -138,7 +138,7 @@ A single snippet can support up to 10 files, which helps keep related files toge - A `gulpfile.js` file and a `package.json` file, which together can be used to bootstrap a project and manage its dependencies. -If you need more than 10 files for your snippet, we recommend you create a +If you need more than 10 files for your snippet, you should create a [wiki](project/wiki/index.md) instead. Wikis are available for projects at all subscription levels, and [groups](project/wiki/group.md) for [GitLab Premium](https://about.gitlab.com/pricing/). @@ -167,9 +167,9 @@ To delete a file from your snippet through the GitLab UI: ## Clone snippets -Instead of copying a snippet to a local file, you may want to clone a snippet to -preserve its relationship with the repository, so you can receive or make updates -as needed. Select **Clone** on a snippet to display the URLs to clone with SSH or HTTPS: +To ensure you receive updates, clone the snippet instead of copying it locally. Cloning +maintains the snippet's connection with the repository. Select **Clone** on a snippet +to display the URLs to clone with SSH or HTTPS:  @@ -229,7 +229,7 @@ Prerequisites: To do this task: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Code > Snippets**. 1. Select the snippet you want to report as spam. 1. Select **Submit as spam**. diff --git a/doc/user/ssh.md b/doc/user/ssh.md index f418cc5f6b7..0e10fea18ad 100644 --- a/doc/user/ssh.md +++ b/doc/user/ssh.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -22,7 +22,7 @@ SSH uses two keys, a public key and a private key. It is not possible to reveal confidential data by uploading your public key. When you need to copy or upload your SSH public key, make sure you do not accidentally copy or upload your private key instead. -You can use your private key to [sign commits](project/repository/ssh_signed_commits/index.md), +You can use your private key to [sign commits](project/repository/signed_commits/ssh.md), which makes your use of GitLab and your data even more secure. This signature then can be verified by anyone using your public key. @@ -74,11 +74,16 @@ must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later ### RSA SSH keys +> Maximum RSA key length [changed](https://gitlab.com/groups/gitlab-org/-/epics/11186) in GitLab 16.3. + Available documentation suggests ED25519 is more secure than RSA. If you use an RSA key, the US National Institute of Standards and Technology in [Publication 800-57 Part 3 (PDF)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-57Pt3r1.pdf) -recommends a key size of at least 2048 bits. The default key size depends on your version of `ssh-keygen`. +recommends a key size of at least 2048 bits. Due to limitations in Go, +RSA keys [cannot exceed 8192 bits](#tls-server-sent-certificate-containing-rsa-key-larger-than-8192-bits). + +The default key size depends on your version of `ssh-keygen`. Review the `man` page for your installed `ssh-keygen` command for details. ## See if you have an existing SSH key pair @@ -523,6 +528,17 @@ are **explicitly not supported** and may stop working at any time. ## Troubleshooting +### TLS: server sent certificate containing RSA key larger than 8192 bits + +In GitLab 16.3 and later, Go limits RSA keys to a maximum of 8192 bits. +To check the length of a key: + +```shell +openssl rsa -in <your-key-file> -text -noout | grep "Key:" +``` + +Replace any key longer than 8192 bits with a shorter key. + ### Password prompt with `git clone` When you run `git clone`, you may be prompted for a password, like `git@gitlab.example.com's password:`. diff --git a/doc/user/storage_management_automation.md b/doc/user/storage_management_automation.md index 210aca4ee35..9a505d23597 100644 --- a/doc/user/storage_management_automation.md +++ b/doc/user/storage_management_automation.md @@ -28,7 +28,7 @@ You must use the following scopes to [authenticate](../api/rest/index.md#authent - Full API access with the `api` scope. - At least the Maintainer role on all projects. -You can use command line tools or a programming language to interact with the REST API. +You can use command-line tools or a programming language to interact with the REST API. ### Command line @@ -71,7 +71,7 @@ see [Efficient DevSecOps workflows: Hands-on `python-gitlab` API automation](htt For more information about other API client libraries, see [Third-party clients](../api/rest/index.md#third-party-clients). NOTE: -Use [GitLab Duo Code Suggestions](project/repository/code_suggestions.md) to write code more efficiently. +Use [GitLab Duo Code Suggestions](project/repository/code_suggestions/index.md) to write code more efficiently. ## Strategies for storage analysis diff --git a/doc/user/tasks.md b/doc/user/tasks.md index c340bb736ef..f4fec01d42b 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Known limitation: -- [Tasks currently cannot be accessed via REST API.](https://gitlab.com/gitlab-org/gitlab/-/issues/368055) +- [Tasks cannot be accessed via REST API.](https://gitlab.com/gitlab-org/gitlab/-/issues/368055) For the latest updates, check the [Tasks Roadmap](https://gitlab.com/groups/gitlab-org/-/epics/7103). @@ -41,7 +41,7 @@ View tasks in issues, in the **Tasks** section. You can also [filter the list of issues](project/issues/managing_issues.md#filter-the-list-of-issues) for `Type = task`. -If you select a task from an issue, it opens in a modal window. +If you select a task from an issue, it opens in a dialog window. If you select a task to open in a new browser tab, or select it from the issue list, the task opens in a full-page view. @@ -107,7 +107,7 @@ To edit a task: ### Using the rich text editor -> - Rich text editing in the modal view [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363007) in GitLab 15.6 [with a flag](../administration/feature_flags.md) named `work_items_mvc`. Disabled by default. +> - Rich text editing in the dialog view [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363007) in GitLab 15.6 [with a flag](../administration/feature_flags.md) named `work_items_mvc`. Disabled by default. > - Rich text editing in the full page view [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104533) in GitLab 15.7. FLAG: @@ -336,7 +336,7 @@ To refer to a task elsewhere in GitLab, you can use its full URL or a short refe To copy the task reference to your clipboard: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your task to view it. 1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. @@ -356,7 +356,7 @@ For more information about creating comments by sending an email and the necessa To copy the task's email address: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy task email address**. diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index ade99a5fef8..8c6840fae92 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -23,7 +23,7 @@ Prerequisites: - To view storage usage for a project, you must have at least the Maintainer role for the project or Owner role for the namespace. - To view storage usage for a namespace, you must have the Owner role for the namespace. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. On the left sidebar, select **Settings > Usage Quotas**. 1. Select the **Storage** tab. @@ -179,7 +179,11 @@ available decreases. All projects no longer have the read-only status because 40 ## Namespace storage limit Namespaces on GitLab SaaS have a storage limit. For more information, see our [pricing page](https://about.gitlab.com/pricing/). -This limit is not visible on the **Usage quotas** page, but is prior to the limit being [applied](#namespace-storage-limit-application-schedule). Self-managed deployments are not affected. + +After namespace storage limits are enforced, view them in the **Usage quotas** page. +For more information about the namespace storage limit enforcement, see the FAQ pages for the [Free](https://about.gitlab.com/pricing/faq-efficient-free-tier/#storage-limits-on-gitlab-saas-free-tier) and [Paid](https://about.gitlab.com/pricing/faq-paid-storage-transfer/) tiers. + +Namespace storage limits do not apply to self-managed deployments, but administrators can [manage the repository size](../administration/settings/account_and_limit_settings.md#repository-size-limit). Storage types that add to the total namespace storage are: @@ -196,17 +200,20 @@ If your total namespace storage exceeds the available namespace storage quota, a To notify you that you have nearly exceeded your namespace storage quota: -- In the command line interface, a notification displays after each `git push` action when you've reached 95% and 100% of your namespace storage quota. +- In the command-line interface, a notification displays after each `git push` action when you've reached 95% and 100% of your namespace storage quota. - In the GitLab UI, a notification displays when you've reached 75%, 95%, and 100% of your namespace storage quota. - GitLab sends an email to members with the Owner role to notify them when namespace storage usage is at 70%, 85%, 95%, and 100%. -To prevent exceeding the namespace storage quota, you can: +To prevent exceeding the namespace storage limit, you can: -- Reduce storage consumption by following the suggestions in the [Manage Your Storage Usage](#manage-your-storage-usage) section of this page. -- Apply for [GitLab for Education](https://about.gitlab.com/solutions/education/join/), [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/join/), or [GitLab for Startups](https://about.gitlab.com/solutions/startups/) if you meet the eligibility requirements. -- Consider using a [self-managed instance](../subscriptions/self_managed/index.md) of GitLab which does not have these limits on the free tier. +- [Manage your storage usage](#manage-your-storage-usage). +- If you meet the eligibility requirements, you can apply for: + - [GitLab for Education](https://about.gitlab.com/solutions/education/join/) + - [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/join/) + - [GitLab for Startups](https://about.gitlab.com/solutions/startups/) +- Consider using a [self-managed instance](../subscriptions/self_managed/index.md) of GitLab, which does not have these limits on the Free tier. - [Purchase additional storage](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer) units at $60/year for 10 GB of storage. -- [Start a trial](https://about.gitlab.com/free-trial/) or [upgrade to GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) which include higher limits and features that enable growing teams to ship faster without sacrificing on quality. +- [Start a trial](https://about.gitlab.com/free-trial/) or [upgrade to GitLab Premium or Ultimate](https://about.gitlab.com/pricing/), which include higher limits and features to enable growing teams to ship faster without sacrificing on quality. - [Talk to an expert](https://page.gitlab.com/usage_limits_help.html) for more information about your options. ### View project fork storage usage @@ -215,14 +222,10 @@ A cost factor is applied to the storage consumed by project forks so that forks To view the amount of namespace storage the fork has used: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. On the left sidebar, select **Settings > Usage Quotas**. 1. Select the **Storage** tab. The **Total** column displays the amount of namespace storage used by the fork as a portion of the actual size of the fork on disk. The cost factor applies to the project repository, LFS objects, job artifacts, packages, snippets, and the wiki. The cost factor does not apply to private forks in namespaces on the Free plan. - -### Namespace storage limit application schedule - -Information on when namespace-level storage limits are applied is available on these FAQ pages for the [Free](https://about.gitlab.com/pricing/faq-efficient-free-tier/#storage-limits-on-gitlab-saas-free-tier) and [Paid](https://about.gitlab.com/pricing/faq-paid-storage-transfer/) tier. diff --git a/doc/user/workspace/configuration.md b/doc/user/workspace/configuration.md index 3900c95e41a..3684845c2c7 100644 --- a/doc/user/workspace/configuration.md +++ b/doc/user/workspace/configuration.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Workspace configuration (Beta) **(PREMIUM ALL)** +# Workspace configuration **(PREMIUM ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112397) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `remote_development_feature_flag`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/391543) in GitLab 16.0. @@ -25,6 +25,8 @@ which you can customize to meet the specific needs of each project. ## Set up a workspace +> Support for private projects [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124273) in GitLab 16.4. + ### Prerequisites - Set up a Kubernetes cluster that the GitLab agent for Kubernetes supports. @@ -48,8 +50,8 @@ which you can customize to meet the specific needs of each project. You can use any agent defined under the root group of your project, provided that remote development is properly configured for that agent. - You must have at least the Developer role in the root group. -- In each public project you want to use this feature for, create a [devfile](index.md#devfile): - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +- In each project you want to use this feature for, create a [devfile](index.md#devfile): + 1. On the left sidebar, select **Search or go to** and find your project. 1. In the root directory of your project, create a file named `.devfile.yaml`. You can use one of the [example configurations](index.md#example-configurations). - Ensure the container images used in the devfile support [arbitrary user IDs](index.md#arbitrary-user-ids). @@ -58,12 +60,11 @@ which you can customize to meet the specific needs of each project. To create a workspace: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Your work**. 1. Select **Workspaces**. 1. Select **New workspace**. 1. From the **Select project** dropdown list, [select a project with a `.devfile.yaml` file](#prerequisites). - You can only create workspaces for public projects. 1. From the **Select cluster agent** dropdown list, select a cluster agent owned by the group the project belongs to. 1. In **Time before automatic termination**, enter the number of hours until the workspace automatically terminates. This timeout is a safety measure to prevent a workspace from consuming excessive resources or running indefinitely. @@ -75,6 +76,8 @@ You also have access to the terminal and can install any necessary dependencies. ## Connect to a workspace with SSH +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10478) in GitLab 16.3. + Prerequisites: - SSH must be enabled for the workspace. diff --git a/doc/user/workspace/create_image.md b/doc/user/workspace/create_image.md index 43140a622e0..df70ff31194 100644 --- a/doc/user/workspace/create_image.md +++ b/doc/user/workspace/create_image.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Tutorial: Create a custom workspace image that supports arbitrary user IDs (Beta) **(PREMIUM ALL)** +# Tutorial: Create a custom workspace image that supports arbitrary user IDs **(PREMIUM ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112397) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `remote_development_feature_flag`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/391543) in GitLab 16.0. diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index 51e3e905a92..1284067a391 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Workspaces (Beta) **(PREMIUM ALL)** +# Workspaces **(PREMIUM ALL BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112397) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `remote_development_feature_flag`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/391543) in GitLab 16.0. @@ -35,7 +35,8 @@ When you [create a workspace](configuration.md#set-up-a-workspace), you must: - Assign the workspace to a specific project. - Select a project with a [`.devfile.yaml`](#devfile) file. -The workspace can then interact with the GitLab API based on the permissions granted to the current user. +The workspace can interact with the GitLab API, with the access level defined by current user permissions. +A running workspace remains accessible even if user permissions are later revoked. ### Open and manage workspaces from a project @@ -43,7 +44,7 @@ The workspace can then interact with the GitLab API based on the permissions gra To open a workspace from a file or the repository file list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. In the upper right, select **Edit**. 1. From the dropdown list, under **Your workspaces**, select the workspace. @@ -128,13 +129,15 @@ The Web IDE is the only code editor available for workspaces. The Web IDE is powered by the [GitLab VS Code fork](https://gitlab.com/gitlab-org/gitlab-web-ide-vscode-fork). For more information, see [Web IDE](../project/web_ide/index.md). -## Private repositories +## Personal access token -You cannot [create a workspace](configuration.md#set-up-a-workspace) for a private repository -because GitLab does not inject any credentials into the workspace. -You can only create a workspace for public repositories that have a devfile. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/129715) in GitLab 16.4. -From a workspace, you can clone any repository manually. +When you [create a workspace](configuration.md#set-up-a-workspace), you get a personal access token with `write_repository` permission. +This token is used to initially clone the project while starting the workspace. + +Any Git operation you perform in the workspace uses this token for authentication and authorization. +When you terminate the workspace, the token is revoked. ## Pod interaction in a cluster |
