diff options
Diffstat (limited to 'doc')
560 files changed, 11956 insertions, 7386 deletions
diff --git a/doc/.vale/gitlab/BadgeCapitalization.yml b/doc/.vale/gitlab/BadgeCapitalization.yml deleted file mode 100644 index a44bcbc0a7d..00000000000 --- a/doc/.vale/gitlab/BadgeCapitalization.yml +++ /dev/null @@ -1,14 +0,0 @@ ---- -# Error: gitlab.BadgeCapitalization -# -# Verifies that badges are not mixed case, which won't render properly. -# -# For a list of all options, see https://vale.sh/docs/topics/styles/ -extends: existence -message: "Capitalize the '%s' badge." -link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#product-tier-badges -level: error -scope: raw -raw: - - '(?!\*\*\((FREE|PREMIUM|ULTIMATE)( (SELF|SAAS|ALL) (BETA|EXPERIMENT))?\)\*\*)' - - '(?i)\*\*\((free|premium|ultimate)( (self|saas|all) (beta|experiment))?\)\*\*' diff --git a/doc/.vale/gitlab/LatinTerms.yml b/doc/.vale/gitlab/LatinTerms.yml index 0f098979b16..9fbaf278da9 100644 --- a/doc/.vale/gitlab/LatinTerms.yml +++ b/doc/.vale/gitlab/LatinTerms.yml @@ -15,4 +15,4 @@ swap: e\. g\.: for example i\.e\.: that is i\. e\.: that is - via: "Use 'with', 'through', or 'by using' instead." + via: "with', 'through', or 'by using" diff --git a/doc/.vale/gitlab/Uppercase.yml b/doc/.vale/gitlab/Uppercase.yml index b13ebe2c0a8..01837726b91 100644 --- a/doc/.vale/gitlab/Uppercase.yml +++ b/doc/.vale/gitlab/Uppercase.yml @@ -64,6 +64,8 @@ exceptions: - DORA - DSA - DSL + - DUOPRO + - DUOENT - DVCS - DVD - EBS diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index cc02821b6d6..b8cc7d4d890 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -284,6 +284,7 @@ devfiles DevOps Dhall dialogs +Diffblue disambiguates discoverability dismissable diff --git a/doc/administration/admin_area.md b/doc/administration/admin_area.md index ff18ac8ff3a..f9b26cd364d 100644 --- a/doc/administration/admin_area.md +++ b/doc/administration/admin_area.md @@ -470,3 +470,12 @@ The content of each log file is listed in chronological order. To minimize perfo ### Audit Events **(PREMIUM SELF)** The **Audit Events** page lists changes made within the GitLab server. With this information you can control, analyze, and track every change. + +### Statistics + +The **Instance overview** section of the Dashboard lists the current statistics of the GitLab instance. This information is retrieved using the [Application statistics API](../api/statistics.md#get-current-application-statistics). + +NOTE: +These statistics show exact counts for values less than 10,000. For values of 10,000 and higher, these statistics show approximate data +when [TablesampleCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/tablesample_count_strategy.rb?ref_type=heads#L16) and [ReltuplesCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/reltuples_count_strategy.rb?ref_type=heads) strategies are used for calculations. +. diff --git a/doc/administration/analytics/dev_ops_reports.md b/doc/administration/analytics/dev_ops_reports.md index 057dc5d48ee..313e99c1e57 100644 --- a/doc/administration/analytics/dev_ops_reports.md +++ b/doc/administration/analytics/dev_ops_reports.md @@ -49,18 +49,18 @@ feature is available. DevOps Adoption shows feature adoption for development, security, and operations. -| Category | Feature | -| --- | --- | -| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | -| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | -| Operations | Deployments<br>Pipelines<br>Runners | +| Category | Feature | +|-------------|---------| +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | You can use Group DevOps Adoption to: - Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on -their DevOps journey. + their DevOps journey. - Find subgroups that have adopted certain features, and provide guidance to other subgroups on -how to use those features. + how to use those features. - Verify if you are getting the return on investment that you expected from GitLab. ## Add or remove a group diff --git a/doc/administration/appearance.md b/doc/administration/appearance.md index 3599c444134..9ebc9a37407 100644 --- a/doc/administration/appearance.md +++ b/doc/administration/appearance.md @@ -6,107 +6,145 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Appearance **(FREE SELF)** -Several options are available for customizing the appearance of a self-managed instance -of GitLab. To access these settings: +You can update your settings to change the look and feel of your GitLab self-managed instance. + +To open the **Appearance** settings: 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > Appearance**. -## Navigation bar +## Customize your homepage button + +Customize the appearance of your **Homepage** button. + +The **Homepage** button is located on the upper-left corner of the left sidebar. +Replace the default **GitLab logo** **{tanuki}** with any image. + +- The file should be less than 1 MB. +- The image should be 28 pixels high. Images more than 28 px high will be resized. + +To customize your **Homepage** icon image: -By default, the navigation bar has the GitLab logo, but this can be customized with -any image desired. It is optimized for images 28px high (any width), but any image can be -used (less than 1 MB) and it is automatically resized. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Under **Navigation bar**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -After you select and upload an image, select **Update appearance settings** at the bottom -of the page to activate it in the GitLab instance. +Pipeline status emails also show your custom logo. However, some email applications do not support SVG images. If your custom image is in SVG format, pipeline emails show the default logo. -NOTE: -GitLab pipeline emails also display the custom logo, unless the logo is in SVG format. If the -custom logo is in SVG format, the default logo is used instead because the SVG format is not -supported by many email clients. +## Customize the favicon -## Favicon +Customize the appearance of the favicon. A favicon is the icon for a website that shows in your browser tabs. The **GitLab logo** **{tanuki}** is the default browser and CI/CD status favicon. Replace the default icon with any image that is `32 x 32` pixels and in `.png` or `.ico` format. -By default, the favicon (used by the browser as the tab icon and the CI status icon) -uses the GitLab logo. This can be customized with any icon desired. It must be a -32x32 `.png` or `.ico` image. +To change the favicon: -After you select and upload an icon, select **Update appearance settings** at the bottom -of the page to activate it in the GitLab instance. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Under **Favicon**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -## System header and footer messages +## Add system header and footer messages > **Enable header and footer in emails** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344819) in GitLab 15.9. -You can add a small header message, a small footer message, or both, to the interface -of your GitLab instance. These messages appear on all projects and pages of the -instance, including the sign-in/sign-up page. The default color is white text on -an orange background, but this can be customized by selecting **Customize colors**. +Add a small header message, a small footer message, or both, to the interface of your GitLab instance. These messages show on all projects and pages of the instance, such as the sign-in and register pages. -Limited [Markdown](../user/markdown.md) is supported, such as bold, italics, and links, for -example. Other Markdown features, including lists, images, and quotes are not supported -as the header and footer messages can only be a single line. +- You can italicize, bold, or add links to your message with Markdown. +- Markdown lists, images, and quotes are not supported because system messages must be a single line. -You can select **Enable header and footer in emails** to have the text of -the header and footer added to all emails sent by the GitLab instance. +To add a system header, footer message, or both: -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **System header and footer** section. +1. Complete the fields. +1. Optional. Select the **Enable header and footer in emails** checkbox. Add your system messages to all emails sent by your GitLab instance. +1. At the bottom of the page, select **Update appearance settings**. -## Sign-in / Sign-up pages +By default, the system header and footer text is white text on an orange background. To customize the message colors: -You can replace the default message on the sign-in/sign-up page with your own message -and logo. You can make full use of [Markdown](../user/markdown.md) in the description. +- Go to the **System header and footer** section and select **Customize colors**. -The optimal size for the logo is 128 x 128 pixels, but any image can be used (below 1 MB) -and it is resized automatically. The logo image appears between the title and -the description, on the left of the sign-up page. +## Customize your sign-in and register pages -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. You can also select **Sign-in page**, -to review the saved appearance settings: +Customize the title, description, and logo on the sign-in and register page. By default, the register page logo is located on the left of the page, between the title and the description. -NOTE: -You can add also add a [customized help message](settings/help_page.md) below the sign-in message or add [a Sign-in text message](settings/sign_in_restrictions.md#sign-in-information). +To customize your sign-in and register page titles or descriptions: -## Progressive Web App +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Sign in/Sign up pages** section. +1. Complete the fields. You can format the page **Title** and **Description** with Markdown. +1. At the bottom of the page, select **Update appearance settings**. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. +To customize the logo on your sign-in and register pages: -GitLab can be installed as a [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) (PWA). -Use the Progressive Web App settings to customize its appearance, including its name, -description, and icon. +- The file should be less than 1 MB. +- The image should be 128 pixels high. Images more than 128 px high will be resized. -### Configure the PWA settings +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Sign in/Sign up pages** section. +1. Under **Logo**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. -To configure the PWA settings: +You can add also add a [customized help message](settings/help_page.md) below the sign-in message or add [a sign-in text message](settings/sign_in_restrictions.md#sign-in-information). + +## Customize the Progressive Web App + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. + +Customize the icon, display name, short name, and description for your Progessive Web App (PWA). For more information, see [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps). + +To add a Progressive Web App name and short name: 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > Appearance**. -1. Scroll to the **Progressive Web App (PWA)** section. +1. Go to the **Progressive Web App (PWA)** section. 1. Complete the fields. - - **Icon**: If you use the standard GitLab icon, it is available in sizes 192x192 pixels, - 512x512 pixels, also as a maskable icon. If you use a custom icon, it must be in either size - 192x192 pixels, or 512x512 pixels. -1. Select **Update appearance settings**. + - **Name** is the display name of your PWA. + - **Short name** shows on mobile devices and small screens. +1. At the bottom of the page, select **Update appearance settings**. + +To add a Progressive Web App description: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Progressive Web App (PWA)** section. +1. Complete the fields. You can format the **Description** with Markdown. +1. At the bottom of the page, select **Update appearance settings**. + +To customize your Progressive Web App icon: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Progressive Web App (PWA)** section. +1. Under **Icon**, select **Choose file**. +1. At the bottom of the page, select **Update appearance settings**. + +## Add guidelines to the new project page -## New project pages +Add a guideline message to the **New project page**. You can format your message with Markdown. The guideline message shows under the **New Project** message and, on the left side of the **New project page**. -You can add a new project guidelines message to the **New project page** in GitLab. -You can make full use of [Markdown](../user/markdown.md) in the description: +To add a guideline message to the **New project page**: -The message is displayed below the **New Project** message, on the left side -of the **New project page**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **New project pages** section. +1. Complete the fields. You can format your guidelines with Markdown. -After you add a message, select **Update appearance settings** at the bottom of the page -to activate it in the GitLab instance. You can also select **New project page**, -which brings you to the new project page so you can review the change. +## Add profile image guidelines + +Add guidelines for profile images. + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Appearance**. +1. Go to the **Profile image guideline** section. +1. Complete the fields. You can format your text with Markdown. ## Libravatar -[Libravatar](https://www.libravatar.org) is supported by GitLab for avatar images, but you must -[manually enable Libravatar support on the GitLab instance](../administration/libravatar.md) to use the service. +GitLab supports [Libravatar](https://www.libravatar.org) is for avatar images, but you must manually enable Libravatar support on the GitLab instance. For more information, see [Libravatar](../administration/libravatar.md) to use the service. <!-- ## Troubleshooting diff --git a/doc/administration/audit_event_streaming/audit_event_types.md b/doc/administration/audit_event_streaming/audit_event_types.md index 28cab04553a..ff99fc1f6c6 100644 --- a/doc/administration/audit_event_streaming/audit_event_types.md +++ b/doc/administration/audit_event_streaming/audit_event_types.md @@ -64,12 +64,6 @@ Audit event types belong to the following product categories. | [`update_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74632) | Event triggered when an external audit event destination is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) | | [`update_instance_event_streaming_destination`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125846) | Event triggered when an instance level external audit event destination is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.2](https://gitlab.com/gitlab-org/gitlab/-/issues/404730) | -### Authorization - -| Name | Description | Saved to database | Streamed | Introduced in | -|:-----|:------------|:------------------|:---------|:--------------| -| [`member_role_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137087) | Event triggered when a custom role is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/issues/388934) | - ### Code review | Name | Description | Saved to database | Streamed | Introduced in | @@ -181,7 +175,7 @@ Audit event types belong to the following product categories. | [`ci_group_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a group's CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_group_variable_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a group's CI variable is updated| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_instance_variable_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is created| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | -| [`ci_instance_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI varialbe is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | +| [`ci_instance_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | | [`ci_instance_variable_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131882) | When an instance level CI variable is changed| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/8070) | | [`ci_variable_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a CI variable is created at a project level| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | | [`ci_variable_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91983) | Triggered when a project's CI variable is deleted| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.2](https://gitlab.com/gitlab-org/gitlab/-/issues/363090) | @@ -233,6 +227,12 @@ Audit event types belong to the following product categories. | [`feature_flag_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113453) | Triggered when a feature flag is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.10](https://gitlab.com/gitlab-org/gitlab/-/issues/374109) | | [`feature_flag_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113453) | Triggered when a feature flag is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [15.10](https://gitlab.com/gitlab-org/gitlab/-/issues/374109) | +### Fleet visibility + +| Name | Description | Saved to database | Streamed | Introduced in | +|:-----|:------------|:------------------|:---------|:--------------| +| [`ci_runner_usage_export`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139578) | Triggered when a runner usage report is generated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/426560) | + ### Fuzz testing | Name | Description | Saved to database | Streamed | Introduced in | @@ -326,6 +326,14 @@ Audit event types belong to the following product categories. |:-----|:------------|:------------------|:---------|:--------------| | [`experiment_features_enabled_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118222) | Event triggered on toggling setting for enabling experiment AI features| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/404856/) | +### Permissions + +| Name | Description | Saved to database | Streamed | Introduced in | +|:-----|:------------|:------------------|:---------|:--------------| +| [`member_role_created`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137087) | Event triggered when a custom role is created.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/issues/388934) | +| [`member_role_deleted`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141630) | Event triggered when a custom role is deleted.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.9](https://gitlab.com/gitlab-org/gitlab/-/issues/437672) | +| [`member_role_updated`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141630) | Event triggered when a custom role is updated.| **{check-circle}** Yes | **{check-circle}** Yes | GitLab [16.9](https://gitlab.com/gitlab-org/gitlab/-/issues/437672) | + ### Portfolio management | Name | Description | Saved to database | Streamed | Introduced in | diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md index 71ae33b3d87..2cfe71cd4a5 100644 --- a/doc/administration/audit_event_streaming/index.md +++ b/doc/administration/audit_event_streaming/index.md @@ -589,11 +589,8 @@ To delete Google Cloud Logging streaming destinations to an instance: ### AWS S3 destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138245) in GitLab 16.7 [with a flag](../feature_flags.md) named `allow_streaming_instance_audit_events_to_amazon_s3`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To enable the feature, an administrator can [enable the feature flag](../feature_flags.md) named -`allow_streaming_instance_audit_events_to_amazon_s3`. On GitLab.com, this feature is not available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138245) in GitLab 16.7 [with a flag](../feature_flags.md) named `allow_streaming_instance_audit_events_to_amazon_s3`. Disabled by default. +> - [Feature flag `allow_streaming_instance_audit_events_to_amazon_s3`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137391) removed in GitLab 16.8. Manage AWS S3 destinations for entire instance. diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index 95268f6f39b..dd7f9cec77e 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -1,6 +1,7 @@ --- stage: Govern group: Authentication +description: Third-party authentication providers. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index f4e7ea09615..4e000576652 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -4,7 +4,7 @@ group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Use JWT as an OAuth 2.0 authentication provider **(FREE SELF)** +# Use JWT as an authentication provider **(FREE SELF)** To enable the JWT OmniAuth provider, you must register your application with JWT. JWT provides you with a secret key for you to use. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 62395ebdcd2..49841b5e1be 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -1210,10 +1210,9 @@ For more information on synchronizing users and groups between LDAP and GitLab, ## Move from LDAP to SAML -1. [Configure SAML](../../../integration/saml.md). Add `auto_link_ldap_user` to: +1. [Add SAML configuration](../../../integration/saml.md) to: - [`gitlab.rb` for Linux package installations](../../../integration/saml.html?tab=Linux+package+%28Omnibus%29). - [`values.yml` for Helm chart installations](../../../integration/saml.html?tab=Helm+chart+%28Kubernetes%29). - For more information, see the [initial settings for all providers](../../../integration/omniauth.md#configure-initial-settings). 1. Optional. [Disable the LDAP auth from the sign-in page](#disable-ldap-web-sign-in). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 9cd306d979f..eb1ee203469 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -774,10 +774,10 @@ If a connection can't be established, it is likely either because of a problem with your configuration or a firewall blocking the connection. - Ensure you don't have a firewall blocking the -connection, and that the LDAP server is accessible to the GitLab host. + connection, and that the LDAP server is accessible to the GitLab host. - Look for an error message in the Rake check output, which may lead to your LDAP configuration to -confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and -`password`) are correct. + confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and + `password`) are correct. - Look for [errors](#connection) in [the logs](#gitlab-logs) to further debug connection failures. If GitLab can successfully connect to LDAP but doesn't return any diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index d20c9ee4412..80c3a21d953 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -4,7 +4,7 @@ group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Use OpenID Connect as an OAuth 2.0 authentication provider **(FREE SELF)** +# Use OpenID Connect as an authentication provider **(FREE SELF)** You can use GitLab as a client application with [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md index f38358810e3..ac04709f8b2 100644 --- a/doc/administration/backup_restore/backup_gitlab.md +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -136,13 +136,13 @@ For more information, see [Backup and restore Linux package (Omnibus) configurat :::TabTitle Docker - Back up the volume where the configuration files are stored. If you created -the GitLab container according to the documentation, it should be in the -`/srv/gitlab/config` directory. + the GitLab container according to the documentation, it should be in the + `/srv/gitlab/config` directory. :::TabTitle GitLab Helm chart - Follow the [Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#back-up-the-secrets) -instructions. + instructions. ::EndTabs @@ -150,8 +150,8 @@ You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, ` [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. -In the unlikely event that the secrets file is lost, see the -[troubleshooting section](#when-the-secrets-file-is-lost). +In the unlikely event that the secrets file is lost, see +[When the secrets file is lost](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). ### Other data @@ -656,7 +656,8 @@ sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=stora #### Back up specific repositories -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. +> - [Skipping specific repositories added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121865) in GitLab 16.1. You can back up specific repositories using the `REPOSITORIES_PATHS` option. Similarly, you can use `SKIP_REPOSITORIES_PATHS` to skip certain repositories. @@ -1451,618 +1452,3 @@ There are a few possible downsides to this: There is an **experimental** script that attempts to automate this process in [the Geo team Runbooks project](https://gitlab.com/gitlab-org/geo-team/runbooks/-/tree/main/experimental-online-backup-through-rsync). - -## Troubleshooting - -The following are possible problems you might encounter, along with potential -solutions. - -### When the secrets file is lost - -If you didn't [back up the secrets file](#storing-configuration-files), you -must complete several steps to get GitLab working properly again. - -The secrets file is responsible for storing the encryption key for the columns -that contain required, sensitive information. If the key is lost, GitLab can't -decrypt those columns, preventing access to the following items: - -- [CI/CD variables](../../ci/variables/index.md) -- [Kubernetes / GCP integration](../../user/infrastructure/clusters/index.md) -- [Custom Pages domains](../../user/project/pages/custom_domains_ssl_tls_certification/index.md) -- [Project error tracking](../../operations/error_tracking.md) -- [Runner authentication](../../ci/runners/index.md) -- [Project mirroring](../../user/project/repository/mirror/index.md) -- [Integrations](../../user/project/integrations/index.md) -- [Web hooks](../../user/project/integrations/webhooks.md) - -In cases like CI/CD variables and runner authentication, you can experience -unexpected behaviors, such as: - -- Stuck jobs. -- 500 errors. - -In this case, you must reset all the tokens for CI/CD variables and -runner authentication, which is described in more detail in the following -sections. After resetting the tokens, you should be able to visit your project -and the jobs begin running again. - -WARNING: -The steps in this section can potentially lead to **data loss** on the above listed items. -Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. - -#### Verify that all values can be decrypted - -You can determine if your database contains values that can't be decrypted by using a -[Rake task](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -#### Take a backup - -You must directly modify GitLab data to work around your lost secrets file. - -WARNING: -Be sure to create a full database backup before attempting any changes. - -#### Disable user two-factor authentication (2FA) - -Users with 2FA enabled can't sign in to GitLab. In that case, you must -[disable 2FA for everyone](../../security/two_factor_authentication.md#for-all-users), -after which users must reactivate 2FA. - -#### Reset CI/CD variables - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Examine the `ci_group_variables` and `ci_variables` tables: - - ```sql - SELECT * FROM public."ci_group_variables"; - SELECT * FROM public."ci_variables"; - ``` - - These are the variables that you need to delete. - -1. Delete all variables: - - ```sql - DELETE FROM ci_group_variables; - DELETE FROM ci_variables; - ``` - -1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: - - ```sql - DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; - DELETE FROM ci_variables WHERE project_id = <PROJECTID>; - ``` - -You may need to reconfigure or restart GitLab for the changes to take effect. - -#### Reset runner registration tokens - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Clear all tokens for projects, groups, and the entire instance: - - WARNING: - The final `UPDATE` operation stops the runners from being able to pick - up new jobs. You must register new runners. - - ```sql - -- Clear project tokens - UPDATE projects SET runners_token = null, runners_token_encrypted = null; - -- Clear group tokens - UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; - -- Clear instance tokens - UPDATE application_settings SET runners_registration_token_encrypted = null; - -- Clear key used for JWT authentication - -- This may break the $CI_JWT_TOKEN job variable: - -- https://gitlab.com/gitlab-org/gitlab/-/issues/325965 - UPDATE application_settings SET encrypted_ci_jwt_signing_key = null; - -- Clear runner tokens - UPDATE ci_runners SET token = null, token_encrypted = null; - ``` - -#### Reset pending pipeline jobs - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Clear all the tokens for pending jobs: - - For GitLab 15.3 and earlier: - - ```sql - -- Clear build tokens - UPDATE ci_builds SET token = null, token_encrypted = null; - ``` - - For GitLab 15.4 and later: - - ```sql - -- Clear build tokens - UPDATE ci_builds SET token_encrypted = null; - ``` - -A similar strategy can be employed for the remaining features. By removing the -data that can't be decrypted, GitLab can be returned to operation, and the -lost data can be manually replaced. - -#### Fix integrations and webhooks - -If you've lost your secrets, the [integrations settings](../../user/project/integrations/index.md) -and [webhooks settings](../../user/project/integrations/webhooks.md) pages might display `500` error messages. Lost secrets might also produce `500` errors when you try to access a repository in a project with a previously configured integration or webhook. - -The fix is to truncate the affected tables (those containing encrypted columns). -This deletes all your configured integrations, webhooks, and related metadata. -You should verify that the secrets are the root cause before deleting any data. - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - -1. Truncate the following tables: - - ```sql - -- truncate web_hooks table - TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE; - ``` - -### Container registry push failures after restoring from a backup - -If you use the [container registry](../../user/packages/container_registry/index.md), -pushes to the registry may fail after restoring your backup on a Linux package (Omnibus) -instance after restoring the registry data. - -These failures mention permission issues in the registry logs, similar to: - -```plaintext -level=error -msg="response completed with error" -err.code=unknown -err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" -err.message="unknown error" -``` - -This issue is caused by the restore running as the unprivileged user `git`, -which is unable to assign the correct ownership to the registry files during -the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). - -To get your registry working again: - -```shell -sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker -``` - -If you changed the default file system location for the registry, run `chown` -against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. - -### Backup fails to complete with Gzip error - -When running the backup, you may receive a Gzip error message: - -```shell -sudo /opt/gitlab/bin/gitlab-backup create -... -Dumping ... -... -gzip: stdout: Input/output error - -Backup failed -``` - -If this happens, examine the following: - -- Confirm there is sufficient disk space for the Gzip operation. It's not uncommon for backups that - use the [default strategy](#backup-strategy-option) to require half the instance size - in free disk space during backup creation. -- If NFS is being used, check if the mount option `timeout` is set. The - default is `600`, and changing this to smaller values results in this error. - -### Backup fails with `File name too long` error - -During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: - -```plaintext -Problem: <class 'OSError: [Errno 36] File name too long: -``` - -This problem stops the backup script from completing. To fix this problem, you must truncate the file names causing the problem. A maximum of 246 characters, including the file extension, is permitted. - -WARNING: -The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. -Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. - -Truncating file names to resolve the error involves: - -- Cleaning up remote uploaded files that aren't tracked in the database. -- Truncating the file names in the database. -- Rerunning the backup task. - -#### Clean up remote uploaded files - -A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). - -To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. - -1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: - - ```shell - bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production - ``` - -1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: - - WARNING: - The following action is **irreversible**. - - ```shell - bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false - ``` - -#### Truncate the file names referenced by the database - -You must truncate the files referenced by the database that are causing the problem. The file names referenced by the database are stored: - -- In the `uploads` table. -- In the references found. Any reference found from other database tables and columns. -- On the file system. - -Truncate the file names in the `uploads` table: - -1. Enter the database console: - - For the Linux package (Omnibus) GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - For the Linux package (Omnibus) GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - - For self-compiled installations, GitLab 14.2 and later: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production --database main - ``` - - For self-compiled installations, GitLab 14.1 and earlier: - - ```shell - sudo -u git -H bundle exec rails dbconsole -e production - ``` - -1. Search the `uploads` table for file names longer than 246 characters: - - The following query selects the `uploads` records with file names longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - SELECT - u.id, - u.path, - -- Current file name - (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, - -- New file name - CONCAT( - LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) AS new_filename, - -- New path - CONCAT( - COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) AS new_path - FROM uploads_with_long_filenames AS u - WHERE u.row_id > 0 AND u.row_id <= 10000; - ``` - - Output example: - - ```postgresql - -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - id | 34 - path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt - current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt - new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt - new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt - ``` - - Where: - - - `current_filename`: a file name that is currently more than 246 characters long. - - `new_filename`: a file name that has been truncated to 246 characters maximum. - - `new_path`: new path considering the `new_filename` (truncated). - - After you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -1. Rename the files found in the `uploads` table from long file names to new truncated file names. The following query rolls back the update so you can check the results safely in a transaction wrapper: - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - BEGIN; - WITH updated_uploads AS ( - UPDATE uploads - SET - path = - CONCAT( - COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) - FROM - uploads_with_long_filenames AS updatable_uploads - WHERE - uploads.id = updatable_uploads.id - AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 - RETURNING uploads.* - ) - SELECT id, path FROM updated_uploads; - ROLLBACK; - ``` - - After you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -1. Validate that the new file names from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: - - WARNING: - The following action is **irreversible**. - - ```sql - CREATE TEMP TABLE uploads_with_long_filenames AS - SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id - FROM uploads AS u - WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; - - CREATE INDEX ON uploads_with_long_filenames(row_id); - - UPDATE uploads - SET - path = - CONCAT( - COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), - CONCAT( - LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), - COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) - ) - ) - FROM - uploads_with_long_filenames AS updatable_uploads - WHERE - uploads.id = updatable_uploads.id - AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; - ``` - - After you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. - -Truncate the file names in the references found: - -1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and file name: - - 1. To dump your database, you can use the following command as an example: - - ```shell - pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp - ``` - - 1. Then you can search for the references using the `grep` command. Combining the parent directory and the file name can be a good idea. For example: - - ```shell - grep public/alongfilenamehere.txt gitlab-dump.tmp - ``` - -1. Replace those long file names using the new file names obtained from querying the `uploads` table. - -Truncate the file names on the file system. You must manually rename the files in your file system to the new file names obtained from querying the `uploads` table. - -#### Re-run the backup task - -After following all the previous steps, re-run the backup task. - -### Restoring database backup fails when `pg_stat_statements` was previously enabled - -The GitLab backup of the PostgreSQL database includes all SQL statements required to enable extensions that were -previously enabled in the database. - -The `pg_stat_statements` extension can only be enabled or disabled by a PostgreSQL user with `superuser` role. -As the restore process uses a database user with limited permissions, it can't execute the following SQL statements: - -```sql -DROP EXTENSION IF EXISTS pg_stat_statements; -CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; -``` - -When trying to restore the backup in a PostgreSQL instance that doesn't have the `pg_stats_statements` extension, -the following error message is displayed: - -```plaintext -ERROR: permission denied to create extension "pg_stat_statements" -HINT: Must be superuser to create this extension. -ERROR: extension "pg_stat_statements" does not exist -``` - -When trying to restore in an instance that has the `pg_stats_statements` extension enabled, the cleaning up step -fails with an error message similar to the following: - -```plaintext -rake aborted! -ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Caused by: -PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' -/opt/gitlab/embedded/bin/bundle:23:in `load' -/opt/gitlab/embedded/bin/bundle:23:in `<main>' -Tasks: TOP => gitlab:db:drop_tables -(See full trace by running task with --trace) -``` - -#### Prevent the dump file to include `pg_stat_statements` - -To prevent the inclusion of the extension in the PostgreSQL dump file that is part of the backup bundle, -enable the extension in any schema except the `public` schema: - -```sql -CREATE SCHEMA adm; -CREATE EXTENSION pg_stat_statements SCHEMA adm; -``` - -If the extension was previously enabled in the `public` schema, move it to a new one: - -```sql -CREATE SCHEMA adm; -ALTER EXTENSION pg_stat_statements SET SCHEMA adm; -``` - -To query the `pg_stat_statements` data after changing the schema, prefix the view name with the new schema: - -```sql -SELECT * FROM adm.pg_stat_statements limit 0; -``` - -To make it compatible with third-party monitoring solutions that expect it to be enabled in the `public` schema, -you need to include it in the `search_path`: - -```sql -set search_path to public,adm; -``` - -#### Fix an existing dump file to remove references to `pg_stat_statements` - -To fix an existing backup file, do the following changes: - -1. Extract from the backup the following file: `db/database.sql.gz`. -1. Decompress the file or use an editor that is capable of handling it compressed. -1. Remove the following lines, or similar ones: - - ```sql - CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; - ``` - - ```sql - COMMENT ON EXTENSION pg_stat_statements IS 'track planning and execution statistics of all SQL statements executed'; - ``` - -1. Save the changes and recompress the file. -1. Update the backup file with the modified `db/database.sql.gz`. diff --git a/doc/administration/backup_restore/restore_gitlab.md b/doc/administration/backup_restore/restore_gitlab.md index 2dd85602f99..0b5bf3cc0ff 100644 --- a/doc/administration/backup_restore/restore_gitlab.md +++ b/doc/administration/backup_restore/restore_gitlab.md @@ -38,8 +38,7 @@ before restoring the backup. To restore a backup, **you must also restore the GitLab secrets**. These include the database encryption key, [CI/CD variables](../../ci/variables/index.md), and variables used for [two-factor authentication](../../user/profile/account/two_factor_authentication.md). -Without the keys, [multiple issues occur](backup_gitlab.md#when-the-secrets-file-is-lost), -including loss of access by users with [two-factor authentication enabled](../../user/profile/account/two_factor_authentication.md), +Without the keys, [multiple issues occur](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost), including loss of access by users with [two-factor authentication enabled](../../user/profile/account/two_factor_authentication.md), and GitLab Runners cannot log in. Restore: @@ -282,7 +281,8 @@ project or group from there: the backed-up instance from which you want to restore. 1. [Restore the backup](#restore-gitlab) into this new instance, then export your [project](../../user/project/settings/import_export.md) - or [group](../../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). For more information about what is and isn't exported, see the export feature's documentation. + or [group](../../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated). For + more information about what is and isn't exported, see the export feature's documentation. 1. After the export is complete, go to the old instance and then import it. 1. After importing the projects or groups that you wanted is complete, you may delete the new, temporary GitLab instance. diff --git a/doc/administration/backup_restore/troubleshooting_backup_gitlab.md b/doc/administration/backup_restore/troubleshooting_backup_gitlab.md new file mode 100644 index 00000000000..83a02b52741 --- /dev/null +++ b/doc/administration/backup_restore/troubleshooting_backup_gitlab.md @@ -0,0 +1,619 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting GitLab backups + +When you back up GitLab, you might encounter the following issues. + +## When the secrets file is lost + +If you didn't [back up the secrets file](../../administration/backup_restore/backup_gitlab.md#storing-configuration-files), you +must complete several steps to get GitLab working properly again. + +The secrets file is responsible for storing the encryption key for the columns +that contain required, sensitive information. If the key is lost, GitLab can't +decrypt those columns, preventing access to the following items: + +- [CI/CD variables](../../ci/variables/index.md) +- [Kubernetes / GCP integration](../../user/infrastructure/clusters/index.md) +- [Custom Pages domains](../../user/project/pages/custom_domains_ssl_tls_certification/index.md) +- [Project error tracking](../../operations/error_tracking.md) +- [Runner authentication](../../ci/runners/index.md) +- [Project mirroring](../../user/project/repository/mirror/index.md) +- [Integrations](../../user/project/integrations/index.md) +- [Web hooks](../../user/project/integrations/webhooks.md) + +In cases like CI/CD variables and runner authentication, you can experience +unexpected behaviors, such as: + +- Stuck jobs. +- 500 errors. + +In this case, you must reset all the tokens for CI/CD variables and +runner authentication, which is described in more detail in the following +sections. After resetting the tokens, you should be able to visit your project +and the jobs begin running again. + +WARNING: +The steps in this section can potentially lead to **data loss** on the above listed items. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +### Verify that all values can be decrypted + +You can determine if your database contains values that can't be decrypted by using a +[Rake task](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + +### Take a backup + +You must directly modify GitLab data to work around your lost secrets file. + +WARNING: +Be sure to create a full database backup before attempting any changes. + +### Disable user two-factor authentication (2FA) + +Users with 2FA enabled can't sign in to GitLab. In that case, you must +[disable 2FA for everyone](../../security/two_factor_authentication.md#for-all-users), +after which users must reactivate 2FA. + +### Reset CI/CD variables + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Examine the `ci_group_variables` and `ci_variables` tables: + + ```sql + SELECT * FROM public."ci_group_variables"; + SELECT * FROM public."ci_variables"; + ``` + + These are the variables that you need to delete. + +1. Delete all variables: + + ```sql + DELETE FROM ci_group_variables; + DELETE FROM ci_variables; + ``` + +1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: + + ```sql + DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; + DELETE FROM ci_variables WHERE project_id = <PROJECTID>; + ``` + +You may need to reconfigure or restart GitLab for the changes to take effect. + +### Reset runner registration tokens + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all tokens for projects, groups, and the entire instance: + + WARNING: + The final `UPDATE` operation stops the runners from being able to pick + up new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear key used for JWT authentication + -- This may break the $CI_JWT_TOKEN job variable: + -- https://gitlab.com/gitlab-org/gitlab/-/issues/325965 + UPDATE application_settings SET encrypted_ci_jwt_signing_key = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` + +### Reset pending pipeline jobs + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all the tokens for pending jobs: + + For GitLab 15.3 and earlier: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token = null, token_encrypted = null; + ``` + + For GitLab 15.4 and later: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token_encrypted = null; + ``` + +A similar strategy can be employed for the remaining features. By removing the +data that can't be decrypted, GitLab can be returned to operation, and the +lost data can be manually replaced. + +### Fix integrations and webhooks + +If you've lost your secrets, the [integrations settings](../../user/project/integrations/index.md) +and [webhooks settings](../../user/project/integrations/webhooks.md) pages might display `500` error messages. Lost secrets might also produce `500` errors when you try to access a repository in a project with a previously configured integration or webhook. + +The fix is to truncate the affected tables (those containing encrypted columns). +This deletes all your configured integrations, webhooks, and related metadata. +You should verify that the secrets are the root cause before deleting any data. + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Truncate the following tables: + + ```sql + -- truncate web_hooks table + TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE; + ``` + +## Container registry push failures after restoring from a backup + +If you use the [container registry](../../user/packages/container_registry/index.md), +pushes to the registry may fail after restoring your backup on a Linux package (Omnibus) +instance after restoring the registry data. + +These failures mention permission issues in the registry logs, similar to: + +```plaintext +level=error +msg="response completed with error" +err.code=unknown +err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" +err.message="unknown error" +``` + +This issue is caused by the restore running as the unprivileged user `git`, +which is unable to assign the correct ownership to the registry files during +the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). + +To get your registry working again: + +```shell +sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker +``` + +If you changed the default file system location for the registry, run `chown` +against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. + +## Backup fails to complete with Gzip error + +When running the backup, you may receive a Gzip error message: + +```shell +sudo /opt/gitlab/bin/gitlab-backup create +... +Dumping ... +... +gzip: stdout: Input/output error + +Backup failed +``` + +If this happens, examine the following: + +- Confirm there is sufficient disk space for the Gzip operation. It's not uncommon for backups that + use the [default strategy](../../administration/backup_restore/backup_gitlab.md#backup-strategy-option) to require half the instance size + in free disk space during backup creation. +- If NFS is being used, check if the mount option `timeout` is set. The + default is `600`, and changing this to smaller values results in this error. + +## Backup fails with `File name too long` error + +During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: + +```plaintext +Problem: <class 'OSError: [Errno 36] File name too long: +``` + +This problem stops the backup script from completing. To fix this problem, you must truncate the file names causing the problem. A maximum of 246 characters, including the file extension, is permitted. + +WARNING: +The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +Truncating file names to resolve the error involves: + +- Cleaning up remote uploaded files that aren't tracked in the database. +- Truncating the file names in the database. +- Rerunning the backup task. + +### Clean up remote uploaded files + +A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). + +To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. + +1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production + ``` + +1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: + + WARNING: + The following action is **irreversible**. + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false + ``` + +### Truncate the file names referenced by the database + +You must truncate the files referenced by the database that are causing the problem. The file names referenced by the database are stored: + +- In the `uploads` table. +- In the references found. Any reference found from other database tables and columns. +- On the file system. + +Truncate the file names in the `uploads` table: + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For self-compiled installations, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + + For self-compiled installations, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + +1. Search the `uploads` table for file names longer than 246 characters: + + The following query selects the `uploads` records with file names longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + SELECT + u.id, + u.path, + -- Current file name + (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, + -- New file name + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) AS new_filename, + -- New path + CONCAT( + COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) AS new_path + FROM uploads_with_long_filenames AS u + WHERE u.row_id > 0 AND u.row_id <= 10000; + ``` + + Output example: + + ```postgresql + -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + id | 34 + path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + ``` + + Where: + + - `current_filename`: a file name that is currently more than 246 characters long. + - `new_filename`: a file name that has been truncated to 246 characters maximum. + - `new_path`: new path considering the `new_filename` (truncated). + + After you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Rename the files found in the `uploads` table from long file names to new truncated file names. The following query rolls back the update so you can check the results safely in a transaction wrapper: + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + BEGIN; + WITH updated_uploads AS ( + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 + RETURNING uploads.* + ) + SELECT id, path FROM updated_uploads; + ROLLBACK; + ``` + + After you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Validate that the new file names from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: + + WARNING: + The following action is **irreversible**. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; + ``` + + After you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +Truncate the file names in the references found: + +1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and file name: + + 1. To dump your database, you can use the following command as an example: + + ```shell + pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp + ``` + + 1. Then you can search for the references using the `grep` command. Combining the parent directory and the file name can be a good idea. For example: + + ```shell + grep public/alongfilenamehere.txt gitlab-dump.tmp + ``` + +1. Replace those long file names using the new file names obtained from querying the `uploads` table. + +Truncate the file names on the file system. You must manually rename the files in your file system to the new file names obtained from querying the `uploads` table. + +### Re-run the backup task + +After following all the previous steps, re-run the backup task. + +## Restoring database backup fails when `pg_stat_statements` was previously enabled + +The GitLab backup of the PostgreSQL database includes all SQL statements required to enable extensions that were +previously enabled in the database. + +The `pg_stat_statements` extension can only be enabled or disabled by a PostgreSQL user with `superuser` role. +As the restore process uses a database user with limited permissions, it can't execute the following SQL statements: + +```sql +DROP EXTENSION IF EXISTS pg_stat_statements; +CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; +``` + +When trying to restore the backup in a PostgreSQL instance that doesn't have the `pg_stats_statements` extension, +the following error message is displayed: + +```plaintext +ERROR: permission denied to create extension "pg_stat_statements" +HINT: Must be superuser to create this extension. +ERROR: extension "pg_stat_statements" does not exist +``` + +When trying to restore in an instance that has the `pg_stats_statements` extension enabled, the cleaning up step +fails with an error message similar to the following: + +```plaintext +rake aborted! +ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Caused by: +PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => gitlab:db:drop_tables +(See full trace by running task with --trace) +``` + +### Prevent the dump file to include `pg_stat_statements` + +To prevent the inclusion of the extension in the PostgreSQL dump file that is part of the backup bundle, +enable the extension in any schema except the `public` schema: + +```sql +CREATE SCHEMA adm; +CREATE EXTENSION pg_stat_statements SCHEMA adm; +``` + +If the extension was previously enabled in the `public` schema, move it to a new one: + +```sql +CREATE SCHEMA adm; +ALTER EXTENSION pg_stat_statements SET SCHEMA adm; +``` + +To query the `pg_stat_statements` data after changing the schema, prefix the view name with the new schema: + +```sql +SELECT * FROM adm.pg_stat_statements limit 0; +``` + +To make it compatible with third-party monitoring solutions that expect it to be enabled in the `public` schema, +you need to include it in the `search_path`: + +```sql +set search_path to public,adm; +``` + +### Fix an existing dump file to remove references to `pg_stat_statements` + +To fix an existing backup file, do the following changes: + +1. Extract from the backup the following file: `db/database.sql.gz`. +1. Decompress the file or use an editor that is capable of handling it compressed. +1. Remove the following lines, or similar ones: + + ```sql + CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; + ``` + + ```sql + COMMENT ON EXTENSION pg_stat_statements IS 'track planning and execution statistics of all SQL statements executed'; + ``` + +1. Save the changes and recompress the file. +1. Update the backup file with the modified `db/database.sql.gz`. diff --git a/doc/administration/configure.md b/doc/administration/configure.md index 8b4f8a9abc6..7ae088cd783 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Installation settings. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index ef9e53c8fb7..005017c6178 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -1,8 +1,8 @@ --- stage: SaaS Platforms group: GitLab Dedicated -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" -description: 'Learn how to configure your GitLab Dedicated instance.' +description: IP allowlists, SAML, maintenance. +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Configure GitLab Dedicated **(ULTIMATE)** @@ -14,12 +14,15 @@ The instructions on this page guide you through: 1. Onboarding and initial setup of your GitLab Dedicated instance using [Switchboard](https://about.gitlab.com/direction/saas-platforms/switchboard/), the GitLab Dedicated portal. 1. Configuring your GitLab Dedicated instance including enabling and updating the settings for [available functionality](../../subscriptions/gitlab_dedicated/index.md#available-features). -Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Panel](../../administration/admin_area.md). +Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Area](../../administration/admin_area.md). Examples of SaaS environment settings include `gitlab.rb` configurations and access to shell, Rails console, and PostgreSQL console. These environment settings cannot be changed by tenants. GitLab Dedicated Engineers also don't have direct access to tenant environments, except for [break glass situations](../../subscriptions/gitlab_dedicated/index.md#access-controls). +NOTE: +An instance refers to a GitLab Dedicated deployment, whereas a tenant refers to a customer. + ## Onboarding to GitLab Dedicated using Switchboard To create a new GitLab Dedicated environment for your organization, provide the following information to your account team: @@ -29,31 +32,33 @@ To create a new GitLab Dedicated environment for your organization, provide the - Email addresses of the users who are responsible to complete the onboarding and create your GitLab Dedicated instance using [Switchboard](https://about.gitlab.com/direction/saas-platforms/switchboard/). If you've been granted access to Switchboard, you receive an email invitation with temporary credentials to sign in. +Your invitation to Switchboard is valid for seven days. If you are having issues accessing +Switchboard, or if your invitation has expired, please [submit a support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). NOTE: The credentials for Switchboard are separate from any other GitLab credentials you may already have to sign in to a GitLab self-managed or GitLab.com instance. -After you first sign in to Switchboard, you must update your password and set up MFA before you can complete your onboarding to create a tenant. +After you first sign in to Switchboard, you must update your password and set up MFA before you can complete your onboarding to create a new instance. -The following stages guide you through a series of four steps to provide the information required to create your GitLab Dedicated tenant. +The following stages guide you through a series of four steps to provide the information required to create your GitLab Dedicated instance. 1. Confirm account details: Confirm key attributes of your GitLab Dedicated account: - Reference architecture: Corresponds with the number of users you provided to your account team when beginning the onboarding process. For more information, see [reference architectures](../../subscriptions/gitlab_dedicated/index.md#availability-and-scalability). - Total repository storage size: Corresponds with the storage size you provided to your account team when beginning the onboarding process. - If you need to make changes to these attributes, [submit a support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). -1. Tenant configuration: Provides the minimum required information needed to create your GitLab Dedicated tenant: +1. Tenant configuration: Provides the minimum required information needed to create your GitLab Dedicated instance: - Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You choose the subdomain name where your instance is accessible from. For example, `customer_name.gitlab-dedicated.com`. - Desired primary region: Primary AWS region in which your data is stored. Note the [available AWS regions](../../subscriptions/gitlab_dedicated/index.md#available-aws-regions). - Desired secondary region: Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster. - Desired backup region: An AWS region where the primary backups of your data are replicated. This can be the same as the primary or secondary region, or different. - Desired maintenance window: A weekly four-hour time slot that GitLab uses to perform routine maintenance and upgrade operations on all tenant instances. For more information, see [maintenance windows](#maintenance-window). 1. Security: You can provide your own [KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) for encrypted AWS services. If you choose not to provide KMS keys, encryption keys are generated for your instance when it is created. For more information, see [encrypting your data at rest](#encrypted-data-at-rest-byok). -1. Summary: You confirm that the information you've provided in the previous steps is accurate before initiating the creation of your tenant. +1. Summary: You confirm that the information you've provided in the previous steps is accurate before initiating the creation of your instance. NOTE: -Some configuration settings (like the option to bring your own keys and your tenant name) are permanent and cannot be changed once your tenant has been created. +Some configuration settings (like the option to bring your own keys and your tenant name) are permanent and cannot be changed once your instance has been created. -It can take up to 3 hours to create the GitLab Dedicated tenant. When the setup is complete, you will receive a confirmation email with further instructions on how to access your tenant. +It can take up to 3 hours to create the GitLab Dedicated instance. When the setup is complete, you will receive a confirmation email with further instructions on how to access your instance. ### Maintenance window @@ -80,10 +85,11 @@ In an event of a platform outage, degradation or a security event requiring urge emergency maintenance will be carried out per [the emergency change processes](https://about.gitlab.com/handbook/engineering/infrastructure/emergency-change-processes/). -The emergency maintenance is initiated urgently when urgent actions need to be executed by GitLab -on a Dedicated tenant instance. -Communication with the customer will be provided on best effort basis prior to commencing the -maintenance, and full communication will follow after the immediate action is carried out. +The emergency maintenance is initiated when urgent actions need to be executed by GitLab on a +Dedicated tenant instance. Communication with the customer will be provided on best effort basis +prior to commencing the maintenance, and full communication will follow after the immediate action +is carried out. The GitLab Support Team will create a new ticket and send a message to the email +addresses of the users listed in Switchboard during [onboarding](#onboarding-to-gitlab-dedicated-using-switchboard). For example, when a critical security process is initiated to address an S1 vulnerability in GitLab, emergency maintenance is carried out to upgrade GitLab to the non-vulnerable version and that @@ -218,7 +224,7 @@ Make sure the AWS KMS keys are replicated to your desired primary, secondary and Configuration changes requested with a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) are batched up and applied during your environment's weekly four-hour maintenance window. -This policy does not apply to configuration changes made by a GitLab Dedicated tenant admin [using Switchboard](#configuration-changes-in-switchboard). +This policy does not apply to configuration changes made by a GitLab Dedicated instance admin [using Switchboard](#configuration-changes-in-switchboard). To have a change considered for an upcoming weekly maintenance window, all required information must be submitted in full two business days before the start of the window. @@ -232,7 +238,7 @@ Changes requested with a support ticket cannot be applied outside of a weekly ma ### Configuration changes in Switchboard -Switchboard empowers the user to make limited configuration changes to their Dedicated Tenant Instance. As Switchboard matures further configuration changes will be made available. +Switchboard empowers the user to make limited configuration changes to their GitLab Dedicated instance. As Switchboard matures further configuration changes will be made available. To change or update the configuration of your GitLab Dedicated instance, use Switchboard following the instructions in the relevant section or open a [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) with your request. @@ -271,12 +277,14 @@ Consider the following when using Outbound Private Links: and repositories, or use any other GitLab functionality to access private services. - You can only establish Private Links between VPCs in the same region. Therefore, you can only establish a connection in the regions you selected for your Dedicated instance. -- The Network Load Balancer (NLB) that backs the Endpoint Service at your end must be enabled in at least one of the Availability Zones to which your Dedicated tenant was +- The Network Load Balancer (NLB) that backs the Endpoint Service at your end must be enabled in at least one of the Availability Zones to which your Dedicated instance was deployed. This is not the user-facing name such as `us-east-1a`, but the underlying [Availability Zone ID](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html). If you did not specify these during onboarding to Dedicated, you must either: - Ask for the Availability Zone IDs in the ticket you raise to enable the link and ensure the NLB is enabled in those AZs, or - Ensure the NLB has is enabled in every Availability Zone in the region. +You can view the `Reverse Private Link IAM Principal` attribute in the **Tenant Details** section of Switchboard. + To enable an Outbound Private Link: 1. [Create the Endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) through which your internal service @@ -300,7 +308,7 @@ To enable an Outbound Private Link: provide a list of DNS names that should resolve to the Private Link Endpoint. This list can be updated as needed in future. GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any matching outbound -connections made from the tenant GitLab instance are directed through the PrivateLink into your VPC. +connections made from the tenant instance are directed through the PrivateLink into your VPC. ### Custom certificates @@ -348,7 +356,7 @@ Specify a comma separated list of IP addresses that can access your GitLab Dedic ### SAML NOTE: -GitLab Dedicated supports a limited number of SAML parameters. Parameters not shown in the configuration below are unavailable for GitLab Dedicated tenant instances. +GitLab Dedicated supports a limited number of SAML parameters. Parameters not shown in the configuration below are unavailable for GitLab Dedicated instances. Prerequisites: @@ -415,7 +423,7 @@ To activate SAML for your GitLab Dedicated instance: #### Request signing -If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA)). +If [SAML request signing](../../integration/saml.md#sign-saml-authentication-requests-optional) is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA). If you choose to enable SAML request signing, the manual steps below will need to be completed before you are able to use SAML, since it requires certificate signing to happen. To enable SAML request signing, indicate on your SAML [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650) that you want request signing enabled. GitLab works with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA. After the certificate is signed, GitLab adds the certificate and its associated private key to the `security` section of the SAML configuration. Authentication requests from GitLab to your identity provider can then be signed. @@ -434,12 +442,12 @@ To enable group sync: 1. Add the [required elements](../../user/group/saml_sso/group_sync.md#configure-saml-group-sync) to the SAML configuration block you provide in your [support ticket](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=4414917877650). 1. Configure the [Group Links](../../user/group/saml_sso/group_sync.md#configure-saml-group-links). -### Add users to a tenant instance +### Add users to an instance -Tenant administrators can add Switchboard users to their tenant instance. There are two types of users: +Administrators can add Switchboard users to their GitLab Dedicated instance. There are two types of users: -- **Read only**: Users can only view tenant data. -- **Admin**: Users can edit the tenant configuration and manage users. +- **Read only**: Users can only view instance data. +- **Admin**: Users can edit the instance configuration and manage users. To add a new user to your GitLab Dedicated instance: diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 5b9bfe82294..30ec21ef660 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -175,7 +175,7 @@ documentation URL requests as needed. For example, if your GitLab version is - The GitLab documentation URL becomes `http://0.0.0.0:4000/14.5/`. - The link in GitLab displays as `<instance_url>/help/administration/settings/help_page#destination-requirements`. - When you select the link, you are redirected to -`http://0.0.0.0:4000/14.5/ee/administration/settings/help_page/#destination-requirements`. + `http://0.0.0.0:4000/14.5/ee/administration/settings/help_page/#destination-requirements`. To test the setting, in GitLab, select a **Learn more** link. For example: diff --git a/doc/administration/external_users.md b/doc/administration/external_users.md index d8951631b72..b6744e56f4f 100644 --- a/doc/administration/external_users.md +++ b/doc/administration/external_users.md @@ -15,12 +15,8 @@ External users: - Cannot create project, groups, and snippets in their personal namespaces. - Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. -- Can only access public projects and projects to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). -- Can only access public groups and groups to which they are explicitly granted access, - thus hiding all other internal or private ones from them (like being - logged out). +- Can access public groups and public projects. +- Can only access projects and groups to which they are explicitly granted access. External users cannot access internal or private projects or groups that they are not granted access to. - Can only access public snippets. Access can be granted by adding the user as member to the project or group. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f97c892c3eb..7949bd498a7 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -71,12 +71,6 @@ must disable the **primary** site. ### Step 3. Promoting a **secondary** site -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the secondary site is paused, be sure to resume before promoting. -This issue has been fixed in GitLab 13.4 and later. - Note the following when promoting a secondary: - If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion @@ -90,7 +84,7 @@ Note the following when promoting a secondary: [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-site). - You should [point the primary domain DNS at the newly promoted site](#step-4-optional-updating-the-primary-domain-dns-record). Otherwise, runners must be registered again with the newly promoted site, and all Git remotes, bookmarks, and external integrations must be updated. -#### Promoting a **secondary** site running on a single node running GitLab 14.5 and later +#### Promoting a **secondary** site running on a single node 1. SSH in to your **secondary** site and execute: @@ -110,63 +104,7 @@ Note the following when promoting a secondary: previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site running on a single node running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -1. SSH in to your **secondary** site and login as root: - - ```shell - sudo -i - ``` - -1. If you're using GitLab 13.5 and later, skip this step. If not, edit - `/etc/gitlab/gitlab.rb` and remove any of the following lines that - might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - -1. Promote the **secondary** site to the **primary** site: - - - To promote the secondary site to primary along with [preflight checks](planned_failover.md#preflight-checks): - - ```shell - gitlab-ctl promote-to-primary-node - ``` - - - If you have already run the preflight checks separately or don't want to run them, - you can skip them with: - - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` - - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync - and don't skip the preflight checks, promoting the secondary reports - `ERROR - Replication is not up-to-date` even if replication is actually - up-to-date. If replication and verification output - shows that it is complete, you can skip the preflight checks to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. - - - To promote the secondary site to primary **without any further confirmation**, - even when preflight checks fail: - - ```shell - gitlab-ctl promote-to-primary-node --force - ``` - -1. Verify you can connect to the newly-promoted **primary** site using the URL used - previously for the **secondary** site. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with multiple nodes running GitLab 14.5 and later +#### Promoting a **secondary** site with multiple nodes 1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands: @@ -200,49 +138,7 @@ Use `gitlab-ctl geo promote` instead. previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with multiple nodes running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with multiple nodes, as it can only perform changes on -a **secondary** with only a single node. Instead, you must -do this manually. - -1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any of the following - lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) - on each machine so the changes take effect. - -1. Promote the **secondary** to **primary**. SSH into a single application - server and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.5 and later +#### Promoting a **secondary** site with a Patroni standby cluster 1. SSH to every Sidekiq, PostgreSQL, and Gitaly node in the **secondary** site and run one of the following commands: @@ -276,54 +172,7 @@ do this manually. previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with a Patroni standby cluster, as it can only perform changes on -a **secondary** with only a single node. Instead, you must do this manually. - -1. SSH in to the Standby Leader database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every Patroni node in the secondary to disable the standby cluster: - - ```ruby - patroni['standby_cluster']['enable'] = false - ``` - -1. Reconfigure GitLab on each machine for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Promote the **secondary** to **primary**. SSH into a single application server and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - -#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.5 and later +#### Promoting a **secondary** site with an external PostgreSQL database The `gitlab-ctl geo promote` command can be used in conjunction with an external PostgreSQL database. In this case, you must first manually promote the replica database associated @@ -388,68 +237,6 @@ with the **secondary** site: previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.4 and earlier - -WARNING: -The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). -Use `gitlab-ctl geo promote` instead. - -The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with -an external PostgreSQL database, as it can only perform changes on a **secondary** -node with GitLab and the database on the same machine. As a result, a manual process is -required: - -1. Promote the replica database associated with the **secondary** site. This - sets the database to read-write. The instructions vary depending on where your database is hosted: - - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - [Azure PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal#stop-replication) - - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - - For other external PostgreSQL databases, save the following script in your - secondary site, for example `/tmp/geo_promote.sh`, and modify the connection - parameters to match your environment. Then, execute it to promote the replica: - - ```shell - #!/bin/bash - - PG_SUPERUSER=postgres - - # The path to your pg_ctl binary. You may need to adjust this path to match - # your PostgreSQL installation - PG_CTL_BINARY=/usr/lib/postgresql/10/bin/pg_ctl - - # The path to your PostgreSQL data directory. You may need to adjust this - # path to match your PostgreSQL installation. You can also run - # `SHOW data_directory;` from PostgreSQL to find your data directory - PG_DATA_DIRECTORY=/etc/postgresql/10/main - - # Promote the PostgreSQL database and allow read/write operations - sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any of the following - lines that might be present: - - ```ruby - geo_secondary_role['enable'] = true - roles ['geo_secondary_role'] - ``` - - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) - on each node so the changes take effect. - -1. Promote the **secondary** to **primary**. SSH into a single secondary application - node and execute: - - ```shell - sudo gitlab-rake geo:set_secondary_as_primary - ``` - -1. Verify you can connect to the newly-promoted **primary** using the URL used - previously for the **secondary**. -1. If successful, the **secondary** site is now promoted to the **primary** site. - ### Step 4. (Optional) Updating the primary domain DNS record Update DNS records for the primary domain to point to the **secondary** site. @@ -627,12 +414,6 @@ When updating a cloud-native Geo deployment, the process for updating any node t The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the site is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - ### Step 1. Permanently disable the **primary** cluster WARNING: @@ -671,10 +452,6 @@ If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming- a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. -::Tabs - -:::TabTitle For GitLab 14.5 and later - 1. For each node (such as PostgreSQL or Gitaly) outside of the **secondary** Kubernetes cluster using the Linux package, SSH into the node and run one of the following commands: @@ -709,46 +486,6 @@ Data that was created on the primary while the secondary was paused is lost. | ---- | ------------- | ------- | | `ENABLE_SILENT_MODE` | `false` | If `true`, enables [Silent Mode](../../silent_mode/index.md) before promotion (GitLab 16.4 and later) | -:::TabTitle For GitLab 14.4 and earlier - -1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to - promote to read-write: - - ```shell - sudo gitlab-ctl promote-db - ``` - -1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: - - NOTE: - Depending on your architecture, these steps need to run on any GitLab node that is external to the **secondary** Kubernetes cluster. - - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` - - After making these changes, [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) on the database node. - -1. Find the task runner pod: - - ```shell - kubectl --namespace gitlab get pods -lapp=task-runner - ``` - -1. Promote the secondary: - - ```shell - kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary - ``` - -::EndTabs - ### Step 3. Promote the **secondary** cluster 1. Update the existing cluster configuration. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 01e5c33a726..70bbc962ba0 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -68,19 +68,13 @@ site: ``` 1. Copy the backup tarball generated from your primary site to the `/var/opt/gitlab/backups` folder -on your secondary site. + on your secondary site. 1. On your secondary site, restore the registry following the [Restore GitLab](../../../administration/backup_restore/index.md#restore-gitlab) -documentation. + documentation. ## Preflight checks -NOTE: -In GitLab 13.7 and earlier, if you have a data type with zero items to sync, -this command reports `ERROR - Replication is not up-to-date` even if -replication is actually up-to-date. This bug was fixed in GitLab 13.8 and -later. - Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process goes smoothly: ```shell diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 6609bbed5a6..811ac92de27 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -337,18 +337,9 @@ For answers to common questions, see the [Geo FAQ](replication/faq.md). ## Log files -Geo stores structured log messages in a `geo.log` file. For Linux package -installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. +Geo stores structured log messages in a `geo.log` file. -This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. - -For example: - -```json -{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} -``` - -This message shows that Geo detected that a repository update was needed for project `1`. +For more information on how to access and consume Geo logs, see the [Geo section in the log system documentation](../logs/index.md#geolog). ## Troubleshooting diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index b9bee753089..1eacca32902 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -170,3 +170,70 @@ To verify container registry replication is working, on the **secondary** site: The initial replication, or "backfill", is probably still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. + +## Troubleshooting + +### Confirm that container registry replication is enabled + +This can be done with a check using the [Rails console](../../operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Geo::ContainerRepositoryRegistry.replication_enabled? +``` + +### Missing container registry notification event + +1. When an image is pushed to the primary site's container registry, it should trigger a [Container Registry notification](../../../administration/packages/container_registry.md#configure-container-registry-notifications) +1. The primary site's container registry calls the primary site's API on `https://<example.com>/api/v4/container_registry_event/events` +1. The primary site inserts a record to the `geo_events` table with `replicable_name: 'container_repository', model_record_id: <ID of the container repository>`. +1. The record gets replicated by PostgreSQL to the secondary site's database. +1. The Geo Log Cursor service processes the new event and enqueues a Sidekiq job `Geo::EventWorker` + +To verify this is working correctly, push an image to the registry on the primary site, and run the following command on the Rails console to verify that the notification was received, and processed into an event: + +```ruby +Geo::Event.where(replicable_name: 'container_repository') +``` + +You can further verify this by checking `geo.log` for entries from `Geo::ContainerRepositorySyncService`. + +### Registry events logs response status 401 Unauthorized unaccepted + +`401 Unauthorized` errors indicate that the primary site's container registry notification is not accepted by the Rails application, preventing it from notifying GitLab that something was pushed. + +To fix this, make sure that the authorization headers being sent with the registry notification match what's configured on the primary site, as should be done during step [Configure primary site](#configure-primary-site). + +#### Registry error: `token from untrusted issuer: "<token>"` + +To replicate a container image, Sidekiq uses JWT to authenticate itself towards the container registry. Geo replication takes it as a prerequisite that the [container registry configuration](../../../administration/packages/container_registry.md) has been done correctly. + +Make sure that both sites share a single signing key pair, as instructed under [Configure secondary site](#configure-secondary-site), and that both container registries, plus primary and secondary sites are [all configured to use the same token issuer](../../../administration/packages/container_registry.md#configure-gitlab-and-registry-to-run-on-separate-nodes-linux-package-installations). + +On multinode deployments, make sure that the issuer configured on the Sidekiq node matches the value configured on the registries. + +### Manually trigger a container registry sync event + +To help with troubleshooting, you can manually trigger the container registry replication process by running the following commands on the secondary's Rails console: + +```ruby +registry = Geo::ContainerRepositoryRegistry.first # Choose a Geo registry entry +registry.replicator.sync_repository # Resync the container repository +pp registry.reload # Look at replication state fields + +#<Geo::ContainerRepositoryRegistry:0x00007f54c2a36060 + id: 1, + container_repository_id: 1, + state: "2", + retry_count: 0, + last_sync_failure: nil, + retry_at: nil, + last_synced_at: Thu, 28 Sep 2023 19:38:05.823680000 UTC +00:00, + created_at: Mon, 11 Sep 2023 15:38:06.262490000 UTC +00:00> +``` + +The `state` field represents sync state: + +- `"0"`: pending sync (usually means it was never synced) +- `"1"`: started sync (a sync job is currently running) +- `"2"`: successfully synced +- `"3"`: failed to sync diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 6849a648991..2491b7c8bc8 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -51,8 +51,8 @@ In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. 1. Go to the -[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select -**Traffic policies**. + [Route53 dashboard](https://console.aws.amazon.com/route53/home) and select + **Traffic policies**.  diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index aae35d45bde..4a72dcf0145 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -15,7 +15,7 @@ to Geo secondaries. This may result in a larger than expected downtime. Upgrading Geo sites involves performing: -1. [Version-specific upgrade steps](version_specific_upgrades.md), depending on the +1. [Version-specific upgrade steps](../../../update/index.md#version-specific-upgrading-instructions), depending on the version being upgraded to or from. 1. [General upgrade steps](#general-upgrade-steps), for all upgrades. diff --git a/doc/administration/geo/replication/version_specific_upgrades.md b/doc/administration/geo/replication/version_specific_upgrades.md deleted file mode 100644 index 13d00ced3cd..00000000000 --- a/doc/administration/geo/replication/version_specific_upgrades.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../../update/index.md#version-specific-upgrading-instructions' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](../../../update/index.md#version-specific-upgrading-instructions). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 22bb7dd721d..376f3f3958e 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -79,11 +79,6 @@ In Kubernetes, you can [use the same domain under `global.hosts.domain` as for t NOTE: The feature flag described in this section is planned to be deprecated and removed in a future release. Support for read-only Geo secondary sites is proposed in [issue 366810](https://gitlab.com/gitlab-org/gitlab/-/issues/366810), you can upvote and share your use cases in that issue. -There are minor known issues linked in the -["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865). -You can also add feedback in the epic about any use-cases that -are not possible anymore with proxying enabled. - If you run into issues, to disable this feature, disable the `geo_secondary_proxy_separate_urls` feature flag. 1. SSH into one node running Rails on your primary Geo site and run: @@ -121,7 +116,7 @@ for details. To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate the certificate, then copy it to all other sites. -- Using Geo secondary sites to accelerate runners is not officially supported. Support for this functionality is planned and can be tracked in [epic 9779](https://gitlab.com/groups/gitlab-org/-/epics/9779). If a replication lag occurs between the primary and secondary site, and the pipeline ref is not available on the secondary site when the job is executed, the job will fail. +- Using Geo secondary sites to accelerate runners is experimental and is not recommended for production. It can be configured and tested by following the steps in [secondary proxy runners](runners.md). Progress toward general availability can be tracked in [epic 9779](https://gitlab.com/groups/gitlab-org/-/epics/9779). - When secondary proxying is used together with separate URLs, [signing in the secondary site using SAML](../replication/single_sign_on.md#saml-with-separate-url-with-proxying-enabled) @@ -150,25 +145,21 @@ It does not cover all data types. In this context, accelerated reads refer to read requests served from the secondary site, provided that the data is up to date for the component on the secondary site. If the data on the secondary site is determined to be out of date, the request is forwarded to the primary site. Read requests for components not listed in the table below are always automatically forwarded to the primary site. -| Feature / component | Accelerated reads? | -|:----------------------------------------------------|:-----------------------| -| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | -| Project, wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | -| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | -| Project, Personal Snippet (using Git) | **{check-circle}** Yes <sup>1</sup> | -| Group wiki repository (using the web UI) | **{dotted-circle}** No | -| Group wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | -| User uploads | **{dotted-circle}** No | -| LFS objects (using the web UI) | **{dotted-circle}** No | -| LFS objects (using Git) | **{check-circle}** Yes | -| Pages | **{dotted-circle}** No <sup>2</sup> | -| Advanced search (using the web UI) | **{dotted-circle}** No | -| Container registry | **{dotted-circle}** No <sup>3</sup>| - -1. Git reads are served from the local secondary while pushes get proxied to the primary. - Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. -1. Pages can use the same URL (without access control), but must be configured separately and are not proxied. -1. The container registry is only recommended for Disaster Recovery scenarios. If the secondary site's container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. +| Feature / component | Accelerated reads? | Notes | +|:----------------------------------------------------|:------------------------------------| ------------| +| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | | +| Project, wiki repository (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error.| +| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | | +| Project, Personal Snippet (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. | +| Group wiki repository (using the web UI) | **{dotted-circle}** No | | +| Group wiki repository (using Git) | **{check-circle}** Yes | Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. | +| User uploads | **{dotted-circle}** No | | +| LFS objects (using the web UI) | **{dotted-circle}** No | | +| LFS objects (using Git) | **{check-circle}** Yes | | +| Pages | **{dotted-circle}** No | Pages can use the same URL (without access control), but must be configured separately and are not proxied.| +| Advanced search (using the web UI) | **{dotted-circle}** No | | +| Container registry | **{dotted-circle}** No | The container registry is only recommended for Disaster Recovery scenarios. If the secondary site's container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. Accelerating the container registry is planned, please upvote or comment in the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/365864) to indicate your interest or ask your GitLab representative to do so on your behalf. | +| Dependency Proxy | **{dotted-circle}** No | Read requests to a Geo secondary site's Dependency Proxy are always proxied to the primary site. | ## Disable Geo proxying diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md index 4ff349d8741..2f55aa27dff 100644 --- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -47,8 +47,8 @@ In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. To create a traffic policy: 1. Go to the -[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select -**Traffic policies**. + [Route53 dashboard](https://console.aws.amazon.com/route53/home) and select + **Traffic policies**. 1. Select **Create traffic policy**. 1. Fill in the **Policy Name** field with `Single Git Host` and select **Next**. diff --git a/doc/administration/geo/secondary_proxy/runners.md b/doc/administration/geo/secondary_proxy/runners.md new file mode 100644 index 00000000000..6dfd61778e7 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/runners.md @@ -0,0 +1,40 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Secondary runners **(PREMIUM SELF EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415179) in GitLab 16.7 [with a flag](../../feature_flags.md) named `geo_proxy_check_pipeline_refs`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, an administrator can [enable the feature flag](../../feature_flags.md) named `geo_proxy_check_pipeline_refs`. On GitLab.com, this feature is not available. + +With [Geo proxying for secondary sites](index.md), it is possible to register a `gitlab-runner` with a secondary site. This offloads load from the primary instance. + +## Enable or disable secondary runners + +To enable secondary runners, SSH into a Rails node on the **primary** Geo site and run: + +```ruby +sudo gitlab-rails runner 'Feature.enable(:geo_proxy_check_pipeline_refs)' +``` + +To disable secondary runners, SSH into a Rails node on the **primary** Geo site and run: + +```ruby +sudo gitlab-rails runner `Feature.disable(:geo_proxy_check_pipeline_refs)` +``` + +## Use secondary runners with a Location Aware public URL (Unified URL) + +Using a [Location Aware public URL](location_aware_external_url.md), with the feature flag enabled works with no extra configuration. After you install and register a runner in the same location as a secondary site, it automatically talks to the closest site, and only proxies to the primary if the secondary is out of date. + +## Use secondary runners with separate URLs + +Using separate secondary URLs, the runners should be: + +1. Registered with the secondary external URL. +1. Configured with [`clone_url`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#how-clone_url-works) set to the `external_url` of the secondary instance. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 13615825a14..b9c2a69eaf7 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -622,7 +622,7 @@ If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to 1. Before migrating, you should ensure there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and doesn't drop it upon restarting. + to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and doesn't drop it upon restarting. 1. If database replication to the **secondary** site was paused before migration, resume replication after Patroni is confirmed as working on the **primary** site. ### Migrating a single PostgreSQL node to Patroni diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 336ed743a47..41783d55377 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -1,7 +1,8 @@ --- -info: For assistance with this tutorial, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none group: Tutorials +description: Administration overview. +info: For assistance with this tutorial, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # Get started administering GitLab **(FREE SELF)** @@ -150,7 +151,7 @@ Backups of GitLab databases and file systems are taken every 24 hours, and are k - You can use the project export option in: - [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data). - [The API](../api/project_import_export.md#schedule-an-export). -- [Group export by uploading a file export](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) +- [Group export by uploading a file export](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) does **not** export the projects in it, but does export: - Epics - Milestones diff --git a/doc/administration/gitaly/concurrency_limiting.md b/doc/administration/gitaly/concurrency_limiting.md index 321bb9efe20..2b1a2bbef64 100644 --- a/doc/administration/gitaly/concurrency_limiting.md +++ b/doc/administration/gitaly/concurrency_limiting.md @@ -18,7 +18,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. ## Limit RPC concurrency @@ -30,7 +30,7 @@ When cloning or pulling repositories, various RPCs run in the background. In par These RPCs can consume a large amount of resources, which can have a significant impact in situations such as: - Unexpectedly high traffic. -- Running against [large repositories](../../user/project/repository/managing_large_repositories.md) that don't follow best practices. +- Running against [large repositories](../../user/project/repository/monorepos/index.md) that don't follow best practices. You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in the Gitaly configuration file. For example: @@ -94,7 +94,7 @@ number of in-flight pack-object processes per remote IP address. WARNING: Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and -[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +[reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. Example configuration: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index d89413b2cf4..eb620ff7413 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -13,7 +13,7 @@ Configure Gitaly in one of two ways: :::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add or change the - [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). + [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -574,7 +574,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as processes may switch to using that instead of being terminated. This situation could lead to notably compromised @@ -1019,9 +1019,9 @@ meaning that unique requests do not get written into the cache. If you: - Increase this number, your cache hit rate goes down and the -cache uses less disk space. + cache uses less disk space. - Decrease this number, your cache hit -rate goes up and the cache uses more disk space. + rate goes up and the cache uses more disk space. You should set `min_occurrences` to `1`. On GitLab.com, going from 0 to 1 saved us 50% cache disk space while barely affecting @@ -1089,6 +1089,33 @@ Example: } ``` +## `cat-file` cache + +A lot of Gitaly RPCs need to look up Git objects from repositories. +Most of the time we use `git cat-file --batch` processes for that. For +better performance, Gitaly can re-use these `git cat-file` processes +across RPC calls. Previously used processes are kept around in a +["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). +To control how much system resources this uses, we have a maximum number of +cat-file processes that can go into the cache. + +The default limit is 100 `cat-file`s, which constitute a pair of +`git cat-file --batch` and `git cat-file --batch-check` processes. If +you see errors about "too many open files", or an +inability to create new processes, you may want to lower this limit. + +Ideally, the number should be large enough to handle standard +traffic. If you raise the limit, you should measure the cache hit ratio +before and after. If the hit ratio does not improve, the higher limit is +probably not making a meaningful difference. Here is an example +Prometheus query to see the hit rate: + +```plaintext +sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) +``` + +Configure the `cat-file` cache in the [Gitaly configuration file](reference.md). + ## Repository consistency checks Gitaly runs repository consistency checks: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 66e220dacc2..112c441164d 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -949,8 +949,8 @@ You can also appoint an authoritative name server by setting it in this format: 1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in -`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` -with Praefect service discovery address, such as `praefect.service.consul`. + `/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` + with Praefect service discovery address, such as `praefect.service.consul`. ```ruby git_data_dirs({ diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 946be6c8af3..169b6e00813 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -4,256 +4,24 @@ group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Gitaly reference **(FREE SELF)** +# Example configuration files **(FREE SELF)** -Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) -configuration file. Unlike self-compiled installations, in Linux package installations you -would not edit this file directly. For Linux package installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`. +Gitaly and Gitaly Cluster are configured by using configuration files. The default location of the configuration files +depends on the type of installation you have: -The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either your Linux -package installation or your [init](https://en.wikipedia.org/wiki/Init) script. +- For Linux package installations, the default location for Gitaly and Gitaly Cluster configuration is in the + `/etc/gitlab/gitlab.rb` Ruby file. +- For self-compiled, the default location for Gitaly and Gitaly Cluster configuration is in the + `/home/git/gitaly/config.toml` and `/home/git/gitaly/config.prafect.toml` TOML files. -An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) -can be found in the Gitaly project. +You can find example TOML configuration files in the `gitaly` project for: -## Format +- Gitaly: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example> +- Gitaly Cluster: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.praefect.toml.example> -At the top level, `config.toml` defines the items described on the table below. +If you are configuring a Linux package installation, you must convert the examples into Ruby to use them. -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `socket_path` | string | yes (if `listen_addr` is not set) | A path which Gitaly should open a Unix socket. | -| `listen_addr` | string | yes (if `socket_path` is not set) | TCP address for Gitaly to listen on. | -| `tls_listen_addr` | string | no | TCP over TLS address for Gitaly to listen on. | -| `bin_dir` | string | yes | Directory containing Gitaly executables. | -| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started. | +For more information on: -For example: - -```toml -socket_path = "/home/git/gitlab/tmp/sockets/private/gitaly.socket" -listen_addr = "localhost:9999" -tls_listen_addr = "localhost:8888" -bin_dir = "/home/git/gitaly" -prometheus_listen_addr = "localhost:9236" -``` - -### Authentication - -Gitaly can be configured to reject requests that do not contain a -specific bearer token in their headers, which is a security measure to -be used when serving requests over TCP: - -```toml -[auth] -# A non-empty token enables authentication. -token = "the secret token" -``` - -Authentication is disabled when the token setting in `config.toml` is absent or -an empty string. - -It is possible to temporarily disable authentication with the `transitioning` -setting. This allows you to monitor if all clients are -authenticating correctly without causing a service outage for clients -that are still to be configured correctly: - -```toml -[auth] -token = "the secret token" -transitioning = true -``` - -WARNING: -Remember to disable `transitioning` when you are done -changing your token settings. - -All authentication attempts are counted in Prometheus under -the [`gitaly_authentications_total` metric](monitoring.md#queries). - -### TLS - -Gitaly supports TLS encryption. You need to bring your own certificates as -this isn't provided automatically. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `certificate_path` | string | no | Path to the certificate. | -| `key_path` | string | no | Path to the key. | - -```toml -tls_listen_addr = "localhost:8888" - -[tls] -certificate_path = '/home/git/cert.cert' -key_path = '/home/git/key.pem' -``` - -[Read more](tls_support.md) about TLS in Gitaly. - -### Storage - -GitLab repositories are grouped into directories known as storages, such as -`/home/git/repositories`. They contain bare repositories managed -by GitLab with names, such as `default`. - -These names and paths are also defined in the `gitlab.yml` configuration file of -GitLab. When you run Gitaly on the same machine as GitLab (the default -and recommended configuration) storage paths defined in the Gitaly `config.toml` -must match those in `gitlab.yml`. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `storage` | array | yes | An array of storage shards. | -| `path` | string | yes | The path to the storage shard. | -| `name` | string | yes | The name of the storage shard. | - -For example: - -```toml -[[storage]] -path = "/path/to/storage/repositories" -name = "my_shard" - -[[storage]] -path = "/path/to/other/repositories" -name = "other_storage" -``` - -### Git - -The following values can be set in the `[git]` section of the configuration file. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. | -| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | -| `signing_key` | string | no | Path to [GPG signing key](configure_gitaly.md#configure-commit-signing-for-gitlab-ui-commits). If not set, Gitaly doesn't sign commits made using the UI. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. | - -#### `cat-file` cache - -A lot of Gitaly RPCs need to look up Git objects from repositories. -Most of the time we use `git cat-file --batch` processes for that. For -better performance, Gitaly can re-use these `git cat-file` processes -across RPC calls. Previously used processes are kept around in a -["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). -To control how much system resources this uses, we have a maximum number of -cat-file processes that can go into the cache. - -The default limit is 100 `cat-file`s, which constitute a pair of -`git cat-file --batch` and `git cat-file --batch-check` processes. If -you are seeing errors complaining about "too many open files", or an -inability to create new processes, you may want to lower this limit. - -Ideally, the number should be large enough to handle standard -traffic. If you raise the limit, you should measure the cache hit ratio -before and after. If the hit ratio does not improve, the higher limit is -probably not making a meaningful difference. Here is an example -Prometheus query to see the hit rate: - -```plaintext -sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) -``` - -#### Custom Hooks - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly supports [custom Git hooks](../server_hooks.md) that are -used to perform tasks based on changes performed in any repository. The custom -hooks directory is configured in the `[hooks]` section: - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. | - -Example: - -```toml -[hooks] -custom_hooks_dir = "/home/git/custom-hooks" -``` - -If left unset, no custom hooks are used. - -### GitLab - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly must connect to the GitLab application to perform access -checks when a user performs a change. The parameters to connect to GitLab must -be configured in the `[gitlab]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `url` | string | yes | The URL of the GitLab server. -| `secret` | string | no | The secret token used to authenticate with GitLab. | -| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. | - -Only one of `secret` or `secret_file` can be configured. - -### Prometheus - -You can optionally configure Gitaly to record histogram latencies on GRPC method -calls in Prometheus. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `grpc_latency_buckets` | array | no | Prometheus stores each observation in a bucket, which means you'd get an approximation of latency. Optimizing the buckets gives more control over the accuracy of the approximation. | - -Example: - -```toml -prometheus_listen_addr = "localhost:9236" - -[prometheus] -grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0] -``` - -### Logging - -The following values configure logging in Gitaly under the `[logging]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `format` | string | no | Log format: `text` or `json`. Default: `text`. | -| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | -| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. | -| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. | - -While the main Gitaly application logs go to `stdout`, there are some extra log -files that go to a configured directory, like the GitLab Shell logs. -GitLab Shell does not support `panic` or `trace` level logs: - -- `panic` falls back to `error`. -- `trace` falls back to `debug`. -- Any other invalid log levels default to `info`. - -Example: - -```toml -[logging] -level = "warn" -dir = "/home/gitaly/logs" -format = "json" -sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -ruby_sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -``` - -## Concurrency - -You can adjust the `concurrency` of each RPC endpoint. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `concurrency` | array | yes | An array of RPC endpoints. | -| `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). | -| `max_per_repo` | integer | no | Concurrency per RPC per repository. | - -Example: - -```toml -[[concurrency]] -rpc = "/gitaly.RepositoryService/GarbageCollect" -max_per_repo = 1 -``` +- Configuring Gitaly, see [Configure Gitaly](configure_gitaly.md). +- Configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 7d0255a3efb..6c355f79191 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -224,7 +224,7 @@ the application might be fetching this secret from a different file. Your Gitaly If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under `/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`. -### Repository pushes fail +### Repository pushes fail with `401 Unauthorized` and `JWT::VerificationError` When attempting `git push`, you can see: @@ -240,7 +240,7 @@ When attempting `git push`, you can see: } ``` -This error occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. +This combination of errors occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. From GitLab 15.5, GitLab [authenticates with GitLab Shell using a JWT token instead of a shared secret](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86148). You should follow the [recommendations on upgrading external Gitaly](../../update/plan_your_upgrade.md#external-gitaly) and upgrade Gitaly before the GitLab @@ -441,6 +441,23 @@ To resolve this, remove the `noexec` option from the file system mount. An alter Because Gitaly commit signing is headless and not associated with a specific user, the GPG signing key must be created without a passphrase, or the passphrase must be removed before export. +### Gitaly logs show errors in `info` messages + +Because of a bug [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6201) in GitLab 16.3, additional entries were written to the +[Gitaly logs](../logs/index.md#gitaly-logs). These log entries contained `"level":"info"` but the `msg` string appeared to contain an error. + +For example: + +```json +{"level":"info","msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: EOF\"","pid":6145,"system":"system","time":"2023-12-14T21:20:39.999Z"} +``` + +The reason for this log entry is that the underlying gRPC library sometimes output verbose transportation logs. These log entries appear to be errors but are, in general, +safe to ignore. + +This bug was [fixed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6513/) in GitLab 16.4.5, 16.5.5, and 16.6.0, which prevents these types of messages from +being written to the Gitaly logs. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index fbd3e636123..9e8bd323f5a 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -91,7 +91,7 @@ server to generate the diagrams: To run a PlantUML container in Docker, run this command: ```shell -docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat +docker run -d --name plantuml -p 8005:8080 plantuml/plantuml-server:tomcat ``` The **PlantUML URL** is the hostname of the server running the container. @@ -99,7 +99,7 @@ The **PlantUML URL** is the hostname of the server running the container. When running GitLab in Docker, it must have access to the PlantUML container. To achieve that, use [Docker Compose](https://docs.docker.com/compose/). In this basic `docker-compose.yml` file, PlantUML is accessible to GitLab at the URL -`http://plantuml:8080/`: +`http://plantuml:8005/`: ```yaml version: "3" @@ -108,7 +108,7 @@ services: image: 'gitlab/gitlab-ee:12.2.5-ee.0' environment: GITLAB_OMNIBUS_CONFIG: | - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8005/; \n}\n" plantuml: image: 'plantuml/plantuml-server:tomcat' @@ -125,6 +125,8 @@ following: - `http://plantuml:8080/` - `http://localhost:8080/plantuml/` +- `http://plantuml:8005/` +- `http://localhost:8005/plantuml/` If you're running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl/index.html) you must configure this redirection, because PlantUML uses the insecure HTTP protocol. @@ -137,7 +139,7 @@ To enable this redirection: ```ruby # Docker deployment - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8005/; \n}\n" ``` 1. To activate the changes, run the following command: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 33dc2020331..0658af2360b 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -282,7 +282,7 @@ To migrate back to local storage, you must If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. -Otherwise, they expire per the [default artifacts expiration setting](../administration/settings/continuous_integration.md). +Otherwise, they expire per the [default artifacts expiration setting](../administration/settings/continuous_integration.md#default-artifacts-expiration). Artifacts are deleted by the `expire_build_artifacts_worker` cron job which Sidekiq runs every 7 minutes (`*/7 * * * *` in [Cron](../topics/cron/index.md) syntax). diff --git a/doc/administration/job_artifacts_troubleshooting.md b/doc/administration/job_artifacts_troubleshooting.md index 138803a2675..6cdde87cc1d 100644 --- a/doc/administration/job_artifacts_troubleshooting.md +++ b/doc/administration/job_artifacts_troubleshooting.md @@ -24,7 +24,7 @@ reasons are: [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) to remove these. This script should always find work to do, as it also removes empty directories (see above). - [Artifact housekeeping was changed significantly](#housekeeping-disabled-in-gitlab-146-to-152), - and you might need to enable a feature flag to used the updated system. + and you might need to enable a feature flag to use the updated system. - The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs) feature is enabled. @@ -220,7 +220,7 @@ end ### List projects by total size of job artifacts stored List the top 20 projects, sorted by the total size of job artifacts stored, by -running the following code in the Rails console (`sudo gitlab-rails console`): +running the following code in the [Rails console](operations/rails_console.md): ```ruby include ActionView::Helpers::NumberHelper @@ -235,7 +235,7 @@ number you want. ### List largest artifacts in a single project List the 50 largest job artifacts in a single project by running the following -code in the Rails console (`sudo gitlab-rails console`): +code in the [Rails console](operations/rails_console.md): ```ruby include ActionView::Helpers::NumberHelper @@ -272,8 +272,11 @@ To change the number of job artifacts listed, change the number in `limit(50)`. WARNING: These commands remove data permanently from database and storage. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -If you need to manually remove job artifacts associated with multiple jobs while -**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): +You can manually remove job artifacts associated with multiple completed jobs while +**retaining their job logs** from the [Rails console](operations/rails_console.md). +A completed job is any job with the status of success, failed, canceled, or skipped. + +To delete jobs completed before a specific date: 1. Select jobs to be deleted: @@ -297,7 +300,7 @@ If you need to manually remove job artifacts associated with multiple jobs while ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts). ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| Ci::JobArtifacts::DeleteService.new(build).execute build.update!(artifacts_expire_at: Time.now) @@ -307,19 +310,16 @@ If you need to manually remove job artifacts associated with multiple jobs while In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/372537), use the following instead: ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| build.artifacts_expire_at = Time.now build.erase_erasable_artifacts! end ``` - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` + `1.year.ago` is a Rails [`ActiveSupport::Duration`](https://api.rubyonrails.org/classes/ActiveSupport/Duration.html) method. + Start with a long duration to reduce the risk of accidentally deleting artifacts that are still in use. + Rerun the deletion with shorter durations as needed, for example `3.months.ago`, `2.weeks.ago`, or `7.days.ago`. `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed; they are not scheduled by a background queue. @@ -329,8 +329,11 @@ If you need to manually remove job artifacts associated with multiple jobs while WARNING: These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -If you need to manually remove **all** job artifacts associated with multiple jobs, -**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): +You can manually remove job artifacts associated with multiple completed jobs while +**retaining their job logs** from the [Rails console](operations/rails_console.md). +A completed job is any job with the status of success, failed, canceled, or skipped. + +To delete jobs completed before a specific date: 1. Select the jobs to be deleted: @@ -371,7 +374,7 @@ If you need to manually remove **all** job artifacts associated with multiple jo 1. Erase the job artifacts and logs older than a specific date: ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.year.ago) builds_to_clear.find_each do |build| print "Ci::Build ID #{build.id}... " @@ -387,12 +390,9 @@ If you need to manually remove **all** job artifacts associated with multiple jo In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/369132), replace `Ci::BuildEraseService.new(build, admin_user).execute` with `build.erase(erased_by: admin_user)`. - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` + `1.year.ago` is a Rails [`ActiveSupport::Duration`](https://api.rubyonrails.org/classes/ActiveSupport/Duration.html) method. + Start with a long duration to reduce the risk of accidentally deleting artifacts that are still in use. + Rerun the deletion with shorter durations as needed, for example `3.months.ago`, `2.weeks.ago`, or `7.days.ago`. ## Job artifact upload fails with error 500 diff --git a/doc/administration/labels.md b/doc/administration/labels.md index 25a86b8c2fc..dcecbb84c3d 100644 --- a/doc/administration/labels.md +++ b/doc/administration/labels.md @@ -6,7 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Labels administration **(FREE SELF)** -To manage labels for the GitLab instance, in the Admin Area, on the left sidebar, select **Labels**. For more details on how to manage labels, see [Labels](../user/project/labels.md). +To manage labels for the GitLab instance: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Labels**. + +For more details on how to manage labels, see [Labels](../user/project/labels.md). Labels created in the Admin Area are automatically added to new projects. They are not available to new groups. diff --git a/doc/administration/license.md b/doc/administration/license.md index 1dc39d9ac23..9b95af28eed 100644 --- a/doc/administration/license.md +++ b/doc/administration/license.md @@ -36,6 +36,22 @@ To activate your instance with an activation code: The subscription is activated. +### Using one activation code for multiple instances + +You can use one activation code or license key for multiple self-managed instances if the users on +these instances are the same or are a subset of your licensed production instance. This means that if +you have a licensed production instance of GitLab, and other instances with the same list of users, the +production activation code applies, even if these users are configured in different groups and projects. + +### Uploading licenses for scaled architectures + +In a scaled architecture, upload the license file to one application instance only. The license is stored in the +database and is replicated to all your application instances so that you do not need to upload the license to all instances. + +### Uploading licenses for GitLab Geo + +When using GitLab Geo, you only need to upload the license to your primary Geo instance. The license is stored in the database and is replicated to all instances. + If you have an offline environment, [activate GitLab EE with a license file or key](license_file.md) instead. diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 1b1591a36d9..08f31439b26 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -15,8 +15,8 @@ This guide talks about how to read and use these system log files. Read more about the log system and using the logs: - [Customize logging on Linux package installations](https://docs.gitlab.com/omnibus/settings/logs.html) -including adjusting log retention, log forwarding, -switching logs from JSON to plain text logging, and more. + including adjusting log retention, log forwarding, + switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](../logs/log_parsing.md). ## Log Levels @@ -811,7 +811,7 @@ GraphQL queries are recorded in the file. For example: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133371) in GitLab 16.5. The `clickhouse.log` file logs information related to the -ClickHouse database client in GitLab. +[ClickHouse database client](../../integration/clickhouse.md) in GitLab. ## `migrations.log` diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md index f6b341ff114..a5f6b103d1f 100644 --- a/doc/administration/logs/tracing_correlation_id.md +++ b/doc/administration/logs/tracing_correlation_id.md @@ -36,7 +36,7 @@ To locate a relevant request and view its correlation ID: 1. To help isolate the requests you are looking for, you can filter for `document` requests. 1. Select the request of interest to view further detail. 1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a -value that was randomly generated by GitLab for the request. + value that was randomly generated by GitLab for the request. See the following example: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 80121c7c235..f1d1b504c9d 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -180,6 +180,8 @@ The following metrics are available: | `gitlab_connection_pool_size` | Gauge | 16.7 | Size of connection pool | | `gitlab_connection_pool_available_count` | Gauge | 16.7 | Number of available connections in the pool | | `gitlab_security_policies_scan_result_process_duration_seconds` | Histogram | 16.7 | The amount of time to process scan result policies | +| `gitlab_highlight_usage` | Counter | 16.8 | The number of times `Gitlab::Highlight` is used | `used_on` | +| `dependency_linker_usage` | Counter | 16.8 | The number of times dependency linker is used | `used_on` | ## Metrics controlled by a feature flag diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index bd3f2a20006..1fe05ed7538 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -246,14 +246,14 @@ To use an external Prometheus server: ``` 1. To allow the Prometheus server to fetch from the [GitLab metrics](#gitlab-metrics) endpoint, add the Prometheus -server IP address to the [monitoring IP allowlist](../ip_allowlist.md): + server IP address to the [monitoring IP allowlist](../ip_allowlist.md): ```ruby gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` 1. As we are setting each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, -update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports). + update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports). 1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) to apply the changes. 1. Edit the Prometheus server's configuration file. 1. Add each node's exporters to the Prometheus server's diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 9a9b2811cf0..0862921f0d7 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -345,7 +345,7 @@ gitlab_rails['object_store']['connection'] = { If you use ADC, be sure that: - The service account that you use has the -[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). + [`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). Typically this is done by granting the `Service Account Token Creator` role to the service account. - Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show: @@ -431,6 +431,29 @@ gitlab_rails['object_store']['connection'] = { The signature version must be `2`. Using v4 results in a HTTP 411 Length Required error. For more information, see [issue #4419](https://gitlab.com/gitlab-org/gitlab/-/issues/4419). +### Hitachi Vantara HCP + +NOTE: +Connections to HCP may return an error stating `SigntureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your HCP Secret Access key and signing method.` In these cases, set the `endpoint` to the URL of the tenant instead of the namespace, and ensure bucket paths are configured as `<namespace_name>/<bucket_name>`. + +[HCP](https://knowledge.hitachivantara.com/Documents/Storage/HCP_for_Cloud_Scale/1.0.0/Adminstering_HCP_for_cloud_scale/Getting_started/02_Support_for_Amazon_S3_API) provides an S3-compatible API. Use the following configuration example: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'endpoint' => 'https://<tenant_endpoint>', + 'path_style' => true, + 'region' => 'eu1', + 'aws_access_key_id' => 'ACCESS_KEY', + 'aws_secret_access_key' => 'SECRET_KEY', + 'aws_signature_version' => 4, + 'enable_signature_v4_streaming' => false +} + +# Example of <namespace_name/bucket_name> formatting +gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<namespace_name>/<bucket_name>' +``` + ## Full example using the consolidated form and Amazon S3 The following example uses AWS S3 to enable object storage for all supported services: @@ -874,28 +897,28 @@ When not proxying files, GitLab returns an This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate -`https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, generates this error: + `https->http` downgrade errors and refuse to process the redirect. The solution to this + is for GitLab to use HTTPS. LFS, for example, generates this error: - ```plaintext - LFS: lfsapi/client: refusing insecure redirect, https->http - ``` + ```plaintext + LFS: lfsapi/client: refusing insecure redirect, https->http + ``` - Clients need to trust the certificate authority that issued the object storage -certificate, or may return common TLS errors such as: + certificate, or may return common TLS errors such as: - ```plaintext - x509: certificate signed by unknown authority - ``` + ```plaintext + x509: certificate signed by unknown authority + ``` - Clients need network access to the object storage. -Network firewalls could block access. -Errors that might result -if this access is not in place include: + Network firewalls could block access. + Errors that might result + if this access is not in place include: - ```plaintext - Received status code 403 from server: Forbidden - ``` + ```plaintext + Received status code 403 from server: Forbidden + ``` - Object storage buckets need to allow Cross-Origin Resource Sharing (CORS) access from the URL of the GitLab instance. Attempting to load diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 86c6d9d7fba..a14f556af84 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -48,7 +48,7 @@ The following instructions enable `gitlab-sshd` on a different port than OpenSSH ``` 1. Optional. By default, Linux package installations generate SSH host keys for `gitlab-sshd` if -they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line: + they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line: ```ruby gitlab_sshd['generate_host_keys'] = false diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 33f407b2dbf..d1ec818a2a5 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Backup and restore, move repos, maintenance tasks. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 89b2f78f6d9..b85d6b8dd07 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -28,7 +28,6 @@ architecture. | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | | Debian 12 | GitLab CE / GitLab EE 16.1.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | TBD | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.4 | GitLab CE / GitLab EE 15.7.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Nov 2023 | <https://en.opensuse.org/Lifetime> | | OpenSUSE 15.5 | GitLab CE / GitLab EE 16.4.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap) | Dec 2024 | <https://en.opensuse.org/Lifetime> | | RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2029 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | | RHEL 9 | GitLab CE / GitLab EE 16.0.0 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2032 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | @@ -106,12 +105,13 @@ release for them can be found below: | Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.3&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | -| CentOS 8 | [December 2021](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=14.6&filter=all&filter=all&dist=el%2F8) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=14.6&filter=all&filter=all&dist=el%2F8) 14.6 | +| CentOS 8 | [December 2021](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=14.6&filter=all&filter=all&dist=el%2F8) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=14.6&filter=all&filter=all&dist=el%2F8) 14.6 | | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | | Ubuntu 16.04 | [April 2021](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.12&dist=ubuntu%2Fxenial) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.12&dist=ubuntu%2Fxenial) 13.12 | | OpenSUSE 15.2 | [December 2021](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-14.7&dist=opensuse%2F15.2) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-14.7&dist=opensuse%2F15.2) 14.7 | | Debian 9 "Stretch" | [June 2022](https://lists.debian.org/debian-lts-announce/2022/07/msg00002.html) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_15.2&dist=debian%2Fstretch) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_15.2&dist=debian%2Fstretch) 15.2 | -| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 | +| OpenSUSE 15.3 | [December 2022](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-15.10&dist=opensuse%2F15.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-15.10&dist=opensuse%2F15.3) 15.10 | +| OpenSUSE 15.4 | [December 2023](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-16.7&dist=opensuse%2F15.4) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-16.7&dist=opensuse%2F15.4) 16.7 | NOTE: An exception to this deprecation policy is when we are unable to provide diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 0594c4b93bd..150e0115eb7 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -249,7 +249,7 @@ outside world. ``` 1. If you haven't named your certificate and key `example.io.crt` and `example.io.key`, -you must also add the full paths as shown below: + you must also add the full paths as shown below: ```ruby pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" @@ -258,8 +258,8 @@ you must also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) -to use the HTTPS protocol. + [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) + to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. @@ -517,7 +517,7 @@ world. Custom domains and TLS are supported. If you don't have IPv6, you can omit the IPv6 address. 1. If you haven't named your certificate `example.io.crt` and your key `example.io.key`, -then you need to also add the full paths as shown below: + then you need to also add the full paths as shown below: ```ruby gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" @@ -526,8 +526,8 @@ then you need to also add the full paths as shown below: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages -[System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) -to use the HTTPS protocol. + [System OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) + to use the HTTPS protocol. ### Custom domain verification @@ -1011,25 +1011,20 @@ Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. Examples: - Increasing `gitlab_cache_expiry` allows items to exist in the cache longer. -This setting might be useful if the communication between GitLab Pages and GitLab Rails -is not stable. - + This setting might be useful if the communication between GitLab Pages and GitLab Rails + is not stable. - Increasing `gitlab_cache_refresh` reduces the frequency at which GitLab Pages -requests a domain's configuration from GitLab Rails. This setting might be useful -GitLab Pages generates too many requests to GitLab API and content does not change frequently. - + requests a domain's configuration from GitLab Rails. This setting might be useful + GitLab Pages generates too many requests to GitLab API and content does not change frequently. - Decreasing `gitlab_cache_cleanup` removes expired items from the cache more frequently, -reducing the memory usage of your Pages node. - + reducing the memory usage of your Pages node. - Decreasing `gitlab_retrieval_timeout` allows you to stop the request to GitLab Rails -more quickly. Increasing it allows more time to receive a response from the API, -useful in slow networking environments. - + more quickly. Increasing it allows more time to receive a response from the API, + useful in slow networking environments. - Decreasing `gitlab_retrieval_interval` makes requests to the API more frequently, -only when there is an error response from the API, for example a connection timeout. - + only when there is an error response from the API, for example a connection timeout. - Decreasing `gitlab_retrieval_retries` reduces the number of times a domain's -configuration is tried to be resolved automatically before reporting an error. + configuration is tried to be resolved automatically before reporting an error. ## Object storage settings @@ -1212,6 +1207,8 @@ TLS connection-based rate limits are enforced using the following: - `rate_limit_tls_domain`: Set the maximum threshold in number of TLS connections per hosted pages domain per second. Set to 0 to disable this feature. - `rate_limit_tls_domain_burst`: Sets the maximum threshold of number of TLS connections allowed in an initial outburst of TLS connections per hosted pages domain. +An IPv6 address receives a large prefix in the 128-bit address space. The prefix is typically at least size /64. Because of the large number of possible addresses, if the client's IP address is IPv6, the limit is applied to the IPv6 prefix with a length of 64, rather than the entire IPv6 address. + #### Enable HTTP requests rate limits by source-IP > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 2caa4f147c4..b91b0e573f0 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -1,4 +1,5 @@ --- + stage: Data Stores group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments @@ -35,8 +36,95 @@ databases. Some examples: ## Known issues - Once data is migrated to the `ci` database, you cannot migrate it back. +- HA setups or PgBouncer setups are not yet supported by this procedure. + +## Migrate existing installations using a script + +> A script for migrating existing Linux package installations was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368729) in GitLab 16.8. + +NOTE: +If something unexpected happens during the migration, it is safe to start over. + +### Existing Linux package installations + +#### Preparation + +1. Verify available disk space: + + - The database node that will store the `gitlabhq_production_ci` database needs enough space to store a copy of the existing database: we _duplicate_ `gitlabhq_production`. Run the following SQL query to find out how much space is needed. Add 25%, to ensure you will not run out of disk space. + + ```shell + sudo gitlab-psql -c "SELECT pg_size_pretty( pg_database_size('gitlabhq_production') );" + ``` -## Migrate existing installations + - During the process, a dump of the `gitlabhq_production` database needs to be temporarily stored on the filesystem of the node that will run the migration. Execute the following SQL statement to find out how much local disk space will be used. Add 25%, to ensure you will not run out of disk space. + + ```shell + sudo gitlab-psql -c "select sum(pg_table_size(concat(table_schema,'.',table_name))) from information_schema.tables where table_catalog = 'gitlabhq_production' and table_type = 'BASE TABLE'" + ``` + +1. Plan for downtime. The downtime is dependent on the size of the `gitlabhq_production` database. + + - We dump `gitlabhq_production` and restore it into a new `gitlabhq_production_ci` database. Database sizes below 100 GB should be done within 30 minutes. + - We advise to also plan some time for smaller tasks like modifying the configuration. + +1. Create the new `gitlabhq_production_ci` database: + + ```shell + sudo gitlab-psql -c "CREATE DATABASE gitlabhq_production_ci WITH OWNER 'gitlab'" + ``` + +#### Migration + +This process includes downtime. Running the migration script will stop the GitLab instance. After the migration has been finished, the instance is restarted. + +1. Create a backup of the configuration: + + ```shell + sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.org + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and save the changes. Do **not** run the reconfigure command, the migration script will run that for you. + + ```ruby + gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' } + gitlab_rails['databases']['ci']['enable'] = true + gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci' + ``` + +1. Run the migration script: + + ```shell + sudo gitlab-ctl pg-decomposition-migration + ``` + +At this point, the GitLab instance should start and be functional. + +If you want to abort the procedure and you want to start GitLab without changing anything, run the following commands: + +```shell +sudo cp /etc/gitlab/gitlab.rb.org /etc/gitlab/gitlab.rb +sudo gitlab-ctl reconfigure +sudo gitlab-ctl restart +``` + +#### Cleaning up + +If everything works as expected, we can clean up unneeded data: + +- Delete the CI data in Main database: + +```shell +sudo gitlab-rake gitlab:db:truncate_legacy_tables:main +``` + +- Delete the Main data in CI database: + +```shell +sudo gitlab-rake gitlab:db:truncate_legacy_tables:ci +``` + +## Migrate existing installations (manual procedure) To migrate existing data from the `main` database to the `ci` database, you can copy the database across. @@ -46,9 +134,9 @@ If something unexpected happens during the migration, it is safe to start over. ### Existing self-compiled installation -1. [Disable background migrations](../../development/database/batched_background_migrations.md#enable-or-disable-background-migrations) +1. [Disable background migrations](../../development/database/batched_background_migrations.md#enable-or-disable-background-migrations). -1. [Ensure all background migrations are finished](../../update/background_migrations.md#check-the-status-of-batched-background-migrations) +1. [Ensure all background migrations are finished](../../update/background_migrations.md#check-the-status-of-batched-background-migrations). 1. Stop GitLab, except for PostgreSQL: @@ -112,7 +200,7 @@ the other way around. ### Self-compiled installations 1. For existing installations, - [migrate the data](#migrate-existing-installations) first. + [migrate the data](#migrate-existing-installations-manual-procedure) first. 1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. @@ -168,7 +256,7 @@ the other way around. ### Linux package installations 1. For existing installations, - [migrate the data](#migrate-existing-installations) first. + [migrate the data](#migrate-existing-installations-manual-procedure) first. 1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index f4ed9d99b45..e93dfc8336e 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -563,8 +563,7 @@ gitlab-rake gitlab:db:configure > **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to PostgreSQL it may be that your PgBouncer node's IP address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before you proceed. ### Backups @@ -575,8 +574,7 @@ Do not backup or restore GitLab through a PgBouncer connection: this causes a Gi ### Ensure GitLab is running At this point, your GitLab instance should be up and running. Verify you're able -to sign in, and create issues and merge requests. If you encounter issues, see -the [Troubleshooting section](#troubleshooting). +to sign in, and create issues and merge requests. For more information, see [Troubleshooting replication and failover](../../administration/postgresql/replication_and_failover_troubleshooting.md). ## Example configuration @@ -901,7 +899,7 @@ Stopping or restarting the Patroni service on the leader node triggers an automa WARNING: In GitLab 16.5 and earlier, PgBouncer nodes do not automatically fail over alongside Patroni nodes. PgBouncer services -[must be restarted manually](#pgbouncer-errors-error-running-command-gitlabctlerrorsexecutionerror-and-error-database-gitlabhq_production-is-not-paused) +[must be restarted manually](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-errors-error-running-command-gitlabctlerrorsexecutionerror-and-error-database-gitlabhq_production-is-not-paused) for a successful switchover. While Patroni supports automatic failover, you also have the ability to perform @@ -1085,8 +1083,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: ``` If issues are encountered upgrading the replicas, -[there is a troubleshooting section](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) -that might be the solution. +[there is a troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#postgresql-major-version-upgrade-fails-on-a-patroni-replica) that might be the solution. NOTE: Reverting the PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as @@ -1274,468 +1271,3 @@ After completing these steps, then you can clean up the resources of the old Pat They are no longer needed. However, before removing the resources, remove the logical replication subscription on the new leader by running `DROP SUBSCRIPTION patroni_upgrade` with `gitlab-psql`. - -## Troubleshooting - -### Consul and PostgreSQL changes not taking effect - -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. - -To restart either service, run `gitlab-ctl restart SERVICE` - -For PostgreSQL, it is usually safe to restart the leader node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. - -On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. - -### PgBouncer error `ERROR: pgbouncer cannot connect to server` - -You may get this error when running `gitlab-rake gitlab:db:configure` or you -may see the error in the PgBouncer log file. - -```plaintext -PG::ConnectionBad: ERROR: pgbouncer cannot connect to server -``` - -The problem may be that your PgBouncer node's IP address is not included in the -`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. - -You can confirm that this is the issue by checking the PostgreSQL log on the leader -database node. If you see the following error then `trust_auth_cidr_addresses` -is the problem. - -```plaintext -2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off -``` - -To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. - -```ruby -postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) -``` - -[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. - -### PgBouncer errors `Error running command: GitlabCtl::Errors::ExecutionError` and `ERROR: database gitlabhq_production is not paused` - -Due to a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8166) that -affects versions of GitLab prior to 16.5.0, the automatic failover of PgBouncer nodes does not -happen after a [Patroni switchover](#manual-failover-procedure-for-patroni). In this -example, GitLab failed to detect a paused database, then attempted to `RESUME` a -not-paused database: - -```plaintext -INFO -- : Running: gitlab-ctl pgb-notify --pg-database gitlabhq_production --newhost database7.example.com --user pgbouncer --hostuser gitlab-consul -ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError -ERROR -- : STDERR: ERROR: ERROR: database gitlabhq_production is not paused -``` - -To ensure a [Patroni switchover](#manual-failover-procedure-for-patroni) succeeds, -you must manually restart the PgBouncer service on all PgBouncer nodes with this command: - -```shell -gitlab-ctl restart pgbouncer -``` - -### Reinitialize a replica - -If a replica cannot start or rejoin the cluster, or when it lags behind and cannot catch up, it might be necessary to reinitialize the replica: - -1. [Check the replication status](#check-replication-status) to confirm which server - needs to be reinitialized. For example: - - ```plaintext - + Cluster: postgresql-ha (6970678148837286213) ------+---------+--------------+----+-----------+ - | Member | Host | Role | State | TL | Lag in MB | - +-------------------------------------+--------------+---------+--------------+----+-----------+ - | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 55 | 0 | - | gitlab-database-2.example.com | 172.18.0.112 | Replica | start failed | | unknown | - | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 55 | | - +-------------------------------------+--------------+---------+--------------+----+-----------+ - ``` - -1. Sign in to the broken server and reinitialize the database and replication. Patroni shuts - down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch: - - ```shell - sudo gitlab-ctl patroni reinitialize-replica --member gitlab-database-2.example.com - ``` - - This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni - reinitialize-replica` without `--member` restarts the server it is run on. - You should run it locally on the broken server to reduce the risk of - unintended data loss. -1. Monitor the logs: - - ```shell - sudo gitlab-ctl tail patroni - ``` - -### Reset the Patroni state in Consul - -WARNING: -Resetting the Patroni state in Consul is a potentially destructive process. Make sure that you have a healthy database backup first. - -As a last resort you can reset the Patroni state in Consul completely. - -This may be required if your Patroni cluster is in an unknown or bad state and no node can start: - -```plaintext -+ Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ -| Member | Host | Role | State | TL | Lag in MB | -+-------------------------------------+--------------+---------+---------+----+-----------+ -| gitlab-database-1.example.com | 172.18.0.111 | Replica | stopped | | unknown | -| gitlab-database-2.example.com | 172.18.0.112 | Replica | stopped | | unknown | -| gitlab-database-3.example.com | 172.18.0.113 | Replica | stopped | | unknown | -+-------------------------------------+--------------+---------+---------+----+-----------+ -``` - -**Before deleting the Patroni state in Consul**, -[try and resolve the `gitlab-ctl` errors](#errors-running-gitlab-ctl) on the Patroni nodes. - -This process results in a reinitialized Patroni cluster when -the first Patroni node starts. - -To reset the Patroni state in Consul: - -1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, - if the current state shows more than one, or none: - - Look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, - which contains the hostname of the current leader. - - Look in the Patroni logs `/var/log/gitlab/patroni/current` (or the older rotated and - compressed logs `/var/log/gitlab/patroni/@40000*`) on **all** database nodes to see - which server was most recently identified as the leader by the cluster: - - ```plaintext - INFO: no action. I am a secondary (database1.local) and following a leader (database2.local) - ``` - -1. Stop Patroni on all nodes: - - ```shell - sudo gitlab-ctl stop patroni - ``` - -1. Reset the state in Consul: - - ```shell - /opt/gitlab/embedded/bin/consul kv delete -recurse /service/postgresql-ha/ - ``` - -1. Start one Patroni node, which initializes the Patroni cluster to elect as a leader. - It's highly recommended to start the previous leader (noted in the first step), - so as to not lose existing writes that may have not been replicated because - of the broken cluster state: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Start all other Patroni nodes that join the Patroni cluster as replicas: - - ```shell - sudo gitlab-ctl start patroni - ``` - -If you are still seeing issues, the next step is restoring the last healthy backup. - -### Errors in the Patroni log about a `pg_hba.conf` entry for `127.0.0.1` - -The following log entry in the Patroni log indicates the replication is not working -and a configuration change is needed: - -```plaintext -FATAL: no pg_hba.conf entry for replication connection from host "127.0.0.1", user "gitlab_replicator" -``` - -To fix the problem, ensure the loopback interface is included in the CIDR addresses list: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = %w(<other_cidrs> 127.0.0.1/32) - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -1. Check that [all the replicas are synchronized](#check-replication-status) - -### Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position - -This error indicates that the database is not replicating: - -```plaintext -FATAL: could not receive data from WAL stream: ERROR: requested starting point 0/5000000 is ahead of the WAL flush position of this server 0/4000388 -``` - -This example error is from a replica that was initially misconfigured, and had never replicated. - -Fix it [by reinitializing the replica](#reinitialize-a-replica). - -### Patroni fails to start with `MemoryError` - -Patroni may fail to start, logging an error and stack trace: - -```plaintext -MemoryError -Traceback (most recent call last): - File "/opt/gitlab/embedded/bin/patroni", line 8, in <module> - sys.exit(main()) -[..] - File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache - CFUNCTYPE(c_int)(lambda: None) -``` - -If the stack trace ends with `CFUNCTYPE(c_int)(lambda: None)`, this code triggers `MemoryError` -if the Linux server has been hardened for security. - -The code causes Python to write temporary executable files, and if it cannot find a file system in which to do this. For example, if `noexec` is set on the `/tmp` file system, it fails with `MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). - -Workarounds: - -- Remove `noexec` from the mount options for filesystems like `/tmp` and `/var/tmp`. -- If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting - SELinux to permissive. - -Patroni first shipped in the Linux package for GitLab 13.1, along with a build of Python 3.7. -The code which causes this was removed in Python 3.8: this fix shipped in -[the Linux package for GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) -and later, removing the need for a workaround. - -### Errors running `gitlab-ctl` - -Patroni nodes can get into a state where `gitlab-ctl` commands fail -and `gitlab-ctl reconfigure` cannot fix the node. - -If this co-incides with a version upgrade of PostgreSQL, [follow a different procedure](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) - -One common symptom is that `gitlab-ctl` cannot determine -information it needs about the installation if the database server is failing to start: - -```plaintext -Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/<HOSTNAME>.json. -This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. -``` - -```plaintext -Error while reinitializing replica on the current node: Attributes not found in -/opt/gitlab/embedded/nodes/<HOSTNAME>.json, has reconfigure been run yet? -``` - -Similarly, the nodes file (`/opt/gitlab/embedded/nodes/<HOSTNAME>.json`) should contain a lot of information, -but might get created with only: - -```json -{ - "name": "<HOSTNAME>" -} -``` - -The following process for fixing this includes reinitializing this replica: -the current state of PostgreSQL on this node is discarded: - -1. Shut down the Patroni and (if present) PostgreSQL services: - - ```shell - sudo gitlab-ctl status - sudo gitlab-ctl stop patroni - sudo gitlab-ctl stop postgresql - ``` - -1. Remove `/var/opt/gitlab/postgresql/data` in case its state prevents - PostgreSQL from starting: - - ```shell - cd /var/opt/gitlab/postgresql - sudo rm -rf data - ``` - - **Take care with this step to avoid data loss**. - This step can be also achieved by renaming `data/`: - make sure there's enough free disk for a new copy of the primary database, - and remove the extra directory when the replica is fixed. - -1. With PostgreSQL not running, the nodes file now gets created successfully: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Start Patroni: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Monitor the logs and check the cluster state: - - ```shell - sudo gitlab-ctl tail patroni - sudo gitlab-ctl patroni members - ``` - -1. Re-run `reconfigure` again: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Reinitialize the replica if `gitlab-ctl patroni members` indicates this is needed: - - ```shell - sudo gitlab-ctl patroni reinitialize-replica - ``` - -If this procedure doesn't work **and** if the cluster is unable to elect a leader, -[there is a another fix](#reset-the-patroni-state-in-consul) which should only be -used as a last resort. - -### PostgreSQL major version upgrade fails on a Patroni replica - -A Patroni **replica** can get stuck in a loop during `gitlab-ctl pg-upgrade`, and -the upgrade fails. - -An example set of symptoms is as follows: - -1. A `postgresql` service is defined, - which shouldn't usually be present on a Patroni node. It is present because - `gitlab-ctl pg-upgrade` adds it to create a new empty database: - - ```plaintext - run: patroni: (pid 1972) 1919s; run: log: (pid 1971) 1919s - down: postgresql: 1s, normally up, want up; run: log: (pid 1973) 1919s - ``` - -1. PostgreSQL generates `PANIC` log entries in - `/var/log/gitlab/postgresql/current` as Patroni is removing - `/var/opt/gitlab/postgresql/data` as part of reinitializing the replica: - - ```plaintext - DETAIL: Could not open file "pg_xact/0000": No such file or directory. - WARNING: terminating connection because of crash of another server process - LOG: all server processes terminated; reinitializing - PANIC: could not open file "global/pg_control": No such file or directory - ``` - -1. In `/var/log/gitlab/patroni/current`, Patroni logs the following. - The local PostgreSQL version is different from the cluster leader: - - ```plaintext - INFO: trying to bootstrap from leader 'HOSTNAME' - pg_basebackup: incompatible server version 12.6 - pg_basebackup: removing data directory "/var/opt/gitlab/postgresql/data" - ERROR: Error when fetching backup: pg_basebackup exited with code=1 - ``` - -**Important**: This workaround applies when the Patroni cluster is in the following state: - -- The [leader has been successfully upgraded to the new major version](#upgrading-postgresql-major-version-in-a-patroni-cluster). -- The step to upgrade PostgreSQL on replicas is failing. - -This workaround completes the PostgreSQL upgrade on a Patroni replica -by setting the node to use the new PostgreSQL version, and then reinitializing -it as a replica in the new cluster that was created -when the leader was upgraded: - -1. Check the cluster status on all nodes to confirm which is the leader - and what state the replicas are in - - ```shell - sudo gitlab-ctl patroni members - ``` - -1. Replica: check which version of PostgreSQL is active: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Replica: ensure the nodes file is correct and `gitlab-ctl` can run. This resolves - the [errors running `gitlab-ctl`](#errors-running-gitlab-ctl) issue if the replica - has any of those errors as well: - - ```shell - sudo gitlab-ctl stop patroni - sudo gitlab-ctl reconfigure - ``` - -1. Replica: relink the PostgreSQL binaries to the required version - to fix the `incompatible server version` error: - - 1. Edit `/etc/gitlab/gitlab.rb` and specify the required version: - - ```ruby - postgresql['version'] = 13 - ``` - - 1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. Check the binaries are relinked. The binaries distributed for - PostgreSQL vary between major releases, it's typical to - have a small number of incorrect symbolic links: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Replica: ensure PostgreSQL is fully reinitialized for the specified version: - - ```shell - cd /var/opt/gitlab/postgresql - sudo rm -rf data - sudo gitlab-ctl reconfigure - ``` - -1. Replica: optionally monitor the database in two additional terminal sessions: - - - Disk use increases as `pg_basebackup` runs. Track progress of the - replica initialization with: - - ```shell - cd /var/opt/gitlab/postgresql - watch du -sh data - ``` - - - Monitor the process in the logs: - - ```shell - sudo gitlab-ctl tail patroni - ``` - -1. Replica: Start Patroni to reinitialize the replica: - - ```shell - sudo gitlab-ctl start patroni - ``` - -1. Replica: After it completes, remove the hardcoded version from `/etc/gitlab/gitlab.rb`: - - 1. Edit `/etc/gitlab/gitlab.rb` and remove `postgresql['version']`. - 1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. Check the correct binaries are linked: - - ```shell - sudo ls -al /opt/gitlab/embedded/bin | grep postgres - ``` - -1. Check the cluster status on all nodes: - - ```shell - sudo gitlab-ctl patroni members - ``` - -Repeat this procedure on the other replica if required. - -### Issues with other components - -If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: - -- [Consul](../consul.md#troubleshooting-consul) -- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) diff --git a/doc/administration/postgresql/replication_and_failover_troubleshooting.md b/doc/administration/postgresql/replication_and_failover_troubleshooting.md new file mode 100644 index 00000000000..1b5aa5fc3b5 --- /dev/null +++ b/doc/administration/postgresql/replication_and_failover_troubleshooting.md @@ -0,0 +1,472 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting PostgreSQL replication and failover for Linux package installations **(PREMIUM SELF)** + +When working with PostgreSQL replication and failover, you might encounter the following issues. + +## Consul and PostgreSQL changes not taking effect + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the leader node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. + +On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. + +## PgBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PgBouncer log file. + +```plaintext +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PgBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the leader +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +```plaintext +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +```ruby +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. + +## PgBouncer errors `Error running command: GitlabCtl::Errors::ExecutionError` and `ERROR: database gitlabhq_production is not paused` + +Due to a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8166) that +affects versions of GitLab prior to 16.5.0, the automatic failover of PgBouncer nodes does not +happen after a [Patroni switchover](../../administration/postgresql/replication_and_failover.md#manual-failover-procedure-for-patroni). In this +example, GitLab failed to detect a paused database, then attempted to `RESUME` a +not-paused database: + +```plaintext +INFO -- : Running: gitlab-ctl pgb-notify --pg-database gitlabhq_production --newhost database7.example.com --user pgbouncer --hostuser gitlab-consul +ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError +ERROR -- : STDERR: ERROR: ERROR: database gitlabhq_production is not paused +``` + +To ensure a [Patroni switchover](../../administration/postgresql/replication_and_failover.md#manual-failover-procedure-for-patroni) succeeds, +you must manually restart the PgBouncer service on all PgBouncer nodes with this command: + +```shell +gitlab-ctl restart pgbouncer +``` + +## Reinitialize a replica + +If a replica cannot start or rejoin the cluster, or when it lags behind and cannot catch up, it might be necessary to reinitialize the replica: + +1. [Check the replication status](../../administration/postgresql/replication_and_failover.md#check-replication-status) to confirm which server + needs to be reinitialized. For example: + + ```plaintext + + Cluster: postgresql-ha (6970678148837286213) ------+---------+--------------+----+-----------+ + | Member | Host | Role | State | TL | Lag in MB | + +-------------------------------------+--------------+---------+--------------+----+-----------+ + | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 55 | 0 | + | gitlab-database-2.example.com | 172.18.0.112 | Replica | start failed | | unknown | + | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 55 | | + +-------------------------------------+--------------+---------+--------------+----+-----------+ + ``` + +1. Sign in to the broken server and reinitialize the database and replication. Patroni shuts + down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch: + + ```shell + sudo gitlab-ctl patroni reinitialize-replica --member gitlab-database-2.example.com + ``` + + This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni + reinitialize-replica` without `--member` restarts the server it is run on. + You should run it locally on the broken server to reduce the risk of + unintended data loss. +1. Monitor the logs: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +## Reset the Patroni state in Consul + +WARNING: +Resetting the Patroni state in Consul is a potentially destructive process. Make sure that you have a healthy database backup first. + +As a last resort you can reset the Patroni state in Consul completely. + +This may be required if your Patroni cluster is in an unknown or bad state and no node can start: + +```plaintext ++ Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ +| Member | Host | Role | State | TL | Lag in MB | ++-------------------------------------+--------------+---------+---------+----+-----------+ +| gitlab-database-1.example.com | 172.18.0.111 | Replica | stopped | | unknown | +| gitlab-database-2.example.com | 172.18.0.112 | Replica | stopped | | unknown | +| gitlab-database-3.example.com | 172.18.0.113 | Replica | stopped | | unknown | ++-------------------------------------+--------------+---------+---------+----+-----------+ +``` + +**Before deleting the Patroni state in Consul**, +[try and resolve the `gitlab-ctl` errors](#errors-running-gitlab-ctl) on the Patroni nodes. + +This process results in a reinitialized Patroni cluster when +the first Patroni node starts. + +To reset the Patroni state in Consul: + +1. Take note of the Patroni node that was the leader, or that the application thinks is the current leader, + if the current state shows more than one, or none: + - Look on the PgBouncer nodes in `/var/opt/gitlab/consul/databases.ini`, + which contains the hostname of the current leader. + - Look in the Patroni logs `/var/log/gitlab/patroni/current` (or the older rotated and + compressed logs `/var/log/gitlab/patroni/@40000*`) on **all** database nodes to see + which server was most recently identified as the leader by the cluster: + + ```plaintext + INFO: no action. I am a secondary (database1.local) and following a leader (database2.local) + ``` + +1. Stop Patroni on all nodes: + + ```shell + sudo gitlab-ctl stop patroni + ``` + +1. Reset the state in Consul: + + ```shell + /opt/gitlab/embedded/bin/consul kv delete -recurse /service/postgresql-ha/ + ``` + +1. Start one Patroni node, which initializes the Patroni cluster to elect as a leader. + It's highly recommended to start the previous leader (noted in the first step), + so as to not lose existing writes that may have not been replicated because + of the broken cluster state: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Start all other Patroni nodes that join the Patroni cluster as replicas: + + ```shell + sudo gitlab-ctl start patroni + ``` + +If you are still seeing issues, the next step is restoring the last healthy backup. + +## Errors in the Patroni log about a `pg_hba.conf` entry for `127.0.0.1` + +The following log entry in the Patroni log indicates the replication is not working +and a configuration change is needed: + +```plaintext +FATAL: no pg_hba.conf entry for replication connection from host "127.0.0.1", user "gitlab_replicator" +``` + +To fix the problem, ensure the loopback interface is included in the CIDR addresses list: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = %w(<other_cidrs> 127.0.0.1/32) + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. +1. Check that [all the replicas are synchronized](../../administration/postgresql/replication_and_failover.md#check-replication-status) + +## Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position + +This error indicates that the database is not replicating: + +```plaintext +FATAL: could not receive data from WAL stream: ERROR: requested starting point 0/5000000 is ahead of the WAL flush position of this server 0/4000388 +``` + +This example error is from a replica that was initially misconfigured, and had never replicated. + +Fix it [by reinitializing the replica](#reinitialize-a-replica). + +## Patroni fails to start with `MemoryError` + +Patroni may fail to start, logging an error and stack trace: + +```plaintext +MemoryError +Traceback (most recent call last): + File "/opt/gitlab/embedded/bin/patroni", line 8, in <module> + sys.exit(main()) +[..] + File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache + CFUNCTYPE(c_int)(lambda: None) +``` + +If the stack trace ends with `CFUNCTYPE(c_int)(lambda: None)`, this code triggers `MemoryError` +if the Linux server has been hardened for security. + +The code causes Python to write temporary executable files, and if it cannot find a file system in which to do this. For example, if `noexec` is set on the `/tmp` file system, it fails with `MemoryError` ([read more in the issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6184)). + +Workarounds: + +- Remove `noexec` from the mount options for filesystems like `/tmp` and `/var/tmp`. +- If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting + SELinux to permissive. + +Patroni first shipped in the Linux package for GitLab 13.1, along with a build of Python 3.7. +The code which causes this was removed in Python 3.8: this fix shipped in +[the Linux package for GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) +and later, removing the need for a workaround. + +## Errors running `gitlab-ctl` + +Patroni nodes can get into a state where `gitlab-ctl` commands fail +and `gitlab-ctl reconfigure` cannot fix the node. + +If this co-incides with a version upgrade of PostgreSQL, [follow a different procedure](#postgresql-major-version-upgrade-fails-on-a-patroni-replica) + +One common symptom is that `gitlab-ctl` cannot determine +information it needs about the installation if the database server is failing to start: + +```plaintext +Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/<HOSTNAME>.json. +This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. +``` + +```plaintext +Error while reinitializing replica on the current node: Attributes not found in +/opt/gitlab/embedded/nodes/<HOSTNAME>.json, has reconfigure been run yet? +``` + +Similarly, the nodes file (`/opt/gitlab/embedded/nodes/<HOSTNAME>.json`) should contain a lot of information, +but might get created with only: + +```json +{ + "name": "<HOSTNAME>" +} +``` + +The following process for fixing this includes reinitializing this replica: +the current state of PostgreSQL on this node is discarded: + +1. Shut down the Patroni and (if present) PostgreSQL services: + + ```shell + sudo gitlab-ctl status + sudo gitlab-ctl stop patroni + sudo gitlab-ctl stop postgresql + ``` + +1. Remove `/var/opt/gitlab/postgresql/data` in case its state prevents + PostgreSQL from starting: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + ``` + + **Take care with this step to avoid data loss**. + This step can be also achieved by renaming `data/`: + make sure there's enough free disk for a new copy of the primary database, + and remove the extra directory when the replica is fixed. + +1. With PostgreSQL not running, the nodes file now gets created successfully: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Start Patroni: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Monitor the logs and check the cluster state: + + ```shell + sudo gitlab-ctl tail patroni + sudo gitlab-ctl patroni members + ``` + +1. Re-run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Reinitialize the replica if `gitlab-ctl patroni members` indicates this is needed: + + ```shell + sudo gitlab-ctl patroni reinitialize-replica + ``` + +If this procedure doesn't work **and** if the cluster is unable to elect a leader, +[there is a another fix](#reset-the-patroni-state-in-consul) which should only be +used as a last resort. + +## PostgreSQL major version upgrade fails on a Patroni replica + +A Patroni **replica** can get stuck in a loop during `gitlab-ctl pg-upgrade`, and +the upgrade fails. + +An example set of symptoms is as follows: + +1. A `postgresql` service is defined, + which shouldn't usually be present on a Patroni node. It is present because + `gitlab-ctl pg-upgrade` adds it to create a new empty database: + + ```plaintext + run: patroni: (pid 1972) 1919s; run: log: (pid 1971) 1919s + down: postgresql: 1s, normally up, want up; run: log: (pid 1973) 1919s + ``` + +1. PostgreSQL generates `PANIC` log entries in + `/var/log/gitlab/postgresql/current` as Patroni is removing + `/var/opt/gitlab/postgresql/data` as part of reinitializing the replica: + + ```plaintext + DETAIL: Could not open file "pg_xact/0000": No such file or directory. + WARNING: terminating connection because of crash of another server process + LOG: all server processes terminated; reinitializing + PANIC: could not open file "global/pg_control": No such file or directory + ``` + +1. In `/var/log/gitlab/patroni/current`, Patroni logs the following. + The local PostgreSQL version is different from the cluster leader: + + ```plaintext + INFO: trying to bootstrap from leader 'HOSTNAME' + pg_basebackup: incompatible server version 12.6 + pg_basebackup: removing data directory "/var/opt/gitlab/postgresql/data" + ERROR: Error when fetching backup: pg_basebackup exited with code=1 + ``` + +**Important**: This workaround applies when the Patroni cluster is in the following state: + +- The [leader has been successfully upgraded to the new major version](../../administration/postgresql/replication_and_failover.md#upgrading-postgresql-major-version-in-a-patroni-cluster). +- The step to upgrade PostgreSQL on replicas is failing. + +This workaround completes the PostgreSQL upgrade on a Patroni replica +by setting the node to use the new PostgreSQL version, and then reinitializing +it as a replica in the new cluster that was created +when the leader was upgraded: + +1. Check the cluster status on all nodes to confirm which is the leader + and what state the replicas are in + + ```shell + sudo gitlab-ctl patroni members + ``` + +1. Replica: check which version of PostgreSQL is active: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure the nodes file is correct and `gitlab-ctl` can run. This resolves + the [errors running `gitlab-ctl`](#errors-running-gitlab-ctl) issue if the replica + has any of those errors as well: + + ```shell + sudo gitlab-ctl stop patroni + sudo gitlab-ctl reconfigure + ``` + +1. Replica: relink the PostgreSQL binaries to the required version + to fix the `incompatible server version` error: + + 1. Edit `/etc/gitlab/gitlab.rb` and specify the required version: + + ```ruby + postgresql['version'] = 13 + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the binaries are relinked. The binaries distributed for + PostgreSQL vary between major releases, it's typical to + have a small number of incorrect symbolic links: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Replica: ensure PostgreSQL is fully reinitialized for the specified version: + + ```shell + cd /var/opt/gitlab/postgresql + sudo rm -rf data + sudo gitlab-ctl reconfigure + ``` + +1. Replica: optionally monitor the database in two additional terminal sessions: + + - Disk use increases as `pg_basebackup` runs. Track progress of the + replica initialization with: + + ```shell + cd /var/opt/gitlab/postgresql + watch du -sh data + ``` + + - Monitor the process in the logs: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +1. Replica: Start Patroni to reinitialize the replica: + + ```shell + sudo gitlab-ctl start patroni + ``` + +1. Replica: After it completes, remove the hardcoded version from `/etc/gitlab/gitlab.rb`: + + 1. Edit `/etc/gitlab/gitlab.rb` and remove `postgresql['version']`. + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. Check the correct binaries are linked: + + ```shell + sudo ls -al /opt/gitlab/embedded/bin | grep postgres + ``` + +1. Check the cluster status on all nodes: + + ```shell + sudo gitlab-ctl patroni members + ``` + +Repeat this procedure on the other replica if required. + +## Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: + +- [Consul](../consul.md#troubleshooting-consul) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 207dbc7b509..39d7cae5dde 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -215,7 +215,7 @@ secrets file (`gitlab-secrets.json`). Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our -documentation on what to do [when the secrets file is lost](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost). +documentation on what to do [when the secrets file is lost](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). This can take a very long time, depending on the size of your database, as it checks all rows in all tables. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 3a136e773d9..21a7ba258c1 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -245,7 +245,7 @@ clear Redis' cache. To do this, run: Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. -This Rake task only applies to self-compiled installations. [Read more](../../update/package/index.md#missing-asset-files) +This Rake task only applies to self-compiled installations. [Read more](../../update/package/package_troubleshooting.md#missing-asset-files) about troubleshooting this problem when running the Linux package. The guidance for Linux package might be applicable for Kubernetes and Docker deployments of GitLab, though in general, container-based installations @@ -400,15 +400,13 @@ To manually rebuild a database index: ### Notes - Rebuilding database indexes is a disk-intensive task, so you should perform the -task during off-peak hours. Running the task during peak hours can lead to -_increased_ bloat, and can also cause certain queries to perform slowly. - + task during off-peak hours. Running the task during peak hours can lead to + _increased_ bloat, and can also cause certain queries to perform slowly. - The task requires free disk space for the index being restored. The created -indexes are appended with `_ccnew`. If the reindexing task fails, re-running the -task cleans up the temporary indexes. - + indexes are appended with `_ccnew`. If the reindexing task fails, re-running the + task cleans up the temporary indexes. - The time it takes for database index rebuilding to complete depends on the size -of the target database. It can take between several hours and several days. + of the target database. It can take between several hours and several days. ## Dump the database schema diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index defc74b7e9e..41897a999ec 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -707,7 +707,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -721,6 +721,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -820,7 +824,8 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1765,6 +1770,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1880,6 +1889,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1896,11 +1914,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2023,6 +2036,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2266,7 +2287,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2290,7 +2311,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 6520f63957b..2f0b51e6662 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -715,7 +715,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -729,6 +729,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -828,7 +832,8 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1500,7 +1505,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1773,6 +1778,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1888,6 +1897,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1904,11 +1922,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2033,6 +2046,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2276,7 +2297,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2300,7 +2321,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d3f53ce3e14..dd71190b76d 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -24,7 +24,7 @@ For a full list of reference architectures, see | Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | -| Gitaly | 1 | 4 vCPU, 15 GB memory<sup>5</sup> | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory<sup>5</sup> | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Sidekiq<sup>6</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails<sup>6</sup> | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -35,7 +35,6 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instance](#provide-your-own-redis-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. -4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly specifications are based on the use of normal-sized repositories in good health. However, if you have large monorepos (larger than several gigabytes) this can **significantly** impact Git and Gitaly performance and an increase of specifications will likely be required. Refer to [large monorepos](index.md#large-monorepos) for more information. @@ -52,7 +51,7 @@ skinparam linetype ortho card "**External Load Balancer**" as elb #6a9be7 -collections "**GitLab Rails** x3" as gitlab #32CD32 +collections "**GitLab Rails** x2" as gitlab #32CD32 card "**Prometheus**" as monitor #7FFFD4 card "**Gitaly**" as gitaly #FF8C00 card "**PostgreSQL**" as postgres #4EA7FF @@ -340,6 +339,10 @@ are supported and can be added if needed. In this section, you'll be guided through configuring an external Redis instance to be used with GitLab. +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + ### Provide your own Redis instance You can optionally use a [third party external service for the Redis instance](../redis/replication_and_failover_external.md#redis-as-a-managed-service-in-a-cloud-provider) with the following guidance: @@ -603,6 +606,10 @@ Sidekiq requires connection to the [Redis](#configure-redis), [PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances. It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + To configure the Sidekiq server, on the server node you want to use for Sidekiq: 1. SSH in to the Sidekiq server. @@ -687,6 +694,14 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -717,11 +732,6 @@ Updates to example must be made at: run: sidekiq: (pid 26870) 92996s; run: log: (pid 26391) 93042s ``` -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -820,6 +830,15 @@ On each node perform the following: } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -1098,7 +1117,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 1.9 vCPU, 5.5 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index c01564a3e7a..5d0ab62d4a2 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -433,6 +433,10 @@ NOTE: Multi-node Redis must be deployed in an odd number of 3 nodes or more to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -977,7 +981,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -991,6 +995,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1415,7 +1423,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1692,6 +1700,10 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.71`: Sidekiq 1 @@ -1800,6 +1812,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1831,11 +1852,6 @@ Updates to example must be made at: run: sidekiq: (pid 30142) 77351s; run: log: (pid 29638) 77386s ``` -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -1969,6 +1985,14 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2250,7 +2274,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 92421a35273..ed66620119e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -43,7 +43,7 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. @@ -717,7 +717,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -731,6 +731,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -830,7 +834,9 @@ NOTE: Redis clusters must each be deployed in an odd number of 3 nodes or more. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. NOTE: -Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. +Redis is primarily single threaded and doesn't significantly benefit from increasing CPU cores. +For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance at this scale. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more @@ -1505,7 +1511,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1778,6 +1784,11 @@ NOTE: [Because it's recommended to use Object storage](../object_storage.md) instead of NFS for data objects, the following examples include the Object storage configuration. +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues +you can scale it accordingly. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1893,6 +1904,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1909,11 +1929,6 @@ Updates to example must be made at: 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -NOTE: -If you find that the environment's Sidekiq job processing is slow with long queues, -more nodes can be added as required. You can also tune your Sidekiq nodes to -run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -2044,7 +2059,16 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -2284,7 +2308,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to specific nodes. A minimum of three nodes per node group in three different availability zones is strongly recommended to align with resilient cloud architecture practices. @@ -2308,7 +2332,7 @@ services where applicable): <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) for more information. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) for more information. - - Redis is primarily single threaded. It's strongly recommended separating out the instances as specified into Cache and Persistent data to achieve optimum performance at this scale. + - Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. 4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 16a92944984..65b46474f7a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -429,6 +429,14 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Multi-node Redis must be deployed in an odd number of 3 nodes or more to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, +such as a cloud provider service. + +NOTE: +Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -972,7 +980,7 @@ SSH in to any of the Patroni nodes on the **primary site**: ``` If the 'State' column for any node doesn't say "running", check the -[PostgreSQL replication and failover troubleshooting section](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +[PostgreSQL replication and failover troubleshooting section](../../administration/postgresql/replication_and_failover_troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) before proceeding. <div align="right"> @@ -986,6 +994,10 @@ before proceeding. Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. +NOTE: +PgBouncer is single threaded and doesn't significantly benefit from an increase in CPU cores. +Refer to the [scaling documentation](index.md#scaling-an-environment) for more information. + The following IPs are used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1411,7 +1423,7 @@ Updates to example must be made at: ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace -the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. 1. Praefect requires to run some database migrations, much like the main GitLab application. For this you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node @@ -1687,7 +1699,7 @@ examples include the Object storage configuration. - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -To configure the Sidekiq nodes, one each one: +To configure the Sidekiq nodes, on each one: 1. SSH in to the Sidekiq server. 1. [Download and install](https://about.gitlab.com/install/) the Linux package @@ -1791,6 +1803,15 @@ Updates to example must be made at: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace @@ -1948,7 +1969,15 @@ On each node perform the following: 'google_json_key_location' => '<path-to-gcp-service-account-key>' } gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "gcp-ci_secure_files-bucket-name" + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -2220,7 +2249,7 @@ the overall makeup as desired as long as the minimum CPU and Memory requirements | Supporting services | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) -[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. + [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index c4827695716..7e4e929f80d 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Recommended deployments at scale. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -13,8 +14,7 @@ GitLab Quality Engineering and Support teams to provide recommended deployments The following Reference Architectures are available as recommended starting points for your environment. -The architectures are named in terms of user count, which in this case means the architecture is designed against -the _total_ load that comes with such a user count based on real data along with substantial headroom added to cover most scenarios such as CI or other automated workloads. +The architectures are named in terms of _total_ load, both manual and automated, correlated to user count and based on real data along with substantial headroom added to add additional coverage for most scenarios. However, it should be noted that in some cases, known heavy scenarios such as [large monorepos](#large-monorepos) or notable [additional workloads](#additional-workloads) may require adjustments to be made. @@ -679,32 +679,44 @@ You should upgrade a Reference Architecture in the same order as you created it. ### Scaling an environment -Scaling a GitLab environment is designed to be as flexible and seamless as possible. +The Reference Architectures have been designed to support scaling in various ways depending on your use case and circumstances. +This can be done iteratively or wholesale to the next size of architecture depending on if metrics suggest a component is being exhausted. -This can be done iteratively or wholesale to the next size of architecture depending on your circumstances. -For example, if any of your GitLab Rails, Sidekiq, Gitaly, Redis or PostgreSQL nodes are consistently oversaturated, then increase their resources accordingly while leaving the rest of the environment as is. +NOTE: +If you're seeing a component continuously exhausting it's given resources it's strongly recommended for you to reach out to our [Support team](https://about.gitlab.com/support/) before performing any scaling. This is especially so if you're planning to scale any component significantly. -If expecting a large increase in users, you may elect to scale up the whole environment to the next -size of architecture. +For most components vertical and horizontal scaling can be applied as normal. However, before doing so though please be aware of the below caveats: -If the overall design is being followed, you can scale the environment vertically as required. +- When scaling Puma or Sidekiq vertically the amount of workers will need to be adjusted to use the additional specs. Puma will be scaled automatically on the next reconfigure but Sidekiq will need [its configuration changed beforehand](../sidekiq/extra_sidekiq_processes.md#start-multiple-processes). +- Redis and PgBouncer are primarily single threaded. If these components are seeing CPU exhaustion they may need to be scaled out horizontally. +- Scaling certain components significantly can result in notable knock on effects that affect the performance of the environment. [Refer to the dedicated section below for more guidance](#scaling-knock-on-effects). -If robust metrics are in place that show the environment is over-provisioned, you can apply the same process for -scaling downwards. You should take an iterative approach when scaling downwards to ensure there are no issues. +Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can scale downwards similarly. +You should take an iterative approach when scaling downwards, however, to ensure there are no issues. -#### Scaling from a non-HA to an HA architecture +#### Scaling knock on effects -While in most cases vertical scaling is only required to increase an environment's resources, if you are moving to an HA environment, -there may be some additional steps required as shown below: +In some cases scaling a component significantly may result in knock on effects for downstream components, impacting performance. The Reference Architectures were designed with balance in mind to ensure components that depend on each other are congruent in terms of specs. As such you may find when notably scaling a component that it's increase may result in additional throughput being passed to the other components it depends on and that they, in turn, may need to be scaled as well. -- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: - - [Redis to multi-node Redis w/ Redis Sentinel](../redis/replication_and_failover.md#switching-from-an-existing-single-machine-installation) - - [Postgres to multi-node Postgres w/ Consul + PgBouncer](../postgresql/moving.md) - - [Gitaly to Gitaly Cluster w/ Praefect](../gitaly/index.md#migrate-to-gitaly-cluster) -- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. +NOTE: +As a general rule most components have good headroom to accommodate an upstream component being scaled, so this is typically on a case by case basis and specific to what has been changed. It's recommended for you to reach out to our [Support team](https://about.gitlab.com/support/) before you make any significant changes to the environment. -Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for -scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues. +The following components can impact others when they have been significantly scaled: + +- Puma and Sidekiq - Notable scale ups of either Puma or Sidekiq workers will result in higher concurrent connections to the internal load balancer, PostgreSQL (via PgBouncer if present), Gitaly (via Praefect if present) and Redis respectively. + - Redis is primarily single threaded and in some cases may need to be split up into different instances (Cache / Persistent) if the increased throughput causes CPU exhaustion if a combined cluster is currently being used. + - PgBouncer is also single threaded but note that a scale out will result in a new pool being added that in turn will increase total connections to Postgres. It's strongly recommended to only do this if you have experience in managing Postgres connections and to seek assistance if in doubt. +- Gitaly Cluster / PostgreSQL - A notable scale out of additional nodes can have a detrimental effect on the HA system and performance due to increased replication calls to the primary node. + +#### Scaling from a non-HA to an HA architecture + +While in most cases vertical scaling is only required to increase an environment's resources, if you are moving to an HA environment +additional steps will be required for the following components to switch over to their HA forms respectively by following the given +documentation for each as follows + +- [Redis to multi-node Redis w/ Redis Sentinel](../redis/replication_and_failover.md#switching-from-an-existing-single-machine-installation) +- [Postgres to multi-node Postgres w/ Consul + PgBouncer](../postgresql/moving.md) +- [Gitaly to Gitaly Cluster w/ Praefect](../gitaly/index.md#migrate-to-gitaly-cluster) ### Monitoring @@ -721,6 +733,7 @@ You can find a full history of changes [on the GitLab project](https://gitlab.co **2023:** - [2023-12-12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Updated notes on Load Balancers to be more reflective that any reputable offering is expected to work. +- [2023-12-12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Updated notes on Load Balancers to be more reflective that any reputable offering is expected to work. - [2023-11-03](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133457): Expanded details on what each Reference Architecture is designed for, the testing methodology used as well as added details on how to scale environments. - [2023-11-03](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134518): Added expanded notes on disk types, object storage and monitoring. - [2023-10-25](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134518): Adjusted Sidekiq configuration example to use Linux Package role. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md deleted file mode 100644 index 844dc76b23d..00000000000 --- a/doc/administration/repository_storage_types.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'repository_storage_paths.md' -remove_date: '2023-11-29' ---- - -This document was moved to [another location](repository_storage_paths.md). - -<!-- This redirect file can be deleted after <2023-11-29>. --> diff --git a/doc/administration/settings/email.md b/doc/administration/settings/email.md index cc80b082139..a010a855ba1 100644 --- a/doc/administration/settings/email.md +++ b/doc/administration/settings/email.md @@ -10,7 +10,7 @@ You can customize some of the content in emails sent from your GitLab instance. ## Custom logo -The logo in the header of some emails can be customized, see the [logo customization section](../../administration/appearance.md#navigation-bar). +The logo in the header of some emails can be customized, see the [logo customization section](../../administration/appearance.md#customize-your-homepage-button). ## Include author name in email notification email body **(PREMIUM SELF)** diff --git a/doc/administration/settings/help_page.md b/doc/administration/settings/help_page.md index 1077bbec316..bb7f214a106 100644 --- a/doc/administration/settings/help_page.md +++ b/doc/administration/settings/help_page.md @@ -55,7 +55,7 @@ GitLab marketing-related entries are occasionally shown on the Help page. To hid You can specify a custom URL to which users are directed when they: -- Select **Support** from the Help dropdown list. +- Select **Help > Support**. - Select **See our website for help** on the Help page. 1. On the left sidebar, at the bottom, select **Admin Area**. diff --git a/doc/administration/settings/import_and_export_settings.md b/doc/administration/settings/import_and_export_settings.md index ddb31e49483..54995cdc686 100644 --- a/doc/administration/settings/import_and_export_settings.md +++ b/doc/administration/settings/import_and_export_settings.md @@ -121,7 +121,7 @@ To modify this setting: > - **Maximum decompressed file size for archives from imports** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130081) from **Maximum decompressed size** in GitLab 16.4. When you import a project using [file exports](../../user/project/settings/import_export.md) or -[direct transfer](../../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended), you can specify the +[direct transfer](../../user/group/import/index.md), you can specify the maximum decompressed file size for imported archives. The default value is 25 GiB. When you import a compressed file, the decompressed size cannot exceed the maximum decompressed file size limit. If the diff --git a/doc/administration/settings/index.md b/doc/administration/settings/index.md index 531073887c5..2df9d1cd52d 100644 --- a/doc/administration/settings/index.md +++ b/doc/administration/settings/index.md @@ -1,6 +1,7 @@ --- stage: none group: unassigned +description: Product settings. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/settings/jira_cloud_app_troubleshooting.md b/doc/administration/settings/jira_cloud_app_troubleshooting.md index 1b6f88d43ae..cc17c620724 100644 --- a/doc/administration/settings/jira_cloud_app_troubleshooting.md +++ b/doc/administration/settings/jira_cloud_app_troubleshooting.md @@ -10,7 +10,7 @@ When administering the GitLab for Jira Cloud app for self-managed instances, you For GitLab.com, see [GitLab for Jira Cloud app](../../integration/jira/connect-app.md#troubleshooting). -## Browser displays a sign-in message when already signed in +## Sign-in message displayed when already signed in You might get the following message prompting you to sign in to GitLab.com when you're already signed in: @@ -26,12 +26,17 @@ To resolve this issue, set up [OAuth authentication](jira_cloud_app.md#set-up-oa ## Manual installation fails -You might get an error if you have installed the GitLab for Jira Cloud app from the official marketplace listing and replaced it with [manual installation](jira_cloud_app.md#install-the-gitlab-for-jira-cloud-app-manually): +You might get one of the following errors if you've installed the GitLab for Jira Cloud app +from the official marketplace listing and replaced it with [manual installation](jira_cloud_app.md#install-the-gitlab-for-jira-cloud-app-manually): ```plaintext The app "gitlab-jira-connect-gitlab.com" could not be installed as a local app as it has previously been installed from Atlassian Marketplace ``` +```plaintext +The app host returned HTTP response code 401 when we tried to contact it during installation. Please try again later or contact the app vendor. +``` + To resolve this issue, disable the **Jira Connect Proxy URL** setting. - In GitLab 15.7: @@ -47,7 +52,7 @@ To resolve this issue, disable the **Jira Connect Proxy URL** setting. 1. Clear the **Jira Connect Proxy URL** text box. 1. Select **Save changes**. -## Data sync fails with `Invalid JWT` error +## Data sync fails with `Invalid JWT` If the GitLab for Jira Cloud app continuously fails to sync data, it may be due to an outdated secret token. Atlassian can send new secret tokens that must be processed and stored by GitLab. If GitLab fails to store the token or misses the new token request, an `Invalid JWT` error occurs. @@ -101,13 +106,16 @@ due to a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388943). To ### Debugging Jira Connect Proxy issues -If you are using a self-managed GitLab instance and you have configured `https://gitlab.com` for the Jira Connect Proxy URL when -[setting up the OAuth authentication](jira_cloud_app.md#set-up-oauth-authentication), you can inspect the network traffic in your browser's development -tools while reproducing the `Failed to update the GitLab instance` error to see a more precise error. +If you set **Jira Connect Proxy URL** to `https://gitlab.com` when you +[set up your instance](jira_cloud_app.md#set-up-your-instance), you can: + +- Inspect the network traffic in your browser's development tools. +- Reproduce the `Failed to update the GitLab instance` error for more information. You should see a `GET` request to `https://gitlab.com/-/jira_connect/installations`. -This request should return a `200` status code, but it can return a `422` status code if there was a problem. The response body can be checked for the error. +This request should return a `200 OK`, but it might return a `422 Unprocessable Entity` if there was a problem. +You can check the response body for the error. If you cannot resolve the problem and you are a GitLab customer, contact [GitLab Support](https://about.gitlab.com/support/) for assistance. Provide GitLab Support with: @@ -119,7 +127,7 @@ GitLab Support with: The GitLab Support team can then look up why this is failing in the GitLab.com server logs. -#### Process for GitLab Support +#### GitLab Support NOTE: These steps can only be completed by GitLab Support. @@ -138,12 +146,12 @@ For the second log: - `json.message` is `Proxy lifecycle event received error response` or similar. - `json.jira_status_code` and `json.jira_body` might contain details on why GitLab.com wasn't able to connect back to the self-managed instance. -- If `json.jira_status_code` is `401` and `json.jira_body` is empty, this might indicate that the [**Jira Connect Proxy URL**](jira_cloud_app.md#set-up-your-instance) is not set to +- If `json.jira_status_code` is `401` and `json.jira_body` is empty, [**Jira Connect Proxy URL**](jira_cloud_app.md#set-up-your-instance) might not be set to `https://gitlab.com`. -## `Failed to link group` +## Error when connecting the app -After you connect the GitLab for Jira Cloud app for self-managed instances, you might get one of these errors: +When you connect the GitLab for Jira Cloud app, you might get one of these errors: ```plaintext Failed to load Jira Connect Application ID. Please try again. @@ -159,6 +167,7 @@ When you check the browser console, you might see the following message: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://gitlab.example.com/-/jira_connect/oauth_application_id. (Reason: CORS header 'Access-Control-Allow-Origin' missing). Status code: 403. ``` -`403` status code is returned if the user information cannot be fetched from Jira because of insufficient permissions. +A `403 Forbidden` is returned if the user information cannot be fetched from Jira because of insufficient permissions. -To resolve this issue, ensure that the Jira user that installs and configures the GitLab for Jira Cloud app meets certain [requirements](jira_cloud_app.md#jira-user-requirements). +To resolve this issue, ensure the Jira user that installs and configures the app meets certain +[requirements](jira_cloud_app.md#jira-user-requirements). diff --git a/doc/administration/settings/rate_limit_on_members_api.md b/doc/administration/settings/rate_limit_on_members_api.md new file mode 100644 index 00000000000..3e8868adc91 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_members_api.md @@ -0,0 +1,33 @@ +--- +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limit on Members API **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140633) in GitLab 16.9. + +You can configure the rate limit per group (or project) per user to the +[delete members API](../../api/members.md#remove-a-member-from-a-group-or-project). + +To change the rate limit: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Members API rate limit**. +1. In the **Maximum requests per minute per group / project** text box, enter the new value. +1. Select **Save changes**. + +The rate limit: + +- Applies per group or project per user. +- Can be set to 0 to disable rate limiting. + +The default value of the rate limit is `60`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 60, requests sent to the +[delete members API](../../api/members.md#remove-a-member-from-a-group-or-project) exceeding a rate of 300 per minute +are blocked. Access to the endpoint is allowed after one minute. diff --git a/doc/administration/settings/scim_setup.md b/doc/administration/settings/scim_setup.md index 52061150fa7..2e1b40d58b8 100644 --- a/doc/administration/settings/scim_setup.md +++ b/doc/administration/settings/scim_setup.md @@ -33,6 +33,51 @@ To configure GitLab SCIM: - Token from the **Your SCIM token** field. - URL from the **SCIM API endpoint URL** field. +## Configure an identity provider + +You can configure the following as an identity provider: + +- [Okta](#configure-okta). + +NOTE: +Other identity providers can work with GitLab but they have not been tested and are not supported. You should contact the provider for support. GitLab support can assist by reviewing related log entries. + +### Configure Okta + +The SAML application created during [single sign-on](index.md) set up for Okta must be set up for SCIM. + +Prerequisites: + +- You must use the [Okta Lifecycle Management](https://www.okta.com/products/lifecycle-management/) product. This + product tier is required to use SCIM on Okta. +- [GitLab is configured](#configure-gitlab) for SCIM. +- The SAML application for [Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/main/) set up as + described in the [Okta setup notes](../../integration/saml.md#set-up-okta). +- Your Okta SAML setup matches the [configuration steps](index.md), especially the NameID configuration. + +To configure Okta for SCIM: + +1. Sign in to Okta. +1. In the upper-right corner, select **Admin**. The button is not visible from the Admin Area. +1. In the **Application** tab, select **Browse App Catalog**. +1. Find and select the **GitLab** application. +1. On the GitLab application overview page, select **Add Integration**. +1. Under **Application Visibility**, select both checkboxes. The GitLab application does not support SAML + authentication so the icon should not be shown to users. +1. Select **Done** to finish adding the application. +1. In the **Provisioning** tab, select **Configure API integration**. +1. Select **Enable API integration**. + - For **Base URL**, paste the URL you copied from **SCIM API endpoint URL** on the GitLab SCIM configuration page. + - For **API Token**, paste the SCIM token you copied from **Your SCIM token** on the GitLab SCIM + configuration page. +1. To verify the configuration, select **Test API Credentials**. +1. Select **Save**. +1. After saving the API integration details, new settings tabs appear on the left. Select **To App**. +1. Select **Edit**. +1. Select the **Enable** checkbox for both **Create Users** and **Deactivate Users**. +1. Select **Save**. +1. Assign users in the **Assignments** tab. Assigned users are created and managed in your GitLab group. + ## Remove access Removing or deactivating a user on the identity provider blocks the user on diff --git a/doc/administration/settings/sign_in_restrictions.md b/doc/administration/settings/sign_in_restrictions.md index 60994bc6e43..6888994580c 100644 --- a/doc/administration/settings/sign_in_restrictions.md +++ b/doc/administration/settings/sign_in_restrictions.md @@ -85,7 +85,7 @@ To enable Admin Mode through the UI: To turn on Admin Mode for your current session and access potentially dangerous resources: -1. On the left sidebar, select **Search or go to**. +1. On the left sidebar, select your avatar. 1. Select **Enter Admin Mode**. 1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). @@ -103,7 +103,7 @@ authentication are supported by Admin Mode. Admin Mode status is stored in the c To turn off Admin Mode for your current session: -1. On the left sidebar, select **Search or go to**. +1. On the left sidebar, select your avatar. 1. Select **Leave Admin Mode**. ### Limitations of Admin Mode diff --git a/doc/administration/settings/slack_app.md b/doc/administration/settings/slack_app.md index 5e54294b720..48352a74060 100644 --- a/doc/administration/settings/slack_app.md +++ b/doc/administration/settings/slack_app.md @@ -106,9 +106,11 @@ When administering the GitLab for Slack app for self-managed instances, you migh For GitLab.com, see [GitLab for Slack app](../../user/project/integrations/gitlab_slack_app_troubleshooting.md). -### Slash commands return an error in Slack +### Slash commands return `dispatch_failed` in Slack -Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. To resolve this issue, ensure: +Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. + +To resolve this issue, ensure: - The GitLab for Slack app is properly [configured](#configure-the-settings) and the **Enable GitLab for Slack app** checkbox is selected. - Your GitLab instance [allows requests to and from Slack](#connectivity-requirements). diff --git a/doc/administration/settings/usage_statistics.md b/doc/administration/settings/usage_statistics.md index 20479a7dd8a..b661bcd6746 100644 --- a/doc/administration/settings/usage_statistics.md +++ b/doc/administration/settings/usage_statistics.md @@ -100,15 +100,6 @@ for all authenticated users, and on the Admin Area pages. The statuses are:  -GitLab Inc. collects your instance's version and hostname (through the HTTP -referer) as part of the version check. No other information is collected. - -This information is used, among other things, to identify to which versions -patches must be backported, making sure active GitLab instances remain -secure. - -If you disable version check, this information isn't collected. - ### Enable or disable version check 1. On the left sidebar, at the bottom, select **Admin Area**. diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md index 6dcc97f2f7a..ff823bb6c56 100644 --- a/doc/administration/sidekiq/sidekiq_memory_killer.md +++ b/doc/administration/sidekiq/sidekiq_memory_killer.md @@ -28,7 +28,7 @@ run as a process group leader (for example, using `chpst -P`). If using a Linux ## Configuring the limits -Sidekiq memory limits are controlled using environment variables. +Sidekiq memory limits are controlled using [environment variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html#setting-custom-environment-variables) - `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): defines the Sidekiq process soft limit for allowed RSS. If the Sidekiq process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_MAX_RSS`, diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 315a68fb126..75fd807643c 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -103,7 +103,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Managing PostgreSQL versions on Linux package installations [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html). - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) + - Including [troubleshooting](../../administration/postgresql/replication_and_failover_troubleshooting.md) `gitlab-ctl patroni check-leader` and PgBouncer errors. - [Developer database documentation](../../development/feature_development.md#database-guides), diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 31d0781ee79..95454ef629c 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -19,7 +19,7 @@ This is the default configuration. To change the location where the uploads are stored locally, use the steps in this section based on your installation method: NOTE: -For historical reasons, uploads for the whole instance (for example the [favicon](../administration/appearance.md#favicon)) are stored in a base directory, +For historical reasons, uploads for the whole instance (for example the [favicon](../administration/appearance.md#customize-the-favicon)) are stored in a base directory, which by default is `uploads/-/system`. Changing the base directory on an existing GitLab installation is strongly discouraged. diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 6e0963c5325..cf08c34655b 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -72,7 +72,8 @@ The following API resources are available in the project context: | [NPM repository](packages/npm.md) | `/projects/:id/packages/npm` | | [NuGet packages](packages/nuget.md) | `/projects/:id/packages/nuget` (also available for groups) | | [Packages](packages.md) | `/projects/:id/packages` | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pages domains](pages_domains.md) | `/projects/:id/pages/domains` (also available standalone) | +| [Pages settings](pages.md) | `/projects/:id/pages` | | [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | | [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | | [Pipelines](pipelines.md) | `/projects/:id/pipelines` | diff --git a/doc/api/award_emoji.md b/doc/api/award_emoji.md deleted file mode 100644 index 09f7b4c77fa..00000000000 --- a/doc/api/award_emoji.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'emoji_reactions.md' -remove_date: '2023-12-20' ---- - -This document was moved to [another location](emoji_reactions.md). - -<!-- This redirect file can be deleted after <2023-12-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/api/bulk_imports.md b/doc/api/bulk_imports.md index 1d7556f863a..bebfdb80a35 100644 --- a/doc/api/bulk_imports.md +++ b/doc/api/bulk_imports.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Project migration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390515) in GitLab 15.11. With the group migration by direct transfer API, you can start and view the progress of migrations initiated with -[group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended). +[group migration by direct transfer](../user/group/import/index.md). WARNING: Migrating projects with this API is in [Beta](../policy/experiment-beta-support.md#beta). This feature is not diff --git a/doc/api/code_suggestions.md b/doc/api/code_suggestions.md index 91c5c988acd..307dc112028 100644 --- a/doc/api/code_suggestions.md +++ b/doc/api/code_suggestions.md @@ -38,6 +38,7 @@ Example response: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415581) in GitLab 16.2 [with a flag](../administration/feature_flags.md) named `code_suggestions_completion_api`. Disabled by default. This feature is an Experiment. > - Requirement to generate a JWT before calling this endpoint was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127863) in GitLab 16.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/416371) in GitLab 16.8. [Feature flag `code_suggestions_completion_api`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138174) removed. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `code_suggestions_completion_api`. diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index ab8be6abab7..9ad52fa3c3a 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -6,28 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container registry API **(FREE ALL)** -> The use of `CI_JOB_TOKEN` scoped to the current project was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49750) in GitLab 13.12. +> - The ability to authenticate with a CI/CD job token [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49750) in GitLab 13.12 [with a flag](../administration/feature_flags.md) named `ci_job_token_scope`. Disabled by default. +> - CI/CD job token authentication [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/300821) in GitLab 16.8. Feature flag `ci_job_token_scope` removed. -This API documentation is about the [GitLab container registry](../user/packages/container_registry/index.md). +Use these API endpoints to work with the [GitLab container registry](../user/packages/container_registry/index.md). -When the `ci_job_token_scope` feature flag is enabled (it is **disabled by default**), you can use the below endpoints -from a CI/CD job, by passing the `$CI_JOB_TOKEN` variable as the `JOB-TOKEN` header. -The job token only has access to its own project. - -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can opt to enable it. - -To enable it: - -```ruby -Feature.enable(:ci_job_token_scope) -``` - -To disable it: - -```ruby -Feature.disable(:ci_job_token_scope) -``` +You can authenticate with these endpoints from a CI/CD job by passing the [`$CI_JOB_TOKEN`](../ci/jobs/ci_job_token.md) +variable as the `JOB-TOKEN` header. The job token only has access to the container registry +of the project that created the pipeline. ## Change the visibility of the container registry @@ -47,13 +33,11 @@ PUT /projects/:id/ Descriptions of the possible values for `container_registry_access_level`: - **enabled** (Default): The container registry is visible to everyone with access to the project. -If the project is public, the container registry is also public. If the project is internal or -private, the container registry is also internal or private. - + If the project is public, the container registry is also public. If the project is internal or + private, the container registry is also internal or private. - **private**: The container registry is visible only to project members with Reporter role or -higher. This behavior is similar to that of a private project with container registry visibility set -to **enabled**. - + higher. This behavior is similar to that of a private project with container registry visibility set + to **enabled**. - **disabled**: The container registry is disabled. See the [container registry visibility permissions](../user/packages/container_registry/index.md#container-registry-visibility-permissions) diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 8640f3b45c6..b12131e4746 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -712,7 +712,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "updated_at": "2018-03-03T21:54:39.668Z", "system": false, "noteable_id": 3, - "noteable_type": "Merge request", + "noteable_type": "MergeRequest", "project_id": 5, "noteable_iid": null, "resolved": false, @@ -737,7 +737,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "updated_at": "2018-03-04T13:38:02.127Z", "system": false, "noteable_id": 3, - "noteable_type": "Merge request", + "noteable_type": "MergeRequest", "project_id": 5, "noteable_iid": null, "resolved": false, @@ -767,7 +767,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "updated_at": "2018-03-04T09:17:22.520Z", "system": false, "noteable_id": 3, - "noteable_type": "Merge request", + "noteable_type": "MergeRequest", "project_id": 5, "noteable_iid": null, "resolved": false, @@ -804,7 +804,7 @@ Diff comments also contain position: "updated_at": "2018-03-04T09:17:22.520Z", "system": false, "noteable_id": 3, - "noteable_type": "Merge request", + "noteable_type": "MergeRequest", "project_id": 5, "noteable_iid": null, "commit_id": "4803c71e6b1833ca72b8b26ef2ecd5adc8a38031", @@ -865,8 +865,6 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" \ ### Create new merge request thread -> The `commit id` entry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47130) in GitLab 13.7. - Creates a new thread to a single project merge request. Similar to creating a note but other comments (replies) can be added to it later. For other approaches, see [Post comment to commit](commits.md#post-comment-to-commit) in the Commits API, @@ -878,27 +876,27 @@ POST /projects/:id/merge_requests/:merge_request_iid/discussions Parameters for all comments: -| Attribute | Type | Required | Description | -| ---------------------------------------- | -------------- | -------- | ----------- | -| `body` | string | yes | The content of the thread. | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `merge_request_iid` | integer | yes | The IID of a merge request. | -| `position[base_sha]` | string | yes | Base commit SHA in the source branch. | -| `position[head_sha]` | string | yes | SHA referencing HEAD of this merge request. | -| `position[start_sha]` | string | yes | SHA referencing commit in target branch. | +| Attribute | Type | Required | Description | +| ---------------------------------------- | -------------- |--------------------------------------| ----------- | +| `body` | string | yes | The content of the thread. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `merge_request_iid` | integer | yes | The IID of a merge request. | +| `position[base_sha]` | string | yes (if `position*` is supplied) | Base commit SHA in the source branch. | +| `position[head_sha]` | string | yes (if `position*` is supplied) | SHA referencing HEAD of this merge request. | +| `position[start_sha]` | string | yes (if `position*` is supplied) | SHA referencing commit in target branch. | | `position[new_path]` | string | yes (if the position type is `text`) | File path after change. | | `position[old_path]` | string | yes (if the position type is `text`) | File path before change. | -| `position[position_type]` | string | yes | Type of the position reference. Allowed values: `text` or `image`. | -| `commit_id` | string | no | SHA referencing commit to start this thread on. | -| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | -| `position` | hash | no | Position when creating a diff note. | -| `position[new_line]` | integer | no | For `text` diff notes, the line number after change. | -| `position[old_line]` | integer | no | For `text` diff notes, the line number before change. | -| `position[line_range]` | hash | no | Line range for a multi-line diff note. | -| `position[width]` | integer | no | For `image` diff notes, width of the image. | -| `position[height]` | integer | no | For `image` diff notes, height of the image. | -| `position[x]` | float | no | For `image` diff notes, X coordinate. | -| `position[y]` | float | no | For `image` diff notes, Y coordinate. | +| `position[position_type]` | string | yes (if position* is supplied) | Type of the position reference. Allowed values: `text` or `image`. | +| `commit_id` | string | no | SHA referencing commit to start this thread on. | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | +| `position` | hash | no | Position when creating a diff note. | +| `position[new_line]` | integer | no | For `text` diff notes, the line number after change. | +| `position[old_line]` | integer | no | For `text` diff notes, the line number before change. | +| `position[line_range]` | hash | no | Line range for a multi-line diff note. | +| `position[width]` | integer | no | For `image` diff notes, width of the image. | +| `position[height]` | integer | no | For `image` diff notes, height of the image. | +| `position[x]` | float | no | For `image` diff notes, X coordinate. | +| `position[y]` | float | no | For `image` diff notes, Y coordinate. | #### Create a new thread on the overview page @@ -1289,17 +1287,17 @@ POST /projects/:id/repository/commits/:commit_id/discussions Parameters: -| Attribute | Type | Required | Description | -| ------------------------- | -------------- | -------- | ----------- | -| `body` | string | yes | The content of the thread. | -| `commit_id` | string | yes | The SHA of a commit. | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `position[base_sha]` | string | yes | SHA of the parent commit. | -| `position[head_sha]` | string | yes | The SHA of this commit. Same as `commit_id`. | -| `position[start_sha]` | string | yes | SHA of the parent commit. | -| `position[position_type]` | string | yes | Type of the position reference. Allowed values: `text` or `image`. | -| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | -| `position` | hash | no | Position when creating a diff note. | +| Attribute | Type | Required | Description | +| ------------------------- | -------------- |----------------------------------| ----------- | +| `body` | string | yes | The content of the thread. | +| `commit_id` | string | yes | The SHA of a commit. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `position[base_sha]` | string | yes (if `position*` is supplied) | SHA of the parent commit. | +| `position[head_sha]` | string | yes (if `position*` is supplied) | The SHA of this commit. Same as `commit_id`. | +| `position[start_sha]` | string | yes (if `position*` is supplied) | SHA of the parent commit. | +| `position[position_type]` | string | yes (if `position*` is supplied) | Type of the position reference. Allowed values: `text` or `image`. | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z`. Requires administrator or project/group owner rights. | +| `position` | hash | no | Position when creating a diff note. | | `position[new_path]` | string | no | File path after change. | | `position[new_line]` | integer | no | Line number after change. | diff --git a/doc/api/graphql/getting_started.md b/doc/api/graphql/getting_started.md index cf756027e01..868059fb979 100644 --- a/doc/api/graphql/getting_started.md +++ b/doc/api/graphql/getting_started.md @@ -42,8 +42,7 @@ You can run GraphQL queries in a `curl` request on the command line on your local computer. The requests `POST` to `/api/graphql` with the query as the payload. You can authorize your request by generating a [personal access token](../../user/profile/personal_access_tokens.md) to use as -a bearer token. -This token requires at least the `read_api` scope. +a bearer token. Read more about [GraphQL Authentication](index.md#authentication). Example: @@ -162,10 +161,9 @@ More about queries: ### Authorization -Authorization uses the same engine as the GitLab application (and GitLab.com). If you've signed in to GitLab and use [GraphiQL](#graphiql), all queries are performed as -you, the authenticated user. For more information, read the -[GitLab API documentation](../rest/index.md#authentication). +you, the authenticated user. For more information, read about +[GraphQL Authentication](index.md#authentication). ### Mutations @@ -285,10 +283,9 @@ in `CI_JOB_TOKEN` scoping behavior. ```graphql mutation DisableCI_JOB_TOKENscope { - projectCiCdSettingsUpdate(input:{fullPath: "<namespace>/<project-name>", inboundJobTokenScopeEnabled: false, jobTokenScopeEnabled: false}) { + projectCiCdSettingsUpdate(input:{fullPath: "<namespace>/<project-name>", inboundJobTokenScopeEnabled: false}) { ciCdSettings { inboundJobTokenScopeEnabled - jobTokenScopeEnabled } errors } diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 37bd9fc38ed..349b667f595 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -1,6 +1,7 @@ --- stage: Manage group: Import and Integrate +description: Programmatic interaction with GitLab. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -43,6 +44,77 @@ You can work with sample queries that pull data from public projects on GitLab.c The [get started](getting_started.md) page includes different methods to customize GraphQL queries. +### Authentication + +Some queries can be accessed anonymously without the request needing to be authenticated, +but others require it. Mutations always require authentication. + +Authentication can happen by: + +- [Token](#token-authentication) +- [Session cookie](#session-cookie-authentication) + +If the authentication information is not valid, GitLab returns an error message with a status code of 401: + +{"errors":[{"message":"Invalid token"}]} + +#### Token authentication + +Use any of the following tokens to authenticate with the GraphQL API: + +- [OAuth 2.0 tokens](../../api/oauth2.md) +- [Personal access tokens](../../user/profile/personal_access_tokens.md) +- [Project access tokens](../../user/project/settings/project_access_tokens.md) +- [Group access tokens](../../user/group/settings/group_access_tokens.md) + +Authenticate with a token by passing it through in a [request header](#header-authentication) or as a [parameter](#parameter-authentication). + +Tokens require the correct [scope](#token-scopes). + +##### Header authentication + +Example of token authentication using an `Authorization: Bearer <token>` request header: + +```shell +curl "https://gitlab.com/api/graphql" --header "Authorization: Bearer <token>" \ + --header "Content-Type: application/json" --request POST \ + --data "{\"query\": \"query {currentUser {name}}\"}" +``` + +##### Parameter authentication + +Alternatively, OAuth 2.0 tokens can be passed in using the `access_token` parameter: + +```shell +curl "https://gitlab.com/api/graphql?access_token=<oauth_token>" \ + --header "Content-Type: application/json" --request POST \ + --data "{\"query\": \"query {currentUser {name}}\"}" +``` + +Personal, project, or group access tokens can be passed in using the `private_token` parameter: + +```shell +curl "https://gitlab.com/api/graphql?private_token=<access_token>" \ + --header "Content-Type: application/json" --request POST \ + --data "{\"query\": \"query {currentUser {name}}\"}" +``` + +##### Token scopes + +Tokens must have the correct scope to access the GraphQL API, either: + +| Scope | Access | +|------------|---------| +| `read_api` | Grants read access to the API. Sufficient for queries. | +| `api` | Grants read and write access to the API. Required by mutations. | + +#### Session cookie authentication + +Signing in to the main GitLab application sets a `_gitlab_session` session cookie. + +The [interactive GraphQL explorer](#interactive-graphql-explorer) and the web frontend of +GitLab itself use this method of authentication. + ### Global IDs In the GitLab GraphQL API, an `id` field is nearly always a [Global ID](https://graphql.org/learn/global-object-identification/) diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 46ddad63360..a8a29e2f01c 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -394,7 +394,7 @@ Returns [`Group`](#group). | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="querygroupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the project, group, or namespace. For example, `gitlab-org/gitlab-foss`. | +| <a id="querygroupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the group. For example, `gitlab-org/gitlab-foss`. | ### `Query.groups` @@ -521,7 +521,7 @@ Returns [`Iteration`](#iteration). ### `Query.jobs` -All jobs on this GitLab instance. +All jobs on this GitLab instance. Returns an empty result for users without administrator access. Returns [`CiJobConnection`](#cijobconnection). @@ -562,6 +562,8 @@ Returns [`MemberRole`](#memberrole). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="querymemberroleid"></a>`id` | [`MemberRoleID`](#memberroleid) | Global ID of the member role to look up. | +| <a id="querymemberroleorderby"></a>`orderBy` | [`MemberRolesOrderBy`](#memberrolesorderby) | Ordering column. Default is NAME. | +| <a id="querymemberrolesort"></a>`sort` | [`SortDirectionEnum`](#sortdirectionenum) | Ordering column. Default is ASC. | ### `Query.memberRolePermissions` @@ -577,6 +579,28 @@ This field returns a [connection](#connections). It accepts the four standard [pagination arguments](#connection-pagination-arguments): `before: String`, `after: String`, `first: Int`, and `last: Int`. +### `Query.memberRoles` + +Member roles available for the instance. + +WARNING: +**Introduced** in 16.7. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`MemberRoleConnection`](#memberroleconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, and `last: Int`. + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="querymemberrolesid"></a>`id` | [`MemberRoleID`](#memberroleid) | Global ID of the member role to look up. | +| <a id="querymemberrolesorderby"></a>`orderBy` | [`MemberRolesOrderBy`](#memberrolesorderby) | Ordering column. Default is NAME. | +| <a id="querymemberrolessort"></a>`sort` | [`SortDirectionEnum`](#sortdirectionenum) | Ordering column. Default is ASC. | + ### `Query.mergeRequest` Find a merge request. @@ -633,7 +657,7 @@ Returns [`Namespace`](#namespace). | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="querynamespacefullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the project, group, or namespace. For example, `gitlab-org/gitlab-foss`. | +| <a id="querynamespacefullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. For example, `gitlab-org/gitlab-foss`. | ### `Query.note` @@ -667,6 +691,20 @@ Returns [`Organization`](#organization). | ---- | ---- | ----------- | | <a id="queryorganizationid"></a>`id` | [`OrganizationsOrganizationID!`](#organizationsorganizationid) | ID of the organization. | +### `Query.organizations` + +List organizations. + +WARNING: +**Introduced** in 16.8. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`OrganizationConnection`](#organizationconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, and `last: Int`. + ### `Query.package` Find a package. This field can only be resolved for one query in any single request. Returns `null` if a package has no `default` status. @@ -689,7 +727,7 @@ Returns [`Project`](#project). | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="queryprojectfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the project, group, or namespace. For example, `gitlab-org/gitlab-foss`. | +| <a id="queryprojectfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the project. For example, `gitlab-org/gitlab-foss`. | ### `Query.projects` @@ -957,6 +995,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="queryusersadmins"></a>`admins` | [`Boolean`](#boolean) | Return only admin users. | +| <a id="queryusersgroupid"></a>`groupId` | [`GroupID`](#groupid) | Return users member of a given group. | | <a id="queryusersids"></a>`ids` | [`[ID!]`](#id) | List of user Global IDs. | | <a id="queryuserssearch"></a>`search` | [`String`](#string) | Query to search users by name, username, or primary email. | | <a id="queryuserssort"></a>`sort` | [`Sort`](#sort) | Sort users by this criteria. | @@ -984,6 +1023,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="queryvulnerabilitieshasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have remediations. | | <a id="queryvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="queryvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="queryvulnerabilitiesowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="queryvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="queryvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="queryvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -1357,6 +1397,31 @@ Input type: `AiActionInput` | <a id="mutationaiactionerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationaiactionrequestid"></a>`requestId` | [`String`](#string) | ID of the request. | +### `Mutation.aiAgentCreate` + +WARNING: +**Introduced** in 16.8. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `AiAgentCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationaiagentcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationaiagentcreatename"></a>`name` | [`String!`](#string) | Name of the agent. | +| <a id="mutationaiagentcreateprojectpath"></a>`projectPath` | [`ID!`](#id) | Project to which the agent belongs. | +| <a id="mutationaiagentcreateprompt"></a>`prompt` | [`String!`](#string) | Prompt for the agent. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationaiagentcreateagent"></a>`agent` | [`AiAgent`](#aiagent) | Agent after mutation. | +| <a id="mutationaiagentcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationaiagentcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.alertSetAssignees` Input type: `AlertSetAssigneesInput` @@ -1945,6 +2010,30 @@ Input type: `BoardListUpdateLimitMetricsInput` | <a id="mutationboardlistupdatelimitmetricserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationboardlistupdatelimitmetricslist"></a>`list` | [`BoardList`](#boardlist) | Updated list. | +### `Mutation.branchRuleCreate` + +WARNING: +**Introduced** in 16.7. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `BranchRuleCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationbranchrulecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbranchrulecreatename"></a>`name` | [`String!`](#string) | Branch name, with wildcards, for the branch rules. | +| <a id="mutationbranchrulecreateprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path to the project that the branch is associated with. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationbranchrulecreatebranchrule"></a>`branchRule` | [`BranchRule`](#branchrule) | Branch rule after mutation. | +| <a id="mutationbranchrulecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbranchrulecreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.branchRuleUpdate` WARNING: @@ -5324,20 +5413,12 @@ Input type: `MemberRoleCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationmemberrolecreateadmingroupmember"></a>`adminGroupMember` | [`Boolean`](#boolean) | Permission to admin group members. | -| <a id="mutationmemberrolecreateadminmergerequest"></a>`adminMergeRequest` | [`Boolean`](#boolean) | Permission to admin merge requests. | -| <a id="mutationmemberrolecreateadminvulnerability"></a>`adminVulnerability` | [`Boolean`](#boolean) | Permission to admin vulnerability. | -| <a id="mutationmemberrolecreatearchiveproject"></a>`archiveProject` | [`Boolean`](#boolean) | Permission to archive projects. | | <a id="mutationmemberrolecreatebaseaccesslevel"></a>`baseAccessLevel` | [`MemberAccessLevel!`](#memberaccesslevel) | Base access level for the custom role. | | <a id="mutationmemberrolecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationmemberrolecreatedescription"></a>`description` | [`String`](#string) | Description of the member role. | | <a id="mutationmemberrolecreategrouppath"></a>`groupPath` | [`ID`](#id) | Group the member role to mutate is in. Required for SaaS. | -| <a id="mutationmemberrolecreatemanageprojectaccesstokens"></a>`manageProjectAccessTokens` | [`Boolean`](#boolean) | Permission to admin project access tokens. | | <a id="mutationmemberrolecreatename"></a>`name` | [`String`](#string) | Name of the member role. | | <a id="mutationmemberrolecreatepermissions"></a>`permissions` | [`[MemberRolePermission!]`](#memberrolepermission) | List of all customizable permissions. | -| <a id="mutationmemberrolecreatereadcode"></a>`readCode` | [`Boolean`](#boolean) | Permission to read code. | -| <a id="mutationmemberrolecreatereaddependency"></a>`readDependency` | [`Boolean`](#boolean) | Permission to read dependency. | -| <a id="mutationmemberrolecreatereadvulnerability"></a>`readVulnerability` | [`Boolean`](#boolean) | Permission to read vulnerability. | #### Fields @@ -5382,6 +5463,7 @@ Input type: `MemberRoleUpdateInput` | <a id="mutationmemberroleupdatedescription"></a>`description` | [`String`](#string) | Description of the member role. | | <a id="mutationmemberroleupdateid"></a>`id` | [`MemberRoleID!`](#memberroleid) | ID of the member role to mutate. | | <a id="mutationmemberroleupdatename"></a>`name` | [`String`](#string) | Name of the member role. | +| <a id="mutationmemberroleupdatepermissions"></a>`permissions` | [`[MemberRolePermission!]`](#memberrolepermission) | List of all customizable permissions. | #### Fields @@ -5673,6 +5755,31 @@ Input type: `MergeRequestUpdateApprovalRuleInput` | <a id="mutationmergerequestupdateapprovalruleerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationmergerequestupdateapprovalrulemergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | +### `Mutation.mlModelCreate` + +WARNING: +**Introduced** in 16.8. +This feature is an Experiment. It can be changed or removed at any time. + +Input type: `MlModelCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationmlmodelcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmlmodelcreatedescription"></a>`description` | [`String`](#string) | Description of the model. | +| <a id="mutationmlmodelcreatename"></a>`name` | [`String!`](#string) | Name of the model. | +| <a id="mutationmlmodelcreateprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the model to mutate is in. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationmlmodelcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmlmodelcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationmlmodelcreatemodel"></a>`model` | [`MlModel`](#mlmodel) | Model after mutation. | + ### `Mutation.namespaceBanDestroy` Input type: `NamespaceBanDestroyInput` @@ -6184,7 +6291,7 @@ Input type: `ProjectCiCdSettingsUpdateInput` | <a id="mutationprojectcicdsettingsupdateinboundjobtokenscopeenabled"></a>`inboundJobTokenScopeEnabled` | [`Boolean`](#boolean) | Indicates CI/CD job tokens generated in other projects have restricted access to this project. | | <a id="mutationprojectcicdsettingsupdatejobtokenscopeenabled"></a>`jobTokenScopeEnabled` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** Outbound job token scope is being removed. This field can now only be set to false. Deprecated in 16.0. | | <a id="mutationprojectcicdsettingsupdatekeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Indicates if the latest artifact should be kept for the project. | -| <a id="mutationprojectcicdsettingsupdatemergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Indicates if merge pipelines are enabled for the project. | +| <a id="mutationprojectcicdsettingsupdatemergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Indicates if merged results pipelines are enabled for the project. | | <a id="mutationprojectcicdsettingsupdatemergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Indicates if merge trains are enabled for the project. | | <a id="mutationprojectcicdsettingsupdatemergetrainsskiptrainallowed"></a>`mergeTrainsSkipTrainAllowed` | [`Boolean`](#boolean) | Indicates whether an option is allowed to merge without refreshing the merge train. Ignored unless the `merge_trains_skip_train` feature flag is also enabled. | @@ -6722,6 +6829,9 @@ Input type: `RunnersExportUsageInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationrunnersexportusageclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnersexportusagefromdate"></a>`fromDate` | [`ISO8601Date`](#iso8601date) | UTC start date of the period to report on. Defaults to the start of last full month. | +| <a id="mutationrunnersexportusagemaxprojectcount"></a>`maxProjectCount` | [`Int`](#int) | Maximum number of projects to return. All other runner usage will be attributed to an `<Other projects>` entry. Defaults to 1000 projects. | +| <a id="mutationrunnersexportusagetodate"></a>`toDate` | [`ISO8601Date`](#iso8601date) | UTC end date of the period to report on. " \ "Defaults to the end of the month specified by `fromDate`. | | <a id="mutationrunnersexportusagetype"></a>`type` | [`CiRunnerType`](#cirunnertype) | Scope of the runners to include in the report. | #### Fields @@ -7522,11 +7632,6 @@ Input type: `UpdateDependencyProxyImageTtlGroupPolicyInput` Updates or creates dependency proxy for packages settings. Requires the packages and dependency proxy to be enabled in the config. Requires the packages feature to be enabled at the project level. -Error is raised if `packages_dependency_proxy_maven` feature flag is disabled. - -WARNING: -**Introduced** in 16.5. -This feature is an Experiment. It can be changed or removed at any time. Input type: `UpdateDependencyProxyPackagesSettingsInput` @@ -7535,10 +7640,10 @@ Input type: `UpdateDependencyProxyPackagesSettingsInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationupdatedependencyproxypackagessettingsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationupdatedependencyproxypackagessettingsenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates whether the dependency proxy for packages is enabled for the project. Introduced in 16.5: This feature is an Experiment. It can be changed or removed at any time. | +| <a id="mutationupdatedependencyproxypackagessettingsenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates whether the dependency proxy for packages is enabled for the project. | | <a id="mutationupdatedependencyproxypackagessettingsmavenexternalregistrypassword"></a>`mavenExternalRegistryPassword` | [`String`](#string) | Password for the external Maven packages registry. Introduced in 16.5: This feature is an Experiment. It can be changed or removed at any time. | -| <a id="mutationupdatedependencyproxypackagessettingsmavenexternalregistryurl"></a>`mavenExternalRegistryUrl` | [`String`](#string) | URL for the external Maven packages registry. Introduced in 16.5: This feature is an Experiment. It can be changed or removed at any time. | -| <a id="mutationupdatedependencyproxypackagessettingsmavenexternalregistryusername"></a>`mavenExternalRegistryUsername` | [`String`](#string) | Username for the external Maven packages registry. Introduced in 16.5: This feature is an Experiment. It can be changed or removed at any time. | +| <a id="mutationupdatedependencyproxypackagessettingsmavenexternalregistryurl"></a>`mavenExternalRegistryUrl` | [`String`](#string) | URL for the external Maven packages registry. | +| <a id="mutationupdatedependencyproxypackagessettingsmavenexternalregistryusername"></a>`mavenExternalRegistryUsername` | [`String`](#string) | Username for the external Maven packages registry. | | <a id="mutationupdatedependencyproxypackagessettingsprojectpath"></a>`projectPath` | [`ID!`](#id) | Project path for the dependency proxy for packages settings. | #### Fields @@ -7739,6 +7844,8 @@ Input type: `UpdateNamespacePackageSettingsInput` | <a id="mutationupdatenamespacepackagesettingsnugetduplicatesallowed"></a>`nugetDuplicatesAllowed` | [`Boolean`](#boolean) | Indicates whether duplicate NuGet packages are allowed for this namespace. | | <a id="mutationupdatenamespacepackagesettingsnugetsymbolserverenabled"></a>`nugetSymbolServerEnabled` | [`Boolean`](#boolean) | Indicates wheather the NuGet symbol server is enabled for this namespace. | | <a id="mutationupdatenamespacepackagesettingspypipackagerequestsforwarding"></a>`pypiPackageRequestsForwarding` | [`Boolean`](#boolean) | Indicates whether PyPI package forwarding is allowed for this namespace. | +| <a id="mutationupdatenamespacepackagesettingsterraformmoduleduplicateexceptionregex"></a>`terraformModuleDuplicateExceptionRegex` | [`UntrustedRegexp`](#untrustedregexp) | When terraform_module_duplicates_allowed is false, you can publish duplicate packages with names that match this regex. Otherwise, this setting has no effect. | +| <a id="mutationupdatenamespacepackagesettingsterraformmoduleduplicatesallowed"></a>`terraformModuleDuplicatesAllowed` | [`Boolean`](#boolean) | Indicates whether duplicate Terraform packages are allowed for this namespace. | #### Fields @@ -8584,6 +8691,7 @@ Input type: `WorkItemUpdateInput` | <a id="mutationworkitemupdateassigneeswidget"></a>`assigneesWidget` | [`WorkItemWidgetAssigneesInput`](#workitemwidgetassigneesinput) | Input for assignees widget. | | <a id="mutationworkitemupdateawardemojiwidget"></a>`awardEmojiWidget` | [`WorkItemWidgetAwardEmojiUpdateInput`](#workitemwidgetawardemojiupdateinput) | Input for emoji reactions widget. | | <a id="mutationworkitemupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemupdatecolorwidget"></a>`colorWidget` | [`WorkItemWidgetColorInput`](#workitemwidgetcolorinput) | Input for color widget. | | <a id="mutationworkitemupdateconfidential"></a>`confidential` | [`Boolean`](#boolean) | Sets the work item confidentiality. | | <a id="mutationworkitemupdatecurrentusertodoswidget"></a>`currentUserTodosWidget` | [`WorkItemWidgetCurrentUserTodosInput`](#workitemwidgetcurrentusertodosinput) | Input for to-dos widget. | | <a id="mutationworkitemupdatedescriptionwidget"></a>`descriptionWidget` | [`WorkItemWidgetDescriptionInput`](#workitemwidgetdescriptioninput) | Input for description widget. | @@ -8593,6 +8701,7 @@ Input type: `WorkItemUpdateInput` | <a id="mutationworkitemupdateiterationwidget"></a>`iterationWidget` | [`WorkItemWidgetIterationInput`](#workitemwidgetiterationinput) | Input for iteration widget. | | <a id="mutationworkitemupdatelabelswidget"></a>`labelsWidget` | [`WorkItemWidgetLabelsUpdateInput`](#workitemwidgetlabelsupdateinput) | Input for labels widget. | | <a id="mutationworkitemupdatemilestonewidget"></a>`milestoneWidget` | [`WorkItemWidgetMilestoneInput`](#workitemwidgetmilestoneinput) | Input for milestone widget. | +| <a id="mutationworkitemupdatenoteswidget"></a>`notesWidget` | [`WorkItemWidgetNotesInput`](#workitemwidgetnotesinput) | Input for notes widget. | | <a id="mutationworkitemupdatenotificationswidget"></a>`notificationsWidget` | [`WorkItemWidgetNotificationsUpdateInput`](#workitemwidgetnotificationsupdateinput) | Input for notifications widget. | | <a id="mutationworkitemupdateprogresswidget"></a>`progressWidget` | [`WorkItemWidgetProgressInput`](#workitemwidgetprogressinput) | Input for progress widget. | | <a id="mutationworkitemupdatestartandduedatewidget"></a>`startAndDueDateWidget` | [`WorkItemWidgetStartAndDueDateUpdateInput`](#workitemwidgetstartandduedateupdateinput) | Input for start and due date widget. | @@ -11801,6 +11910,43 @@ The edge type for [`MlCandidate`](#mlcandidate). | <a id="mlcandidateedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="mlcandidateedgenode"></a>`node` | [`MlCandidate`](#mlcandidate) | The item at the end of the edge. | +#### `MlModelConnection` + +The connection type for [`MlModel`](#mlmodel). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlmodelconnectionedges"></a>`edges` | [`[MlModelEdge]`](#mlmodeledge) | A list of edges. | +| <a id="mlmodelconnectionnodes"></a>`nodes` | [`[MlModel]`](#mlmodel) | A list of nodes. | +| <a id="mlmodelconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +##### Fields with arguments + +###### `MlModelConnection.count` + +Limited count of collection. Returns limit + 1 for counts greater than the limit. + +Returns [`Int!`](#int). + +####### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlmodelconnectioncountlimit"></a>`limit` | [`Int`](#int) | Limit value to be applied to the count query. Default is 1000. | + +#### `MlModelEdge` + +The edge type for [`MlModel`](#mlmodel). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlmodeledgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="mlmodeledgenode"></a>`node` | [`MlModel`](#mlmodel) | The item at the end of the edge. | + #### `MlModelVersionConnection` The connection type for [`MlModelVersion`](#mlmodelversion). @@ -14290,6 +14436,43 @@ Information about a connected Agent. | <a id="agentmetadatapodnamespace"></a>`podNamespace` | [`String`](#string) | Namespace of the pod running the Agent. | | <a id="agentmetadataversion"></a>`version` | [`String`](#string) | Agent version tag. | +### `AiAgent` + +An AI agent. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="aiagent_links"></a>`_links` | [`AiAgentLinks!`](#aiagentlinks) | Map of links to perform actions on the agent. | +| <a id="aiagentcreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | +| <a id="aiagentid"></a>`id` | [`ID!`](#id) | ID of the agent. | +| <a id="aiagentname"></a>`name` | [`String!`](#string) | Name of the agent. | +| <a id="aiagentversions"></a>`versions` | [`[AiAgentVersion!]`](#aiagentversion) | Versions of the agent. | + +### `AiAgentLinks` + +Represents links to perform actions on the agent. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="aiagentlinksshowpath"></a>`showPath` | [`String`](#string) | Path to the details page of the agent. | + +### `AiAgentVersion` + +Version of an AI Agent. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="aiagentversioncreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp when the agent version was created. | +| <a id="aiagentversionid"></a>`id` | [`ID!`](#id) | ID of the agent version. | +| <a id="aiagentversionmodel"></a>`model` | [`String!`](#string) | Model of the agent. | +| <a id="aiagentversionprompt"></a>`prompt` | [`String!`](#string) | Prompt of the agent. | + ### `AiMessage` AI features communication message. @@ -15430,6 +15613,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="cicatalogresourceversionsname"></a>`name` | [`String`](#string) | Name of the version. | | <a id="cicatalogresourceversionssort"></a>`sort` | [`CiCatalogResourceVersionSort`](#cicatalogresourceversionsort) | Sort versions by given criteria. | ### `CiCatalogResourceComponent` @@ -15439,9 +15623,9 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="cicatalogresourcecomponentid"></a>`id` **{warning-solid}** | [`CiCatalogResourcesComponentID!`](#cicatalogresourcescomponentid) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. ID of the component. | +| <a id="cicatalogresourcecomponentincludepath"></a>`includePath` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Path used to include the component. | | <a id="cicatalogresourcecomponentinputs"></a>`inputs` **{warning-solid}** | [`[CiCatalogResourceComponentInput!]`](#cicatalogresourcecomponentinput) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Inputs for the component. | | <a id="cicatalogresourcecomponentname"></a>`name` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Name of the component. | -| <a id="cicatalogresourcecomponentpath"></a>`path` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Path used to include the component. | ### `CiCatalogResourceComponentInput` @@ -15464,9 +15648,10 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="cicatalogresourceversioncomponents"></a>`components` **{warning-solid}** | [`CiCatalogResourceComponentConnection`](#cicatalogresourcecomponentconnection) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Components belonging to the catalog resource. | | <a id="cicatalogresourceversioncreatedat"></a>`createdAt` **{warning-solid}** | [`Time`](#time) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Timestamp of when the version was created. | | <a id="cicatalogresourceversionid"></a>`id` **{warning-solid}** | [`CiCatalogResourcesVersionID!`](#cicatalogresourcesversionid) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Global ID of the version. | +| <a id="cicatalogresourceversionname"></a>`name` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. Name that uniquely identifies the version within the catalog resource. | +| <a id="cicatalogresourceversionpath"></a>`path` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. Relative web path to the version. | +| <a id="cicatalogresourceversionreadmehtml"></a>`readmeHtml` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. GitLab Flavored Markdown rendering of README.md. This field can only be resolved for one version in any single request. | | <a id="cicatalogresourceversionreleasedat"></a>`releasedAt` **{warning-solid}** | [`Time`](#time) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Timestamp of when the version was released. | -| <a id="cicatalogresourceversiontagname"></a>`tagName` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Name of the tag associated with the version. | -| <a id="cicatalogresourceversiontagpath"></a>`tagPath` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Relative web path to the tag associated with the version. | ### `CiConfig` @@ -15626,6 +15811,7 @@ CI/CD variables for a GitLab instance. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="ciinstancevariabledescription"></a>`description` | [`String`](#string) | Description of the variable. | | <a id="ciinstancevariableenvironmentscope"></a>`environmentScope` **{warning-solid}** | [`String`](#string) | **Deprecated** in 15.3. No longer used, only available for GroupVariableType and ProjectVariableType. | | <a id="ciinstancevariableid"></a>`id` | [`ID!`](#id) | ID of the variable. | | <a id="ciinstancevariablekey"></a>`key` | [`String`](#string) | Name of the variable. | @@ -16388,6 +16574,8 @@ Represents a ComplianceFramework associated with a Project. | <a id="complianceframeworkname"></a>`name` | [`String!`](#string) | Name of the compliance framework. | | <a id="complianceframeworkpipelineconfigurationfullpath"></a>`pipelineConfigurationFullPath` | [`String`](#string) | Full path of the compliance pipeline configuration stored in a project repository, such as `.gitlab/.compliance-gitlab-ci.yml@compliance/hipaa` **(ULTIMATE ALL)**. | | <a id="complianceframeworkprojects"></a>`projects` | [`ProjectConnection`](#projectconnection) | Projects associated with the compliance framework. (see [Connections](#connections)) | +| <a id="complianceframeworkscanexecutionpolicies"></a>`scanExecutionPolicies` | [`ScanExecutionPolicyConnection`](#scanexecutionpolicyconnection) | Scan Execution Policies of the compliance framework. (see [Connections](#connections)) | +| <a id="complianceframeworkscanresultpolicies"></a>`scanResultPolicies` | [`ScanResultPolicyConnection`](#scanresultpolicyconnection) | Scan Result Policies of the compliance framework. (see [Connections](#connections)) | ### `ComplianceStandardsAdherence` @@ -16582,6 +16770,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="containerrepositorydetailstagsname"></a>`name` | [`String`](#string) | Search by tag name. | +| <a id="containerrepositorydetailstagsreferrers"></a>`referrers` | [`Boolean`](#boolean) | Include tag referrers. | | <a id="containerrepositorydetailstagssort"></a>`sort` | [`ContainerRepositoryTagSort`](#containerrepositorytagsort) | Sort tags by these criteria. | ### `ContainerRepositoryPermissions` @@ -16592,6 +16781,18 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="containerrepositorypermissionsdestroycontainerrepository"></a>`destroyContainerRepository` | [`Boolean!`](#boolean) | If `true`, the user can perform `destroy_container_image` on this resource. | +### `ContainerRepositoryReferrer` + +A referrer for a container repository tag. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="containerrepositoryreferrerartifacttype"></a>`artifactType` | [`String`](#string) | Artifact type of the referrer. | +| <a id="containerrepositoryreferrerdigest"></a>`digest` | [`String`](#string) | Digest of the referrer. | +| <a id="containerrepositoryreferreruserpermissions"></a>`userPermissions` | [`ContainerRepositoryTagPermissions!`](#containerrepositorytagpermissions) | Permissions for the current user on the resource. | + ### `ContainerRepositoryRegistry` Represents the Geo replication and verification state of an Container Repository. @@ -16630,6 +16831,8 @@ A tag from a container repository. | <a id="containerrepositorytaglocation"></a>`location` | [`String!`](#string) | URL of the tag. | | <a id="containerrepositorytagname"></a>`name` | [`String!`](#string) | Name of the tag. | | <a id="containerrepositorytagpath"></a>`path` | [`String!`](#string) | Path of the tag. | +| <a id="containerrepositorytagpublishedat"></a>`publishedAt` | [`Time`](#time) | Timestamp when the tag was published. | +| <a id="containerrepositorytagreferrers"></a>`referrers` | [`[ContainerRepositoryReferrer!]`](#containerrepositoryreferrer) | Referrers for this tag. | | <a id="containerrepositorytagrevision"></a>`revision` | [`String`](#string) | Revision of the tag. | | <a id="containerrepositorytagshortrevision"></a>`shortRevision` | [`String`](#string) | Short revision of the tag. | | <a id="containerrepositorytagtotalsize"></a>`totalSize` | [`BigInt`](#bigint) | Size of the tag. | @@ -16712,6 +16915,8 @@ The currently authenticated GitLab user. | <a id="currentusercommitemail"></a>`commitEmail` | [`String`](#string) | User's default commit email. | | <a id="currentusercreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of when the user was created. | | <a id="currentuserdiscord"></a>`discord` | [`String`](#string) | Discord ID of the user. | +| <a id="currentuserduochatavailable"></a>`duoChatAvailable` **{warning-solid}** | [`Boolean!`](#boolean) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. User access to AI chat feature. | +| <a id="currentuserduocodesuggestionsavailable"></a>`duoCodeSuggestionsAvailable` **{warning-solid}** | [`Boolean!`](#boolean) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. User access to code suggestions feature. | | <a id="currentuseremail"></a>`email` **{warning-solid}** | [`String`](#string) | **Deprecated** in 13.7. This was renamed. Use: [`User.publicEmail`](#userpublicemail). | | <a id="currentuseremails"></a>`emails` | [`EmailConnection`](#emailconnection) | User's email addresses. (see [Connections](#connections)) | | <a id="currentusergitpodenabled"></a>`gitpodEnabled` | [`Boolean`](#boolean) | Whether Gitpod is enabled at the user level. | @@ -17092,8 +17297,8 @@ Represents a product analytics dashboard visualization. | <a id="customizablepermissionavailablefor"></a>`availableFor` | [`[String!]!`](#string) | Objects the permission is available for. | | <a id="customizablepermissiondescription"></a>`description` | [`String`](#string) | Description of the permission. | | <a id="customizablepermissionname"></a>`name` | [`String!`](#string) | Localized name of the permission. | -| <a id="customizablepermissionrequirement"></a>`requirement` | [`String`](#string) | Requirement of the permission. | -| <a id="customizablepermissionvalue"></a>`value` | [`String!`](#string) | Value of the permission. | +| <a id="customizablepermissionrequirement"></a>`requirement` | [`MemberRolePermission`](#memberrolepermission) | Requirement of the permission. | +| <a id="customizablepermissionvalue"></a>`value` | [`MemberRolePermission!`](#memberrolepermission) | Value of the permission. | ### `DastPreScanVerification` @@ -17395,9 +17600,9 @@ Project-level Dependency Proxy for packages settings. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="dependencyproxypackagessettingenabled"></a>`enabled` **{warning-solid}** | [`Boolean!`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Indicates whether the dependency proxy for packages is enabled for the project. | -| <a id="dependencyproxypackagessettingmavenexternalregistryurl"></a>`mavenExternalRegistryUrl` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. URL for the external Maven packages registry. | -| <a id="dependencyproxypackagessettingmavenexternalregistryusername"></a>`mavenExternalRegistryUsername` **{warning-solid}** | [`String`](#string) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Username for the external Maven packages registry. | +| <a id="dependencyproxypackagessettingenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether the dependency proxy for packages is enabled for the project. | +| <a id="dependencyproxypackagessettingmavenexternalregistryurl"></a>`mavenExternalRegistryUrl` | [`String`](#string) | URL for the external Maven packages registry. | +| <a id="dependencyproxypackagessettingmavenexternalregistryusername"></a>`mavenExternalRegistryUsername` | [`String`](#string) | Username for the external Maven packages registry. | ### `DependencyProxySetting` @@ -18584,6 +18789,7 @@ Check permissions for the current user on an epic. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="epicpermissionsadminepic"></a>`adminEpic` | [`Boolean!`](#boolean) | If `true`, the user can perform `admin_epic` on this resource. | +| <a id="epicpermissionsadminepicrelation"></a>`adminEpicRelation` | [`Boolean!`](#boolean) | If `true`, the user can perform `admin_epic_relation` on this resource. | | <a id="epicpermissionsawardemoji"></a>`awardEmoji` | [`Boolean!`](#boolean) | If `true`, the user can perform `award_emoji` on this resource. | | <a id="epicpermissionscreateepic"></a>`createEpic` | [`Boolean!`](#boolean) | If `true`, the user can perform `create_epic` on this resource. | | <a id="epicpermissionscreatenote"></a>`createNote` | [`Boolean!`](#boolean) | If `true`, the user can perform `create_note` on this resource. | @@ -19166,7 +19372,7 @@ GPG signature for a signed commit. | <a id="groupautodevopsenabled"></a>`autoDevopsEnabled` | [`Boolean`](#boolean) | Indicates whether Auto DevOps is enabled for all projects within this group. | | <a id="groupavatarurl"></a>`avatarUrl` | [`String`](#string) | Avatar URL of the group. | | <a id="groupcontainerrepositoriescount"></a>`containerRepositoriesCount` | [`Int!`](#int) | Number of container repositories in the group. | -| <a id="groupcontainslockedprojects"></a>`containsLockedProjects` | [`Boolean!`](#boolean) | Includes at least one project where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | +| <a id="groupcontainslockedprojects"></a>`containsLockedProjects` | [`Boolean`](#boolean) | Includes at least one project where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | | <a id="groupcrossprojectpipelineavailable"></a>`crossProjectPipelineAvailable` | [`Boolean!`](#boolean) | Indicates if the cross_project_pipeline feature is available for the namespace. | | <a id="groupdependencyproxyblobcount"></a>`dependencyProxyBlobCount` | [`Int!`](#int) | Number of dependency proxy blobs cached in the group. | | <a id="groupdependencyproxyblobs"></a>`dependencyProxyBlobs` | [`DependencyProxyBlobConnection`](#dependencyproxyblobconnection) | Dependency Proxy blobs. (see [Connections](#connections)) | @@ -19178,6 +19384,7 @@ GPG signature for a signed commit. | <a id="groupdependencyproxytotalsize"></a>`dependencyProxyTotalSize` | [`String!`](#string) | Total size of the dependency proxy cached images. | | <a id="groupdependencyproxytotalsizebytes"></a>`dependencyProxyTotalSizeBytes` | [`BigInt!`](#bigint) | Total size of the dependency proxy cached images in bytes, encoded as a string. | | <a id="groupdependencyproxytotalsizeinbytes"></a>`dependencyProxyTotalSizeInBytes` **{warning-solid}** | [`Int!`](#int) | **Deprecated** in 16.1. Use `dependencyProxyTotalSizeBytes`. | +| <a id="groupdescendantgroupscount"></a>`descendantGroupsCount` | [`Int!`](#int) | Count of direct descendant groups of this group. | | <a id="groupdescription"></a>`description` | [`String`](#string) | Description of the namespace. | | <a id="groupdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="groupdora"></a>`dora` | [`Dora`](#dora) | Group's DORA metrics. | @@ -19190,8 +19397,9 @@ GPG signature for a signed commit. | <a id="groupfullname"></a>`fullName` | [`String!`](#string) | Full name of the namespace. | | <a id="groupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. | | <a id="groupgooglecloudloggingconfigurations"></a>`googleCloudLoggingConfigurations` | [`GoogleCloudLoggingConfigurationTypeConnection`](#googlecloudloggingconfigurationtypeconnection) | Google Cloud logging configurations that receive audit events belonging to the group. (see [Connections](#connections)) | +| <a id="groupgroupmemberscount"></a>`groupMembersCount` | [`Int!`](#int) | Count of direct members of this group. | | <a id="groupid"></a>`id` | [`ID!`](#id) | ID of the namespace. | -| <a id="groupistemporarystorageincreaseenabled"></a>`isTemporaryStorageIncreaseEnabled` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 16.7. Feature removal, will be completely removed in 17.0. | +| <a id="groupistemporarystorageincreaseenabled"></a>`isTemporaryStorageIncreaseEnabled` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 16.7. Feature removal, will be completely removed in 17.0. | | <a id="grouplfsenabled"></a>`lfsEnabled` | [`Boolean`](#boolean) | Indicates if Large File Storage (LFS) is enabled for namespace. | | <a id="groupmentionsdisabled"></a>`mentionsDisabled` | [`Boolean`](#boolean) | Indicates if a group is disabled from getting mentioned. | | <a id="groupname"></a>`name` | [`String!`](#string) | Name of the namespace. | @@ -19200,8 +19408,9 @@ GPG signature for a signed commit. | <a id="grouppath"></a>`path` | [`String!`](#string) | Path of the namespace. | | <a id="grouppendingmembers"></a>`pendingMembers` **{warning-solid}** | [`PendingGroupMemberConnection`](#pendinggroupmemberconnection) | **Introduced** in 16.6. This feature is an Experiment. It can be changed or removed at any time. A pending membership of a user within this group. | | <a id="groupprojectcreationlevel"></a>`projectCreationLevel` | [`String`](#string) | Permission level required to create projects in the group. | +| <a id="groupprojectscount"></a>`projectsCount` | [`Int!`](#int) | Count of direct projects in this group. | | <a id="grouprecentissueboards"></a>`recentIssueBoards` | [`BoardConnection`](#boardconnection) | List of recently visited boards of the group. Maximum size is 4. (see [Connections](#connections)) | -| <a id="grouprepositorysizeexcessprojectcount"></a>`repositorySizeExcessProjectCount` | [`Int!`](#int) | Number of projects in the root namespace where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | +| <a id="grouprepositorysizeexcessprojectcount"></a>`repositorySizeExcessProjectCount` | [`Int`](#int) | Number of projects in the root namespace where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | | <a id="grouprequestaccessenabled"></a>`requestAccessEnabled` | [`Boolean`](#boolean) | Indicates if users can request access to namespace. | | <a id="grouprequiretwofactorauthentication"></a>`requireTwoFactorAuthentication` | [`Boolean`](#boolean) | Indicates if all users in this group are required to set up two-factor authentication. | | <a id="grouprootstoragestatistics"></a>`rootStorageStatistics` | [`RootStorageStatistics`](#rootstoragestatistics) | Aggregated storage statistics of the namespace. Only available for root namespaces. | @@ -19856,6 +20065,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="groupmemberrolesid"></a>`id` | [`MemberRoleID`](#memberroleid) | Global ID of the member role to look up. | +| <a id="groupmemberrolesorderby"></a>`orderBy` | [`MemberRolesOrderBy`](#memberrolesorderby) | Ordering column. Default is NAME. | +| <a id="groupmemberrolessort"></a>`sort` | [`SortDirectionEnum`](#sortdirectionenum) | Ordering column. Default is ASC. | ##### `Group.mergeRequestViolations` @@ -20165,6 +20376,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupvulnerabilitieshasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have remediations. | | <a id="groupvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="groupvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="groupvulnerabilitiesowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="groupvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="groupvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="groupvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -20220,6 +20432,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | <a id="groupvulnerabilityseveritiescounthasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have remediations. | | <a id="groupvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="groupvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="groupvulnerabilityseveritiescountowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="groupvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="groupvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="groupvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | @@ -20670,6 +20883,7 @@ CI/CD variables a project inherites from its parent group and ancestors. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="inheritedcivariabledescription"></a>`description` | [`String`](#string) | Description of the variable. | | <a id="inheritedcivariableenvironmentscope"></a>`environmentScope` | [`String`](#string) | Scope defining the environments that can use the variable. | | <a id="inheritedcivariablegroupcicdsettingspath"></a>`groupCiCdSettingsPath` | [`String`](#string) | Indicates the path to the CI/CD settings of the group the variable belongs to. | | <a id="inheritedcivariablegroupname"></a>`groupName` | [`String`](#string) | Indicates group the variable belongs to. | @@ -20796,6 +21010,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | <a id="instancesecuritydashboardvulnerabilityseveritiescounthasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have remediations. | | <a id="instancesecuritydashboardvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="instancesecuritydashboardvulnerabilityseveritiescountowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | @@ -20984,6 +21199,7 @@ Check permissions for the current user on a issue. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="issuepermissionsadminissue"></a>`adminIssue` | [`Boolean!`](#boolean) | If `true`, the user can perform `admin_issue` on this resource. | +| <a id="issuepermissionsadminissuerelation"></a>`adminIssueRelation` | [`Boolean!`](#boolean) | If `true`, the user can perform `admin_issue_relation` on this resource. | | <a id="issuepermissionscreatedesign"></a>`createDesign` | [`Boolean!`](#boolean) | If `true`, the user can perform `create_design` on this resource. | | <a id="issuepermissionscreatenote"></a>`createNote` | [`Boolean!`](#boolean) | If `true`, the user can perform `create_note` on this resource. | | <a id="issuepermissionsdestroydesign"></a>`destroyDesign` | [`Boolean!`](#boolean) | If `true`, the user can perform `destroy_design` on this resource. | @@ -21288,6 +21504,16 @@ Represents links to perform actions on the candidate. | <a id="mlcandidatelinksartifactpath"></a>`artifactPath` | [`String`](#string) | Path to the artifact. | | <a id="mlcandidatelinksshowpath"></a>`showPath` | [`String`](#string) | Path to the details page of the candidate. | +### `MLModelLinks` + +Represents links to perform actions on the model. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlmodellinksshowpath"></a>`showPath` | [`String`](#string) | Path to the details page of the model. | + ### `MLModelVersionLinks` Represents links to perform actions on the model version. @@ -21296,6 +21522,7 @@ Represents links to perform actions on the model version. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="mlmodelversionlinkspackagepath"></a>`packagePath` | [`String`](#string) | Path to the package of the model version. | | <a id="mlmodelversionlinksshowpath"></a>`showPath` | [`String`](#string) | Path to the details page of the model version. | ### `MavenMetadata` @@ -21322,20 +21549,12 @@ Represents a member role. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="memberroleadmingroupmember"></a>`adminGroupMember` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to admin group members. | -| <a id="memberroleadminmergerequest"></a>`adminMergeRequest` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to admin merge requests. | -| <a id="memberroleadminvulnerability"></a>`adminVulnerability` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to admin vulnerability. | -| <a id="memberrolearchiveproject"></a>`archiveProject` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.6. This feature is an Experiment. It can be changed or removed at any time. Permission to archive projects. | | <a id="memberrolebaseaccesslevel"></a>`baseAccessLevel` **{warning-solid}** | [`AccessLevel!`](#accesslevel) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Base access level for the custom role. | | <a id="memberroledescription"></a>`description` | [`String`](#string) | Description of the member role. | -| <a id="memberroleenabledpermissions"></a>`enabledPermissions` **{warning-solid}** | [`[MemberRolePermission!]`](#memberrolepermission) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Array of all permissions enabled for the custom role. | +| <a id="memberroleenabledpermissions"></a>`enabledPermissions` **{warning-solid}** | [`CustomizablePermissionConnection`](#customizablepermissionconnection) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Array of all permissions enabled for the custom role. | | <a id="memberroleid"></a>`id` | [`MemberRoleID!`](#memberroleid) | ID of the member role. | -| <a id="memberrolemanageprojectaccesstokens"></a>`manageProjectAccessTokens` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to admin project access tokens. | | <a id="memberrolememberscount"></a>`membersCount` **{warning-solid}** | [`Int!`](#int) | **Introduced** in 16.7. This feature is an Experiment. It can be changed or removed at any time. Total number of members with the custom role. | | <a id="memberrolename"></a>`name` | [`String!`](#string) | Name of the member role. | -| <a id="memberrolereadcode"></a>`readCode` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to read code. | -| <a id="memberrolereaddependency"></a>`readDependency` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to read dependency. | -| <a id="memberrolereadvulnerability"></a>`readVulnerability` **{warning-solid}** | [`Boolean`](#boolean) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Permission to read vulnerability. | ### `MergeAccessLevel` @@ -21357,6 +21576,8 @@ Defines which user roles, users, or groups can merge into a protected branch. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestallowcollaboration"></a>`allowCollaboration` | [`Boolean`](#boolean) | Indicates if members of the target project can push to the fork. | +| <a id="mergerequestallowsmultipleassignees"></a>`allowsMultipleAssignees` | [`Boolean!`](#boolean) | Allows assigning multiple users to a merge request. | +| <a id="mergerequestallowsmultiplereviewers"></a>`allowsMultipleReviewers` | [`Boolean!`](#boolean) | Allows assigning multiple reviewers to a merge request. | | <a id="mergerequestapprovalstate"></a>`approvalState` | [`MergeRequestApprovalState!`](#mergerequestapprovalstate) | Information relating to rules that must be satisfied to merge this merge request. | | <a id="mergerequestapprovalsleft"></a>`approvalsLeft` | [`Int`](#int) | Number of approvals left. | | <a id="mergerequestapprovalsrequired"></a>`approvalsRequired` | [`Int`](#int) | Number of approvals required. | @@ -22923,9 +23144,14 @@ Machine learning model in the model registry. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="mlmodel_links"></a>`_links` | [`MLModelLinks!`](#mlmodellinks) | Map of links to perform actions on the model. | | <a id="mlmodelcandidates"></a>`candidates` | [`MlCandidateConnection`](#mlcandidateconnection) | Version candidates of the model. (see [Connections](#connections)) | +| <a id="mlmodelcreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | +| <a id="mlmodeldescription"></a>`description` | [`String!`](#string) | Description of the model. | | <a id="mlmodelid"></a>`id` | [`MlModelID!`](#mlmodelid) | ID of the model. | +| <a id="mlmodellatestversion"></a>`latestVersion` | [`MlModelVersion`](#mlmodelversion) | Latest version of the model. | | <a id="mlmodelname"></a>`name` | [`String!`](#string) | Name of the model. | +| <a id="mlmodelversioncount"></a>`versionCount` | [`Int`](#int) | Count of versions in the model. | | <a id="mlmodelversions"></a>`versions` | [`MlModelVersionConnection`](#mlmodelversionconnection) | Versions of the model. (see [Connections](#connections)) | ### `MlModelVersion` @@ -22962,19 +23188,19 @@ Product analytics events for a specific month and year. | <a id="namespaceactualrepositorysizelimit"></a>`actualRepositorySizeLimit` | [`Float`](#float) | Size limit for repositories in the namespace in bytes. This limit only applies to namespaces under Project limit enforcement. | | <a id="namespaceactualsizelimit"></a>`actualSizeLimit` | [`Float`](#float) | The actual storage size limit (in bytes) based on the enforcement type of either repository or namespace. This limit is agnostic of enforcement type. | | <a id="namespaceadditionalpurchasedstoragesize"></a>`additionalPurchasedStorageSize` | [`Float`](#float) | Additional storage purchased for the root namespace in bytes. | -| <a id="namespacecontainslockedprojects"></a>`containsLockedProjects` | [`Boolean!`](#boolean) | Includes at least one project where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | +| <a id="namespacecontainslockedprojects"></a>`containsLockedProjects` | [`Boolean`](#boolean) | Includes at least one project where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | | <a id="namespacecrossprojectpipelineavailable"></a>`crossProjectPipelineAvailable` | [`Boolean!`](#boolean) | Indicates if the cross_project_pipeline feature is available for the namespace. | | <a id="namespacedescription"></a>`description` | [`String`](#string) | Description of the namespace. | | <a id="namespacedescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="namespacefullname"></a>`fullName` | [`String!`](#string) | Full name of the namespace. | | <a id="namespacefullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. | | <a id="namespaceid"></a>`id` | [`ID!`](#id) | ID of the namespace. | -| <a id="namespaceistemporarystorageincreaseenabled"></a>`isTemporaryStorageIncreaseEnabled` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 16.7. Feature removal, will be completely removed in 17.0. | +| <a id="namespaceistemporarystorageincreaseenabled"></a>`isTemporaryStorageIncreaseEnabled` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated** in 16.7. Feature removal, will be completely removed in 17.0. | | <a id="namespacelfsenabled"></a>`lfsEnabled` | [`Boolean`](#boolean) | Indicates if Large File Storage (LFS) is enabled for namespace. | | <a id="namespacename"></a>`name` | [`String!`](#string) | Name of the namespace. | | <a id="namespacepackagesettings"></a>`packageSettings` | [`PackageSettings`](#packagesettings) | Package settings for the namespace. | | <a id="namespacepath"></a>`path` | [`String!`](#string) | Path of the namespace. | -| <a id="namespacerepositorysizeexcessprojectcount"></a>`repositorySizeExcessProjectCount` | [`Int!`](#int) | Number of projects in the root namespace where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | +| <a id="namespacerepositorysizeexcessprojectcount"></a>`repositorySizeExcessProjectCount` | [`Int`](#int) | Number of projects in the root namespace where the repository size exceeds the limit. This only applies to namespaces under Project limit enforcement. | | <a id="namespacerequestaccessenabled"></a>`requestAccessEnabled` | [`Boolean`](#boolean) | Indicates if users can request access to namespace. | | <a id="namespacerootstoragestatistics"></a>`rootStorageStatistics` | [`RootStorageStatistics`](#rootstoragestatistics) | Aggregated storage statistics of the namespace. Only available for root namespaces. | | <a id="namespacesharedrunnerssetting"></a>`sharedRunnersSetting` | [`SharedRunnersSetting`](#sharedrunnerssetting) | Shared runners availability for the namespace and its descendants. | @@ -23283,6 +23509,7 @@ Active period time range for on-call rotation. | <a id="organizationname"></a>`name` **{warning-solid}** | [`String!`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Name of the organization. | | <a id="organizationorganizationusers"></a>`organizationUsers` **{warning-solid}** | [`OrganizationUserConnection!`](#organizationuserconnection) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Users with access to the organization. | | <a id="organizationpath"></a>`path` **{warning-solid}** | [`String!`](#string) | **Introduced** in 16.4. This feature is an Experiment. It can be changed or removed at any time. Path of the organization. | +| <a id="organizationprojects"></a>`projects` **{warning-solid}** | [`ProjectConnection!`](#projectconnection) | **Introduced** in 16.8. This feature is an Experiment. It can be changed or removed at any time. Projects within this organization that the user has access to. | | <a id="organizationweburl"></a>`webUrl` **{warning-solid}** | [`String!`](#string) | **Introduced** in 16.6. This feature is an Experiment. It can be changed or removed at any time. Web URL of the organization. | #### Fields with arguments @@ -23605,6 +23832,8 @@ Namespace-level Package Registry settings. | <a id="packagesettingsnugetsymbolserverenabled"></a>`nugetSymbolServerEnabled` | [`Boolean!`](#boolean) | Indicates wheather the NuGet symbol server is enabled for this namespace. | | <a id="packagesettingspypipackagerequestsforwarding"></a>`pypiPackageRequestsForwarding` | [`Boolean`](#boolean) | Indicates whether PyPI package forwarding is allowed for this namespace. | | <a id="packagesettingspypipackagerequestsforwardinglocked"></a>`pypiPackageRequestsForwardingLocked` | [`Boolean!`](#boolean) | Indicates whether PyPI package forwarding settings are locked by a parent namespace. | +| <a id="packagesettingsterraformmoduleduplicateexceptionregex"></a>`terraformModuleDuplicateExceptionRegex` | [`UntrustedRegexp`](#untrustedregexp) | When terraform_module_duplicates_allowed is false, you can publish duplicate packages with names that match this regex. Otherwise, this setting has no effect. | +| <a id="packagesettingsterraformmoduleduplicatesallowed"></a>`terraformModuleDuplicatesAllowed` | [`Boolean!`](#boolean) | Indicates whether duplicate Terraform packages are allowed for this namespace. | ### `PackageTag` @@ -23839,7 +24068,7 @@ Returns [`PipelineSecurityReportFinding`](#pipelinesecurityreportfinding). ##### `Pipeline.securityReportFindings` -Vulnerability findings reported on the pipeline. +Vulnerability findings reported on the pipeline. By default all the states except dismissed are included in the response. Returns [`PipelineSecurityReportFindingConnection`](#pipelinesecurityreportfindingconnection). @@ -24085,6 +24314,8 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectactualrepositorysizelimit"></a>`actualRepositorySizeLimit` | [`Float`](#float) | Size limit for the repository in bytes. | | <a id="projectagentconfigurations"></a>`agentConfigurations` | [`AgentConfigurationConnection`](#agentconfigurationconnection) | Agent configurations defined by the project. (see [Connections](#connections)) | | <a id="projectallowmergeonskippedpipeline"></a>`allowMergeOnSkippedPipeline` | [`Boolean`](#boolean) | If `only_allow_merge_if_pipeline_succeeds` is true, indicates if merge requests of the project can also be merged with skipped jobs. | +| <a id="projectallowsmultiplemergerequestassignees"></a>`allowsMultipleMergeRequestAssignees` | [`Boolean!`](#boolean) | Project allows assigning multiple users to a merge request. | +| <a id="projectallowsmultiplemergerequestreviewers"></a>`allowsMultipleMergeRequestReviewers` | [`Boolean!`](#boolean) | Project allows assigning multiple reviewers to a merge request. | | <a id="projectapifuzzingciconfiguration"></a>`apiFuzzingCiConfiguration` | [`ApiFuzzingCiConfiguration`](#apifuzzingciconfiguration) | API fuzzing configuration for the project. | | <a id="projectarchived"></a>`archived` | [`Boolean`](#boolean) | Indicates the archived status of the project. | | <a id="projectautoclosereferencedissues"></a>`autocloseReferencedIssues` | [`Boolean`](#boolean) | Indicates if issues referenced by merge requests and commits within the default branch are closed automatically. | @@ -24094,8 +24325,8 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectcicdsettings"></a>`ciCdSettings` | [`ProjectCiCdSetting`](#projectcicdsetting) | CI/CD settings for the project. | | <a id="projectciconfigpathordefault"></a>`ciConfigPathOrDefault` | [`String!`](#string) | Path of the CI configuration file. | | <a id="projectcijobtokenscope"></a>`ciJobTokenScope` | [`CiJobTokenScopeType`](#cijobtokenscopetype) | The CI Job Tokens scope of access. | -| <a id="projectcisubscribedprojects"></a>`ciSubscribedProjects` | [`CiSubscriptionsProjectConnection`](#cisubscriptionsprojectconnection) | Triggers a new pipeline in the downstream project when a pipeline successfullycompletes on the(upstream) project. (see [Connections](#connections)) | -| <a id="projectcisubscriptionsprojects"></a>`ciSubscriptionsProjects` | [`CiSubscriptionsProjectConnection`](#cisubscriptionsprojectconnection) | Triggers a new pipeline in the(downstream) project when a pipeline successfullycompletes on the upstream project. (see [Connections](#connections)) | +| <a id="projectcisubscribedprojects"></a>`ciSubscribedProjects` | [`CiSubscriptionsProjectConnection`](#cisubscriptionsprojectconnection) | Pipeline subscriptions for projects subscribed to the project. (see [Connections](#connections)) | +| <a id="projectcisubscriptionsprojects"></a>`ciSubscriptionsProjects` | [`CiSubscriptionsProjectConnection`](#cisubscriptionsprojectconnection) | Pipeline subscriptions for the project. (see [Connections](#connections)) | | <a id="projectcodecoveragesummary"></a>`codeCoverageSummary` | [`CodeCoverageSummary`](#codecoveragesummary) | Code coverage summary associated with the project. | | <a id="projectcomplianceframeworks"></a>`complianceFrameworks` | [`ComplianceFrameworkConnection`](#complianceframeworkconnection) | Compliance frameworks associated with the project. (see [Connections](#connections)) | | <a id="projectcontainerexpirationpolicy"></a>`containerExpirationPolicy` | [`ContainerExpirationPolicy`](#containerexpirationpolicy) | Container expiration policy of the project. | @@ -24105,7 +24336,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectcreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of the project creation. | | <a id="projectdastscannerprofiles"></a>`dastScannerProfiles` | [`DastScannerProfileConnection`](#dastscannerprofileconnection) | DAST scanner profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdastsiteprofiles"></a>`dastSiteProfiles` | [`DastSiteProfileConnection`](#dastsiteprofileconnection) | DAST Site Profiles associated with the project. (see [Connections](#connections)) | -| <a id="projectdependencyproxypackagessetting"></a>`dependencyProxyPackagesSetting` **{warning-solid}** | [`DependencyProxyPackagesSetting`](#dependencyproxypackagessetting) | **Introduced** in 16.5. This feature is an Experiment. It can be changed or removed at any time. Packages Dependency Proxy settings for the project. Requires the packages and dependency proxy to be enabled in the config. Requires the packages feature to be enabled at the project level. Returns `null` if `packages_dependency_proxy_maven` feature flag is disabled. | +| <a id="projectdependencyproxypackagessetting"></a>`dependencyProxyPackagesSetting` | [`DependencyProxyPackagesSetting`](#dependencyproxypackagessetting) | Packages Dependency Proxy settings for the project. Requires the packages and dependency proxy to be enabled in the config. Requires the packages feature to be enabled at the project level. | | <a id="projectdescription"></a>`description` | [`String`](#string) | Short description of the project. | | <a id="projectdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="projectdetailedimportstatus"></a>`detailedImportStatus` | [`DetailedImportStatus`](#detailedimportstatus) | Detailed import status of the project. | @@ -25033,6 +25264,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectmemberrolesid"></a>`id` | [`MemberRoleID`](#memberroleid) | Global ID of the member role to look up. | +| <a id="projectmemberrolesorderby"></a>`orderBy` | [`MemberRolesOrderBy`](#memberrolesorderby) | Ordering column. Default is NAME. | +| <a id="projectmemberrolessort"></a>`sort` | [`SortDirectionEnum`](#sortdirectionenum) | Ordering column. Default is ASC. | ##### `Project.mergeRequest` @@ -25103,6 +25336,28 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectmilestonestimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | | <a id="projectmilestonestitle"></a>`title` | [`String`](#string) | Title of the milestone. | +##### `Project.mlModels` + +Finds machine learning models. + +WARNING: +**Introduced** in 16.8. +This feature is an Experiment. It can be changed or removed at any time. + +Returns [`MlModelConnection`](#mlmodelconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, and `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectmlmodelsname"></a>`name` | [`String`](#string) | Search for names that include the string. | +| <a id="projectmlmodelsorderby"></a>`orderBy` | [`MlModelsOrderBy`](#mlmodelsorderby) | Ordering column. Default is created_at. | +| <a id="projectmlmodelssort"></a>`sort` | [`SortDirectionEnum`](#sortdirectionenum) | Ordering column. Default is desc. | + ##### `Project.nestedEnvironments` Environments for this project with nested folders, can only be resolved for one project in any single request. @@ -25546,6 +25801,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectvulnerabilitieshasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have remediations. | | <a id="projectvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="projectvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="projectvulnerabilitiesowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="projectvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="projectvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="projectvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -25588,6 +25844,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | <a id="projectvulnerabilityseveritiescounthasremediations"></a>`hasRemediations` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have remediations. | | <a id="projectvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="projectvulnerabilityseveritiescountimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | +| <a id="projectvulnerabilityseveritiescountowasptopten"></a>`owaspTopTen` | [`[VulnerabilityOwaspTop10!]`](#vulnerabilityowasptop10) | Filter vulnerabilities by OWASP Top 10 category. | | <a id="projectvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="projectvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="projectvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | @@ -25674,7 +25931,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectcicdsettinginboundjobtokenscopeenabled"></a>`inboundJobTokenScopeEnabled` | [`Boolean`](#boolean) | Indicates CI/CD job tokens generated in other projects have restricted access to this project. | | <a id="projectcicdsettingjobtokenscopeenabled"></a>`jobTokenScopeEnabled` | [`Boolean`](#boolean) | Indicates CI/CD job tokens generated in this project have restricted access to other projects. | | <a id="projectcicdsettingkeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Whether to keep the latest builds artifacts. | -| <a id="projectcicdsettingmergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Whether merge pipelines are enabled. | +| <a id="projectcicdsettingmergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Whether merged results pipelines are enabled. | | <a id="projectcicdsettingmergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Whether merge trains are enabled. | | <a id="projectcicdsettingmergetrainsskiptrainallowed"></a>`mergeTrainsSkipTrainAllowed` | [`Boolean!`](#boolean) | Whether merge immediately is allowed for merge trains. | | <a id="projectcicdsettingproject"></a>`project` | [`Project`](#project) | Project the CI/CD settings belong to. | @@ -28776,6 +29033,18 @@ Represents the emoji reactions widget. | <a id="workitemwidgetawardemojitype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | | <a id="workitemwidgetawardemojiupvotes"></a>`upvotes` | [`Int!`](#int) | Number of upvotes the work item has received. | +### `WorkItemWidgetColor` + +Represents a color widget. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetcolorcolor"></a>`color` | [`String`](#string) | Color of the Work Item. | +| <a id="workitemwidgetcolortextcolor"></a>`textColor` | [`String`](#string) | Text color generated for the Work Item. | +| <a id="workitemwidgetcolortype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | + ### `WorkItemWidgetCurrentUserTodos` Represents a todos widget. @@ -28965,6 +29234,7 @@ Represents a notes widget. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="workitemwidgetnotesdiscussionlocked"></a>`discussionLocked` | [`Boolean`](#boolean) | Discussion lock attribute of the work item. | | <a id="workitemwidgetnotestype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | #### Fields with arguments @@ -29022,6 +29292,24 @@ Represents a legacy requirement widget. | <a id="workitemwidgetrequirementlegacylegacyiid"></a>`legacyIid` **{warning-solid}** | [`Int`](#int) | **Deprecated** in 15.9. Use Work Item IID instead. | | <a id="workitemwidgetrequirementlegacytype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | +### `WorkItemWidgetRolledupDates` + +Represents the rolledup dates widget. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetrolledupdatesduedate"></a>`dueDate` | [`Date`](#date) | Due date for the work item. | +| <a id="workitemwidgetrolledupdatesduedateisfixed"></a>`dueDateIsFixed` | [`Boolean`](#boolean) | Indicates if the due date for the work item is fixed. | +| <a id="workitemwidgetrolledupdatesduedatesourcingmilestone"></a>`dueDateSourcingMilestone` | [`Milestone`](#milestone) | Indicates which milestone sources the rolledup due date. | +| <a id="workitemwidgetrolledupdatesduedatesourcingworkitem"></a>`dueDateSourcingWorkItem` | [`WorkItem`](#workitem) | Indicates which work_item sources the rolledup due date. | +| <a id="workitemwidgetrolledupdatesstartdate"></a>`startDate` | [`Date`](#date) | Start date for the work item. | +| <a id="workitemwidgetrolledupdatesstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates if the start date for the work item is fixed. | +| <a id="workitemwidgetrolledupdatesstartdatesourcingmilestone"></a>`startDateSourcingMilestone` | [`Milestone`](#milestone) | Indicates which milestone sources the rolledup start date. | +| <a id="workitemwidgetrolledupdatesstartdatesourcingworkitem"></a>`startDateSourcingWorkItem` | [`WorkItem`](#workitem) | Indicates which work_item sources the rolledup start date. | +| <a id="workitemwidgetrolledupdatestype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | + ### `WorkItemWidgetStartAndDueDate` Represents a start and due date widget. @@ -30646,14 +30934,27 @@ Member role permission. | Value | Description | | ----- | ----------- | -| <a id="memberrolepermissionadmin_group_member"></a>`ADMIN_GROUP_MEMBER` | Allows to admin group members. | -| <a id="memberrolepermissionadmin_merge_request"></a>`ADMIN_MERGE_REQUEST` | Allows to approve merge requests. | +| <a id="memberrolepermissionadmin_group_member"></a>`ADMIN_GROUP_MEMBER` | Allows admin of group members. | +| <a id="memberrolepermissionadmin_merge_request"></a>`ADMIN_MERGE_REQUEST` | Allows approval of merge requests. | +| <a id="memberrolepermissionadmin_terraform_state"></a>`ADMIN_TERRAFORM_STATE` | Allows to admin terraform state. | | <a id="memberrolepermissionadmin_vulnerability"></a>`ADMIN_VULNERABILITY` | Allows admin access to the vulnerability reports. | -| <a id="memberrolepermissionarchive_project"></a>`ARCHIVE_PROJECT` | Allows to archive projects. | +| <a id="memberrolepermissionarchive_project"></a>`ARCHIVE_PROJECT` | Allows archiving of projects. | +| <a id="memberrolepermissionmanage_group_access_tokens"></a>`MANAGE_GROUP_ACCESS_TOKENS` | Allows manage access to the group access tokens. | | <a id="memberrolepermissionmanage_project_access_tokens"></a>`MANAGE_PROJECT_ACCESS_TOKENS` | Allows manage access to the project access tokens. | | <a id="memberrolepermissionread_code"></a>`READ_CODE` | Allows read-only access to the source code. | | <a id="memberrolepermissionread_dependency"></a>`READ_DEPENDENCY` | Allows read-only access to the dependencies. | | <a id="memberrolepermissionread_vulnerability"></a>`READ_VULNERABILITY` | Allows read-only access to the vulnerability reports. | +| <a id="memberrolepermissionremove_project"></a>`REMOVE_PROJECT` | Allows deletion of projects. | + +### `MemberRolesOrderBy` + +Values for ordering member roles by a specific field. + +| Value | Description | +| ----- | ----------- | +| <a id="memberrolesorderbycreated_at"></a>`CREATED_AT` | Ordered by creation time. | +| <a id="memberrolesorderbyid"></a>`ID` | Ordered by id. | +| <a id="memberrolesorderbyname"></a>`NAME` | Ordered by name. | ### `MemberSort` @@ -30820,6 +31121,17 @@ Milestone ID wildcard values. | <a id="milestonewildcardidstarted"></a>`STARTED` | Milestone assigned is open and started (start date <= today). | | <a id="milestonewildcardidupcoming"></a>`UPCOMING` | Milestone assigned is due in the future (due date > today). | +### `MlModelsOrderBy` + +Values for ordering machine learning models by a specific field. + +| Value | Description | +| ----- | ----------- | +| <a id="mlmodelsorderbycreated_at"></a>`CREATED_AT` | Ordered by creation time. | +| <a id="mlmodelsorderbyid"></a>`ID` | Ordered by id. | +| <a id="mlmodelsorderbyname"></a>`NAME` | Ordered by name. | +| <a id="mlmodelsorderbyupdated_at"></a>`UPDATED_AT` | Ordered by update time. | + ### `MoveType` The position to which the adjacent object should be moved. @@ -31327,6 +31639,7 @@ State of a Sentry error. | <a id="servicetypeconfluence_service"></a>`CONFLUENCE_SERVICE` | ConfluenceService type. | | <a id="servicetypecustom_issue_tracker_service"></a>`CUSTOM_ISSUE_TRACKER_SERVICE` | CustomIssueTrackerService type. | | <a id="servicetypedatadog_service"></a>`DATADOG_SERVICE` | DatadogService type. | +| <a id="servicetypediffblue_cover_service"></a>`DIFFBLUE_COVER_SERVICE` | DiffblueCoverService type. | | <a id="servicetypediscord_service"></a>`DISCORD_SERVICE` | DiscordService type. | | <a id="servicetypedrone_ci_service"></a>`DRONE_CI_SERVICE` | DroneCiService type. | | <a id="servicetypeemails_on_push_service"></a>`EMAILS_ON_PUSH_SERVICE` | EmailsOnPushService type. | @@ -31540,6 +31853,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumcloud_licensing_subscription_activation_banner"></a>`CLOUD_LICENSING_SUBSCRIPTION_ACTIVATION_BANNER` | Callout feature name for cloud_licensing_subscription_activation_banner. | | <a id="usercalloutfeaturenameenumcluster_security_warning"></a>`CLUSTER_SECURITY_WARNING` | Callout feature name for cluster_security_warning. | | <a id="usercalloutfeaturenameenumcode_suggestions_ga_non_owner_alert"></a>`CODE_SUGGESTIONS_GA_NON_OWNER_ALERT` | Callout feature name for code_suggestions_ga_non_owner_alert. | +| <a id="usercalloutfeaturenameenumcode_suggestions_ga_owner_alert"></a>`CODE_SUGGESTIONS_GA_OWNER_ALERT` | Callout feature name for code_suggestions_ga_owner_alert. | | <a id="usercalloutfeaturenameenumduo_chat_callout"></a>`DUO_CHAT_CALLOUT` | Callout feature name for duo_chat_callout. | | <a id="usercalloutfeaturenameenumeoa_bronze_plan_banner"></a>`EOA_BRONZE_PLAN_BANNER` | Callout feature name for eoa_bronze_plan_banner. | | <a id="usercalloutfeaturenameenumfeature_flags_new_version"></a>`FEATURE_FLAGS_NEW_VERSION` | Callout feature name for feature_flags_new_version. | @@ -31563,6 +31877,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumpipeline_needs_banner"></a>`PIPELINE_NEEDS_BANNER` | Callout feature name for pipeline_needs_banner. | | <a id="usercalloutfeaturenameenumpipeline_needs_hover_tip"></a>`PIPELINE_NEEDS_HOVER_TIP` | Callout feature name for pipeline_needs_hover_tip. | | <a id="usercalloutfeaturenameenumpreview_user_over_limit_free_plan_alert"></a>`PREVIEW_USER_OVER_LIMIT_FREE_PLAN_ALERT` | Callout feature name for preview_user_over_limit_free_plan_alert. | +| <a id="usercalloutfeaturenameenumproduct_analytics_dashboard_feedback"></a>`PRODUCT_ANALYTICS_DASHBOARD_FEEDBACK` | Callout feature name for product_analytics_dashboard_feedback. | | <a id="usercalloutfeaturenameenumprofile_personal_access_token_expiry"></a>`PROFILE_PERSONAL_ACCESS_TOKEN_EXPIRY` | Callout feature name for profile_personal_access_token_expiry. | | <a id="usercalloutfeaturenameenumproject_quality_summary_feedback"></a>`PROJECT_QUALITY_SUMMARY_FEEDBACK` | Callout feature name for project_quality_summary_feedback. | | <a id="usercalloutfeaturenameenumproject_repository_limit_alert_alert_threshold"></a>`PROJECT_REPOSITORY_LIMIT_ALERT_ALERT_THRESHOLD` | Callout feature name for project_repository_limit_alert_alert_threshold. | @@ -31661,7 +31976,7 @@ Stage event identifiers. ### `VerificationStatus` -Verification status of a GPG or X.509 signature for a commit. +Verification status of a GPG, X.509 or SSH signature for a commit. | Value | Description | | ----- | ----------- | @@ -31673,6 +31988,7 @@ Verification status of a GPG or X.509 signature for a commit. | <a id="verificationstatusunverified"></a>`UNVERIFIED` | unverified verification status. | | <a id="verificationstatusunverified_key"></a>`UNVERIFIED_KEY` | unverified_key verification status. | | <a id="verificationstatusverified"></a>`VERIFIED` | verified verification status. | +| <a id="verificationstatusverified_ca"></a>`VERIFIED_CA` | verified_ca verification status. | | <a id="verificationstatusverified_system"></a>`VERIFIED_SYSTEM` | verified_system verification status. | ### `VisibilityLevelsEnum` @@ -31763,6 +32079,33 @@ The type of the issue link related to a vulnerability. | <a id="vulnerabilityissuelinktypecreated"></a>`CREATED` | Issue is created for the vulnerability. | | <a id="vulnerabilityissuelinktyperelated"></a>`RELATED` | Has a related issue. | +### `VulnerabilityOwaspTop10` + +OwaspTop10 category of the vulnerability. + +| Value | Description | +| ----- | ----------- | +| <a id="vulnerabilityowasptop10a10_2017"></a>`A10_2017` | A10:2017-Insufficient Logging & Monitoring, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a10_2021"></a>`A10_2021` | A10:2021-Server-Side Request Forgery, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a1_2017"></a>`A1_2017` | A1:2017-Injection, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a1_2021"></a>`A1_2021` | A1:2021-Broken Access Control, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a2_2017"></a>`A2_2017` | A2:2017-Broken Authentication, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a2_2021"></a>`A2_2021` | A2:2021-Cryptographic Failures, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a3_2017"></a>`A3_2017` | A3:2017-Sensitive Data Exposure, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a3_2021"></a>`A3_2021` | A3:2021-Injection, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a4_2017"></a>`A4_2017` | A4:2017-XML External Entities (XXE), OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a4_2021"></a>`A4_2021` | A4:2021-Insecure Design, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a5_2017"></a>`A5_2017` | A5:2017-Broken Access Control, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a5_2021"></a>`A5_2021` | A5:2021-Security Misconfiguration, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a6_2017"></a>`A6_2017` | A6:2017-Security Misconfiguration, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a6_2021"></a>`A6_2021` | A6:2021-Vulnerable and Outdated Components, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a7_2017"></a>`A7_2017` | A7:2017-Cross-Site Scripting (XSS), OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a7_2021"></a>`A7_2021` | A7:2021-Identification and Authentication Failures, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a8_2017"></a>`A8_2017` | A8:2017-Insecure Deserialization, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a8_2021"></a>`A8_2021` | A8:2021-Software and Data Integrity Failures, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a9_2017"></a>`A9_2017` | A9:2017-Using Components with Known Vulnerabilities, OWASP top 10 category. | +| <a id="vulnerabilityowasptop10a9_2021"></a>`A9_2021` | A9:2021-Security Logging and Monitoring Failures, OWASP top 10 category. | + ### `VulnerabilityReportType` The type of the security scan that found the vulnerability. @@ -31895,6 +32238,7 @@ Type of a work item widget. | ----- | ----------- | | <a id="workitemwidgettypeassignees"></a>`ASSIGNEES` | Assignees widget. | | <a id="workitemwidgettypeaward_emoji"></a>`AWARD_EMOJI` | Award Emoji widget. | +| <a id="workitemwidgettypecolor"></a>`COLOR` | Color widget. | | <a id="workitemwidgettypecurrent_user_todos"></a>`CURRENT_USER_TODOS` | Current User Todos widget. | | <a id="workitemwidgettypedescription"></a>`DESCRIPTION` | Description widget. | | <a id="workitemwidgettypehealth_status"></a>`HEALTH_STATUS` | Health Status widget. | @@ -31907,6 +32251,7 @@ Type of a work item widget. | <a id="workitemwidgettypenotifications"></a>`NOTIFICATIONS` | Notifications widget. | | <a id="workitemwidgettypeprogress"></a>`PROGRESS` | Progress widget. | | <a id="workitemwidgettyperequirement_legacy"></a>`REQUIREMENT_LEGACY` | Requirement Legacy widget. | +| <a id="workitemwidgettyperolledup_dates"></a>`ROLLEDUP_DATES` | Rolledup Dates widget. | | <a id="workitemwidgettypestart_and_due_date"></a>`START_AND_DUE_DATE` | Start And Due Date widget. | | <a id="workitemwidgettypestatus"></a>`STATUS` | Status widget. | | <a id="workitemwidgettypetest_reports"></a>`TEST_REPORTS` | Test Reports widget. | @@ -33580,6 +33925,7 @@ Implementations: - [`WorkItemWidgetAssignees`](#workitemwidgetassignees) - [`WorkItemWidgetAwardEmoji`](#workitemwidgetawardemoji) +- [`WorkItemWidgetColor`](#workitemwidgetcolor) - [`WorkItemWidgetCurrentUserTodos`](#workitemwidgetcurrentusertodos) - [`WorkItemWidgetDescription`](#workitemwidgetdescription) - [`WorkItemWidgetHealthStatus`](#workitemwidgethealthstatus) @@ -33592,6 +33938,7 @@ Implementations: - [`WorkItemWidgetNotifications`](#workitemwidgetnotifications) - [`WorkItemWidgetProgress`](#workitemwidgetprogress) - [`WorkItemWidgetRequirementLegacy`](#workitemwidgetrequirementlegacy) +- [`WorkItemWidgetRolledupDates`](#workitemwidgetrolledupdates) - [`WorkItemWidgetStartAndDueDate`](#workitemwidgetstartandduedate) - [`WorkItemWidgetStatus`](#workitemwidgetstatus) - [`WorkItemWidgetTestReports`](#workitemwidgettestreports) @@ -34394,6 +34741,7 @@ Attributes for value stream stage. | <a id="workitemupdatedtaskinputid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | | <a id="workitemupdatedtaskinputlabelswidget"></a>`labelsWidget` | [`WorkItemWidgetLabelsUpdateInput`](#workitemwidgetlabelsupdateinput) | Input for labels widget. | | <a id="workitemupdatedtaskinputmilestonewidget"></a>`milestoneWidget` | [`WorkItemWidgetMilestoneInput`](#workitemwidgetmilestoneinput) | Input for milestone widget. | +| <a id="workitemupdatedtaskinputnoteswidget"></a>`notesWidget` | [`WorkItemWidgetNotesInput`](#workitemwidgetnotesinput) | Input for notes widget. | | <a id="workitemupdatedtaskinputnotificationswidget"></a>`notificationsWidget` | [`WorkItemWidgetNotificationsUpdateInput`](#workitemwidgetnotificationsupdateinput) | Input for notifications widget. | | <a id="workitemupdatedtaskinputstartandduedatewidget"></a>`startAndDueDateWidget` | [`WorkItemWidgetStartAndDueDateUpdateInput`](#workitemwidgetstartandduedateupdateinput) | Input for start and due date widget. | | <a id="workitemupdatedtaskinputstateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | @@ -34416,6 +34764,14 @@ Attributes for value stream stage. | <a id="workitemwidgetawardemojiupdateinputaction"></a>`action` | [`WorkItemAwardEmojiUpdateAction!`](#workitemawardemojiupdateaction) | Action for the update. | | <a id="workitemwidgetawardemojiupdateinputname"></a>`name` | [`String!`](#string) | Emoji name. | +### `WorkItemWidgetColorInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetcolorinputcolor"></a>`color` | [`Color!`](#color) | Color of the work item. | + ### `WorkItemWidgetCurrentUserTodosInput` #### Arguments @@ -34485,6 +34841,14 @@ Attributes for value stream stage. | ---- | ---- | ----------- | | <a id="workitemwidgetmilestoneinputmilestoneid"></a>`milestoneId` | [`MilestoneID`](#milestoneid) | Milestone to assign to the work item. | +### `WorkItemWidgetNotesInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetnotesinputdiscussionlocked"></a>`discussionLocked` | [`Boolean!`](#boolean) | Discussion lock attribute for notes widget of the work item. | + ### `WorkItemWidgetNotificationsUpdateInput` #### Arguments diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index 7a3b031f18a..7e065dd87d2 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -32,7 +32,7 @@ To preserve the member list and their respective permissions on imported groups, ## Prerequisites - For information on prerequisites for group import and export API, see prerequisites for - [migrating groups by uploading an export file](../user/group/import/index.md#preparation). + [migrating groups by uploading an export file](../user/project/settings/import_export.md#preparation). ## Schedule new export diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index 109f3ba5f6e..aeab76b7212 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -49,7 +49,10 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ## Show variable details -Get the details of a group's specific variable. +> The `filter` parameter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340185) in GitLab 16.9. + +Get the details of a group's specific variable. If there are multiple variables with the same key, +use `filter` to select the correct `environment_scope`. ```plaintext GET /groups/:id/variables/:key @@ -59,6 +62,7 @@ GET /groups/:id/variables/:key |-----------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | | `key` | string | Yes | The `key` of a variable | +| `filter` | hash | No | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables/TEST_VARIABLE_1" @@ -85,17 +89,17 @@ Create a new variable. POST /groups/:id/variables ``` -| Attribute | Type | Required | Description | -|-----------------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | -| `key` | string | Yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | -| `value` | string | Yes | The `value` of a variable | -| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | No | Whether the variable is protected | -| `masked` | boolean | No | Whether the variable is masked | -| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| Attribute | Type | Required | Description | +|---------------------------------------|----------------|----------|-------------| +| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | +| `key` | string | Yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `value` | string | Yes | The `value` of a variable | +| `description` | string | No | The `description` of the variable. Default: `null` | | `environment_scope` **(PREMIUM ALL)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | -| `description` | string | No | The `description` of the variable. Default: `null` | +| `masked` | boolean | No | Whether the variable is masked | +| `protected` | boolean | No | Whether the variable is protected | +| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -117,23 +121,27 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ## Update variable -Update a group's variable. +> The `filter` parameter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340185) in GitLab 16.9. + +Update a group's variable. If there are multiple variables with the same key, +use `filter` to select the correct `environment_scope`. ```plaintext PUT /groups/:id/variables/:key ``` -| Attribute | Type | Required | Description | -|-----------------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | -| `key` | string | Yes | The `key` of a variable | -| `value` | string | Yes | The `value` of a variable | -| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | No | Whether the variable is protected | -| `masked` | boolean | No | Whether the variable is masked | -| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| Attribute | Type | Required | Description | +|---------------------------------------|----------------|----------|-------------| +| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | +| `key` | string | Yes | The `key` of a variable | +| `value` | string | Yes | The `value` of a variable | +| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | | `environment_scope` **(PREMIUM ALL)** | string | No | The [environment scope](../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | -| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | +| `filter` | hash | No | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | +| `masked` | boolean | No | Whether the variable is masked | +| `protected` | boolean | No | Whether the variable is protected | +| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -155,7 +163,10 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ ## Remove variable -Remove a group's variable. +> The `filter` parameter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340185) in GitLab 16.9. + +Remove a group's variable. If there are multiple variables with the same key, +use `filter` to select the correct `environment_scope`. ```plaintext DELETE /groups/:id/variables/:key @@ -163,10 +174,47 @@ DELETE /groups/:id/variables/:key | Attribute | Type | Required | Description | |-----------|----------------|----------|-------------| -| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | +| `id` | integer/string | Yes | The ID of a group or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) | | `key` | string | Yes | The `key` of a variable | +| `filter` | hash | No | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/groups/1/variables/VARIABLE_1" ``` + +## The `filter` parameter **(PREMIUM ALL)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340185) in GitLab 16.9. + +When multiple variables have the same `key`, [GET](#show-variable-details), [PUT](#update-variable), +or [DELETE](#remove-variable) requests might return: + +```plaintext +There are multiple variables with provided parameters. Please use 'filter[environment_scope]'. +``` + +Use `filter[environment_scope]` to select the variable with the matching `environment_scope` attribute. + +For example: + +- GET: + + ```shell + curl --globoff --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables/SCOPED_VARIABLE_1?filter[environment_scope]=production" + ``` + +- PUT: + + ```shell + curl --request PUT --globoff --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables/SCOPED_VARIABLE_1?value=scoped-variable-updated-value&environment_scope=production&filter[environment_scope]=production" + ``` + +- DELETE: + + ```shell + curl --request DELETE --globoff --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables/SCOPED_VARIABLE_1?filter[environment_scope]=production" + ``` diff --git a/doc/api/group_relations_export.md b/doc/api/group_relations_export.md index e1c0cd81bd6..bd48fb13ec6 100644 --- a/doc/api/group_relations_export.md +++ b/doc/api/group_relations_export.md @@ -13,7 +13,7 @@ top-level relation (for example, milestones, boards, and labels). The group relations export API is primarily used in -[group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) +[group migration by direct transfer](../user/group/import/index.md) and can't be used with the [group import and export API](group_import_export.md). diff --git a/doc/api/groups.md b/doc/api/groups.md index 2dbfb5a4937..90820727040 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -147,7 +147,7 @@ If you request this list as: - An unauthenticated user, the response returns only public groups. - An authenticated user, the response returns only the groups you're -a member of and does not include public groups. + a member of and does not include public groups. Parameters: @@ -306,7 +306,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user | | `archived` | boolean | no | Limit by archived status | | `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | -| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `similarity` (1), or `last_activity_at` fields. Default is `created_at` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `similarity` <sup>1</sup>, or `last_activity_at` fields. Default is `created_at` | | `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Return list of authorized projects matching the search criteria | | `simple` | boolean | no | Return only limited fields for each project. This is a no-op without authentication where only simple fields are returned. | @@ -316,14 +316,18 @@ Parameters: | `with_issues_enabled` | boolean | no | Limit by projects with issues feature enabled. Default is `false` | | `with_merge_requests_enabled` | boolean | no | Limit by projects with merge requests feature enabled. Default is `false` | | `with_shared` | boolean | no | Include projects shared to this group. Default is `true` | -| `include_subgroups` | boolean | no | Include projects in subgroups of this group. Default is `false` | +| `include_subgroups` | boolean | no | Include projects in subgroups of this group. Default is `false` | | `min_access_level` | integer | no | Limit to projects where current user has at least this [role (`access_level`)](members.md#roles) | | `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (administrators only) | -| `with_security_reports` **(ULTIMATE ALL)** | boolean | no | Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | +| `with_security_reports` **(ULTIMATE ALL)** | boolean | no | Return only projects that have security reports artifacts present in any of their builds. This means "projects with security reports enabled". Default is `false` | -1. Order by similarity: Orders the results by a similarity score calculated from the provided `search` -URL parameter. When using `order_by=similarity`, the `sort` parameter is ignored. When the `search` -parameter is not provided, the API returns the projects ordered by `name`. +<html> +<small>Footnotes: + <ol> + <li>Order by similarity: Orders the results by a similarity score calculated from the provided `search` URL parameter. When using `order_by=similarity`, the `sort` parameter is ignored. When the `search` parameter is not provided, the API returns the projects ordered by `name`.</li> + </ol> +</small> +</html> Example response: @@ -816,30 +820,32 @@ POST /groups Parameters: -| Attribute | Type | Required | Description | -| ------------------------------------------------------- | ------- | -------- | ----------- | -| `name` | string | yes | The name of the group. | -| `path` | string | yes | The path of the group. | -| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | -| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | -| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | -| `description` | string | no | The group's description. | -| `emails_disabled` | boolean | no | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127899) in GitLab 16.5.)_ Disable email notifications. Use `emails_enabled` instead. | -| `emails_enabled` | boolean | no | Enable email notifications. | -| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | -| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned. | -| `parent_id` | integer | no | The parent group ID for creating nested group. | +| Attribute | Type | Required | Description | +| ------------------------------------------------------- | ------- | -------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `name` | string | yes | The name of the group. | +| `path` | string | yes | The path of the group. | +| `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | +| `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | +| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | +| `default_branch_protection_defaults` | hash | no | See [Options for `default_branch_protection_defaults`](#options-for-default_branch_protection_defaults). | +| `description` | string | no | The group's description. | +| `emails_disabled` | boolean | no | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127899) in GitLab 16.5.)_ Disable email notifications. Use `emails_enabled` instead. | +| `emails_enabled` | boolean | no | Enable email notifications. | +| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | +| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned. | +| `organization_id` | integer | no | The organization ID for the group. | +| `parent_id` | integer | no | The parent group ID for creating nested group. | | `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | -| `request_access_enabled` | boolean | no | Allow users to request member access. | -| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | -| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | -| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | -| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | -| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `membership_lock` **(PREMIUM ALL)** | boolean | no | Users cannot be added to projects in this group. | -| `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Additional compute minutes for this group. | -| `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Maximum number of monthly compute minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | -| `wiki_access_level` **(PREMIUM ALL)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | +| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | +| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | +| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | +| `membership_lock` **(PREMIUM ALL)** | boolean | no | Users cannot be added to projects in this group. | +| `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Additional compute minutes for this group. | +| `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Maximum number of monthly compute minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | +| `wiki_access_level` **(PREMIUM ALL)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | ### Options for `default_branch_protection` @@ -853,6 +859,18 @@ The `default_branch_protection` attribute determines whether users with the Deve | `3` | Protected against pushes. Users with the Maintainer role can: <br>- Push new commits<br>- Force push changes<br>- Accept merge requests<br>Users with the Developer role can:<br>- Accept merge requests| | `4` | Full protection after initial push. User with the Developer role can: <br>- Push commit to empty repository.<br> Users with the Maintainer role can: <br>- Push new commits<br>- Accept merge requests| +### Options for `default_branch_protection_defaults` + +The `default_branch_protection_defaults` attribute describes the default branch +protection defaults. All parameters are optional. + +| Key | Type | Description | +|------------------------------|---------|-----------------------------------------------------------------------------------------| +| `allowed_to_push` | array | An array of access levels allowed to push. Supports Developer (30) or Maintainer (40). | +| `allow_force_push` | boolean | Allow force push for all users with push access. | +| `allowed_to_merge` | array | An array of access levels allowed to merge. Supports Developer (30) or Maintainer (40). | +| `developer_can_initial_push` | boolean | Allow developers to initial push. | + ## New Subgroup This is similar to creating a [New group](#new-group). You need the `parent_id` from the [List groups](#list-groups) call. You can then enter the desired: @@ -981,6 +999,7 @@ PUT /groups/:id | `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | | `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | | `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). | +| `default_branch_protection_defaults` | hash | no | See [Options for `default_branch_protection_defaults`](#options-for-default_branch_protection_defaults). | | `description` | string | no | The description of the group. | | `emails_disabled` | boolean | no | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127899) in GitLab 16.5.)_ Disable email notifications. Use `emails_enabled` instead. | | `emails_enabled` | boolean | no | Enable email notifications. | @@ -1689,6 +1708,7 @@ To delete the LDAP group link, provide either a `cn` or a `filter`, but not both > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/290367) in GitLab 15.3.0. > - `access_level` type [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95607) from `string` to `integer` in GitLab 15.3.3. > - `member_role_id` type [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/417201) in GitLab 16.7 [with a flag](../administration/feature_flags.md) named `custom_roles_for_saml_group_links`. Disabled by default. +> - `member_role_id` type [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/417201) in GitLab 16.8. Feature flag `custom_roles_for_saml_group_links` removed. List, get, add, and delete SAML group links. diff --git a/doc/api/instance_level_ci_variables.md b/doc/api/instance_level_ci_variables.md index a4018a46525..466479e0cdb 100644 --- a/doc/api/instance_level_ci_variables.md +++ b/doc/api/instance_level_ci_variables.md @@ -22,6 +22,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a [ { "key": "TEST_VARIABLE_1", + "description": null, "variable_type": "env_var", "value": "TEST_1", "protected": false, @@ -30,6 +31,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a }, { "key": "TEST_VARIABLE_2", + "description": null, "variable_type": "env_var", "value": "TEST_2", "protected": false, @@ -58,6 +60,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ```json { "key": "TEST_VARIABLE_1", + "description": null, "variable_type": "env_var", "value": "TEST_1", "protected": false, @@ -93,6 +96,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ```json { "key": "NEW_VARIABLE", + "description": null, "value": "new value", "variable_type": "env_var", "protected": false, @@ -126,6 +130,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ ```json { "key": "NEW_VARIABLE", + "description": null, "value": "updated value", "variable_type": "env_var", "protected": true, diff --git a/doc/api/integrations.md b/doc/api/integrations.md index f713d845762..3e2286fa0a5 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -98,17 +98,17 @@ Parameters: | `app_store_private_key` | string | true | Apple App Store Connect private key. | | `app_store_protected_refs` | boolean | false | Set variables on protected branches and tags only. | -### Disable Apple App Store +### Disable Apple App Store Connect -Disable the Apple App Store integration for a project. Integration settings are reset. +Disable the Apple App Store Connect integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/apple_app_store ``` -### Get Apple App Store settings +### Get Apple App Store Connect settings -Get the Apple App Store integration settings for a project. +Get the Apple App Store Connect integration settings for a project. ```plaintext GET /projects/:id/integrations/apple_app_store @@ -198,8 +198,8 @@ Parameters: | --------- | ---- | -------- | ----------- | | `bamboo_url` | string | true | Bamboo root URL (for example, `https://bamboo.example.com`). | | `enable_ssl_verification` | boolean | false | Enable SSL verification. Defaults to `true` (enabled). | -| `build_key` | string | true | Bamboo build plan key like `KEY`. | -| `username` | string | true | A user with API access, if applicable. | +| `build_key` | string | true | Bamboo build plan key (for example, `KEY`). | +| `username` | string | true | User with API access to the Bamboo server. | | `password` | string | true | Password of the user. | ### Disable Atlassian Bamboo @@ -232,9 +232,9 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New issue URL. | -| `issues_url` | string | true | Issue URL. | -| `project_url` | string | true | Project URL. | +| `new_issue_url` | string | true | URL of the new issue. | +| `issues_url` | string | true | URL of the issue. | +| `project_url` | string | true | URL of the project. | ### Disable Bugzilla @@ -303,9 +303,9 @@ Parameters: | Parameter | Type | Required | Description | |---------------|---------|----------|---------------------------------------------------------------------------------------------| -| `token` | string | true | Campfire API token. To find it, sign in to Campfire and select **My info**. | -| `subdomain` | string | false | Campfire subdomain. Text between `https://` and `.campfirenow.com` when you're logged in. | -| `room` | string | false | Campfire room. The last part of the URL when you're in a room. | +| `token` | string | true | API authentication token from Campfire. To get the token, sign in to Campfire and select **My info**. | +| `subdomain` | string | false | `.campfirenow.com` subdomain when you're signed in. | +| `room` | string | false | ID portion of the Campfire room URL. | ### Disable Campfire @@ -339,8 +339,8 @@ Parameters: | Parameter | Type | Required | Description | | ------------- | ------ | -------- | -------------- | -| `issues_url` | string | true | Issue URL. | -| `project_url` | string | true | Project URL. | +| `issues_url` | string | true | URL of the issue. | +| `project_url` | string | true | URL of the project. | ### Disable ClickUp @@ -372,7 +372,7 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `confluence_url` | string | true | The URL of the Confluence Workspace hosted on `atlassian.net`. | +| `confluence_url` | string | true | URL of the Confluence Workspace hosted on `atlassian.net`. | ### Disable Confluence Workspace @@ -404,9 +404,9 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New issue URL. | -| `issues_url` | string | true | Issue URL. | -| `project_url` | string | true | Project URL. | +| `new_issue_url` | string | true | URL of the new issue. | +| `issues_url` | string | true | URL of the issue. | +| `project_url` | string | true | URL of the project. | ### Disable a custom issue tracker @@ -462,6 +462,40 @@ Get the Datadog integration settings for a project. GET /projects/:id/integrations/datadog ``` +## Diffblue Cover + +### Set up Diffblue Cover + +Set up the Diffblue Cover integration for a project. + +```plaintext +PUT /projects/:id/integrations/diffblue-cover +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `diffblue_license_key` | string | true | Diffblue Cover license key. | +| `diffblue_access_token_name` | string | true | Access token name used by Diffblue Cover in pipelines. | +| `diffblue_access_token_secret` | string | true | Access token secret used by Diffblue Cover in pipelines. | + +### Disable Diffblue Cover + +Disable the Diffblue Cover integration for a project. Integration settings are reset. + +```plaintext +DELETE /projects/:id/integrations/diffblue-cover +``` + +### Get Diffblue Cover settings + +Get the Diffblue Cover integration settings for a project. + +```plaintext +GET /projects/:id/integrations/diffblue-cover +``` + ## Discord Notifications ### Set up Discord Notifications @@ -478,7 +512,7 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `webhook` | string | true | Discord webhook (for example, `https://discord.com/api/webhooks/…`). | +| `webhook` | string | true | Discord webhook (for example, `https://discord.com/api/webhooks/...`). | | `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are `all`, `default`, `protected`, and `default_and_protected`. The default value is `default`. | | `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | | `confidential_issue_channel` | string | false | The webhook override to receive notifications for confidential issue events. | @@ -610,9 +644,9 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | The URL to create an issue in EWM. | -| `project_url` | string | true | The URL to the project in EWM. | -| `issues_url` | string | true | The URL to view an issue in EWM. Must contain `:id`. | +| `new_issue_url` | string | true | URL of the new issue. | +| `project_url` | string | true | URL of the project. | +| `issues_url` | string | true | URL of the issue. | ### Disable EWM @@ -644,7 +678,7 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `external_wiki_url` | string | true | The URL of the external wiki. | +| `external_wiki_url` | string | true | URL of the external wiki. | ### Disable an external wiki @@ -696,111 +730,118 @@ Get the GitHub integration settings for a project. GET /projects/:id/integrations/github ``` -## GitLab for Slack app +## Google Chat -### Set up the GitLab for Slack app +### Set up Google Chat -Set up the GitLab for Slack app for a project. +Set up the Google Chat integration for a project. ```plaintext -PUT /projects/:id/integrations/slack +PUT /projects/:id/integrations/hangouts-chat ``` Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `webhook` | string | true | `https://hooks.slack.com/services/...`. | -| `username` | string | false | username. | -| `channel` | string | false | Default channel to use if others are not configured. | +| `webhook` | string | true | The Hangouts Chat webhook (for example, `https://chat.googleapis.com/v1/spaces...`). | | `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines. | | `notify_only_default_branch` | boolean | false | **Deprecated:** This parameter has been replaced with `branches_to_be_notified`. | | `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are `all`, `default`, `protected`, and `default_and_protected`. The default value is `default`. | -| `alert_channel` | string | false | The name of the channel to receive notifications for alert events. | -| `alert_events` | boolean | false | Enable notifications for alert events. | -| `commit_events` | boolean | false | Enable notifications for commit events. | -| `confidential_issue_channel` | string | false | The name of the channel to receive notifications for confidential issue events. | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | -| `confidential_note_channel` | string | false | The name of the channel to receive notifications for confidential note events. | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events. | -| `deployment_channel` | string | false | The name of the channel to receive notifications for deployment events. | -| `deployment_events` | boolean | false | Enable notifications for deployment events. | -| `incident_channel` | string | false | The name of the channel to receive notifications for incident events. | -| `incidents_events` | boolean | false | Enable notifications for incident events. | -| `issue_channel` | string | false | The name of the channel to receive notifications for issue events. | +| `push_events` | boolean | false | Enable notifications for push events. | | `issues_events` | boolean | false | Enable notifications for issue events. | -| `job_events` | boolean | false | Enable notifications for job events. | -| `merge_request_channel` | string | false | The name of the channel to receive notifications for merge request events. | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | | `merge_requests_events` | boolean | false | Enable notifications for merge request events. | -| `note_channel` | string | false | The name of the channel to receive notifications for note events. | +| `tag_push_events` | boolean | false | Enable notifications for tag push events. | | `note_events` | boolean | false | Enable notifications for note events. | -| `pipeline_channel` | string | false | The name of the channel to receive notifications for pipeline events. | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events. | | `pipeline_events` | boolean | false | Enable notifications for pipeline events. | -| `push_channel` | string | false | The name of the channel to receive notifications for push events. | -| `push_events` | boolean | false | Enable notifications for push events. | -| `tag_push_channel` | string | false | The name of the channel to receive notifications for tag push events. | -| `tag_push_events` | boolean | false | Enable notifications for tag push events. | -| `wiki_page_channel` | string | false | The name of the channel to receive notifications for wiki page events. | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events. | -### Disable the GitLab for Slack app +### Disable Google Chat -Disable the GitLab for Slack app for a project. Integration settings are reset. +Disable the Google Chat integration for a project. Integration settings are reset. ```plaintext -DELETE /projects/:id/integrations/slack +DELETE /projects/:id/integrations/hangouts-chat ``` -### Get the GitLab for Slack app settings +### Get Google Chat settings -Get the GitLab for Slack app settings for a project. +Get the Google Chat integration settings for a project. ```plaintext -GET /projects/:id/integrations/slack +GET /projects/:id/integrations/hangouts-chat ``` -## Google Chat +## Google Play -### Set up Google Chat +### Set up Google Play -Set up the Google Chat integration for a project. +Set up the Google Play integration for a project. ```plaintext -PUT /projects/:id/integrations/hangouts-chat +PUT /projects/:id/integrations/google-play ``` Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Hangouts Chat webhook (for example, `https://chat.googleapis.com/v1/spaces...`). | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines. | -| `notify_only_default_branch` | boolean | false | **Deprecated:** This parameter has been replaced with `branches_to_be_notified`. | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are `all`, `default`, `protected`, and `default_and_protected`. The default value is `default`. | -| `push_events` | boolean | false | Enable notifications for push events. | -| `issues_events` | boolean | false | Enable notifications for issue events. | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events. | -| `tag_push_events` | boolean | false | Enable notifications for tag push events. | -| `note_events` | boolean | false | Enable notifications for note events. | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events. | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events. | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events. | +| `package_name` | string | true | Package name of the app in Google Play. | +| `service_account_key` | string | true | Google Play service account key. | +| `service_account_key_file_name` | string | true | File name of the Google Play service account key. | +| `google_play_protected_refs` | boolean | false | Set variables on protected branches and tags only. | -### Disable Google Chat +### Disable Google Play -Disable the Google Chat integration for a project. Integration settings are reset. +Disable the Google Play integration for a project. Integration settings are reset. ```plaintext -DELETE /projects/:id/integrations/hangouts-chat +DELETE /projects/:id/integrations/google-play ``` -### Get Google Chat settings +### Get Google Play settings -Get the Google Chat integration settings for a project. +Get the Google Play integration settings for a project. ```plaintext -GET /projects/:id/integrations/hangouts-chat +GET /projects/:id/integrations/google-play +``` + +## Harbor + +### Set up Harbor + +Set up the Harbor integration for a project. + +```plaintext +PUT /projects/:id/integrations/harbor +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `url` | string | true | The base URL to the Harbor instance linked to the GitLab project. For example, `https://demo.goharbor.io`. | +| `project_name` | string | true | The name of the project in the Harbor instance. For example, `testproject`. | +| `username` | string | true | The username created in the Harbor interface. | +| `password` | string | true | The password of the user. | + +### Disable Harbor + +Disable the Harbor integration for a project. Integration settings are reset. + +```plaintext +DELETE /projects/:id/integrations/harbor +``` + +### Get Harbor settings + +Get the Harbor integration settings for a project. + +```plaintext +GET /projects/:id/integrations/harbor ``` ## irker (IRC gateway) @@ -977,12 +1018,14 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Mattermost webhook (for example, `http://mattermost_host/hooks/...`). | -| `username` | string | false | username. | -| `channel` | string | false | Default channel to use if others are not configured. | +| `webhook` | string | true | Mattermost notifications webhook (for example, `http://mattermost.example.com/hooks/...`). | +| `username` | string | false | Mattermost notifications username. | +| `channel` | string | false | Default channel to use if no other channel is configured. | | `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines. | | `notify_only_default_branch` | boolean | false | **Deprecated:** This parameter has been replaced with `branches_to_be_notified`. | | `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are `all`, `default`, `protected`, and `default_and_protected`. The default value is `default`. | +| `labels_to_be_notified` | string | false | Labels to send notifications for. Leave blank to receive notifications for all events. | +| `labels_to_be_notified_behavior` | string | false | Labels to be notified for. Valid options are `match_any` and `match_all`. The default value is `match_any`. | | `push_events` | boolean | false | Enable notifications for push events. | | `issues_events` | boolean | false | Enable notifications for issue events. | | `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | @@ -1329,9 +1372,9 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New issue URL. | -| `project_url` | string | true | Project URL. | -| `issues_url` | string | true | Issue URL. | +| `new_issue_url` | string | true | URL of the new issue. | +| `project_url` | string | true | URL of the project. | +| `issues_url` | string | true | URL of the issue. | ### Disable Redmine @@ -1349,6 +1392,71 @@ Get the Redmine integration settings for a project. GET /projects/:id/integrations/redmine ``` +## Slack notifications + +### Set up Slack notifications + +Set up Slack notifications for a project. + +```plaintext +PUT /projects/:id/integrations/slack +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | Slack notifications webhook (for example, `https://hooks.slack.com/services/...`). | +| `username` | string | false | Slack notifications username. | +| `channel` | string | false | Default channel to use if no other channel is configured. | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines. | +| `notify_only_default_branch` | boolean | false | **Deprecated:** This parameter has been replaced with `branches_to_be_notified`. | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are `all`, `default`, `protected`, and `default_and_protected`. The default value is `default`. | +| `labels_to_be_notified` | string | false | Labels to send notifications for. Leave blank to receive notifications for all events. | +| `labels_to_be_notified_behavior` | string | false | Labels to be notified for. Valid options are `match_any` and `match_all`. The default value is `match_any`. | +| `alert_channel` | string | false | The name of the channel to receive notifications for alert events. | +| `alert_events` | boolean | false | Enable notifications for alert events. | +| `commit_events` | boolean | false | Enable notifications for commit events. | +| `confidential_issue_channel` | string | false | The name of the channel to receive notifications for confidential issue events. | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events. | +| `confidential_note_channel` | string | false | The name of the channel to receive notifications for confidential note events. | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events. | +| `deployment_channel` | string | false | The name of the channel to receive notifications for deployment events. | +| `deployment_events` | boolean | false | Enable notifications for deployment events. | +| `incident_channel` | string | false | The name of the channel to receive notifications for incident events. | +| `incidents_events` | boolean | false | Enable notifications for incident events. | +| `issue_channel` | string | false | The name of the channel to receive notifications for issue events. | +| `issues_events` | boolean | false | Enable notifications for issue events. | +| `job_events` | boolean | false | Enable notifications for job events. | +| `merge_request_channel` | string | false | The name of the channel to receive notifications for merge request events. | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events. | +| `note_channel` | string | false | The name of the channel to receive notifications for note events. | +| `note_events` | boolean | false | Enable notifications for note events. | +| `pipeline_channel` | string | false | The name of the channel to receive notifications for pipeline events. | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events. | +| `push_channel` | string | false | The name of the channel to receive notifications for push events. | +| `push_events` | boolean | false | Enable notifications for push events. | +| `tag_push_channel` | string | false | The name of the channel to receive notifications for tag push events. | +| `tag_push_events` | boolean | false | Enable notifications for tag push events. | +| `wiki_page_channel` | string | false | The name of the channel to receive notifications for wiki page events. | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events. | + +### Disable Slack notifications + +Disable Slack notifications for a project. Integration settings are reset. + +```plaintext +DELETE /projects/:id/integrations/slack +``` + +### Get Slack notifications settings + +Get the Slack notifications settings for a project. + +```plaintext +GET /projects/:id/integrations/slack +``` + ## Slack slash commands ### Set up Slack slash commands @@ -1423,7 +1531,7 @@ Parameters: | Parameter | Type | Required | Description | |-------------------------|--------|----------|-------------------------------| | `url` | string | yes | URL of the Squash TM webhook. | -| `token` | string | no | Optional token. | +| `token` | string | no | Secret token. | ### Disable Squash TM @@ -1585,8 +1693,8 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `issues_url` | string | true | Issue URL. | -| `project_url` | string | true | Project URL. | +| `issues_url` | string | true | URL of the issue. | +| `project_url` | string | true | URL of the project. | ### Disable YouTrack diff --git a/doc/api/issues.md b/doc/api/issues.md index b2c6f0cb739..3c4ad7b6685 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -2020,7 +2020,8 @@ The `assignee` column is deprecated. We now show it as a single-sized array `ass Promotes an issue to an epic by adding a comment with the `/promote` [quick action](../user/project/quick_actions.md). -For more information about promoting issues to epics, see [Manage epics](../user/group/epics/manage_epics.md#promote-an-issue-to-an-epic). +For more information about promoting issues to epics, see +[Promote an issue to an epic](../user/project/issues/managing_issues.md#promote-an-issue-to-an-epic). ```plaintext POST /projects/:id/issues/:issue_iid/notes diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index 31a72c44d02..c7f0dfc28a1 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -24,11 +24,11 @@ as the request might redirect through a CDN. GET /projects/:id/jobs/:job_id/artifacts ``` -| Attribute | Type | Required | Description | -|---------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `job_id` | integer | Yes | ID of a job. | -| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| Attribute | Type | Required | Description | +|-------------------------------|----------------|----------|-------------| +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `job_id` | integer | Yes | ID of a job. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request using the `PRIVATE-TOKEN` header: @@ -65,10 +65,10 @@ Use either: Possible response status codes: -| Status | Description | -|-----------|---------------------------------| -| 200 | Serves the artifacts file. | -| 404 | Build not found or no artifacts.| +| Status | Description | +|--------|-------------| +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts. | ## Download the artifacts archive @@ -93,12 +93,12 @@ GET /projects/:id/jobs/artifacts/:ref_name/download?job=name Parameters -| Attribute | Type | Required | Description | -|---------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `ref_name` | string | Yes | Branch or tag name in repository. HEAD or SHA references are not supported. | -| `job` | string | Yes | The name of the job. | -| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| Attribute | Type | Required | Description | +|-------------------------------|----------------|----------|-------------| +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `job` | string | Yes | The name of the job. | +| `ref_name` | string | Yes | Branch or tag name in repository. HEAD or SHA references are not supported. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request using the `PRIVATE-TOKEN` header: @@ -136,10 +136,10 @@ Use either: Possible response status codes: -| Status | Description | -|-----------|---------------------------------| -| 200 | Serves the artifacts file. | -| 404 | Build not found or no artifacts.| +| Status | Description | +|--------|-------------| +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts. | ## Download a single artifact file by job ID @@ -156,12 +156,12 @@ GET /projects/:id/jobs/:job_id/artifacts/*artifact_path Parameters -| Attribute | Type | Required | Description | -|---------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `job_id` | integer | Yes | The unique job identifier. | -| `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | -| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| Attribute | Type | Required | Description | +|-------------------------------|----------------|----------|-------------| +| `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `job_id` | integer | Yes | The unique job identifier. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request: @@ -174,11 +174,11 @@ in a CI/CD job by using a [CI/CD job token](../ci/jobs/ci_job_token.md). Possible response status codes: -| Status | Description | -|-----------|--------------------------------------| -| 200 | Sends a single artifact file | -| 400 | Invalid path provided | -| 404 | Build not found or no file/artifacts | +| Status | Description | +|--------|-------------| +| 200 | Sends a single artifact file | +| 400 | Invalid path provided | +| 404 | Build not found or no file/artifacts | ## Download a single artifact file from specific tag or branch @@ -202,13 +202,13 @@ GET /projects/:id/jobs/artifacts/:ref_name/raw/*artifact_path?job=name Parameters: -| Attribute | Type | Required | Description | -|---------------------------|----------------|----------|-------------| -| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `ref_name` | string | Yes | Branch or tag name in repository. `HEAD` or `SHA` references are not supported. | -| `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | -| `job` | string | Yes | The name of the job. | -| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | +| Attribute | Type | Required | Description | +|-------------------------------|----------------|----------|-------------| +| `artifact_path` | string | Yes | Path to a file inside the artifacts archive. | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `job` | string | Yes | The name of the job. | +| `ref_name` | string | Yes | Branch or tag name in repository. `HEAD` or `SHA` references are not supported. | +| `job_token` **(PREMIUM ALL)** | string | No | To be used with [triggers](../ci/jobs/ci_job_token.md#download-an-artifact-from-a-different-pipeline) for multi-project pipelines. It should be invoked only in a CI/CD job defined in the `.gitlab-ci.yml` file. The value is always `$CI_JOB_TOKEN`. The job associated with the `$CI_JOB_TOKEN` must be running when this token is used. | Example request: @@ -221,11 +221,11 @@ in a CI/CD job by using a [CI/CD job token](../ci/jobs/ci_job_token.md). Possible response status codes: -| Status | Description | -|-----------|--------------------------------------| -| 200 | Sends a single artifact file | -| 400 | Invalid path provided | -| 404 | Build not found or no file/artifacts | +| Status | Description | +|--------|-------------| +| 200 | Sends a single artifact file | +| 400 | Invalid path provided | +| 404 | Build not found or no file/artifacts | ## Keep artifacts diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 594add64b0a..886f209520d 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -46,6 +46,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.802Z", "started_at": "2015-12-24T17:54:27.722Z", @@ -115,6 +116,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.727Z", "started_at": "2015-12-24T17:54:24.729Z", @@ -187,8 +189,8 @@ GET /projects/:id/pipelines/:pipeline_id/jobs |-------------------|--------------------------------|----------|-------------| | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `pipeline_id` | integer | Yes | ID of a pipeline. Can also be obtained in CI jobs via the [predefined CI variable](../ci/variables/predefined_variables.md) `CI_PIPELINE_ID`. | -| `scope` | string **or** array of strings | No | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, `waiting_for_resource`, or `manual`. All jobs are returned if `scope` is not provided. | | `include_retried` | boolean | No | Include retried jobs in the response. Defaults to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/272627) in GitLab 13.9. | +| `scope` | string **or** array of strings | No | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, `waiting_for_resource`, or `manual`. All jobs are returned if `scope` is not provided. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines/6/jobs?scope[]=pending&scope[]=running" @@ -209,6 +211,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.727Z", "started_at": "2015-12-24T17:54:24.729Z", @@ -269,6 +272,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.802Z", "started_at": "2015-12-24T17:54:27.722Z", @@ -363,6 +367,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.802Z", "started_at": "2015-12-24T17:54:27.722Z", @@ -450,6 +455,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.880Z", "started_at": "2015-12-24T17:54:30.733Z", @@ -600,6 +606,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2015-12-24T15:51:21.880Z", "started_at": "2015-12-24T17:54:30.733Z", @@ -669,10 +676,10 @@ curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.ex Possible response status codes: -| Status | Description | -|-----------|-------------------------------| -| 200 | Serves the log file | -| 404 | Job not found or no log file | +| Status | Description | +|--------|-------------| +| 200 | Serves the log file | +| 404 | Job not found or no log file | ## Cancel a job @@ -705,6 +712,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": "2016-01-11T10:14:09.526Z", @@ -759,6 +767,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": null, @@ -817,6 +826,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "download_url": null, "id": 1, @@ -899,6 +909,7 @@ Example response: "title": "Test the CI integration." }, "coverage": null, + "archived": false, "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": null, diff --git a/doc/api/member_roles.md b/doc/api/member_roles.md index 24ac7099004..2bfbc29081f 100644 --- a/doc/api/member_roles.md +++ b/doc/api/member_roles.md @@ -17,6 +17,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Admin group members introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131914) in GitLab 16.5 [with a flag](../administration/feature_flags.md) named `admin_group_member`. Disabled by default. The feature flag has been removed in GitLab 16.6. > - [Manage project access tokens introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132342) in GitLab 16.5 in [with a flag](../administration/feature_flags.md) named `manage_project_access_tokens`. Disabled by default. > - [Archive project introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134998) in GitLab 16.7. +> - [Delete project introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139696) in GitLab 16.8. +> - [Manage group access tokens introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140115) in GitLab 16.8. +> - [Admin terraform state introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140759) in GitLab 16.8. FLAG: On self-managed GitLab, by default these features are not available. To make them available, an administrator can [enable the feature flags](../administration/feature_flags.md) named `admin_group_member` and `manage_project_access_tokens`. @@ -44,6 +47,7 @@ If successful, returns [`200`](rest/index.md#status-codes) and the following res | `[].group_id` | integer | The ID of the group that the member role belongs to. | | `[].base_access_level` | integer | Base access level for member role. Valid values are 10 (Guest), 20 (Reporter), 30 (Developer), 40 (Maintainer), or 50 (Owner).| | `[].admin_merge_request` | boolean | Permission to admin project merge requests and enables the ability to `download_code`. | +| `[].admin_terraform_state` | boolean | Permission to admin project terraform state. | | `[].admin_vulnerability` | boolean | Permission to admin project vulnerabilities. | | `[].read_code` | boolean | Permission to read project code. | | `[].read_dependency` | boolean | Permission to read project dependencies. | @@ -51,6 +55,8 @@ If successful, returns [`200`](rest/index.md#status-codes) and the following res | `[].admin_group_member` | boolean | Permission to admin members of a group. | | `[].manage_project_access_tokens` | boolean | Permission to manage project access tokens. | | `[].archive_project` | boolean | Permission to archive projects. | +| `[].remove_project` | boolean | Permission to delete projects. | +| `[].manage_group_access_tokens` | boolean | Permission to manage group access tokens. | Example request: @@ -69,12 +75,15 @@ Example response: "group_id": 84, "base_access_level": 10, "admin_merge_request": false, + "admin_terraform_state": false, "admin_vulnerability": false, "read_code": true, "read_dependency": false, "read_vulnerability": false, + "manage_group_access_tokens": false, "manage_project_access_tokens": false, - "archive_project": false + "archive_project": false, + "remove_project": false }, { "id": 3, @@ -82,13 +91,16 @@ Example response: "description: "Custom guest that read and admin security entities", "group_id": 84, "base_access_level": 10, - "admin_merge_request": false, "admin_vulnerability": true, + "admin_merge_request": false, + "admin_terraform_state": false, "read_code": false, "read_dependency": true, "read_vulnerability": true, + "manage_group_access_tokens": false, "manage_project_access_tokens": false, - "archive_project": false + "archive_project": false, + "remove_project": false } ] ``` @@ -112,6 +124,7 @@ To add a member role to a group, the group must be at root-level (have no parent | `description` | string | no | The description of the member role. | | `base_access_level` | integer | yes | Base access level for configured role. Valid values are 10 (Guest), 20 (Reporter), 30 (Developer), 40 (Maintainer), or 50 (Owner).| | `admin_merge_request` | boolean | no | Permission to admin project merge requests. | +| `admin_terraform_state` | boolean | no | Permission to admin project terraform state. | | `admin_vulnerability` | boolean | no | Permission to admin project vulnerabilities. | | `read_code` | boolean | no | Permission to read project code. | | `read_dependency` | boolean | no | Permission to read project dependencies. | @@ -127,6 +140,7 @@ If successful, returns [`201`](rest/index.md#status-codes) and the following att | `group_id` | integer | The ID of the group that the member role belongs to. | | `base_access_level` | integer | Base access level for member role. | | `admin_merge_request` | boolean | Permission to admin project merge requests. | +| `admin_terraform_state` | boolean | Permission to admin project terraform state. | | `admin_vulnerability` | boolean | Permission to admin project vulnerabilities. | | `read_code` | boolean | Permission to read project code. | | `read_dependency` | boolean | Permission to read project dependencies. | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index a852c7c0b96..1e26a23272d 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1107,6 +1107,8 @@ Example response: ## List merge request diffs +> `generated_file` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141576) in GitLab 16.9 [with a flag](../administration/feature_flags.md) named `collapse_generated_diff_files`. Disabled by default. + List diffs of the files changed in a merge request. ```plaintext @@ -1136,6 +1138,7 @@ following response attributes: | `new_file` | boolean | Indicates if the file has just been added. | | `renamed_file` | boolean | Indicates if the file has been renamed. | | `deleted_file` | boolean | Indicates if the file has been removed. | +| `generated_file` | boolean | Indicates if the file is marked as generated. | Example request: @@ -1156,7 +1159,8 @@ Example response: "diff": "@@ -1 +1 @@\ -Title\ +README", "new_file": false, "renamed_file": false, - "deleted_file": false + "deleted_file": false, + "generated_file": false }, { "old_path": "VERSION", @@ -1166,7 +1170,8 @@ Example response: "diff": "@@\ -1.9.7\ +1.9.8", "new_file": false, "renamed_file": false, - "deleted_file": false + "deleted_file": false, + "generated_file": false } ] ``` @@ -2014,7 +2019,9 @@ This API returns specific HTTP status codes: | HTTP Status | Message | Reason | |-------------|--------------------------------------------|--------| | `202` | *(no message)* | Successfully enqueued. | -| `403` | <ul><li>`Source branch does not exist`</li><li>`Cannot push to source branch`</li><li>`Source branch is protected from force push`</li></ul> | You don't have permission to push to the merge request's source branch. | +| `403` | `Cannot push to source branch` | You don't have permission to push to the merge request's source branch. | +| `403` | `Source branch does not exist` | You don't have permission to push to the merge request's source branch. | +| `403` | `Source branch is protected from force push` | You don't have permission to push to the merge request's source branch. | | `409` | `Failed to enqueue the rebase operation` | A long-lived transaction might have blocked your request. | If the request is enqueued successfully, the response contains: diff --git a/doc/api/merge_trains.md b/doc/api/merge_trains.md index dff65e5786b..7fdc145f67d 100644 --- a/doc/api/merge_trains.md +++ b/doc/api/merge_trains.md @@ -232,9 +232,9 @@ Supported attributes: |--------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `merge_request_iid` | integer | Yes | The internal ID of the merge request. | -| `when_pipeline_succeeds` | boolean | No | If true, the merge request is added to the merge train when the pipeline succeeds. When false or unspecified, the merge request is added directly to the merge train. | | `sha` | string | No | If present, the SHA must match the `HEAD` of the source branch, otherwise the merge fails. | | `squash` | boolean | No | If true, the commits are squashed into a single commit on merge. | +| `when_pipeline_succeeds` | boolean | No | If true, the merge request is added to the merge train when the pipeline succeeds. When false or unspecified, the merge request is added directly to the merge train. | Example request: diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 4e285cde975..6d5e15934e9 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -18,6 +18,8 @@ You might also want to view documentation for: ## List namespaces +> `top_level_only` [introduced](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/7600) in GitLab 16.8. + Get a list of the namespaces of the authenticated user. If the user is an administrator, a list of all namespaces in the GitLab instance is shown. @@ -25,12 +27,14 @@ administrator, a list of all namespaces in the GitLab instance is shown. GET /namespaces GET /namespaces?search=foobar GET /namespaces?owned_only=true +GET /namespaces?top_level_only=true ``` -| Attribute | Type | Required | Description | -| ------------ | ------- | -------- | ----------- | -| `search` | string | no | Returns a list of namespaces the user is authorized to view based on the search criteria | -| `owned_only` | boolean | no | In GitLab 14.2 and later, returns a list of owned namespaces only | +| Attribute | Type | Required | Description | +| ---------------- | ------- | -------- | ----------- | +| `search` | string | no | Returns a list of namespaces the user is authorized to view based on the search criteria | +| `owned_only` | boolean | no | In GitLab 14.2 and later, returns a list of owned namespaces only | +| `top_level_only` | boolean | no | In GitLab 16.8 and later, returns a list of top level namespaces only | Example request: diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 2f8e030374f..71d387f7dd5 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -1,6 +1,7 @@ --- stage: Govern group: Authentication +description: Third-party authorization to GitLab. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/packages/terraform-modules.md b/doc/api/packages/terraform-modules.md index 3645fc03e9a..079b7652e67 100644 --- a/doc/api/packages/terraform-modules.md +++ b/doc/api/packages/terraform-modules.md @@ -208,6 +208,8 @@ X-Terraform-Get: /api/v4/packages/terraform/modules/v1/group/hello-world/local/1 ## Download module +### From a namespace + ```plaintext GET packages/terraform/modules/v1/:module_namespace/:module_name/:module_system/:module_version/file ``` @@ -229,6 +231,29 @@ To write the output to file: curl --header "Authorization: Bearer <personal_access_token>" "https://gitlab.example.com/api/v4/packages/terraform/modules/v1/group/hello-world/local/1.0.0/file" --output hello-world-local.tgz ``` +### From a project + +```plaintext +GET /projects/:id/packages/terraform/modules/:module_name/:module_system/:module_version +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or URL-encoded path of the project. | +| `module_name` | string | yes | The module name. | +| `module_system` | string | yes | The name of the module system or [provider](https://www.terraform.io/registry/providers). | +| `module_version` | string | no | Specific module version to download. If omitted, the latest version is downloaded. | + +```shell +curl --user "<username>:<personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/terraform/modules/hello-world/local/1.0.0" +``` + +To write the output to file: + +```shell +curl --user "<username>:<personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/terraform/modules/hello-world/local/1.0.0" --output hello-world-local.tgz +``` + ## Upload module ```plaintext diff --git a/doc/api/pages.md b/doc/api/pages.md index 69c96f2aeb9..5467b5112df 100644 --- a/doc/api/pages.md +++ b/doc/api/pages.md @@ -29,3 +29,70 @@ DELETE /projects/:id/pages ```shell curl --request 'DELETE' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/2/pages" ``` + +## Get pages settings for a project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/436932) in GitLab 16.8. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +List Pages settings for the project. + +```plaintext +GET /projects/:id/pages +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | + +If successful, returns [`200`](rest/index.md#status-codes) and the following +response attributes: + +| Attribute | Type | Description | +| ----------------------------------------- | ---------- | ----------------------- | +| `url` | string | URL to access this project pages. | +| `is_unique_domain_enabled` | boolean | If [unique domain](../user/project/pages/introduction.md) is enabled. | +| `force_https` | boolean | `true` if the project is set to force HTTPS. | +| `deployments[]` | array | List of current active deployments. | + +| `deployments[]` attribute | Type | Description | +| ----------------------------------------- | ---------- | ----------------------- | +| `created_at` | date | Date deployment was created. | +| `url` | string | URL for this deployment. | +| `path_prefix` | string | Path prefix of this deployment when using [multiple deployments](../user/project/pages/index.md#create-multiple-deployments). | +| `root_directory` | string | Root directory. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/2/pages" +``` + +Example response: + +```json +{ + "url": "http://html-root-4160ce5f0e9a6c90ccb02755b7fc80f5a2a09ffbb1976cf80b653.pages.gdk.test:3010", + "is_unique_domain_enabled": true, + "force_https": false, + "deployments": [ + { + "created_at": "2024-01-05T18:58:14.916Z", + "url": "http://html-root-4160ce5f0e9a6c90ccb02755b7fc80f5a2a09ffbb1976cf80b653.pages.gdk.test:3010/", + "path_prefix": "", + "root_directory": null + }, + { + "created_at": "2024-01-05T18:58:46.042Z", + "url": "http://html-root-4160ce5f0e9a6c90ccb02755b7fc80f5a2a09ffbb1976cf80b653.pages.gdk.test:3010/mr3", + "path_prefix": "mr3", + "root_directory": null + } + ] +} +``` diff --git a/doc/api/pipeline_schedules.md b/doc/api/pipeline_schedules.md index 9992231dc7f..8a2d38a4266 100644 --- a/doc/api/pipeline_schedules.md +++ b/doc/api/pipeline_schedules.md @@ -166,12 +166,12 @@ POST /projects/:id/pipeline_schedules | Attribute | Type | Required | Description | |-----------------|----------------|----------|-------------| -| `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `cron` | string | Yes | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | | `description` | string | Yes | The description of the pipeline schedule. | +| `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `ref` | string | Yes | The branch or tag name that is triggered. | -| `cron` | string | Yes | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | -| `cron_timezone` | string | No | The time zone supported by `ActiveSupport::TimeZone`, for example: `Pacific Time (US & Canada)` (default: `UTC`). | | `active` | boolean | No | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated (default: `true`). | +| `cron_timezone` | string | No | The time zone supported by `ActiveSupport::TimeZone`, for example: `Pacific Time (US & Canada)` (default: `UTC`). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -214,11 +214,11 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id |------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID. | +| `active` | boolean | No | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated. | +| `cron_timezone` | string | No | The time zone supported by `ActiveSupport::TimeZone` (for example `Pacific Time (US & Canada)`), or `TZInfo::Timezone` (for example `America/Los_Angeles`). | +| `cron` | string | No | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | | `description` | string | No | The description of the pipeline schedule. | | `ref` | string | No | The branch or tag name that is triggered. | -| `cron` | string | No | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | -| `cron_timezone` | string | No | The time zone supported by `ActiveSupport::TimeZone` (for example `Pacific Time (US & Canada)`), or `TZInfo::Timezone` (for example `America/Los_Angeles`). | -| `active` | boolean | No | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -384,8 +384,8 @@ POST /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables | Attribute | Type | Required | Description | |------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | -| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | | `key` | string | Yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | | `value` | string | Yes | The `value` of a variable | | `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | @@ -413,8 +413,8 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables/:key | Attribute | Type | Required | Description | |------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | -| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | | `key` | string | Yes | The `key` of a variable | +| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | | `value` | string | Yes | The `value` of a variable | | `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | @@ -443,8 +443,8 @@ DELETE /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables/:key | Attribute | Type | Required | Description | |------------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | -| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | | `key` | string | Yes | The `key` of a variable | +| `pipeline_schedule_id` | integer | Yes | The pipeline schedule ID | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables/NEW_VARIABLE" diff --git a/doc/api/pipeline_triggers.md b/doc/api/pipeline_triggers.md index c17227a3d41..5ac59854f23 100644 --- a/doc/api/pipeline_triggers.md +++ b/doc/api/pipeline_triggers.md @@ -80,8 +80,8 @@ POST /projects/:id/triggers | Attribute | Type | Required | Description | |---------------|----------------|----------|-------------| -| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | | `description` | string | Yes | The trigger name | +| `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index d8b4b4cf41f..1b75a3f74b5 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -36,18 +36,18 @@ GET /projects/:id/pipelines | Attribute | Type | Required | Description | |------------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | -| `scope` | string | No | The scope of pipelines, one of: `running`, `pending`, `finished`, `branches`, `tags` | -| `status` | string | No | The status of pipelines, one of: `created`, `waiting_for_resource`, `preparing`, `pending`, `running`, `success`, `failed`, `canceled`, `skipped`, `manual`, `scheduled` | -| `source` | string | No | In [GitLab 14.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/325439), how the pipeline was triggered, one of: `api`, `chat`, `external`, `external_pull_request_event`, `merge_request_event`, `ondemand_dast_scan`, `ondemand_dast_validation`, `parent_pipeline`, `pipeline`, `push`, `schedule`, `security_orchestration_policy`, `trigger`, `web`, or `webide`. | +| `name` | string | No | Return pipelines with the specified name. Introduced in GitLab 15.11, not available by default. | +| `order_by` | string | No | Order pipelines by `id`, `status`, `ref`, `updated_at` or `user_id` (default: `id`) | | `ref` | string | No | The ref of pipelines | +| `scope` | string | No | The scope of pipelines, one of: `running`, `pending`, `finished`, `branches`, `tags` | | `sha` | string | No | The SHA of pipelines | -| `yaml_errors` | boolean | No | Returns pipelines with invalid configurations | -| `username` | string | No | The username of the user who triggered pipelines | +| `sort` | string | No | Sort pipelines in `asc` or `desc` order (default: `desc`) | +| `source` | string | No | In [GitLab 14.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/325439), how the pipeline was triggered, one of: `api`, `chat`, `external`, `external_pull_request_event`, `merge_request_event`, `ondemand_dast_scan`, `ondemand_dast_validation`, `parent_pipeline`, `pipeline`, `push`, `schedule`, `security_orchestration_policy`, `trigger`, `web`, or `webide`. | +| `status` | string | No | The status of pipelines, one of: `created`, `waiting_for_resource`, `preparing`, `pending`, `running`, `success`, `failed`, `canceled`, `skipped`, `manual`, `scheduled` | | `updated_after` | datetime | No | Return pipelines updated after the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | | `updated_before` | datetime | No | Return pipelines updated before the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | -| `name` | string | No | Return pipelines with the specified name. Introduced in GitLab 15.11, not available by default. | -| `order_by` | string | No | Order pipelines by `id`, `status`, `ref`, `updated_at` or `user_id` (default: `id`) | -| `sort` | string | No | Sort pipelines in `asc` or `desc` order (default: `desc`) | +| `username` | string | No | The username of the user who triggered pipelines | +| `yaml_errors` | boolean | No | Returns pipelines with invalid configurations | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines" @@ -530,8 +530,8 @@ PUT /projects/:id/pipelines/:pipeline_id/metadata | Attribute | Type | Required | Description | |---------------|----------------|----------|-------------| | `id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | -| `pipeline_id` | integer | Yes | The ID of a pipeline | | `name` | string | Yes | The new name of the pipeline | +| `pipeline_id` | integer | Yes | The ID of a pipeline | Sample request: diff --git a/doc/api/project_access_tokens.md b/doc/api/project_access_tokens.md index f8e7c0f3633..2f7be015e75 100644 --- a/doc/api/project_access_tokens.md +++ b/doc/api/project_access_tokens.md @@ -141,9 +141,6 @@ Rotate a project access token. Revokes the previous token and creates a new toke In GitLab 16.6 and later, you can use the `expires_at` parameter to set a different expiry date. This non-default expiry date can be up to a maximum of one year from the rotation date. -WARNING: -When you rotate a project access token, the new token retains the expiry date of the old token. For more information, see [issue 423362](https://gitlab.com/gitlab-org/gitlab/-/issues/423362). - ```plaintext POST /projects/:id/access_tokens/:token_id/rotate ``` diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 5031279d5bb..e6d3050d59c 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -94,12 +94,12 @@ POST /projects/:id/variables | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | | `key` | string | Yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | | `value` | string | Yes | The `value` of a variable | -| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | No | Whether the variable is protected. Default: `false` | +| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | +| `environment_scope` | string | No | The `environment_scope` of the variable. Default: `*` | | `masked` | boolean | No | Whether the variable is masked. Default: `false` | +| `protected` | boolean | No | Whether the variable is protected. Default: `false` | | `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | -| `environment_scope` | string | No | The `environment_scope` of the variable. Default: `*` | -| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | +| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -133,13 +133,13 @@ PUT /projects/:id/variables/:key | `id` | integer/string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | | `key` | string | Yes | The `key` of a variable | | `value` | string | Yes | The `value` of a variable | -| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | No | Whether the variable is protected | -| `masked` | boolean | No | Whether the variable is masked | -| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | | `environment_scope` | string | No | The `environment_scope` of the variable | | `filter` | hash | No | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | -| `description` | string | No | The description of the variable. Default: `null`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409641) in GitLab 16.2. | +| `masked` | boolean | No | Whether the variable is masked | +| `protected` | boolean | No | Whether the variable is protected | +| `raw` | boolean | No | Whether the variable is treated as a raw string. Default: `false`. When `true`, variables in the value are not [expanded](../ci/variables/index.md#prevent-cicd-variable-expansion). | +| `variable_type` | string | No | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ diff --git a/doc/api/project_relations_export.md b/doc/api/project_relations_export.md index 5fe3923edc8..a586c940b8e 100644 --- a/doc/api/project_relations_export.md +++ b/doc/api/project_relations_export.md @@ -16,7 +16,7 @@ top-level relation (for example, milestones, issues, and labels). The project relations export API is primarily used in -[group migration](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) can't +[group migration](../user/group/import/index.md) can't be used with the [project import and export API](project_import_export.md). diff --git a/doc/api/projects.md b/doc/api/projects.md index 3d3a11fc59e..257dd88a770 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -51,38 +51,38 @@ are returned. GET /projects ``` -| Attribute | Type | Required | Description | -|--------------------------------------------|----------|------------------------|-------------| -| `archived` | boolean | No | Limit by archived status. | -| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | -| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | -| `imported` | boolean | No | Limit results to projects which were imported from external systems by current user. | -| `include_hidden` **(PREMIUM ALL)** | boolean | No | Include hidden projects. _(administrators only)_ | -| `include_pending_delete` | boolean | No | Include projects pending deletion. _(administrators only)_ | -| `last_activity_after` | datetime | No | Limit results to projects with last activity after specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | -| `last_activity_before` | datetime | No | Limit results to projects with last activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | -| `membership` | boolean | No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | -| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | -| `repository_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the repository checksum calculation has failed. | -| `repository_storage` | string | No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | -| `search_namespaces` | boolean | No | Include ancestor namespaces when matching search criteria. Default is `false`. | -| `search` | string | No | Return list of projects matching the search criteria. | -| `simple` | boolean | No | Return only limited fields for each project. This operation is a no-op without authentication where only simple fields are returned. | -| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | No | Limit by projects starred by the current user. | -| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | -| `topic` | string | No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | -| `topic_id` | integer | No | Limit results to projects with the assigned topic given by the topic ID. | -| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | -| `wiki_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the wiki checksum calculation has failed. | -| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | -| `with_programming_language` | string | No | Limit by projects which use the given programming language. | -| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | -| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | +| Attribute | Type | Required | Description | +|------------------------------------------------|----------|----------|-------------| +| `archived` | boolean | No | Limit by archived status. | +| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | +| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | +| `imported` | boolean | No | Limit results to projects which were imported from external systems by current user. | +| `include_hidden` **(PREMIUM ALL)** | boolean | No | Include hidden projects. _(administrators only)_ | +| `include_pending_delete` | boolean | No | Include projects pending deletion. _(administrators only)_ | +| `last_activity_after` | datetime | No | Limit results to projects with last activity after specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | +| `last_activity_before` | datetime | No | Limit results to projects with last activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `repository_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the repository checksum calculation has failed. | +| `repository_storage` | string | No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | +| `search_namespaces` | boolean | No | Include ancestor namespaces when matching search criteria. Default is `false`. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. This operation is a no-op without authentication where only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `topic_id` | integer | No | Limit results to projects with the assigned topic given by the topic ID. | +| `topic` | string | No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. For this filter to work, you must also provide `updated_at` as the `order_by` attribute. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `wiki_checksum_failed` **(PREMIUM ALL)** | boolean | No | Limit projects where the wiki checksum calculation has failed. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `with_programming_language` | string | No | Limit by projects which use the given programming language. | This endpoint supports [keyset pagination](rest/index.md#keyset-based-pagination) for selected `order_by` options. @@ -326,28 +326,28 @@ for selected `order_by` options. GET /users/:user_id/projects ``` -| Attribute | Type | Required | Description | -|-------------------------------|---------|------------------------|-------------| -| `user_id` | string | Yes | The ID or username of the user. | -| `archived` | boolean | No | Limit by archived status. | -| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | -| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | -| `membership` | boolean | No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | -| `search` | string | No | Return list of projects matching the search criteria. | -| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | No | Limit by projects starred by the current user. | -| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | -| `with_programming_language` | string | No | Limit by projects which use the given programming language. | -| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| Attribute | Type | Required | Description | +|-------------------------------|----------|----------|-------------| +| `user_id` | string | Yes | The ID or username of the user. | +| `archived` | boolean | No | Limit by archived status. | +| `id_after` | integer | No | Limit results to projects with IDs greater than the specified ID. | +| `id_before` | integer | No | Limit results to projects with IDs less than the specified ID. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | +| `with_programming_language` | string | No | Limit by projects which use the given programming language. | ```json [ @@ -602,12 +602,12 @@ Get a list of visible projects a given user has contributed to. GET /users/:user_id/contributed_projects ``` -| Attribute | Type | Required | Description | -|-------------------------------|---------|------------------------|-------------| -| `user_id` | string | Yes | The ID or username of the user. | -| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| Attribute | Type | Required | Description | +|------------|---------|----------|-------------| +| `user_id` | string | Yes | The ID or username of the user. | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/5/contributed_projects" @@ -847,25 +847,25 @@ authentication, only public projects are returned. GET /users/:user_id/starred_projects ``` -| Attribute | Type | Required | Description | -|-------------------------------|---------|------------------------|-------------| -| `user_id` | string | Yes | The ID or username of the user. | -| `archived` | boolean | No | Limit by archived status. | -| `membership` | boolean | No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | -| `search` | string | No | Return list of projects matching the search criteria. | -| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | No | Limit by projects starred by the current user. | -| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | -| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | -| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| Attribute | Type | Required | Description | +|-------------------------------|----------|----------|-------------| +| `user_id` | string | Yes | The ID or username of the user. | +| `archived` | boolean | No | Limit by archived status. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/5/starred_projects" @@ -1105,12 +1105,12 @@ the project is publicly accessible. GET /projects/:id ``` -| Attribute | Type | Required | Description | -|--------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `license` | boolean | No | Include project license data. | -| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | -| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | +| Attribute | Type | Required | Description | +|--------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `license` | boolean | No | Include project license data. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | ```json { @@ -1373,11 +1373,11 @@ Get the users list of a project. GET /projects/:id/users ``` -| Attribute | Type | Required | Description | -|--------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | No | Search for specific users. | -| `skip_users` | integer array | No | Filter out users with the specified IDs. | +| Attribute | Type | Required | Description | +|--------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific users. | +| `skip_users` | integer array | No | Filter out users with the specified IDs. | ```json [ @@ -1408,14 +1408,14 @@ Get a list of ancestor groups for this project. GET /projects/:id/groups ``` -| Attribute | Type | Required | Description | -|-----------------------------|-------------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | No | Search for specific groups. | -| `shared_min_access_level` | integer | No | Limit to shared groups with at least this [role (`access_level`)](members.md#roles). | -| `shared_visible_only` | boolean | No | Limit to shared groups user has access to. | -| `skip_groups` | array of integers | No | Skip the group IDs passed. | -| `with_shared` | boolean | No | Include projects shared with this group. Default is `false`. | +| Attribute | Type | Required | Description | +|---------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific groups. | +| `shared_min_access_level` | integer | No | Limit to shared groups with at least this [role (`access_level`)](members.md#roles). | +| `shared_visible_only` | boolean | No | Limit to shared groups user has access to. | +| `skip_groups` | array of integers | No | Skip the group IDs passed. | +| `with_shared` | boolean | No | Include projects shared with this group. Default is `false`. | ```json [ @@ -1446,10 +1446,10 @@ Get a list of groups that can be shared with a project GET /projects/:id/share_locations ``` -| Attribute | Type | Required | Description | -|-----------------------------|-------------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | No | Search for specific groups. | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | Search for specific groups. | ```json [ @@ -1501,81 +1501,81 @@ curl --request POST --header "PRIVATE-TOKEN: <your-token>" \ --url "https://gitlab.example.com/api/v4/projects/" ``` -| Attribute | Type | Required | Description | -|-------------------------------------------------------------|---------|------------------------|-------------| -| `name` | string | Yes (if `path` isn't provided) | The name of the new project. Equals path if not provided. | -| `path` | string | Yes (if `name` isn't provided) | Repository name for new project. Generated based on name if not provided (generated as lowercase with dashes). Starting with GitLab 14.9, path must not start or end with a special character and must not contain consecutive special characters. | -| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | -| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | -| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | -| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | No | Image file for avatar of the project. | -| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | No | The path to CI configuration file. | -| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). See the [container registry](../user/packages/container_registry/reduce_container_registry_storage.md#use-the-cleanup-policy-api) documentation for more information on `cadence`, `keep_n` and `older_than` values. | -| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | -| `description` | string | No | Short project description. | -| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | No | Enable email notifications. | -| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | -| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | No | Enable group runners for this project. | -| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | -| `import_url` | string | No | URL to import repository from. When the URL value isn't empty, you must not set `initialize_with_readme` to `true`. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | -| `initialize_with_readme` | boolean | No | Whether to create a Git repository with just a `README.md` file. Default is `false`. When this boolean is true, you must not pass `import_url` or other attributes of this endpoint which specify alternative contents for the repository. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | -| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `lfs_enabled` | boolean | No | Enable LFS. | -| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | -| `merge_pipelines_enabled` | boolean | No | Enable or disable merge pipelines. | -| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | -| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | -| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | -| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) in the project settings. | -| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | -| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | -| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | -| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `model_experiments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | No | Which storage shard the repository is on. _(administrator only)_ | -| `request_access_enabled` | boolean | No | Allow users to request member access. | -| `requirements_access_level` | string | No | One of `disabled`, `private` or `enabled` | -| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | -| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | -| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | -| `template_project_id` **(PREMIUM ALL)** | integer | No | When used with `use_custom_template`, project ID of a custom project template. Using a project ID is preferable to using `template_name` since `template_name` may be ambiguous. | -| `topics` | array | No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | -| `visibility` | string | No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| Attribute | Type | Required | Description | +|-------------------------------------------------------------------|---------|--------------------------------|-------------| +| `name` | string | Yes (if `path` isn't provided) | The name of the new project. Equals path if not provided. | +| `path` | string | Yes (if `name` isn't provided) | Repository name for new project. Generated based on name if not provided (generated as lowercase with dashes). Starting with GitLab 14.9, path must not start or end with a special character and must not contain consecutive special characters. | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). See the [container registry](../user/packages/container_registry/reduce_container_registry_storage.md#use-the-cleanup-policy-api) documentation for more information on `cadence`, `keep_n` and `older_than` values. | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead | +| `emails_enabled` | boolean | No | Enable email notifications. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | +| `import_url` | string | No | URL to import repository from. When the URL value isn't empty, you must not set `initialize_with_readme` to `true`. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `initialize_with_readme` | boolean | No | Whether to create a Git repository with just a `README.md` file. Default is `false`. When this boolean is true, you must not pass `import_url` or other attributes of this endpoint which specify alternative contents for the repository. Doing so might result in the [following error](https://gitlab.com/gitlab-org/gitlab/-/issues/360266): `not a git repository`. | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | No | Enable or disable merged results pipelines. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `model_experiments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful pipelines. This setting is named [**Pipelines must succeed**](../user/project/merge_requests/merge_when_pipeline_succeeds.md#require-a-successful-pipeline-for-merge) in the project settings. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrator only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | +| `template_project_id` **(PREMIUM ALL)** | integer | No | When used with `use_custom_template`, project ID of a custom project template. Using a project ID is preferable to using `template_name` since `template_name` may be ambiguous. | +| `topics` | array | No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Create project for user @@ -1592,83 +1592,83 @@ where `password` is a public access key with the `api` scope enabled. POST /projects/user/:user_id ``` -| Attribute | Type | Required | Description | -|-------------------------------------------------------------|---------|------------------------|-------------| -| `user_id` | integer | Yes | The user ID of the project owner. | -| `name` | string | Yes | The name of the new project. | -| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | -| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | -| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | -| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | No | Image file for avatar of the project. | -| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | No | The path to CI configuration file. | -| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | -| `description` | string | No | Short project description. | -| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | No | Enable email notifications. | -| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | -| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | -| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | No | Enable group runners for this project. | -| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | -| `import_url` | string | No | URL to import repository from. | -| `initialize_with_readme` | boolean | No | `false` by default. | -| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `lfs_enabled` | boolean | No | Enable LFS. | -| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | -| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | -| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | -| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | -| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | -| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | -| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | -| `path` | string | No | Custom repository name for new project. By default generated based on name. | -| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project-members. | -| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `model_experiments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | -| `request_access_enabled` | boolean | No | Allow users to request member access. | -| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | -| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | -| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | -| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | -| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | -| `suggestion_commit_message` | string | No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | -| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | -| `topics` | array | No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | -| `visibility` | string | No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| Attribute | Type | Required | Description | +|-------------------------------------------------------------------|---------|----------|-------------| +| `name` | string | Yes | The name of the new project. | +| `user_id` | integer | Yes | The user ID of the project owner. | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead | +| `emails_enabled` | boolean | No | Enable email notifications. | +| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `group_with_project_templates_id` **(PREMIUM ALL)** | integer | No | For group-level custom templates, specifies ID of group from which all the custom project templates are sourced. Leave empty for instance-level templates. Requires `use_custom_template` to be true. | +| `import_url` | string | No | URL to import repository from. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `initialize_with_readme` | boolean | No | `false` by default. | +| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `model_experiments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `namespace_id` | integer | No | Namespace for the new project (defaults to the current user's namespace). | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `path` | string | No | Custom repository name for new project. By default generated based on name. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project-members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `suggestion_commit_message` | string | No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `template_name` | string | No | When used without `use_custom_template`, name of a [built-in project template](../user/project/index.md#create-a-project-from-a-built-in-template). When used with `use_custom_template`, name of a custom project template. | +| `topics` | array | No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `use_custom_template` **(PREMIUM ALL)** | boolean | No | Use either custom [instance](../administration/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Edit project @@ -1696,96 +1696,98 @@ curl --request PUT --header "PRIVATE-TOKEN: <your-token>" \ Supported attributes: -| Attribute | Type | Required | Description | -|-------------------------------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | -| `allow_pipeline_trigger_approve_deployment` **(PREMIUM ALL)** | boolean | No | Set whether or not a pipeline triggerer is allowed to approve deployments. | -| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false.<br/><br/>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. The feature flag was enabled by default in GitLab 15.9. | -| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | -| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | -| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual`, or `timed_incremental`). | -| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | -| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | -| `avatar` | mixed | No | Image file for avatar of the project. | -| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | -| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | -| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `ci_config_path` | string | No | The path to CI configuration file. | -| `ci_default_git_depth` | integer | No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | -| `ci_forward_deployment_enabled` | boolean | No | Enable or disable [prevent outdated deployment jobs](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | -| `ci_forward_deployment_rollback_allowed` | boolean | No | Enable or disable [allow job retries for rollback deployments](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | -| `ci_allow_fork_pipelines_to_run_in_parent_project` | boolean | No | Enable or disable [running pipelines in the parent project for merge requests from forks](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325189) in GitLab 15.3.)_ | -| `ci_separated_caches` | boolean | No | Set whether or not caches should be [separated](../ci/caching/index.md#cache-key-names) by branch protection status. | -| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | -| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | -| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | -| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. | -| `description` | string | No | Short project description. | -| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead| -| `emails_enabled` | boolean | No | Enable email notifications. | -| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | -| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | -| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `group_runners_enabled` | boolean | No | Enable group runners for this project. | -| `import_url` | string | No | URL the repository was imported from. | -| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | -| `issues_template` **(PREMIUM ALL)** | string | No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | -| `keep_latest_artifact` | boolean | No | Disable or enable the ability to keep the latest artifact for this project. | -| `lfs_enabled` | boolean | No | Enable LFS. | -| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | -| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | -| `merge_pipelines_enabled` | boolean | No | Enable or disable merge pipelines. | -| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | -| `merge_requests_template` **(PREMIUM ALL)** | string | No | Default description for merge requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | -| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | -| `mirror_overwrites_diverged_branches` **(PREMIUM ALL)** | boolean | No | Pull mirror overwrites diverged branches. | -| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | -| `mirror_user_id` **(PREMIUM ALL)** | integer | No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | -| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | -| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | -| `name` | string | No | The name of the project. | -| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | -| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | -| `only_mirror_protected_branches` **(PREMIUM ALL)** | boolean | No | Only mirror protected branches. | -| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | -| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | -| `path` | string | No | Custom repository name for the project. By default generated based on name. | -| `prevent_merge_without_jira_issue` **(PREMIUM ALL)** | boolean | No | Set whether merge requests require an associated issue from Jira. -| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | -| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | -| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | -| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | -| `request_access_enabled` | boolean | No | Allow users to request member access. | -| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | -| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | -| `restrict_user_defined_variables` | boolean | No | Allow only users with the Maintainer role to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | -| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | -| `service_desk_enabled` | boolean | No | Enable or disable Service Desk feature. | -| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | -| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | -| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | -| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | -| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | -| `suggestion_commit_message` | string | No | The commit message used to apply merge request suggestions. | -| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | -| `topics` | array | No | The list of topics for the project. This replaces any existing topics that are already added to the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | -| `visibility` | string | No | See [project visibility level](#project-visibility-level). | -| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | -| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | +| Attribute | Type | Required | Description | +|-------------------------------------------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `allow_merge_on_skipped_pipeline` | boolean | No | Set whether or not merge requests can be merged with skipped jobs. | +| `allow_pipeline_trigger_approve_deployment` **(PREMIUM ALL)** | boolean | No | Set whether or not a pipeline triggerer is allowed to approve deployments. | +| `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE ALL)** | boolean | No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false.<br/><br/>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. The feature flag was enabled by default in GitLab 15.9. | +| `analytics_access_level` | string | No | One of `disabled`, `private` or `enabled` | +| `approvals_before_merge` **(PREMIUM ALL)** | integer | No | How many approvers should approve merge requests by default. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/353097) in GitLab 16.0. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | +| `auto_cancel_pending_pipelines` | string | No | Auto-cancel pending pipelines. This action toggles between an enabled state and a disabled state; it is not a boolean. | +| `auto_devops_deploy_strategy` | string | No | Auto Deploy strategy (`continuous`, `manual`, or `timed_incremental`). | +| `auto_devops_enabled` | boolean | No | Enable Auto DevOps for this project. | +| `autoclose_referenced_issues` | boolean | No | Set whether auto-closing referenced issues on default branch. | +| `avatar` | mixed | No | Image file for avatar of the project. | +| `build_git_strategy` | string | No | The Git strategy. Defaults to `fetch`. | +| `build_timeout` | integer | No | The maximum amount of time, in seconds, that a job can run. | +| `builds_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `ci_config_path` | string | No | The path to CI configuration file. | +| `ci_default_git_depth` | integer | No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | +| `ci_forward_deployment_enabled` | boolean | No | Enable or disable [prevent outdated deployment jobs](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | +| `ci_forward_deployment_rollback_allowed` | boolean | No | Enable or disable [allow job retries for rollback deployments](../ci/pipelines/settings.md#prevent-outdated-deployment-jobs). | +| `ci_allow_fork_pipelines_to_run_in_parent_project` | boolean | No | Enable or disable [running pipelines in the parent project for merge requests from forks](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325189) in GitLab 15.3.)_ | +| `ci_separated_caches` | boolean | No | Set whether or not caches should be [separated](../ci/caching/index.md#cache-key-names) by branch protection status. | +| `ci_restrict_pipeline_cancellation_role` **(PREMIUM ALL)** | string | No | Set the [role required to cancel a pipeline or job](../ci/pipelines/settings.md#restrict-roles-that-can-cancel-pipelines-or-jobs). One of `developer`, `maintainer`, or `no_one`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429921) in GitLab 16.8.)_ | +| `container_expiration_policy_attributes` | hash | No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | +| `container_registry_access_level` | string | No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | +| `container_registry_enabled` | boolean | No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | +| `default_branch` | string | No | The [default branch](../user/project/repository/branches/default.md) name. | +| `description` | string | No | Short project description. | +| `emails_disabled` | boolean | No | _(Deprecated)_ Disable email notifications. Use `emails_enabled` instead | +| `emails_enabled` | boolean | No | Enable email notifications. | +| `enforce_auth_checks_on_uploads` | boolean | No | Enforce [auth checks](../security/user_file_uploads.md#enable-authorization-checks-for-all-media-files) on uploads. | +| `external_authorization_classification_label` **(PREMIUM ALL)** | string | No | The classification label for the project. | +| `forking_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `group_runners_enabled` | boolean | No | Enable group runners for this project. | +| `import_url` | string | No | URL the repository was imported from. | +| `issues_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `issues_enabled` | boolean | No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | +| `issues_template` **(PREMIUM ALL)** | string | No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | +| `jobs_enabled` | boolean | No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | +| `keep_latest_artifact` | boolean | No | Disable or enable the ability to keep the latest artifact for this project. | +| `lfs_enabled` | boolean | No | Enable LFS. | +| `merge_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | +| `merge_method` | string | No | Set the [merge method](#project-merge-method) used. | +| `merge_pipelines_enabled` | boolean | No | Enable or disable merged results pipelines. | +| `merge_requests_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `merge_requests_enabled` | boolean | No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | +| `merge_requests_template` **(PREMIUM ALL)** | string | No | Default description for merge requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | +| `merge_trains_enabled` | boolean | No | Enable or disable merge trains. | +| `mirror_overwrites_diverged_branches` **(PREMIUM ALL)** | boolean | No | Pull mirror overwrites diverged branches. | +| `mirror_trigger_builds` **(PREMIUM ALL)** | boolean | No | Pull mirroring triggers builds. | +| `mirror_user_id` **(PREMIUM ALL)** | integer | No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | +| `mirror` **(PREMIUM ALL)** | boolean | No | Enables pull mirroring in a project. | +| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | +| `name` | string | No | The name of the project. | +| `only_allow_merge_if_all_discussions_are_resolved` | boolean | No | Set whether merge requests can only be merged when all the discussions are resolved. | +| `only_allow_merge_if_pipeline_succeeds` | boolean | No | Set whether merge requests can only be merged with successful jobs. | +| `only_mirror_protected_branches` **(PREMIUM ALL)** | boolean | No | Only mirror protected branches. | +| `packages_enabled` | boolean | No | Enable or disable packages repository feature. | +| `pages_access_level` | string | No | One of `disabled`, `private`, `enabled`, or `public`. | +| `path` | string | No | Custom repository name for the project. By default generated based on name. | +| `prevent_merge_without_jira_issue` **(PREMIUM ALL)** | boolean | No | Set whether merge requests require an associated issue from Jira. | +| `printing_merge_request_link_enabled` | boolean | No | Show link to create/view merge request when pushing from the command line. | +| `public_builds` | boolean | No | If `true`, jobs can be viewed by non-project members. | +| `releases_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `environments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `feature_flags_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `infrastructure_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `monitor_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `model_experiments_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `model_registry_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `remove_source_branch_after_merge` | boolean | No | Enable `Delete source branch` option by default for all new merge requests. | +| `repository_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `repository_storage` | string | No | Which storage shard the repository is on. _(administrators only)_ | +| `request_access_enabled` | boolean | No | Allow users to request member access. | +| `requirements_access_level` | string | No | One of `disabled`, `private`, `enabled` or `public` | +| `resolve_outdated_diff_discussions` | boolean | No | Automatically resolve merge request diffs discussions on lines changed with a push. | +| `restrict_user_defined_variables` | boolean | No | Allow only users with the Maintainer role to pass user-defined variables when triggering a pipeline. For example when the pipeline is triggered in the UI, with the API, or by a trigger token. | +| `security_and_compliance_access_level` | string | No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | +| `service_desk_enabled` | boolean | No | Enable or disable Service Desk feature. | +| `shared_runners_enabled` | boolean | No | Enable shared runners for this project. | +| `show_default_award_emojis` | boolean | No | Show default emoji reactions. | +| `snippets_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `snippets_enabled` | boolean | No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `issue_branch_template` | string | No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | +| `squash_commit_template` | string | No | [Template](../user/project/merge_requests/commit_templates.md) used to create squash commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345275) in GitLab 14.6.)_ | +| `squash_option` | string | No | One of `never`, `always`, `default_on`, or `default_off`. | +| `suggestion_commit_message` | string | No | The commit message used to apply merge request suggestions. | +| `tag_list` | array | No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `topics` | array | No | The list of topics for the project. This replaces any existing topics that are already added to the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | +| `visibility` | string | No | See [project visibility level](#project-visibility-level). | +| `wiki_access_level` | string | No | One of `disabled`, `private`, or `enabled`. | +| `wiki_enabled` | boolean | No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | ## Fork project @@ -1799,18 +1801,18 @@ fork of the project has completed, query the `import_status` for the new project POST /projects/:id/fork ``` -| Attribute | Type | Required | Description | -|------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `description` | string | No | The description assigned to the resultant project after forking. | -| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | -| `name` | string | No | The name assigned to the resultant project after forking. | -| `namespace_id` | integer | No | The ID of the namespace that the project is forked to. | -| `namespace_path` | string | No | The path of the namespace that the project is forked to. | -| `namespace` | integer or string | No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | -| `path` | string | No | The path assigned to the resultant project after forking. | -| `visibility` | string | No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | -| `branches` | string | No | Branches to fork (empty for all branches). | +| Attribute | Type | Required | Description | +|--------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `branches` | string | No | Branches to fork (empty for all branches). | +| `description` | string | No | The description assigned to the resultant project after forking. | +| `mr_default_target_self` | boolean | No | For forked projects, target merge requests to this project. If `false`, the target is the upstream project. | +| `name` | string | No | The name assigned to the resultant project after forking. | +| `namespace_id` | integer | No | The ID of the namespace that the project is forked to. | +| `namespace_path` | string | No | The path of the namespace that the project is forked to. | +| `namespace` | integer or string | No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | +| `path` | string | No | The path assigned to the resultant project after forking. | +| `visibility` | string | No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | ## List forks of a project @@ -1823,25 +1825,25 @@ forked relationship with the specified project GET /projects/:id/forks ``` -| Attribute | Type | Required | Description | -|-------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `archived` | boolean | No | Limit by archived status. | -| `membership` | boolean | No | Limit by projects that the current user is a member of. | -| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | -| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | -| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | -| `search` | string | No | Return list of projects matching the search criteria. | -| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | -| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | -| `starred` | boolean | No | Limit by projects starred by the current user. | -| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | -| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | -| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | -| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | -| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | -| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| Attribute | Type | Required | Description | +|-------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `archived` | boolean | No | Limit by archived status. | +| `membership` | boolean | No | Limit by projects that the current user is a member of. | +| `min_access_level` | integer | No | Limit by current user minimal [role (`access_level`)](members.md#roles). | +| `order_by` | string | No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `owned` | boolean | No | Limit by projects explicitly owned by the current user. | +| `search` | string | No | Return list of projects matching the search criteria. | +| `simple` | boolean | No | Return only limited fields for each project. Without authentication, this operation is a no-op; only simple fields are returned. | +| `sort` | string | No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `starred` | boolean | No | Limit by projects starred by the current user. | +| `statistics` | boolean | No | Include project statistics. Available only to users with at least the Reporter role. | +| `updated_after` | datetime | No | Limit results to projects last updated after the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `updated_before` | datetime | No | Limit results to projects last updated before the specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393979) in GitLab 15.10. | +| `visibility` | string | No | Limit by visibility `public`, `internal`, or `private`. | +| `with_custom_attributes` | boolean | No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | +| `with_issues_enabled` | boolean | No | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | No | Limit by enabled merge requests feature. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/forks" @@ -1942,9 +1944,9 @@ starred. POST /projects/:id/star ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/star" @@ -2202,9 +2204,9 @@ Get languages used in a project with percentage value. GET /projects/:id/languages ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/languages" @@ -2233,9 +2235,9 @@ does not change the project. POST /projects/:id/archive ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/archive" @@ -2366,9 +2368,9 @@ doesn't change the project. POST /projects/:id/unarchive ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/unarchive" @@ -2514,11 +2516,11 @@ The option to delete projects immediately from deletion protection settings in t DELETE /projects/:id ``` -| Attribute | Type | Required | Description | -|------------------------------------|-------------------|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `permanently_remove` **(PREMIUM ALL)** | boolean/string | no | Immediately deletes a project if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11 | -| `full_path` **(PREMIUM ALL)** | string | no | Full path of project to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11. To find the project path, use `path_with_namespace` from [get single project](projects.md#get-single-project) | +| Attribute | Type | Required | Description | +|----------------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `full_path` **(PREMIUM ALL)** | string | no | Full path of project to use with `permanently_remove`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11. To find the project path, use `path_with_namespace` from [get single project](projects.md#get-single-project) | +| `permanently_remove` **(PREMIUM ALL)** | boolean/string | no | Immediately deletes a project if it is marked for deletion. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/396500) in GitLab 15.11 | ## Restore project marked for deletion **(PREMIUM ALL)** @@ -2530,9 +2532,9 @@ Restores project marked for deletion. POST /projects/:id/restore ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Upload a file @@ -2547,10 +2549,10 @@ description, or a comment. POST /projects/:id/uploads ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `file` | string | Yes | The file to be uploaded. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `file` | string | Yes | The file to be uploaded. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | To upload a file from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -2585,10 +2587,10 @@ Uploads an avatar to the specified project. PUT /projects/:id ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `avatar` | string | Yes | The file to be uploaded. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `avatar` | string | Yes | The file to be uploaded. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | To upload an avatar from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -2631,12 +2633,12 @@ Allow to share project with group. POST /projects/:id/share ``` -| Attribute | Type | Required | Description | -|----------------|----------------|------------------------|-------------| -| `group_access` | integer | Yes | The [role (`access_level`)](members.md#roles) to grant the group. | -| `group_id` | integer | Yes | The ID of the group to share with. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `expires_at` | string | No | Share expiration date in ISO 8601 format: 2016-09-26 | +| Attribute | Type | Required | Description | +|----------------|-------------------|----------|-------------| +| `group_access` | integer | Yes | The [role (`access_level`)](members.md#roles) to grant the group. | +| `group_id` | integer | Yes | The ID of the group to share with. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `expires_at` | string | No | Share expiration date in ISO 8601 format. For example, `2016-09-26`. | ## Delete a shared project link within a group @@ -2646,10 +2648,10 @@ Unshare the project from the group. Returns `204` and no content on success. DELETE /projects/:id/share/:group_id ``` -| Attribute | Type | Required | Description | -|------------|----------------|------------------------|-------------| -| `group_id` | integer | Yes | The ID of the group. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-------------| +| `group_id` | integer | Yes | The ID of the group. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/share/17" @@ -2668,10 +2670,10 @@ If the importing member's role in the target project is: POST /projects/:id/import_project_members/:project_id ``` -| Attribute | Type | Required | Description | -|--------------|-------------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the target project to receive the members. | -| `project_id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the source project to import the members from. | +| Attribute | Type | Required | Description | +|--------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the target project to receive the members. | +| `project_id` | integer or string | Yes | The ID or [URL-encoded path](rest/index.md#namespaced-path-encoding) of the source project to import the members from. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/import_project_members/32" @@ -2723,9 +2725,9 @@ Get a list of project hooks. GET /projects/:id/hooks ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ### Get project hook @@ -2735,10 +2737,10 @@ Get a specific hook for a project. GET /projects/:id/hooks/:hook_id ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|---------------------------| -| `hook_id` | integer | Yes | The ID of a project hook. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `hook_id` | integer | Yes | The ID of a project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```json { @@ -2775,25 +2777,25 @@ Adds a hook to a specified project. POST /projects/:id/hooks ``` -| Attribute | Type | Required | Description | -|------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `url` | string | Yes | The hook URL. | -| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | -| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | -| `deployment_events` | boolean | No | Trigger hook on deployment events. | -| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | -| `issues_events` | boolean | No | Trigger hook on issues events. | -| `job_events` | boolean | No | Trigger hook on job events. | -| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | -| `note_events` | boolean | No | Trigger hook on note events. | -| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | -| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | -| `push_events` | boolean | No | Trigger hook on push events. | -| `releases_events` | boolean | No | Trigger hook on release events. | -| `tag_push_events` | boolean | No | Trigger hook on tag push events. | -| `token` | string | No | Secret token to validate received payloads; the token isn't returned in the response. | -| `wiki_page_events` | boolean | No | Trigger hook on wiki events. | +| Attribute | Type | Required | Description | +|------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `url` | string | Yes | The hook URL. | +| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | +| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | +| `deployment_events` | boolean | No | Trigger hook on deployment events. | +| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | +| `issues_events` | boolean | No | Trigger hook on issues events. | +| `job_events` | boolean | No | Trigger hook on job events. | +| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | +| `note_events` | boolean | No | Trigger hook on note events. | +| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | +| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | +| `push_events` | boolean | No | Trigger hook on push events. | +| `releases_events` | boolean | No | Trigger hook on release events. | +| `tag_push_events` | boolean | No | Trigger hook on tag push events. | +| `token` | string | No | Secret token to validate received payloads; the token isn't returned in the response. | +| `wiki_page_events` | boolean | No | Trigger hook on wiki events. | ### Edit project hook @@ -2803,26 +2805,26 @@ Edits a hook for a specified project. PUT /projects/:id/hooks/:hook_id ``` -| Attribute | Type | Required | Description | -|------------------------------|----------------|------------------------|-------------| -| `hook_id` | integer | Yes | The ID of the project hook. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `url` | string | Yes | The hook URL. | -| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | -| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | -| `deployment_events` | boolean | No | Trigger hook on deployment events. | -| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | -| `issues_events` | boolean | No | Trigger hook on issues events. | -| `job_events` | boolean | No | Trigger hook on job events. | -| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | -| `note_events` | boolean | No | Trigger hook on note events. | -| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | -| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | -| `push_events` | boolean | No | Trigger hook on push events. | -| `releases_events` | boolean | No | Trigger hook on release events. | -| `tag_push_events` | boolean | No | Trigger hook on tag push events. | -| `token` | string | No | Secret token to validate received payloads. Not returned in the response. When you change the webhook URL, the secret token is reset and not retained. | -| `wiki_page_events` | boolean | No | Trigger hook on wiki page events. | +| Attribute | Type | Required | Description | +|------------------------------|-------------------|----------|-------------| +| `hook_id` | integer | Yes | The ID of the project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `url` | string | Yes | The hook URL. | +| `confidential_issues_events` | boolean | No | Trigger hook on confidential issues events. | +| `confidential_note_events` | boolean | No | Trigger hook on confidential note events. | +| `deployment_events` | boolean | No | Trigger hook on deployment events. | +| `enable_ssl_verification` | boolean | No | Do SSL verification when triggering the hook. | +| `issues_events` | boolean | No | Trigger hook on issues events. | +| `job_events` | boolean | No | Trigger hook on job events. | +| `merge_requests_events` | boolean | No | Trigger hook on merge requests events. | +| `note_events` | boolean | No | Trigger hook on note events. | +| `pipeline_events` | boolean | No | Trigger hook on pipeline events. | +| `push_events_branch_filter` | string | No | Trigger hook on push events for matching branches only. | +| `push_events` | boolean | No | Trigger hook on push events. | +| `releases_events` | boolean | No | Trigger hook on release events. | +| `tag_push_events` | boolean | No | Trigger hook on tag push events. | +| `token` | string | No | Secret token to validate received payloads. Not returned in the response. When you change the webhook URL, the secret token is reset and not retained. | +| `wiki_page_events` | boolean | No | Trigger hook on wiki page events. | ### Delete project hook @@ -2833,10 +2835,10 @@ multiple times. Either the hook is available or not. DELETE /projects/:id/hooks/:hook_id ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `hook_id` | integer | Yes | The ID of the project hook. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `hook_id` | integer | Yes | The ID of the project hook. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Note the JSON response differs if the hook is available or not. If the project hook is available before it's returned in the JSON response or an empty response @@ -2853,10 +2855,10 @@ Available only for project owners and administrators. POST /projects/:id/fork/:forked_from_id ``` -| Attribute | Type | Required | Description | -|------------------|----------------|------------------------|-------------| -| `forked_from_id` | ID | Yes | The ID of the project that was forked from. | -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|------------------|-------------------|----------|-------------| +| `forked_from_id` | ID | Yes | The ID of the project that was forked from. | +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ### Delete an existing forked from relationship @@ -2864,9 +2866,9 @@ POST /projects/:id/fork/:forked_from_id DELETE /projects/:id/fork ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Search for projects by name @@ -2878,11 +2880,11 @@ accessible. GET /projects ``` -| Attribute | Type | Required | Description | -|------------|--------|------------------------|-------------| -| `search` | string | Yes | A string contained in the project name. | -| `order_by` | string | No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | -| `sort` | string | No | Return requests sorted in `asc` or `desc` order. | +| Attribute | Type | Required | Description | +|------------|--------|----------|-------------| +| `search` | string | Yes | A string contained in the project name. | +| `order_by` | string | No | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields. | +| `sort` | string | No | Return requests sorted in `asc` or `desc` order. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?search=test" @@ -2894,10 +2896,10 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a POST /projects/:id/housekeeping ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `task` | string | No | `prune` to trigger manual prune of unreachable objects or `eager` to trigger eager housekeeping. | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `task` | string | No | `prune` to trigger manual prune of unreachable objects or `eager` to trigger eager housekeeping. | ## Push rules **(PREMIUM ALL)** @@ -2910,9 +2912,9 @@ project. GET /projects/:id/push_rule ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) | ```json { @@ -2942,21 +2944,21 @@ Adds a push rule to a specified project. POST /projects/:id/push_rule ``` -| Attribute | Type | Required | Description | -|-----------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | -| `branch_name_regex` | string | No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | -| `commit_committer_name_check` | boolean | No | Users can only push commits to this repository if the commit author name is consistent with their GitLab account name. | -| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | -| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | -| `deny_delete_tag` | boolean | No | Deny deleting a tag. | -| `file_name_regex` | string | No | All committed file names must **not** match this, for example `(jar|exe)$`. | -| `max_file_size` | integer | No | Maximum file size (MB). | -| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | -| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | -| `reject_unsigned_commits` | boolean | No | Reject commit when it's not signed through GPG. | +| Attribute | Type | Required | Description | +|---------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | +| `branch_name_regex` | string | No | All branch names must match this, for example `(feature | +| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | +| `commit_committer_name_check` | boolean | No | Users can only push commits to this repository if the commit author name is consistent with their GitLab account name. | +| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | +| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | +| `deny_delete_tag` | boolean | No | Deny deleting a tag. | +| `file_name_regex` | string | No | All committed file names must **not** match this, for example `(jar | +| `max_file_size` | integer | No | Maximum file size (MB). | +| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | +| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | +| `reject_unsigned_commits` | boolean | No | Reject commit when it's not signed through GPG. | ### Edit project push rule @@ -2966,21 +2968,21 @@ Edits a push rule for a specified project. PUT /projects/:id/push_rule ``` -| Attribute | Type | Required | Description | -|-----------------------------------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | -| `branch_name_regex` | string | No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | -| `commit_committer_name_check` | boolean | No | Users can only push commits to this repository if the commit author name is consistent with their GitLab account name. | -| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | -| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | -| `deny_delete_tag` | boolean | No | Deny deleting a tag. | -| `file_name_regex` | string | No | All committed file names must **not** match this, for example `(jar|exe)$`. | -| `max_file_size` | integer | No | Maximum file size (MB). | -| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | -| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | -| `reject_unsigned_commits` | boolean | No | Reject commits when they are not GPG signed. | +| Attribute | Type | Required | Description | +|---------------------------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `author_email_regex` | string | No | All commit author emails must match this, for example `@my-company.com$`. | +| `branch_name_regex` | string | No | All branch names must match this, for example `(feature | +| `commit_committer_check` | boolean | No | Users can only push commits to this repository if the committer email is one of their own verified emails. | +| `commit_committer_name_check` | boolean | No | Users can only push commits to this repository if the commit author name is consistent with their GitLab account name. | +| `commit_message_negative_regex` | string | No | No commit message is allowed to match this, for example `ssh\:\/\/`. | +| `commit_message_regex` | string | No | All commit messages must match this, for example `Fixed \d+\..*`. | +| `deny_delete_tag` | boolean | No | Deny deleting a tag. | +| `file_name_regex` | string | No | All committed file names must **not** match this, for example `(jar | +| `max_file_size` | integer | No | Maximum file size (MB). | +| `member_check` | boolean | No | Restrict commits by author (email) to existing GitLab users. | +| `prevent_secrets` | boolean | No | GitLab rejects any files that are likely to contain secrets. | +| `reject_unsigned_commits` | boolean | No | Reject commits when they are not GPG signed. | ### Delete project push rule @@ -2992,9 +2994,9 @@ Removes a push rule from a project. DELETE /projects/:id/push_rule ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ## Get groups to which a user can transfer a project @@ -3006,10 +3008,10 @@ Retrieve a list of groups to which the user can transfer a project. GET /projects/:id/transfer_locations ``` -| Attribute | Type | Required | Description | -|-------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `search` | string | No | The group names to search for. | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `search` | string | No | The group names to search for. | Example request: @@ -3051,10 +3053,10 @@ for prerequisites to transfer a project. PUT /projects/:id/transfer ``` -| Attribute | Type | Required | Description | -|-------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `namespace` | integer or string | Yes | The ID or path of the namespace to transfer to project to. | +| Attribute | Type | Required | Description | +|-------------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `namespace` | integer or string | Yes | The ID or path of the namespace to transfer to project to. | Example request: @@ -3202,9 +3204,9 @@ GET /projects/:id/mirror/pull Supported attributes: -| Attribute | Type | Required | Description | -|:----------|:------|:------------|:------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: @@ -3242,13 +3244,13 @@ you can add the authentication information to the URL: where `token` is a [personal access token](../user/profile/personal_access_tokens.md) with the API scope enabled. -| Attribute | Type | Required | Description | -|--------------|---------|------------------------|-------------| -| `import_url` | string | Yes | URL of remote repository being mirrored (with `user:token` if needed). | -| `mirror` | boolean | Yes | Enables pull mirroring on project when set to `true`. | -| `mirror_trigger_builds`| boolean | No | Trigger pipelines for mirror updates when set to `true`. | -| `only_mirror_protected_branches`| boolean | No | Limits mirroring to only protected branches when set to `true`. | -| `mirror_branch_regex` | String | No | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_mirror_protected_branches` to be disabled. | +| Attribute | Type | Required | Description | +|----------------------------------|---------|----------|-------------| +| `import_url` | string | Yes | URL of remote repository being mirrored (with `user:token` if needed). | +| `mirror` | boolean | Yes | Enables pull mirroring on project when set to `true`. | +| `mirror_trigger_builds` | boolean | No | Trigger pipelines for mirror updates when set to `true`. | +| `only_mirror_protected_branches` | boolean | No | Limits mirroring to only protected branches when set to `true`. | +| `mirror_branch_regex` | String | No | Contains a regular expression. Only branches with names matching the regex are mirrored. Requires `only_mirror_protected_branches` to be disabled. | Example creating a project with pull mirroring: @@ -3288,9 +3290,9 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ POST /projects/:id/mirror/pull ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/mirror/pull" @@ -3315,10 +3317,10 @@ snapshot may allow some of the data to be retrieved. GET /projects/:id/snapshot ``` -| Attribute | Type | Required | Description | -|-----------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `wiki` | boolean | No | Whether to download the wiki, rather than project, repository. | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `wiki` | boolean | No | Whether to download the wiki, rather than project, repository. | ## Get the path to repository storage @@ -3333,9 +3335,9 @@ Available for administrators only. GET /projects/:id/storage ``` -| Attribute | Type | Required | Description | -|--------------|----------------|------------------------|-------------| -| `id` | integer or string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-------------| +| `id` | integer or string | Yes | ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | ```json [ diff --git a/doc/api/rest/index.md b/doc/api/rest/index.md index 76db6273399..1b943f37e3f 100644 --- a/doc/api/rest/index.md +++ b/doc/api/rest/index.md @@ -1,6 +1,7 @@ --- stage: Manage group: Import and Integrate +description: Programmatic interaction with GitLab. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/runners.md b/doc/api/runners.md index ac854a477d3..373fc4e4344 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -14,6 +14,7 @@ This page describes endpoints for runners registered to an instance. To create a GET /runners GET /runners/all GET /runners/:id/jobs +GET /runners/:id/managers/:system_id/jobs GET /projects/:id/runners GET /groups/:id/runners ``` @@ -367,7 +368,7 @@ NOTE: The `active` form attribute was deprecated [in GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/347211). and will be removed in [a future version of the REST API](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). It is replaced by the `paused` attribute. -## List runner's jobs +## List jobs processed by a runner > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15432) in GitLab 10.3. @@ -378,12 +379,13 @@ to projects where the user has at least the Reporter role. GET /runners/:id/jobs ``` -| Attribute | Type | Required | Description | -|-----------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a runner | -| `status` | string | no | Status of the job; one of: `running`, `success`, `failed`, `canceled` | -| `order_by`| string | no | Order jobs by `id` | -| `sort` | string | no | Sort jobs in `asc` or `desc` order (default: `desc`). Specify `order_by` as well, including for `id`. | +| Attribute | Type | Required | Description | +|-------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a runner | +| `system_id` | string | no | System ID of the machine where the runner manager is running | +| `status` | string | no | Status of the job; one of: `running`, `success`, `failed`, `canceled` | +| `order_by` | string | no | Order jobs by `id` | +| `sort` | string | no | Sort jobs in `asc` or `desc` order (default: `desc`). If `sort` is specified, `order_by` must be specified as well | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/1/jobs?status=running" diff --git a/doc/api/search.md b/doc/api/search.md index f452d7f0398..68b845227db 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -384,7 +384,7 @@ NOTE: This scope is available only if [Elasticsearch](../integration/advanced_search/elasticsearch.md) is enabled. ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/search?scope=notes&search=maxime" ``` Example response: @@ -775,7 +775,7 @@ NOTE: This scope is available only if [Elasticsearch](../integration/advanced_search/elasticsearch.md) is enabled. ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/6/search?scope=notes&search=maxime" ``` Example response: @@ -1023,8 +1023,6 @@ Example response: > Moved to GitLab Premium in 13.9. -This scope is available only if [Elasticsearch](../integration/advanced_search/elasticsearch.md) is enabled. - ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime" ``` @@ -1060,8 +1058,6 @@ Example response: > Moved to GitLab Premium in 13.9. -This scope is available only if [Elasticsearch](../integration/advanced_search/elasticsearch.md) is enabled. - Filters are available for this scope: - filename @@ -1144,8 +1140,6 @@ Example response: > Moved to GitLab Premium in 13.9. -This scope is available only if [Elasticsearch](../integration/advanced_search/elasticsearch.md) is enabled. - Filters are available for this scope: - Filename diff --git a/doc/api/secure_files.md b/doc/api/secure_files.md index 2e133522cb9..87c39520942 100644 --- a/doc/api/secure_files.md +++ b/doc/api/secure_files.md @@ -91,8 +91,8 @@ Supported attributes: | Attribute | Type | Required | Description | |--------------|----------------|----------|-------------| -| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `id` | integer | Yes | The ID of a secure file. | +| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: @@ -126,9 +126,9 @@ Supported attributes: | Attribute | Type | Required | Description | |-----------------|----------------|----------|-------------| -| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | -| `name` | string | Yes | The name of the file being uploaded. The file name must be unique in the project. | | `file` | file | Yes | The file being uploaded (5 MB limit). | +| `name` | string | Yes | The name of the file being uploaded. The file name must be unique in the project. | +| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: @@ -163,8 +163,8 @@ Supported attributes: | Attribute | Type | Required | Description | |--------------|----------------|----------|-------------| -| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `id` | integer | Yes | The ID of a secure file. | +| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: @@ -184,8 +184,8 @@ Supported attributes: | Attribute | Type | Required | Description | |--------------|----------------|----------|-------------| -| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `id` | integer | Yes | The ID of a secure file. | +| `project_id` | integer/string | Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | Example request: diff --git a/doc/api/settings.md b/doc/api/settings.md index cf34cf3d65e..bde4c769b92 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -527,7 +527,7 @@ listed in the descriptions of the relevant settings. | `protected_ci_variables` | boolean | no | CI/CD variables are protected by default. | | `disable_overriding_approvers_per_merge_request` | boolean | no | Prevent editing approval rules in projects and merge requests | | `prevent_merge_requests_author_approval` | boolean | no | Prevent approval by author | -| `prevent_merge_requests_committers_approval` | boolean | no | Prevent editing approval rules in projects and merge requests | +| `prevent_merge_requests_committers_approval` | boolean | no | Prevent approval by committers to merge requests | | `push_event_activities_limit` | integer | no | Maximum number of changes (branches or tags) in a single push above which a [bulk push event is created](../administration/settings/push_event_activities_limit.md). Setting to `0` does not disable throttling. | | `push_event_hooks_limit` | integer | no | Maximum number of changes (branches or tags) in a single push above which webhooks and integrations are not triggered. Setting to `0` does not disable throttling. | | `rate_limiting_response_text` | string | no | When rate limiting is enabled via the `throttle_*` settings, send this plain text response when a rate limit is exceeded. 'Retry later' is sent if this is blank. | diff --git a/doc/api/statistics.md b/doc/api/statistics.md index 8868f6d5190..fc0aa9a0b39 100644 --- a/doc/api/statistics.md +++ b/doc/api/statistics.md @@ -12,7 +12,8 @@ List the current statistics of the GitLab instance. You have to be an administrator to perform this action. NOTE: -These statistics are approximate. +These statistics show exact counts for values less than 10,000. For values of 10,000 and higher, these statistics show approximate data +when [TablesampleCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/tablesample_count_strategy.rb?ref_type=heads#L16) and [ReltuplesCountStrategy](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/count/reltuples_count_strategy.rb?ref_type=heads) strategies are used for calculations. ```plaintext GET /application/statistics diff --git a/doc/api/templates/gitlab_ci_ymls.md b/doc/api/templates/gitlab_ci_ymls.md index 96b2247600d..25667a2e7f7 100644 --- a/doc/api/templates/gitlab_ci_ymls.md +++ b/doc/api/templates/gitlab_ci_ymls.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w In GitLab, there is an API endpoint available to work with GitLab CI/CD YAML. For more information on CI/CD pipeline configuration in GitLab, see the -[configuration reference documentation](../../ci/yaml/index.md). +[CI/CD YAML syntax reference](../../ci/yaml/index.md). ## List GitLab CI YAML templates diff --git a/doc/api/users.md b/doc/api/users.md index 31fe6234ad2..cd911196f74 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -25,8 +25,6 @@ GET /users | Attribute | Type | Required | Description | | ------------------ | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | | `username` | string | no | Get a single user with a specific username. | -| `extern_uid` | string | no | Get a single user with a specific external authentication provider UID. | -| `provider` | string | no | The external provider. | | `search` | string | no | Search for a username. | | `active` | boolean | no | Filters only active users. Default is `false`. | | `external` | boolean | no | Filters only external users. Default is `false`. | @@ -146,6 +144,8 @@ You can use all [parameters available for everyone](#for-non-administrator-users | Attribute | Type | Required | Description | | ------------------ | ------- | -------- | --------------------------------------------------------------------------------------------------------------------- | +| `extern_uid` | string | no | Get a single user with a specific external authentication provider UID. | +| `provider` | string | no | The external provider. | | `order_by` | string | no | Return users ordered by `id`, `name`, `username`, `created_at`, or `updated_at` fields. Default is `id` | | `sort` | string | no | Return users sorted in `asc` or `desc` order. Default is `desc` | | `two_factor` | string | no | Filter users by Two-factor authentication. Filter values are `enabled` or `disabled`. By default it returns all users | @@ -554,6 +554,7 @@ Parameters: | `bio` | No | User's biography | | `can_create_group` | No | User can create top-level groups - true or false | | `color_scheme_id` | No | User's color scheme for the file viewer (for more information, see the [user preference documentation](../user/profile/preferences.md#change-the-syntax-highlighting-theme)) | +| `commit_email` | No | User's commit email address | | `email` | Yes | Email | | `extern_uid` | No | External UID | | `external` | No | Flags the user as external - true or false (default) | @@ -568,7 +569,9 @@ Parameters: | `password` | No | Password | | `private_profile` | No | User's profile is private - true or false. The default value is determined by [this](../administration/settings/account_and_limit_settings.md#set-profiles-of-new-users-to-private-by-default) setting. | | `projects_limit` | No | Number of projects user can create | +| `pronouns` | No | User's pronouns | | `provider` | No | External provider name | +| `public_email` | No | User's public email address | | `reset_password` | No | Send user password reset link - true or false(default) | | `shared_runners_minutes_limit` **(PREMIUM ALL)** | No | Can be set by administrators only. Maximum number of monthly compute minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | | `skip_confirmation` | No | Skip confirmation - true or false (default) | @@ -910,11 +913,6 @@ Example response: } ``` -Users on [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) also see these -preferences if `code_suggestions_used_by_default` feature flag is disabled: - -- `code_suggestions` - Parameters: - **none** @@ -945,12 +943,6 @@ Parameters: | `show_whitespace_in_diffs` | Yes | Flag indicating the user sees whitespace changes in diffs. | | `pass_user_identities_to_ci_jwt` | Yes | Flag indicating the user passes their external identities as CI information. This attribute does not contain enough information to identify or authorize the user in an external system. The attribute is internal to GitLab, and must not be passed to third-party services. For more information and examples, see [Token Payload](../ci/secrets/id_token_authentication.md#token-payload). | -Users on [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/) also can update these parameters: - -| Attribute | Required | Description | -|:---------------------------------|:---------|:---------------------------------------------------| -| `code_suggestions` | No | Flag indicating the user allows code suggestions. Argument is experimental and can be removed in the future without notice. In GitLab 16.8 and later, this attribute is ignored if `code_suggestions_used_by_default` feature flag is enabled. | - ## User follow ### Follow and unfollow users diff --git a/doc/api/vulnerability_exports.md b/doc/api/vulnerability_exports.md index 3be1dccea29..f2e0784cda7 100644 --- a/doc/api/vulnerability_exports.md +++ b/doc/api/vulnerability_exports.md @@ -187,11 +187,12 @@ The response is `404 Not Found` if the vulnerability export is not finished yet Example response: ```csv -Group Name,Project Name,Tool,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers,Detected At,Location,Activity,Comments,Full Path -Gitlab.org,Defend,container_scanning,Trivy,resolved,CVE-2019-14697 in musl-utils-1.1.20-r4,"musl libc through 1.1.23 has an x87 floating-point stack adjustment imbalance, related to the math/i386/ directory. In some cases, use of this library could introduce out-of-bounds writes that are not present in an application's source code.",CVE-2019-14697 in musl-utils-1.1.20-r4,critical,CVE-2019-14697,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl-utils""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"2022-10-07 13:41:08 UTC|root|resolved|changed vulnerability status to resolved",group/project/1 -Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2019-19242 in sqlite-libs-3.26.0-r3,"SQLite 3.30.1 mishandles pExpr->y.pTab, as demonstrated by the TK_COLUMN case in sqlite3ExprCodeTarget in expr.c.",CVE-2019-19242 in sqlite-libs-3.26.0-r3,medium,CVE-2019-19242,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""sqlite-libs""}, ""version""=>""3.26.0-r3""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/2 -Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2020-28928 in musl-1.1.20-r4,"In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow).",CVE-2020-28928 in musl-1.1.20-r4,medium,CVE-2020-28928,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/3 -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,Carefully crafted requests can cause shell escape sequences to be written to the terminal via Rack's Lint middleware and CommonLogger middleware. These escape sequences can be leveraged to possibly execute commands in the victim's terminal.,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,unknown,Gemfile.lock:rack:gemnasium:60b5a27f-4e4d-4ab4-8ae7-74b4b212e177,,Gemnasium-60b5a27f-4e4d-4ab4-8ae7-74b4b212e177; GHSA-wq4h-7r42-5hrr,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"",group/project/4 -Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Denial of Service Vulnerability in Rack Multipart Parsing in rack,"Carefully crafted multipart POST requests can cause Rack's multipart parser to take much longer than expected, leading to a possible denial of service vulnerability. Impacted code will use Rack's multipart parser to parse multipart posts.",Denial of Service Vulnerability in Rack Multipart Parsing in rack,unknown,Gemfile.lock:rack:gemnasium:20daa17a-47b5-4f79-80c2-cd8f2db9805c,,Gemnasium-20daa17a-47b5-4f79-80c2-cd8f2db9805c; GHSA-hxqx-xwvh-44m2,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,"",group/project/5 -Gitlab.org,Defend,sast,Brakeman,detected,Possible SQL injection,,Possible SQL injection,medium,e52f23a259cd489168b4313317ac94a3f13bffde57b9635171c1a44a9f329e9a,,"""Brakeman Warning Code 0""",2022-10-13 15:16:36 UTC,"{""file""=>""main.rb"", ""class""=>""User"", ""method""=>""index"", ""start_line""=>3}",false,"",group/project/6 +Group Name,Project Name,Tool,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers,Detected At,Location,Activity,Comments,Full Path,CVSS Vectors,Dismissal Reason +Gitlab.org,Defend,container_scanning,Trivy,resolved,CVE-2019-14697 in musl-utils-1.1.20-r4,"musl libc through 1.1.23 has an x87 floating-point stack adjustment imbalance, related to the math/i386/ directory. In some cases, use of this library could introduce out-of-bounds writes that are not present in an application's source code.",CVE-2019-14697 in musl-utils-1.1.20-r4,critical,CVE-2019-14697,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl-utils""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"2022-10-07 13:41:08 UTC|root|resolved|changed vulnerability status to resolved",group/project/1,,, +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2019-19242 in sqlite-libs-3.26.0-r3,"SQLite 3.30.1 mishandles pExpr->y.pTab, as demonstrated by the TK_COLUMN case in sqlite3ExprCodeTarget in expr.c.",CVE-2019-19242 in sqlite-libs-3.26.0-r3,medium,CVE-2019-19242,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""sqlite-libs""}, ""version""=>""3.26.0-r3""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/2,,, +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2020-28928 in musl-1.1.20-r4,"In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow).",CVE-2020-28928 in musl-1.1.20-r4,medium,CVE-2020-28928,,"",2022-10-07 13:34:41 UTC,"{""image""=>""python:3.4-alpine"", ""dependency""=>{""package""=>{""name""=>""musl""}, ""version""=>""1.1.20-r4""}, ""operating_system""=>""alpine 3.9.2""}",true,"",group/project/3,,, +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,Carefully crafted requests can cause shell escape sequences to be written to the terminal via Rack's Lint middleware and CommonLogger middleware. These escape sequences can be leveraged to possibly execute commands in the victim's terminal.,Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') in rack,unknown,Gemfile.lock:rack:gemnasium:60b5a27f-4e4d-4ab4-8ae7-74b4b212e177,,Gemnasium-60b5a27f-4e4d-4ab4-8ae7-74b4b212e177; GHSA-wq4h-7r42-5hrr,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,group/project/4,,, +Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Denial of Service Vulnerability in Rack Multipart Parsing in rack,"Carefully crafted multipart POST requests can cause Rack's multipart parser to take much longer than expected, leading to a possible denial of service vulnerability. Impacted code will use Rack's multipart parser to parse multipart posts.",Denial of Service Vulnerability in Rack Multipart Parsing in rack,unknown,Gemfile.lock:rack:gemnasium:20daa17a-47b5-4f79-80c2-cd8f2db9805c,,Gemnasium-20daa17a-47b5-4f79-80c2-cd8f2db9805c; GHSA-hxqx-xwvh-44m2,2022-10-14 13:16:00 UTC,"{""file""=>""Gemfile.lock"", ""dependency""=>{""package""=>{""name""=>""rack""}, ""version""=>""2.2.3""}}",false,group/project/5,,, +Gitlab.org,Defend,sast,Brakeman,detected,Possible SQL injection,,Possible SQL injection,medium,e52f23a259cd489168b4313317ac94a3f13bffde57b9635171c1a44a9f329e9a,,"""Brakeman Warning Code 0""",2022-10-13 15:16:36 UTC,"{""file""=>""main.rb"", ""class""=>""User"", ""method""=>""index"", ""start_line""=>3}",false,"",group/project/6,,, +Gitlab.org,Defend,sast,Semgrep,dismissed,Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection'),"SQL Injection is a critical vulnerability that can lead to data or system compromise...",,critical,,CWE-89,SCS0002,2023-12-28 10:48:34 UTC,"{""file""=>""WebGoat/App_Code/DB/SqliteDbProvider.cs"", ""start_line""=>274}",false,"2023-12-28 10:51:32 UTC|root|Dismissed|""changed vulnerability status to Dismissed: Not Applicable and the following comment: ""dismiss 5""",gitlab-org/defend/579,,Not applicable, ``` diff --git a/doc/architecture/blueprints/ai_gateway/index.md b/doc/architecture/blueprints/ai_gateway/index.md index c09f8aaa621..e40861139d6 100644 --- a/doc/architecture/blueprints/ai_gateway/index.md +++ b/doc/architecture/blueprints/ai_gateway/index.md @@ -103,7 +103,7 @@ GitLab instances, JSON API, and gRPC differ on these items: | + A new Ruby-gRPC server for vscode: likely faster because we can limit dependencies to load ([modular monolith](https://gitlab.com/gitlab-org/gitlab/-/issues/365293)) | - Existing Grape API for vscode: meaning slow boot time and unneeded resources loaded | | + Bi-directional streaming | - Straight forward way to stream requests and responses (could still be added) | | - A new Python-gRPC server: we don't have experience running gRPC-Python servers | + Existing Python fastapi server, already running for Code Suggestions to extend | -| - Hard to pass on unknown messages from vscode through GitLab to ai-gateway | + Easier support for newer vscode + newer ai-gatway, through old GitLab instance | +| - Hard to pass on unknown messages from vscode through GitLab to ai-gateway | + Easier support for newer VS Code + newer AI-gateway, through old GitLab instance | | - Unknown support for gRPC in other clients (vscode, jetbrains, other editors) | + Support in all external clients | | - Possible protocol mismatch (VSCode --REST--> Rails --gRPC--> AI gateway) | + Same protocol across the stack | @@ -264,7 +264,7 @@ Another example use case includes 2 versions of a prompt passed in the `prompt_c a field in the gateway, and keep them around for at least 2 major versions of GitLab.** -A good practise that might help support backwards compatibility is to provide building blocks for the prompt inside the `prompt_components` rather then a complete prompt. By moving responsibility of compiling prompt out of building blocks on the AI-Gateway, one can achive more flexibility in terms of prompt adjustments in the future. +A good practice that might help support backward compatibility: provide building blocks for the prompt inside the `prompt_components`, rather then a complete prompt. By moving responsibility of compiling the prompt out of building blocks and into the AI-Gateway, more flexible prompt adjustments are possible in the future. #### Example feature: Code Suggestions @@ -503,7 +503,7 @@ It is deployed to a Kubernetes cluster in it's own project. There is a staging environment that is currently used directly by engineers for testing. -In the future, this will be deloyed using +In the future, this will be deployed using [Runway](https://gitlab.com/gitlab-com/gl-infra/platform/runway/). At that time, there will be a production and staging deployment. The staging deployment can be used for automated QA-runs that will have diff --git a/doc/architecture/blueprints/cells/impacted_features/git-access.md b/doc/architecture/blueprints/cells/impacted_features/git-access.md index 611b4db5f43..d2d357d4178 100644 --- a/doc/architecture/blueprints/cells/impacted_features/git-access.md +++ b/doc/architecture/blueprints/cells/impacted_features/git-access.md @@ -6,12 +6,10 @@ description: 'Cells: Git Access' <!-- vale gitlab.FutureTense = NO --> -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. # Cells: Git Access @@ -146,11 +144,34 @@ Where: Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. The one major complication is supporting snippets, but this likely falls in the same category as for the approach to support a user's Personal Namespace. -## 4.1. Pros +### 4.1. Pros 1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. -## 4.2. Cons +### 4.2. Cons 1. The sharing of repositories objects is limited to the given Cell and Gitaly node. 1. Cross-Cells forks are likely impossible to be supported (discover: How this works today across different Gitaly node). + +## 5. Forking and object pools + +One of the biggest struggles that needs to be addressed with the Cells architecture is how to handle forking. At present, Gitaly utilizes object pools to provide deduplication of fork storage. If forks are not created on the same storage node as the upstream repository that is being forked, we end up with significant storage inefficiencies as we will effectively have two complete copies of the repository and we will not be able to utilize object pools to improve performance. + +The storage nodes from one Cell cannot talk to the storage nodes of another Cell, making forking across Cells impossible. Therefore, it will be necessary to ensure that forked repositories end up in the same Cell (and on the same Gitaly nodes) as their upstream parent repository. This will also enable Gitaly to continue to utilize object pools to provide storage and performance efficiency. + +### 5.1. How this works today + +**Single Gitaly storage node** + +Currently, for a GitLab instance backed with a single Gitaly storage node, forking works just fine. +Any forks must reside on the same storage node as there is only one, and therefore object deduplication (and object pools) all function as expected. + +**Sharded Gitaly storage** + +A sharded Gitaly storage is when multiple Gitaly storage nodes are attached to a single instance, and repositories are assigned based on a priority weighting between the nodes. + +Since Gitaly knows how to do cross-storage fetches, forking across shards works without issue. + +**Gitaly Cluster** + +For Gitaly cluster, we recently resolved [the issue](https://gitlab.com/gitlab-org/gitaly/-/issues/5094) of object pools not being created on the same storage nodes as the parent repository. This enables forking to work correctly from an efficiency perspective (can share an object pool) and from an object deduplication perspective (Git can properly deduplicate storage). diff --git a/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md index 757f83c32d3..d80f5c44b98 100644 --- a/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md +++ b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md @@ -138,6 +138,6 @@ Cons: ## 4. Evaluation -The most straightforward solution requiring the least engineering effort is to create [one personal Namespace in each Organization](#33-one-personal-namespace-in-each-organization). -We recognize that this solution is not ideal for users working across multiple Organizations, but find this acceptable due to our expectation that most users will mainly work in one Organization. -At a later point, this concept will be reviewed and possibly replaced with a better solution. +We will begin by [making the personal namespace optional for Organizations](https://gitlab.com/groups/gitlab-org/-/epics/12179). The goal of this iteration is to disable personal namespaces for any Organization other than the default Organization, so that customers who do not want to use personal namespaces can already move to Organizations. The first phase will only change the Ruby on Rails model relationships in preparation for further changes at the user-facing level. + +We need to [split the concept of a User Profile and a personal namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/432654) now that a User is cluster-wide and a User's personal namespace must be Cell-local. It is likely we will [discontinue personal namespaces](#34-discontinue-personal-namespaces) in favor of Groups. diff --git a/doc/architecture/blueprints/cells/index.md b/doc/architecture/blueprints/cells/index.md index 3b800a54781..6f00fe4e61e 100644 --- a/doc/architecture/blueprints/cells/index.md +++ b/doc/architecture/blueprints/cells/index.md @@ -305,8 +305,7 @@ It is expected that initial iterations will be rather slow, because they require The Cells architecture has long lasting implications to data processing, location, scalability and the GitLab architecture. This section links all different technical proposals that are being evaluated. -- [Stateless Router That Uses a Cache to Pick Cell and Is Redirected When Wrong Cell Is Reached](proposal-stateless-router-with-buffering-requests.md) -- [Stateless Router That Uses a Cache to Pick Cell and pre-flight `/api/v4/internal/cells/learn`](proposal-stateless-router-with-routes-learning.md) +- [Routing Service](routing-service.md) ## Impacted features diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md index 847532a36dc..699a41879a9 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md @@ -2,8 +2,11 @@ stage: enablement group: Tenant Scale description: 'Cells Stateless Router Proposal' +status: rejected --- +_This proposal was superseded by the [routing service proposal](routing-service.md)_ + <!-- vale gitlab.FutureTense = NO --> This document is a work-in-progress and represents a very early state of the @@ -13,7 +16,7 @@ contrast this with alternatives before deciding which approach to implement. This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. -# Proposal: Stateless Router +# Proposal: Stateless Router using Requests Buffering We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related tables so that they can be shared between all cells and allow any cell to diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md index cdcb5b8b21f..72b96e9ab8c 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md @@ -2,8 +2,11 @@ stage: enablement group: Tenant Scale description: 'Cells Stateless Router Proposal' +status: rejected --- +_This proposal was superseded by the [routing service proposal](routing-service.md)_ + <!-- vale gitlab.FutureTense = NO --> This document is a work-in-progress and represents a very early state of the @@ -13,7 +16,7 @@ contrast this with alternatives before deciding which approach to implement. This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. -# Proposal: Stateless Router +# Proposal: Stateless Router using Routes Learning We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related tables so that they can be shared between all cells and allow any cell to @@ -35,7 +38,7 @@ Organization can only be on a single Cell. ## Differences The main difference between this proposal and one [with buffering requests](proposal-stateless-router-with-buffering-requests.md) -is that this proposal uses a pre-flight API request (`/pi/v4/internal/cells/learn`) to redirect the request body to the correct Cell. +is that this proposal uses a pre-flight API request (`/api/v4/internal/cells/learn`) to redirect the request body to the correct Cell. This means that each request is sent exactly once to be processed, but the URI is used to decode which Cell it should be directed. ## Summary in diagrams diff --git a/doc/architecture/blueprints/cells/routing-service.md b/doc/architecture/blueprints/cells/routing-service.md index 9efdbdf3f91..bd5570b68f4 100644 --- a/doc/architecture/blueprints/cells/routing-service.md +++ b/doc/architecture/blueprints/cells/routing-service.md @@ -59,20 +59,23 @@ For example: ## Requirements -| Requirement | Description | Priority | -|---------------|-------------------------------------------------------------------|----------| -| Discovery | needs to be able to discover and monitor the health of all Cells. | high | -| Security | only authorized cells can be routed to | high | -| Single domain | e.g. GitLab.com | high | -| Caching | can cache routing information for performance | high | -| [50 ms of increased latency](#low-latency) | | high | -| Path-based | can make routing decision based on path | high | -| Complexity | the routing service should be configuration-driven and small | high | -| Stateless | does not need database, Cells provide all routing information | medium | -| Secrets-based | can make routing decision based on secret (e.g. JWT) | medium | -| Observability | can use existing observability tooling | low | -| Self-managed | can be eventually used by [self-managed](goals.md#self-managed) | low | -| Regional | can route requests to different [regions](goals.md#regions) | low | +| Requirement | Description | Priority | +| ------------------- | ----------------------------------------------------------------- | -------- | +| Discovery | needs to be able to discover and monitor the health of all Cells. | high | +| Security | only authorized cells can be routed to | high | +| Single domain | for example GitLab.com | high | +| Caching | can cache routing information for performance | high | +| Low latency | [50 ms of increased latency](#low-latency) | high | +| Path-based | can make routing decision based on path | high | +| Complexity | the routing service should be configuration-driven and small | high | +| Rolling | the routing service works with Cells running mixed versions | high | +| Feature Flags | features can be turned on, off, and % rollout | high | +| Progressive Rollout | we can slowly rollout a change | medium | +| Stateless | does not need database, Cells provide all routing information | medium | +| Secrets-based | can make routing decision based on secret (for example JWT) | medium | +| Observability | can use existing observability tooling | low | +| Self-managed | can be eventually used by [self-managed](goals.md#self-managed) | low | +| Regional | can route requests to different [regions](goals.md#regions) | low | ### Low Latency @@ -91,7 +94,7 @@ The main SLI we use is the [rails requests](../../../development/application_sli It has multiple `satisfied` targets (apdex) depending on the [request urgency](../../../development/application_slis/rails_request.md#how-to-adjust-the-urgency): | Urgency | Duration in ms | -|------------|----------------| +| ---------- | -------------- | | `:high` | 250 _ms_ | | `:medium` | 500 _ms_ | | `:default` | 1000 _ms_ | @@ -108,7 +111,7 @@ The way we calculate the headroom we have is by using the following: **`web`**: | Target Duration | Percentile | Headroom | -|-----------------|------------|-----------| +| --------------- | ---------- | --------- | | 5000 _ms_ | p99 | 4000 _ms_ | | 5000 _ms_ | p95 | 4500 _ms_ | | 5000 _ms_ | p90 | 4600 _ms_ | @@ -131,7 +134,7 @@ _Analysis was done in <https://gitlab.com/gitlab-org/gitlab/-/issues/432934#note **`api`**: | Target Duration | Percentile | Headroom | -|-----------------|------------|-----------| +| --------------- | ---------- | --------- | | 5000 _ms_ | p99 | 3500 _ms_ | | 5000 _ms_ | p95 | 4300 _ms_ | | 5000 _ms_ | p90 | 4600 _ms_ | @@ -154,7 +157,7 @@ _Analysis was done in <https://gitlab.com/gitlab-org/gitlab/-/issues/432934#note **`git`**: | Target Duration | Percentile | Headroom | -|-----------------|------------|-----------| +| --------------- | ---------- | --------- | | 5000 _ms_ | p99 | 3760 _ms_ | | 5000 _ms_ | p95 | 4280 _ms_ | | 5000 _ms_ | p90 | 4430 _ms_ | @@ -180,7 +183,585 @@ Not yet defined. ## Proposal -TBD +The Routing Service implements the following design guidelines: + +1. Simple: + - Routing service does not buffer requests. + - Routing service can only proxy to a single Cell based on request headers. +1. Stateless: + - Routing service does not have permanent storage. + - Routing service uses multi-level cache: in-memory, external shared cache. +1. Zero-trust: + - Routing service signs each request that is being proxied. + - The trust is established by using JWT token, or mutual authentication scheme. + - Cells can be available over public internet, as long as they follow the zero-trust model. +1. Configuration-based: + - Routing service is configured with a static list of Cells. + - Routing service configuration is applied as part of service deployment. +1. Rule-based: + - Routing service is deployed with a routing rules gathered from all Cells. + - Routing service does support rules lists generated by different versions of GitLab. + - rules allows to match by any criteria: header, content of the header, or route path. +1. Agnostic: + - Routing service is not aware of high-level concepts like organizations. + - The classification is done per-specification provided in a rules, to find the sharding key. + - The sharding key result is cached. + - The single sharding key cached is used to handle many similar requests. + +The following diagram shows how a user request routes through DNS to the Routing Service deployed +as Cloudflare Worker and the router chooses a cell to send the request to. + +```mermaid +graph TD; + user((User)); + router[Routing Service]; + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + user-->router; + router-->cell_eu0; + router-->cell_eu1; + router-->cell_us0; + router-->cell_us1; + subgraph Europe + cell_eu0; + cell_eu1; + end + subgraph United States + cell_us0; + cell_us1; + end +``` + +### Routing rules + +Each Cell will publish a precompiled list of routing rules that will be consumed by the Routing Service: + +- The routing rules describe how to decode the request, find the sharding key, and make the routing decision. +- The routing rules are compiled during the deployment of the Routing Service. + - The deployment process fetches latest version of the routing rules from each Cell + that is part of Routing Service configuration. + - The compilation process merges the routing rules from all Cells. + - The conflicting rules prevent routing service from being compiled / started. + - Each routing rule entry has a unique identifier to ease the merge. + - The Routing Service would be re-deployed only if the list of rules was changed, + which shouldn't happen frequently, because we expect the majority of newly added endpoints + to already adhere to the prior route rules. +- The configuration describes from which Cells the routing rules need to be fetched during deploy. +- The published routing rules might make routing decision based on the secret. For example, if the session cookie + or authentication token has prefix `c100-` all requests are to be forwarded to the given Cell. +- The Cell does publish routing rules at `/api/v4/internal/cells/route_rules.json`. +- The rules published by Cell only include endpoints that the particular Cell can process. +- The Cell might request to perform dynamic classification based on sharding key, by configuring + routing rules to call `/api/v4/internal/cells/classify`. +- The routing rules should use `prefix` as a way to speed up classification. During the compilation phase + the routing service transforms all found prefixes into a decision tree to speed up any subsequent regex matches. +- The routing rules is ideally compiled into source code to avoid expensive parsing and evaluation of the rules + dynamically as part of deployment. + +The routing rules JSON structure describes all matchers: + +```json +{ + "rules": [ + { + "id": "<unique-identifier>", + "cookies": { + "<cookie_name>": { + "prefix": "<match-given-prefix>", + "match_regex": "<regex_match>" + }, + "<cookie_name2>": { + "prefix": "<match-given-prefix>", + "match_regex": "<regex_match>" + } + }, + "headers": { + "<header_name>": { + "prefix": "<match-given-prefix>", + "match_regex": "<regex_match>" + }, + "<header_name2>": { + "prefix": "<match-given-prefix>", + "match_regex": "<regex_match>" + }, + }, + "path": { + "prefix": "<match-given-prefix>", + "match_regex": "<regex_match>" + }, + "method": ["<list_of_accepted_methods>"], + + // If many rules are matched, define which one wins + "priority": 1000, + + // Accept request and proxy to the Cell in question + "action": "proxy", + + // Classify request based on regex matching groups + "action": "classify", + "classify": { + "keys": ["list_of_regex_match_capture_groups"] + } + } + ] +} +``` + +Example of the routing rules published by the Cell 100 that makes routing decision based session cookie, and secret. +The high priority is assigned since the routing rules is secret-based, and should take precedence before all other matchers: + +```json +{ + "rules": [ + { + "id": "t4mkd5ndsk58si6uwwz7rdavil9m2hpq", + "cookies": { + "_gitlab_session": { + "prefix": "c100-" // accept `_gitlab_session` that are prefixed with `c100-` + } + }, + "action": "proxy", + "priority": 1000 + }, + { + "id": "jcshae4d4dtykt8byd6zw1ecccl5dkts", + "headers": { + "GITLAB_TOKEN": { + "prefix": "C100_" // accept `GITLAB_TOKEN` that are prefixed with `C100_` + } + }, + "action": "proxy", + "priority": 1000 + } + ] +} +``` + +Example of the routing rules published by all Cells that makes routing decision based on the path: + +```json +{ + "rules": [ + { + "id": "c9scvaiwj51a75kzoh917uwtnw8z4ebl", + "path": { + "prefix": "/api/v4/projects/", // speed-up rule matching + "match_regex": "^/api/v4/projects/(?<project_id_or_path_encoded>[^/]+)(/.*)?$" + }, + "action": "classify", + "classify": { + "keys": ["project_id_or_path_encoded"] + } + } + ] +} +``` + +### Classification + +Each Cell does implement classification endpoint: + +- The classification endpoint is at `/api/v4/internal/cells/classify` (or gRPC endpoint). +- The classification endpoint accepts a list of the sharding keys. Sharding keys are decoded from request, + based on the routing rules provided by the Cell. +- The endpoint returns other equivalent sharding keys to pollute cache for similar requests. + This is to ensure that all similar requests can be handled quickly without having to classify each time. +- Routing Service tracks the health of Cells, and issues a `classify` request to Cells based on weights, + health of the Cell, or other defined criteria. Weights would indicate which Cell is preferred to perform the + classification of sharding keys. +- Routing Service retries the `classify` call for a reasonable amount of time. + The repetitive failure of Cell to `classify` is indicative of Cell being unhealthy. +- The `classify` result is cached regardless of returned `action` (proxy or reject). + The rejected classification is cached to prevent excessive amount of + requests for sharding keys that are not found. +- The cached response is for time defined by `expiry` and `refresh`. + - The `expiry` defines when the item is removed from cache unless used. + - The `refresh` defines when the item needs to be reclassified if used. + - The refresh is done asynchronously as the request should be served without a delay if they were classified. The refresh is done to ensure that cache is always hot and up-to date. + +For the above example: + +1. The router sees request to `/api/v4/projects/1000/issues`. +1. It selects the above `rule` for this request, which requests `classify` for `project_id_or_path_encoded`. +1. It decodes `project_id_or_path_encoded` to be `1000`. +1. Checks the cache if there's `project_id_or_path_encoded=1000` associated to any Cell. +1. Sends the request to `/api/v4/internal/cells/classify` if no Cells was found in cache. +1. Rails responds with the Cell holding the given project, and also all other equivalent sharding keys + for the resource that should be put in the cache. +1. Routing Service caches for the duration specified in configuration, or response. + +```json +# POST /api/v4/internal/cells/classify +## Request: +{ + "metadata": { + "rule_id": "c9scvaiwj51a75kzoh917uwtnw8z4ebl", + "headers": { + "all_request_headers": "value" + }, + "method": "GET", + "path": "/api/v4/projects/100/issues" + }, + "keys": { + "project_id_or_path_encoded": 100 + } +} + +## Response: +{ + "action": "proxy", + "proxy": { + "name": "cell_1", + "url": "https://cell1.gitlab.com" + }, + "ttl": "10 minutes", + "matched_keys": [ // list of all equivalent keys that should be put in the cache + { "project_id_or_path_encoded": 100 }, + { "project_id_or_path_encoded": "gitlab-org%2Fgitlab" }, + { "project_full_path": "gitlab-org/gitlab" }, + { "namespace_full_path": "gitlab-org" }, + { "namespace_id": 10 }, + { "organization_full_path": "gitlab-inc" }, + { "organization_id": 50 }, + ] +} +``` + +The following code represents a negative response when a sharding key was not found: + +```json +# POST /api/v4/internal/cells/classify +## Request: +{ + "metadata": { + "rule_id": "c9scvaiwj51a75kzoh917uwtnw8z4ebl", + "headers": { + "all_request_headers": "value" + }, + "method": "GET", + "path": "/api/v4/projects/100/issues" + }, + "keys": { + "project_id_or_path_encoded": 100 + } +} + +## Response: +{ + "action": "reject", + "reject": { + "http_status": 404 + }, + "cache": { + "refresh": "10 minutes", + "expiry": "10 minutes" + }, + "matched_keys": [ // list of all equivalent keys that should be put in the cache + { "project_id_or_path_encoded": 100 }, + ] +} +``` + +### Configuration + +The Routing Service will use the configuration similar to this: + +```toml +[[cells]] +name=cell_1 +url=https://cell1.gitlab.com +key=ABC123 +classify_weight=100 + +[[cells]] +name=cell_2 +url=https://cell2.gitlab.com +key=CDE123 +classify_weight=1 + +[cache.memory.classify] +refresh_time=10 minutes +expiry_time=1 hour + +[cache.external.classify] +refresh_time=30 minutes +expiry_time=6 hour +``` + +We assume that this is acceptable to provide a static list of Cells, because: + +1. Static: Cells provisioned are unlikely to be dynamically provisioned and decommissioned. +1. Good enough: We can manage such list even up to 100 Cells. +1. Simple: We don't have to implement robust service discovery in the service, + and we have guarantee that this list is always exhaustive. + +The configuration describes all Cells, URLs, zero-trust keys, and weights, +and how long requests should be cached. The `classify_weight` defines how often +the Cell should receive classification requests versus other Cells. + +## Request flows + +1. There are two Cells. +1. `gitlab-org` is a top-level namespace and lives in `Cell US0` in the `GitLab.com Public` organization. +1. `my-company` is a top-level namespace and lives in `Cell EU0` in the `my-organization` organization. + +### Router configured to perform static routing + +1. The Cell US0 supports all other public-facing projects. +1. The Cells is configured to generate all secrets and session cookies with a prefix like `eu0_` for Cell EU0. + 1. The Personal Access Token is scoped to Organization, and because the Organization is part only of a single Cell, + the PATs generated are prefixed with Cell identifier. + 1. The Session Cookie encodes Organization in-use, and because the Organization is part only of a single Cell, + the session cookie generated is prefixed with Cell identifier. +1. The Cell EU0 allows only private organizations, groups, and projects. +1. The Cell US0 is a target Cell for all requests unless explicitly prefixed. + +Cell US0: + +```json +{ + "rules": [ + { + "id": "tjh147se67wadjzum7onwqiad2b75uft", + "path": { + "prefix": "/" + }, + "action": "proxy", + "priority": 1 + } + ] +} +``` + +Cell EU0: + +```json +{ + "rules": [ + { + "id": "t4mkd5ndsk58si6uwwz7rdavil9m2hpq", + "cookies": { + "_gitlab_session": { + "prefix": "eu0_" + } + }, + "path": { + "prefix": "/" + }, + "action": "proxy", + "priority": 1000 + }, + { + "id": "jcshae4d4dtykt8byd6zw1ecccl5dkts", + "headers": { + "GITLAB_TOKEN": { + "prefix": "eu0_" + } + }, + "path": { + "prefix": "/" + }, + "action": "proxy", + "priority": 1000 + } + ] +} +``` + +#### Navigates to `/my-company/my-project` while logged in into Cell EU0 + +1. Because user switched the Organization to `my-company`, its session cookie is prefixed with `eu0_`. +1. User sends request `/my-company/my-project`, and because the cookie is prefixed with `eu0_` it is directed to Cell EU0. +1. `Cell EU0` returns the correct response. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router: GET /my-company/my-project<br/>_gitlab_session=eu0_uwwz7rdavil9 + router->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/my-company/my-project` while not logged in + +1. User visits `/my-company/my-project`, and because it does not have session cookie, the request is forwarded to `Cell US0`. +1. User signs in. +1. GitLab sees that user default organization is `my-company`, so it assigns session cookie with `eu0_` to indicate that + user is meant to interact with `my-company`. +1. User sends request to `/my-company/my-project` again, now with the session cookie that proxies to `Cell EU0`. +1. `Cell EU0` returns the correct response. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cell_us0 as Cell US0 + participant cell_eu0 as Cell EU0 + user->>router: GET /my-company/my-project + router->>cell_us0: GET /my-company/my-project + cell_us0->>user: HTTP 302 /users/sign_in?redirect=/my-company/my-project + user->>router: GET /users/sign_in?redirect=/my-company/my-project + router->>cell_us0: GET /users/sign_in?redirect=/my-company/my-project + cell_us0-->>user: <h1>Sign in... + user->>router: POST /users/sign_in?redirect=/my-company/my-project + router->>cell_us0: POST /users/sign_in?redirect=/my-company/my-project + cell_us0->>user: HTTP 302 /my-company/my-project<br/>_gitlab_session=eu0_uwwz7rdavil9 + user->>router: GET /my-company/my-project<br/>_gitlab_session=eu0_uwwz7rdavil9 + router->>cell_eu0: GET /my-company/my-project<br/>_gitlab_session=eu0_uwwz7rdavil9 + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/gitlab-org/gitlab` after last step + +User visits `/my-company/my-project`, and because it does not have a session cookie, the request is forwarded to `Cell US0`. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cell_eu0 as Cell EU0 + participant cell_us0 as Cell US0 + user->>router: GET /gitlab-org/gitlab<br/>_gitlab_session=eu0_uwwz7rdavil9 + router->>cell_eu0: GET /gitlab-org/gitlab + cell_eu0->>user: HTTP 404 +``` + +### Router configured to perform dynamic routing based on classification + +The Cells publish route rules that allows to classify the requests. + +Cell US0 and EU0: + +```json +{ + "rules": [ + { + "id": "tjh147se67wadjzum7onwqiad2b75uft", + "path": { + "prefix": "/", + "regex": "^/(?top_level_group)[^/]+(/.*)?$", + }, + "action": "classify", + "classify": { + "keys": ["top_level_group"] + } + }, + { + "id": "jcshae4d4dtykt8byd6zw1ecccl5dkts", + "path": { + "prefix": "/" + }, + "action": "proxy" + } + ] +} +``` + +#### Navigates to `/my-company/my-project` while logged in into Cell EU0 + +1. The `/my-company/my-project/` is visited. +1. Router decodes sharding key `top_level_group=my-company`. +1. Router checks if this sharding key is cached. +1. Because it is not, the classification request is sent to a random Cell to `/classify`. +1. The response of classify is cached. +1. The request is then proxied to Cell returned by classification. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cache as Cache + participant cell_us0 as Cell US0 + participant cell_eu0 as Cell EU0 + user->>router: GET /my-company/my-project + router->>cache: CACHE_GET: top_level_group=my-company + cache->>router: CACHE_NOT_FOUND + router->>cell_us0: POST /api/v4/internal/cells/classify<br/>top_level_group=my-company + cell_us0->>router: CLASSIFY: top_level_group=my-company, cell=cell_eu0 + router->>cache: CACHE_SET: top_level_group=my-company, cell=cell_eu0 + router->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/my-company/my-project` while not logged in + +1. The `/my-company/my-project/` is visited. +1. Router decodes sharding key `top_level_group=my-company`. +1. Router checks if this sharding key is cached. +1. Because it is not, the classification request is sent to a random Cell to `/classify`. +1. The response of `classify` is cached. +1. The request is then proxied to Cell returned by classification. +1. Because project is private, user is redirected to sign in. +1. The sign-in since is defined to be handled by all Cells, so it is proxied to a random Cell. +1. User visits the `/my-company/my-project/` again after logging in. +1. The `top_level_group=my-company` is proxied to the correct Cell. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cache as Cache + participant cell_us0 as Cell US0 + participant cell_eu0 as Cell EU0 + user->>router: GET /my-company/my-project + router->>cache: CACHE_GET: top_level_group=my-company + cache->>router: CACHE_NOT_FOUND + router->>cell_us0: POST /api/v4/internal/cells/classify<br/>top_level_group=my-company + cell_us0->>router: CLASSIFY: top_level_group=my-company, cell=cell_eu0 + router->>cache: CACHE_SET: top_level_group=my-company, cell=cell_eu0 + router->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: HTTP 302 /users/sign_in?redirect=/my-company/my-project + user->>router: GET /users/sign_in?redirect=/my-company/my-project + router->>cell_us0: GET /users/sign_in?redirect=/my-company/my-project + cell_us0-->>user: <h1>Sign in... + user->>router: POST /users/sign_in?redirect=/my-company/my-project + router->>cell_eu0: POST /users/sign_in?redirect=/my-company/my-project + cell_eu0->>user: HTTP 302 /my-company/my-project + user->>router: GET /my-company/my-project + router->>cache: CACHE_GET: top_level_group=my-company + cache->>router: CACHE_FOUND: cell=cell_eu0 + router->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/gitlab-org/gitlab` after last step + +1. Because the `/gitlab-org` is not found in cache, it will be classified and then directed to correct Cell. + +```mermaid +sequenceDiagram + participant user as User + participant router as Router + participant cache as Cache + participant cell_us0 as Cell US0 + participant cell_eu0 as Cell EU0 + user->>router: GET /gitlab-org/gitlab + router->>cache: CACHE_GET: top_level_group=gitlab-org + cache->>router: CACHE_NOT_FOUND + router->>cell_us0: POST /api/v4/internal/cells/classify<br/>top_level_group=gitlab-org + cell_us0->>router: CLASSIFY: top_level_group=gitlab-org, cell=cell_us0 + router->>cache: CACHE_SET: top_level_group=gitlab-org, cell=cell_us0 + router->>cell_us0: GET /gitlab-org/gitlab + cell_us0->>user: <h1>My Project... +``` + +### Performance and reliability considerations + +- It is expected that each Cell can classify all sharding keys. +- Alternatively the classification could be done by Cluster-wide Data Provider + if it would own all data required to classify. +- The published routing rules allow to define static criteria, allowing to make routing decision + only on a secret. As a result, the Routing Service doesn't add any latency + for request processing, and superior resiliency. +- It is expected that there will be penalty when learning new sharding key. However, + it is expected that multi-layer cache should provide a very high cache-hit-ratio, + due to low cardinality of sharding key. The sharding key would effectively be mapped + into resource (organization, group, or project), and there's a finite amount of those. ## Technology @@ -188,7 +769,30 @@ TBD ## Alternatives -TBD +### Buffering requests + +The [Stateless Router using Requests Buffering](proposal-stateless-router-with-buffering-requests.md) +describes an approach where Cell answers with `X-Gitlab-Cell-Redirect` to redirect request to another Cell: + +- This is based on a need to buffer the whole request (headers + body) which is very memory intensive. +- This proposal does not provide an easy way to handle mixed deployment of Cells, where Cells might be running different versions. +- This proposal likely requires caching significantly more information, since it is based on requests, rather than on decoded sharding keys. + +### Learn request + +The [Stateless Router using Routes Learning](proposal-stateless-router-with-routes-learning.md) +describes an approach similar to the one in this document. Except the route rules and classification +is done in a single go in a form of pre-flight check `/api/v4/internal/cells/learn`: + +- This makes the whole routes learning dynamic, and dependent on availability of the Cells. +- This proposal does not provide an easy way to handle mixed deployment of Cells, where Cells might be running different versions. +- This proposal likely requires caching significantly more information, since it is based on requests, rather than on decoded sharding keys. + +## FAQ + +1. How and when will Routing Service compile set of rules? + +To be defined. ## Links diff --git a/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/ci_insights.md b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/ci_insights.md new file mode 100644 index 00000000000..72d82558eb7 --- /dev/null +++ b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/ci_insights.md @@ -0,0 +1,154 @@ +--- +status: proposed +creation-date: "2023-01-25" +authors: [ "@pedropombeiro", "@vshushlin"] +coach: "@grzesiek" +approvers: [ ] +stage: Verify +group: Runner +participating-stages: [] +description: 'CI Insights design' +--- + +# CI Insights + +## Summary + +As part of the Fleet Metrics, we would like to have a section dedicated to CI insights to help users monitor pipelines and summarize findings about pipelines speed, common job failures and more. It would eventually offer actionables to help users optimize and fix issues with their CI/CD. + +## Motivation + +We have a [page for CI/CD Analytics](https://gitlab.com/gitlab-org/gitlab/-/pipelines/charts?chart=pipelines) that contain some very basic analytics on pipelines. Most of this information relates to the **total** number of pipelines over time, which does not give any real value to customers: projects will always see an increase of pipelines number over time, so the total number of pipelines is of little consequence. + + + +Because this page lacks real insights, it makes understanding pipelines slowdowns or failures hard to track and becomes a very manual task. We want to empower users to optimize their workflow in a centralized place to avoid all of the manual labor associated with either querying the API for data and then manually parsing it or navigating the UI through dozens of pages utils the insights or action required can be found. + +As we are going to process large quantities of data relating to a proejct pipelines, there is potential to eventually summarize findings with an AI tool to give insights into job failures, pipeline slowdowns and flaky specs. As AI has become a crucial part of our product roadmap and Verify lacks any promising lead in that area, this page could be the center of this new addition. + +- Deliver a new Pipelines Analysis Dashbord page +- Have excellent data visualization to help digest information quickly +- Flexible querying to let users get the information they want + +- Clear actionables based on information presented in the page +- Show some default information on landing like pipelines duration over time and slowest jobs +- Make the CI/CD Analytics more accessible, liked and remembered (AKA, more page views) + +### Non-Goals + +We do not aim to improve the GitLab project's pipeline speed. This feature could help us achieve this, but it is not a direct objective of this blueprint. + +We also are not aiming to have AI in the first iteration and should instead focus on making as much information available and disgestible as possible. + +## Proposal + +Revamp the [page for CI/CD Analytics](https://gitlab.com/gitlab-org/gitlab/-/pipelines/charts?chart=pipelines) to include more meaningful data so that users can troubleshoot their pipelines with ease. Here is a list of the main improvements: + +### Overall statistics + +The current "overall statistics" will become a one line header in a smaller font to keep this information available, but without taking as much visual space. For the pipelines chart, we will replace it with a stacked bar plot where each stack of a bar represents a status and each bar is a unit (in days, a day, in month a month and in years, a year) so users can keep track of how many pipelines ran in that specific unit of time and what percent of these pipelines ended up in failling or succeeding. + +### Pipeline duration graph + +A new pipeline duration graph that can be customized by type (MR pipelines, pipeline on a specific branch, etc), number of runs and status (success, failed, etc) and will replace the current `Pipeline durations for the last 30 commits` chart. The existing chart checks the latest 30 commits made on the repository with no filtering so the results presented are not very valuable. + +We also add jobs that failed multiple times and jobs that are the slowest in the last x pipelines on master. All of this is to support the effort of allowing users to query their pipelines data to figure out what they need to improve on or what kind of problems they are facing with their CI/CD configuration. + +### Visibility + +Add a link in the `pipelines` page to increase the visibility of this feature. We can add a new option with the `Run pipeline` primary button. + +### Master Broken + +Add a "Is master broken?" quick option that scans the last x pipelines on the main branch and check for failed jobs. All jobs that failed multiple times will be listed in a table with the option to create an incident from that list. + +### Color scheme + +Rethink our current color schemes for data visuliaztion when it comes to pipelines statuses. We currently use the default visualization colors, but they don't actually match with that colors user have grown accustomed to for pipeline/jobs statuses. There is an opportunity here to help user better understand their data through more relevant color schemes and better visualization. + +### Routing + +Change the routing from `pipelines/charts` to `pipelines/analytics` since `charts` is a really restrictive terminology when talking about data visualization. It also doesn't really convey what this page is, which is a way to get information, not just nice charts. Then we can also get rid of the query parameter for the tabs and instead support first-class routing. + +## Design and implementation details + +### New API for aggregated data + +This feature depends on having a new set of data available to us that aggregates jobs and pipelines insights and make them available to the client. + +We'll start by aggregating data from ClickHouse, and probably only for `gitlab.com`, as the MVC. We will aggregate the data on the backend on the fly. So far ClickHouse has been very capable of such things. + +We won't store the aggregated data anywhere (we'll probably have the materialized views in ClickHouse, but nothing more complex). Then if the features get traction, we can explore ways to bring these features to environments without ClickHouse + +This way we can move fast, test our ideas with real users, and get feedback. + +### Feature flag + +To develop this new analytic page, we will gate the new page behind a feature flag `ci_insights`, and conditionally render the old or new analytics page. Potentially, we could even add the flag on the controller to decide which route to render: the new `/analytic` when the flag is one, and the old `/charts` when it isn't. + +### Add analytics on page view + +Make sure that we can get information on how often this page is viewed. If we do not have it, then let's implment some to know how visible this page is. The changes to this section should make the view count go up and we want to track this as a measure of success. + +### Routing + +We are planning to have new routes for the page and some redicts to setup. To read more about the routing proposal, see the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/437556) + +### Pipelines duration graph + +We want a way for user to query data about pipelines with a lot of different criterias. Most notably, query for only pipelines with the scope `finished` or by status `success` or `failed`. There is also the possibility to scope this to a ref, so users could either test for the main branch or maybe even a branch that has introduced a CI/CD change. We want branch comparaison for pipeline speed. + +To get more accurate data, we want to increase the count of pipelines requested. In graphQL, we have a limit of 100 items and we will probably get performance degradations quite quickly. We need to define how we could get more data set for more accurate data visualization. + +### Jobs insights + +Currently, there is no way to query a single job across multiple pipelines and it prevent us from doing a query that would look like this: + +```graphql +query getJob($projectPath: ID!, $jobName: String!){ + project(fullPath:$projectPath){ + job(name: $jobName, last: 100){ + nodes{ + id + duration + } + } + } +} +``` + +There are plans to create a new unified table to log job analytics and it is not yet defined what this API will look like. Without comitting yet to an API definiton, we want so unified way to query information for nalytics that may look rougly like so: + +```ruby +get_jobs(project_id:, job_name: nil, stage: nil, stage_index: nil, *etc) +# > +[{id: 1, duration: 134, status: 'failed'}, *etc] + +get_jobs_statistics(project_id, job_name:, *etc) +# > +[{time_bucket: '2024-01-01:00:00:00', avg_duration: 234, count: 123, statuses_count: {success: 123, failed: 45, cancelled: 45}}] +``` + +### Revamping our charts + +Explore new color scheme and nicer look on our charts. Colaborate with UX to determine whether this is something we had on our mind or not and support any iniative to have nicer, more modern looking charts as our charts are quite forgettable. + +## Alternative Solutions + +### New page + +We could create a brand new page and leave this section as it is. The pro would be that we could perhaps have a more prominent placement in the Navigation under `Build`, while the cons are that we'd have clear overlap with the section. + +### Pipeline analysis per pipeline + +There was an [experiment](https://gitlab.com/gitlab-org/gitlab/-/issues/365902) in the past to add performance insights **per pipeline**. The experiment was removed and deemed not viable. Some of the findings were that: + +- Users did not interact with the page as much as thought and would not click on the button to view insights +- Users who did click on the button did not try to get more insights into a job. +- Users did not leave feedback in the issue. + +This experiment reveals to us mostly that users who go on the pipeline graph page `pipelines/:id` are **not** trying to imrpove the performance of pipelines. Instead, it is most likely that this page is used to debug pipeline failures, which means that they are from the IC/developer persona, not the DevOps engineer trying to improve the workflow. By having this section in a more "broad" area, we expect a much better adoption and more useful actionables. + +### Do nothing + +We could leave this section untouched and not add any new form of analytics. The pro here would be the saved resources and time. The cons are that we currently have no way to help customers improve their CI/CD configurations speed except reading our documentation. This revamped section would also be a great gateway for AI features and help user iteration on their setup. diff --git a/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/img/current_page.png b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/img/current_page.png Binary files differnew file mode 100644 index 00000000000..42b09d37785 --- /dev/null +++ b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/img/current_page.png diff --git a/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/index.md b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/index.md index 104a6ee2136..016db5f5766 100644 --- a/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/index.md +++ b/doc/architecture/blueprints/ci_builds_runner_fleet_metrics/index.md @@ -61,6 +61,10 @@ The following customer problems should be solved when addressing this question. #### Which runners have failures in the past hour? +## CI Insights + +CI Insights is a page that would mostly expose data on pipelines and jobs duration, with a multitude of different filters, search and dynamic graphs. To read more on this, see [this related sub-section](ci_insights.md). + ## Implementation The current implementation plan is based on a diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 9a225c9cd97..78d9401d5a5 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -232,7 +232,7 @@ The version of the component can be (in order of highest priority first): 1. A commit SHA - For example: `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` 1. A tag - For example: `gitlab.com/gitlab-org/dast@1.0` -1. A special moving target version that points to the most recent released tag - For example: `gitlab.com/gitlab-org/dast@~latest` +1. A special moving target version that points to the most recent published release - For example: `gitlab.com/gitlab-org/dast@~latest` 1. A branch name - For example: `gitlab.com/gitlab-org/dast@master` If a tag and branch exist with the same name, the tag takes precedence over the branch. @@ -244,6 +244,30 @@ As we want to be able to reference any revisions (even those not released), a co When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches the commit SHA of the current pipeline context. +#### The `~latest` version + +The use of `~latest` version qualifier is restricted to only those releases that are published in the Catalog. + +We debated whether `~latest` should be supported for projects that are not marked as catalog resources. + +There are various reasons for this decision: + +1. Versions could be unlisted from the Catalog and `~latest` need to reflect that. +1. The Catalog will support private resources. There are currently no valid requirements for component projects to + have releases but not being published in the Catalog. +1. In the future we will be separating the process of releasing and publishing a release, allowing users to choose + what release is published in the catalog. +1. We could enforce better the use of semantic versioning when publishing a release, rejecting it if not following + the standard. We won't be able to enforce semantic versioning on the release model because it won't be backwards + compatible. +1. Latest version will likely be denormalized for a catalog resource data structure for more performant queries both + when displaying the Catalog resources and when fetching a component version. + +In all the points above, if we were supporting `~latest` for both catalog resources and unpublished component projects we could +introduce discrepancies and surprising behaviors. Starting with a stricter approach, like supporting +only published versions, we have the freedom to expand this in the future to support unpublished component projects based on +user demand. + ### Note about future resource types In the future, to support multiple types of resources in the Catalog we could diff --git a/doc/architecture/blueprints/cloud_connector/index.md b/doc/architecture/blueprints/cloud_connector/index.md index 9aef8bc7a98..50e233a6089 100644 --- a/doc/architecture/blueprints/cloud_connector/index.md +++ b/doc/architecture/blueprints/cloud_connector/index.md @@ -170,16 +170,16 @@ It will have the following responsibilities: We suggest to use one of the following language stacks: 1. **Go.** There is substantial organizational knowledge in writing and running -Go systems at GitLab, and it is a great systems language that gives us efficient ways to handle requests where -they merely need to be forwarded (request proxying) and a powerful concurrency mechanism through goroutines. This makes the -service easier to scale and cheaper to run than Ruby or Python, which scale largely at the process level due to their use -of Global Interpreter Locks, and use inefficient memory models especially as regards byte stream handling and manipulation. -A drawback of Go is that resource requirements such as memory use are less predictable because Go is a garbage collected language. + Go systems at GitLab, and it is a great systems language that gives us efficient ways to handle requests where + they merely need to be forwarded (request proxying) and a powerful concurrency mechanism through goroutines. This makes the + service easier to scale and cheaper to run than Ruby or Python, which scale largely at the process level due to their use + of Global Interpreter Locks, and use inefficient memory models especially as regards byte stream handling and manipulation. + A drawback of Go is that resource requirements such as memory use are less predictable because Go is a garbage collected language. 1. **Rust.** We are starting to build up knowledge in Rust at GitLab. Like Go, it is a great systems language that is -also starting to see wider adoption in the Ruby ecosystem to write CRuby extensions. A major benefit is more predictable -resource consumption because it is not garbage collected and allows for finer control of memory use. -It is also very fast; we found that the Rust implementation for `prometheus-client-mmap` outperformed the original -extension written in C. + also starting to see wider adoption in the Ruby ecosystem to write CRuby extensions. A major benefit is more predictable + resource consumption because it is not garbage collected and allows for finer control of memory use. + It is also very fast; we found that the Rust implementation for `prometheus-client-mmap` outperformed the original + extension written in C. ## Alternative solutions diff --git a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md index f3335a0935e..7f451b4f92b 100644 --- a/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md +++ b/doc/architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md @@ -43,14 +43,14 @@ configurations, especially the value of the concurrency limit, are static. There are some drawbacks to this: - It's tedious to maintain a sane value for the concurrency limit. Looking at -this [production configuration](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/db11ef95859e42d656bb116c817402635e946a32/roles/gprd-base-stor-gitaly-common.json), -each limit is heavily calibrated based on clues from different sources. When the -overall scene changes, we need to tweak them again. + this [production configuration](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/db11ef95859e42d656bb116c817402635e946a32/roles/gprd-base-stor-gitaly-common.json), + each limit is heavily calibrated based on clues from different sources. When the + overall scene changes, we need to tweak them again. - Static limits are not good for all usage patterns. It's not feasible to pick a -fit-them-all value. If the limit is too low, big users will be affected. If the -value is too loose, the protection effect is lost. + fit-them-all value. If the limit is too low, big users will be affected. If the + value is too loose, the protection effect is lost. - A request may be rejected even though the server is idle as the rate is not -necessarily an indicator of the load induced on the server. + necessarily an indicator of the load induced on the server. To overcome all of those drawbacks while keeping the benefits of concurrency limiting, one promising solution is to make the concurrency limit adaptive to @@ -78,18 +78,18 @@ occurs. There are various criteria for determining whether Gitaly is in trouble. In this proposal, we focus on two things: - Lack of resources, particularly memory and CPU, which are essential for -handling Git processes. + handling Git processes. - Serious latency degradation. The proposed solution is heavily inspired by many materials about this subject shared by folks from other companies in the industry, especially the following: - TCP Congestion Control ([RFC-2581](https://www.rfc-editor.org/rfc/rfc2581), [RFC-5681](https://www.rfc-editor.org/rfc/rfc5681), -[RFC-9293](https://www.rfc-editor.org/rfc/rfc9293.html#name-tcp-congestion-control), [Computer Networks: A Systems Approach](https://book.systemsapproach.org/congestion/tcpcc.html)). + [RFC-9293](https://www.rfc-editor.org/rfc/rfc9293.html#name-tcp-congestion-control), [Computer Networks: A Systems Approach](https://book.systemsapproach.org/congestion/tcpcc.html)). - Netflix adaptive concurrency limit ([blog post](https://tech.olx.com/load-shedding-with-nginx-using-adaptive-concurrency-control-part-1-e59c7da6a6df) -and [implementation](https://github.com/Netflix/concurrency-limits)) + and [implementation](https://github.com/Netflix/concurrency-limits)) - Envoy Adaptive Concurrency -([doc](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/adaptive_concurrency_filter#config-http-filters-adaptive-concurrency)) + ([doc](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/adaptive_concurrency_filter#config-http-filters-adaptive-concurrency)) We cannot blindly apply a solution without careful consideration and expect it to function flawlessly. The suggested approach considers Gitaly's specific @@ -116,12 +116,11 @@ process functioning but quickly reducing it when an issue occurs. During initialization, we configure the following parameters: - `initialLimit`: Concurrency limit to start with. This value is essentially -equal to the current static concurrency limit. + equal to the current static concurrency limit. - `maxLimit`: Maximum concurrency limit. - `minLimit`: Minimum concurrency limit so that the process is considered as -functioning. If it's equal to 0, it rejects all upcoming requests. -- `backoffFactor`: how fast the limit decreases when a backoff event occurs (`0 -< backoff < 1`, default to `0.75`) + functioning. If it's equal to 0, it rejects all upcoming requests. +- `backoffFactor`: how fast the limit decreases when a backoff event occurs (`0 < backoff < 1`, default to `0.75`) When the Gitaly process starts, it sets `limit = initialLimit`, in which `limit` is the maximum in-flight requests allowed at a time. @@ -130,9 +129,9 @@ Periodically, maybe once per 15 seconds, the value of the `limit` is re-calibrated: - `limit = limit + 1` if there is no backoff event since the last -calibration. The new limit cannot exceed `maxLimit`. + calibration. The new limit cannot exceed `maxLimit`. - `limit = limit * backoffFactor` otherwise. The new limit cannot be lower than -`minLimit`. + `minLimit`. When a process can no longer handle requests or will not be able to handle them soon, it is referred to as a back-off event. Ideally, we would love to see the @@ -151,16 +150,16 @@ The concurrency limit restricts the total number of in-flight requests (IFR) at a time. - When `IFR < limit`, Gitaly handles new requests without waiting. After an -increment, Gitaly immediately handles the subsequent request in the queue, if -any. + increment, Gitaly immediately handles the subsequent request in the queue, if + any. - When `IFR = limit`, it means the limit is reached. Subsequent requests are -queued, waiting for their turn. If the queue length reaches a configured limit, -Gitaly rejects new requests immediately. When a request stays in the queue long -enough, it is also automatically dropped by Gitaly. + queued, waiting for their turn. If the queue length reaches a configured limit, + Gitaly rejects new requests immediately. When a request stays in the queue long + enough, it is also automatically dropped by Gitaly. - When `IRF > limit`, it's appropriately a consequence of backoff events. It -means Gitaly handles more requests than the newly appointed limits. In addition -to queueing upcoming requests similarly to the above case, Gitaly may start -load-shedding in-flight requests if this situation is not resolved long enough. + means Gitaly handles more requests than the newly appointed limits. In addition + to queueing upcoming requests similarly to the above case, Gitaly may start + load-shedding in-flight requests if this situation is not resolved long enough. At several points in time we have discussed whether we want to change queueing semantics. Right now we admit queued processes from the head of the queue @@ -181,16 +180,16 @@ Each system has its own set of signals, and in the case of Gitaly, there are two aspects to consider: - Lack of resources, particularly memory and CPU, which are essential for -handling Git processes like `git-pack-objects(1)`. When these resources are limited -or depleted, it doesn't make sense for Gitaly to accept more requests. Doing so -would worsen the saturation, and Gitaly addresses this issue by applying cgroups -extensively. The following section outlines how accounting can be carried out -using cgroup. + handling Git processes like `git-pack-objects(1)`. When these resources are limited + or depleted, it doesn't make sense for Gitaly to accept more requests. Doing so + would worsen the saturation, and Gitaly addresses this issue by applying cgroups + extensively. The following section outlines how accounting can be carried out + using cgroup. - Serious latency degradation. Gitaly offers various RPCs for different purposes -besides serving Git data that is hard to reason about latencies. A significant -overall latency decline is an indication that Gitaly should not accept more -requests. Another section below describes how to assert latency degradation -reasonably. + besides serving Git data that is hard to reason about latencies. A significant + overall latency decline is an indication that Gitaly should not accept more + requests. Another section below describes how to assert latency degradation + reasonably. Apart from the above signals, we can consider adding more signals in the future to make the system smarter. Some examples are Go garbage collector statistics, diff --git a/doc/architecture/blueprints/gitlab_housekeeper/index.md b/doc/architecture/blueprints/gitlab_housekeeper/index.md new file mode 100644 index 00000000000..fcb5590772d --- /dev/null +++ b/doc/architecture/blueprints/gitlab_housekeeper/index.md @@ -0,0 +1,133 @@ +--- +status: implemented +creation-date: "2023-10-18" +authors: [ "@DylanGriffith" ] +coach: +approvers: [ "@rymai", "@tigerwnz" ] +owning-stage: "~devops::tenant scale" +participating-stages: [] +--- + +<!-- vale gitlab.FutureTense = NO --> + +# GitLab Housekeeper - automating merge requests + +## Summary + +This blueprint documents the philosophy behind the +["GitLab Housekeeper" gem](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper) +which was introduced in +<https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139492> and has already +been used to create many merge requests. + +The tool should be used to save developers from mundane repetitive tasks that +can be automated. The tool is scoped to any task where a developer needs to +create a straightforward merge request and is known ahead of time. + +This tool should be useful for at least the following kinds of mundane MRs +we create: + +1. Remove a feature flag after X date +1. Remove an unused index where the unused index is identified by some + automation +1. Remove an `ignore_column` after X date (part of renaming/removing columns + multi-step procedure) +1. Populate sharding keys for organizations/cells on tables that are missing a + sharding key + +## Motivation + +We've observed there are many cases where developers are doing a lot of +manual work for tasks that are entirely predictable and automatable. Often +these manual tasks are done after waiting some known period of time. As such we +usually create an issue and set the future milestone. Then in the future the +developer remembers to followup on that issue and opens an MR to make the +manual change. + +The biggest examples we've seen lately are: + +1. Feature flag removal: <https://gitlab.com/groups/gitlab-org/-/epics/5325>. We + have many opportunities for automation with feature flags but this blueprint + focuses on removing the feature flag after it's fully rolled out. A step + that is often forgotten leading to growing technical debt. +1. Removing duplicated or unused indexes in Postgres: + <https://gitlab.com/gitlab-org/gitlab/-/issues/385701>. For now we're + developing automation that creates issues and assigns them to groups to + follow up and manually open MRs to remove them. This blueprint would take it + a step further and the automation would just create the MRs to remove them + once we have identified them. +1. Removing out of date `ignore_column` references: + <https://docs.gitlab.com/ee/development/database/avoiding_downtime_in_migrations.html#removing-the-ignore-rule-release-m2> + . For now we leave a note in our code telling us the date it needs to be + removed and often create an issue as a reminder. This blueprint proposes + that automation just reads this note and opens the MR to remove it after the + date. +1. Adding and backfilling sharding keys for organizations for Cells: + <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133796>. The cells + architecture depends on all tables having a sharding key that is attributed + to an organization. We will need to backfill this for ~300 tables. Much of + this will be repetitive and mundane work that we can automate provided that + groups just identify what the name of the sharding key should be and how we + will backfill it. As such we can automate the creation of MRs that guess the + sharding key and owning groups can check and correct those MRs. Then we can + automate the MR creation for adding the columns and backfilling the data. + Some kind of automation like this will be necessary to finish this work in a + reasonable timeframe. + +### Goals + +1. Identify the common tasks that take development time and automate them. +1. Focus on MR creation rather than issue creation as MRs are the results we + want and issues are a process for reminding us to get those results. +1. Improve developer job satisfication by knowing that automation is doing the + busy work while we get to do the challenging and creative work. +1. Developers should be encouraged to contribute to the automation framework + when they see a pattern rather than documenting the manual work for future + developers to do it again. +1. Automation MRs should be very easily identified and reviewed and merged much + more quickly than other MRs. If our automation MRs cause too much effort for + reviewers we maybe will outweigh the benefits. This might mean that some + automations get disabled when they are just noisy. + +## Solution + +The +[GitLab Housekeeper gem](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper) +should be used to automate creation of mundane merge requests. + +Using this tool reflects our +[bias for action](https://handbook.gitlab.com/handbook/values/#bias-for-action) +subvalue. As such, developers should preference contributing a new +[keep](https://gitlab.com/gitlab-org/gitlab/-/tree/master/keeps) over the following: + +1. Documenting a process that involves creating several merge requests over a + period of time +1. Setting up periodic reminders for developers (in Slack or issues) to create + some merge request + +The keeps may sometimes take more work to implement than documentation or +reminders so judgement should be used to assess the likely time savings from +using automation. The `gitlab-housekeeper` gem will evolve over time with many +utilities that make it simpler to contribute new keeps and it is expected that +over time the cost to implementing a keep should be small enough that we will +mostly prefer this whenever developers need to do a repeatable task more than a +few times. + +## Design and implementation details + +The key details for this architecture is: + +1. The design of this tool is like a combination of `rubocop -a` and Renovate + bot. It extends on `rubocop -a` to understand when things need to be removed + after certain deadlines as well as creating a steady stream of manageable + merge requests for the reviewer rather than leaving those decisions to the + developer. Like the renovate bot it attempts to create MRs periodically and + assign them to the right people to review. +1. The keeps live in the GitLab repo which means that there are no + dependencies to update and the keeps can use code inside the + GitLab codebase. +1. The script can be run locally by a developer or can be run periodically + in some automated way. +1. The keeps are able to use any data sources (eg. local code, Prometheus, + Postgres database archive, logs) needed to determine whether and how to make + the change. diff --git a/doc/architecture/blueprints/gitlab_steps/data.drawio.png b/doc/architecture/blueprints/gitlab_steps/data.drawio.png Binary files differindex 59436093fb7..5ffe2964134 100644 --- a/doc/architecture/blueprints/gitlab_steps/data.drawio.png +++ b/doc/architecture/blueprints/gitlab_steps/data.drawio.png diff --git a/doc/architecture/blueprints/gitlab_steps/step-runner-sequence.drawio.png b/doc/architecture/blueprints/gitlab_steps/step-runner-sequence.drawio.png Binary files differindex 9f6a6dcad9f..57029733b3c 100644 --- a/doc/architecture/blueprints/gitlab_steps/step-runner-sequence.drawio.png +++ b/doc/architecture/blueprints/gitlab_steps/step-runner-sequence.drawio.png diff --git a/doc/architecture/blueprints/runner_admission_controller/index.md b/doc/architecture/blueprints/runner_admission_controller/index.md index 21dc1d53303..0a62b271901 100644 --- a/doc/architecture/blueprints/runner_admission_controller/index.md +++ b/doc/architecture/blueprints/runner_admission_controller/index.md @@ -140,7 +140,7 @@ Each runner has a tag identifier unique to that runner, e.g. `DiscoveryOne`, `tu 1. The `preparing` state will wait for a response from the webhook or until timeout. 1. The UI should be updated with the current status of the job prerequisites and admission 1. For jobs where the webhook times out (1 hour) their status should be set as though the admission was denied with a timeout reasoning. This should -be rare in typical circumstances. + be rare in typical circumstances. 1. Jobs with denied admission can be retried. Retried jobs will be resent to the admission controller without tag mutations or runner filtering reset. 1. [`allow_failure`](../../../ci/yaml/index.md#allow_failure) should be updated to support jobs that fail on denied admissions, for example: diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index f2e9d624d20..c667a460f5c 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -284,11 +284,11 @@ not an issue per-se. New records are created in 2 situations: -- when the runner calls the `POST /api/v4/runners/verify` endpoint as part of the -`gitlab-runner register` command, if the specified runner token is prefixed with `glrt-`. -This allows the frontend to determine whether the user has successfully completed the registration and take an -appropriate action; -- when GitLab is pinged for new jobs and a record matching the `token`+`system_id` does not already exist. +- When the runner calls the `POST /api/v4/runners/verify` endpoint as part of the + `gitlab-runner register` command, if the specified runner token is prefixed with `glrt-`. + This allows the frontend to determine whether the user has successfully completed the registration and take an + appropriate action; +- When GitLab is pinged for new jobs and a record matching the `token`+`system_id` does not already exist. Due to the time-decaying nature of the `ci_runner_machines` records, they are automatically cleaned after 7 days after the last contact from the respective runner. diff --git a/doc/architecture/blueprints/runway/index.md b/doc/architecture/blueprints/runway/index.md index becb7914feb..af7f466cdc9 100644 --- a/doc/architecture/blueprints/runway/index.md +++ b/doc/architecture/blueprints/runway/index.md @@ -169,7 +169,7 @@ In order for runway to function, there are two JSON/YAML documents in use. They 1. The Runway Inventory Model. This covers what service projects are currently onboarded into Runway. It's located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/blob/main/inventory.json?ref_type=heads). The schema used to validate the docuemnt is located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/blob/main/schemas/service-inventory/v1.0.0-beta/inventory.schema.json?ref_type=heads). There is no backwards compatibility guarenteed to changes to this document schema. This is because it's only used internally by the Runway team, and there is only a single document actually being used by Runway to provision/deprovision Runway services. 1. The runway Service Model. This is used by Runway users to pass through configuration needed to Runway in order to deploy their service. It's located inside their Service project, at `.runway/runway.yml`. [An example is here](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/.runway/runway.yml?ref_type=heads). The schema used to validate the document is located [here](https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/blob/main/schemas/service-manifest/v1.0.0-beta/manifest.schema.json?ref_type=heads). We aim to continue to make improvements and changes to the model, but all changes to the model within the same `kind/apiVersion` must be backwards compatible. In order to -make breaking changes, a new `apiVersion` of the schema will be released. The overall goal is to copy the [Kubernetes model for making API changes](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md). + make breaking changes, a new `apiVersion` of the schema will be released. The overall goal is to copy the [Kubernetes model for making API changes](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md). There are also [GitLab CI templates](https://gitlab.com/gitlab-com/gl-infra/platform/runway/ci-tasks) used by Runway users in order to automate deployments via Runway through GitLab CI. Users will be encouraged to use tools such as [Renovate bot](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/renovate-bot.md) in order to make sure the CI templates and version of Runway they are using is up to date. The Runway team will support all released versions of Runway, with the exception of when a security issue is identified. When this happens, Runway users will be expected to update to a version of Runway that contains a fix for the issue as soon as possible (once notification is received). diff --git a/doc/architecture/blueprints/secret_detection/index.md b/doc/architecture/blueprints/secret_detection/index.md index fb77fffee40..3e9421539e6 100644 --- a/doc/architecture/blueprints/secret_detection/index.md +++ b/doc/architecture/blueprints/secret_detection/index.md @@ -29,19 +29,18 @@ job logs, and project management features such as issues, epics, and MRs. - Support platform-wide detection of tokens to avoid secret leaks - Prevent exposure by rejecting detected secrets - Provide scalable means of detection without harming end user experience +- Unified list of token patterns and masking See [target types](#target-types) for scan target priorities. ### Non-Goals -Initial proposal is limited to detection and alerting across platform, with rejection only -during [preceive Git interactions and browser-based detection](#iterations). +Phase1 is limited to detection and alerting across platform, with rejection only +during [prereceive Git interactions and browser-based detection](#iterations). Secret revocation and rotation is also beyond the scope of this new capability. -Scanned object types beyond the scope of this MVC include: - -See [target types](#target-types) for scan target priorities. +Scanned object types beyond the scope of this MVC are included within [target types](#target-types). #### Management UI @@ -67,7 +66,7 @@ Target object types refer to the scanning targets prioritized for detection of l In order of priority this includes: -1. non-binary Git blobs +1. non-binary Git blobs under 1 megabyte 1. job logs 1. issuable creation (issues, MRs, epics) 1. issuable updates (issues, MRs, epics) @@ -75,30 +74,60 @@ In order of priority this includes: Targets out of scope for the initial phases include: +- non-binary Git blobs over 1 megabyte +- binary Git blobs - Media types (JPEG, PDF, ...) - Snippets - Wikis - Container images +- External media (Youtube platform videos) ### Token types -The existing Secret Detection configuration covers ~100 rules across a variety +The existing Secret Detection configuration covers 100+ rules across a variety of platforms. To reduce total cost of execution and likelihood of false positives -the dedicated service targets only well-defined tokens. A well-defined token is -defined as a token with a precise definition, most often a fixed substring prefix or -suffix and fixed length. +the dedicated service targets only well-defined, low-FP tokens. Token types to identify in order of importance: 1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens) 1. Verified Partner tokens (including AWS) -1. Remainder tokens currently included in Secret Detection CI configuration +1. Well-defined low-FP third party tokens +1. Remainder tokens currently included in Secret Detection analyzer configuration -## Proposal +A well-defined token is a token with a precise definition, most often a fixed +substring prefix (or suffix) and fixed length. -### Decisions +For GitLab and partner tokens, we have good domain understanding of our own tokens +and by collaborating with partners verified the accuracy of their provided patterns. -- [001: Use Ruby Push Check approach within monolith](decisions/001_use_ruby_push_check_approach_within_monolith.md) +An observed low-FP token relies on user reports and dismissal reports. With delivery of +[this data issue](https://gitlab.com/gitlab-data/product-analytics/-/issues/1225) +we will have aggregates on FP-rates but primarily this is user-reported data, at present. + +In order to minimize false positives, there are no plans to introduce or alert on high-entropy, +arbitrary strings; i.e. patterns such as `3lsjkw3a22`. + +#### Uniformity of rule configuration + +Rule pattern configuration should remain centralized in the `secrets` analyzer's packaged `gitleaks.toml` +configuration, vendored to the monolith for Phase 1, and checksum-checked to ensure it matches the +specific release version to avoid drift. Each token can be filtered by `tags` to form both high-confidence +and blocking groupings. For example: + +```ruby +prereceive_blocking_rules = toml.load_file('gitleaks.toml')['rules'].select do |r| + r.tags.include?('gitlab_blocking_p1') && + r.tags.include?('gitlab_blocking') +end +``` + +### Auditability + +A critical aspect of both secret detection and [suppression](#detection-suppression) is administrative visibility. +With each phase we must include audit capabilities (events or logging) to enable event discovery. + +## Proposal The first iteration of the experimental capability will feature a blocking pre-receive hook implemented in the Rails application. This iteration @@ -119,6 +148,10 @@ This service must be: Platform-wide secret detection should be enabled by-default on GitLab SaaS as well as self-managed instances. +### Decisions + +- [001: Use Ruby Push Check approach within monolith](decisions/001_use_ruby_push_check_approach_within_monolith.md) + ## Challenges - Secure authentication to GitLab.com infrastructure @@ -136,17 +169,15 @@ In expansion phases we must explore chunking or alternative strategies like the ## Design and implementation details +The detection capability relies on a multiphase rollout, from an experimental component implemented directly in the monolith to a standalone service capable of scanning text blobs generically. + The implementation of the secret scanning service is highly dependent on the outcomes of our benchmarking and capacity planning against both GitLab.com and our [Reference Architectures](../../../administration/reference_architectures/index.md). As the scanning capability must be an on-by-default component of both our SaaS and self-managed -instances [the PoC](#iterations), the deployment characteristics must be considered to determine whether -this is a standalone component or executed as a subprocess of the existing Sidekiq worker fleet -(similar to the implementation of our Elasticsearch indexing service). - -Similarly, the scan target volume will require a robust and scalable enqueueing system to limit resource consumption. - -The detection capability relies on a multiphase rollout, from an experimental component implemented directly in the monolith to a standalone service capable of scanning text blobs generically. +instances, [each iteration's](#iterations) deployment characteristic defines whether +the service will act as a standalone component, or executed as a subprocess of the Rails architecture +(as mirrors the implementation of our Elasticsearch indexing service). See [technical discovery](https://gitlab.com/gitlab-org/gitlab/-/issues/376716) for further background exploration. @@ -154,14 +185,35 @@ for further background exploration. See [this thread](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105142#note_1194863310) for past discussion around scaling approaches. +### Detection engine + +Our current secret detection offering uses [Gitleaks](https://github.com/zricethezav/gitleaks/) +for all secret scanning in pipeline contexts. By using its `--no-git` configuration +we can scan arbitrary text blobs outside of a repository context and continue to +use it for non-pipeline scanning. + +Changes to the detection engine are out of scope until benchmarking unveils performance concerns. + +For the long-term direction of GitLab Secret Detection, the scope is greater than that of the Gitleaks tool. As such, we should consider feature encapsulation to limit the Gitleaks domain to the relevant build context only. + +In the case of pre-receive detection, we rely on a combination of keyword/substring matches +for pre-filtering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks. + +Notable alternatives include high-performance regex engines such as [Hyperscan](https://github.com/intel/hyperscan) or it's portable fork [Vectorscan](https://github.com/VectorCamp/vectorscan). +These systems may be worth exploring in the future if our performance characteristics show a need to grow beyond the existing stack, however the team's velocity in building an independently scalable and generic scanning engine was prioritized, see [ADR 001](decisions/001_use_ruby_push_check_approach_within_monolith.md) for more on the implementation language considerations. + +### Organization-level Controls + +Configuration and workflows should be oriented around [Organizations](../organization/index.md). Detection controls and governance patterns should support configuration across multiple projects and groups in a uniform way that emphasizes shared allowlists, organization-wide policies (i.e. disablement of push option bypass), and auditability. + +Each phase documents the paradigm used as we iterate from Instance-level to Organization-level controls. + ### Phase 1 - Ruby pushcheck pre-receive integration The critical paths as outlined under [goals above](#goals) cover two major object types: Git text blobs (corresponding to push events) and arbitrary text blobs. In Phase 1, we focus entirely on Git text blobs. -This phase will be considered "Experimental" with limited availability for customer opt-in, through instance level application settings. - The detection flow for push events relies on subscribing to the PreReceive hook to scan commit data using the [PushCheck interface](https://gitlab.com/gitlab-org/gitlab/blob/3f1653f5706cd0e7bbd60ed7155010c0a32c681d/lib/gitlab/checks/push_check.rb). This `SecretScanningService` service fetches the specified blob contents from Gitaly, scans @@ -170,6 +222,10 @@ See [Push event detection flow](#push-event-detection-flow) for sequence. In the case of a push detection, the commit is rejected inline and error returned to the end user. +#### Configuration + +This phase will be considered "Experimental" with limited availability for customer opt-in, through instance level application settings. + #### High-Level Architecture The Phase 1 architecture involves no additional components and is entirely encapsulated in the Rails application server. This provides a rapid deployment with tight integration within auth boundaries and no distribution coordination. @@ -204,7 +260,7 @@ sidekiq .[#ff8dd1]----> postgres @enduml ``` -#### Push event detection flow +#### Push Event Detection Flow ```mermaid sequenceDiagram @@ -237,7 +293,7 @@ sequenceDiagram The critical paths as outlined under [goals above](#goals) cover two major object types: Git text blobs (corresponding to push events) and arbitrary text blobs. In Phase 2, -we focus entirely on Git text blobs. +we continue to focus on Git text blobs. This phase emphasizes scaling the service outside of the monolith for general availability and to allow an on-by-default behavior. The architecture is adapted to provide an isolated and independently @@ -245,13 +301,17 @@ scalable service outside of the Rails monolith. In the case of a push detection, the commit is rejected inline and error returned to the end user. +#### Configuration + +This phase will be considered "Generally Available" and on-by-default, with disablement configuration through organization-level settings. + #### High-Level Architecture The Phase 2 architecture involves extracting the secret detection logic into a standalone service which communicates directly with both the Rails application and Gitaly. This provides a means to scale the secret detection nodes independently, and reduce resource usage overhead on the rails application. -Scans still runs synchronously as a (potentially) blocking pre-receive transaction. +Scans still runs synchronously as a (potentially) blocking pre-receive transaction. The blob size remains limited to 1MB. Note that the node count is purely illustrative, but serves to emphasize the independent scaling requirements for the scanning service. @@ -308,7 +368,7 @@ consul .[#e76a9b]-> prsd_cluster @enduml ``` -#### Push event detection flow +#### Push Event Detection Flow ```mermaid sequenceDiagram @@ -345,7 +405,7 @@ sequenceDiagram Rails->>User: accepted ``` -### Phase 3 - Expansion beyond pre- +### Phase 3 - Expansion beyond pre-receive service The detection flow for arbitrary text blobs, such as issue comments, relies on subscribing to `Notes::PostProcessService` (or equivalent service) to enqueue @@ -364,11 +424,15 @@ In any other case of detection, the Rails application manually creates a vulnera using the `Vulnerabilities::ManuallyCreateService` to surface the finding in the existing Vulnerability Management UI. -#### Architecture +#### Configuration + +This phase will be considered "Generally Available" and on-by-default, with disablement configuration through organization-level settings. + +#### High-Level Architecture There is no change to the architecture defined in Phase 2, however the individual load requirements may require scaling up the node counts for the detection service. -#### Detection flow +#### Push Event Detection Flow There is no change to the push event detection flow defined in Phase 2, however the added capability to scan arbitary text blobs directly from Rails allows us to emulate a pre-receive behavior for issuable creations, @@ -403,52 +467,42 @@ sequenceDiagram Rails->>User: rejected: secret found ``` -### Target types +### Future Phases -Target object types refer to the scanning targets prioritized for detection of leaked secrets. +These are key items for delivering a feature-complete always-on experience but have not have yet been prioritized into phases. -In order of priority this includes: +### Large blob sizes (1mb+) -1. non-binary Git blobs -1. job logs -1. issuable creation (issues, MRs, epics) -1. issuable updates (issues, MRs, epics) -1. issuable comments (issues, MRs, epics) +Current phases do not include expansions of blob sizes beyond 1mb. While the main limitation was chosen [to conform to RPC transfer limits for future iterations](#transfer-optimizations-for-large-git-data-blobs) we should expand to supporting additional blob sizes. This can be achieved in two ways: -Targets out of scope for the initial phases include: +1. *Post-receive processing* -- Media types (JPEG, PDF, ...) -- Snippets -- Wikis -- Container images + Accept blobs in a non-blocking fashion, process scanning as background job and alert passively on detection of a given secret. -### Token types +1. *Improvements to scanning logic batching* -The existing Secret Detection configuration covers ~100 rules across a variety -of platforms. To reduce total cost of execution and likelihood of false positives -the dedicated service targets only well-defined tokens. A well-defined token is -defined as a token with a precise definition, most often a fixed substring prefix or -suffix and fixed length. + Maintaining the constraint of 1MB is primarily futureproofing to match an expected transport protocol. This can be mitigated by using separate transport (http, reads from disk, ...) or by slicing blob sizes. -Token types to identify in order of importance: +### Detection Suppression -1. Well-defined GitLab tokens (including Personal Access Tokens and Pipeline Trigger Tokens) -1. Verified Partner tokens (including AWS) -1. Remainder tokens included in Secret Detection CI configuration +Suppression of detection and action on leaked secrets will be supported at several levels. -### Detection engine +1. *Global suppression* - If a secret is highly-likely to be a false token (i.e. `EXAMPLE`) it should be suppressed in workflow contexts where user would be seriously inconvenienced. -Our current secret detection offering uses [Gitleaks](https://github.com/zricethezav/gitleaks/) -for all secret scanning in pipeline contexts. By using its `--no-git` configuration -we can scan arbitrary text blobs outside of a repository context and continue to -use it for non-pipeline scanning. + We should still provide some means of triaging these results, whether via [audit events](#auditability) or as [automatic vulnerability resolution](../../../user/application_security/sast/index.md#automatic-vulnerability-resolution). -In the case of pre-receive detection, we rely on a combination of keyword/substring matches -for pre-filtering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks +1. *Organization suppression* - If a secret matches an organization's allowlist (or was previously flagged and remediated as irrelevant) it should not reoccur. See [Organization-level controls](#organization-level-controls). -Changes to the detection engine are out of scope until benchmarking unveils performance concerns. +1. *Inline suppression* - Inline annotations should be supported in later phases with the Organization-level configuration to ignore annotations. -Notable alternatives include high-performance regex engines such as [Hyperscan](https://github.com/intel/hyperscan) or it's portable fork [Vectorscan](https://github.com/VectorCamp/vectorscan). +### External Token Verification + +As a post-processing step for detection we should explore verification of detected secrets. This requires processors per supported token type in which we can distinguish tokens that are valid leaks from false positives. Similar to our [automatic response to leaked secrets](../../../user/application_security/secret_detection/automatic_response.md), we must externally verify a given token to give a high degree of confidence in our alerting. + +There are two token types: internal and external: + +- Internal tokens are verifiable and revocable as part of `ScanSecurityReportSecretsWorker` worker +- External tokens require external verification, in which [the architecture](../../../user/application_security/secret_detection/automatic_response.md#high-level-architecture) will closely match the [Secret Revocation Service](https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/secret-revocation-service/) ## Iterations @@ -459,14 +513,14 @@ Notable alternatives include high-performance regex engines such as [Hyperscan]( - [Pre-Production Performance Profiling for pre-receive PoCs](https://gitlab.com/gitlab-org/gitlab/-/issues/428499) - Profiling service capabilities - ✓ [Benchmarking regex performance between Ruby and Go approaches](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) - - gRPC commit retrieval from Gitaly - transfer latency, CPU, and memory footprint -- Implementation of secret scanning service MVC (targeting individual commits) +- ✓ Implementation of secret scanning gem integration MVC (targeting individual commits) +- Phase1 - Deployment and monitoring - Capacity planning for addition of service component to Reference Architectures headroom - Security and readiness review -- Deployment and monitoring -- Implementation of secret scanning service MVC (targeting arbitrary text blobs) -- Deployment and monitoring +- Phase2 - Deployment and monitoring +- Implementation of secret scanning service (targeting arbitrary text blobs) +- Phase3 - Deployment and monitoring - High priority domain object rollout (priority `TBD`) - Issuable comments - Issuable bodies diff --git a/doc/architecture/blueprints/secret_manager/index.md b/doc/architecture/blueprints/secret_manager/index.md index ac30f3399d8..3a538f58dde 100644 --- a/doc/architecture/blueprints/secret_manager/index.md +++ b/doc/architecture/blueprints/secret_manager/index.md @@ -114,10 +114,10 @@ the data keys mentioned above. ### Further investigations required 1. Management of identities stored in GCP Key Management. -We need to investigate how we can correlate and de-multiplex GitLab identities into -GCP identities that are used to allow access to cryptographic operations on GCP Key Management. + We need to investigate how we can correlate and de-multiplex GitLab identities into + GCP identities that are used to allow access to cryptographic operations on GCP Key Management. 1. Authentication of clients. Clients to the Secrets Manager could be GitLab Runner or external clients. -For each of these, we need a secure and reliable method to authenticate requests to decrypt a secret. + For each of these, we need a secure and reliable method to authenticate requests to decrypt a secret. 1. Assignment of GCP backed private keys to each identity. ### Availability on SaaS and Self-Managed diff --git a/doc/architecture/blueprints/tailwindcss/index.md b/doc/architecture/blueprints/tailwindcss/index.md new file mode 100644 index 00000000000..0409f802038 --- /dev/null +++ b/doc/architecture/blueprints/tailwindcss/index.md @@ -0,0 +1,172 @@ +--- +status: proposed +creation-date: "2023-12-21" +authors: [ "@peterhegman", "@svedova", "@pgascouvaillancourt" ] +approvers: [ "@samdbeckham" ] +owning-stage: "~devops::manage" +participating-stages: [] +--- + +<!-- Blueprints often contain forward-looking statements --> +<!-- vale gitlab.FutureTense = NO --> + +# Delegating CSS utility classes generation to Tailwind CSS + +## Summary + +Styling elements in GitLab primarily relies on CSS utility classes. Those are classes that +generally define a single CSS property and that can be applied additively to change an element's look. +We have developed our own tooling in the [GitLab UI](https://gitlab.com/gitlab-org/gitlab-ui) project +to generate the utils we need, but our approach has demonstrated a number of flaws that can be +circumvented by delegating that task to the [Tailwind CSS](https://tailwindcss.com/) framework. + +This initiative requires that we deprecate existing utilities so that Tailwind CSS can replace them. + +## Motivation + +In June 2019, we have consolidated our usage of CSS utility classes through [RFC#4](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/4) +which introduced the concept of silent classes, where utilities would be generated from a collection +of manually defined SCSS mixins. + +This has served us well, but came with some caveats: + +- **Increased development overhead:** whenever a new utility is needed, it has to be manually added + to the [GitLab UI](https://gitlab.com/gitlab-org/gitlab-ui) project. One then needs to wait on a + new version of `@gitlab/ui` to be released and installed in the consumer project. +- **Inconsistencies:** Without any tooling in place to check how utilities are named, we have seen + many inconsistencies make their way in the library, making it quite unpredictable. The most striking + example of this was the introduction of desktop-first utilities among a majority of mobile-first + utils, without any way of distinguishing the former from the latter, other than looking at the source. +- **Disconnection between the utilities library and its consumers:** When a utility is added to the + library, it is made available to _any_ project that uses `@gitlab/ui`. As a result, some utils are + included in projects that don't need them. Conversely, if all consumers stop using a given util, + it could potentially be removed to decrease the CSS bundle size, but we have no visibility over this. +- **Limited autocompletion:** Although it's possible to configure autocomplete for the existing + library, it is restricted to the utilities bundle. In contrast, Tailwind CSS autocomplete aligns + with an on-demand approach, ensuring that all utilities are readily available. Additionally, IDE + extensions can enhance understanding by revealing the values applied by a specific utility. + +As part of this architectural change, we are alleviating these issues by dropping our custom built +solution for generating CSS utils, and delegating this task to [Tailwind CSS](https://tailwindcss.com/). + +It is worth noting that this was previously debated in [RFC#107](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/107). +The RFC was well received. The few concerns that were raised were about the CSS utility approach as +a whole, not the way we implemented it. This initiative's purpose _is not_ to question our reliance +on utility classes, but to consolidate its implementation to improve engineers' efficiency when working +with CSS utils. + +### Why Tailwind CSS? + +Here are a few reasons that led us to choosing Tailwind CSS over similar tools: + +- It is a long-standing project that has been battle-tested in many production apps and has a + healthy community around it. +- Tailwind CSS is well maintained and keeps evolving without getting bloated. +- It integrates well in all of our tech stacks + - Ruby on Rails projects can leverage the [`tailwindcss-rails` Gem](https://tailwindcss.com/docs/guides/ruby-on-rails). + - Nuxt apps can setup the [`tailwindcss` module](https://nuxt.com/modules/tailwindcss). + - More generic frontend stacks can use the [`tailwindcss` Node module](https://tailwindcss.com/docs/installation). + +### Goals + +This blueprint's goal is to improve the developer experience (DX) when working with CSS utility classes. +As a result of this initiative, frontend engineers' efficiency should be increased thanks to a much +lower development overhead. + +### Non-Goals + +As stated in the motivations above, this focuses on improving an existing architectural decision, +not on replacing it with a new design. So this therefore: + +- _Is not_ aimed at revisiting the way we write CSS or how we apply styles within our projects. +- _Does not_ focus on user-facing improvements. This change is mostly a developer experience enhancement. + The resulting increase in efficiency could certainly indirectly improve user experience, but that + is not our primary intent. + +## Proposal + +We will be setting up Tailwind CSS in GitLab UI _and_ GitLab. The intent is to have the main +Tailwind CSS configuration in GitLab UI. This step is where we'll be maintaining the Pajamas-compliant +configuration properties (color, spacing scale, etc.). The Tailwind CSS setup in GitLab will inherit from +GitLab UI's setup. The subtlety here is that, in GitLab, we will be scanning both the GitLab codebase +and the `@gitlab/ui` Node module. This will ensure that GitLab UI does not need to expose any CSS +utilities anymore, but the ones it relies on are still generated in GitLab. A similar setup will +need to be introduced in other projects that use CSS utilities and need to upgrade to the Tailwind +CSS-based version. + +### Pros + +- We are removing the cumbersome workflow for adding new utilities. One should be able to use any + utility right away without contributing to another project and waiting through the release cycle. +- We are introducing a predictable library, where the naming is decided upon in the overarching + Tailwind CSS project. As engineers know, naming things is difficult, and it's best that we defer + this to a well-established project. +- Engineers should be able to refer to Tailwind CSS documentation to know what utils are available + and how to use them. No need to read through GitLab UI's source code anymore. +- Because Tailwind CSS generates the required utils by scanning the consumer's codebase, we'll be + sure to only generate the utilities we actually need, keeping CSS bundle sizes under control. This + must be taken with a grain of salt though: Tailwind CSS is extremely flexible and makes it possible + to generate all sorts of utils, sometimes with developer-defined values, which could result in + large utils bundles depending on how we'll adopt Tailwind CSS' features. +- We'll benefit from a robust IDE integration providing auto-completion and previews for the utils + we support. + +### Cons + +- More setup: each project that requires CSS utils would need to have Tailwind CSS set up, + which might be more or less tedious depending on the environment. +- One more dev dependency in each project. +- Inability to use string interpolation to build class names dynamically (Tailwind CSS + needs to see the full names to generate the required classes). +- We'll need a migration: we'll need to ensure usages of the existing CSS utilities library + don't break, which implies a deprecation/migration process. + +## Design and implementation details + +In order to prevent breakages, we are taking an iterative approach to moving away from the current +library. The proposed path here is purposefully rough around the edges. We acknowledge that it's +not a one size fits all solution and that we might need to adjust to some cases along the way. + +Here's the basic process: + +1. Deprecate a collection of utility mixins in GitLab UI. This entails replacing the `gl-` prefix + with `gl-deprecated-` in the mixin's name, and updating all usages in both GitLab UI _and_ GitLab + accordingly. We will typically focus on a single mixins file at a time, though we might want to + deprecate several files at once if they are small enough. Conversely, some files might be too big + to be deprecated in one go and would require several iterations. +1. Enable the corresponding [Tailwind CSS core plugins](https://tailwindcss.com/docs/configuration#core-plugins) so that we can immediately start using the + newer utilities. +1. Migrate deprecated utilities to their Tailwind CSS equivalents. + +```mermaid +flowchart TD + RequiresDeprecation(Is the mixins collection widely used in GitLab?) + DeprecateMixins[Mark the mixins as deprecated with the `gl-deprecated-` prefix] + + HasTailwindEq(Does Tailwind CSS have equivalents?) + EnableCorePlugin["Enable the corresponding Tailwind CSS core plugin(s)"] + WriteCustomUtil[Write a custom Tailwind CSS utility] + + MigrateUtils[Migrate legacy utils to Tailwind CSS] + + RequiresDeprecation -- Yes --> DeprecateMixins + DeprecateMixins --> HasTailwindEq + RequiresDeprecation -- No --> HasTailwindEq + HasTailwindEq -- Yes --> EnableCorePlugin + HasTailwindEq -- No --> WriteCustomUtil + EnableCorePlugin --> MigrateUtils + WriteCustomUtil --> MigrateUtils +``` + +The deprecation step gives us some margin to evaluate each migration without risking breakages in +production. It does have some drawbacks: + +- We might cause merge conflicts for others as we will be touching several areas of the product in + our deprecation MRs. We will make sure to communicate these changes efficiently to not make things + too confusing. We will also use our best judgement to split MRs when we feel like their scope gets + too large. +- Deprecation MRs might require approval from several departments, which is another reason to + be transparent and iterative throughout the process. +- We are purposefully introducing technical debt which we are committed to pay in a reasonable time frame. + We acknowledge that the actual duration of this initiative may be affected by a number of factors (uncovering + edge-cases, DRIs' capacity, department-wide involvement, etc.), but we expect to have it completed in 6-12 months. diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 55f18987490..737b18fb6c2 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -738,5 +738,5 @@ instance. To share the cache between concurrent runners, you can either: - Use the `[runners.docker]` section of the runners' `config.toml` to configure a single mount point on the host that -is mapped to `/cache` in each container, preventing the runner from creating unique volume names. + is mapped to `/cache` in each container, preventing the runner from creating unique volume names. - Use a distributed cache. diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index b80bbbc5c98..82f92cdc938 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -276,7 +276,7 @@ include: To use DAST on the default branch: 1. Set up a new [service](#create-an-ecs-service). This service will be used to deploy a temporary -DAST environment. + DAST environment. 1. Use the `CI_AWS_ECS_SERVICE` variable to set the name. 1. Set the scope to the `dast-default` environment. 1. Add the following to your `.gitlab-ci.yml` file: diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index d3bd8e187ba..b3fcaabfdde 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -153,7 +153,7 @@ you should verify: - For the `gitlab-group/gitlab-project` project and `main` branch it would be: `project_path:gitlab-group/gitlab-project:ref_type:branch:ref:main`. - The correct values of `mygroup` and `myproject` can be retrieved by checking the URL - when accessing your GitLab project or by selecting the **Clone** option in the project. + when accessing your GitLab project or, in the upper-right corner of the project's overview page, selecting **Code**. - The `Audience` defined in the Azure AD federated identity credentials, for example `https://gitlab.com` or your own GitLab URL. diff --git a/doc/ci/debugging.md b/doc/ci/debugging.md index 8a60b5f649e..84715cda2f5 100644 --- a/doc/ci/debugging.md +++ b/doc/ci/debugging.md @@ -143,7 +143,7 @@ configuration into more independent [parent-child pipelines](../ci/pipelines/pip Pipeline configuration warnings are shown when you: -- [Validate configuration with the CI Lint tool](yaml/index.md). +- [Validate configuration with the CI Lint tool](lint.md). - [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually). ### `Job may allow multiple pipelines to run for a single action` warning diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index beaa2291eea..bc157c7ae08 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -853,7 +853,7 @@ This indicates the GitLab Runner does not have permission to start the 1. Check that `privileged = true` is set in the `config.toml`. 1. Make sure the CI job has the right Runner tags to use these -privileged runners. + privileged runners. ### Error: `cgroups: cgroup mountpoint does not exist: unknown` diff --git a/doc/ci/environments/img/kubernetes_summary_ui.png b/doc/ci/environments/img/kubernetes_summary_ui.png Binary files differindex ce51cd8e96f..f8eae88745e 100644 --- a/doc/ci/environments/img/kubernetes_summary_ui.png +++ b/doc/ci/environments/img/kubernetes_summary_ui.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 03c9a152d98..c9148e04bf3 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Environments describe where code is deployed. -Each time [GitLab CI/CD](../yaml/index.md) deploys a version of code to an environment, +Each time [GitLab CI/CD](../index.md) deploys a version of code to an environment, a deployment is created. GitLab: @@ -24,7 +24,7 @@ associated with your project, you can use it to assist with your deployments. Prerequisites: -- You must have at least the Reporter role. +- In a private project, you must have at least the Reporter role. See [Environment permissions](#environment-permissions). There are a few ways to view a list of environments for a given project: @@ -108,7 +108,7 @@ To create a static environment, in your `.gitlab-ci.yml` file: 1. Define a job in the `deploy` stage. 1. In the job, define the environment `name` and `url`. If an -environment of that name doesn't exist when the pipeline runs, it is created. + environment of that name doesn't exist when the pipeline runs, it is created. NOTE: Some characters cannot be used in environment names. For more information about the @@ -912,6 +912,38 @@ like [Review Apps](../review_apps/index.md) (`review/*`). The most specific spec takes precedence over the other wildcard matching. In this case, the `review/feature-1` spec takes precedence over `review/*` and `*` specs. +## Environment permissions + +Depending on your role, you can interact with environments in public +and private projects. + +### View environments + +- In public projects, anyone can view a list of environments, including non-members. +- In private projects, you must have at least the Reporter role to view a list of environments. + +### Create and update environments + +- You must have at least the Developer role to create a new environment, or update an existing unprotected environment. +- If an existing environment is protected and you don't have access to it, you cannot update the environment. + +### Stop and delete environments + +- You must have at least the Developer role to stop or delete an unprotected environment. +- If an environment is protected and you don't have access to it, you cannot stop or delete the environment. + +### Run deployment jobs in protected environments + +If you can push or merge to the protected branch: + +- You must have at least the Reporter role. + +If you can't push to the protected branch: + +- You must be a part of a group with the Reporter role. + +See [Deployment-only access to protected environments](protected_environments.md#deployment-only-access-to-protected-environments). + ## Related topics - [Dashboard for Kubernetes](kubernetes_dashboard.md) @@ -1007,7 +1039,7 @@ deploy: Since `$ENVIRONMENT` variable does not exist in the pipeline, GitLab tries to create an environment with a name `production/`, which is invalid in -[the environment name constraint](../yaml/index.md). +[the environment name constraint](../yaml/index.md#environmentname). To fix this, use one of the following solutions: diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 556af355b1a..8872602a027 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -144,7 +144,7 @@ new browser window interacting with your app as you specified. Which brings us to the exciting part: how do we run this in GitLab CI/CD? There are two things we need to do for this: -1. Set up [CI/CD jobs](../../yaml/index.md) that actually have a browser available. +1. Set up [CI/CD jobs](../../jobs/index.md) that actually have a browser available. 1. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/index.md#stages) diff --git a/doc/ci/index.md b/doc/ci/index.md index beb7fffeb8a..429db0beede 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -2,7 +2,6 @@ stage: Verify group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." --- # Get started with GitLab CI/CD **(FREE ALL)** @@ -22,7 +21,7 @@ If you're new to GitLab CI/CD, start by reviewing some of the commonly used term To use GitLab CI/CD, you start with a `.gitlab-ci.yml` file at the root of your project which contains the configuration for your CI/CD pipeline. This file follows the YAML format -and has its own special syntax. +and has its own syntax. You can name this file anything you want, but `.gitlab-ci.yml` is the most common name. @@ -38,8 +37,8 @@ In the `.gitlab-ci.yml` file, you can define: **Get started:** - [Create your first `.gitlab-ci.yml` file](quick_start/index.md). -- [View all the possible keywords that you can use in the `.gitlab-ci.yml` file](yaml/index.md). -the configuration. +- View all the possible keywords that you can use in the `.gitlab-ci.yml` file in + the [CI/CD YAML syntax reference](../ci/yaml/index.md). - Use the [pipeline editor](pipeline_editor/index.md) to edit or [visualize](pipeline_editor/index.md#visualize-ci-configuration) your CI/CD configuration. diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index ba2d63c12e4..3a787189ac0 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -17,7 +17,7 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - [Container registry](../../user/packages/container_registry/build_and_push_images.md#use-gitlab-cicd) (the `$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`). - [Container registry API](../../api/container_registry.md) - (scoped to the job's project, when the `ci_job_token_scope` feature flag is enabled). + (scoped to the job's project). - [Get job artifacts](../../api/job_artifacts.md#get-job-artifacts). - [Get job token's job](../../api/jobs.md#get-job-tokens-job). - [Pipeline triggers](../../api/pipeline_triggers.md), using the `token=` parameter diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index f4836f93234..3cba8787821 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -134,7 +134,7 @@ jobs. Select to expand them.  -To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/index.md), +To create a group of jobs, in the [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file), separate each job name with a number and one of the following: - A slash (`/`), for example, `slash-test 1/3`, `slash-test 2/3`, `slash-test 3/3`. diff --git a/doc/ci/jobs/job_artifacts.md b/doc/ci/jobs/job_artifacts.md index f93068faf01..34da3be9370 100644 --- a/doc/ci/jobs/job_artifacts.md +++ b/doc/ci/jobs/job_artifacts.md @@ -261,9 +261,17 @@ from: - The **Artifacts** page. On the right of the job, select **Browse** (**{folder-open}**). If [GitLab Pages](../../administration/pages/index.md) is enabled in the project, you can preview -HTML files in the artifacts directly in your browser. If the project is internal or private, you must -enable [GitLab Pages access control](../../administration/pages/index.md#access-control) to preview -HTML files. +some artifacts file extensions directly in your browser. If the project is internal or private, you must enable [GitLab Pages access control](../../administration/pages/index.md#access-control) to enable the preview. + +The following extensions are supported: + +| File extension | GitLab.com | Linux package with built-in NGINX | +|----------|---------------------|--------------| +| `.html` | **{check-circle}** Yes | **{check-circle}** Yes | +| `.json` | **{check-circle}** Yes | **{check-circle}** Yes | +| `.xml` | **{check-circle}** Yes | **{check-circle}** Yes | +| `.txt` | **{dotted-circle}** No | **{check-circle}** Yes | +| `.log` | **{dotted-circle}** No | **{check-circle}** Yes | ### From a URL diff --git a/doc/ci/jobs/job_artifacts_troubleshooting.md b/doc/ci/jobs/job_artifacts_troubleshooting.md index 0b7777d2d82..470c1bf4b55 100644 --- a/doc/ci/jobs/job_artifacts_troubleshooting.md +++ b/doc/ci/jobs/job_artifacts_troubleshooting.md @@ -155,3 +155,26 @@ To troubleshoot this error, verify that: parent-child pipeline hierarchy. - The `pipeline` and `job` combination exists and resolves to an existing pipeline. - `dependency-job` has run and finished successfully. + +## Jobs show `UnlockPipelinesInQueueWorker` after an upgrade + +Jobs might stall and show an error that states `UnlockPipelinesInQueueWorker`. + +This issue occurs after an upgrade. + +The workaround is to enable the `ci_unlock_pipelines_extra_low` feature flag. +To toggle feature flags, you must be an administrator. + +On GitLab SaaS: + +- Run the following [ChatOps](../chatops/index.md) command: + + ```ruby + /chatops run feature set ci_unlock_pipelines_extra_low true + ``` + +On GitLab self-managed: + +- [Enable the feature flag](../../administration/feature_flags.md) named `ci_unlock_pipelines_extra_low`. + +For more information see the comment in [merge request 140318](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140318#note_1718600424). diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index b432106a2d8..f7b3fed7d74 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -1218,3 +1218,49 @@ a branch that has an open merge request associated with it. To [prevent duplicate pipelines](#avoid-duplicate-pipelines), use [`workflow: rules`](../yaml/index.md#workflow) or rewrite your rules to control which pipelines can run. + +### `This GitLab CI configuration is invalid` for variable expressions + +You might receive one of several `This GitLab CI configuration is invalid` errors +when working with [CI/CD variable expressions](#cicd-variable-expressions). +These syntax errors can be caused by incorrect usage of quote characters. + +In variable expressions, strings should be quoted, while variables should not be quoted. +For example: + +```yaml +variables: + ENVIRONMENT: production + +job: + script: echo + rules: + - if: $ENVIRONMENT == "production" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + +In this example, both `if:` clauses are valid because the `production` string is quoted, +and the CI/CD variables are unquoted. + +On the other hand, these `if:` clauses are all invalid: + +```yaml +variables: + ENVIRONMENT: production + +job: + script: echo + rules: # These rules all cause YAML syntax errors: + - if: ${ENVIRONMENT} == "production" + - if: "$ENVIRONMENT" == "production" + - if: $ENVIRONMENT == production + - if: "production" == "production" +``` + +In this example: + +- `if: ${ENVIRONMENT} == "production"` is invalid, because `${ENVIRONMENT}` is not valid + formatting for CI/CD variables in `if:`. +- `if: "$ENVIRONMENT" == "production"` is invalid, because the variable is quoted. +- `if: $ENVIRONMENT == production` is invalid, because the string is not quoted. +- `if: "production" == "production"` is invalid, because there is no CI/CD variable to compare. diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md deleted file mode 100644 index cef0554bf6c..00000000000 --- a/doc/ci/large_repositories/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../user/project/repository/managing_large_repositories.md' -remove_date: '2023-11-30' ---- - -This document was moved to [another location](../../user/project/repository/managing_large_repositories.md). - -<!-- This redirect file can be deleted after <2023-11-30>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/migration/bamboo.md b/doc/ci/migration/bamboo.md index ea9dd666176..b2594e7571e 100644 --- a/doc/ci/migration/bamboo.md +++ b/doc/ci/migration/bamboo.md @@ -14,7 +14,7 @@ exported from the Bamboo UI or stored in Spec repositories. If you are new to GitLab CI/CD, use the [Getting started guide](../index.md) to learn the basic concepts and how to create your first [`.gitlab-ci.yml` file](../quick_start/index.md). -If you already have some experience using GitLab CI/CD, you can review [keywords reference documentation](../yaml/index.md) +If you already have some experience using GitLab CI/CD, you can review [CI/CD YAML syntax reference](../yaml/index.md) to see the full list of available keywords. You can also take a look at [Auto DevOps](../../topics/autodevops/index.md), which automatically @@ -77,7 +77,7 @@ Bamboo Specs can also be [repository-stored](https://confluence.atlassian.com/ba #### `.gitlab-ci.yml` configuration file -GitLab, by default, uses a [`.gitlab-ci.yml` file](../yaml/index.md) for CI/CD configuration. +GitLab, by default, uses a [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file) for CI/CD configuration. Alternatively, [Auto DevOps](../../topics/autodevops/index.md) can automatically build, test, and deploy your application without a manually configured `.gitlab-ci.yml` file. @@ -754,7 +754,7 @@ Before doing any migration work, you should first: - Follow tutorials to create [your first GitLab pipeline](../quick_start/index.md) and [more complex pipelines](../quick_start/tutorial.md) that build, test, and deploy a static site. - - Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). + - Review the [CI/CD YAML syntax reference](../yaml/index.md). 1. Set up and configure GitLab. 1. Test your GitLab instance. - Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners or installing new runners. diff --git a/doc/ci/migration/github_actions.md b/doc/ci/migration/github_actions.md index ec55f129c1e..6afb6cebbff 100644 --- a/doc/ci/migration/github_actions.md +++ b/doc/ci/migration/github_actions.md @@ -672,7 +672,7 @@ Before doing any migration work, you should first: 1. Get familiar with GitLab. - Read about the [key GitLab CI/CD features](../../ci/index.md). - Follow tutorials to create [your first GitLab pipeline](../quick_start/index.md) and [more complex pipelines](../quick_start/tutorial.md) that build, test, and deploys a static site. - - Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). + - Review the [CI/CD YAML syntax reference](../yaml/index.md). 1. Set up and configure GitLab. 1. Test your GitLab instance. - Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners or installing new runners. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index f430b1ac7b9..961799b9564 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -699,7 +699,7 @@ Before doing any migration work, you should first: 1. Get familiar with GitLab. - Read about the [key GitLab CI/CD features](../../ci/index.md). - Follow tutorials to create [your first GitLab pipeline](../quick_start/index.md) and [more complex pipelines](../quick_start/tutorial.md) that build, test, and deploys a static site. - - Review the [`.gitlab-ci.yml` keyword reference](../yaml/index.md). + - Review the [CI/CD YAML syntax reference](../yaml/index.md). 1. Set up and configure GitLab. 1. Test your GitLab instance. - Ensure [runners](../runners/index.md) are available, either by using shared GitLab.com runners or installing new runners. diff --git a/doc/ci/mobile_devops.md b/doc/ci/mobile_devops.md index e871a95d29f..4639967fb1d 100644 --- a/doc/ci/mobile_devops.md +++ b/doc/ci/mobile_devops.md @@ -338,26 +338,34 @@ To create an iOS distribution with the Apple Store integration and fastlane, you 1. Generate an API Key for App Store Connect API. In the Apple App Store Connect portal, [generate a new private key for your project](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api). -1. [Enable the Apple App Store integration](#enable-apple-app-store-integration). +1. [Enable the Apple App Store Connect integration](#enable-the-apple-app-store-connect-integration). 1. Add the release step to your pipeline and fastlane configuration. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Apple App Store integration demo](https://youtu.be/CwzAWVgJeK8). +For an overview, see [Apple App Store Connect integration demo](https://youtu.be/CwzAWVgJeK8). +<!-- Video published on 2023-03-17 --> -#### Enable Apple App Store Integration +#### Enable the Apple App Store Connect integration -Use the [Apple App Store integration](../user/project/integrations/apple_app_store.md) -to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com/) -to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. To enable the integration: +Prerequisites: + +- You must have an Apple ID enrolled in the [Apple Developer Program](https://developer.apple.com/programs/enroll/). +- You must [generate a new private key](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api) for your project in the Apple App Store Connect portal. + +Use the Apple App Store Connect integration to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com). +With this integration, you can build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. + +To enable the Apple App Store Connect integration in GitLab: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. -1. Select **Apple App Store**. +1. Select **Apple App Store Connect**. 1. Under **Enable integration**, select the **Active** checkbox. 1. Provide the Apple App Store Connect configuration information: - - **Issuer ID**: You can find the Apple App Store Connect Issuer ID in the **Keys** section under **Users and Access** in the Apple App Store Connect portal. - - **Key ID**: The key ID of the new private key that was just generated. - - **Private Key**: The private key that was just generated. You can only download this key one time. + - **Issuer ID**: The Apple App Store Connect issuer ID. + - **Key ID**: The key ID of the generated private key. + - **Private key**: The generated private key. You can download this key only once. + - **Protected branches and tags only**: Enable to set variables on protected branches and tags only. 1. Select **Save changes**. With the integration enabled, you can use fastlane to distribute a build to TestFlight diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 699136ccf97..a6c3bb835d0 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -1,6 +1,7 @@ --- stage: Verify group: Pipeline Execution +description: Calculations, quotas, purchase information. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -321,7 +322,7 @@ Jobs on project runners are not affected by the compute quota. ### GitLab SaaS usage notifications -On GitLab SaaS an email notification is sent to the namespace owners when: +On GitLab SaaS an in-app banner is displayed and an email notification sent to the namespace owners when: - The remaining compute minutes is below 30% of the quota. - The remaining compute minutes is below 5% of the quota. diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index 61862a20436..ae2ca74e6f8 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -379,7 +379,9 @@ trigger_job: ::EndTabs -### View multi-project pipelines in pipeline graphs **(PREMIUM ALL)** +### View multi-project pipelines in pipeline graphs + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/422282) from GitLab Premium to GitLab Free in 16.8. After you trigger a multi-project pipeline, the downstream pipeline displays to the right of the [pipeline graph](index.md#visualize-pipelines). diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 957e0e0de27..0a4f4c3762b 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -67,7 +67,7 @@ Pipelines and their component jobs and stages are defined in the CI/CD pipeline - [Jobs](../jobs/index.md) are the basic configuration component. - Stages are defined by using the [`stages`](../yaml/index.md#stages) keyword. -For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/index.md). +For a list of configuration options in the CI pipeline file, see the [CI/CD YAML syntax reference](../yaml/index.md). You can also configure specific aspects of your pipelines through the GitLab UI. For example: diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 0de55d2a488..25358ecd602 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -58,7 +58,7 @@ The three types of merge request pipelines are: To use merge request pipelines: -- Your project's [CI/CD configuration file](../yaml/index.md) must be configured with +- Your project's [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file) must be configured with jobs that run in merge request pipelines. To do this, you can use: - [`rules`](#use-rules-to-add-jobs). - [`only/except`](#use-only-to-add-jobs). @@ -160,7 +160,7 @@ GitLab shows a warning that you must accept before the pipeline runs. Otherwise, Prerequisites: -- The parent project's [CI/CD configuration file](../yaml/index.md) must be configured to +- The parent project's [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file) must be configured to [run jobs in merge request pipelines](#prerequisites). - You must be a member of the parent project with [permissions to run CI/CD pipelines](../../user/permissions.md#gitlab-cicd-permissions). You might need additional permissions if the branch is protected. diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md index 213a07f49c4..691de7c3f3c 100644 --- a/doc/ci/pipelines/merged_results_pipelines.md +++ b/doc/ci/pipelines/merged_results_pipelines.md @@ -28,7 +28,7 @@ and [is labeled as `merge request`](merge_request_pipelines.md#types-of-merge-re To use merged results pipelines: -- Your project's [CI/CD configuration file](../yaml/index.md) must be configured to +- Your project's [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file) must be configured to [run jobs in merge request pipelines](merge_request_pipelines.md#prerequisites). - Your repository must be a GitLab repository, not an [external repository](../ci_cd_for_external_repos/index.md). diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 4a548c8c0af..07fc7c2dfd6 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -27,7 +27,7 @@ The easiest indicators to check for inefficient pipelines are the runtimes of th stages, and the total runtime of the pipeline itself. The total pipeline duration is heavily influenced by the: -- [Size of the repository](../../user/project/repository/managing_large_repositories.md) +- [Size of the repository](../../user/project/repository/monorepos/index.md) - Total number of stages and jobs. - Dependencies between jobs. - The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index f83868952f5..9ea635792f3 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -15,7 +15,7 @@ For a scheduled pipeline to run: - The schedule owner must have the Developer role. For pipelines on protected branches, the schedule owner must be [allowed to merge](../../user/project/protected_branches.md#add-protection-to-existing-branches) to the branch. -- The [CI/CD configuration](../yaml/index.md) must be valid. +- The [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file) must have valid syntax. Otherwise, the pipeline is not created. No error message is displayed. diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 3da4e5ad023..0dc07a36ef5 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -136,7 +136,7 @@ Now you can get started customizing your `.gitlab-ci.yml` and defining more adva Here are some tips to get started working with the `.gitlab-ci.yml` file. -For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` keyword reference](../yaml/index.md). +For the complete `.gitlab-ci.yml` syntax, see the full [CI/CD YAML syntax reference](../yaml/index.md). - Use the [pipeline editor](../pipeline_editor/index.md) to edit your `.gitlab-ci.yml` file. - Each job contains a script section and belongs to a stage: diff --git a/doc/ci/quick_start/tutorial.md b/doc/ci/quick_start/tutorial.md index 389309538e9..bb766436379 100644 --- a/doc/ci/quick_start/tutorial.md +++ b/doc/ci/quick_start/tutorial.md @@ -45,7 +45,7 @@ on GitLab.com: - In the **Project name** field, enter the name of your project, for example `My Pipeline Tutorial Project`. - Select **Initialize repository with a README**. 1. Select **Create project**. -1. On the right of the **Project Overview** page for your project, select **Clone** +1. On the project's overview page, in the upper-right corner, select **Code** to find the clone paths for your project. Copy the SSH or HTTP path and use the path to clone the project locally. @@ -502,5 +502,5 @@ Use a merge request to commit this pipeline configuration to the default branch. The file is simpler, but it should have the same behavior as the previous step. You've just created a full pipeline and streamlined it to be more efficient. Nice work! -Now you can take this knowledge, learn about [the rest of the `.gitlab-ci.yml` keywords](../yaml/index.md), -and build your own pipelines. +Now you can take this knowledge, learn about the rest of the `.gitlab-ci.yml` keywords +in the [CI/CD YAML syntax reference](../yaml/index.md), and build your own pipelines. diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 3ff25cc8bab..5854704521b 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -108,7 +108,7 @@ The following are example projects that demonstrate review app configuration: Other examples of review apps: - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -[Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw). + [Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw). - [Review apps for Android](https://about.gitlab.com/blog/2020/05/06/how-to-create-review-apps-for-android-with-gitlab-fastlane-and-appetize-dot-io/). ## Route Maps diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 3b21d865d8b..6212c07ce47 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -903,18 +903,41 @@ variables: | `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | | `CACHE_REQUEST_TIMEOUT` | Configure the maximum duration of cache upload and download operations for a single job in minutes. Default is `10` minutes. | -## Artifact attestation +## Artifact provenance metadata > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28940) in GitLab Runner 15.1. NOTE: Zip archives are the only supported artifact type. Follow [the issue for details](https://gitlab.com/gitlab-org/gitlab/-/issues/367203). -GitLab Runner can generate and produce attestation metadata for all build artifacts. To enable this feature, you must set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment variable to `true`. This variable can either be set globally or it can be set for individual jobs. The metadata is in rendered in a plain text `.json` file that's stored with the artifact. The file name is as follows: `{ARTIFACT_NAME}-metadata.json` where `ARTIFACT_NAME` is what was defined as the [name for the artifact](../jobs/job_artifacts.md#with-a-dynamically-defined-name) in the CI file. The file name, however, defaults to `artifacts-metadata.json` if no name was given to the build artifacts. +Runners can generate and produce provenance metadata for all build artifacts. -### Attestation format +To enable artifact provenance data, set the `RUNNER_GENERATE_ARTIFACTS_METADATA` environment +variable to `true`. You can set the variable as global or for individual jobs: -The attestation metadata is generated in the [in-toto attestation format](https://github.com/in-toto/attestation) for spec version [v0.1](https://github.com/in-toto/attestation/tree/v0.1.0/spec). The following fields are populated by default: +```yaml +variables: + RUNNER_GENERATE_ARTIFACTS_METADATA: "true" + +job1: + variables: + RUNNER_GENERATE_ARTIFACTS_METADATA: "true" +``` + +The metadata renders in a plain text `.json` file stored with the artifact. The +file name is `{ARTIFACT_NAME}-metadata.json`. `ARTIFACT_NAME` is the +[name for the artifact](../jobs/job_artifacts.md#with-a-dynamically-defined-name) +defined in the `.gitlab-ci.yml` file. If the name is not defined, the default file name is +`artifacts-metadata.json`. + +### Provenance metadata format + +The provenance metadata is generated in the [in-toto attestation format](https://github.com/in-toto/attestation) for spec version [0.1](https://github.com/in-toto/attestation/tree/v0.1.0/spec). +The runner also produces a statement that adheres to SLSA v0.2 by default. + +To opt-in to an SLSA v1.0 statement, set the `SLSA_PROVENANCE_SCHEMA_VERSION=v1` variable in the `.gitlab-ci.yml` file. The v0.2 statement is deprecated and is planned to be removed in the GitLab 17.0 and the v1.0 statement is planned to become the new default format. + +The following fields are populated by default: | Field | Value | | ------ | ------ | @@ -938,7 +961,7 @@ The attestation metadata is generated in the [in-toto attestation format](https: | `metadata.completeness.environment` | Whether the builder's environment is reported. Always `true`. | | `metadata.completeness.materials` | Whether the build materials are reported. Always `false`. | -An example of an attestation that the GitLab Runner might generate is as follows: +An example of provenance metadata that the GitLab Runner might generate is as follows: ```yaml { diff --git a/doc/ci/runners/img/runner_fleet_dashboard.png b/doc/ci/runners/img/runner_fleet_dashboard.png Binary files differnew file mode 100644 index 00000000000..8c77b5a1aa9 --- /dev/null +++ b/doc/ci/runners/img/runner_fleet_dashboard.png diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 1df93ed8896..bfb30a36be2 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -33,7 +33,7 @@ When you use SaaS runners: - The VM is active only for the duration of the job and immediately deleted. This means that any changes that your job makes to the virtual machine will not be available to a subsequent job. - The virtual machine where your job runs has `sudo` access with no password. - The storage is shared by the operating system, the image with pre-installed software, and a copy of your cloned repository. -This means that the available free disk space for your jobs to use is reduced. + This means that the available free disk space for your jobs to use is reduced. NOTE: Jobs handled by SaaS runners on GitLab.com **time out after 3 hours**, regardless of the timeout configured in a project. diff --git a/doc/ci/runners/new_creation_workflow.md b/doc/ci/runners/new_creation_workflow.md index c870a89a77a..12bffb79e33 100644 --- a/doc/ci/runners/new_creation_workflow.md +++ b/doc/ci/runners/new_creation_workflow.md @@ -39,7 +39,7 @@ The new runner registration workflow has the following benefits: - Preserved ownership records for runners, and minimized impact on users. - The addition of a unique system ID ensures that you can reuse the same authentication token across -multiple runners. For more information, see [Reusing a GitLab Runner configuration](https://docs.gitlab.com/runner/fleet_scaling/#reusing-a-gitlab-runner-configuration). + multiple runners. For more information, see [Reusing a GitLab Runner configuration](https://docs.gitlab.com/runner/fleet_scaling/#reusing-a-gitlab-runner-configuration). ## Estimated time frame for planned changes @@ -61,7 +61,7 @@ To avoid a broken workflow, you must: 1. [Create a shared runner](runners_scope.md#create-a-shared-runner-with-a-runner-authentication-token) and obtain the authentication token. 1. Replace the registration token in your runner registration workflow with the -authentication token. + authentication token. ## Using registration tokens after GitLab 17.0 @@ -159,7 +159,9 @@ Several runner configuration options cannot be set during runner registration. T The following configuration options are no longer supported in [`values.yaml`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml): ```yaml -## All these fields are DEPRECATED and the runner WILL FAIL TO START if you specify them +## All these fields are DEPRECATED and the runner WILL FAIL TO START with GitLab Runner 18.0 and later if you specify them. +## If a runner authentication token is specified in runnerRegistrationToken, the registration will succeed, however the +## other values will be ignored. runnerRegistrationToken: "" locked: true tags: "" diff --git a/doc/ci/runners/runner_fleet_dashboard.md b/doc/ci/runners/runner_fleet_dashboard.md new file mode 100644 index 00000000000..f329561cf4b --- /dev/null +++ b/doc/ci/runners/runner_fleet_dashboard.md @@ -0,0 +1,90 @@ +--- +stage: Verify +group: Runner +info: >- + To determine the technical writer assigned to the Stage/Group associated with + this page, see + https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- +# Runner Fleet Dashboard **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/424495) in GitLab 16.6 + +GitLab administrators can use the Runner Fleet Dashboard to assess the health of your instance runners. +The Runner Fleet Dashboard shows: + +- Recent CI errors related caused by runner infrastructure. +- Number of concurrent jobs executed on most busy runners. +- Histogram of job queue times [(available only with ClickHouse)](#enable-more-ci-analytics-features-with-clickhouse). + +Support for usage and cost analysis are proposed in [epic 11183](https://gitlab.com/groups/gitlab-org/-/epics/11183). + + + +## View the Runner Fleet Dashboard + +Prerequisites: + +- You must be an administrator. + +To view the runner fleet dashboard: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Runners**. +1. Select **Fleet dashboard**. + +Most of the dashboard works without any additional actions, with the +exception of **Wait time to pick a job** chart and features proposed in [epic 11183](https://gitlab.com/groups/gitlab-org/-/epics/11183). +These features require [setting up an additional infrastructure](#enable-more-ci-analytics-features-with-clickhouse). + +## Export compute minutes used by instance runners + +Prerequisites: + +- You must be an administrator. + +To analyze runner usage, you can export a CSV file that contains the number of jobs and executed runner minutes. The +CSV file shows the runner type and job status for each project. The CSV is sent to your email when the export is completed. + +To export compute minutes used by instance runners: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Runners**. +1. Select **Fleet dashboard**. +1. Select **Export CSV**. + +## Enable more CI analytics features with ClickHouse **(ULTIMATE EXPERIMENT)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11180) in GitLab 16.7 with the [flags](../../administration/feature_flags.md) named `ci_data_ingestion_to_click_house` and `clickhouse_ci_analytics`. Disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/424866) in GitLab 16.8. Feature flag `clickhouse_ci_analytics` removed. + +This feature is an [Experiment](../../policy/experiment-beta-support.md). +To test it, we have launched an early adopters program. +To join the list of users testing this feature, see +[epic 11180](https://gitlab.com/groups/gitlab-org/-/epics/11180). + +### Enable ClickHouse integration + +To enable additional CI analytics features: + +1. [Configure ClickHouse integration](../../integration/clickhouse.md) +1. [Enable](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) the following feature flags: + + | Feature flag name | Purpose | Status | + |------------------------------------|---------------------------------------------------------------------------|------------------------------------| + | `ci_data_ingestion_to_click_house` | Enables synchronization of new finished CI builds to ClickHouse database. | Enabled by default in GitLab 16.8. | + | `clickhouse_ci_analytics` | Enables the **Wait time to pick a job** chart. | Removed in 16.8. | + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video walkthrough, see [Setting up Runner Fleet Dashboard with ClickHouse](https://www.youtube.com/watch?v=YpGV95Ctbpk). + +## Feedback + +To help us improve the Runner Fleet Dashboard, you can provide feedback in +[issue 421737](https://gitlab.com/gitlab-org/gitlab/-/issues/421737). +In particular: + +- How easy or difficult it was to setup GitLab to make the dashboard work. +- How useful you found the dashboard. +- What other information you would like to see on that dashboard. +- Any other related thoughts and ideas. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index 48f6a22e9b3..d6a556ffd9b 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -576,13 +576,16 @@ A runner can have one of the following statuses. As an administrator, you can view runner statistics to learn about the performance of your runner fleet. -- The **Median job queued time** value is calculated by sampling the queue duration of the +The **Median job queued time** value is calculated by sampling the queue duration of the most recent 100 jobs that were run by Instance runners. Jobs from only the latest 5000 runners are considered. -- The median is a value that falls into the 50th percentile: half of the jobs + +The median is a value that falls into the 50th percentile: half of the jobs queued for longer than the median value, and half of the jobs queued for less than the median value. +To view runner statistics: + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **CI/CD > Runners**. 1. Select **View metrics**. diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 03c23680c1c..0251c523864 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -21,8 +21,7 @@ You can follow our work towards this goal in the ## Machine types available for macOS -GitLab SaaS provides macOS build machines on Apple silicon (M1) chips. -Intel x86-64 runners were deprecated in favor of Apple silicon. To build for an x86-64 target, use Rosetta 2 to emulate an Intel x86-64 build environment. +GitLab SaaS provides macOS build machines on Apple silicon (M1) chips. To build for an x86-64 target, you can use Rosetta 2 to emulate an Intel x86-64 build environment. | Runner Tag | vCPUS | Memory | Storage | | ---------------------- | ----- | ------ | ------- | @@ -40,7 +39,7 @@ in your `.gitlab-ci.yml` file. Each image runs a specific version of macOS and X |----------------------------|--------|--------------| | `macos-12-xcode-14` | `GA` | | | `macos-13-xcode-14` | `GA` | [Preinstalled Software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/job-images/-/blob/main/toolchain/macos-13.yml) | -| `macos-14-xcode-15` | `Beta` | [Preinstalled Software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/job-images/-/blob/main/toolchain/macos-14.yml) | +| `macos-14-xcode-15` | `GA` | [Preinstalled Software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/job-images/-/blob/main/toolchain/macos-14.yml) | If no image is specified, the macOS runner uses `macos-13-xcode-14`. diff --git a/doc/ci/secrets/gcp_secret_manager.md b/doc/ci/secrets/gcp_secret_manager.md new file mode 100644 index 00000000000..ad2a2a269eb --- /dev/null +++ b/doc/ci/secrets/gcp_secret_manager.md @@ -0,0 +1,92 @@ +--- +stage: Verify +group: Pipeline Security +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Use GCP Secret Manager secrets in GitLab CI/CD **(PREMIUM ALL)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11739) in GitLab and GitLab Runner 16.8. + +You can use secrets stored in the [Google Cloud (GCP) Secret Manager](https://cloud.google.com/security/products/secret-manager) +in your GitLab CI/CD pipelines. + +The flow for using GitLab with GCP Secret Manager +is summarized by this diagram: + +1. GitLab issues ID token to CI/CD job. +1. The runner authenticates to GCP using an ID token. +1. GCP verifies the ID token with GitLab. +1. GCP issues a short-lived access token. +1. The runner accesses the secret data using the access token. +1. GCP checks IAM permission on the access token's principal. +1. GCP returns the secret data to Runner. + +To use GitLab with GCP Secret Manager, you must: + +- Have secrets stored in [GCP Secret Manager](https://cloud.google.com/security/products/secret-manager). +- Configure [GCP Workload Identity Federation](#configure-gcp-iam-workload-identify-federation-wif) to include GitLab as an identity provider. +- Configure [GCP IAM](#grant-access-to-gcp-iam-principal) permissions to grant access to GCP Secret Manager. +- Configure [GitLab CI/CD with GCP Secret Manager](#configure-gitlab-cicd-to-use-gcp-secret-manager-secrets). + +## Configure GCP IAM Workload Identify Federation (WIF) + +GCP IAM WIF must be configured to recognize ID tokens issued by GitLab and assign an appropriate principal to them. +The principal is used to authorize access to the Secret Manager resources: + +1. In GCP Console, go to **IAM & Admin > Workload Identity Federation**. +1. Select **CREATE POOL** and create a new identity pool with a unique name, for example `gitlab-pool`. +1. Select **ADD PROVIDER** to add a new OIDC Provider to the Identity Pool with a unique name, for example `gitlab-provider`. + 1. Set **Issuer (URL)** to the GitLab URL, for example `https://gitlab.com`. + 1. Select **Default audience**, or select **Allowed audiences** for a custom audience, which is used in the `aud` for the GitLab CI/CD ID token. +1. Under **Attribute Mapping**, configure provider attributes, which are mappings between the [OIDC claims](id_token_authentication.md#token-payload) + (referred to as "assertion") and Google attributes. These mappings can be used to set fine grained access control. + For example, to grant a GitLab project access to Secret Manager secrets, select **ADD MAPPING** and create a mapping of + `attribute.gitlab_project_id` to `assertion.project_id`. + +## Grant access to GCP IAM principal + +After setting up WIF, you must grant the WIF principal access to the secrets in Secret Manager. + +1. In GCP Console, go to **IAM & Admin > IAM**. +1. Select **GRANT ACCESS** to grant access to the principal set created through the WIF provider. For example, + to grant IAM access to the principal matching the project with ID `123`, add + a new principal like: `principalSet://iam.googleapis.com/projects/[PROJECT_NUMBER]/locations/global/workloadIdentityPools/[POOL_ID]/attribute.gitlab_project_id/[PROJECT_ID]`. +1. Assign the role **Secret Manager Secret Accessor**. +1. (Optional) Select **IAM condition (Optional)** to add an IAM condition. + Under **Condition Builder**, you can add conditions. For example, you could add two `AND` conditions: + - First condition: + - **Condition type**: `Type` + - **Operator**: `is` + - **Resource type**: `secretmanager.googleapis.com/SecretVersion` + - Second condition: + - **Condition type**: `Name` + - **Operator**: `Starts with` + - **Value**: The pattern of secrets that you want to grant access to. + +You can add additional IAM conditions for fine-grained access controls, including +accessing secrets with names starting with the project name. + +## Configure GitLab CI/CD to use GCP Secret Manager secrets + +You can use secrets stored in GCP Secret Manager in CI/CD jobs by defining them with the `gcp_secret_manager` keyword: + +```yaml +job_using_gcp_sm: + id_tokens: + GCP_ID_TOKEN: + # `aud` must match the audience defined in the WIF Identity Pool. + aud: https://iam.googleapis.com/projects/1234/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider + secrets: + DATABASE_PASSWORD: + gcp_secret_manager: + name: my-project-secret # This is the name of the secret defined in GCP Secret Manager + version: 1 # optional: default to `latest`. + token: $GCP_ID_TOKEN +``` + +You must also [add these CI/CD variables](../variables/index.md#for-a-project) to provide details about your GCP Secret Manager: + +- `GCP_PROJECT_NUMBER`: The GCP [Project Number](https://cloud.google.com/resource-manager/docs/creating-managing-projects) +- `GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID`: The WIF Pool ID (e.g `gitlab-pool`) +- `GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID`: The WIF Provider ID (e.g `gitlab-provider`) diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index e452b26d8a9..96b9709bdef 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -18,6 +18,12 @@ Unlike CI/CD variables, which are always presented to a job, secrets must be exp required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/index.md#secrets) for more information about the syntax. +GitLab provides support for the following secret management providers: + +1. [Vault by HashiCorp](#use-vault-secrets-in-a-ci-job) +1. [Google Cloud Secret Manager](gcp_secret_manager.md) +1. [Azure Key Vault](azure_key_vault.md) + GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the first supported provider, and [KV-V2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2) as the first supported secrets engine. diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index 23ae615eeb2..9e6c409a0d3 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -27,14 +27,15 @@ You can extend the code coverage either by using Code Climate Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: -| Capability | In Free | In Premium | In Ultimate | -|:-----------------------------------------------------------------------|:--------------------|:--------------------|:-------------------| -| [Configure scanners](#customizing-scan-settings) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Integrate custom scanners](#implement-a-custom-tool) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request widget](#merge-request-widget) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Generate JSON or HTML report artifacts](#output) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See reports in CI pipelines](#pipeline-details-view) | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request diff view](#merge-request-changes-view) | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | +| Feature | In Free | In Premium | In Ultimate | +|:----------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------| +| [Configure scanners](#customizing-scan-settings) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| [Integrate custom scanners](#implement-a-custom-tool) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| [Generate JSON or HTML report artifacts](#output) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| [Findings in merge request widget](#merge-request-widget) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| [Findings in pipelines](#pipeline-details-view) | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | +| [Findings in merge request changes view](#merge-request-changes-view) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | +| [Summary in project quality view](#project-quality-view) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | ## View Code Quality results @@ -219,63 +220,42 @@ To configure the Code Quality job: 1. Declare a job with the same name as the Code Quality job, after the template's inclusion. 1. Specify additional keys in the job's stanza. -For an example, see [Download output in JSON format](#download-output-in-json-format). +For an example, see [Download output in HTML format](#output-in-only-html-format). -### Available CI/CD variables - -> In [GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/11100), the option to override the Code Quality environment variables was added. +## Available CI/CD variables Code Quality can be customized by defining available CI/CD variables: -| CI/CD variable | Description | -| --------------------------- | ----------- | -| `SOURCE_CODE` | Path to the source code to scan. | -| `TIMEOUT_SECONDS` | Custom timeout per engine container for the `codeclimate analyze` command, default is 900 seconds (15 minutes). | -| `CODECLIMATE_DEBUG` | Set to enable [Code Climate debug mode](https://github.com/codeclimate/codeclimate#environment-variables) | -| `CODECLIMATE_DEV` | Set to enable `--dev` mode which lets you run engines not known to the CLI. | -| `REPORT_STDOUT` | Set to print the report to `STDOUT` instead of generating the usual report file. | -| `REPORT_FORMAT` | Set to control the format of the generated report file. One of: `json\|html`. | -| `ENGINE_MEMORY_LIMIT_BYTES` | Set the memory limit for engines, default is 1,024,000,000 bytes. | -| `CODE_QUALITY_DISABLED` | Prevents the Code Quality job from running. | -| `CODECLIMATE_PREFIX` | Set a prefix to use with all `docker pull` commands in CodeClimate engines. Useful for [offline scanning](https://github.com/codeclimate/codeclimate/pull/948). | +| CI/CD variable | Description | +|---------------------------------|-------------| +| `CODECLIMATE_DEBUG` | Set to enable [Code Climate debug mode](https://github.com/codeclimate/codeclimate#environment-variables). | +| `CODECLIMATE_DEV` | Set to enable `--dev` mode which lets you run engines not known to the CLI. | +| `CODECLIMATE_PREFIX` | Set a prefix to use with all `docker pull` commands in CodeClimate engines. Useful for [offline scanning](https://github.com/codeclimate/codeclimate/pull/948). For more information, see [Use a private container registry](#use-a-private-container-image-registry). | +| `CODECLIMATE_REGISTRY_USERNAME` | Set to specify the username for the registry domain parsed from `CODECLIMATE_PREFIX`. | +| `CODECLIMATE_REGISTRY_PASSWORD` | Set to specify the password for the registry domain parsed from `CODECLIMATE_PREFIX`. | +| `CODE_QUALITY_DISABLED` | Prevents the Code Quality job from running. | +| `CODE_QUALITY_IMAGE` | Set to a fully prefixed image name. Image must be accessible from your job environment. | +| `ENGINE_MEMORY_LIMIT_BYTES` | Set the memory limit for engines. Default: 1,024,000,000 bytes. | +| `REPORT_STDOUT` | Set to print the report to `STDOUT` instead of generating the usual report file. | +| `REPORT_FORMAT` | Set to control the format of the generated report file. Either `json` or `html`. | +| `SOURCE_CODE` | Path to the source code to scan. | +| `TIMEOUT_SECONDS` | Custom timeout per engine container for the `codeclimate analyze` command. Default: 900 seconds (15 minutes) | ## Output -Code Quality creates a file named `gl-code-quality-report.json`. The content of this file is -processed internally and the results shown in the UI. To see the raw results, you can -configure the Code Quality job to allow download of this file. Format options are JSON format, HTML -format, or both. Use the HTML format to view the report in a more human-readable -format. For example, you could publish the HTML format file on GitLab Pages for even easier +Code Quality outputs a report containing details of issues found. The content of this report is +processed internally and the results shown in the UI. The report is also output as a job artifact of +the `code_quality` job, named `gl-code-quality-report.json`. You can optionally output the report in +HTML format. For example, you could publish the HTML format file on GitLab Pages for even easier reviewing. -### Download output in JSON format - -To be able to download the Code Quality report in JSON format, declare the -`gl-code-quality-report.json` file as an artifact of the `code_quality` job: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - artifacts: - paths: [gl-code-quality-report.json] -``` - -The full JSON file is available as a -[downloadable artifact](../jobs/job_artifacts.md#download-job-artifacts) of the `code_quality` -job. +### Output in JSON and HTML format -### Download output in JSON and HTML format - -> HTML report format [introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/10) in GitLab 13.6. - -NOTE: -To create the HTML format file, the Code Quality job must be run twice, once for each format. -In this configuration, the JSON format file is created but it is only processed internally. +To output the Code Quality report in JSON and HTML format, you create an additional job. This requires +Code Quality to be run twice, once each for file format. -To be able to download the Code Quality report in both JSON and HTML format, add another job to your -template by using `extends: code_quality`: +To output the Code Quality report in HTML format, add another job to your template by using +`extends: code_quality`: ```yaml include: @@ -289,18 +269,17 @@ code_quality_html: paths: [gl-code-quality-report.html] ``` -Both the JSON and HTML files are available as -[downloadable artifacts](../jobs/job_artifacts.md#download-job-artifacts) of the `code_quality` -job. +Both the JSON and HTML files are output as job artifacts. The HTML file is contained in the +`artifacts.zip` job artifact. -### Download output in only HTML format +### Output in only HTML format -To download the Code Quality report in _only_ an HTML format file, set `REPORT_FORMAT` to `html` in -the existing job. +To download the Code Quality report in _only_ HTML format, set `REPORT_FORMAT` to `html`, overriding +the default definition of the `code_quality` job. NOTE: -This does not create a JSON format file, so Code Quality results are not shown in the -merge request widget, pipeline report, or changes view. +This does not create a JSON format file, so Code Quality results are not shown in the merge request +widget, pipeline report, or changes view. ```yaml include: @@ -313,9 +292,7 @@ code_quality: paths: [gl-code-quality-report.html] ``` -The HTML file is available as a -[downloadable artifact](../jobs/job_artifacts.md#download-job-artifacts) of the `code_quality` -job. +The HTML file is output as a job artifact. ## Use Code Quality with merge request pipelines diff --git a/doc/ci/testing/img/code_quality_inline_indicator_v16_7.png b/doc/ci/testing/img/code_quality_inline_indicator_v16_7.png Binary files differindex 0d7d5bb3062..91285493562 100644 --- a/doc/ci/testing/img/code_quality_inline_indicator_v16_7.png +++ b/doc/ci/testing/img/code_quality_inline_indicator_v16_7.png diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md index ff55f37e1ff..ecd5c794344 100644 --- a/doc/ci/testing/test_coverage_visualization.md +++ b/doc/ci/testing/test_coverage_visualization.md @@ -178,7 +178,7 @@ the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverag ### JavaScript example -The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [Mocha](https://mochajs.org/) +The following `.gitlab-ci.yml` example uses [Mocha](https://mochajs.org/) JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to generate the coverage artifact: @@ -198,7 +198,7 @@ test: #### Maven example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/) +The following `.gitlab-ci.yml` example for Java or Kotlin uses [Maven](https://maven.apache.org/) to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to generate the coverage artifact. You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. @@ -236,7 +236,7 @@ coverage-jdk11: #### Gradle example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/) +The following `.gitlab-ci.yml` example for Java or Kotlin uses [Gradle](https://gradle.org/) to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to generate the coverage artifact. You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. @@ -274,7 +274,7 @@ coverage-jdk11: ### Python example -The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data: +The following `.gitlab-ci.yml` example uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data: ```yaml run tests: @@ -293,7 +293,7 @@ run tests: ### PHP example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/) +The following `.gitlab-ci.yml` example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/) to collect test coverage data and generate the report. With a minimal [`phpunit.xml`](https://docs.phpunit.de/en/10.2/configuration.html) file (you may reference @@ -331,7 +331,7 @@ to find Cobertura in the appropriate path. ### C/C++ example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for C/C++ with +The following `.gitlab-ci.yml` example for C/C++ with `gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage output file in Cobertura XML format. @@ -362,7 +362,7 @@ run tests: ### Go example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for Go uses: +The following `.gitlab-ci.yml` example for Go uses: - [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests. - [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. @@ -391,7 +391,7 @@ run tests: ### Ruby example -The following [`.gitlab-ci.yml`](../yaml/index.md) example for Ruby uses +The following `.gitlab-ci.yml` example for Ruby uses - [`rspec`](https://rspec.info/) to run tests. - [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura) diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index f247d9609fe..49f5f1edf41 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -147,7 +147,7 @@ To add or update variables in the project settings: in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). -After you create a variable, you can use it in the [`.gitlab-ci.yml` configuration](../yaml/index.md) +After you create a variable, you can use it in the pipeline configuration or in [job scripts](#use-cicd-variables-in-job-scripts). ### For a group @@ -244,7 +244,7 @@ malicious code can compromise both masked and protected variables. Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and stored in the database. This data can only be read and decrypted with a -valid [secrets file](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost). +valid [secrets file](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). ### Mask a CI/CD variable @@ -638,7 +638,7 @@ To disable variable expansion for the variable: ## CI/CD variable precedence -> Scan Execution Policies variable precedence was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/424028) in GitLab 16.7 [with a flag](../../administration/feature_flags.md) named `security_policies_variables_precedence`. Enabled by default. +> Scan Execution Policies variable precedence was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/424028) in GitLab 16.7 [with a flag](../../administration/feature_flags.md) named `security_policies_variables_precedence`. Enabled by default. [Feature flag removed in GitLab 16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/435727). You can use CI/CD variables with the same name in different places, but the values can overwrite each other. The type of variable and where they are defined determines @@ -967,4 +967,4 @@ As a workaround you can either: - Use [File-type](#use-file-type-cicd-variables) CI/CD variables for large environment variables where possible. - If a single large variable is larger than `ARG_MAX`, try using [Secure Files](../secure_files/index.md), or -bring the file to the job through some other mechanism. + bring the file to the job through some other mechanism. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index e8ed47cedd0..470982c7d26 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -164,8 +164,9 @@ These variables are available when: | `CI_MERGE_REQUEST_DIFF_BASE_SHA` | 13.7 | all | The base SHA of the merge request diff. | | `CI_MERGE_REQUEST_DIFF_ID` | 13.7 | all | The version of the merge request diff. | | `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | +| `CI_MERGE_REQUEST_DESCRIPTION` | 16.7 | all | The description of the merge request. If the description is more than 2700 characters long, only the first 2700 characters are stored in the variable. | +| `CI_MERGE_REQUEST_DESCRIPTION_IS_TRUNCATED` | 16.8 | all | `true` if `CI_MERGE_REQUEST_DESCRIPTION` is truncated down to 2700 characters because the description of the merge request is too long. | | `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. This is a unique ID across all projects on the GitLab instance. | -| `CI_MERGE_REQUEST_DESCRIPTION` | 16.7 | all | The description of the merge request. | | `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. This ID is unique for the current project, and is the number used in the merge request URL, page title, and other visible locations. | | `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request. | | `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index bddb0947fac..4ea45c9bae4 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -16,7 +16,7 @@ This document describes where and how the different types of variables can be us There are two places defined variables can be used. On the: -1. GitLab side, in the [`.gitlab-ci.yml` file](../yaml/index.md). +1. GitLab side, in the [`.gitlab-ci.yml` file](../index.md#the-gitlab-ciyml-file). 1. The GitLab Runner side, in `config.toml`. ### `.gitlab-ci.yml` file @@ -27,6 +27,7 @@ There are two places defined variables can be used. On the: |:----------------------------------------------------------------------|:-----------------|:-----------------------|:------------| | [`after_script`](../yaml/index.md#after_script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment). | | [`artifacts:name`](../yaml/index.md#artifactsname) | yes | Runner | The variable expansion is made by GitLab Runner's shell environment. | +| [`artifacts:paths`](../yaml/index.md#artifactspaths) | yes | Runner | The variable expansion is made by GitLab Runner's shell environment. | | [`before_script`](../yaml/index.md#before_script) | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | | [`cache:key`](../yaml/index.md#cachekey) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | | [`cache:policy`](../yaml/index.md#cachepolicy) | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism). | diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 5867a5b3506..131f9e502fe 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -322,13 +322,13 @@ The `repository_xray` report collects information about your repository for use > [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST -report uploads to GitLab as an artifact. +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). +The collected SAST report uploads to GitLab as an artifact. -GitLab can display the results of one or more reports in: +For more information, see: -- The merge request [SAST widget](../../user/application_security/sast/index.md). -- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- [View SAST results](../../user/application_security/sast/index.md#view-sast-results) +- [SAST output](../../user/application_security/sast/index.md#output) ## `artifacts:reports:secret_detection` diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index df2330e04f6..208da96c5ad 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -303,7 +303,9 @@ include: A `not found or access denied` error may be displayed if the user does not have access to any of the included files. - Be careful when including another project's CI/CD configuration file. No pipelines or notifications trigger when CI/CD configuration files change. From a security perspective, this is similar to pulling a third-party dependency. For the `ref`, consider: - - Using a specific SHA hash, which should be the most stable option. + - Using a specific SHA hash, which should be the most stable option. Use the + full 40-character SHA hash to ensure the desired commit is referenced, because + using a short SHA hash for the `ref` might be ambiguous. - Applying both [protected branch](../../user/project/protected_branches.md) and [protected tag](../../user/project/protected_tags.md#prevent-tag-creation-with-the-same-name-as-branches) rules to the `ref` in the other project. Protected tags and branches are more likely to pass through change management before changing. @@ -480,6 +482,46 @@ You can use some [predefined CI/CD variables](../variables/predefined_variables. - [`workflow: rules` examples](workflow.md#workflow-rules-examples) - [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) +#### `workflow:auto_cancel:on_new_commit` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412473) in GitLab 16.8 [with a flag](../../administration/feature_flags.md) named `ci_workflow_auto_cancel_on_new_commit`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or +for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `ci_workflow_auto_cancel_on_new_commit`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +Use `workflow:auto_cancel:on_new_commit` to configure the behavior of +the [auto-cancel redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) feature. + +**Possible inputs**: + +- `conservative`: Cancel the pipeline, but only if no jobs with `interruptible: false` have started yet. Default when not defined. +- `interruptible`: Cancel only jobs with `interruptible: true`. +- `none`: Do not auto-cancel any jobs. + +**Example of `workflow:auto_cancel:on_new_commit`**: + +```yaml +workflow: + auto_cancel: + on_new_commit: interruptible + +job1: + interruptible: true + script: sleep 60 + +job2: + interruptible: false # Default when not defined. + script: sleep 60 +``` + +In this example: + +- When a new commit is pushed to a branch, GitLab creates a new pipeline and `job1` and `job2` start. +- If a new commit is pushed to the branch before the jobs complete, only `job1` is canceled. + #### `workflow:name` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372538) in GitLab 15.5 [with a flag](../../administration/feature_flags.md) named `pipeline_name`. Disabled by default. @@ -657,6 +699,52 @@ When the branch is something else: - Use [`inherit:variables`](#inheritvariables) in the trigger job and list the exact variables you want to forward to the downstream pipeline. +#### `workflow:rules:auto_cancel` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/436467) in GitLab 16.8 [with a flag](../../administration/feature_flags.md) named `ci_workflow_auto_cancel_on_new_commit`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or +for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `ci_workflow_auto_cancel_on_new_commit`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +Use `workflow:rules:auto_cancel` to configure the behavior of +the [`workflow:auto_cancel:on_new_commit`](#workflowauto_cancelon_new_commit) feature. + +**Possible inputs**: + +- `on_new_commit`: [`workflow:auto_cancel:on_new_commit`](#workflowauto_cancelon_new_commit) + +**Example of `workflow:rules:auto_cancel`**: + +```yaml +workflow: + auto_cancel: + on_new_commit: interruptible + rules: + - if: $CI_COMMIT_REF_PROTECTED == 'true' + auto_cancel: + on_new_commit: none + - when: always # Run the pipeline in other cases + +test-job1: + script: sleep 10 + interruptible: false + +test-job2: + script: sleep 10 + interruptible: true +``` + +In this example, [`workflow:auto_cancel:on_new_commit`](#workflowauto_cancelon_new_commit) +is set to `interruptible` for all jobs by default. But if a pipeline runs for a protected branch, +the rule overrides the default with `on_new_commit: none`. For example, if a pipeline +is running for: + +- A non-protected branch and a new commit is pushed, `test-job1` continues to run and `test-job2` is canceled. +- A protected branch and a new commit is pushed, both `test-job1` and `test-job2` continue to run. + ## Header keywords Some keywords must be defined in a header section of a YAML configuration file. @@ -719,12 +807,12 @@ scan-website: Inputs are mandatory when included, unless you set a default value with `spec:inputs:default`. -Use `default: null` to have no default value. +Use `default: ''` to have no default value. **Keyword type**: Header keyword. `specs` must be declared at the top of the configuration file, in a header section. -**Possible inputs**: A string representing the default value, or `null`. +**Possible inputs**: A string representing the default value, or `''`. **Example of `spec:inputs:default`**: @@ -735,7 +823,7 @@ spec: user: default: 'test-user' flags: - default: null + default: '' --- # The pipeline configuration would follow... @@ -887,7 +975,8 @@ The following topics explain how to use keywords to configure CI/CD pipelines. ### `after_script` -Use `after_script` to define an array of commands that run after each job, including failed jobs. +Use `after_script` to define an array of commands that run after a job's `script` section, including failed jobs with failure type of `script_failure`. +`after_script` commands do not run after [other failure types](#retrywhen). **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1077,6 +1166,8 @@ link outside it. [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +CI/CD variables [are supported](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). + **Example of `artifacts:paths`**: ```yaml @@ -1272,15 +1363,7 @@ job: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223273) in GitLab 13.8 [with a flag](../../user/feature_flags.md) named `non_public_artifacts`, disabled by default. > - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/322454) in GitLab 15.10. Artifacts created with `artifacts:public` before 15.10 are not guaranteed to remain private after this update. -> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/294503) in GitLab 16.7. Rolled out and removed a feature flag named `non_public_artifacts` - -WARNING: -On self-managed GitLab, by default this feature is not available. To make it available, -an administrator can [enable the feature flag](../../administration/feature_flags.md) named `non_public_artifacts`. On -GitLab.com, this feature is not available. Due to [issue 413822](https://gitlab.com/gitlab-org/gitlab/-/issues/413822), -the keyword can be used when the feature flag is disabled, but the feature does not work. -Do not attempt to use this feature when the feature flag is disabled, and always test -with non-production data first. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/294503) in GitLab 16.7. Feature flag `non_public_artifacts` removed. Use `artifacts:public` to determine whether the job artifacts should be publicly available. @@ -1593,7 +1676,7 @@ use the new cache, instead of rebuilding the dependencies. **Additional details**: - The cache `key` is a SHA computed from the most recent commits -that changed each listed file. + that changed each listed file. If neither file is changed in any commits, the fallback key is `default`. ##### `cache:key:prefix` @@ -1909,8 +1992,8 @@ to select a specific site profile and scanner profile. **Related topics**: -- [Site profile](../../user/application_security/dast/proxy-based.md#site-profile). -- [Scanner profile](../../user/application_security/dast/proxy-based.md#scanner-profile). +- [Site profile](../../user/application_security/dast/on-demand_scan.md#site-profile). +- [Scanner profile](../../user/application_security/dast/on-demand_scan.md#scanner-profile). ### `dependencies` @@ -2468,7 +2551,8 @@ image: #### `image:docker` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27919) in GitLab 16.7. Requires GitLab Runner 16.7 or later. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27919) in GitLab 16.7. Requires GitLab Runner 16.7 or later. +> - `user` input option [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137907) in GitLab 16.8. Use `image:docker` to pass options to the Docker executor of a GitLab Runner. @@ -2481,6 +2565,7 @@ A hash of options for the Docker executor, which can include: - `platform`: Selects the architecture of the image to pull. When not specified, the default is the same platform as the host runner. +- `user`: Specify the username or UID to use when running the container. **Example of `image:docker`**: @@ -2491,11 +2576,13 @@ arm-sql-job: name: super/sql:experimental docker: platform: arm64/v8 + user: dave ``` **Additional details**: - `image:docker:platform` maps to the [`docker pull --platform` option](https://docs.docker.com/engine/reference/commandline/pull/#options). +- `image:docker:user` maps to the [`docker run --user` option](https://docs.docker.com/engine/reference/commandline/run/#options). #### `image:pull_policy` @@ -2620,7 +2707,8 @@ job2: ### `interruptible` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32022) in GitLab 12.3. +> - Support for `trigger` jobs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412473) in GitLab 16.8 [with a flag](../../administration/feature_flags.md) named `ci_workflow_auto_cancel_on_new_commit`. Disabled by default. Use `interruptible` to configure the [auto-cancel redundant pipelines](../pipelines/settings.md#auto-cancel-redundant-pipelines) feature to cancel a job before it completes if a new pipeline on the same ref starts for a newer commit. If the feature @@ -2685,6 +2773,12 @@ In this example, a new pipeline causes a running pipeline to be: a pipeline to allow users to manually prevent a pipeline from being automatically cancelled. After a user starts the job, the pipeline cannot be canceled by the **Auto-cancel redundant pipelines** feature. +- When using `interruptible` with a [trigger job](#trigger): + - The triggered downstream pipeline is never affected by the trigger job's `interruptible` configuration. + - If [`workflow:auto_cancel`](#workflowauto_cancelon_new_commit) is set to `conservative`, + the trigger job's `interruptible` configuration has no effect. + - If [`workflow:auto_cancel`](#workflowauto_cancelon_new_commit) is set to `interruptible`, + a trigger job with `interruptible: true` can be automatically cancelled. ### `needs` @@ -4203,6 +4297,34 @@ job: vault: production/db/password@ops # Translates to secret: `ops/data/production/db`, field: `password` ``` +#### `secrets:gcp_secret_manager` + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11739) in GitLab 16.8 and GitLab Runner 16.8. + +Use `secrets:gcp_secret_manager` to specify secrets provided by [GCP Secret Manager](https://cloud.google.com/security/products/secret-manager). + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- `name`: Name of the secret. +- `version`: Version of the secret. + +**Example of `secrets:gcp_secret_manager`**: + +```yaml +job: + secrets: + DATABASE_PASSWORD: + gcp_secret_manager: + name: 'test' + version: 2 +``` + +**Related topics**: + +- [Use GCP Secret Manager secrets in GitLab CI/CD](../secrets/gcp_secret_manager.md). + #### `secrets:azure_key_vault` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271271) in GitLab 16.3 and GitLab Runner 16.3. @@ -4350,7 +4472,8 @@ In this example, GitLab launches two containers for the job: #### `services:docker` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27919) in GitLab 16.7. Requires GitLab Runner 16.7 or later. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27919) in GitLab 16.7. Requires GitLab Runner 16.7 or later. +> - `user` input option [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137907) in GitLab 16.8. Use `services:docker` to pass options to the Docker executor of a GitLab Runner. @@ -4363,6 +4486,7 @@ A hash of options for the Docker executor, which can include: - `platform`: Selects the architecture of the image to pull. When not specified, the default is the same platform as the host runner. +- `user`: Specify the username or UID to use when running the container. **Example of `services:docker`**: @@ -4374,11 +4498,13 @@ arm-sql-job: - name: super/sql:experimental docker: platform: arm64/v8 + user: dave ``` **Additional details**: - `services:docker:platform` maps to the [`docker pull --platform` option](https://docs.docker.com/engine/reference/commandline/pull/#options). +- `services:docker:user` maps to the [`docker run --user` option](https://docs.docker.com/engine/reference/commandline/run/#options). #### `services:pull_policy` diff --git a/doc/ci/yaml/inputs.md b/doc/ci/yaml/inputs.md index 0869be6da9f..18dcb865c06 100644 --- a/doc/ci/yaml/inputs.md +++ b/doc/ci/yaml/inputs.md @@ -89,7 +89,7 @@ spec: default: true --- -"$[[ job-prefix ]]-scan-website": +"$[[ inputs.job-prefix ]]-scan-website": stage: $[[ inputs.job-stage ]] script: - echo "scanning website -e $[[ inputs.environment ]] -c $[[ inputs.concurrency ]] -v $[[ inputs.version ]]" @@ -233,11 +233,18 @@ Only variables you can [use with the `include` keyword](includes.md#use-variable Example: ```yaml -$[[ inputs.test | expand_vars ]] +spec: + inputs: + test: + default: 'test $MY_VAR' +--- + +test-job: + script: echo $[[ inputs.test | expand_vars ]] ``` -Assuming the value of `inputs.test` is `test $MY_VAR`, and the variable `$MY_VAR` is unmasked -with a value of `my value`, then the output would be `test my value`. +In this example, if `$MY_VAR` is unmasked (exposed in job logs) with a value of `my value`, then the input +would expand to `test my value`. #### `truncate` @@ -259,3 +266,56 @@ $[[ inputs.test | truncate(3,5) ]] ``` Assuming the value of `inputs.test` is `0123456789`, then the output would be `34567`. + +## Troubleshooting + +### YAML syntax errors when using `inputs` + +[CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions) +in `rules:if` expect a comparison of a CI/CD variable with a string, otherwise +[a variety of syntax errors could be returned](../jobs/job_control.md#this-gitlab-ci-configuration-is-invalid-for-variable-expressions). + +You must ensure that expressions remain properly formatted after input values are +inserted into the configuration, which might require the use of additional quote characters. + +For example: + +```yaml +spec: + inputs: + branch: + default: $CI_DEFAULT_BRANCH +--- + +job-name: + rules: + - if: $CI_COMMIT_REF_NAME == $[[ inputs.branch ]] +``` + +In this example: + +- Using `include: inputs: branch: $CI_DEFAULT_BRANCH` is valid. The `if:` clause evaluates to + `if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH`, which is a valid variable expression. +- Using `include: inputs: branch: main` is **invalid**. The `if:` clause evaluates to + `if: $CI_COMMIT_REF_NAME == main`, which is invalid because `main` is a string but is not quoted. + +Alternatively, add quotes to resolve some variable expression issues. For example: + +```yaml +spec: + inputs: + environment: + default: "$ENVIRONMENT" +--- + +$[[ inputs.environment | expand_vars ]] job: + script: echo + rules: + - if: '"$[[ inputs.environment1 | expand_vars ]]" == "production"' +``` + +In this example, quoting the input block and also the entire variable expression +ensures valid `if:` syntax after the input is evaluated. The internal and external quotes +in the expression must not be the same character. You can use `"` for the internal quotes +and `'` for the external quotes, or the inverse. On the other hand, the job name does +not require any quoting. diff --git a/doc/development/activitypub/actor.md b/doc/development/activitypub/actor.md deleted file mode 100644 index 1d10e421df7..00000000000 --- a/doc/development/activitypub/actor.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'actors/index.md' -remove_date: '2023-12-08' ---- - -This document was moved to [another location](actors/index.md). - -<!-- This redirect file can be deleted after <2023-12-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/activitypub/actors/index.md b/doc/development/activitypub/actors/index.md index 6d82e79b9a0..e0367d71871 100644 --- a/doc/development/activitypub/actors/index.md +++ b/doc/development/activitypub/actors/index.md @@ -72,7 +72,7 @@ render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) ``` - `outbox` is the endpoint where to find the activities feed for this -actor. + actor. - `inbox` is where to POST to subscribe to the feed. Not yet implemented, so pass `nil`. ## Outbox page diff --git a/doc/development/ai_features.md b/doc/development/ai_features.md deleted file mode 100644 index a952d8f2804..00000000000 --- a/doc/development/ai_features.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'ai_features/index.md' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](ai_features/index.md). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/ai_features/index.md b/doc/development/ai_features/index.md index e99a49f3bb9..f8680ef91c9 100644 --- a/doc/development/ai_features/index.md +++ b/doc/development/ai_features/index.md @@ -48,7 +48,7 @@ To implement a new AI action, connect to the preferred AI provider. You can conn All AI features are experimental. -## Test AI features locally +## Test SaaS-only AI features locally **One-line setup** @@ -97,11 +97,11 @@ In order to obtain a GCP service key for local development, follow the steps bel - Create a sandbox GCP project by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group GCP project by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). - If you are using an individual GCP project, you may also need to enable the Vertex AI API: - 1. Visit [welcome page](https://console.cloud.google.com/welcome), choose your project (e.g. jdoe-5d23dpe). - 1. Go to **APIs & Services > Enabled APIs & services**. - 1. Select **+ Enable APIs and Services**. - 1. Search for `Vertex AI API`. - 1. Select **Vertex AI API**, then select **Enable**. + 1. Visit [welcome page](https://console.cloud.google.com/welcome), choose your project (e.g. jdoe-5d23dpe). + 1. Go to **APIs & Services > Enabled APIs & services**. + 1. Select **+ Enable APIs and Services**. + 1. Search for `Vertex AI API`. + 1. Select **Vertex AI API**, then select **Enable**. - Install the [`gcloud` CLI](https://cloud.google.com/sdk/docs/install) - Authenticate locally with GCP using the [`gcloud auth application-default login`](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command. - Open the Rails console. Update the settings to: @@ -166,6 +166,132 @@ end View [guidelines](duo_chat.md) for working with GitLab Duo Chat. +## Test AI features with AI Gateway locally + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11251) in GitLab 16.8. + +In order to develop an AI feature that is compatible with both SaaS and Self-managed GitLab instances, +the feature must request to the [AI Gateway](../../architecture/blueprints/ai_gateway/index.md) instead of directly requesting to the 3rd party model providers. +Therefore, a different setup is required from the [SaaS-only AI features](#test-saas-only-ai-features-locally). + +### Setup + +1. Set up AI Gateway: + 1. [Install it](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally). + 1. Ensure that the following environment variables are set in the `.env` file: + + ```shell + AIGW_AUTH__BYPASS_EXTERNAL=true + ANTHROPIC_API_KEY="[REDACTED]" # IMPORTANT: Ensure you use Corp account. See https://gitlab.com/gitlab-org/gitlab/-/issues/435911#note_1701762954. + AIGW_VERTEX_TEXT_MODEL__PROJECT="[REDACTED]" + ``` + + 1. Run `poetry run ai_gateway`. + 1. Visit OpenAPI playground (`http://0.0.0.0:5052/docs`), try an endpoint (e.g. `/v1/chat/agent`) and make sure you get a successful response. + If something went wrong, check `modelgateway_debug.log` if it contains error information. +1. Setup GitLab Development Kit (GDK): + 1. [Install it](https://gitlab.com/gitlab-org/gitlab-development-kit#installation). + 1. [Set up `gdk.test` hostname](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#set-up-gdktest-hostname). + 1. [Activate GitLab Enterprise license](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#use-gitlab-enterprise-features) (e.g. Ultimate). + 1. Export these environment variables in the same terminal session with `gdk start`: + + ```shell + export CODE_SUGGESTIONS_BASE_URL=http://0.0.0.0:5052 # URL to the local AI Gateway instance + export LLM_DEBUG=1 # Enable debug logging + ``` + + Alternatively, you can create an `env.runit` file in the root of your GDK with the above snippet. + 1. Enable the following feature flags via `gdk rails console`: + + ```ruby + # NOTE: This feature flag name might be changed. See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140352. + ::Feature.enable(:ai_global_switch) + + # This is to request to AI Gateway instead of built-in Anthropic client. See https://gitlab.com/gitlab-org/gitlab/-/issues/433213 for more info. + ::Feature.enable(:gitlab_duo_chat_requests_to_ai_gateway) + ``` + + 1. Create a dummy access token via `gdk rails console` OR skip this step and setup GitLab or Customer Dot as OIDC provider (See the following section): + + ```ruby + # Creating dummy token, and this will work as long as `AIGW_AUTH__BYPASS_EXTERNAL=true` in AI Gateway. + ::Ai::ServiceAccessToken.create!(token: 'dummy', expires_at: 1.month.from_now) + ``` + + 1. Ensure GitLab-Rails can talk to the AI Gateway. Run `gdk rails console` and execute: + + ```ruby + user = User.first + Gitlab::Llm::AiGateway::Client.new(user).stream(prompt: "\n\nHuman: Hi, how are you?\n\nAssistant:") + ``` + +#### Verify the setup with GraphQL + +1. Visit [GraphQL explorer](../../api/graphql/index.md#interactive-graphql-explorer). +1. Execute the `aiAction` mutation. Here is an example: + + ```graphql + mutation { + aiAction( + input: { + chat: { + resourceId: "gid://gitlab/User/1", + content: "Hello" + } + } + ){ + requestId + errors + } + } + ``` + +1. (GitLab Duo Chat only) Execute the following query to fetch the response: + + ```graphql + query { + aiMessages { + nodes { + requestId + content + role + timestamp + chunkId + errors + } + } + } + ``` + + If you can't fetch the response, check `graphql_json.log`, `sidekiq_json.log`, `llm.log` or `modelgateway_debug.log` if it contains error information. + +### Use GitLab as OIDC provider in AI Gateway + +1. Reconfigure AI Gateway: + 1. Additionally, ensure that the following environment variables are set in the `.env` file: + + ```shell + AIGW_GITLAB_URL="http://gdk.test:3000/" + AIGW_GITLAB_API_URL="http://gdk.test:3000/api/v4/" + AIGW_AUTH__BYPASS_EXTERNAL=False + ``` + + 1. Restart AI Gateway. +1. Reconfigure GitLab Development Kit (GDK): + 1. Additionally, export the following environment variables: + + ```shell + export GITLAB_SIMULATE_SAAS=1 # Simulate a SaaS instance. See https://docs.gitlab.com/ee/development/ee_features.html#simulate-a-saas-instance. + ``` + + 1. Restart GDK. + +### Use Customer Dot as OIDC provider in AI Gateway + +1. AI Gateway: + 1. Ensure `AIGW_CUSTOMER_PORTAL_BASE_URL` in the `.env` file points to your Customer Dot URL. + 1. Restart + ## Experimental REST API Use the [experimental REST API endpoints](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/experimentation) to quickly experiment and prototype AI features. @@ -210,7 +336,7 @@ As an example, assume we want to build an "explain code" action. To do this, we ```graphql mutation { - aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log()" }}) { + aiAction(input: {explainCode: {resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log() }" }}) { clientMutationId } } diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index cfe82fe9b81..606d8c77432 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -178,7 +178,7 @@ Breaking changes are: allowed so long as all scalar type fields of the object continue to serialize in the same way. - Raising the [complexity](#max-complexity) of a field or complexity multipliers in a resolver. - Changing a field from being _not_ nullable (`null: false`) to nullable (`null: true`), as -discussed in [Nullable fields](#nullable-fields). + discussed in [Nullable fields](#nullable-fields). - Changing an argument from being optional (`required: false`) to being required (`required: true`). - Changing the [max page size](#page-size-limit) of a connection. - Lowering the global limits for query complexity and depth. @@ -1515,8 +1515,8 @@ To find the parent object in your `Presenter` class: ``` 1. Declare your field's method in your Presenter class and have it accept the `parent` keyword argument. -This argument contains the parent **GraphQL context**, so you have to access the parent object with -`parent[:parent_object]` or whatever key you used in your `Resolver`: + This argument contains the parent **GraphQL context**, so you have to access the parent object with + `parent[:parent_object]` or whatever key you used in your `Resolver`: ```ruby # in ChildPresenter diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index d5a011e7e55..94dbcea514b 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -18,7 +18,7 @@ is also available on YouTube. Auto DevOps builds on top of GitLab CI/CD to create an automatic pipeline based on your project contents. When Auto DevOps is enabled for a project, the user does not need to explicitly include any pipeline configuration -through a [`.gitlab-ci.yml` file](../ci/yaml/index.md). +through a [`.gitlab-ci.yml` file](../ci/index.md#the-gitlab-ciyml-file). In the absence of a `.gitlab-ci.yml` file, the [Auto DevOps CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 6c0bed8e204..752bcbbac19 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -12,7 +12,7 @@ NOTE: To use direct transfer, ensure your GitLab installation is accessible from [GitLab IP addresses](../user/gitlab_com/index.md#ip-range) and has a public DNS entry. -[Group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) is the +[Group migration by direct transfer](../user/group/import/index.md) is the evolution of migrating groups and projects using file exports. The goal is to have an easier way for the user to migrate a whole group, including projects, from one GitLab instance to another. @@ -36,8 +36,8 @@ idea is to have one ETL pipeline for each relation to be imported. ### API -The current [project](../user/project/settings/import_export.md) and -[group](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so +The current [project](../user/project/settings/import_export.md#migrate-projects-by-uploading-an-export-file) and +[group](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so they require an export step to generate the file to be imported. Group migration by direct transfer leverages the [GitLab API](../api/rest/index.md) to speed the migration. diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index ccd952f586c..3832ba182b6 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -6,7 +6,7 @@ info: Any user with at least the Maintainer role can merge updates to this conte # Documenting the `.gitlab-ci.yml` keywords -The [CI/CD YAML reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. +The [CI/CD YAML syntax reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. The reference information should be kept as simple as possible, and expanded details and examples should be documented on other pages. diff --git a/doc/development/cicd/components.md b/doc/development/cicd/components.md new file mode 100644 index 00000000000..56ab5a24bd1 --- /dev/null +++ b/doc/development/cicd/components.md @@ -0,0 +1,80 @@ +--- +stage: Verify +group: Pipeline Authoring +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Development guide for GitLab CI/CD components + +This document explains how to develop [CI/CD components](../../ci/components/index.md) that are maintained by GitLab. + +The official location for all GitLab-maintained component projects is the [`gitlab.com/components`](https://gitlab.com/components) group. +This group contains all components that are designed to be generic, served to all GitLab users, and maintained by GitLab. + +A component project can initially be created under a different group (for example `gitlab-org`) +but they need to be moved into the `components` group before the first version gets published to the catalog. + +Components that are for GitLab internal use only, for example specific to `gitlab-org/gitlab` project, should be +implemented under `gitlab-org` group. + +Component projects that are expected to be published in the [CI/CD catalog](../../ci/components/index.md#cicd-catalog) +should first be dogfooded it to ensure we stay on top of the project quality and have first-hand +experience with it. + +## Define ownership + +GitLab-maintained components are trusted by the community and require a high degree of quality and timely maintenance. +Components must be kept up to date, monitored for security vulnerabilities, and bugs fixed. + +Each component project must have a set of owners and maintainers that are also domain experts. +Experts can be from any department in GitLab, from Engineering to Support, Customer Success, and Developer Relations. + +If a component is related to a GitLab feature (for example Secret Detection), the team that owns the +feature category or is most closely related to it should maintain the project. + +The component project can be created by a separate team or individual initially but it must be transitioned +to a set of owners before the first version gets published to the catalog. + +The `README.md` file in the project repository must indicate the main owners of the project so that +they can be contacted by the wider community if needed. + +NOTE: +If a set of project owners cannot be guaranteed or the components cannot be dogfooded, we strongly recommend +not creating a GitLab-maintained component project and instead let the wider community fulfill the demand +in the catalog. + +## Development process + +1. Create a project under [`gitlab.com/components`](https://gitlab.com/components) + or ask one of the group owners to create an empty project for you. +1. Follow the [standard guide for creating components](../../ci/components/index.md). +1. Add a concise project description that clearly describes the capabilities offered by the component project. +1. Ensure that the [best practices](../../ci/components/index.md#best-practices) are followed. +1. Use [semantic versioning](https://semver.org) in the form `MAJOR.MINOR` or `MAJOR.MINOR.PATCH`. +1. Add a `LICENSE.md` file with the MIT license. +1. The project must have a `.gitlab-ci.yml` file that: + - Validates all the components in the project correctly. + - Contains a `release` job to publish newly released tags to the catalog. +1. Ensure that the `README.md` contains at least the sections below (for example, see the [Code quality component](https://gitlab.com/components/code-quality)): + - **Overview**: The capabilities offered by the component project. + - **Components**: Sub-sections for each component, each with: + - **Usage**: Examples with and without inputs (when optional). + - **Inputs**: A table showing the input names, types, default values (if any) and descriptions. + - **Variables** (when applicable): The variable names, possible values, and descriptions. + - **Contribute**: Notes and how to get in touch with the maintainers. + Usually the contribution process should follow the [official guide](../../ci/components/index.md). +1. Upload the [official avatar image](img/avatar_component_project.png) to the component project. + +## Review and contribution process + +It's possible that components in the project have a related [CI/CD template](templates.md) in the GitLab codebase. +In that case we need to cross link the component project and CI/CD template: + +- Add a comment in the CI/CD template with the location of the related component project. +- Add a section in the `README.md` of the component project with the location of the existing CI/CD template. + +When changes are applied to these components, check whether we can integrate the changes in the CI/CD template too. +This might not be possible due to the rigidity of versioning in CI/CD templates. + +Ping [`@gitlab-org/maintainers/ci-components`](https://gitlab.com/groups/gitlab-org/maintainers/ci-components/-/group_members?with_inherited_permissions=exclude) +for reviews to ensure that the components are written in consistent style and follow the best practices. diff --git a/doc/development/cicd/configuration.md b/doc/development/cicd/configuration.md new file mode 100644 index 00000000000..60f55174651 --- /dev/null +++ b/doc/development/cicd/configuration.md @@ -0,0 +1,100 @@ +--- +stage: Verify +group: Pipeline Authoring +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Contribute to the CI/CD configuration + +## Glossary + +- **CI/CD configuration**: The YAML file that defines the CI/CD configuration for a project. +- **keyword**: Each keyword in the CI/CD configuration. +- **entry**: An `Entry` class that represents a keyword in the CI/CD configuration. + +Not every keyword in the CI/CD configuration is represented by an `Entry` class. +We create `Entry` classes for keywords that have a complex structure or reusable parts. + +For example; + +- The `image` keyword is represented by the [`Entry::Image`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/image.rb) class. +- The `name` subkeyword of the `image` keyword is not represented by an `Entry` class. +- The `pull_policy` subkeyword of the `image` keyword is represented by the [`Entry::PullPolicy`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/pull_policy.rb) class. + +## Adding New Keywords + +CI config keywords are added in the [`lib/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/config/entry) directory. +For EE-specific changes, use the [`ee/lib/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/ci/config/entry) +or [`ee/lib/ee/gitlab/ci/config/entry`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/ee/gitlab/ci/config/entry) directory. + +### Inheritance + +An entry is represented by a class that inherits from; + +- `Entry::Node`: for simple keywords. + (e.g. [`Entry::Stage`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/stage.rb)) +- `Entry::Simplifiable`: for keywords that have multiple structures. + For example, [`Entry::Retry`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/retry.rb) can be a simple number or a hash configuration. +- `Entry::ComposableArray`: for keywords that have a list of single-type sub-elements. + For example, [`Entry::Includes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/includes.rb) has a list of `Entry::Include` elements. +- `Entry::ComposableHash`: for keywords that have single-type sub-elements with user-defined keys. + For example, [`Entry::Variables`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/config/entry/variables.rb) has a list of `Entry::Variable` elements with user-defined keys. + +### Helper Classes + +The following helper classes are available for use in entries: + +- `Entry::Validatable`: Enables the `validations` block in an entry class and provides validations. +- `Entry::Attributable`: Enables the `attributes` method in an entry class. It creates these methods for each attribute; `xxx`, `has_xxx?`, `has_xxx_value?`. +- `Entry::Configurable`: Enables the `entry` method in an entry class. It creates these methods for each entry; `xxx_defined?`, `xxx_entry`, `xxx_value`. + +### The `value` Method + +The `value` method is the main method of an entry class. It returns the actual value of the entry. +By default, from the `Entry::Node` class, the `value` method returns the hash configuration of the entry unless it has nested entries. +It can be useful for simple entries. For example, `Entry::Paths` has an array of strings as its value. So, it can return the array of strings directly. + +In some keywords, we override the `value` method. In this method, we return what and how we want to return from the entry. +The usage of `Entry::Attributable` and `Entry::Configurable` may have a significant role here. For example, +in `Entry::Secret`, we have this; + +```ruby +attributes %i[vault file token].freeze + +entry :vault, Entry::Vault::Secret +entry :file, ::Gitlab::Config::Entry::Boolean + +def value + { + vault: vault_value, + file: file_value, + token: token + }.compact +end +``` + +- `vault_value` is the value of the nested `vault` entry. +- `file_value` is the value of the nested `file` entry. +- `token` is the value of the basic `token` attribute. + +**It is important** that we should always use the `xxx_value` method to get the value of a nested entry. + +## Feature Flag Usage + +When adding new CI/CD configuration keywords, it is important to use feature flags to control the rollout of the change. +This allows us to test the change in production without affecting all users. For more information, see the [feature flags documentation](../feature_flags/index.md). + +### Feature Flag Actor + +In entry classes, we have no access to the current project or user. However, it's discouraged to use feature flags without [an actor](../feature_flags/index.md#feature-actors). +To solve this problem, we have three options; + +1. Use `Feature.enabled?(:feature_flag, Feature.current_request)`. +1. Use `YamlProcessor::FeatureFlags.enabled?(:feature_flag)` +1. Do not use feature flags in entry classes and use them in other parts of the code. + +## Testing and Validation + +When adding or modifying an entry, the corresponding spec file must be either added or updated. +Besides, to have a fully integrated test, it's also important to add/modify tests in the `spec/lib/gitlab/ci/yaml_processor_spec.rb` file or +the files in `spec/lib/gitlab/ci/yaml_processor/test_cases/*` directory. diff --git a/doc/development/cicd/img/avatar_component_project.png b/doc/development/cicd/img/avatar_component_project.png Binary files differnew file mode 100644 index 00000000000..e5c20d108fa --- /dev/null +++ b/doc/development/cicd/img/avatar_component_project.png diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 18781f9315a..a8dfdeb30a3 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -9,10 +9,12 @@ info: Any user with at least the Maintainer role can merge updates to this conte Development guides that are specific to CI/CD are listed here: - If you are creating new CI/CD templates, read [the development guide for GitLab CI/CD templates](templates.md). -- If you are adding a new keyword or changing the CI schema, check the [CI schema guide](schema.md) +- If you are adding a new keyword or changing the CI schema, refer to the following guides: + - [The CI configuration guide](configuration.md) + - [The CI schema guide](schema.md) See the [CI/CD YAML reference documentation guide](cicd_reference_documentation_guide.md) -to learn how to update the [reference page](../../ci/yaml/index.md). +to learn how to update the [CI/CD YAML syntax reference page](../../ci/yaml/index.md). ## Examples of CI/CD usage diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md index 0c0c0f3cc45..b534fede6e9 100644 --- a/doc/development/cicd/pipeline_wizard.md +++ b/doc/development/cicd/pipeline_wizard.md @@ -138,7 +138,7 @@ is planned to add the ability to create a MR from here. ### Events - `done` - Emitted after the file has been committed. Use this to redirect the -user to the pipeline, for example. + user to the pipeline, for example. ### Template file location diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md index e9a0b93b5f3..c24b7d21286 100644 --- a/doc/development/cicd/schema.md +++ b/doc/development/cicd/schema.md @@ -30,7 +30,7 @@ a step-by-step introduction on how to work with JSON schemas. The CI/CD schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json). It contains all the keywords available for authoring CI/CD configuration files. -Check the [keyword reference](../../ci/yaml/index.md) for a comprehensive list of +Check the [CI/CD YAML syntax reference](../../ci/yaml/index.md) for a comprehensive list of all available keywords. All keywords are defined under `definitions`. We use these definitions as diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index a2b490b9106..bd3023ebf8d 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -13,7 +13,7 @@ we encourage team members to create [CI/CD components](../../ci/components/index for the catalog. This transition enhances the modularity and maintainability of our shared CI/CD resources, and avoids the complexities of contributing new CI/CD templates. If you need to update an existing template, you must also update the matching CI/CD component. -If no component exists that matches the CI/CD template yet, consider creating the matching component. +If no component exists that matches the CI/CD template yet, consider [creating the matching component](components.md). This ensures that template and component functionality remain in sync, aligning with our new development practices. diff --git a/doc/development/code_owners/index.md b/doc/development/code_owners/index.md index 45c632d5adc..b8e99475dd3 100644 --- a/doc/development/code_owners/index.md +++ b/doc/development/code_owners/index.md @@ -53,12 +53,27 @@ namespace. Code Owners is an EE-only feature, so the files only exist in the `./ have been changed when a user pushes to a protected branch with `require_code_owner_approval` enabled. - Defined in `./ee/lib/gitlab/code_owners/validator.rb`. +## Where Code Owners sits in the Git access check execution order + +`Gitlab::Checks::DiffCheck#file_paths_validations` returns either an empty array, or an array with a single member of the results of `#lfs_file_locks_validation` if LFS is enabled and file locks are present. The return result of `#validate_code_owners` in the EE version of this file is inserted at the end of this list in the `EE::Gitlab::Checks::DiffCheck#file_paths_validations`. LFS checks are performed before Code Owners checks. + +These checks are executed after those listed in `#validations_for_path`, which exists only in the EE version, and include `#path_locks_validation` and `#file_name_validation`. This means that checks for Path Locks precede checks for Code Owners in the flow. + +The check order is as follows in `EE` (only LFS exists as a non-EE feature): + +- Path Locks +- File Names + - Blocks files containing secrets for example `id_rsa` + - Blocks files matching the `PushRule#file_name_regex` +- LFS File Locks +- Code Owners + ## Related models ### `ProtectedBranch` The `ProtectedBranch` model is defined in `app/models/protected_branch.rb` and -extended in `ee/app/ee/models/protected_branch.rb`. The EE version includes a column +extended in `ee/app/models/concerns/ee/protected_branch.rb`. The EE version includes a column named `require_code_owner_approval` which prevents changes from being pushed directly to the branch being protected if the file is listed in `CODEOWNERS`. @@ -108,7 +123,9 @@ This service is defined in `services/merge_requests/sync_code_owner_approval_rul These flowcharts should help explain the flow from the controllers down to the models for different features. -### Push changes to a protected branch with `require_code_owner_approval` enabled +Note that many of the Code Owners implementations exist in the `EE` variants of the classes. + +### Push changes to a protected branch with `require_code_owner_approval` enabled, over SSH ```mermaid graph TD @@ -120,6 +137,19 @@ graph TD Gitlab::CodeOwners::Loader --> Gitlab::CodeOwners::Entry ``` +### Push changes to a protected branch with `require_code_owner_approval` enabled, over HTTPS + +```mermaid +graph TD + Repositories::GitHttpController --> Gitlab::GlRepository + Gitlab::GlRepository --> Gitlab::GitAccessProject + Gitlab::GitAccessProject --> Gitlab::Checks::DiffCheck + Gitlab::Checks::DiffCheck --> Gitlab::CodeOwners::Validator + Gitlab::CodeOwners::Validator --> ProtectedBranch + Gitlab::CodeOwners::Validator --> Gitlab::CodeOwners::Loader + Gitlab::CodeOwners::Loader --> Gitlab::CodeOwners::Entry +``` + ### Sync code owner rules to merge request approval rules ```mermaid diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 84d2537d058..cad71d4b843 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -376,7 +376,7 @@ Avoid: [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). - Requesting maintainer reviews of merge requests with failed tests. If the tests are failing and you have to request a review, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable -through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. + through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. This saves reviewers time and helps authors catch mistakes earlier. @@ -412,7 +412,7 @@ that it meets all requirements, you should: - Select **Approve**. - `@` mention the author to generate a to-do notification, and advise them that their merge request has been reviewed and approved. - Request a review from a maintainer. Default to requests for a maintainer with [domain expertise](#domain-experts), -however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. + however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. - Remove yourself as a reviewer. ### The responsibility of the maintainer @@ -580,7 +580,7 @@ experience, refactors the existing code). Then: optionally resolve within the merge request or follow-up at a later stage. - There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. - Ensure there are no open dependencies. Check [linked issues](../user/project/issues/related_issues.md) for blockers. Clarify with the authors -if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/dependencies.md). + if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/dependencies.md). - After a round of line notes, it can be helpful to post a summary note such as "Looks good to me", or "Just a couple things to address." - Let the author know if changes are required following your review. diff --git a/doc/development/code_suggestions/index.md b/doc/development/code_suggestions/index.md index bdf3bcdd520..2bf36664437 100644 --- a/doc/development/code_suggestions/index.md +++ b/doc/development/code_suggestions/index.md @@ -12,7 +12,7 @@ The recommended setup for locally developing and debugging Code Suggestions is t - IDE Extension (e.g. VS Code Extension) - Main application configured correctly -- [Model gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist) +- [AI Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist) This should enable everyone to see locally any change in an IDE being sent to the main application transformed to a prompt which is then sent to the respective model. @@ -24,26 +24,26 @@ This should enable everyone to see locally any change in an IDE being sent to th 1. Open the extension settings by clicking a small cog icon and select "Extension Settings" option 1. Check a "GitLab: Debug" checkbox. 1. Main Application - 1. Enable Feature Flags ```code_suggestions_completion_api``` and ```code_suggestions_tokens_api``` + 1. Enable Feature Flag ```code_suggestions_tokens_api``` 1. In your terminal, navigate to a `gitlab` inside your `gitlab-development-kit` directory 1. Run `bundle exec rails c` to start a Rails console - 1. Call `Feature.enable(:code_suggestions_completion_api)` and `Feature.enable(:code_suggestions_tokens_api)` from the console + 1. Call `Feature.enable(:code_suggestions_tokens_api)` from the console 1. Run the GDK with ```export CODE_SUGGESTIONS_BASE_URL=http://localhost:5052``` -1. [Setup Model Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally) +1. [Setup AI Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally) 1. Build tree sitter libraries ```poetry run scripts/build-tree-sitter-lib.py``` - 1. Extra .env Changes for all debugging insights - 1. LOG_LEVEL=DEBUG - 1. LOG_FORMAT_JSON=false - 1. LOG_TO_FILE=true + 1. Extra .env changes for all debugging insights + 1. `AIGW_LOGGING__LEVEL=DEBUG` + 1. `AIGW_LOGGING__FORMAT_JSON=false` + 1. `AIGW_LOGGING__TO_FILE=true` 1. Watch the new log file ```modelgateway_debug.log``` , e.g. ```tail -f modelgateway_debug.log | fblog -a prefix -a suffix -a current_file_name -a suggestion -a language -a input -a parameters -a score -a exception``` -### Setup instructions to use staging Model Gateway +### Setup instructions to use staging AI Gateway -When testing interactions with the Model Gateway, you might want to integrate your local GDK -with the deployed staging Model Gateway. To do this: +When testing interactions with the AI Gateway, you might want to integrate your local GDK +with the deployed staging AI Gateway. To do this: 1. You need a [cloud staging license](../../user/project/repository/code_suggestions/self_managed.md#upgrade-gitlab) that has the Code Suggestions add-on, because add-ons are enabled on staging. Drop a note in the `#s_fulfillment` internal Slack channel to request an add-on to your license. See this [handbook page](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee-developer-licenses) for how to request a license for local development. -1. Set environment variables to point customers-dot to staging, and the Model Gateway to staging: +1. Set environment variables to point customers-dot to staging, and the AI Gateway to staging: ```shell export GITLAB_LICENSE_MODE=test @@ -52,5 +52,5 @@ with the deployed staging Model Gateway. To do this: ``` 1. Restart the GDK. -1. Ensure you followed the necessary [steps to enable the Code Suggestions feature](../../user/project/repository/code_suggestions/self_managed.md#gitlab-163-and-later). +1. Ensure you followed the necessary [steps to enable the Code Suggestions feature](../../user/project/repository/code_suggestions/self_managed.md). 1. Test out the Code Suggestions feature by opening the Web IDE for a project. diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 0e922550856..c6027e27310 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -19,12 +19,12 @@ with additions and improvements. As a merge request (MR) author, you must: - Include _Before_ and _After_ -screenshots (or videos) of your changes in the description, as explained in our -[MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful -for all reviewers and can speed up the review process, especially if the changes -are small. + screenshots (or videos) of your changes in the description, as explained in our + [MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful + for all reviewers and can speed up the review process, especially if the changes + are small. - Attach the ~UX label to any merge request that has any user facing changes. This will trigger our -Reviewer Roulette to suggest a UX [reviewer](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs). + Reviewer Roulette to suggest a UX [reviewer](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs). If you are a **team member**: We recommend assigning the Product Designer suggested by the [Reviewer Roulette](../code_review.md#reviewer-roulette) as reviewer. [This helps us](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#benefits) spread work evenly, improve communication, and make our UI more @@ -41,10 +41,10 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. ### Writing -- Follow [Pajamas](https://design.gitlab.com/content/punctuation/) as the primary +- Follow [Pajamas](https://design.gitlab.com/content/ui-text/) as the primary guidelines for UI text and [documentation style guide](../documentation/styleguide/index.md) as the secondary. -- Use clear and consistent [terminology](https://design.gitlab.com/content/terminology/). +- Use clear and consistent terminology. - Check grammar and spelling. - Consider help content and follow its [guidelines](https://design.gitlab.com/usability/contextual-help). - Request review from the [appropriate Technical Writer](https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments), diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 2c8b5b2af20..d4541558d23 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -134,6 +134,14 @@ Lastly, keep the following in mind when submitting merge requests: be merged, as well as some guidance. The maintainers will be open to discussion about how to change the code so it can be approved and merged in the future. +## Tips + +- [Small MRs are the main key to a great review](https://about.gitlab.com/blog/2021/03/18/iteration-and-code-review/). +- Make sure to read our [merge request guidelines for contributors before you start for the first time](merge_request_workflow.md#merge-request-guidelines-for-contributors). +- Automated testing is required. Take your time to understand the different [testing levels](../testing_guide/testing_levels.md#how-to-test-at-the-correct-level) and apply them accordingly. +- Make sure to have a great description that includes steps to reproduce your implementation. +- [Make sure to follow our commit message guidelines](merge_request_workflow.md#commit-messages-guidelines). + ## Closing policy for issues and merge requests - For the criteria for closing issues, see [the Issue Triage handbook page](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#outdated-issues). diff --git a/doc/development/contributing/verify/index.md b/doc/development/contributing/verify/index.md index dbc48121dff..5f9e2e3acb3 100644 --- a/doc/development/contributing/verify/index.md +++ b/doc/development/contributing/verify/index.md @@ -53,7 +53,7 @@ and they serve us and our users well. Some examples of these principles are that If a job fails and we notify a user that it was successful, it can have severe negative consequences. - Feedback needs to be available when a user needs it and data cannot disappear unexpectedly when engineers need it. - It all doesn't matter if the platform is not secure and we -are leaking credentials or secrets. + are leaking credentials or secrets. - When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. - If it is fast, simple to use and has a great UX it will serve our users well. diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index bf75b7c2ab2..57c71cbaec6 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -588,27 +588,27 @@ swap the columns. Swapping is done with post-deployment migration. The exact pro table being converted, but in general it's done in the following steps: 1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched -migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). -If the migration has not completed, the subsequent steps fail anyway. By checking in advance we -aim to have more helpful error message. + migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). + If the migration has not completed, the subsequent steps fail anyway. By checking in advance we + aim to have more helpful error message. 1. Use the `add_bigint_column_indexes` helper method from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module to create indexes with the `bigint` columns that match the existing indexes using the `integer` column. - The helper method is expected to create all required `bigint` indexes, but it's advised to recheck to make sure we are not missing any of the existing indexes. More information about the helper can be found in merge request [135781](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135781). 1. Create foreign keys (FK) using the `bigint` columns that match the existing FK using the -`integer` column. Do this both for FK referencing other tables, and FK that reference the table -that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). + `integer` column. Do this both for FK referencing other tables, and FK that reference the table + that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). 1. Inside a transaction, swap the columns: - 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). - 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) - 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). - 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). - 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). - 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). - - Names of the `bigint` indexes created using `add_bigint_column_indexes` helper can be retrieved by calling - `bigint_index_name` from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module. - 1. Remove old foreign keys (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). + 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) + 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). + 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). + 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). + 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). + - Names of the `bigint` indexes created using `add_bigint_column_indexes` helper can be retrieved by calling + `bigint_index_name` from `Gitlab::Database::MigrationHelpers::ConvertToBigint` module. + 1. Remove old foreign keys (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). diff --git a/doc/development/database/clickhouse/clickhouse_within_gitlab.md b/doc/development/database/clickhouse/clickhouse_within_gitlab.md index a459f89b185..8988ae182f7 100644 --- a/doc/development/database/clickhouse/clickhouse_within_gitlab.md +++ b/doc/development/database/clickhouse/clickhouse_within_gitlab.md @@ -272,3 +272,7 @@ end ``` Additionally, to view the executed ClickHouse queries in web interactions, on the performance bar, next to the `ch` label select the count. + +### Getting help + +For additional information or specific questions, please reach out to the ClickHouse Datastore working group in the `#f_clickhouse` Slack channel, or mention `@gitlab-org/maintainers/clickhouse` in a comment on GitLab.com. diff --git a/doc/development/database/clickhouse/merge_request_analytics.md b/doc/development/database/clickhouse/merge_request_analytics.md index db441ca9051..aa04d9a4fe1 100644 --- a/doc/development/database/clickhouse/merge_request_analytics.md +++ b/doc/development/database/clickhouse/merge_request_analytics.md @@ -100,7 +100,7 @@ LIMIT 1 ## Store merge request data in ClickHouse Several other use cases exist for storing and querying merge request data in -ClickHouse. In this document, we focus on this particular feature. +[ClickHouse](../../../integration/clickhouse.md). In this document, we focus on this particular feature. The core data exists in the `merge_request_metrics` and in the `merge_requests` database tables. Some filters require extra tables to be joined: diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index 8e334e5bb5c..e1a7b56e35e 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -916,16 +916,16 @@ Here's an outline of the steps we take in the recursive CTE query 1. Sort the initial `resultset` according to the `ORDER BY` clause. 1. Pick the top cursor to fetch the record, this is our first record. In the example, -this cursor would be (`2020-01-05`, `3`) for `project_id=9`. + this cursor would be (`2020-01-05`, `3`) for `project_id=9`. 1. We can use (`2020-01-05`, `3`) to fetch the next issue respecting the `ORDER BY` clause -`project_id=9` filter. This produces an updated `resultset`. + `project_id=9` filter. This produces an updated `resultset`. - | `project_ids` | `created_at_values` | `id_values` | - | ------------- | ------------------- | ----------- | - | 2 | 2020-01-10 | 5 | - | 5 | 2020-01-05 | 4 | - | 10 | 2020-01-15 | 7 | - | **9** | **2020-01-06** | **6** | + | `project_ids` | `created_at_values` | `id_values` | + | ------------- | ------------------- | ----------- | + | 2 | 2020-01-10 | 5 | + | 5 | 2020-01-05 | 4 | + | 10 | 2020-01-15 | 7 | + | **9** | **2020-01-06** | **6** | 1. Repeat 1 to 3 with the updated `resultset` until we have fetched `N=20` records. @@ -944,9 +944,9 @@ Example initializer row: | `NULL::issues` | `[9, 2, 5, 10]` | `[...]` | `[...]` | `0` | `NULL` | - The `records` column contains our sorted database records, and the initializer query sets the -first value to `NULL`, which is filtered out later. + first value to `NULL`, which is filtered out later. - The `count` column tracks the number of records found. We use this column to filter out the -initializer row from the result set. + initializer row from the result set. ### Recursive portion of the CTE query @@ -1016,7 +1016,7 @@ As the final step, we need to produce a new row by manipulating the initializer (`data_collector_query` method). Two things happen here: - Read the full row from the DB and return it in the `records` columns. (`result_collector_columns` -method) + method) - Replace the cursor values at the current position with the results from the keyset query. Reading the full row from the database is only one index scan by the primary key. We use the @@ -1091,7 +1091,7 @@ in the PostgreSQL's buffer cache. The optimized `IN` query reads maximum 519 entries (cursor values) from the index: - 500 index-only scans for populating the arrays for each project. The cursor values of the first -record is here. + record is here. - Maximum 19 additional index-only scans for the consecutive records. The optimized `IN` query sorts the array (cursor values per project array) 20 times, which diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index 802b7622592..7192a84508e 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -414,12 +414,12 @@ the queries, where the `issues` table is batched first. When to use `JOINS`: - When there's a 1:1 or 1:N relationship between the tables where we know that the joined record -(almost) always exists. This works well for "extension-like" tables: + (almost) always exists. This works well for "extension-like" tables: - `projects` - `project_settings` - `users` - `user_details` - `users` - `user_statuses` - `LEFT JOIN` works well in this case. Conditions on the joined table need to go to the yielded -relation so the iteration is not affected by the data distribution in the joined table. + relation so the iteration is not affected by the data distribution in the joined table. Example: @@ -555,7 +555,7 @@ table or a range of rows. The scaling and performance characteristics are very s Examples: - Iterate over the table in a specific order (timestamp columns) in combination with a tie-breaker -if column user to sort by does not contain unique values. + if column user to sort by does not contain unique values. - Iterate over the table with composite primary keys. ### Iterate over the issues in a project by creation date @@ -619,8 +619,8 @@ To keep the iteration stable and predictable, avoid updating the columns in the ### Iterate over the `merge_request_diff_commits` table -The `merge_request_diff_commits` table uses a composite primary key (`merge_request_diff_id, -relative_order`), which makes `EachBatch` impossible to use efficiently. +The `merge_request_diff_commits` table uses a composite primary key (`merge_request_diff_id, relative_order`), +which makes `EachBatch` impossible to use efficiently. To paginate over the `merge_request_diff_commits` table, you can use the following snippet: @@ -654,7 +654,7 @@ end ### Order object configuration Keyset pagination works well with simple `ActiveRecord` `order` scopes -([first example](#iterate-over-the-issues-in-a-project-by-creation-date). +([first example](#iterate-over-the-issues-in-a-project-by-creation-date)). However, in special cases, you need to describe the columns in the `ORDER BY` clause (second example) for the underlying keyset pagination library. When the `ORDER BY` configuration cannot be automatically determined by the keyset pagination library, an error is raised. diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 9bb44d2ed71..6b2c4f9d146 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -123,7 +123,7 @@ For the REST API, the `paginate_with_strategies` helper can be used on a relatio In order for keyset pagination to be used, the following conditions must be met: -1. `params[:keyset]` must return `'keyset'` +1. `params[:pagination]` must return `'keyset'` 1. `params[:order_by]` and `params[:sort]` must both appear in the object returned by the `supported_keyset_orderings` class method on the model. In the following example, `Thing` supports keyset pagination when ordering by ID in either ascending or descending order. diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 3003ee970ce..9ecc70c06bc 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -499,7 +499,7 @@ cron job where the schedule depends on the configuration of the GitLab instance. - Non-decomposed GitLab (1 database): invoked every minute. - Decomposed GitLab (2 databases, CI and Main): invoked every minute, cleaning up one database -at a time. For example, the cleanup worker for the main database runs every two minutes. + at a time. For example, the cleanup worker for the main database runs every two minutes. To avoid lock contention and the processing of the same database rows, the worker does not run parallel. This behavior is ensured with a Redis lock. @@ -509,13 +509,13 @@ parallel. This behavior is ensured with a Redis lock. 1. Acquire the Redis lock. 1. Determine which database to clean up. 1. Collect all database tables where the deletions are tracked (parent tables). - - This is achieved by reading the `config/gitlab_loose_foreign_keys.yml` file. - - A table is considered "tracked" when a loose foreign key definition exists for the table and - the `DELETE` trigger is installed. + - This is achieved by reading the `config/gitlab_loose_foreign_keys.yml` file. + - A table is considered "tracked" when a loose foreign key definition exists for the table and + the `DELETE` trigger is installed. 1. Cycle through the tables with an infinite loop. 1. For each table, load a batch of deleted parent records to clean up. 1. Depending on the YAML configuration, build `DELETE` or `UPDATE` (nullify) queries for the -referenced child tables. + referenced child tables. 1. Invoke the queries. 1. Repeat until all child records are cleaned up or the maximum limit is reached. 1. Remove the deleted parent records when all child records are cleaned up. @@ -530,13 +530,13 @@ The inserted record stores the following information about the deleted record: - `fully_qualified_table_name`: name of the database table where the record was located. - `primary_key_value`: the ID of the record, the value is present in the child tables as -the foreign key value. At the moment, composite primary keys are not supported, the parent table -must have an `id` column. + the foreign key value. At the moment, composite primary keys are not supported, the parent table + must have an `id` column. - `status`: defaults to pending, represents the status of the cleanup process. - `consume_after`: defaults to the current time. - `cleanup_attempts`: defaults to 0. The number of times the worker tried to clean up this record. -A non-zero number would mean that this record has many child records and cleaning it up requires -several runs. + A non-zero number would mean that this record has many child records and cleaning it up requires + several runs. #### Database decomposition @@ -692,7 +692,7 @@ The loop-based batch processing is preferred over `EachBatch` for the following - The records in the batch are modified, so the next batch contains different records. - There is always an index on the foreign key column however, the column is usually not unique. -`EachBatch` requires a unique column for the iteration. + `EachBatch` requires a unique column for the iteration. - The record order doesn't matter for the cleanup. Notice that we have two loops. The initial loop processes records with the `SKIP LOCKED` clause. @@ -747,23 +747,23 @@ For example, a project with millions of `ci_builds` records is deleted. The `ci_ is deleted by the loose foreign keys feature. 1. The cleanup worker is scheduled and picks up a batch of deleted `projects` records. The large -project is part of the batch. + project is part of the batch. 1. Deletion of the orphaned `ci_builds` rows has started. 1. The time limit is reached, but the cleanup is not complete. 1. The `cleanup_attempts` column is incremented for the deleted records. 1. Go to step 1. The next cleanup worker continues the cleanup. 1. When the `cleanup_attempts` reaches 3, the batch is re-scheduled 10 minutes later by updating -the `consume_after` column. + the `consume_after` column. 1. The next cleanup worker processes a different batch. We have Prometheus metrics in place to monitor the deleted record cleanup: - `loose_foreign_key_processed_deleted_records`: Number of processed deleted records. When large -cleanup happens, this number would decrease. + cleanup happens, this number would decrease. - `loose_foreign_key_incremented_deleted_records`: Number of deleted records which were not -finished processing. The `cleanup_attempts` column was incremented. + finished processing. The `cleanup_attempts` column was incremented. - `loose_foreign_key_rescheduled_deleted_records`: Number of deleted records that had to be -rescheduled at a later time after 3 cleanup attempts. + rescheduled at a later time after 3 cleanup attempts. Example Thanos query: diff --git a/doc/development/database/maintenance_operations.md b/doc/development/database/maintenance_operations.md index 2701c228b74..fcdf132aa09 100644 --- a/doc/development/database/maintenance_operations.md +++ b/doc/development/database/maintenance_operations.md @@ -18,7 +18,7 @@ There are certain situations in which you might want to disable an index before To disable an index before removing it: 1. Open a [production infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new) -and use the "Production Change" template. + and use the "Production Change" template. 1. Inform the database team in the issue `@gl-database` or in Slack `#database`. 1. Add a step to verify the index is used (this would likely be an `EXPLAIN` command known to use the index). 1. Add the step to disable the index: diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index e453a7f27ea..a8d088a372e 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -936,7 +936,7 @@ I, [2023-07-14T17:08:06.761709 #92505] INFO -- : SELECT set_config('lock_writes I, [2023-07-14T17:08:06.765272 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_pending_states', 'false', false) I, [2023-07-14T17:08:06.768220 #92505] INFO -- : SELECT set_config('lock_writes.ci_build_report_results', 'false', false) [...] -I, [2023-07-14T17:08:06.957294 #92505] INFO -- : TRUNCATE TABLE ci_build_needs, ci_build_pending_states, ci_build_report_results, ci_build_trace_chunks, ci_build_trace_metadata, ci_builds, ci_builds_metadata, ci_builds_runner_session, ci_cost_settings, ci_daily_build_group_report_results, ci_deleted_objects, ci_editor_ai_conversation_messages, ci_freeze_periods, ci_group_variables, ci_instance_variables, ci_job_artifact_states, ci_job_artifacts, ci_job_token_project_scope_links, ci_job_variables, ci_minutes_additional_packs, ci_namespace_mirrors, ci_namespace_monthly_usages, ci_partitions, ci_pending_builds, ci_pipeline_artifacts, ci_pipeline_chat_data, ci_pipeline_messages, ci_pipeline_metadata, ci_pipeline_schedule_variables, ci_pipeline_schedules, ci_pipeline_variables, ci_pipelines, ci_pipelines_config, ci_platform_metrics, ci_project_mirrors, ci_project_monthly_usages, ci_refs, ci_resource_groups, ci_resources, ci_runner_machines, ci_runner_namespaces, ci_runner_projects, ci_runner_versions, ci_runners, ci_running_builds, ci_secure_file_states, ci_secure_files, ci_sources_pipelines, ci_sources_projects, ci_stages, ci_subscriptions_projects, ci_trigger_requests, ci_triggers, ci_unit_test_failures, ci_unit_tests, ci_variables, external_pull_requests, p_ci_builds, p_ci_builds_metadata, p_ci_job_annotations, p_ci_runner_machine_builds, taggings, tags RESTRICT +I, [2023-07-14T17:08:06.957294 #92505] INFO -- : TRUNCATE TABLE ci_build_needs, ci_build_pending_states, ci_build_report_results, ci_build_trace_chunks, ci_build_trace_metadata, ci_builds, ci_builds_metadata, ci_builds_runner_session, ci_cost_settings, ci_daily_build_group_report_results, ci_deleted_objects, ci_freeze_periods, ci_group_variables, ci_instance_variables, ci_job_artifact_states, ci_job_artifacts, ci_job_token_project_scope_links, ci_job_variables, ci_minutes_additional_packs, ci_namespace_mirrors, ci_namespace_monthly_usages, ci_partitions, ci_pending_builds, ci_pipeline_artifacts, ci_pipeline_chat_data, ci_pipeline_messages, ci_pipeline_metadata, ci_pipeline_schedule_variables, ci_pipeline_schedules, ci_pipeline_variables, ci_pipelines, ci_pipelines_config, ci_platform_metrics, ci_project_mirrors, ci_project_monthly_usages, ci_refs, ci_resource_groups, ci_resources, ci_runner_machines, ci_runner_namespaces, ci_runner_projects, ci_runner_versions, ci_runners, ci_running_builds, ci_secure_file_states, ci_secure_files, ci_sources_pipelines, ci_sources_projects, ci_stages, ci_subscriptions_projects, ci_trigger_requests, ci_triggers, ci_unit_test_failures, ci_unit_tests, ci_variables, external_pull_requests, p_ci_builds, p_ci_builds_metadata, p_ci_job_annotations, p_ci_runner_machine_builds, taggings, tags RESTRICT ``` The tasks will first find out the tables that need to be truncated. Truncation will diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index ad641797496..babd51f91a0 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -78,18 +78,18 @@ Execute a standard migration (not a post-migration): - Add a note in the Engineering Week-in-Review document: `table_name` is going to be renamed in N.M. Modifications to this table are not allowed in release N.M and N.M+1. - The helper method uses the standard `rename_table` helper from Rails for renaming the table. - The helper renames the sequence and the indexes. Sometimes it diverges from the standard Rails convention -when naming indexes, so there is a possibility that not all indexes are properly renamed. After running -the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be -renamed manually in a separate migration, which can be also part of the release M.N+1. + when naming indexes, so there is a possibility that not all indexes are properly renamed. After running + the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be + renamed manually in a separate migration, which can be also part of the release M.N+1. - Foreign key columns might still contain the old table name. For smaller tables, follow our [standard column rename process](avoiding_downtime_in_migrations.md#renaming-columns) - Avoid renaming database tables which are using with triggers. - Table modifications (add or remove columns) are not allowed during the rename process. Make sure that all changes to the table happen before the rename migration is started (or in the next release). - As the index names might change, verify that the model does not use bulk insert -(for example, `insert_all` and `upsert_all`) with the `unique_by: index_name` option. -Renaming an index while using these methods may break functionality. + (for example, `insert_all` and `upsert_all`) with the `unique_by: index_name` option. + Renaming an index while using these methods may break functionality. - Modify the model code to point to the new database table. Do this by -renaming the model directly or setting the `self.table_name` variable. + renaming the model directly or setting the `self.table_name` variable. At this point, we don't have applications using the old database table name in their queries. @@ -107,7 +107,7 @@ At this point, we don't have applications using the old database table name in t 1. Additionally the table definition from `TABLES_TO_BE_RENAMED` **must** be removed. -To do so, edit the `TABLES_TO_BE_RENAMED` constant in `lib/gitlab/database.rb`: + To do so, edit the `TABLES_TO_BE_RENAMED` constant in `lib/gitlab/database.rb`: From: diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 70d59603e0c..50188dba14e 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -12,8 +12,8 @@ When adding new columns to store strings or other textual information: 1. We always use the `text` data type instead of the `string` data type. 1. `text` columns should always have a limit set, either by using the `create_table` with -the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` -when altering an existing table. + the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` + when altering an existing table. The standard Rails `text` column type cannot be defined with a limit, but we extend `create_table` to add a `limit: 255` option. Outside of `create_table`, `add_text_limit` can be used to add a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 4d3101b40fb..cb159a404fd 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -19,15 +19,15 @@ table. If the application is designed to work with partitioning in mind, there can be multiple benefits, such as: - Query performance can be improved greatly, because the database can -cheaply eliminate much of the data from the search space, while still -providing full SQL capabilities. + cheaply eliminate much of the data from the search space, while still + providing full SQL capabilities. - Bulk deletes can be achieved with minimal impact on the database by -dropping entire partitions. This is a natural fit for features that need -to periodically delete data that falls outside the retention window. + dropping entire partitions. This is a natural fit for features that need + to periodically delete data that falls outside the retention window. - Administrative tasks like `VACUUM` and index rebuilds can operate on -individual partitions, rather than across a single massive table. + individual partitions, rather than across a single massive table. Unfortunately, not all models fit a partitioning scheme, and there are significant drawbacks if implemented incorrectly. Additionally, tables @@ -632,3 +632,73 @@ class AsyncPrepareTableConstraintsForListPartitioning < Gitlab::Database::Migrat end end ``` + +### Step 7 - Re-point foreign keys to parent table + +The tables that reference the initial partition must be updated to point to the +parent table now. Without this change, the records from those tables +will not be able to locate the rows in the next partitions because they will look +for them in the initial partition. + +Steps: + +- Add the foreign key to the partitioned table and validate it asynchronously, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230524135543_replace_ci_build_pending_states_foreign_key.rb). +- Validate it synchronously after the asynchronously validation was completed on GitLab.com, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230530140456_validate_fk_ci_build_pending_states_p_ci_builds.rb). +- Remove the old foreign key and rename the new one to the old name, + [for example](https://gitlab.com/gitlab-org/gitlab/-/blob/65d63f6a00196c3a7d59f15191920f271ab2b145/db/post_migrate/20230615083713_replace_old_fk_ci_build_pending_states_to_builds.rb#L9). + +### Step 8 - Ensure ID uniqueness across partitions + +All uniqueness constraints must include the partitioning key, so we can have +duplicate IDs across partitions. To solve this we enforce that only the database +can set the ID values and use a sequence to generate them because sequences are +guaranteed to generate unique values. + +For example: + +```ruby +class EnsureIdUniquenessForPCiBuilds < Gitlab::Database::Migration[2.1] + include Gitlab::Database::PartitioningMigrationHelpers::UniquenessHelpers + + enable_lock_retries! + + TABLE_NAME = :p_ci_builds + FUNCTION_NAME = :assign_p_ci_builds_id_value + + def up + ensure_unique_id(TABLE_NAME) + end + + def down + execute(<<~SQL.squish) + ALTER TABLE #{TABLE_NAME} + ALTER COLUMN id SET DEFAULT nextval('ci_builds_id_seq'::regclass); + + DROP FUNCTION IF EXISTS #{FUNCTION_NAME} CASCADE; + SQL + end +``` + +### Step 9 - Analyze the partitioned table and create new partitions + +The autovacuum daemon does not process partitioned tables. It is necessary to +periodically run a manual `ANALYZE` to keep the statistics of the table hierarchy +up to date. + +Models that implement `Ci::Partitionable` with `partitioned: true` option are +analyzed by default on a weekly basis. To enable this and create new partitions +you need to register the model in the [PostgreSQL initializer](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f0e3f1bcd2ffc220768bbc373364151775ca8e/config/initializers/postgres_partitioning.rb). + +### Step 10 - Update the application to use the partitioned table + +Now that the parent table is ready, we can update the application to use it: + +```ruby +class Model < ApplicationRecord + self.table_name = :partitioned_table +end +``` + +Depending on the model, it might be safer to use a [change management issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16387). diff --git a/doc/development/database_review.md b/doc/development/database_review.md index ec4c4f9f2c5..2bb2a6fc267 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -29,7 +29,7 @@ A database review is required for: These metrics could have complex queries over large tables. See the [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) for implementation details. -- Changes that use [`update`, `delete`, `update_all`, `delete_all` or `destroy_all`](#preparation-when-using-update-delete-update_all-delete_all-or-destroy_all) +- Changes that use [`update`, `upsert`, `delete`, `update_all`, `upsert_all`, `delete_all` or `destroy_all`](#preparation-when-using-bulk-update-operations) methods on an ActiveRecord object. A database reviewer is expected to look out for overly complex @@ -113,7 +113,7 @@ the following preparations into account. #### Preparation when adding migrations - Ensure `db/structure.sql` is updated as [documented](migration_style_guide.md#schema-changes), and additionally ensure that the relevant version files under -`db/schema_migrations` were added or removed. + `db/schema_migrations` were added or removed. - Ensure that the Database Dictionary is updated as [documented](database/database_dictionary.md). - Make migrations reversible by using the `change` method or include a `down` method when using `up`. - Include either a rollback procedure or describe how to rollback changes. @@ -153,7 +153,7 @@ Include in the MR description: - Write the raw SQL in the MR description. Preferably formatted nicely with [pgFormatter](https://sqlformat.darold.net) or <https://paste.depesz.com> and using regular quotes - (for example, `"projects"."id"`) and avoiding smart quotes (for example, `"projects"."id"`). + (for example, `"projects"."id"`) and avoiding smart quotes (for example, `“projects”.“id”`). - In case of queries generated dynamically by using parameters, there should be one raw SQL query for each variation. For example, a finder for issues that may take as a parameter an optional filter on projects, @@ -227,9 +227,10 @@ Include in the MR description: - If you're adding a composite index, another index might become redundant, so remove that in the same migration. For example adding `index(column_A, column_B, column_C)` makes the indexes `index(column_A, column_B)` and `index(column_A)` redundant. -#### Preparation when using `update`, `delete`, `update_all`, `delete_all` or `destroy_all` +#### Preparation when using bulk update operations -Using these ActiveRecord methods requires extra care because they modify data and can perform poorly, or they +Using `update`, `upsert`, `delete`, `update_all`, `upsert_all`, `delete_all` or `destroy_all` +ActiveRecord methods requires extra care because they modify data and can perform poorly, or they can destroy data if improperly scoped. These methods are also [incompatible with Common Table Expression (CTE) statements](sql.md#when-to-use-common-table-expressions). Danger will comment on a Merge Request Diff when these methods are used. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 86732c3a8ac..ad41d21a9e7 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -93,9 +93,9 @@ concern, some instrumentations are disabled by default. To enable those disabled instrumentations, set the following environment variables: - `GITLAB_TRACING_TRACK_CACHES`: enable tracking cache operations, such as cache -read, write, or delete. + read, write, or delete. - `GITLAB_TRACING_TRACK_REDIS`: enable tracking Redis operations. Most Redis -operations are for caching, though. + operations are for caching, though. ## Using Jaeger in the GitLab Development Kit diff --git a/doc/development/documentation/alpha_beta.md b/doc/development/documentation/alpha_beta.md deleted file mode 100644 index 4579c57b448..00000000000 --- a/doc/development/documentation/alpha_beta.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'experiment_beta.md' -remove_date: '2023-11-29' ---- - -This document was moved to [another location](experiment_beta.md). - -<!-- This redirect file can be deleted after <2023-11-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/documentation/experiment_beta.md b/doc/development/documentation/experiment_beta.md index 85f6dc38621..96e8d6fb638 100644 --- a/doc/development/documentation/experiment_beta.md +++ b/doc/development/documentation/experiment_beta.md @@ -32,7 +32,7 @@ On self-managed GitLab, by default this feature is not available. To make it available, an administrator can enable the feature flag named `example_flag`. On GitLab.com, this feature is not available. This feature is not ready for production use. -Use this great new feature when you need to do this new thing. +Use this new feature when you need to do this new thing. This feature is an [Experiment](<link_to>/policy/experiment-beta-support.md). To join the list of users testing this feature, do this thing. If you find a bug, diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 617691e8628..637b2d163e7 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -17,7 +17,7 @@ When the state of a feature flag changes, the developer who made the change Every feature introduced to the codebase, even if it's behind a disabled feature flag, must be documented. For more information, see -[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Experiment or Beta](../../policy/experiment-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Experiment or Beta features](alpha_beta.md). +[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Experiment or Beta](../../policy/experiment-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Experiment or Beta features](experiment_beta.md). When the feature is [implemented in multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development), discuss the plan with your technical writer. diff --git a/doc/development/documentation/metadata.md b/doc/development/documentation/metadata.md index 4e80ea154b3..e6eff56d66d 100644 --- a/doc/development/documentation/metadata.md +++ b/doc/development/documentation/metadata.md @@ -32,11 +32,48 @@ To populate the metadata, include this information: - `info`: How to find the Technical Writer associated with the page's stage and group. +### Exceptions + +Documents in the `/development` directory get this metadata: + +```yaml +--- +stage: Example Stage +group: Example Group +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- +``` + +Documents in the `/solutions` directory get this metadata: + +```yaml +--- +stage: Solutions Architecture +group: Solutions Architecture +info: This page is owned by the Solutions Architecture team. +--- +``` + +## Description metadata + +The `description` tag: + +- Is used to populate text on the docs home page. +- Is shown in social media previews. +- Can be used in search result snippets. + +For the top-level pages, like **Use GitLab** and one level underneath, +the descriptions are lists of nouns. For example, for **Set up your organization**, +the description is `Users, groups, namespaces, SSH keys.` + +For other pages, descriptions are not actively maintained. However, if you want to add one, +use a short description of what the page is about. +See the Google [Best practices for creating quality meta descriptions](https://developers.google.com/search/docs/appearance/snippet#meta-descriptions) for tips. + ## Additional metadata The following metadata is optional and is not actively maintained. -- `description`: A short description of what the page is about. See the Google [Best practices for creating quality meta descriptions](https://developers.google.com/search/docs/appearance/snippet#meta-descriptions) for writing tips. This content can be used in search result snippets and is shown in social media previews. - `feedback`: Set to `false` to not include the "Help & Feedback" footer. - `noindex`: Set to `false` to prevent the page from being indexed by search engines. - `redirect_to`: Used to control redirects. For more information, see [Redirects in GitLab documentation](redirects.md). diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 0dfe538cfbd..be10688766f 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -26,6 +26,11 @@ For example: While some older sections of the nav are alphabetical, the nav should primarily be workflow-based. +Without a navigation entry: + +- The navigation closes when the page is opened, and the reader loses their place. +- The page isn't visible in a group with other pages. + ## Choose the right words for your navigation entry Before you add an item to the left nav, choose the parts of speech you want to use. @@ -41,35 +46,14 @@ as helpful as **Get started with runners**. ## Add a navigation entry -To add a topic to the global nav, edit -[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) -and add your item. - -Without a navigation entry: - -- The navigation closes when the page is opened, and the reader loses their place. -- The page isn't visible in a group with other pages. - -### Pages you don't need to add - -Exclude these pages from the global nav: - -- Legal notices. -- Pages in the `architecture/blueprints` directory. -- Pages in the `user/application_security/dast/checks/` directory. - -The following pages should probably be in the global nav, but the technical writers -do not actively work to add them: - -- Pages in the `/development` directory. -- Pages authored by the support team, which are under the `doc/administration/troubleshooting` directory. +**Do not** add items to the global nav without +the consent of one of the technical writers. -Sometimes pages for deprecated features are not in the global nav, depending on how long ago the feature was deprecated. +To add a topic to the global navigation: -All other pages should be in the global nav. - -The technical writing team runs a report to determine which pages are not in the nav. -The team reviews this list each month. +1. In the [`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) + file, add the item. +1. Assign the MR to a technical writer for review and merge. ### Where to add @@ -110,7 +94,28 @@ mechanics of what is required is [documented below](#data-file) but, in principl substitution for **Continuous Integration**. - Navigation links must follow the rules documented in the [data file](#data-file). -## How it works +### Pages you don't need to add + +Exclude these pages from the global nav: + +- Legal notices. +- Pages in the `architecture/blueprints` directory. +- Pages in the `user/application_security/dast/checks/` directory. + +The following pages should probably be in the global nav, but the technical writers +do not actively work to add them: + +- Pages in the `/development` directory. +- Pages authored by the support team, which are under the `doc/administration/troubleshooting` directory. + +Sometimes pages for deprecated features are not in the global nav, depending on how long ago the feature was deprecated. + +All other pages should be in the global nav. + +The technical writing team runs a report to determine which pages are not in the nav. +The team reviews this list each month. + +## Navigation structure The global nav has five levels: @@ -122,9 +127,6 @@ The global nav has five levels: You can view this structure in [the `navigation.yml` file](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/navigation.yaml). -**Do not** [add items](#add-a-navigation-entry) to the global nav without -the consent of one of the technical writers. - ## Composition The global nav is built from two files: diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 26660d2eba1..a18b376a1cc 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -243,7 +243,7 @@ by default. Capitalize names of: - GitLab [product tiers](https://about.gitlab.com/pricing/). For example, - GitLab Free and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) + GitLab Free and GitLab Ultimate. - Third-party organizations, software, and products. For example, Prometheus, Kubernetes, Git, and The Linux Foundation. - Methods or methodologies. For example, Continuous Integration, @@ -351,10 +351,10 @@ For numbers in text, spell out zero through nine and use numbers for 10 and grea - [Write in Markdown](#markdown). - Insert an empty line for new paragraphs. - Insert an empty line between different markups (for example, after every - paragraph, header, list, and so on). Example: + paragraph, heading, list, and so on). Example: ```markdown - ## Header + ## Heading Paragraph. @@ -692,9 +692,9 @@ Markdown tables naturally fall out of alignment over time, but still render corr on `docs.gitlab.com`. The technical writing team can realign cells the next time the page is refactored. -### Table headings +### Table headers -Use sentence case for table headings. For example, `Keyword value` or `Project name`. +Use sentence case for table headers. For example, `Keyword value` or `Project name`. ### Feature tables @@ -1140,7 +1140,7 @@ When you take screenshots: Reduce the size of your browser window as much as possible to keep elements close together and reduce empty space. Try to keep the screenshot dimensions as small as possible. - **Review how the image renders on the page.** Preview the image locally or use the -review app in the merge request. Make sure the image isn't blurry or overwhelming. + review app in the merge request. Make sure the image isn't blurry or overwhelming. - **Be consistent.** Coordinate screenshots with the other screenshots already on a documentation page for a consistent reading experience. Ensure your navigation theme is **Indigo** and the syntax highlighting theme is **Light**. These are the default preferences. @@ -1237,8 +1237,7 @@ and annoying for users. If you're describing a complicated interaction in the user interface and want to include a visual representation to help readers understand it, you can: -- Use a static image (screenshot) and if necessary, add callouts to emphasize an - an area of the screen. +- Use a static image (screenshot) and if necessary, add callouts to emphasize an area of the screen. - Create a short video of the interaction and link to it. ### Automatic screenshot generator @@ -1349,12 +1348,14 @@ Do not upload videos to the product repositories. [Link](#link-to-video) or ### Link to video -To link out to a video, include a YouTube icon so that readers can scan the page -for videos before reading: +To link to a video, include a YouTube icon so that readers can scan the page +for videos before reading. Include the video's publication date after the link, to help identify +videos that might be out-of-date. ```markdown <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Video Title](link-to-video). +<!-- Video published on YYYY-MM-DD --> ``` You can link any up-to-date video that's useful to the GitLab user. @@ -1384,6 +1385,8 @@ To embed a video: (`https://www.youtube-nocookie.com/embed/VIDEO-ID`), and paste it, replacing the content of the `src` field in the `iframe` tag. +1. Include the video's publication date below the link, to help identify + videos that might be out-of-date. ```html leave a blank line here @@ -1393,6 +1396,7 @@ leave a blank line here <figure class="video-container"> <iframe src="https://www.youtube-nocookie.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen> </iframe> </figure> +<!-- Video published on YYYY-MM-DD --> leave a blank line here ``` @@ -1434,7 +1438,7 @@ NOTE: This is something to note. ``` -To display an alert box for multiple paragraphs, lists, or headers, use +To display an alert box for multiple paragraphs, lists, or headings, use [blockquotes](#blockquotes) instead. Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). @@ -1636,13 +1640,37 @@ When names change, it is more complicated to search or grep text that has line b Tier badges provide information about a feature and are displayed next to the topic title. -You should assign a tier badge: +#### When to add tier badges -- To all H1 topic titles, except the pages under `doc/development/*`. -- To topic titles that don't apply to the same tier as the H1. +Assign tier badges to: + +- All H1 topic titles, except the pages under `doc/development/*` and `doc/solutions/*`. +- Topic titles that don't apply to the same tier as the H1. The H1 tier badge should be the badge that applies to the lowest tier for the features on the page. +#### When not to add tier badges + +Do not assign tier badges: + +- When a feature does not have one obvious subscription tier or offering. + For example, if a feature applies to one tier for SaaS and a different tier for self-managed. + +In this case, do any or all of the following: + +- Use a `NOTE` in an alert box to describe the tiers. +- Add tier badges to other topic titles where this information makes more sense. +- Do not add tier badges to the H1. + +##### Pages that don't need a tier badge + +Some pages won't have a tier badge, because no obvious tier badge applies. For example: + +- Tutorials. +- Pages that compare features from different tiers. +- Pages in the `/development` folder. These pages are automatically assigned a `Contribute` badge. +- Pages in the `/solutions` folder. These pages are automatically assigned a `Solutions` badge. + #### Available product tier badges Tier badges should include two components, in this order: a subscription tier and an offering. @@ -1667,7 +1695,9 @@ You can also add a third component for the feature's status: For example, `**(FREE ALL EXPERIMENT)**`. -A tier or status can stand alone. An offering should always have a tier. +- A tier or status can stand alone. +- An offering should always have a tier. +- Do not add more than one offering, tier, or status. Multiples do not render properly in the documentation. #### Add a tier badge @@ -1697,15 +1727,6 @@ Do not add tier badges inline with other text, except for [API attributes](../re The single source of truth for a feature should be the topic where the functionality is described. -##### Pages that don't need a tier badge - -Some pages won't have a tier badge, because no obvious tier badge applies. For example: - -- Tutorials. -- Pages that compare features from different tiers. -- Pages in the `/development` folder. These pages are automatically assigned a `Contribute` badge. -- Pages in the `/solutions` folder. These pages are automatically assigned a `Solutions` badge. - ##### Administrator documentation tier badges Topics that are only for instance administrators should be badged `<TIER> SELF`. Instance diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 9d0d59e27c5..fed295b8ec9 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -130,6 +130,10 @@ The token generated when you create an agent for Kubernetes. Use **agent access Use **AI**. Do not spell out **artificial intelligence**. +## AI-powered DevSecOps platform + +If preceded by GitLab, capitalize **Platform**. For example, the GitLab AI-powered DevSecOps Platform. + ## air gap, air-gapped Use **offline environment** to describe installations that have physical barriers or security policies that prevent or limit internet access. Do not use **air gap**, **air gapped**, or **air-gapped**. For example: @@ -239,6 +243,22 @@ Use **text box** to refer to the UI field. Do not use **field** or **box**. For - In the **Variable name** text box, enter a value. +## branch + +Use **branch** by itself to describe a branch. For specific branches, use these terms only: + +- **default branch**: The primary branch in the repository. Users can use the UI to set the default + branch. For examples that use the default branch, use `main` instead of [`master`](#master). +- **source branch**: The branch you're merging from. +- **target branch**: The branch you're merging to. +- **current branch**: The branch you have checked out. + This branch might be the default branch, a branch you've created, a source branch, or some other branch. + +Do not use the terms **feature branch** or **merge request branch**. Be as specific as possible. For example: + +- The branch you have checked out... +- The branch you added commits to... + ## bullet Don't refer to individual items in an ordered or unordered list as **bullets**. Use **list item** instead. If you need to be less ambiguous, you can use: @@ -428,13 +448,6 @@ Instead of: - Data are collected. - The data show a performance increase. -## default branch - -Use **default branch** to refer generically to the primary branch in the repository. -Users can set the default branch by using a UI setting. - -For examples that use the default branch, use `main` instead of [`master`](#master). - ## delete Use **delete** when an object is completely deleted. **Delete** is the opposite of **create**. @@ -464,6 +477,10 @@ When writing about the Developer role: Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. +## DevSecOps platform + +If preceded by GitLab, capitalize **Platform**. For example, the GitLab DevSecOps Platform. + ## dialog Use **dialog** rather than any of these alternatives: @@ -661,6 +678,10 @@ Instead of: - Use the merge request feature to incorporate changes into the target branch. +## feature branch + +Do not use **feature branch**. See [branch](#branch). + ## field Use **text box** instead of **field** or **box**. @@ -1083,7 +1104,7 @@ Do not use **manpower**. Use words like **workforce** or **GitLab team members** ## master -Do not use `master`. Use `main` when you need a sample [default branch name](#default-branch). +Do not use **master**. Use **main** when you need a sample [default branch name](#branch). ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) ## may, might @@ -1115,6 +1136,10 @@ For **MB** and **GB**, follow the [Microsoft guidance](https://learn.microsoft.c When you add a [user account](#user-account) to a group or project, the user account becomes a **member**. +## merge request branch + +Do not use **merge request branch**. See [branch](#branch). + ## merge requests Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it out on first use. @@ -1570,7 +1595,7 @@ Searching is different from [filtering](#filter). When referring to the subscription billing model: - For GitLab SaaS, use **seats**. Customers purchase seats. Users occupy seats when they are invited -to a group, with some [exceptions](../../../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined). + to a group, with some [exceptions](../../../subscriptions/gitlab_com/index.md#how-seat-usage-is-determined). - For GitLab self-managed, use **users**. Customers purchase subscriptions for a specified number of **users**. ## section diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 62df2395fbb..8a38b0ed0e4 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -492,6 +492,7 @@ command line. To configure markdownlint in your editor, install one of the following as appropriate: +- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). - Sublime Text [`SublimeLinter-contrib-markdownlint` package](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint). This package uses `markdownlint-cli` by default, but can be configured to use `markdownlint-cli2` with this SublimeLinter configuration: @@ -502,11 +503,20 @@ To configure markdownlint in your editor, install one of the following as approp } ``` -- Visual Studio Code [`DavidAnson.vscode-markdownlint` extension](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). - Vim [ALE plugin](https://github.com/dense-analysis/ale). +- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). `Flycheck` supports + `markdownlint-cli` out of the box, but you must add a `.dir-locals.el` file to + point it to the `.markdownlint.yml` at the base of the project directory: + + ```lisp + ;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project. + ((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml")))) + ``` To configure Vale in your editor, install one of the following as appropriate: +- Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). + You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Sublime Text [`SublimeLinter-vale` package](https://packagecontrol.io/packages/SublimeLinter-vale). To have Vale suggestions appears as blue instead of red (which is how errors appear), add `vale` configuration to your [SublimeLinter](http://sublimelinter.readthedocs.org) configuration: @@ -522,26 +532,12 @@ To configure Vale in your editor, install one of the following as appropriate: ``` - [LSP for Sublime Text](https://lsp.sublimetext.io) package [`LSP-vale-ls`](https://packagecontrol.io/packages/LSP-vale-ls). -- Visual Studio Code [`ChrisChinchilla.vale-vscode` extension](https://marketplace.visualstudio.com/items?itemName=ChrisChinchilla.vale-vscode). - You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). - JetBrains IDEs - No plugin exists, but [this issue comment](https://github.com/errata-ai/vale-server/issues/39#issuecomment-751714451) contains tips for configuring an external tool. -- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). - This requires some configuration: - - - `Flycheck` supports `markdownlint-cli` out of the box, but you must point it - to the `.markdownlint.yml` at the base of the project directory. A `.dir-locals.el` - file can accomplish this: - - ```lisp - ;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project. - ((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml")))) - - ``` - - - A minimal configuration for Flycheck to work with Vale could look like this: +- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). A minimal configuration + for Flycheck to work with Vale could look like: ```lisp (flycheck-define-checker vale diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md index aee5bd1377c..f970b58e4fc 100644 --- a/doc/development/documentation/topic_types/troubleshooting.md +++ b/doc/development/documentation/topic_types/troubleshooting.md @@ -64,7 +64,7 @@ The workaround is... If multiple causes or solutions exist, consider putting them into a table format. If you use the exact error message, surround it in backticks so it's styled as code. -For more guidance on solution types, see [workaround](../../documentation/styleguide/word_list.md#workaround) and [resolution, resolve](../../documentation/styleguide/word_list.md#resolution-resolve). +For more guidance on solution types, see [workaround](../../documentation/styleguide/word_list.md#workaround) and [resolution, resolve](../../documentation/styleguide/word_list.md#resolution-resolve). ## Troubleshooting topic titles diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 08045675295..78177612aa9 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -200,7 +200,7 @@ To guard your licensed feature: ``` 1. Optional. If your global feature is also available to namespaces with a paid plan, combine two -feature identifiers to allow both administrators and group users. For example: + feature identifiers to allow both administrators and group users. For example: ```ruby License.feature_available?(:my_feature_name) || group.licensed_feature_available?(:my_feature_name_for_namespace) # Both admins and group members can see this EE feature diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 971c527ef68..75a815925ab 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -24,7 +24,7 @@ This structure allows the group to think through a proposed change, gather feedb ### Design Documents -When the work ahead may affect more than a single group, stage or potentially an entirement department (for example, all of the Frontend team) then it is likely that there is need for a [Design Document](https://about.gitlab.com/handbook/engineering/architecture/workflow/). +When the work ahead may affect more than a single group, stage or potentially an entire department (for example, all of the Frontend team) then it is likely that there is need for a [Design Document](https://about.gitlab.com/handbook/engineering/architecture/workflow/). This is well documented in the handbook, but to touch on it shortly, it is **the best way** to propose large changes and gather the required feedback and support to move forward. These documents are version controlled, keep evolving with time and are a great way to share a complex understanding across the entire organization. They also require a coach, which is a great way to involve someone with a lot of experience with larger changes. This process is shared across all engineering departments and is owned by the CTO. @@ -57,9 +57,9 @@ Very small changes may have a very broad impact. For example, a change to any ES For recommending certain code patterns in our documentation, you can write the MR that apply your proposed change, share it broadly with the department and if no strong objections are raised, merge your change. This is more efficient than RFCs because of the bias for action, while also gathering all the feedback necessary for everyone to feel included. -If you'd like to propose a major change to the technological stack (Vue to React, JavaScript to TypeScript, etc.), start by reaching out on Slack to gauge interest. Always ask yourself whether or not the problems that you see can be fixed from our current tech stack, as we should always try to fix our problems with the tools we already have. Other departments, such as Backend and QA, do not have a clear process to propose technological changes either. That is because these changes would require huge investements from the company and probably cannot be decided without involving high-ranking executives from engineering. +If you'd like to propose a major change to the technological stack (Vue to React, JavaScript to TypeScript, etc.), start by reaching out on Slack to gauge interest. Always ask yourself whether or not the problems that you see can be fixed from our current tech stack, as we should always try to fix our problems with the tools we already have. Other departments, such as Backend and QA, do not have a clear process to propose technological changes either. That is because these changes would require huge investments from the company and probably cannot be decided without involving high-ranking executives from engineering. -Instead, consider starting a Design Document that explains the problem and try to solve it with our current tools. Invite contribution from the department and research this thoroughly as there can only be two outcomes. Either the problem **can** be solved with our current tools or it cannot. If it can, this is a huge with for our teams since we've fixed and issue without the need to completly change our stack, and if it cannot, then the Design Document can be the start of the larger conversation around the technological change. +Instead, consider starting a Design Document that explains the problem and try to solve it with our current tools. Invite contribution from the department and research this thoroughly as there can only be two outcomes. Either the problem **can** be solved with our current tools or it cannot. If it can, this is a huge win for our teams since we've fixed an issue without the need to completely change our stack, and if it cannot, then the Design Document can be the start of the larger conversation around the technological change. ## Widget Architecture diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 7a76b39edbc..64cf7de75ab 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -46,7 +46,7 @@ export default { The rich text editor requires two properties: - `renderMarkdown` is an asynchronous function that returns the response (String) of invoking the -[Markdown API](../../api/markdown.md). + [Markdown API](../../api/markdown.md). - `uploadsPath` is a URL that points to a [GitLab upload service](../uploads/index.md) with `multipart/form-data` support. diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index 93b5b20b047..9ba0d4bb40c 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -98,7 +98,7 @@ export const pageViewsOverTime = { To add a new visualization render type: 1. Create a new Vue component that accepts `data` and `options` properties. -See [`line_chart.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/analytics/analytics_dashboards/components/visualizations/line_chart.vue) as an example. + See [`line_chart.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/analytics/analytics_dashboards/components/visualizations/line_chart.vue) as an example. 1. Add your component to the list of conditional imports in [`panel_base.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/panels_base.vue#L13). 1. Add your component to the schema's list of `AnalyticsVisualization` types in [`analytics_visualizations.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/analytics_visualization.json). diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md deleted file mode 100644 index e0d3a80d9c8..00000000000 --- a/doc/development/fe_guide/design_anti_patterns.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -redirect_to: 'design_patterns.md' -remove_date: '2023-12-07' ---- - -This document was moved to [another location](design_patterns.md). - -<!-- This redirect file can be deleted after <2023-12-07>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index b8e98b47cac..0ab64841365 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -202,3 +202,10 @@ To see what polyfills are being used: ### 9. Why is my page broken in dark mode? See [dark mode docs](dark_mode.md) + +### 10. How to render GitLab-flavored Markdown? + +If you need to render [GitLab-flavored Markdown](../gitlab_flavored_markdown/index.md), then there are two things that you require: + +- Pass the GLFM content with the `v-safe-html` directive to a `div` HTML element inside your Vue component +- Add the `md` class to the root div, which will apply the appropriate CSS styling diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 22b977519be..bd35d8f237f 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -1671,7 +1671,7 @@ When [using Vuex](#using-with-vuex), disable the cache when: - The data is being cached elsewhere - The use case does not need caching -if the data is being cached elsewhere, or if there is no need for it for the given use case. + if the data is being cached elsewhere, or if there is no need for it for the given use case. ```javascript import createDefaultClient from '~/lib/graphql'; @@ -1767,12 +1767,12 @@ To improve performance, sometimes we want to make initial GraphQL queries early. ``` - Add startup calls with correct variables to the HAML file that serves as a view -for your application. To add GraphQL startup calls, we use -`add_page_startup_graphql_call` helper where the first parameter is a path to the -query, the second one is an object containing query variables. Path to the query is -relative to `app/graphql/queries` folder: for example, if we need a -`app/graphql/queries/repository/files.query.graphql` query, the path is -`repository/files`. + for your application. To add GraphQL startup calls, we use + `add_page_startup_graphql_call` helper where the first parameter is a path to the + query, the second one is an object containing query variables. Path to the query is + relative to `app/graphql/queries` folder: for example, if we need a + `app/graphql/queries/repository/files.query.graphql` query, the path is + `repository/files`. ## Troubleshooting diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index 88f4a785a70..4a6b349bbc8 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -20,8 +20,8 @@ used in HAML: - Some of the Pajamas components are available as a [ViewComponent](view_component.md#pajamas-components). Use these when possible. - If no ViewComponent exists, why not go ahead and create it? Talk to the Foundations team if you need help. - As a fallback, this can be done by applying the correct CSS classes to the elements. -- A custom -[Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) exists to help use GitLab UI components in HAML forms. +- A custom [Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) + exists to help use GitLab UI components in HAML forms. ### Use the GitLab UI form builder diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index 05d4397933f..f0233f71ad2 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -68,7 +68,7 @@ To add a story with API access: If you test against `gitlab.com`, make sure to use a token with `read_api` if possible and to make the token short-lived. 1. Create an `.env` file in the `storybook` directory. Use the `storybook/.env.template` file as -a starting point. + a starting point. 1. Set the `API_ACCESS_TOKEN` variable to the access token that you created. diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index 83d725d453b..870a4c51915 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -123,7 +123,7 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor ## Naming 1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension -([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)). + ([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)). 1. **Reference Naming**: Use PascalCase for their default imports: ```javascript @@ -142,7 +142,7 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor }; ``` -1. **Props Naming:** Avoid using DOM component prop names. +1. **Props Naming:** Avoid using DOM component prop names. 1. **Props Naming:** Use kebab-case instead of camelCase to provide props in templates. ```html @@ -582,8 +582,8 @@ describe('MyComponent', () => { ### Setting component state 1. Avoid using [`setProps`](https://v1.test-utils.vuejs.org/api/wrapper/#setprops) to set -component state wherever possible. Instead, set the component's -[`propsData`](https://v1.test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: + component state wherever possible. Instead, set the component's + [`propsData`](https://v1.test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: ```javascript // bad @@ -603,7 +603,7 @@ component state wherever possible. Instead, set the component's ### Accessing component state 1. When accessing props or attributes, prefer the `wrapper.props('myProp')` syntax over -`wrapper.props().myProp` or `wrapper.vm.myProp`: + `wrapper.props().myProp` or `wrapper.vm.myProp`: ```javascript // good @@ -616,7 +616,7 @@ component state wherever possible. Instead, set the component's ``` 1. When asserting multiple props, check the deep equality of the `props()` object with -[`toEqual`](https://jestjs.io/docs/expect#toequalvalue): + [`toEqual`](https://jestjs.io/docs/expect#toequalvalue): ```javascript // good @@ -633,8 +633,8 @@ component state wherever possible. Instead, set the component's ``` 1. If you are only interested in some of the props, you can use -[`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` -over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): + [`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` + over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): ```javascript // good @@ -652,7 +652,7 @@ over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectconta ### Testing props validation -1. When checking component props use `assertProps` helper. Props validation failures will be thrown as errors +When checking component props use `assertProps` helper. Props validation failures will be thrown as errors: ```javascript import { assertProps } from 'helpers/assert_props' @@ -668,12 +668,12 @@ The goal of this accord is to make sure we are all on the same page. 1. When writing Vue, you may not use jQuery in your application. 1. If you need to grab data from the DOM, you may query the DOM 1 time while bootstrapping your - application to grab data attributes using `dataset`. You can do this without jQuery. + application to grab data attributes using `dataset`. You can do this without jQuery. 1. You may use a jQuery dependency in Vue.js following [this example from the docs](https://vuejs.org/v2/examples/select2.html). 1. If an outside jQuery Event needs to be listen to inside the Vue application, you may use - jQuery event listeners. + jQuery event listeners. 1. We avoid adding new jQuery events when they are not required. Instead of adding new jQuery - events take a look at [different methods to do the same task](https://v2.vuejs.org/v2/api/#vm-emit). + events take a look at [different methods to do the same task](https://v2.vuejs.org/v2/api/#vm-emit). 1. You may query the `window` object one time, while bootstrapping your application for application specific data (for example, `scrollTo` is ok to access anytime). Do this access during the bootstrapping of your application. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 69967a5a2be..cf7659d0fc0 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -543,7 +543,7 @@ Based on the Vue guidance: - **Do not** use or create a JavaScript class in your [data function](https://v2.vuejs.org/v2/api/#data). - **Do not** add new JavaScript class implementations. - **Do** use [GraphQL](../api_graphql_styleguide.md), [Vuex](vuex.md) or a set of components if -cannot use primitives or objects. + cannot use primitives or objects. - **Do** maintain existing implementations using such approaches. - **Do** Migrate components to a pure object model when there are substantial changes to it. - **Do** add business logic to helpers or utilities, so you can test them separately from your component. @@ -555,7 +555,7 @@ Additional reasons why having a JavaScript class presents maintainability issues - After a class is created, it can be extended in a way that can infringe Vue reactivity and best practices. - A class adds a layer of abstraction, which makes the component API and its inner workings less clear. - It makes it harder to test. Because the class is instantiated by the component data function, it is -harder to 'manage' component and class separately. + harder to 'manage' component and class separately. - Adding Object Oriented Principles (OOP) to a functional codebase adds another way of writing code, reducing consistency and clarity. ## Style guide @@ -831,7 +831,7 @@ describe('~/todos/app.vue', () => { 1. Test any directive that defines if/how child component is rendered (for example, `v-if` and `v-for`). 1. Test any props we are passing to child components (especially if the prop is calculated in the -component under test, with the `computed` property, for example). Remember to use `.props()` and not `.vm.someProp`. + component under test, with the `computed` property, for example). Remember to use `.props()` and not `.vm.someProp`. 1. Test we react correctly to any events emitted from child components: ```javascript diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index a5a6439b420..59785554ce6 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -97,6 +97,7 @@ Consult these topics for information on contributing to specific GitLab features - [Application limits](application_limits.md) - [AI features](ai_features/index.md) - [Application settings](application_settings.md) +- [Remote Development](remote_development/index.md) ### Import and Export diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 17dd8432f7b..bea42d6340d 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -7,7 +7,7 @@ info: "See the Technical Writers assigned to Development Guidelines: https://han # Feature flags in the development of GitLab NOTE: -This document explains how to contribute to the development of the GitLab product. +This document explains how to contribute to the development and operations of the GitLab product. If you want to use feature flags to show and hide functionality in your own applications, view [this feature flags information](../../operations/feature_flags.md) instead. @@ -17,6 +17,11 @@ All newly-introduced feature flags should be [disabled by default](https://about WARNING: All newly-introduced feature flags should be [used with an actor](controls.md#percentage-based-actor-selection). +Blueprints: + +- (Latest) [Feature Flags usage in GitLab development and operations](../../architecture/blueprints/feature_flags_usage_in_dev_and_ops/index.md) +- [Development Feature Flags Architecture](../../architecture/blueprints/feature_flags_development/index.md) + This document is the subject of continued work as part of an epic to [improve internal usage of feature flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. For an [overview of the feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), or if you need help deciding [if you should use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) or not, see the [feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) handbook page. @@ -44,77 +49,147 @@ should be leveraged: When the feature implementation is delivered over multiple merge requests: 1. [Create a new feature flag](#create-a-new-feature-flag) - which is **off** by default, in the first merge request which uses the flag. - Flags [should not be added separately](#risk-of-a-broken-main-branch). + which is **disabled** by default, in the first merge request which uses the flag. + Flags [should not be added separately](#risk-of-a-broken-default-branch). 1. Submit incremental changes via one or more merge requests, ensuring that any - new code added can only be reached if the feature flag is **on**. + new code added can only be reached if the feature flag is **enabled**. You can keep the feature flag enabled on your local GDK during development. 1. When the feature is ready to be tested by other team members, [create the initial documentation](../documentation/feature_flags.md#when-to-document-features-behind-a-feature-flag). Include details about the status of the [feature flag](../documentation/feature_flags.md#how-to-add-feature-flag-documentation). -1. Enable the feature flag for a specific project and ensure that there are no issues +1. Enable the feature flag for a specific group/project/user and ensure that there are no issues with the implementation. Do not enable the feature flag for a public project - like `gitlab` if there is no documentation. Team members and contributors might search for + like `gitlab-org/gitlab` if there is no documentation. Team members and contributors might search for documentation on how to use the feature if they see it enabled in a public project. 1. When the feature is ready for production use, open a merge request to: - Update the documentation to describe the latest flag status. - Add a [changelog entry](#changelog). - - Flip the feature flag to be **on by default** or remove it entirely - to enable the new behavior. + - Remove the feature flag to enable the new behavior, or flip the feature flag to be **enabled by default** (only for `ops` and `beta` feature flags). One might be tempted to think that feature flags will delay the release of a feature by at least one month (= one release). This is not the case. A feature flag does not have to stick around for a specific amount of time (for example, at least one release), instead they should stick around until the feature -is deemed stable. Stable means it works on GitLab.com without causing any -problems, such as outages. +is deemed stable. **Stable means it works on GitLab.com without causing any +problems, such as outages.** -## Risk of a broken main branch +## Risk of a broken default branch Feature flags must be used in the MR that introduces them. Not doing so causes a -[broken main branch](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due -to the `rspec:feature-flags` job that only runs on the `main` branch. +[broken default branch](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due +to the `rspec:feature-flags` job that only runs on the default branch. ## Types of feature flags Choose a feature flag type that matches the expected usage. -### `development` type +### `gitlab_com_derisk` type + +`gitlab_com_derisk` feature flags are short-lived feature flags, +used to de-risk GitLab.com deployments. Most feature flags used at +GitLab are of the `gitlab_com_derisk` type. -`development` feature flags are short-lived feature flags, -used for deploying unfinished code to production. Most feature flags used at -GitLab are the `development` type. +#### Constraints -A `development` feature flag must have a rollout issue -created from the [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). +- `default_enabled`: **Must not** be set to true. This kind of feature flag is meant to lower the risk on GitLab.com, thus there's no need to keep the flag in the codebase after it's been enabled on GitLab.com. `default_enabled: true` will not have any effect for this type of feature flag. +- Maximum Lifespan: 2 months after it's merged into the default branch +- Documentation: This type of feature flag don't need to be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page given they're short-lived and deployment-related +- Rollout issue: **Must** have a rollout issue created from the + [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) + +#### Usage + +The format for `gitlab_com_derisk` feature flags is `Feature.<state>(:<dev_flag_name>)`. -The format for `development` feature flags is `Feature.<state>(:<dev_flag_name>)`. To enable and disable them, run on the GitLab Rails console: ```ruby # To enable it for the instance: -Feature.enable(:<dev_flag_name>) +Feature.enable(:<dev_flag_name>, type: :gitlab_com_derisk) # To disable it for the instance: -Feature.disable(:<dev_flag_name>) +Feature.disable(:<dev_flag_name>, type: :gitlab_com_derisk) # To enable for a specific project: -Feature.enable(:<dev_flag_name>, Project.find(<project id>)) +Feature.enable(:<dev_flag_name>, Project.find(<project id>), type: :gitlab_com_derisk) # To disable for a specific project: -Feature.disable(:<dev_flag_name>, Project.find(<project id>)) +Feature.disable(:<dev_flag_name>, Project.find(<project id>), type: :gitlab_com_derisk) ``` -To check a `development` feature flag's state: +To check a `gitlab_com_derisk` feature flag's state: ```ruby # Check if the feature flag is enabled -Feature.enabled?(:dev_flag_name) +Feature.enabled?(:dev_flag_name, type: :gitlab_com_derisk) # Check if the feature flag is disabled -Feature.disabled?(:dev_flag_name) +Feature.disabled?(:dev_flag_name, type: :gitlab_com_derisk) +``` + +### `wip` type + +Some features are complex and need to be implemented through several MRs. Until they're fully implemented, +it needs to be hidden from anyone. In that case, the `wip` (for "Work In Progress") feature flag allows +to merge all the changes to the main branch without actually using the feature yet. + +Once the feature is complete, the feature flag type can be changed to the `gitlab_com_derisk` or +`beta` type depending on how the feature will be presented/documented to customers. + +#### Constraints + +- `default_enabled`: **Must not** be set to true. If needed, this type can be changed to beta once the feature is complete. +- Maximum Lifespan: 4 months after it's merged into the default branch +- Documentation: This type of feature flag don't need to be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page given they're mostly hiding unfinished code +- Rollout issue: Likely no need for a rollout issues, as `wip` feature flags should be transitioned to + another type before being enabled + +#### Usage + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_wip_flag, project, type: :wip) + +# Check if feature flag is disabled +Feature.disabled?(:my_wip_flag, project, type: :wip) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_wip_flag, project, type: :wip) ``` -For `development` feature flags, the type doesn't need to be specified (they're the default type). +### `beta` type + +We might +[not be confident we'll be able to scale, support, and maintain a feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#experiment-beta-ga) +in its current form for every designed use case ([example](https://gitlab.com/gitlab-org/gitlab/-/issues/336070#note_1523983444)). +There are also scenarios where a feature is not complete enough to be considered an MVC. +Providing a flag in this case allows engineers and customers to disable the new feature until it's performant enough. + +#### Constraints + +- `default_enabled`: Can be set to `true` so that a feature can be "released" to everyone in Beta with the + possibility to disable it in the case of scalability issues (ideally it should only be disabled for this + reason on specific on-premise installations) +- Maximum Lifespan: 6 months after it's merged into the default branch +- Documentation: This type of feature flag **must** be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page +- Rollout issue: **Must** have a rollout issue + created from the + [Feature flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) + +#### Usage + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_beta_flag, project, type: :beta) + +# Check if feature flag is disabled +Feature.disabled?(:my_beta_flag, project, type: :beta) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_beta_flag, project, type: :beta) +``` ### `ops` type @@ -122,10 +197,20 @@ For `development` feature flags, the type doesn't need to be specified (they're of GitLab product behavior. For example, feature flags that disable features that might have a performance impact such as Sidekiq worker behavior. -`ops` feature flags likely do not have rollout issues, as it is hard to -predict when they are enabled or disabled. +Remember that using this type should follow a conscious decision not to introduce an +instance/group/project/user setting. + +#### Constraints + +- `default_enabled`: Can be set to `true` so that a feature can be "released" to everyone in Beta with the + possibility to disable it in the case of scalability issues (ideally it should only be disabled for this + reason on specific on-premise installations) +- Maximum Lifespan: Unlimited +- Documentation: This type of feature flag **must** be documented in the + [All feature flags in GitLab](../../user/feature_flags.md) page +- Rollout issue: Likely no need for a rollout issues, as it is hard to predict when they are enabled or disabled -To invoke `ops` feature flags, you must append `type: :ops`: +#### Usage ```ruby # Check if feature flag is enabled @@ -142,18 +227,27 @@ push_frontend_feature_flag(:my_ops_flag, project, type: :ops) `experiment` feature flags are used for A/B testing on GitLab.com. -An `experiment` feature flag should conform to the same standards as a `development` feature flag, +An `experiment` feature flag should conform to the same standards as a `beta` feature flag, although the interface has some differences. An experiment feature flag should have a rollout issue, created using the [Experiment Tracking template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Experiment%20Rollout.md). More information can be found in the [experiment guide](../experiment_guide/index.md). +#### Constraints + +- `default_enabled`: **Must not** be set to `true`. +- Maximum Lifespan: 6 months after it's merged into the default branch + ### `worker` type -`worker` feature flags are used for controlling Sidekiq workers behavior, such as deferring Sidekiq jobs. +`worker` feature flags are special `ops` flags that allow to control Sidekiq workers behavior, such as deferring Sidekiq jobs. `worker` feature flags likely do not have any YAML definition as the name could be dynamically generated using the worker name itself, for example, `run_sidekiq_jobs_AuthorizedProjectsWorker`. Some examples for using `worker` type feature flags can be found in [deferring Sidekiq jobs](#deferring-sidekiq-jobs). +### (Deprecated) `development` type + +The `development` type is deprecated in favor of the `gitlab_com_derisk`, `wip`, and `beta` feature flag types. + ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -179,10 +273,12 @@ Each feature flag is defined in a separate YAML file consisting of a number of f | `name` | yes | Name of the feature flag. | | `type` | yes | Type of feature flag. | | `default_enabled` | yes | The default state of the feature flag. | -| `introduced_by_url` | no | The URL to the merge request that introduced the feature flag. | +| `introduced_by_url` | yes | The URL to the merge request that introduced the feature flag. | +| `milestone` | yes | Milestone in which the feature flag was created. | +| `group` | yes | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | +| `feature_issue_url` | no | The URL to the original feature issue. | | `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | -| `milestone` | no | Milestone in which the feature flag was created. | -| `group` | no | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | +| `log_state_changes` | no | Used to log the state of the feature flag | NOTE: All validations are skipped when running in `RAILS_ENV=production`. @@ -571,6 +667,19 @@ as follows: Feature.remove(:feature_flag_name) ``` +### Logging + +Usage and state of the feature flag is logged if either: + +- `log_state_changes` is set to `true` in the feature flag definition. +- `milestone` refers to a milestone that is greater than or equal to the current GitLab version. + +When the state of a feature flag is logged, it can be identified by using the `"json.feature_flag_states": "feature_flag_name:1"` or `"json.feature_flag_states": "feature_flag_name:0"` condition in Kibana. +You can see an example in [this](https://log.gprd.gitlab.net/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d%2Fd,to:now))&_a=(columns:!(json.feature_flag_states),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:json.feature_flag_states,index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',key:json.feature_flag_states,negate:!f,params:(query:'optimize_where_full_path_in:1'),type:phrase),query:(match_phrase:(json.feature_flag_states:'optimize_where_full_path_in:1')))),hideChart:!f,index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:''),sort:!(!(json.time,desc)))) link. + +NOTE: +Only 20% of the requests log the state of the feature flags. This is controlled with the [`feature_flag_state_logs`](https://gitlab.com/gitlab-org/gitlab/-/blob/6deb6ecbc69f05a80d920a295dfc1a6a303fc7a0/config/feature_flags/ops/feature_flag_state_logs.yml) feature flag. + ## Changelog We want to avoid introducing a changelog when features are not accessible by an end-user either directly (example: ability to use the feature) or indirectly (examples: ability to take advantage of background jobs, performance improvements, or database migration updates). diff --git a/doc/development/gems.md b/doc/development/gems.md index 476ed8f916b..cd1a57fe136 100644 --- a/doc/development/gems.md +++ b/doc/development/gems.md @@ -150,6 +150,25 @@ You can see example adding a new gem: [!121676](https://gitlab.com/gitlab-org/gi gem '<name-of-gem>', path: 'gems/<name-of-gem>' ``` +### Specifying dependencies for the Gem + +It is important to note that while the gem has its own `Gemfile`, in the +actual application the top-level `Gemfile` for the monolith GitLab will be +used instead of the individual `Gemfile` sitting in the directory of the gem. + +This means we should be aware that the `Gemfile` for the gem should not use +any versions of dependencies which might be conflicting with the top-level +`Gemfile`, and we should try to use the same dependencies if possible. + +An example of this is [Rack](https://rubygems.org/gems/rack). If the monolith +is using Rack 2 and we're in the process of +[upgrading to Rack 3](https://gitlab.com/gitlab-org/gitlab/-/issues/396273), +all gems we develop should also be tested against Rack 2, optionally also with +Rack 3 if a separate `Gemfile` is used in CI. See an +[example here](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140463). + +Note that this does not limit to just Rack, but any dependencies. + ### Examples of Gem extractions The `gitlab-utils` is a Gem containing as of set of class that implement common intrinsic functions @@ -254,12 +273,12 @@ The project for a new Gem should always be created in [`gitlab-org/ruby/gems` na 1. Create a project in the [`gitlab-org/ruby/gems` group](https://gitlab.com/gitlab-org/ruby/gems/) (or in a subgroup of it): 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). - 1. Use the [gem-release CI component](https://gitlab.com/gitlab-org/quality/pipeline-common/-/tree/master/gem-release) + 1. Use the [`gem-release` CI component](https://gitlab.com/gitlab-org/components/gem-release) to release and publish new gem versions by adding the following to their `.gitlab-ci.yml`: ```yaml include: - - component: gitlab.com/gitlab-org/quality/pipeline-common/gem-release@<REPLACE WITH LATEST TAG FROM https://gitlab.com/gitlab-org/quality/pipeline-common/-/releases> + - component: gitlab.com/gitlab-org/components/gem-release/gem-release@~latest ``` This job will handle building and publishing the gem (it uses a `gitlab_rubygems` Rubygems.org diff --git a/doc/development/geo.md b/doc/development/geo.md index d8b000c090a..365893b3be7 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -472,6 +472,10 @@ and the administrator can see this in the [Geo Admin Area](../administration/geo Geo secondaries can proxy web requests to the primary. Read more on the [Geo proxying (development) page](geo/proxying.md). +## Geo API + +Geo uses the external [API](geo/api.md) to facilitate communication between various components. + ## Glossary ### Primary site diff --git a/doc/development/geo/api.md b/doc/development/geo/api.md new file mode 100644 index 00000000000..b9f3f7ce5c9 --- /dev/null +++ b/doc/development/geo/api.md @@ -0,0 +1,32 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Geo API **(PREMIUM SELF)** + +The Geo API is used internally by GitLab components to assist in coordinating Geo actions. It is inaccessible to admins or users. + +## Fetch pipeline refs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/415179) in GitLab 16.7. + +This method returns a list of branches matching `pipeline/refs/X` that exist on the repository for `gl_repository` on the current Geo node. This endpoint is used by runners registered with a secondary Geo instance to check if a repository is up to date. + +```plaintext +GET /geo/repositories/:gl_repository/pipeline_refs +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|--------------------------|----------|----------|-----------------------| +| `gl_repository` | string | Yes | The `gl_repository` ID of the repository to query | + +If successful, returns [`200`](../../api/rest/index.md#status-codes) and the following +response attributes: + +| Attribute | Type | Description | +|--------------------------|----------|-----------------------| +| `attribute` | 'array' | An array of ids matching `refs/pipeline/X` created for running pipelines. | diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index f3e35f9cdf8..5bfa77e2aa3 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -42,7 +42,7 @@ repositories that depend on the object pool. The danger lies in `git prune`, and `git gc` calls `git prune`. The problem is that `git prune`, when running in a pool repository, cannot -reliable decide if an object is no longer needed. +reliably decide if an object is no longer needed. ### Git alternates in GitLab: pool repositories @@ -51,7 +51,7 @@ which are hidden from the user. We then use Git alternates to let a collection of project repositories borrow from a single pool repository. We call such a collection of project repositories a pool. Pools form star-shaped networks of repositories -that borrow from a single pool, which resemble (but not be +that borrow from a single pool, which resemble (but are not identical to) the fork networks that get formed when users fork projects. diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 56a4ea898df..9e9216c4f66 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -347,6 +347,23 @@ execute services that: - [Continue their loop](https://gitlab.com/gitlab-org/gitlab/-/blob/487521cc26c1e2bdba4fc67c14478d2b2a5f2bfa/lib/gitlab/github_import/importer/attachments/issues_importer.rb#L27) from where it left off. +## `sidekiq_options dead: false` + +Typically when a worker's retries are exhausted they go to the Sidekiq dead set +and can be retried by an instance admin. + +`GithubImport::Queue` sets the Sidekiq worker option `dead: false` to prevent +this from happening to GitHub importer workers. + +The reason is: + +- The dead set has a max limit and if object importer workers (ones that include + `ObjectImporter`) fail en masse they can spam the dead set and push other workers out. +- Stage workers (ones that include `StageMethods`) + [fail the import](https://gitlab.com/gitlab-org/gitlab/-/blob/dd7cde8d6a28254b9c7aff27f9bf6b7be1ac7532/app/workers/concerns/gitlab/github_import/stage_methods.rb#L23) + when their retries are exhausted, so a retry would be guaranteed to + [be a no-op](https://gitlab.com/gitlab-org/gitlab/-/blob/dd7cde8d6a28254b9c7aff27f9bf6b7be1ac7532/app/workers/concerns/gitlab/github_import/stage_methods.rb#L55-63). + ## Mapping labels and milestones To reduce pressure on the database we do not query it when setting labels and diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index a9f4b22c778..c6471d9720c 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -26,7 +26,7 @@ can still have specifics. They are described in their respective The Go upgrade documentation [provides an overview](go_upgrade.md#overview) of how GitLab manages and ships Go binary support. -If a GitLab component requires a newer version of Go, +If a GitLab component requires a newer version of Go, follow the [upgrade process](go_upgrade.md#updating-go-version) to ensure no customer, team, or component is adversely impacted. Sometimes, individual projects must also [manage builds with multiple versions of Go](go_upgrade.md#supporting-multiple-go-versions). diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 35e423b28e9..16f96687c4f 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -16,7 +16,7 @@ are very appreciative of the work done by translators and proofreaders! - Albanian - Proofreaders needed. - Amharic - - Tsegaselassie Tadesse - [GitLab](https://gitlab.com/tsega), [Crowdin](https://crowdin.com/profile/tsegaselassi) + - Proofreaders needed. - Arabic - Proofreaders needed. - Basque @@ -24,51 +24,38 @@ are very appreciative of the work done by translators and proofreaders! - Belarusian - Anton Katsuba - [GitLab](https://gitlab.com/coinvariant), [Crowdin](https://crowdin.com/profile/aerialfiddle) - Bosnian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Bulgarian - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Catalan - - David Planella - [GitLab](https://gitlab.com/dplanella), [Crowdin](https://crowdin.com/profile/dplanella) + - Proofreaders needed. - Chinese Simplified 简体中文 - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Xiaogang Wen - [GitLab](https://gitlab.com/xiaogang_cn), [Crowdin](https://crowdin.com/profile/xiaogang_gitlab) - Zhiyuan Lu - [GitLab](https://gitlab.com/luzhiyuan.deer), [Crowdin](https://crowdin.com/profile/luzhiyuan.deer) - Chinese Traditional 繁體中文 - - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) - - Yi-Jyun Pan - [GitLab](https://gitlab.com/pan93412), [Crowdin](https://crowdin.com/profile/pan93412) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - Hansel Wang - [GitLab](https://gitlab.com/airness), [Crowdin](https://crowdin.com/profile/airness) - Chinese Traditional, Hong Kong 繁體中文 (香港) - - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - - Ivan Ip - [GitLab](https://gitlab.com/lifehome), [Crowdin](https://crowdin.com/profile/lifehome) + - Proofreaders needed. - Croatian - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Czech - - Jan Urbanec - [GitLab](https://gitlab.com/TatranskyMedved), [Crowdin](https://crowdin.com/profile/Tatranskymedved) + - Proofreaders needed. - Danish - - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) - scootergrisen - [GitLab](https://gitlab.com/scootergrisen), [Crowdin](https://crowdin.com/profile/scootergrisen) - Dutch - - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) + - Proofreaders needed. - English (UK) - - Andrew Smith - [GitLab](https://gitlab.com/espadav8), [Crowdin](https://crowdin.com/profile/espadav8) + - Proofreaders needed. - Esperanto - - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) + - Proofreaders needed. - Estonian - Proofreaders needed. - Filipino - - Andrei Jiroh Halili - [GitLab](https://gitlab.com/ajhalili2006), [Crowdin](https://crowdin.com/profile/AndreiJirohHaliliDev2006) + - Proofreaders needed. - French - - Davy Defaud - [GitLab](https://gitlab.com/DevDef), [Crowdin](https://crowdin.com/profile/DevDef) - - Germain Gorisse - [GitLab](https://gitlab.com/ggorisse), [Crowdin](https://crowdin.com/profile/germaingorisse) - Xavier Delatour - [GitLab](https://gitlab.com/xdelatour), [Crowdin](https://crowdin.com/profile/xdelatour) - Galician - - Antón Méixome - [Crowdin](https://crowdin.com/profile/meixome) - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) + - Proofreaders needed. - German - - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) - - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Vladislav Wanner - [GitLab](https://gitlab.com/RumBugen), [Crowdin](https://crowdin.com/profile/RumBugen) - Daniel Ziegenberg - [GitLab](https://gitlab.com/ziegenberg), [Crowdin](https://crowdin.com/profile/ziegenberg) - Greek @@ -80,69 +67,47 @@ are very appreciative of the work done by translators and proofreaders! - Hungarian - Proofreaders needed. - Indonesian - - Adi Ferdian - [GitLab](https://gitlab.com/adiferd), [Crowdin](https://crowdin.com/profile/adiferd) - - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) + - Proofreaders needed. - Italian - - Massimiliano Cuttini - [GitLab](https://gitlab.com/maxcuttins), [Crowdin](https://crowdin.com/profile/maxcuttins) - - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) + - Proofreaders needed. - Japanese - - Hiroyuki Sato - [GitLab](https://gitlab.com/hiroponz), [Crowdin](https://crowdin.com/profile/hiroponz) - Tomo Dote - [GitLab](https://gitlab.com/fu7mu4), [Crowdin](https://crowdin.com/profile/fu7mu4) - - Hiromi Nozawa - [GitLab](https://gitlab.com/hir0mi), [Crowdin](https://crowdin.com/profile/hir0mi) - - Takuya Noguchi - [GitLab](https://gitlab.com/tnir), [Crowdin](https://crowdin.com/profile/tnir) - Tsukasa Komatsubara - [GitLab](https://gitlab.com/tkomatsubara), [Crowdin](https://crowdin.com/profile/tkomatsubara) - Korean - - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - - Ji Hun Oh - [GitLab](https://gitlab.com/Baw-Appie), [Crowdin](https://crowdin.com/profile/BawAppie) - - Jeongwhan Choi - [GitLab](https://gitlab.com/jeongwhanchoi), [Crowdin](https://crowdin.com/profile/jeongwhanchoi) - Sunjung Park - [GitLab](https://gitlab.com/sunjungp), [Crowdin](https://crowdin.com/profile/sunjungp) + - Hwanyong Lee - [GitLab](https://gitlab.com/hwan_ajou), [Crowdin](https://crowdin.com/profile/grbear) - Mongolian - Proofreaders needed. - Norwegian Bokmal - Imre Kristoffer Eilertsen - [GitLab](https://gitlab.com/DandelionSprout), [Crowdin](https://crowdin.com/profile/DandelionSprout) - Polish - - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) - - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [Crowdin](https://crowdin.com/profile/villaincandle) - - Jakub Gładykowski - [GitLab](https://gitlab.com/gladykov), [Crowdin](https://crowdin.com/profile/gladykov) + - Proofreaders needed. - Portuguese - - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [Crowdin](https://crowdin.com/profile/ldiogotrindade) + - Proofreaders needed. - Portuguese, Brazilian - - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra) - - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) - - Horberlan Brito - [GitLab](https://gitlab.com/horberlan), [Crowdin](https://crowdin.com/profile/horberlan) - Romanian - - Mircea Pop - [GitLab](https://gitlab.com/eeex), [Crowdin](https://crowdin.com/profile/eex) - - Rareș Pița - [GitLab](https://gitlab.com/dlphin) - - Nicolae Liviu - [GitLab](https://gitlab.com/nicklcanada), [Crowdin](https://crowdin.com/profile/nicklcanada) + - Proofreaders needed. - Russian - - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) - Mark Minakou - [GitLab](https://gitlab.com/sandzhaj), [Crowdin](https://crowdin.com/profile/sandzhaj) - - NickVolynkin - [Crowdin](https://crowdin.com/profile/NickVolynkin) - Andrey Komarov - [GitLab](https://gitlab.com/elkamarado), [Crowdin](https://crowdin.com/profile/kamarado) - - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Latin and Cyrillic) - - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) + - Proofreaders needed. - Sinhalese/Sinhala සිංහල - හෙළබස (HelaBasa) - [GitLab](https://gitlab.com/helabasa), [Crowdin](https://crowdin.com/profile/helabasa) - Slovak - Proofreaders needed. - Spanish - - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - David Elizondo - [GitLab](https://gitlab.com/daelmo), [Crowdin](https://crowdin.com/profile/daelmo) - Pablo Reyes - [GitLab](https://gitlab.com/pabloryst9n), [Crowdin](https://crowdin.com/profile/pabloryst9n) - - Swedish - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [Crowdin](https://crowdin.com/profile/nlssn) - Turkish - - Ali Demirtaş - [GitLab](https://gitlab.com/alidemirtas), [Crowdin](https://crowdin.com/profile/alidemirtas) - - Rıfat Ünalmış (Rifat Unalmis) - [GitLab](https://gitlab.com/runalmis), [Crowdin](https://crowdin.com/profile/runalmis) - - İsmail Arılık - [GitLab](https://gitlab.com/ismailarilik), [Crowdin](https://crowdin.com/profile/ismailarilik) + - Proofreaders needed. - Ukrainian - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) - Welsh - - Delyth Prys - [GitLab](https://gitlab.com/Delyth), [Crowdin](https://crowdin.com/profile/DelythPrys) + - Proofreaders needed. <!-- vale gitlab.Spelling = YES --> ## Become a proofreader diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 7149d431c30..ed3afb8efa6 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -96,7 +96,7 @@ For example, in German, the word _user_ can be translated into _Benutzer_ (male) ### Updating the glossary -To propose additions to the glossary, +To propose additions to the glossary, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). ## French translation guidelines diff --git a/doc/development/img/runner_fleet_dashboard.png b/doc/development/img/runner_fleet_dashboard.png Binary files differdeleted file mode 100644 index 242ebf4aea9..00000000000 --- a/doc/development/img/runner_fleet_dashboard.png +++ /dev/null diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 3fb89605bdd..6709c748994 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -7,16 +7,16 @@ info: Any user with at least the Maintainer role can merge updates to this conte # Security scanner integration Integrating a security scanner into GitLab consists of providing end users -with a [CI job definition](../../ci/yaml/index.md) -they can add to their CI configuration files to scan their GitLab projects. -This CI job should then output its results in a GitLab-specified format. These results are then +with a [CI/CD job definition](../../ci/jobs/index.md) +they can add to their CI/CD configuration files to scan their GitLab projects. +This job should then output its results in a GitLab-specified format. These results are then automatically presented in various places in GitLab, such as the Pipeline view, merge request widget, and Security Dashboard. The scanning job is usually based on a [Docker image](https://docs.docker.com/) that contains the scanner and all its dependencies in a self-contained environment. -This page documents requirements and guidelines for writing CI jobs that implement a security +This page documents requirements and guidelines for writing CI/CD jobs that implement a security scanner, as well as requirements and guidelines for the Docker image. ## Job definition @@ -285,12 +285,12 @@ See the [`logutil` README](https://gitlab.com/gitlab-org/security-products/analy The report is a JSON document that combines vulnerabilities with possible remediations. -This documentation gives an overview of the report JSON format, -as well as recommendations and examples to help integrators set its fields. +This documentation gives an overview of the report JSON format, recommendations, and examples to +help integrators set its fields. The format is extensively described in the documentation of -[SAST](../../user/application_security/sast/index.md#reports-json-format), +[SAST](../../user/application_security/sast/index.md#output), [DAST](../../user/application_security/dast/proxy-based.md#reports), -[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), +[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#output), and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format) You can find the schemas for these scanners here: diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 53c333a6f13..d0a2c8b828f 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -87,8 +87,8 @@ and complete an integration with the Secure stage. - Read about [job artifacts](../../ci/jobs/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). - - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). - - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#reports-json-format). + - Documentation for [SAST output](../../user/application_security/sast/index.md#output). + - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#output). - Documentation for [Container Scanning reports](../../user/application_security/container_scanning/index.md#reports-json-format). - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Container-Scanning.gitlab-ci.yml). - If you need a new kind of scan or report, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new#) diff --git a/doc/development/internal_analytics/index.md b/doc/development/internal_analytics/index.md index b5403f56600..29d1a46c3cf 100644 --- a/doc/development/internal_analytics/index.md +++ b/doc/development/internal_analytics/index.md @@ -17,8 +17,8 @@ when developing new features or instrumenting existing ones. <div class="video-fallback"> See the video about <a href="https://www.youtube.com/watch?v=GtFNXbjygWo">the concepts of events and metrics.</a> </div> -<figure class="video_container"> - <iframe src="https://www.youtube-nocookie.com/embed/GtFNXbjygWo" frameborder="0" allowfullscreen="true"> </iframe> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/GtFNXbjygWo" frameborder="0" allowfullscreen> </iframe> </figure> Events and metrics are the foundation of the internal analytics system. @@ -95,7 +95,6 @@ SELECT COUNT(*) as event_occurences FROM common_mart.mart_behavior_structured_event WHERE event_action = 'feature_used' -AND event_category = 'InternalEventTracking' AND behavior_date > '2023-08-01' --restricted minimum date for performance AND app_id='gitlab' -- use gitlab for production events and gitlab-staging for events from staging GROUP BY 1 ORDER BY 1 desc diff --git a/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md b/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md index 2c53185a5db..9417861eee9 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/event_definition_guide.md @@ -26,7 +26,7 @@ Each event is defined in a separate YAML file consisting of the following fields |------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `description` | yes | A description of the event. | | `category` | yes | Always InternalEventTracking (only different for legacy events). | -| `action` | yes | A unique name for the event. | +| `action` | yes | A unique name for the event. Use the format `<operation>_<target_of_operation>_<where/when>`. <br/><br/> Ex: `publish_go_module_to_the_registry_from_pipeline` <br/>`<operation> = publish`<br/>`<target> = go_module`<br/>`<when/where> = to_the_registry_from_pipeline`. | | `identifiers` | no | A list of identifiers sent with the event. Can be set to one or more of `project`, `user`, or `namespace`. | | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the event. | @@ -43,7 +43,7 @@ This is an example YAML file for an internal event: ```yaml description: A user visited a product analytics dashboard category: InternalEventTracking -action: user_visited_dashboard +action: visit_product_analytics_dashboard identifiers: - project - user diff --git a/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md b/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md index 56e83184060..244482a4c2e 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/local_setup_and_debugging.md @@ -48,6 +48,15 @@ By default, self-managed instances do not collect event data via Snowplow. We ca 1. You can now see all events being sent by your local instance in the [Snowplow Micro UI](http://localhost:9091/micro/ui) and can filter for specific events. +### Introduction to Snowplow Micro UI and API + +<div class="video-fallback"> + Watch the video about <a href="https://www.youtube.com/watch?v=netZ0TogNcA">Snowplow Micro</a> +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/netZ0TogNcA" frameborder="0" allowfullscreen> </iframe> +</figure> + ## Configure a remote event collector On GitLab.com events are sent to a collector configured by GitLab. By default, self-managed instances do not have a collector configured and do not collect data with Snowplow. @@ -72,8 +81,8 @@ You can configure your self-managed GitLab instance to use a custom Snowplow col <div class="video-fallback"> Watch the demo video about the <a href="https://www.youtube.com/watch?v=R7vT-VEzZOI">Internal Events Tracking Monitor</a> </div> -<figure class="video_container"> - <iframe src="https://www.youtube-nocookie.com/embed/R7vT-VEzZOI" frameborder="0" allowfullscreen="true"> </iframe> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/R7vT-VEzZOI" frameborder="0" allowfullscreen> </iframe> </figure> To understand how events are triggered and metrics are updated while you use the Rails app locally or `rails console`, diff --git a/doc/development/internal_analytics/internal_event_instrumentation/migration.md b/doc/development/internal_analytics/internal_event_instrumentation/migration.md index 2ef439e21e9..79ca45ed84c 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/migration.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/migration.md @@ -125,6 +125,7 @@ To start using Internal Events Tracking, follow these steps: 1. Create an event definition that describes `git_write_action` ([guide](event_definition_guide.md)). 1. Find metric definitions that list `git_write_action` in the events section (`20210216182041_action_monthly_active_users_git_write.yml` and `20210216184045_git_write_action_weekly.yml`). 1. Change the `data_source` from `redis_hll` to `internal_events` in the metric definition files. +1. Remove the `instrumentation_class` property. It's not used for Internal Events metrics. 1. Add an `events` section to both metric definition files. ```yaml diff --git a/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md b/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md index 6f48f83e7ca..c70f7debe41 100644 --- a/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md +++ b/doc/development/internal_analytics/internal_event_instrumentation/quick_start.md @@ -31,6 +31,13 @@ Triggering an event and thereby updating a metric is slightly different on backe ### Backend tracking +<div class="video-fallback"> + Watch the video about <a href="https://www.youtube.com/watch?v=Teid7o_2Mmg">Backend instrumentation using Internal Events</a> +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/Teid7o_2Mmg" frameborder="0" allowfullscreen> </iframe> +</figure> + To trigger an event, call the `Gitlab::InternalEvents.track_event` method with the desired arguments: ```ruby @@ -50,6 +57,53 @@ It is encouraged to fill out as many of `user`, `namespace` and `project` as pos If a `project` but no `namespace` is provided, the `project.namespace` is used as the `namespace` for the event. +#### Controller and API helpers + +There is a helper module `ProductAnalyticsTracking` for controllers you can use to track internal events for particular controller actions by calling `#track_internal_event`: + +```ruby +class Projects::PipelinesController < Projects::ApplicationController + include ProductAnalyticsTracking + + track_internal_event :charts, name: 'p_analytics_ci_cd_pipelines', conditions: -> { should_track_ci_cd_pipelines? } + + def charts + ... + end + + private + + def should_track_ci_cd_pipelines? + params[:chart].blank? || params[:chart] == 'pipelines' + end +end +``` + +You need to add these two methods to the controller body, so that the helper can get the current project and namespace for the event: + +```ruby + private + + def tracking_namespace_source + project.namespace + end + + def tracking_project_source + project + end +``` + +Also, there is an API helper: + +```ruby +track_event( + event_name, + user: current_user, + namespace_id: namespace_id, + project_id: project_id +) +``` + ### Frontend tracking Any frontend tracking call automatically passes the values `user.id`, `namespace.id`, and `project.id` from the current context of the page. diff --git a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md b/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md deleted file mode 100644 index 8793e317cdc..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/event_definition_guide.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/event_definition_guide.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/event_definition_guide.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/index.md b/doc/development/internal_analytics/internal_event_tracking/index.md deleted file mode 100644 index 03b539d2a03..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/index.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/index.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/introduction.md b/doc/development/internal_analytics/internal_event_tracking/introduction.md deleted file mode 100644 index 3f769e7935e..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/introduction.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/introduction.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/introduction.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/migration.md b/doc/development/internal_analytics/internal_event_tracking/migration.md deleted file mode 100644 index 9d78f544f4c..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/migration.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/migration.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/migration.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/internal_event_tracking/quick_start.md b/doc/development/internal_analytics/internal_event_tracking/quick_start.md deleted file mode 100644 index 4c378c4d7bb..00000000000 --- a/doc/development/internal_analytics/internal_event_tracking/quick_start.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../internal_event_instrumentation/quick_start.md' -remove_date: '2023-12-27' ---- - -This document was moved to [another location](../internal_event_instrumentation/quick_start.md). - -<!-- This redirect file can be deleted after <2023-12-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/metrics/metrics_dictionary.md b/doc/development/internal_analytics/metrics/metrics_dictionary.md index 463d5be9100..ba3fb28743a 100644 --- a/doc/development/internal_analytics/metrics/metrics_dictionary.md +++ b/doc/development/internal_analytics/metrics/metrics_dictionary.md @@ -81,9 +81,9 @@ Metric definitions can have one of the following values for `value_type`: - `number` - `string` - `object`: A metric with `value_type: object` must have `value_json_schema` with a link to the JSON schema for the object. -In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. -An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), -which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. + In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. + An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), + which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. ### Metric `time_frame` diff --git a/doc/development/internal_analytics/review_guidelines.md b/doc/development/internal_analytics/review_guidelines.md index eb59b834cbc..802a6f410ed 100644 --- a/doc/development/internal_analytics/review_guidelines.md +++ b/doc/development/internal_analytics/review_guidelines.md @@ -29,7 +29,7 @@ In most cases, an Analytics Instrumentation review is automatically added, but i #### The merge request **author** should - Decide whether a Analytics Instrumentation review is needed. You can skip the Analytics Instrumentation -review and remove the labels if the changes are not related to the Analytics Instrumentation domain. + review and remove the labels if the changes are not related to the Analytics Instrumentation domain. - If an Analytics Instrumentation review is needed and was not assigned automatically, add the labels `~analytics instrumentation` and `~analytics instrumentation::review pending`. - Use reviewer roulette to assign an [Analytics Instrumentation reviewer](https://gitlab-org.gitlab.io/gitlab-roulette/?hourFormat24=true&visible=reviewer%7Canalytics+instrumentation) who is not the author. diff --git a/doc/development/internal_analytics/service_ping/review_guidelines.md b/doc/development/internal_analytics/service_ping/review_guidelines.md deleted file mode 100644 index 0ca7b084fc4..00000000000 --- a/doc/development/internal_analytics/service_ping/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../review_guidelines.md' -remove_date: '2023-12-29' ---- - -This document was moved to [another location](../review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-12-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/troubleshooting.md b/doc/development/internal_analytics/service_ping/troubleshooting.md index 1b74921fb2f..637aec384c0 100644 --- a/doc/development/internal_analytics/service_ping/troubleshooting.md +++ b/doc/development/internal_analytics/service_ping/troubleshooting.md @@ -90,39 +90,7 @@ However, it has the following limitations: always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping. -## Service Ping Payload drop - -### Symptoms - -You will be alerted by the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) and their [Monte Carlo alerting](https://about.gitlab.com/handbook/business-technology/data-team/platform/monte-carlo/). - -### Locating the problem - -First you need to identify at which stage in Service Ping data pipeline the drop is occurring. - -Start at [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489) on Sisense. - -You can use [this query](https://gitlab.com/gitlab-org/gitlab/-/issues/347298#note_836685350) as an example, to start detecting when the drop started. - -### Troubleshoot the GitLab application layer - -We conducted an investigation into an unexpected drop in Service ping Payload events volume. -GitLab team members can view more information in this confidential issue: -`https://gitlab.com/gitlab-data/analytics/-/issues/11071` - -### Troubleshoot VersionApp layer - -Check if the [export jobs](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/tree/main/#data-export-using-pipeline-schedules) are successful. - -Check [Service Ping errors](https://app.periscopedata.com/app/gitlab/968489?widget=14609989&udv=0) in the [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489). - -### Troubleshoot Google Storage layer - -Check if the files are present in [Google Storage](https://console.cloud.google.com/storage/browser/cloudsql-gs-production-efd5e8-cloudsql-exports;tab=objects?project=gs-production-efd5e8&prefix=&forceOnObjectsSortingFiltering=false). - -### Troubleshoot the data warehouse layer - -Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). +## Troubleshooting ### Cannot disable Service Ping with the configuration file diff --git a/doc/development/internal_analytics/snowplow/review_guidelines.md b/doc/development/internal_analytics/snowplow/review_guidelines.md deleted file mode 100644 index 0ca7b084fc4..00000000000 --- a/doc/development/internal_analytics/snowplow/review_guidelines.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../review_guidelines.md' -remove_date: '2023-12-29' ---- - -This document was moved to [another location](../review_guidelines.md). - -<!-- This redirect file can be deleted after <2023-12-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 84f5d09418f..fc73fcb6c2c 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -534,7 +534,6 @@ metric counters. | `counters["k8s_api_proxy_requests_via_user_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_user_access` counter by | | `counters["k8s_api_proxy_requests_via_pat_access"]` | integer | no | The number to increase the `k8s_api_proxy_requests_via_pat_access` counter by | | `unique_counters` | hash | no | Array of unique numbers | -| `unique_counters["agent_users_using_ci_tunnel"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel to track the `agent_users_using_ci_tunnel` metric event | | `unique_counters["k8s_api_proxy_requests_unique_users_via_ci_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_users_via_ci_access` metric event | | `unique_counters["k8s_api_proxy_requests_unique_agents_via_ci_access"]` | integer array | no | The set of unique agent ids that have interacted a CI Tunnel via `ci_access` to track the `k8s_api_proxy_requests_unique_agents_via_ci_access` metric event | | `unique_counters["k8s_api_proxy_requests_unique_users_via_user_access"]` | integer array | no | The set of unique user ids that have interacted a CI Tunnel via `user_access` to track the `k8s_api_proxy_requests_unique_users_via_user_access` metric event | diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 289c258fafe..2fb7d20cabc 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -180,32 +180,32 @@ sequenceDiagram 1. The user requests the project archive from the UI. 1. Workhorse forwards this request to Rails. 1. If the user is authorized to download the archive, Rails replies with -an HTTP header of `Gitlab-Workhorse-Send-Data` with a base64-encoded -JSON payload prefaced with `git-archive`. This payload includes the -`SendArchiveRequest` binary message, which is encoded again in base64. + an HTTP header of `Gitlab-Workhorse-Send-Data` with a base64-encoded + JSON payload prefaced with `git-archive`. This payload includes the + `SendArchiveRequest` binary message, which is encoded again in base64. 1. Workhorse decodes the `Gitlab-Workhorse-Send-Data` payload. If the -archive already exists in the archive cache, Workhorse sends that -file. Otherwise, Workhorse sends the `SendArchiveRequest` to the -appropriate Gitaly server. + archive already exists in the archive cache, Workhorse sends that + file. Otherwise, Workhorse sends the `SendArchiveRequest` to the + appropriate Gitaly server. 1. The Gitaly server calls `git archive <ref>` to begin generating -the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled, -Gitaly enables a custom LFS smudge filter via the `-c -filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option. + the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled, + Gitaly enables a custom LFS smudge filter via the `-c + filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option. 1. When `git` identifies a possible LFS pointer using the -`.gitattributes` file, `git` calls `gitaly-lfs-smudge` and provides the -LFS pointer via the standard input. Gitaly provides `GL_PROJECT_PATH` -and `GL_INTERNAL_CONFIG` as environment variables to enable lookup of -the LFS object. + `.gitattributes` file, `git` calls `gitaly-lfs-smudge` and provides the + LFS pointer via the standard input. Gitaly provides `GL_PROJECT_PATH` + and `GL_INTERNAL_CONFIG` as environment variables to enable lookup of + the LFS object. 1. If a valid LFS pointer is decoded, `gitaly-lfs-smudge` makes an -internal API call to Workhorse to download the LFS object from GitLab. + internal API call to Workhorse to download the LFS object from GitLab. 1. Workhorse forwards this request to Rails. If the LFS object exists -and is associated with the project, Rails sends `ArchivePath` either -with a path where the LFS object resides (for local disk) or a -pre-signed URL (when object storage is enabled) via the -`Gitlab-Workhorse-Send-Data` HTTP header with a payload prefaced with -`send-url`. + and is associated with the project, Rails sends `ArchivePath` either + with a path where the LFS object resides (for local disk) or a + pre-signed URL (when object storage is enabled) via the + `Gitlab-Workhorse-Send-Data` HTTP header with a payload prefaced with + `send-url`. 1. Workhorse retrieves the file and send it to the `gitaly-lfs-smudge` -process, which writes the contents to the standard output. + process, which writes the contents to the standard output. 1. `git` reads this output and sends it back to the Gitaly process. 1. Gitaly sends the data back to Rails. 1. The archive data is sent back to the client. diff --git a/doc/development/logging.md b/doc/development/logging.md index 2af914d76ef..1cfcc6530c5 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -330,7 +330,7 @@ Entry points can be seen at: When adding new attributes, make sure they're exposed within the context of the entry points above and: - Pass them within the hash to the `with_context` (or `push`) method (make sure to pass a Proc if the -method or variable shouldn't be evaluated right away) + method or variable shouldn't be evaluated right away) - Change `Gitlab::ApplicationContext` to accept these new values - Make sure the new attributes are accepted at [`Labkit::Context`](https://gitlab.com/gitlab-org/labkit-ruby/blob/master/lib/labkit/context.rb) diff --git a/doc/development/merge_request_concepts/diffs/index.md b/doc/development/merge_request_concepts/diffs/index.md index ad0e8603983..eb7c836e610 100644 --- a/doc/development/merge_request_concepts/diffs/index.md +++ b/doc/development/merge_request_concepts/diffs/index.md @@ -28,6 +28,7 @@ codebase in the future: - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Recording on YouTube](https://www.youtube.com/watch?v=K6G3gMcFyek) + <!-- Video published on 2019-01-29 --> - Slides on [Google Slides](https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit) - [PDF slides](https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/) @@ -192,7 +193,7 @@ has been introduced. One of the key challenges to deal with when working on merge ref diffs are merge conflicts. If the target and source branch contains a merge conflict, the branches cannot be automatically merged. The -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) <!-- Video published on 2020-07-24 --> is a quick introduction to the problem and the motivation behind the [epic](https://gitlab.com/groups/gitlab-org/-/epics/854). In 13.5 a solution for both-modified merge diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 30f598ef736..be76680c44c 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -723,7 +723,7 @@ The `with_lock_retries` method **cannot** be used within the `change` method, yo 1. For each iteration, set a pre-configured `lock_timeout`. 1. Try to execute the given block. (`remove_column`). 1. If `LockWaitTimeout` error is raised, sleep for the pre-configured `sleep_time` -and retry the block. + and retry the block. 1. If no error is raised, the current iteration has successfully executed the block. For more information check the [`Gitlab::Database::WithLockRetries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/with_lock_retries.rb) class. The `with_lock_retries` helper method is implemented in the [`Gitlab::Database::MigrationHelpers`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers.rb) module. @@ -733,7 +733,7 @@ In a worst-case scenario, the method: - Executes the block for a maximum of 50 times over 40 minutes. - Most of the time is spent in a pre-configured sleep period after each iteration. - After the 50th retry, the block is executed without `lock_timeout`, just -like a standard migration invocation. + like a standard migration invocation. - If a lock cannot be acquired, the migration fails with `statement timeout` error. The migration might fail if there is a very long running transaction (40+ minutes) diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 40a30aa4926..716e42655d0 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -234,7 +234,7 @@ And these deployments align perfectly with application changes. 1. At the beginning we have `Version N` on `Schema A`. 1. Then we have a _long_ transition period with both `Version N` and `Version N+1` on `Schema B`. 1. When we only have `Version N+1` on `Schema B` the schema changes again. -1. Finally we have `Version N+1` on `Schema C`. +1. Finally we have `Version N+1` on `Schema C`. With all those details in mind, let's imagine we need to replace a query, and this query has an index to support it. @@ -310,10 +310,10 @@ variable `CI_NODE_TOTAL` being an integer failed. This was caused because after 1. New code: Sidekiq created a new pipeline and new build. `build.options[:parallel]` is a `Hash`. 1. Old code: Runners requested a job from an API node that is running the previous version. 1. As a result, the [new code](https://gitlab.com/gitlab-org/gitlab/-/blob/42b82a9a3ac5a96f9152aad6cbc583c42b9fb082/app/models/concerns/ci/contextable.rb#L104) -was not run on the API server. The runner's request failed because the -older API server tried return the `CI_NODE_TOTAL` CI/CD variable, but -instead of sending an integer value (for example, 9), it sent a serialized -`Hash` value (`{:number=>9, :total=>9}`). + was not run on the API server. The runner's request failed because the + older API server tried return the `CI_NODE_TOTAL` CI/CD variable, but + instead of sending an integer value (for example, 9), it sent a serialized + `Hash` value (`{:number=>9, :total=>9}`). If you look at the [deployment pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/202212), you see all nodes were updated in parallel: @@ -322,11 +322,11 @@ you see all nodes were updated in parallel: However, even though the updated started around the same time, the completion time varied significantly: -|Node type|Duration (min)| -|---------|--------------| -|API |54 | -|Sidekiq |21 | -|K8S |8 | +| Node type | Duration (min) | +|-----------|----------------| +| API | 54 | +| Sidekiq | 21 | +| K8S | 8 | Builds that used the `parallel` keyword and depended on `CI_NODE_TOTAL` and `CI_NODE_INDEX` would fail during the time after Sidekiq was diff --git a/doc/development/packages/cleanup_policies.md b/doc/development/packages/cleanup_policies.md index bbec6ce0623..789bb408dd7 100644 --- a/doc/development/packages/cleanup_policies.md +++ b/doc/development/packages/cleanup_policies.md @@ -27,7 +27,7 @@ The parameters are split into two groups: - The parameters that define tags to destroy: - `older_than`. Destroy tags older than this timestamp. - `name_regex`. Destroy tags matching this regular expression. - + The remaining parameters impact when the policy is executed: - `enabled`. Defines if the policy is enabled or not. @@ -41,8 +41,7 @@ follows this design. - Policy executions are limited in time. - Policy executions are either complete or partial. -- The background jobs will consider the next job to be executed based on two -priorities: +- The background jobs will consider the next job to be executed based on two priorities: - Policy with a `next_run_at` in the past. - Partially executed policies. @@ -54,7 +53,7 @@ Background jobs for this execution are organized on: - A cron background job that runs every hour. - A set of background jobs that will loop on container repositories that need -a policy execution. + a policy execution. #### The cron background job @@ -63,7 +62,7 @@ is quite simple. Its main tasks are: 1. Check if there are any container repositories in need of a cleanup. If any, -enqueue as many limited capacity jobs as necessary, up to a limit. + enqueue as many limited capacity jobs as necessary, up to a limit. 1. Compute metrics for cleanup policies and log them. #### The limited capacity job @@ -97,14 +96,14 @@ flowchart TD ``` - [`ContainerExpirationPolicies::CleanupService`](https://gitlab.com/gitlab-org/gitlab/-/blob/6546ffc6fe4e9b447a1b7f050edddb8926fe4a3d/app/services/container_expiration_policies/cleanup_service.rb). -This service mainly deals with container repository `expiration_policy_cleanup_status` -updates and will call the cleanup tags service. + This service mainly deals with container repository `expiration_policy_cleanup_status` + updates and will call the cleanup tags service. - [`Projects::ContainerRepository::CleanupTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/cleanup_tags_service.rb). -This service receives the policy parameters and builds the list of tags to -destroy on the container registry. + This service receives the policy parameters and builds the list of tags to + destroy on the container registry. - [`Projects::ContainerRepository::DeleteTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/delete_tags_service.rb). -This service receives a list of tags and loops on that list. For each tag, -the service will call the container registry API endpoint to destroy the target tag. + This service receives a list of tags and loops on that list. For each tag, + the service will call the container registry API endpoint to destroy the target tag. The cleanup tags service uses a very specific [execution order](../../user/packages/container_registry/reduce_container_registry_storage.md#how-the-cleanup-policy-works) to build the list of tags to destroy. diff --git a/doc/development/packages/settings.md b/doc/development/packages/settings.md index 89f91f41f4c..690f9ccae93 100644 --- a/doc/development/packages/settings.md +++ b/doc/development/packages/settings.md @@ -70,6 +70,8 @@ Setting | Table | Description `nuget_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate NuGet packages. `nuget_duplicate_exception_regex` | `namespace_package_settings` | Regex defining NuGet packages that are allowed to be duplicate when duplicates are not allowed. `nuget_symbol_server_enabled` | `namespace_package_settings` | Enable or disable the NuGet symbol server. +`terraform_module_duplicates_allowed` | `namespace_package_settings` | Allow or prevent duplicate Terraform module packages. +`terraform_module_duplicate_exception_regex` | `namespace_package_settings` | Regex defining Terraform module packages that are allowed to be duplicate when duplicates are not allowed. Dependency Proxy Cleanup Policies - `ttl` | `dependency_proxy_image_ttl_group_policies` | Number of days to retain an unused Dependency Proxy file before it is removed. Dependency Proxy - `enabled` | `dependency_proxy_image_ttl_group_policies` | Enable or disable the Dependency Proxy cleanup policy. diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md index f4710c951ed..7bf89789ee0 100644 --- a/doc/development/pages/index.md +++ b/doc/development/pages/index.md @@ -22,14 +22,14 @@ subdomain. You can set the GitLab Pages hostname: As `/etc/hosts` don't support wildcard hostnames, you must configure one entry for GitLab Pages, and then one entry for each page site: - ```plaintext - 127.0.0.1 gdk.test # If you're using GDK - 127.0.0.1 pages.gdk.test # Pages host - # Any namespace/group/user needs to be added - # as a subdomain to the pages host. This is because - # /etc/hosts doesn't accept wildcards - 127.0.0.1 root.pages.gdk.test # for the root pages - ``` +```plaintext +127.0.0.1 gdk.test # If you're using GDK +127.0.0.1 pages.gdk.test # Pages host +# Any namespace/group/user needs to be added +# as a subdomain to the pages host. This is because +# /etc/hosts doesn't accept wildcards +127.0.0.1 root.pages.gdk.test # for the root pages +``` ### With DNS wildcard alternatives @@ -151,8 +151,8 @@ GitLab Pages access control is disabled by default. To enable it: 1. Create an [Instance-wide OAuth application](../../integration/oauth_provider.md#create-an-instance-wide-application) with the `api` scope. 1. Set the value of your `redirect-uri` to the `pages-domain` authorization endpoint -(for example, `http://pages.gdk.test:3010/auth`). -The `redirect-uri` must not contain any GitLab Pages site domain. + (for example, `http://pages.gdk.test:3010/auth`). + The `redirect-uri` must not contain any GitLab Pages site domain. 1. Add the auth client configuration: diff --git a/doc/development/permissions/custom_roles.md b/doc/development/permissions/custom_roles.md index 9d02c0c6f39..53589658f56 100644 --- a/doc/development/permissions/custom_roles.md +++ b/doc/development/permissions/custom_roles.md @@ -34,7 +34,7 @@ Like static roles, custom roles are [inherited](../../user/project/members/index - A Group or project membership can be associated with any custom role that is defined on the root-level group of the group or project. - The `member_roles` table includes individual permissions and a `base_access_level` value. - The `base_access_level` must be a [valid access level](../../api/access_requests.md#valid-access-levels). -The `base_access_level` determines which abilities are included in the custom role. For example, if the `base_access_level` is `10`, the custom role will include any abilities that a static Guest role would receive, plus any additional abilities that are enabled by the `member_roles` record by setting an attribute, such as `read_code`, to true. + The `base_access_level` determines which abilities are included in the custom role. For example, if the `base_access_level` is `10`, the custom role will include any abilities that a static Guest role would receive, plus any additional abilities that are enabled by the `member_roles` record by setting an attribute, such as `read_code`, to true. - A custom role can enable additional abilities for a `base_access_level` but it cannot disable a permission. As a result, custom roles are "additive only". The rationale for this choice is [in this comment](https://gitlab.com/gitlab-org/gitlab/-/issues/352891#note_1059561579). - Custom role abilities are supported at project level and group level. @@ -169,7 +169,7 @@ For example, you see in `GroupPolicy` that there is an ability called than adding a row to the `member_roles` table for each ability, consider renaming them to `read_security_dashboard` and adding `read_security_dashboard` to the `member_roles` table. This is more expected because it means that -enabling `read_security_dashboard` on the parent group will enable the custom +enabling `read_security_dashboard` on the parent group will enable the custom role. For example, `GroupPolicy` has an ability called `read_group_security_dashboard` and `ProjectPolicy` has an ability called `read_project_security_dashboard`. If you would like to make both customizable, rather than adding a row to the `member_roles` table for each ability, consider renaming them to `read_security_dashboard` and adding @@ -183,23 +183,116 @@ security dashboard. To add a new ability to a custom role: - Generate YAML file by running `./ee/bin/custom-ability` generator -- Add a new column to `member_roles` table, for example in [this change in merge request 114734](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734/diffs#diff-content-5c53d6f1c29a272a87eecea3f62d017ab6635275). -- Add the ability to the `MemberRole` model, `ALL_CUSTOMIZABLE_PERMISSIONS` hash, for example in [this change in merge request 121534](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534/diffs#ce5ec769500a53ce2b603467d9984fc2b33ca71d_8_8). There are following possible keys in the `ALL_CUSTOMIZABLE_PERMISSIONS` hash: - - - `description` - description of the ability. - - `minimal_level` - minimal level a user has to have in order to be able to be assigned to the ability. - - `requirement` - required ability for the ability defined in the hash, in case the requirement is `false`, the ability can not be `true`. - +- Add a new column to `member_roles` table, either manually or by running `custom_roles:code` generator, eg. by running `rails generate gitlab:custom_roles:code --ability new_ability_name`. The ability parameter is case sensitive and has to exactly match the permission name from the YAML file. - Add the ability to the respective Policy for example in [this change in merge request 114734](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734/diffs#diff-content-edcbe28bdecbd848d4d9efdc5b5e9bddd2a7299e). -- Update the specs. +- Update the specs. Don't forget to add a spec to `ee/spec/requests/custom_roles` - the spec template file was pre-generated if you used the code generator +- Compile the documentation by running `bundle exec rake gitlab:custom_roles:compile_docs` +- Update the GraphQL documentation by running `bundle exec rake gitlab:graphql:compile_docs` Examples of merge requests adding new abilities to custom roles: - [Read code](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) - [Read vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114734) -- [Admin vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) - this is the newest MR implementing a new custom role ability. Some changes from the previous MRs are not necessary anymore (such as a change of the Preloader query or adding a method to `User` model). +- [Admin vulnerability](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) + +The above merge requests don't use YAML files and code generators. Some of the changes are not needed anymore. We will update the documentation once we have a permission implemented using the generators. + +If you have any concerns, put the new ability behind a feature flag. + +#### Documenting handling the feature flag + +- When you introduce a new custom ability under a feature flag, add the `feature_flag` attribute to the appropriate ability YAML file. +- When you enable the ability by default, add the `feature_flag_enabled_milestone` and `feature_flag_enabled_mr` attributes to the appropriate ability YAML file and regenerate the documentation. +- You do not have to include these attributes in the YAML file if the feature flag is enabled by default in the same release as the ability is introduced. + +#### Testing + +Unit tests are preferred to test out changes to any policies affected by the +addition of new custom permissions. Custom Roles is an Ultimate tier feature so +these tests can be found in the `ee/spec/policies` directory. The [spec file](https://gitlab.com/gitlab-org/gitlab/-/blob/13baa4e8c92a56260591a5bf0a58d3339890ee10/ee/spec/policies/project_policy_spec.rb#L2726-2740) for +the `ProjectPolicy` contains shared examples that can be used to test out the +following conditions: + +- when the `custom_roles` licensed feature is not enabled +- when the `custom_roles` licensed feature is enabled + - when a user is a member of a custom role via an inherited group member + - when a user is a member of a custom role via a direct group member + - when a user is a member of a custom role via a direct project membership + +Below is an example for testing out `ProjectPolicy` related changes. + +```ruby + context 'for a role with `custom_permission` enabled' do + let(:member_role_abilities) { { custom_permission: true } } + let(:allowed_abilities) { [:custom_permission] } + + it_behaves_like 'custom roles abilities' + end +``` + +Request specs are preferred to test out any endpoint that allow access via a custom role permission. +This includes controllers, REST API, and GraphQL. Examples of request specs can be found in `ee/spec/requests/custom_roles/`. In this directory you will find a sub-directory named after each permission that can be enabled via a custom role. +The `custom_roles` licensed feature must be enabled to test this functionality. + +Below is an example of the typical setup that is required to test out a +Rails Controller endpoint. + +```ruby + let_it_be(:user) { create(:user) } + let_it_be(:project) { create(:project, :repository, :in_group) } + let_it_be(:role) { create(:member_role, :guest, namespace: project.group, custom_permission: true) } + let_it_be(:membership) { create(:project_member, :guest, member_role: role, user: user, project: project) } + + before do + stub_licensed_features(custom_roles: true) + sign_in(user) + end + + describe MyController do + describe '#show' do + it 'allows access' do + get my_controller_path(project) + + expect(response).to have_gitlab_http_status(:ok) + expect(response).to render_template(:show) + end + end + end +``` + +Below is an example of the typical setup that is required to test out a GraphQL +mutation. + +```ruby + let_it_be(:user) { create(:user) } + let_it_be(:project) { create(:project, :repository, :in_group) } + let_it_be(:role) { create(:member_role, :guest, namespace: project.group, custom_permission: true) } + let_it_be(:membership) { create(:project_member, :guest, member_role: role, user: user, project: project) } + + before do + stub_licensed_features(custom_roles: true) + end + + describe MyMutation do + include GraphqlHelpers + + describe '#show' do + it 'allows access' do + post_graphql_mutation(graphql_mutation(:my_mutation, { + example: "Example" + }), current_user: user) + + expect(response).to have_gitlab_http_status(:success) + mutation_response = graphql_mutation_response(:my_mutation) + expect(mutation_response).to be_present + expect(mutation_response["errors"]).to be_empty + end + end + end +``` -You should make sure a new custom roles ability is under a feature flag. +[`GITLAB_DEBUG_POLICIES=true`](#finding-existing-abilities-checks) can be used +to troubleshoot runtime policy decisions. ## Custom abilities definition @@ -213,6 +306,7 @@ To add a new custom ability: - Use the `ee/bin/custom-ability` CLI to create the YAML definition automatically. - Perform manual steps to create a new file in `ee/config/custom_abilities/` with the filename matching the name of the ability name. 1. Add contents to the file that conform to the [schema](#schema) defined in `ee/config/custom_abilities/types/type_schema.json`. +1. Add [tests](#testing) for the new ability in `ee/spec/requests/custom_roles/` with a new directory named after the ability name. ### Schema @@ -222,7 +316,7 @@ To add a new custom ability: | `description` | yes | Human-readable description of the custom ability. | | `feature_category` | yes | Name of the feature category. For example, `vulnerability_management`. | | `introduced_by_issue` | yes | Issue URL that proposed the addition of this custom ability. | -| `introduced_by_mr` | yes | MR URL that added this custom ability. | +| `introduced_by_mr` | no | MR URL that added this custom ability. | | `milestone` | yes | Milestone in which this custom ability was added. | | `group_ability` | yes | Indicate whether this ability is checked on group level. | | `project_ability` | yes | Indicate whether this ability is checked on project level. | diff --git a/doc/development/permissions/predefined_roles.md b/doc/development/permissions/predefined_roles.md index 7cb00977e1f..577cf6192c4 100644 --- a/doc/development/permissions/predefined_roles.md +++ b/doc/development/permissions/predefined_roles.md @@ -83,10 +83,10 @@ module): - Maintainer (`40`) - Owner (`50`) -If a user is the member of both a project and the project parent groups, the +If a user is a member of both a project and the project parent groups, the highest permission is the applied access level for the project. -If a user is the member of a project, but not the parent groups, they +If a user is a member of a project, but not the parent groups, they can still view the groups and their entities (like epics). Project membership (where the group membership is already taken into account) diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index c4dfda9466a..91f4ae702ac 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -13,7 +13,7 @@ which itself includes files under for easier maintenance. We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/development/principles/#dogfooding) -GitLab [CI/CD features and best-practices](../../ci/yaml/index.md) +GitLab [CI/CD features and best-practices](../../ci/index.md) as much as possible. ## Predictive test jobs before a merge request is approved @@ -268,6 +268,9 @@ the specific list of rules. If you want to force `e2e:package-and-test` to run regardless of your changes, you can add the `pipeline:run-all-e2e` label to the merge request. +The [`e2e:test-on-gdk`](../testing_guide/end_to_end/index.md#using-the-test-on-gdk-job) child pipeline runs `:reliable` +E2E specs automatically for all `code patterns changes`. See `.qa:rules:e2e-blocking` [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for specific set of rules. + Consult the [End-to-end Testing](../testing_guide/end_to_end/index.md) dedicated page for more information. ### Review app jobs @@ -297,6 +300,23 @@ set and get the `ee/` folder removed before the tests start running. The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. +#### As-if-FOSS cross project downstream pipeline + +As an alternative to the `* as-if-foss` jobs, we can also run a cross project +FOSS pipeline exactly in the `gitlab-org/gitlab-foss` project. We trigger it +in the following cases: + +- when the `pipeline:run-as-if-foss-cross-project` label is set on the merge request + +This is still working-in-progress to replace the `* as-if-foss` jobs. The +goal is to simplify pipeline rules and make it more clear about the intention. + +##### Tokens set in the project variables + +- `AS_IF_FOSS_TOKEN`: This is a [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss) + project token with `developer` role and `write_repository` permission, + to push generated `as-if-foss/*` branch. + ### As-if-JH cross project downstream pipeline #### What it is @@ -396,7 +416,8 @@ flowchart TD - `ADD_JH_FILES_TOKEN`: This is a [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) project token with `read_api` permission, to be able to download JiHu files. - `AS_IF_JH_TOKEN`: This is a [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) - project token with `write_repository` permission, to push generated `as-if-jh/*` branch. + project token with `developer` role and `write_repository` permission, + to push generated `as-if-jh/*` branch. ##### How we generate the as-if-JH branch @@ -610,30 +631,30 @@ Exceptions to this general guideline should be motivated and documented. ### Ruby versions testing -We're running Ruby 3.0 on GitLab.com, as well as for the default branch. -To prepare for the next Ruby version, we run merge requests in Ruby 3.1. +We're running Ruby 3.1 on GitLab.com, as well as for the default branch. +To prepare for the next Ruby version, we will run merge requests in Ruby 3.2, +starting on February 2024. Please see the roadmap at +[Ruby 3.2 epic](https://gitlab.com/groups/gitlab-org/-/epics/9684#plan) +for more details. -This takes effects at the time when -[Run merge requests in Ruby 3.1 by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134290) -is merged. See -[Ruby 3.1 epic](https://gitlab.com/groups/gitlab-org/-/epics/10034) -for the roadmap to fully make Ruby 3.1 the default. +To make sure all supported Ruby versions are working, we also run our test +suite on dedicated 2-hourly scheduled pipelines for each supported versions. -To make sure both Ruby versions are working, we also run our test suite -against both Ruby 3.0 and Ruby 3.1 on dedicated 2-hourly scheduled pipelines. +For merge requests, you can add the following labels to run the respective +Ruby version only: -For merge requests, you can add the `pipeline:run-in-ruby3_0` label to switch -the Ruby version to 3.0. When you do this, the test suite will no longer run -in Ruby 3.1 (default for merge requests). +- `pipeline:run-in-ruby3_0` +- `pipeline:run-in-ruby3_1` +- `pipeline:run-in-ruby3_2` -When the pipeline is running in a Ruby version not considered default, an -additional job `verify-default-ruby` will also run and always fail to remind -us to remove the label and run in default Ruby before merging the merge -request. At the moment both Ruby 3.0 and Ruby 3.1 are considered default. +Note that when you do this, the test suite will no longer run in the default +Ruby version for merge requests. In this case, an additional job +`verify-default-ruby` will also run and always fail to remind us to remove +the label and run in default Ruby before merging the merge request. This should let us: -- Test changes for Ruby 3.1 +- Test changes for any supported Ruby versions - Make sure it will not break anything when it's merged into the default branch ### PostgreSQL versions testing @@ -649,24 +670,27 @@ We also run our test suite against PostgreSQL 13 upon specific database library | Where? | PostgreSQL version | Ruby version | |--------------------------------------------------------------------------------------------------|-------------------------------------------------|-----------------------| -| Merge requests | 14 (default version), 13 for DB library changes | 3.1 | -| `master` branch commits | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `ruby3_1` branch (every odd-numbered hour), see below. | 14 (default version), 13 for DB library changes | 3.1 | -| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.0 (default version) | - -There are 2 pipeline schedules used for testing Ruby 3.1. One is triggering a -pipeline in `ruby3_1-sync` branch, which updates the `ruby3_1` branch with latest -`master`, and no pipelines will be triggered by this push. The other schedule -is triggering a pipeline in `ruby3_1` 5 minutes after it, which is considered -the maintenance schedule to run test suites and update cache. - -The `ruby3_1` branch must not have any changes. The branch is only there to set -`RUBY_VERSION` to `3.1` in the maintenance pipeline schedule. - -The `gitlab` job in the `ruby3_1-sync` branch uses a `gitlab-org/gitlab` project -token with `write_repository` scope and `Maintainer` role with no expiration. -The token is stored in the `RUBY3_1_SYNC_TOKEN` variable in `gitlab-org/gitlab`. +| Merge requests | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `master` branch commits | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour at XX:05) | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `ruby3_0` branch (every odd-numbered hour at XX:40) | 14 (default version), 13 for DB library changes | 3.0 | +| `maintenance` scheduled pipelines for the `ruby3_2` branch (every odd-numbered hour at XX:10) | 14 (default version), 13 for DB library changes | 3.2 | +| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.1 (default version) | + +For each current Ruby versions we're testing against with, we run +maintenance scheduled pipelines every 2 hours on their respective `ruby\d_\d` +branches. All these branches must not have any changes. These branches are +only there to run pipelines with their respective Ruby versions in the +scheduled maintenance pipelines. + +Additionally, we have scheduled pipelines running on `ruby-sync` branch also +every 2 hours, updating all the `ruby\d_\d` branches to be up-to-date with +the default branch `master`. No pipelines will be triggered by this push. + +The `gitlab` job in the `ruby-sync` branch uses a `gitlab-org/gitlab` project +token named `RUBY_SYNC` with `write_repository` scope and `Maintainer` role, +expiring on 2024-12-01. The token is stored in the `RUBY_SYNC_TOKEN` variable +in the pipeline schedule for `ruby-sync` branch. ### Redis versions testing @@ -692,9 +716,9 @@ We also run tests with a single database in nightly scheduled pipelines, and in Single database tests run in two modes: 1. **Single database with one connection**. Where GitLab connects to all the tables using one connection pool. -This runs through all the jobs that end with `-single-db` + This runs through all the jobs that end with `-single-db` 1. **Single database with two connections**. Where GitLab connects to `gitlab_main`, `gitlab_ci` database tables -using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. + using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. @@ -747,28 +771,29 @@ graph LR ### Backend pipeline -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063). +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/1118782302). ```mermaid graph RL; classDef criticalPath fill:#f66; - 1-3["compile-test-assets (5.5 minutes)"]; - class 1-3 criticalPath; + 1-1["clone-gitlab-repo (1 minute)"]; + 1-3["compile-test-assets (3 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" - 1-6["setup-test-env (3.6 minutes)"]; + 1-6["setup-test-env (4 minutes)"]; + class 1-6 criticalPath; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" - 1-14["retrieve-tests-metadata"]; + 1-14["retrieve-tests-metadata (50 seconds)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" - 1-15["detect-tests"]; + 1-15["detect-tests (1 minute)"]; click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - 2_5-1["rspec & db jobs (24 minutes)"]; + 2_5-1["rspec & db jobs (30~50 minutes)"]; class 2_5-1 criticalPath; click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" - 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; + 2_5-1 --> 1-1 & 1-3 & 1-6 & 1-14 & 1-15; - ac-1["rspec:artifact-collector (2 minutes)<br/>(workaround for 'needs' limitation)"]; + ac-1["rspec:artifact-collector (30 seconds)<br/>(workaround for 'needs' limitation)"]; class ac-1 criticalPath; ac-1 --> 2_5-1; @@ -781,7 +806,6 @@ graph RL; class 4_3-1 criticalPath; click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" 4_3-1 --> 3_2-1; - ``` ### Review app pipeline diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 1fdb2014c1f..16d0bfdfa30 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -48,6 +48,25 @@ from using `$FORCE_GITLAB_CI`. - [JiHu validation pipeline](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/jihu-validation-pipelines.html) - [Gitaly downstream GitLab pipeline](https://gitlab.com/gitlab-org/gitaly/-/issues/4615) +See the next section for how we can enable pipelines without using +`$FORCE_GITLAB_CI`. + +#### Alternative to `$FORCE_GITLAB_CI` + +Essentially, we use different variables to enable different pipelines. +An example doing this is `$START_AS_IF_FOSS`. When we want to trigger a +cross project FOSS pipeline, we set `$START_AS_IF_FOSS`, along with a set of +other variables like `$ENABLE_RSPEC_UNIT`, `$ENABLE_RSPEC_SYSTEM`, and so on +so forth to enable each jobs we want to run in the as-if-foss cross project +downstream pipeline. + +The advantage of this over `$FORCE_GITLAB_CI` is that we have full control +over how we want to run the pipeline because `$START_AS_IF_FOSS` is only used +for this purpose, and changing how the pipeline behaves under this variable +will not affect other types of pipelines, while using `$FORCE_GITLAB_CI` we +do not know what exactly the pipeline is because it's used for multiple +purposes. + ## Default image The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). @@ -149,6 +168,7 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k |------------------|-------------| | `.default-retry` | Allows a job to [retry](../../ci/yaml/index.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (for example, tests). | +| `.repo-from-artifacts` | Allows a job to fetch the repository from artifacts in `clone-gitlab-repo` instead of cloning. This should reduce GitLab.com Gitaly load and also slightly improve the speed because downloading from artifacts is faster than cloning. Note that this should be avoided to be used with jobs having `needs: []` because otherwise it'll start later and we normally want all jobs to start as soon as possible. Use this only on jobs which has other dependencies so that we don't wait longer than just cloning. Note that this behavior can be controlled via `CI_FETCH_REPO_GIT_STRATEGY`. See [Fetch repository via artifacts instead of cloning/fetching from Gitaly](performance.md#fetch-repository-via-artifacts-instead-of-cloningfetching-from-gitaly) for more details. | | `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | | `.ruby-cache` | Allows a job to use a default `cache` definition suitable for Ruby tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | @@ -199,6 +219,7 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | | `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | | `if-merge-request-labels-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | +| `if-merge-request-labels-run-cs-evaluation` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-CS-evaluation". | | | `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | | `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | @@ -414,6 +435,8 @@ For this scenario, you have to: - `scripts/merge-simplecov` - `spec/simplecov_env_core.rb` - `spec/simplecov_env.rb` + - `prepare-as-if-foss-env` for: + - `scripts/setup/generate-as-if-foss-env.rb` Additionally, `scripts/utils.sh` is always downloaded from the API when this pattern is used (this file contains the code for `.fast-no-clone-job`). @@ -425,7 +448,7 @@ projects, only one of the following tags should be added to a job: - `gitlab-org`: Jobs randomly use privileged and unprivileged runners. - `gitlab-org-docker`: Jobs must use a privileged runner. If you need [Docker-in-Docker support](../../ci/docker/using_docker_build.md#use-docker-in-docker), -use `gitlab-org-docker` instead of `gitlab-org`. + use `gitlab-org-docker` instead of `gitlab-org`. The `gitlab-org-docker` tag is added by the `.use-docker-in-docker` job definition above. diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 0cd4f4270fe..d9019e6053b 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -29,6 +29,45 @@ This works well for the following reasons: - We use [shallow clone](../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git history for every job. +### Fetch repository via artifacts instead of cloning/fetching from Gitaly + +Lately we see errors from Gitaly look like this: (see [the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/435456)) + +```plaintext +fatal: remote error: GitLab is currently unable to handle this request due to load. +``` + +While GitLab.com uses [pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache), +sometimes the load is still too heavy for Gitaly to handle, and +[thundering herds](https://gitlab.com/gitlab-org/gitlab/-/issues/423830) can +also be a concern that we have a lot of jobs cloning the repository around +the same time. + +To mitigate and reduce loads for Gitaly, we changed some jobs to fetch the +repository from artifacts in a job instead of all cloning from Gitaly at once. + +For now this applies to most of the RSpec jobs, which has the most concurrent +jobs in most pipelines. This also slightly improved the speed because fetching +from the artifacts is also slightly faster than cloning, at the cost of saving +more artifacts for each pipeline. + +Based on the numbers on 2023-12-20 at [Fetch repo from artifacts for RSpec jobs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140330), +the extra storage cost was about 280M for each pipeline, and we save 15 seconds +for each RSpec jobs. + +We do not apply this to jobs having no other job dependencies because we don't +want to delay any jobs from starting. + +This behavior can be controlled by variable `CI_FETCH_REPO_GIT_STRATEGY`: + +- Set to `none` means jobs using `.repo-from-artifacts` fetch repository from + artifacts in job `clone-gitlab-repo` rather than cloning. +- Set to `clone` means jobs using `.repo-from-artifacts` clone repository + as usual. Job `clone-gitlab-repo` does not run in this case. + +To disable it, set `CI_FETCH_REPO_GIT_STRATEGY` to `clone`. To enable it, +set `CI_FETCH_REPO_GIT_STRATEGY` to `none`. + ## Caching strategy 1. All jobs must only pull caches by default. diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 6a4a85f14ff..cef55c800e1 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -72,10 +72,10 @@ This section describes how to add new metrics for self-monitoring for [Prometheus metric names](https://prometheus.io/docs/practices/naming/#metric-names). 1. Update the list of [GitLab Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md). 1. Carefully choose what labels you want to add to your metric. Values with high cardinality, -like `project_path`, or `project_id` are strongly discouraged because they can affect our services -availability due to the fact that each set of labels is exposed as a new entry in the `/metrics` endpoint. -For example, a histogram with 10 buckets and a label with 100 values would generate 1000 -entries in the export endpoint. + like `project_path`, or `project_id` are strongly discouraged because they can affect our services + availability due to the fact that each set of labels is exposed as a new entry in the `/metrics` endpoint. + For example, a histogram with 10 buckets and a label with 100 values would generate 1000 + entries in the export endpoint. 1. Trigger the relevant page or code that records the new metric. 1. Check that the new metric appears at `/-/metrics`. diff --git a/doc/development/push_rules/index.md b/doc/development/push_rules/index.md index 343b199e613..96d16f5eb35 100644 --- a/doc/development/push_rules/index.md +++ b/doc/development/push_rules/index.md @@ -42,9 +42,6 @@ change the push behavior. - `EE::Gitlab::Checks::PushRules::FileSizeCheck`: Executes push rule checks related to file size rules. - Defined in `ee/lib/ee/gitlab/checks/push_rules/file_size_check.rb`. -- `EE::Gitlab::Checks::PushRules::SecretsCheck`: Executes push rule checks - related to secrets rules. - - Defined in `ee/lib/ee/gitlab/checks/push_rules/secrets_check.rb`. - `EE::Gitlab::Checks::PushRules::TagCheck`: Executes push rule checks related to tag rules. - Defined in `ee/lib/ee/gitlab/checks/push_rules/tag_check.rb`. @@ -83,11 +80,9 @@ graph TD EE::Gitlab::Checks::PushRuleCheck -->|Only if pushing to a tag| EE::Gitlab::Checks::PushRules::TagCheck EE::Gitlab::Checks::PushRuleCheck -->|Only if pushing to a branch| EE::Gitlab::Checks::PushRules::BranchCheck EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::FileSizeCheck - EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::SecretsCheck - EE::Gitlab::Checks::PushRuleCheck --> EE::Gitlab::Checks::PushRules::SecretsCheck ``` NOTE: The `PushRuleCheck` only triggers checks in parallel if the `parallel_push_checks` feature flag is enabled. Otherwise tag or branch check -runs first, then file size, then secrets. +runs first, then file size. diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 00110d21dc0..dfe1e2d1b05 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -255,14 +255,14 @@ self.reactive_cache_hard_limit = 5.megabytes #### `self.reactive_cache_work_type` - This is the type of work performed by the `calculate_reactive_cache` method. Based on this attribute, -it's able to pick the right worker to process the caching job. Make sure to -set it as `:external_dependency` if the work performs any external request -(for example, Kubernetes, Sentry); otherwise set it to `:no_dependency`. + it's able to pick the right worker to process the caching job. Make sure to + set it as `:external_dependency` if the work performs any external request + (for example, Kubernetes, Sentry); otherwise set it to `:no_dependency`. #### `self.reactive_cache_worker_finder` - This is the method used by the background worker to find or generate the object on -which `calculate_reactive_cache` can be called. + which `calculate_reactive_cache` can be called. - By default it uses the model primary key to find the object: ```ruby diff --git a/doc/development/remote_development/index.md b/doc/development/remote_development/index.md new file mode 100644 index 00000000000..3d85c44961c --- /dev/null +++ b/doc/development/remote_development/index.md @@ -0,0 +1,16 @@ +--- +stage: Create +group: Web IDE +info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review. +--- + +# Remote Development developer guidelines + +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105783) in GitLab 16.0. + +## Workspaces feature developer documentation + +Currently, the majority of the developer documentation for the [Remote Development Workspaces feature](../../user/workspace/index.md) +is located in the separate [`gitlab-remote-development-docs` project](https://gitlab.com/gitlab-org/remote-development/gitlab-remote-development-docs/-/blob/main/README.md). + +Parts of that documentation will eventually be migrated here. diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index 110bc6076b0..d1dd6d793d4 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -50,31 +50,31 @@ To help you estimate the scope of future upgrades, see the efforts required for Before any upgrade, consider all audiences and targets, ordered by how immediately they are affected by Ruby upgrades: 1. **Developers.** We have many contributors to GitLab and related projects both inside and outside the company. Changing files such as `.ruby-version` affects everyone using tooling that interprets these files. -The developers are affected as soon as they pull from the repository containing the merged changes. + The developers are affected as soon as they pull from the repository containing the merged changes. 1. **GitLab CI/CD.** We heavily lean on CI/CD for code integration and testing. CI/CD jobs do not interpret files such as `.ruby-version`. -Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. -The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. -When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). + Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. + The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. + When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). 1. **GitLab SaaS**. GitLab.com is deployed from customized Helm charts that use Docker images from [Cloud Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG). -Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. -GitLab SaaS is affected with the next deployment. + Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. + GitLab SaaS is affected with the next deployment. 1. **Self-managed GitLab.** Customers installing GitLab via [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab) use none of the above. -Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. -Self-managed customers are affected as soon as they upgrade to the release containing this change. + Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. + Self-managed customers are affected as soon as they upgrade to the release containing this change. ## Ruby upgrade approach Timing all steps in a Ruby upgrade correctly is critical. As a general guideline, consider the following: - For smaller upgrades where production behavior is unlikely to change, aim to keep the version gap between -repositories and production minimal. Coordinate with stakeholders to merge all changes closely together -(within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and -environments first, production second. + repositories and production minimal. Coordinate with stakeholders to merge all changes closely together + (within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and + environments first, production second. - For larger changes, the risk of going to production with a new Ruby is significant. In this case, try to get into a -position where all known incompatibilities with the new Ruby version are already fixed, then work -with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario -the likely order is to update production first, developer tooling and environments second. This makes rollbacks -easier in case of critical regressions in production. + position where all known incompatibilities with the new Ruby version are already fixed, then work + with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario + the likely order is to update production first, developer tooling and environments second. This makes rollbacks + easier in case of critical regressions in production. Either way, we found that from past experience the following approach works well, with some steps likely only necessary for minor and major upgrades. Note that some of these steps can happen in parallel or may have their @@ -110,17 +110,17 @@ for a smoother transition by supporting both old and new Ruby versions for a per There are two places that require changes: 1. **[GitLab Build Images](https://gitlab.com/gitlab-org/gitlab-build-images).** These are Docker images -we use for runners and other Docker-based pre-production environments. The kind of change necessary -depends on the scope. + we use for runners and other Docker-based pre-production environments. The kind of change necessary + depends on the scope. - For [patch level updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/418), it should suffice to increment the patch level of `RUBY_VERSION`. -All projects building against the same minor release automatically download the new patch release. + All projects building against the same minor release automatically download the new patch release. - For [major and minor updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/320), create a new set of Docker images that can be used side-by-side with existing images during the upgrade process. **Important:** Make sure to copy over all Ruby patch files -in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. + in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. 1. **[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit).** -Update GDK to add the new Ruby as an additional option for -developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` -users will benefit from this. Other users will have to install it manually -([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) + Update GDK to add the new Ruby as an additional option for + developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` + users will benefit from this. Other users will have to install it manually + ([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) For larger version upgrades, consider working with [Quality Engineering](https://about.gitlab.com/handbook/engineering/quality/) to identify and set up a test plan. @@ -265,16 +265,16 @@ For GitLab SaaS, GitLab team members can inspect these log events in Kibana During the upgrade process, consider the following recommendations: - **Front-load as many changes as possible.** Especially for minor and major releases, it is likely that application -code will break or change. Any changes that are backward compatible should be merged into the main branch and -released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and -get feedback from production environments early. + code will break or change. Any changes that are backward compatible should be merged into the main branch and + released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and + get feedback from production environments early. - **Create an experimental branch for larger updates.** We generally try to avoid long-running topic branches, -but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular -feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems -we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. -These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out -and merged back independently. + but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular + feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems + we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. + These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out + and merged back independently. - **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. -As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week -before release day. This gives us extra time to act if something breaks. If in doubt, it is better to -postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/development/principles/#prioritizing-technical-decisions). + As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week + before release day. This gives us extra time to act if something breaks. If in doubt, it is better to + postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/development/principles/#prioritizing-technical-decisions). diff --git a/doc/development/runner_fleet_dashboard.md b/doc/development/runner_fleet_dashboard.md index 70499e5a087..77c89f2adb5 100644 --- a/doc/development/runner_fleet_dashboard.md +++ b/doc/development/runner_fleet_dashboard.md @@ -1,197 +1,11 @@ --- -stage: Verify -group: Runner -info: >- - To determine the technical writer assigned to the Stage/Group associated with - this page, see - https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../ci/runners/runner_fleet_dashboard.md' +remove_date: '2024-03-01' --- -# Runner Fleet Dashboard **(ULTIMATE EXPERIMENT)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/424495) in GitLab 16.6 behind several [feature flags](#enable-feature-flags). +This document was moved to [another location](../ci/runners/runner_fleet_dashboard.md). -This feature is an [Experiment](../policy/experiment-beta-support.md). -To join the list of users testing this feature, contact us in -[epic 11180](https://gitlab.com/groups/gitlab-org/-/epics/11180). - -GitLab administrators can use the Runner Fleet Dashboard to assess the health of your instance runners. -The Runner Fleet Dashboard shows: - -- Recent CI errors related caused by runner infrastructure. -- Number of concurrent jobs executed on most busy runners. -- Histogram of job queue times (available only with ClickHouse). - -There is a proposal to introduce [more features](#whats-next) to the Runner Fleet Dashboard. - - - -## View the Runner Fleet Dashboard - -Prerequisites: - -- You must be an administrator. - -To view the runner fleet dashboard: - -1. On the left sidebar, at the bottom, select **Admin Area**. -1. Select **Runners**. -1. Click **Fleet dashboard**. - -Most of the dashboard works without any additional actions, with the -exception of **Wait time to pick a job** chart and [proposed features](#whats-next). -These features require setting up an additional infrastructure, described in this page. - -To test the Runner Fleet Dashboard and gather feedback, we have launched an early adopters program -for some customers to try this feature. - -## Requirements - -To test the Runner Fleet Dashboard as part of the early adopters program, you must: - -- Run GitLab 16.7 or above. -- Have an [Ultimate license](https://about.gitlab.com/pricing/). -- Be able to run ClickHouse database. We recommend using [ClickHouse Cloud](https://clickhouse.cloud/). - -## Setup - -To setup ClickHouse as the GitLab data storage: - -1. [Run ClickHouse Cluster and configure database](#run-and-configure-clickhouse). -1. [Configure GitLab connection to Clickhouse](#configure-the-gitlab-connection-to-clickhouse). -1. [Run ClickHouse migrations](#run-clickhouse-migrations). -1. [Enable the feature flags](#enable-feature-flags). - -### Run and configure ClickHouse - -The most straightforward way to run ClickHouse is with [ClickHouse Cloud](https://clickhouse.cloud/). -You can also [run ClickHouse on your own server](https://clickhouse.com/docs/en/install). Refer to the ClickHouse -documentation regarding [recommendations for self-managed instances](https://clickhouse.com/docs/en/install#recommendations-for-self-managed-clickhouse). - -When you run ClickHouse on a hosted server, various data points might impact the resource consumption, like the number -of builds that run on your instance each month, the selected hardware, the data center choice to host ClickHouse, and more. -Regardless, the cost should not be significant. - -NOTE: -ClickHouse is a secondary data store for GitLab. All your data is still stored in Postgres, -and only duplicated in ClickHouse for analytics purposes. - -To create necessary user and database objects: - -1. Generate a secure password and save it. -1. Sign in to the ClickHouse SQL console. -1. Execute the following command. Replace `PASSWORD_HERE` with the generated password. - - ```sql - CREATE DATABASE gitlab_clickhouse_main_production; - CREATE USER gitlab IDENTIFIED WITH sha256_password BY 'PASSWORD_HERE'; - CREATE ROLE gitlab_app; - GRANT SELECT, INSERT, ALTER, CREATE, UPDATE, DROP, TRUNCATE, OPTIMIZE ON gitlab_clickhouse_main_production.* TO gitlab_app; - GRANT gitlab_app TO gitlab; - ``` - -### Configure the GitLab connection to ClickHouse - -::Tabs - -:::TabTitle Linux package - -To provide GitLab with ClickHouse credentials: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['clickhouse_databases']['main']['database'] = 'gitlab_clickhouse_main_production' - gitlab_rails['clickhouse_databases']['main']['url'] = 'https://example.com/path' - gitlab_rails['clickhouse_databases']['main']['username'] = 'gitlab' - gitlab_rails['clickhouse_databases']['main']['password'] = 'PASSWORD_HERE' # replace with the actual password - ``` - -1. Save the file and reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -:::TabTitle Helm chart (Kubernetes) - -1. Save the ClickHouse password as a Kubernetes Secret: - - ```shell - kubectl create secret generic gitlab-clickhouse-password --from-literal="main_password=PASSWORD_HERE" - ``` - -1. Export the Helm values: - - ```shell - helm get values gitlab > gitlab_values.yaml - ``` - -1. Edit `gitlab_values.yaml`: - - ```yaml - global: - clickhouse: - enabled: true - main: - username: default - password: - secret: gitlab-clickhouse-password - key: main_password - database: gitlab_clickhouse_main_production - url: 'http://example.com' - ``` - -1. Save the file and apply the new values: - - ```shell - helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab - ``` - -::EndTabs - -To verify that your connection is set up successfully: - -1. Log in to [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session) -1. Execute the following: - - ```ruby - ClickHouse::Client.select('SELECT 1', :main) - ``` - - If successful, the command returns `[{"1"=>1}]` - -### Run ClickHouse migrations - -To create the required database objects execute: - -```shell -sudo gitlab-rake gitlab:clickhouse:migrate -``` - -### Enable feature flags - -Features that use ClickHouse are currently under development and are disabled by feature flags. - -To enable these features, [enable](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) -the following feature flags: - -| Feature flag name | Purpose | -|------------------------------------|---------------------------------------------------------------------------| -| `ci_data_ingestion_to_click_house` | Enables synchronization of new finished CI builds to Clickhouse database. | -| `clickhouse_ci_analytics` | Enables the **Wait time to pick a job** chart. | - -## What's next - -Support for usage and cost analysis are proposed in -[epic 11183](https://gitlab.com/groups/gitlab-org/-/epics/11183). - -## Feedback - -To help us improve the Runner Fleet Dashboard, you can provide feedback in -[issue 421737](https://gitlab.com/gitlab-org/gitlab/-/issues/421737). -In particular: - -- How easy or difficult it was to setup GitLab to make the dashboard work. -- How useful you found the dashboard. -- What other information you would like to see on that dashboard. -- Any other related thoughts and ideas. +<!-- This redirect file can be deleted after <2024-03-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/search/advanced_search_migration_styleguide.md b/doc/development/search/advanced_search_migration_styleguide.md index 87083d1a36f..b79ab6561d5 100644 --- a/doc/development/search/advanced_search_migration_styleguide.md +++ b/doc/development/search/advanced_search_migration_styleguide.md @@ -225,6 +225,53 @@ class MigrationName < Elastic::Migration end ``` +#### `Search::Elastic::MigrationDeleteBasedOnSchemaVersion` + +Deletes all documents in the index that stores the specified document type and has `schema_version` less than the given value. + +Requires the `DOCUMENT_TYPE` constant and `schema_version` method. +The index mapping must have a `schema_version` integer field in a `YYMM` format. + +```ruby +class MigrationName < Elastic::Migration + include ::Search::Elastic::MigrationDeleteBasedOnSchemaVersion + + DOCUMENT_TYPE = Issue + + batch_size 10_000 + batched! + throttle_delay 1.minute + retry_on_failure + + def schema_version + 23_12 + end +end +``` + +#### `Search::Elastic::MigrationDatabaseBackfillHelper` + +Reindexes all documents in the database to the elastic search index respecting the `limited_indexing` setting. + +Requires the `DOCUMENT_TYPE` constant and `respect_limited_indexing?` method. + +```ruby +class MigrationName < Elastic::Migration + include ::Search::Elastic::MigrationDatabaseBackfillHelper + + batch_size 10_000 + batched! + throttle_delay 1.minute + retry_on_failure + + DOCUMENT_TYPE = Issue + + def respect_limited_indexing? + true + end +end +``` + #### `Elastic::MigrationHelper` Contains methods you can use when a migration doesn't fit the previous examples. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index eb59d8fcaf5..843f41300a2 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -128,7 +128,7 @@ To use Docker with `replace` in the `go.mod` file: 1. Copy the contents of `command` into the directory of the analyzer. `cp -r /path/to/command path/to/analyzer/command`. 1. Add a copy statement in the analyzer's `Dockerfile`: `COPY command /command`. 1. Update the `replace` statement to make sure it matches the destination of the `COPY` statement in the step above: -`replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` + `replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` ## Analyzer scripts @@ -189,11 +189,11 @@ are integrated with the existing application, iteration should not be blocked by 1. Ensure that the release source (typically the `master` or `main` branch) has a passing pipeline. 1. Create a new release for the analyzer project by selecting the **Deployments** menu on the left-hand side of the project window, then selecting the **Releases** sub-menu. 1. Select **New release** to open the **New Release** page. - 1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here). - 1. In the **Release title** text box enter the same version used above, for example `v2.4.2`. - 1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`. - 1. Leave all other settings as the default values. - 1. Select **Create release**. + 1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here). + 1. In the **Release title** text box enter the same version used above, for example `v2.4.2`. + 1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`. + 1. Leave all other settings as the default values. + 1. Select **Create release**. After following the above process and creating a new release, a new Git tag is created with the `Tag name` provided above. This triggers a new pipeline with the given tag version and a new analyzer Docker image is built. @@ -229,7 +229,7 @@ After the above steps have been completed, the automatic release process execute 1. After a new version of the analyzer Docker image has been tagged and deployed, test it with the corresponding test project. 1. Announce the release on the relevant group Slack channel. Example message: - > FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE` + > FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE` **Never delete a Git tag that has been pushed** as there is a good chance that the tag will be used and/or cached by the Go package registry. diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index a575d1ff890..d8fad6deb9c 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -182,7 +182,7 @@ For other regular expressions, here are a few guidelines: - If there's a clean non-regex solution, such as `String#start_with?`, consider using it - Ruby supports some advanced regex features like [atomic groups](https://www.regular-expressions.info/atomic.html) -and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eliminate backtracking + and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that eliminate backtracking - Avoid nested quantifiers if possible (for example `(a+)+`) - Try to be as precise as possible in your regex and avoid the `.` if there's an alternative - For example, Use `_[^_]+_` instead of `_.*_` to match `_text here_` diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index f625dc466b9..f940cf289d9 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -48,6 +48,7 @@ Matches use: | The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly scheduled pipeline"](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | | The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | | Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | +| Local development environment | `only: :local` | Any environment where `Runtime::Env.running_in_ci?` is false | | Any run where condition evaluates to a truthy value | `only: { condition: -> { ENV['TEST_ENV'] == 'true' } }` | Any run where `TEST_ENV` is set to true ```ruby diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index e11119d2c0b..189b21ea607 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -24,14 +24,14 @@ Be sure to include the `feature_flag` tag so that the test can be skipped on the - Format: `feature_flag: { name: 'feature_flag_name' }` - Used only for informational purposes at this time. It should be included to help quickly determine what -feature flag is under test. + feature flag is under test. `scope` - Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` - When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. - When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and pre-production**. -This is due to the fact that administrator access is not available there. + This is due to the fact that administrator access is not available there. **WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), or [feature group](../../feature_flags/index.md#feature-groups). diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index aaea3f8958d..eabc722a26f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -303,6 +303,75 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r - You can increase the wait time for replication by setting `GEO_MAX_FILE_REPLICATION_TIME` and `GEO_MAX_DB_REPLICATION_TIME`. The default is 120 seconds. - To save time during tests, create a Personal Access Token with API access on the Geo primary node, and pass that value in as `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN`. +## Group SAML Tests + +Tests that are tagged with `:group_saml` meta are orchestrated tests where the user accesses a group via SAML SSO. + +These tests depend on a SAML IDP Docker container ([jamedjo/test-SAML-idp](https://hub.docker.com/r/jamedjo/test-saml-idp)). The tests spin up the container themselves. + +To run these tests on your computer against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + omniauth: + enabled: true + providers: + - { name: 'group_saml' } + ``` + +1. Run a group SAML test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_DEBUG=true CHROME_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/ee/browser_ui/1_manage/group/group_saml_enforced_sso_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationgroupsaml-eefull-image-address). + +## Instance SAML Tests + +Tests that are tagged with `:instance_saml` meta are orchestrated tests where the instance level sign-in happens using SAML SSO. + +These tests require a SAML IDP Docker container ([jamedjo/test-SAML-idp](https://hub.docker.com/r/jamedjo/test-saml-idp)) to be configured and running. + +To run these tests on your computer against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ["saml"] + block_auto_created_users: false + auto_link_saml_user: true + providers: + - { name: 'saml', + args: { + assertion_consumer_service_url: 'http://gdk.test:3000/users/auth/saml/callback', + idp_cert_fingerprint: '11:9b:9e:02:79:59:cd:b7:c6:62:cf:d0:75:d9:e2:ef:38:4e:44:5f', + idp_sso_target_url: 'https://gdk.test:8443/simplesaml/saml2/idp/SSOService.php', + issuer: 'http://gdk.test:3000', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + } } + ``` + +1. Start the SAML IDP Docker container: + + ```shell + docker run --name=group_saml_qa_idp -p 8080:8080 -p 8443:8443 \ + -e SIMPLESAMLPHP_SP_ENTITY_ID=http://localhost:3000 \ + -e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=http://localhost:3000/users/auth/saml/callback \ + -d jamedjo/test-saml-idp + ``` + +1. Run the test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_DEBUG=true CHROME_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/browser_ui/1_manage/login/login_via_instance_wide_saml_sso_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationinstancesaml-ceeefull-image-address). + ## LDAP Tests Tests that are tagged with `:ldap_tls` and `:ldap_no_tls` meta are orchestrated tests where the sign-in happens via LDAP. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 1895b9bdb39..4d6fac57c06 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -119,7 +119,7 @@ Adding a delay in API or controller could help reproducing the issue. - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101728/diffs): A CSS selector only appears after a GraphQL requests has finished, and the UI has updated. - [Example 3](https://gitlab.com/gitlab-org/gitlab/-/issues/408215): A false-positive test, Capybara immediately returns true after -page visit and page is not fully loaded, or if the element is not detectable by webdriver (such as being rendered outside the viewport or behind other elements). + page visit and page is not fully loaded, or if the element is not detectable by webdriver (such as being rendered outside the viewport or behind other elements). ### Datetime-sensitive diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 81399c7ce05..99852645914 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -349,7 +349,7 @@ possible). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver is [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | +| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver is [Selenium](https://github.com/teamcapybara/capybara#selenium), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | ### Frontend feature tests diff --git a/doc/development/ux/index.md b/doc/development/ux/index.md index ab6cd4c64f5..de4491f997c 100644 --- a/doc/development/ux/index.md +++ b/doc/development/ux/index.md @@ -20,7 +20,7 @@ To contribute to [Pajamas design system](https://design.gitlab.com/) and the [UI ## Contributing to other issues -1. Review the list of available issues that are currently [accepting UX contribution](https://gitlab.com/groups/gitlab-org/-/issues/?sort=weight&state=opened&label_name%5B%5D=UX&label_name%5B%5D=workflow%3A%3Aready%20for%20design&label_name%5B%5D=Accepting%20UX%20contributions&first_page_size=20). +1. Review the list of available UX issues that are currently [seeking community contribution](https://gitlab.com/groups/gitlab-org/-/issues/?sort=weight&state=opened&label_name%5B%5D=UX&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=100). 1. Find an issue that does not have an Assignee to ensure someone else is not working on a solution. Add the `~"workflow::design"` and `~"Community contribution"` labels and mention `@gitlab-com/gitlab-ux/reviewers` to request they assign the issue to you. 1. Add your design proposal to the issue description/[design management](../../user/project/issues/design_management.md) section. Remember to keep the scope of the proposal/change small following our [MVCs guidelines](https://about.gitlab.com/handbook/values/#minimal-viable-change-mvc). 1. If you have any questions or are ready for a review of your proposal, mention `@gitlab-com/gitlab-ux/reviewers` in a comment to make your request. diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 725c8aa45d2..83cfb3fb385 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -28,7 +28,7 @@ Apart from the durations, we expose the record count within a stage. ## Feature availability - Group level (licensed): Requires Ultimate or Premium subscription. This version is the most -feature-full. + feature-full. - Project level (licensed): We are continually adding features to project level VSA to bring it in line with group level VSA. - Project level (FOSS): Keep it as is. diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md index 53c6721b01f..7386a83afc1 100644 --- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -45,7 +45,7 @@ Benefits of the aggregated VSA backend: - Possibility to introduce further aggregations for improving the first page load time. - Better performance for large groups (with many subgroups, projects, issues and, merge requests). - Ready for database decomposition. The VSA related database tables could live in a separate -database with a minimal development effort. + database with a minimal development effort. - Ready for keyset pagination which can be useful for exporting the data. - Possibility to implement more complex event definitions. - For example, the start event can be two timestamp columns where the earliest value would be @@ -107,8 +107,7 @@ the service performs operations in batches and enforces strict application limit - Load records in batches. - Insert records in batches. -- Stop processing when a limit is reached, schedule a background job to continue the processing -later. +- Stop processing when a limit is reached, schedule a background job to continue the processing later. - Continue processing data from a specific point. As of GitLab 14.7, the data loading is done manually. Once the feature is ready, the service is @@ -267,16 +266,15 @@ database tables. This change could be implemented using array columns. The feature uses private JSON APIs for delivering the data to the frontend. On the first page load , the following requests are invoked: -- Initial HTML page load which is mostly empty. Some configuration data is exposed via `data` -attributes. +- Initial HTML page load which is mostly empty. Some configuration data is exposed via `data` attributes. - `value_streams` - Load the available value streams for the given group. - `stages` - Load the stages for the currently selected value stream. - `median` - For each stage, request the median duration. - `count` - For each stage, request the number of items in the stage (this is a -[limit count](../merge_request_concepts/performance.md#badge-counters), maximum 1000 rows). + [limit count](../merge_request_concepts/performance.md#badge-counters), maximum 1000 rows). - `average_duration_chart` - Data for the duration chart. - `summary`, `time_summary` - Top-level aggregations, most of the metrics are using different APIs/ -finders and not invoking the aggregated backend. + finders and not invoking the aggregated backend. When selecting a specific stage, the `records` endpoint is invoked, which returns the related records (paginated) for the chosen stage in a specific order. diff --git a/doc/development/work_items.md b/doc/development/work_items.md index 0c3bc4611f5..7fc0c0d9d21 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -245,7 +245,8 @@ Keep the following in mind when you write your migration: necessary if the new work item type is going to use the `Hierarchy` widget. In this table, you must add what work item type can have children and of what type. Also, you should specify the hierarchy depth for work items of the same type. By default a cross-hierarchy (cross group or project) relationship is disabled when creating new restrictions but - it can be enabled by specifying a value for `cross_hierarchy_enabled`. + it can be enabled by specifying a value for `cross_hierarchy_enabled`. Due to the restrictions being cached for the work item type, it's also + required to call `clear_reactive_cache!` on the associated work item types. - Optional. Create linked item restrictions. - Similarly to the `Hierarchy` widget, the `Linked items` widget also supports rules defining which work item types can be linked to other types. A restriction can specify if the source type can be related to or blocking a target type. Current restrictions: diff --git a/doc/editor_extensions/index.md b/doc/editor_extensions/index.md index de1e3f21ee6..052327596c9 100644 --- a/doc/editor_extensions/index.md +++ b/doc/editor_extensions/index.md @@ -1,6 +1,7 @@ --- stage: Create group: Editor Extensions +description: Visual Studio Code, JetBrains, Neovim, GitLab CLI. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/editor_extensions/visual_studio_code/index.md b/doc/editor_extensions/visual_studio_code/index.md index 4e2fb31c863..f6716f96fd8 100644 --- a/doc/editor_extensions/visual_studio_code/index.md +++ b/doc/editor_extensions/visual_studio_code/index.md @@ -37,7 +37,7 @@ you can [configure](https://marketplace.visualstudio.com/items?itemName=GitLab.g - [Features to display or hide](https://gitlab.com/gitlab-org/gitlab-vscode-extension#extension-settings). - [Self-signed certificate](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#self-signed-certificates) information. - [Code Suggestions](../../user/project/repository/code_suggestions/index.md). -- [GitLab Duo Chat](../../user/gitlab_duo_chat.md#gitlab-workflow-extension-for-vs-code). +- [GitLab Duo Chat](../../user/gitlab_duo_chat.md#use-gitlab-duo-chat-in-vs-code). ## Report issues with the extension diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index ecdc4aeed06..a1dd99d811f 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -117,7 +117,7 @@ Clone with SSH when you want to authenticate only one time. 1. Authenticate with GitLab by following the instructions in the [SSH documentation](../user/ssh.md). 1. On the left sidebar, select **Search or go to** and find the project you want to clone. -1. On the right-hand side of the page, select **Clone**, then copy the URL for **Clone with SSH**. +1. On the project's overview page, in the upper-right corner, select **Code**, then copy the URL for **Clone with SSH**. 1. Open a terminal and go to the directory where you want to clone the files. Git automatically creates a folder with the repository name and downloads the files there. 1. Run this command: @@ -142,7 +142,7 @@ between your computer and GitLab. [OAuth credential helpers](../user/profile/account/two_factor_authentication.md#oauth-credential-helpers) can decrease the number of times you must manually authenticate, making HTTPS a seamless experience. 1. On the left sidebar, select **Search or go to** and find the project you want to clone. -1. On the right-hand side of the page, select **Clone**, then copy the URL for **Clone with HTTPS**. +1. On the project's overview page, in the upper-right corner, select **Code**, then copy the URL for **Clone with HTTPS**. 1. Open a terminal and go to the directory where you want to clone the files. 1. Run the following command. Git automatically creates a folder with the repository name and downloads the files there. @@ -255,6 +255,47 @@ existing branch. You can create additional named remotes and branches as necessa You can learn more on how Git manages remote repositories in the [Git Remote documentation](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). +## Add another URL to a remote + +Add another URL to a remote, so both remotes get updated on each push: + +```shell +git remote set-url --add <remote_name> <remote_url> +``` + +## Show the log of reference changes to HEAD + +```shell +git reflog +``` + +## Check the Git history of a file + +The basic command to check the Git history of a file: + +```shell +git log <file> +``` + +If you get this error message: + +```plaintext +fatal: ambiguous argument <file_name>: unknown revision or path not in the working tree. +Use '--' to separate paths from revisions, like this: +``` + +Use this to check the Git history of the file: + +```shell +git log -- <file> +``` + +## Check the content of each change to a file + +```shell +gitk <file> +``` + ## Branches A **branch** is a copy of the files in the repository at the time you create the branch. @@ -423,6 +464,20 @@ In GitLab, you typically use a [merge request](../user/project/merge_requests/in To create a merge request from a fork to an upstream repository, see the [forking workflow](../user/project/repository/forking_workflow.md). +## Reuse recorded resolutions + +To _reuse_ recorded resolutions: + +```shell +git rerere +``` + +To enable `rerere` functionality: + +```shell +git config --global rerere.enabled true +``` + ## Advanced use of Git through the command line For an introduction of more advanced Git techniques, see [Git rebase, force-push, and merge conflicts](../topics/git/git_rebase.md). @@ -439,15 +494,3 @@ changes from the original repository. It is common to call this remote repositor You can now use the `upstream` as a [`<remote>` to `pull` new updates](#download-the-latest-changes-in-the-project) from the original repository, and use the `origin` to [push local changes](#send-changes-to-gitlab) and create merge requests. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> diff --git a/doc/index.md b/doc/index.md index 362bbab3607..37200d94293 100644 --- a/doc/index.md +++ b/doc/index.md @@ -37,14 +37,14 @@ Have a look at some of our most popular topics: |:-------------------------------------------------------------------------------------------|:------------| | [Two-factor authentication](user/profile/account/two_factor_authentication.md) | Improve the security of your GitLab account. | | [GitLab groups](user/group/index.md) | Manage projects together. | -| [Keyword reference for the `.gitlab-ci.yml` file](ci/yaml/index.md) | Available configuration options for `.gitlab-ci.yml` files. | -| [Activate GitLab EE with a license](administration/license.md) | Activate GitLab Enterprise Edition functionality with a license. | -| [Back up and restore GitLab](administration/backup_restore/index.md) | Backing up and restoring GitLab self-managed instances. | +| [CI/CD YAML syntax reference](ci/yaml/index.md) | Available configuration options for `.gitlab-ci.yml` files. | +| [Activate GitLab EE with a license](administration/license.md) | Activate GitLab Enterprise Edition functionality with a license. | +| [Back up and restore GitLab](administration/backup_restore/index.md) | Backing up and restoring GitLab self-managed instances. | | [GitLab release and maintenance policy](policy/maintenance.md) | Policies for version naming and cadence, and also upgrade recommendations. | | [Elasticsearch integration](integration/advanced_search/elasticsearch.md) | Integrate Elasticsearch with GitLab to enable advanced search. | -| [Database settings for Linux package installations](https://docs.gitlab.com/omnibus/settings/database.html) | Database settings for self-managed instances installed using Linux packages. | -| [NGINX settings for Linux package installations](https://docs.gitlab.com/omnibus/settings/nginx.html) | NGINX settings for self-managed instances installed using Linux packages. | -| [SSL configuration for Linux package installations](https://docs.gitlab.com/omnibus/settings/ssl/index.html) | SSL settings for self-managed instances installed using Linux packages. | +| [Database settings for Linux package installations](https://docs.gitlab.com/omnibus/settings/database.html) | Database settings for self-managed instances installed using Linux packages. | +| [NGINX settings for Linux package installations](https://docs.gitlab.com/omnibus/settings/nginx.html) | NGINX settings for self-managed instances installed using Linux packages. | +| [SSL configuration for Linux package installations](https://docs.gitlab.com/omnibus/settings/ssl/index.html) | SSL settings for self-managed instances installed using Linux packages. | | [GitLab.com settings](user/gitlab_com/index.md) | Settings used for GitLab.com. | ## User account @@ -54,7 +54,7 @@ For more information about GitLab account management, see: | Topic | Description | |:-----------------------------------------------------------|:------------| | [User account](user/profile/index.md) | Manage your account. | -| [Authentication](topics/authentication/index.md) | Account security with two-factor authentication, set up your SSH keys, and deploy keys for secure access to your projects. | +| [Authentication](administration/auth/index.md) | Account security with two-factor authentication, set up your SSH keys, and deploy keys for secure access to your projects. | | [User settings](user/profile/index.md#access-your-user-settings) | Manage your user settings, two factor authentication, and more. | | [User permissions](user/permissions.md) | Learn what each role in a project can do. | @@ -62,10 +62,10 @@ For more information about GitLab account management, see: If you are coming to GitLab from another platform, the following information is useful: -| Topic | Description | -|:----------------------------------------------------|:------------| -| [Importing to GitLab](user/project/import/index.md) | Import your projects from GitHub, Bitbucket, GitLab.com, FogBugz, and SVN into GitLab. | -| [Migrating from SVN](user/project/import/index.md#import-from-subversion) | Convert a SVN repository to Git and GitLab. | +| Topic | Description | +|:---------------------------------------------------------------------------------------|:------------| +| [Importing to GitLab](user/project/import/index.md) | Import your projects from GitHub, Bitbucket, GitLab.com, FogBugz, and SVN into GitLab. | +| [Migrating from SVN](user/project/import/index.md#import-repositories-from-subversion) | Convert a SVN repository to Git and GitLab. | ## Build an integration with GitLab diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 6ed3368fb45..25b52c9003c 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -725,13 +725,13 @@ From the EC2 dashboard: 1. We leave our **Health Check Grace Period** as the default `300` seconds. Select **Configure scaling policies**. 1. Check **Use scaling policies to adjust the capacity of this group**. 1. For this group we scale between 2 and 4 instances where one instance is added if CPU -utilization is greater than 60% and one instance is removed if it falls -to less than 45%. + utilization is greater than 60% and one instance is removed if it falls + to less than 45%.  1. Finally, configure notifications and tags as you see fit, review your changes, and create the -auto scaling group. + auto scaling group. As the auto scaling group is created, you see your new instances spinning up in your EC2 dashboard. You also see the new instances added to your load balancer. After the instances pass the heath check, they are ready to start receiving traffic from the load balancer. diff --git a/doc/install/cloud_providers.md b/doc/install/cloud_providers.md index 318895b6d89..ecec2250e11 100644 --- a/doc/install/cloud_providers.md +++ b/doc/install/cloud_providers.md @@ -1,11 +1,11 @@ --- stage: Systems group: Distribution +description: AWS, Google Cloud Platform, Azure. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: Install GitLab on a cloud provider. --- -# Installing GitLab on a cloud provider **(FREE SELF)** +# Install GitLab on a cloud provider **(FREE SELF)** You can install GitLab on several cloud providers. diff --git a/doc/install/docker.md b/doc/install/docker.md index 252f34f7120..76e5b9bd7b3 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -531,7 +531,7 @@ To upgrade GitLab that was [installed using Docker Engine](#install-gitlab-using ``` 1. Create the container once again with the -[previously specified](#install-gitlab-using-docker-engine) options: + [previously specified](#install-gitlab-using-docker-engine) options: ```shell sudo docker run --detach \ @@ -621,7 +621,7 @@ to back up the `gitlab.rb` file. WARNING: [Backing up the GitLab secrets file](../administration/backup_restore/backup_gitlab.md#storing-configuration-files) is required -to avoid [complicated steps](../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost) when recovering +to avoid [complicated steps](../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost) when recovering GitLab from backup. The secrets file is stored at `/etc/gitlab/gitlab-secrets.json` inside the container, or `$GITLAB_HOME/config/gitlab-secrets.json` [on the container host](#set-up-the-volumes-location). diff --git a/doc/install/install_methods.md b/doc/install/install_methods.md index b40d89e957a..99be2709564 100644 --- a/doc/install/install_methods.md +++ b/doc/install/install_methods.md @@ -1,8 +1,8 @@ --- stage: Systems group: Distribution +description: Linux, Helm, Docker, Operator, source, or scripts. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: Read through the GitLab installation methods. --- # Installation methods **(FREE SELF)** diff --git a/doc/install/installation.md b/doc/install/installation.md index 370f67865ed..0ccb475eacc 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -174,7 +174,7 @@ the Git path: ### GraphicsMagick -For the [Custom Favicon](../administration/appearance.md#favicon) to work, GraphicsMagick +For the [Custom Favicon](../administration/appearance.md#customize-the-favicon) to work, GraphicsMagick must be installed. ```shell diff --git a/doc/install/requirements.md b/doc/install/requirements.md index fa6be957d99..bad3a2b09e7 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Prerequisites for installation. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -231,8 +232,8 @@ The recommended number of threads is dependent on several factors, including tot - If the operating system has a maximum 2 GB of memory, the recommended number of threads is `1`. A higher value results in excess swapping, and decrease performance. - In all other cases, the recommended number of threads is `4`. We don't recommend setting this -higher, due to how [Ruby MRI multi-threading](https://en.wikipedia.org/wiki/Global_interpreter_lock) -works. + higher, due to how [Ruby MRI multi-threading](https://en.wikipedia.org/wiki/Global_interpreter_lock) + works. ### Puma per worker maximum memory diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md index 0a456e6c73e..896ad18033b 100644 --- a/doc/integration/advanced_search/elasticsearch.md +++ b/doc/integration/advanced_search/elasticsearch.md @@ -997,7 +997,7 @@ To create both an indexing and a non-indexing Sidekiq process in one node: ``` 1. Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) -for the changes to take effect. + for the changes to take effect. 1. On all other Rails and Sidekiq nodes, ensure that `sidekiq['routing_rules']` is the same as above. 1. Run the Rake task to [migrate existing jobs](../../administration/sidekiq/sidekiq_job_migration.md): @@ -1029,7 +1029,7 @@ To handle these queue groups on two nodes: ``` 1. Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) -for the changes to take effect. + for the changes to take effect. 1. To set up the non-indexing Sidekiq process, on your non-indexing Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: @@ -1052,7 +1052,7 @@ for the changes to take effect. 1. On all other Rails and Sidekiq nodes, ensure that `sidekiq['routing_rules']` is the same as above. 1. Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) -for the changes to take effect. + for the changes to take effect. 1. Run the Rake task to [migrate existing jobs](../../administration/sidekiq/sidekiq_job_migration.md): ```shell diff --git a/doc/integration/clickhouse.md b/doc/integration/clickhouse.md new file mode 100644 index 00000000000..1382488ad57 --- /dev/null +++ b/doc/integration/clickhouse.md @@ -0,0 +1,126 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# ClickHouse integration guidelines **(EXPERIMENT)** + +This feature is an [Experiment](../policy/experiment-beta-support.md). + +Instructions about how to setup integration between GitLab and ClickHouse database. + +## Setup + +To setup ClickHouse as the GitLab data storage: + +1. [Run ClickHouse Cluster and configure database](#run-and-configure-clickhouse). +1. [Configure GitLab connection to Clickhouse](#configure-the-gitlab-connection-to-clickhouse). +1. [Run ClickHouse migrations](#run-clickhouse-migrations). + +### Run and configure ClickHouse + +The most straightforward way to run ClickHouse is with [ClickHouse Cloud](https://clickhouse.cloud/). +You can also [run ClickHouse on your own server](https://clickhouse.com/docs/en/install). Refer to the ClickHouse +documentation regarding [recommendations for self-managed instances](https://clickhouse.com/docs/en/install#recommendations-for-self-managed-clickhouse). + +When you run ClickHouse on a hosted server, various data points might impact the resource consumption, like the number +of builds that run on your instance each month, the selected hardware, the data center choice to host ClickHouse, and more. +Regardless, the cost should not be significant. + +NOTE: +ClickHouse is a secondary data store for GitLab. All your data is still stored in Postgres, +and only duplicated in ClickHouse for analytics purposes. + +To create necessary user and database objects: + +1. Generate a secure password and save it. +1. Sign in to the ClickHouse SQL console. +1. Execute the following command. Replace `PASSWORD_HERE` with the generated password. + + ```sql + CREATE DATABASE gitlab_clickhouse_main_production; + CREATE USER gitlab IDENTIFIED WITH sha256_password BY 'PASSWORD_HERE'; + CREATE ROLE gitlab_app; + GRANT SELECT, INSERT, ALTER, CREATE, UPDATE, DROP, TRUNCATE, OPTIMIZE ON gitlab_clickhouse_main_production.* TO gitlab_app; + GRANT gitlab_app TO gitlab; + ``` + +### Configure the GitLab connection to ClickHouse + +::Tabs + +:::TabTitle Linux package + +To provide GitLab with ClickHouse credentials: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['clickhouse_databases']['main']['database'] = 'gitlab_clickhouse_main_production' + gitlab_rails['clickhouse_databases']['main']['url'] = 'https://example.com/path' + gitlab_rails['clickhouse_databases']['main']['username'] = 'gitlab' + gitlab_rails['clickhouse_databases']['main']['password'] = 'PASSWORD_HERE' # replace with the actual password + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Save the ClickHouse password as a Kubernetes Secret: + + ```shell + kubectl create secret generic gitlab-clickhouse-password --from-literal="main_password=PASSWORD_HERE" + ``` + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + clickhouse: + enabled: true + main: + username: default + password: + secret: gitlab-clickhouse-password + key: main_password + database: gitlab_clickhouse_main_production + url: 'http://example.com' + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +::EndTabs + +To verify that your connection is set up successfully: + +1. Log in to [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session) +1. Execute the following: + + ```ruby + ClickHouse::Client.select('SELECT 1', :main) + ``` + + If successful, the command returns `[{"1"=>1}]` + +### Run ClickHouse migrations + +To create the required database objects execute: + +```shell +sudo gitlab-rake gitlab:clickhouse:migrate +``` diff --git a/doc/integration/diffblue_cover.md b/doc/integration/diffblue_cover.md new file mode 100644 index 00000000000..bd1d026b5ef --- /dev/null +++ b/doc/integration/diffblue_cover.md @@ -0,0 +1,89 @@ +--- +stage: Verify +group: Pipeline Execution +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" +description: >- + How to configure the Diffblue Cover GitLab integration - Cover Pipeline for + GitLab +--- + +# Diffblue Cover **(FREE ALL)** + +You can integrate the [Diffblue Cover](https://www.diffblue.com/) reinforcement learning AI tool into your CI/CD pipelines, to automatically write and maintain Java unit tests for your GitLab projects. +The Diffblue Cover Pipeline for GitLab integration allows you to automatically: + +- Write a baseline unit test suite for your projects. +- Write new unit tests for new code. +- Update existing unit tests in your code. +- Remove existing unit tests in your code when they're no longer required. + + + +## Configure the integration + +To integrate Diffblue Cover into your pipeline: + +1. Find and configure the Diffblue Cover integration. +1. Configure a pipeline for a sample project using the GitLab pipeline editor and the Diffblue Cover pipeline template. +1. Create a full baseline unit test suite for the project. + +### Configure Diffblue Cover + +1. On the left sidebar, select **Search or go to** and find your project. + - If you want to test the integration with a sample project, you can [import](../user/project/import/repo_by_url.md) + the Diffblue [Spring PetClinic sample project](https://github.com/diffblue/demo-spring-petclinic). +1. Select **Settings > Integrations**. +1. Find **Diffblue Cover** and select **Configure**. +1. Complete the fields: + + - Select the **Active** checkbox. + - Enter your Diffblue Cover **License key** provided in your welcome email or by your organization. + If needed, select the [**Try Diffblue Cover**](https://www.diffblue.com/try-cover/gitlab) link to sign up for a free trial. + - Enter details of your GitLab access token (**Name** and **Secret**) to allow Diffblue Cover to access your project. + In general, use a GitLab [project access token](../user/project/settings/project_access_tokens.md) with the `Developer` role, plus `api` and `write_repository` scopes. + If necessary you can use a [group access token](../user/group/settings/group_access_tokens.md) or a [personal access token](../user/profile/personal_access_tokens.md), again with the `Developer` role, plus `api` and `write_repository` scopes. + + NOTE: + Using an access token with excessive permissions is a security risk. + If you use a Personal access token, consider creating a dedicated user with access limited to just the project, minimizing the impact of the token being leaked. + +1. Select **Save changes**. + Your Diffblue Cover integration is now <mark style="color:green;">**Active**</mark> and ready for use in your project. + +### Configure a pipeline + +Here we'll create a merge request pipeline for the project that will download the latest version of Diffblue Cover, build the project, write Java unit tests for the project, and commit the changes to the branch. + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Build > Pipeline editor**. +1. Select **Configure pipeline** to create the `.gitlab-ci.yml` file. +1. Select **Browse templates** and find the `Diffblue-Cover.gitlab-ci.yml` template file. +1. Select the file and copy the contents to your project's `.gitlab-ci.yml` file. + + NOTE: + When using the Diffblue Cover pipeline template with your own project and existing pipeline file, add the Diffblue template content to your file and modify as needed. + For more information, see [Cover Pipeline for GitLab](https://docs.diffblue.com/features/cover-pipeline/cover-pipeline-for-gitlab) in the Diffblue documentation. + +1. Enter a commit message. +1. Enter a new **Branch** name. For example, `add-diffblue-cover-pipeline`. +1. Select **Start a new merge request with these changes**. +1. Select **Commit changes**. + +### Create a baseline unit test suite + +1. In the **New merge request** form, enter a **Title** (for example, "Add Cover pipeline and create baseline unit test suite") and fill out the other fields. +1. Select **Create merge request**. The merge request pipeline runs Diffblue Cover to create the baseline unit test suite for the project. +1. Once the pipeline completes, the changes can be reviewed from the **Changes** tab. When you're happy, merge the updates to your repo. Go to the `src/test` folders in the project repository to see the unit tests created by Diffblue Cover (suffixed with `*DiffblueTest.java`). + +## Subsequent code changes + +When performing subsequent code changes to a project, the merge request pipeline will run Diffblue Cover but will only update the associated tests. +The resulting diff can then be analyzed to check the new behavior, catch regressions, and spot any unplanned behavioral changes to the code. + + + +## Next steps + +This topic demonstrates some of the key features of Cover Pipeline for GitLab and how to use the integration in a pipeline. +The wider and deeper functionality, provided through `dcover` commands in the pipeline template, can be implemented to expand your unit test capabilities even further. +For more information, see [Cover Pipeline for GitLab](https://docs.diffblue.com/features/cover-pipeline/cover-pipeline-for-gitlab) in the Diffblue documentation. diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index 05ffaa98a42..2c074162634 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -16,9 +16,7 @@ your Git branches like a CI/CD server. This means you don't have to wait for dependencies to be downloaded and builds to finish, you can start coding immediately. With Gitpod you can start coding instantly on any project, branch, and merge -request from any device, at any time, from your browser: - - +request from your browser. To use the GitLab Gitpod integration, it must be enabled for your GitLab instance. Users of: diff --git a/doc/integration/img/diffblue_cover_diff_v16_8.png b/doc/integration/img/diffblue_cover_diff_v16_8.png Binary files differnew file mode 100644 index 00000000000..1ee4dd954af --- /dev/null +++ b/doc/integration/img/diffblue_cover_diff_v16_8.png diff --git a/doc/integration/img/diffblue_cover_workflow_after_v16_8.png b/doc/integration/img/diffblue_cover_workflow_after_v16_8.png Binary files differnew file mode 100644 index 00000000000..21345600125 --- /dev/null +++ b/doc/integration/img/diffblue_cover_workflow_after_v16_8.png diff --git a/doc/integration/img/gitpod_web_interface_v13_4.png b/doc/integration/img/gitpod_web_interface_v13_4.png Binary files differdeleted file mode 100644 index 5cd9a6aad0f..00000000000 --- a/doc/integration/img/gitpod_web_interface_v13_4.png +++ /dev/null diff --git a/doc/integration/index.md b/doc/integration/index.md index 736f25f71d8..2a3154a739d 100644 --- a/doc/integration/index.md +++ b/doc/integration/index.md @@ -1,6 +1,7 @@ --- stage: Manage group: Import and Integrate +description: Projects, issues, authentication, security providers. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 6f6a882031a..2dac6dd5cf5 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -104,15 +104,15 @@ When configuring the GitLab for Jira Cloud app on GitLab.com, you might encounte For self-managed GitLab, see [GitLab for Jira Cloud app administration](../../administration/settings/jira_cloud_app_troubleshooting.md). -### `Failed to link group` +### Error when connecting the app -After you connect the GitLab for Jira Cloud app, you might get this error: +When you connect the GitLab for Jira Cloud app, you might get this error: ```plaintext Failed to link group. Please try again. ``` -`403` status code is returned if the user information cannot be fetched from Jira due to insufficient permissions. +A `403 Forbidden` is returned if the user information cannot be fetched from Jira because of insufficient permissions. -To resolve this issue, ensure that the Jira user that installs and configures the GitLab for Jira Cloud app meets certain +To resolve this issue, ensure the Jira user that installs and configures the app meets certain [requirements](../../administration/settings/jira_cloud_app.md#jira-user-requirements). diff --git a/doc/integration/jira/dvcs/troubleshooting.md b/doc/integration/jira/dvcs/troubleshooting.md index ee697f1bffd..04113975f06 100644 --- a/doc/integration/jira/dvcs/troubleshooting.md +++ b/doc/integration/jira/dvcs/troubleshooting.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting Jira DVCS connector **(FREE ALL)** -Refer to the items in this section if you're having problems with your Jira DVCS connector. +When working with the [Jira DVCS connector](index.md), you might encounter the following issues. -## Jira cannot access GitLab server +## Jira cannot access the GitLab server If you complete the **Add New Account** form, authorize access, and you receive this error, Jira and GitLab cannot connect. No other error messages @@ -20,9 +20,10 @@ Error obtaining access token. Cannot access https://gitlab.example.com from Jira ## Session token bug in Jira -When using GitLab 15.0 and later (including GitLab.com) with Jira Server, you might experience -a [session token bug in Jira](https://jira.atlassian.com/browse/JSWSERVER-21389). As a workaround, -ensure Jira Server is version 9.1.0 and later or 8.20.11 and later. +When you use GitLab 15.0 and later with Jira Server, you might encounter a +[session token bug in Jira](https://jira.atlassian.com/browse/JSWSERVER-21389). + +To resolve this issue, ensure Jira Server is version 9.1.0 and later or 8.20.11 and later. ## SSL and TLS problems @@ -42,7 +43,7 @@ Error obtaining access token. Cannot access https://gitlab.example.com from Jira issued by a public certificate authority, add the appropriate certificate (such as your organization's root certificate) to the Java Truststore on Jira Server. -For help with Jira setup, see the Atlassian documentation and Atlassian Support: +For more information about setting up Jira, see the Atlassian documentation and Atlassian Support. - [Add a certificate](https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html) to the trust store. @@ -68,7 +69,7 @@ The message `Successfully connected` indicates a successful TLS handshake. If there are problems, the Java TLS library generates errors that you can look up for more detail. -## Scope error when connecting to Jira using DVCS +## Scope error when connecting to Jira with DVCS ```plaintext The requested scope is invalid, unknown, or malformed. @@ -83,7 +84,7 @@ Potential resolutions: [GitLab account configuration](index.md#create-a-gitlab-application-for-dvcs). Review the **Scopes** field and ensure the `api` checkbox is selected. -## Jira error adding account and no repositories listed +## Error when adding an account in Jira After you complete the **Add New Account** form in Jira and authorize access, you might encounter these issues: @@ -100,13 +101,12 @@ To resolve this issue: [Contact GitLab Support](https://about.gitlab.com/support/) if none of these reasons apply. -## `410 : Gone` error when connecting to Jira - -When you connect to Jira and synchronize repositories, you may receive a `410 : Gone` error. +## `410 Gone` when connecting to Jira +When you connect to Jira and synchronize repositories, you might get a `410 Gone` error. This issue occurs when you use the Jira DVCS connector and your integration is configured to use **GitHub Enterprise**. -For more information and possible fixes, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340160). +For more information, see [issue 340160](https://gitlab.com/gitlab-org/gitlab/-/issues/340160). ## Synchronization issues @@ -123,7 +123,7 @@ resynchronize the information: For more information, see the [Atlassian documentation](https://support.atlassian.com/jira-cloud-administration/docs/integrate-with-development-tools/). -## `Sync Failed` error when refreshing repository data +## `Sync Failed` when refreshing repository data If you get a `Sync Failed` error in Jira when [refreshing repository data](index.md#refresh-data-imported-to-jira) for specific projects, check your Jira DVCS connector logs. Look for errors that occur when executing requests to API resources in GitLab. For example: @@ -132,8 +132,8 @@ Failed to execute request [https://gitlab.com/api/v4/projects/:id/merge_requests {"message":"403 Forbidden"} ``` -If you find a `{"message":"403 Forbidden"}` error, it is possible that this specific project has some [GitLab features disabled](../../../user/project/settings/project_features_permissions.md#configure-project-features-and-permissions). -In the example above, the merge requests feature is disabled. +If you get a `403 Forbidden` error, this project might have some [GitLab features disabled](../../../user/project/settings/project_features_permissions.md#configure-project-features-and-permissions). +In the previous example, the merge requests feature is disabled. To resolve the issue, enable the relevant feature: diff --git a/doc/integration/jira/troubleshooting.md b/doc/integration/jira/troubleshooting.md index 6c8b49b4159..a9f190068b4 100644 --- a/doc/integration/jira/troubleshooting.md +++ b/doc/integration/jira/troubleshooting.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting Jira issue integration **(FREE ALL)** -This page contains a list of common issues you might encounter when working with the [Jira issue integration](configure.md). +When working with the [Jira issue integration](configure.md), you might encounter the following issues. ## GitLab cannot link to a Jira issue @@ -70,7 +70,7 @@ If GitLab cannot close a Jira issue: - Ensure the transition ID you set in the Jira settings matches the one your project must have to close an issue. For more information, see - [automatic issue transitions](issues.md#automatic-issue-transitions) and [custom issue transitions](issues.md#custom-issue-transitions). + [Automatic issue transitions](issues.md#automatic-issue-transitions) and [Custom issue transitions](issues.md#custom-issue-transitions). - Make sure the Jira issue is not already marked as resolved: - Check the Jira issue resolution field is not set. - Check the issue is not struck through in Jira lists. @@ -122,7 +122,7 @@ To resolve this issue, see WARNING: Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. -### Change all projects on the instance +### Change all projects on an instance To change all Jira projects to use instance-level integration settings: @@ -189,7 +189,7 @@ To change all Jira projects in a group (and its subgroups) to use group-level in end ``` -## Update the Jira issue integration password for all projects +## Update the integration password for all projects WARNING: Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. @@ -212,7 +212,7 @@ When [viewing Jira issues](issues.md#view-jira-issues) in GitLab, you might enco ### `500 We're sorry` when accessing a Jira issue in GitLab When accessing a Jira issue in GitLab, you might get a `500 We're sorry. Something went wrong on our end` error. -Check [`production.log`](../../administration/logs/index.md#productionlog) to see if it contains the following exception: +Check [`production.log`](../../administration/logs/index.md#productionlog) to see if the file contains the following exception: ```plaintext :NoMethodError (undefined method 'duedate' for #<JIRA::Resource::Issue:0x00007f406d7b3180>) diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index b5515a730d3..8f2e121cb26 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -4,7 +4,7 @@ group: Authentication info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# Use Kerberos as an OAuth 2.0 authentication provider **(FREE SELF)** +# Integrate GitLab with Kerberos **(FREE SELF)** GitLab can integrate with [Kerberos](https://web.mit.edu/kerberos/) as an authentication mechanism. diff --git a/doc/integration/partner_marketplace.md b/doc/integration/partner_marketplace.md index 36aef0a0a90..cbcb8f70164 100644 --- a/doc/integration/partner_marketplace.md +++ b/doc/integration/partner_marketplace.md @@ -124,8 +124,8 @@ curl \ To create a new customer subscription from a Marketplace partner client application, - Make an authorized POST request to the -[`/api/v1/marketplace/subscriptions`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/post_api_v1_marketplace_subscriptions) -endpoint in the Customers Portal with the following parameters in JSON format: + [`/api/v1/marketplace/subscriptions`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/post_api_v1_marketplace_subscriptions) + endpoint in the Customers Portal with the following parameters in JSON format: | Parameter | Type | Required | Description | |--------------------------|--------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -144,8 +144,8 @@ If the subscription creation is unsuccessful, the response body includes an erro To get the status of a given subscription, - Make an authorized GET request to the -[`/api/v1/marketplace/subscriptions/{external_subscription_id}`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/get_api_v1_marketplace_subscriptions__external_subscription_id_) -endpoint in the Customers Portal. + [`/api/v1/marketplace/subscriptions/{external_subscription_id}`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/get_api_v1_marketplace_subscriptions__external_subscription_id_) + endpoint in the Customers Portal. The request must include the Marketplace partner system ID of the subscription to fetch the status for. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 3423b1bde6d..466c1ec7ed0 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -666,10 +666,12 @@ IdPs, contact your provider's support. Prerequisites: - Make sure you have access to a -[Google Workspace Super Admin account](https://support.google.com/a/answer/2405986#super_admin). + [Google Workspace Super Admin account](https://support.google.com/a/answer/2405986#super_admin). + +To set up a Google Workspace: 1. Use the following information, and follow the instructions in -[Set up your own custom SAML application in Google Workspace](https://support.google.com/a/answer/6087519?hl=en). + [Set up your own custom SAML application in Google Workspace](https://support.google.com/a/answer/6087519?hl=en). | | Typical value | Description | |:-----------------|:---------------------------------------------------|:----------------------------------------------------------------------------------------------| @@ -2437,7 +2439,10 @@ The value given is added to the current time at which the response is validated. ::EndTabs -### Designate a unique attribute for the `uid` +### Designate a unique attribute for the `uid` (optional) + +By default, the users `uid` is set as the `NameID` attribute in the SAML response. To designate +a different attribute for the `uid`, you can set the `uid_attribute`. Before setting the `uid` to a unique attribute, make sure that you have configured the following attributes so your SAML users cannot change them: @@ -2448,10 +2453,7 @@ the following attributes so your SAML users cannot change them: If users can change these attributes, they can sign in as other authorized users. See your SAML IdP documentation for information on how to make these attributes unchangeable. - -By default, the `uid` is set as the `name_id` in the SAML response. To designate -a unique attribute for the `uid`, you can set the `uid_attribute`. In the following -example, the value of `uid` attribute in the SAML response is set as the `uid_attribute`. +In the following example, the value of `uid` attribute in the SAML response is set as the `uid_attribute`. ::Tabs diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index f30f073bf08..0ff99a144c2 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -4,7 +4,7 @@ group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Use Shibboleth as an OAuth 2.0 authentication provider **(FREE SELF)** +# Use Shibboleth as an authentication provider **(FREE SELF)** NOTE: Use the [GitLab SAML integration](saml.md) to integrate specific Shibboleth identity providers (IdPs). For Shibboleth federation support (Discovery Service), use this document. diff --git a/doc/integration/sourcegraph.md b/doc/integration/sourcegraph.md index f6fb387f016..198dda7ec95 100644 --- a/doc/integration/sourcegraph.md +++ b/doc/integration/sourcegraph.md @@ -42,7 +42,7 @@ If you are using an HTTPS connection to GitLab, you must [configure HTTPS](https 1. Navigate to the site Admin Area in Sourcegraph. 1. [Configure your GitLab external service](https://docs.sourcegraph.com/admin/external_service/gitlab). -You can skip this step if you already have your GitLab repositories searchable in Sourcegraph. + You can skip this step if you already have your GitLab repositories searchable in Sourcegraph. 1. Validate that you can search your repositories from GitLab in your Sourcegraph instance by running a test query. 1. Add your GitLab instance URL to the [`corsOrigin` setting](https://docs.sourcegraph.com/admin/config/site_config#corsOrigin) in your site configuration. diff --git a/doc/integration/vault.md b/doc/integration/vault.md index 54ade6c1066..8a76ef02215 100644 --- a/doc/integration/vault.md +++ b/doc/integration/vault.md @@ -4,7 +4,7 @@ group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Use Vault as a GitLab OpenID Connect authentication provider **(FREE ALL)** +# Vault authentication with GitLab OpenID Connect **(FREE ALL)** [Vault](https://www.vaultproject.io/) is a secrets management application offered by HashiCorp. It allows you to store and manage sensitive information such as secret environment diff --git a/doc/operations/img/metrics_details_v16_8.png b/doc/operations/img/metrics_details_v16_8.png Binary files differnew file mode 100644 index 00000000000..ede5ed95e97 --- /dev/null +++ b/doc/operations/img/metrics_details_v16_8.png diff --git a/doc/operations/img/metrics_list_v16_8.png b/doc/operations/img/metrics_list_v16_8.png Binary files differnew file mode 100644 index 00000000000..29a12fa205b --- /dev/null +++ b/doc/operations/img/metrics_list_v16_8.png diff --git a/doc/operations/img/tracing_details_v16_7.png b/doc/operations/img/tracing_details_v16_7.png Binary files differnew file mode 100644 index 00000000000..2e9cb180994 --- /dev/null +++ b/doc/operations/img/tracing_details_v16_7.png diff --git a/doc/operations/img/tracing_drawer_v16_7.png b/doc/operations/img/tracing_drawer_v16_7.png Binary files differnew file mode 100644 index 00000000000..263151edaab --- /dev/null +++ b/doc/operations/img/tracing_drawer_v16_7.png diff --git a/doc/operations/img/tracing_list_v16_3.png b/doc/operations/img/tracing_list_v16_3.png Binary files differdeleted file mode 100644 index 93c336d4bd7..00000000000 --- a/doc/operations/img/tracing_list_v16_3.png +++ /dev/null diff --git a/doc/operations/img/tracing_list_v16_7.png b/doc/operations/img/tracing_list_v16_7.png Binary files differnew file mode 100644 index 00000000000..ad186a33bb1 --- /dev/null +++ b/doc/operations/img/tracing_list_v16_7.png diff --git a/doc/operations/index.md b/doc/operations/index.md index 0ea5922d378..e14d371e75d 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -1,6 +1,7 @@ --- stage: Service Management group: Respond +description: Error tracking, incident management. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -34,6 +35,12 @@ and the work required to fix them - all without leaving GitLab. - Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). +## Manage application performance with distributed tracing + +GitLab can troubleshoot application performance issues with distributed tracing by inspecting how a request moves through different services and systems, the timing of each operation, and any errors or logs as they occur. You can leverage tracing for your microservice applications, which group multiple independent services collaborating to fulfill user requests. + +- See [Distributed tracing](tracing.md). + ## Manage your infrastructure in code GitLab stores and executes your infrastructure as code, whether it's diff --git a/doc/operations/metrics.md b/doc/operations/metrics.md new file mode 100644 index 00000000000..b99c86d1a23 --- /dev/null +++ b/doc/operations/metrics.md @@ -0,0 +1,76 @@ +--- +stage: Monitor +group: Observability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Metrics **(ULTIMATE SAAS EXPERIMENT)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.7 [with a flag](../administration/feature_flags.md) named `observability_metrics`. Disabled by default. This feature is an [Experiment](../policy/experiment-beta-support.md#experiment). + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, an administrator can [enable the feature flag](../administration/feature_flags.md) named `observability_metrics`. +On GitLab.com, this feature is not available. +The feature is not ready for production use. + +Metrics provide insight about the operational health of monitored systems. +Use metrics to learn more about your systems and applications in a given time range. + +Metrics are structured as time series data, and are: + +- Indexed by timestamp +- Continuously expanding as additional data is gathered +- Usually aggregated, downsampled, and queried by range +- Have write-intensive requirements + +## Configure metrics + +Configure metrics to enable them for a project. + +Prerequisites: + +You must have at least the Maintainer role for the project. + +1. Create an access token and enable metrics: + 1. On the left sidebar, select **Search or go to** and find your project. + 1. Select **Settings > Access Tokens**. + 1. Create an access token with the following scopes: `read_api`, `read_observability`, `write_observability`. Be sure to save the access token value for later. + 1. Select **Monitor > Tracing**, and then select **Enable**. +1. To configure your application to send GitLab metrics, set the following environment variables: + + ```shell + OTEL_EXPORTER = "otlphttp" + OTEL_EXPORTER_OTLP_METRICS_ENDPOINT = "https://observe.gitlab.com/v3/<namespace-id>/<gitlab-project-id>/ingest/metrics" + OTEL_EXPORTER_OTLP_METRICS_HEADERS = "PRIVATE-TOKEN=<gitlab-access-token>" + ``` + + Use the following values: + + - `namespace-id` - The top-level group ID that contains the project + - `gitlab-project-id` - The project ID + - `gitlab-access-token` - The access token you created + +Metrics are configured for your project. +When you run your application, the OpenTelemetry exporter sends metrics to GitLab. + +## View metrics + +You can view the metrics for a given project: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Monitor > Metrics**. + +A list of metrics is displayed. +Select a metric to view its details. + + + +### Metric details + +Metrics are displayed as either a sum, a gauge, or a histogram. +The metric details page displays a chart depending on the type of metric. + +On the metric details page, you can also view a metric for a specific time range. + + diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index cb31fdd8025..f6ac0b6174d 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -4,9 +4,9 @@ group: Observability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Distributed tracing **(ULTIMATE SAAS EXPERIMENT)** +# Distributed tracing **(ULTIMATE SAAS BETA)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.2 [with flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. Disabled by default. This feature is an [Experiment](../policy/experiment-beta-support.md#experiment). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.2 [with flags](../administration/feature_flags.md) named `observability_group_tab` and `observability_tracing`. Disabled by default. This feature is in [Beta](../policy/experiment-beta-support.md#beta). > - Feature flag `observability_group_tab` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133264) in GitLab 16.5. FLAG: @@ -16,7 +16,7 @@ The feature is not ready for production use. With distributed tracing, you can troubleshoot application performance issues by inspecting how a request moves through different services and systems, the timing of each operation, and any errors or logs as they occur. Tracing is particularly useful in the context of microservice applications, which group multiple independent services collaborating to fulfill user requests. -This feature is an [Experiment](../policy/experiment-beta-support.md). For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/observability/). To leave feedback about tracing bugs or functionality, comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2363) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). +This feature is in [Beta](../policy/experiment-beta-support.md). For more information, see the [group direction page](https://about.gitlab.com/direction/analytics/observability/). To leave feedback about tracing bugs or functionality, comment in the [feedback issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/2590) or open a [new issue](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/new). ## Configure distributed tracing for a project @@ -70,6 +70,21 @@ To view the list of traces: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Monitor > Traces**. -To see the details of a trace, select it from the list. +To see the details of a trace, select it from the list. You can also select a trace directly from the scatterplot. - + + +The trace details page and a list of spans are displayed. + + + +To view the attributes for a single span, select it from the list. + + + +## Tracing ingestion limits + +Tracing ingests a maximum of 102,400 bytes per minute. +After the limit is exceeded, a `429 Too Many Requests` response is returned. + +To request a limit increase to 104,8576 bytes per minute, contact GitLab support. diff --git a/doc/policy/experiment-beta-support.md b/doc/policy/experiment-beta-support.md index 0c58380d304..a563ce7919d 100644 --- a/doc/policy/experiment-beta-support.md +++ b/doc/policy/experiment-beta-support.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Support details. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 8e4541a58a7..0790cf07ed1 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -72,14 +72,14 @@ GitLab. These two policies are in place because: 1. GitLab has Community and Enterprise distributions, doubling the amount of work -necessary to test/release the software. + necessary to test/release the software. 1. Backporting to more than one release creates a high development, quality assurance, -and support cost. + and support cost. 1. Supporting parallel version discourages incremental upgrades which over time accumulate in -complexity and create upgrade challenges for all users. GitLab has a dedicated team ensuring that -incremental upgrades (and installations) are as simple as possible. + complexity and create upgrade challenges for all users. GitLab has a dedicated team ensuring that + incremental upgrades (and installations) are as simple as possible. 1. The number of changes created in the GitLab application is high, which contributes to backporting complexity to older releases. In several cases, backporting has to go through the same -review process a new change goes through. + review process a new change goes through. 1. Ensuring that tests pass on the older release is a considerable challenge in some cases, and as such is very time-consuming. Including new features in a patch release is not possible as that would break [Semantic Versioning](https://semver.org/). diff --git a/doc/security/hardening_application_recommendations.md b/doc/security/hardening_application_recommendations.md index 857e322191e..4ff1e94deb4 100644 --- a/doc/security/hardening_application_recommendations.md +++ b/doc/security/hardening_application_recommendations.md @@ -108,7 +108,7 @@ If GitLab is in FIPS mode, use the following: - If using `RSA`, set it to **Must be at least 2048 bits**. - Set all other key types to **Are forbidden**. - If you are setting up an instance for a new group of users, define your user SSH -key policy with the maximum bits settings for added security. + key policy with the maximum bits settings for added security. In a hardened environment RSS feeds are typically not required, and in **Feed token**, select the **Disabled feed token** checkbox. @@ -192,14 +192,14 @@ process or authenticated user. The main focus for hardening is **Usage statistics**: - You should make sure **Enable version check** is selected. This checks to see if you -are running the latest version of GitLab, and as new versions with new features and -security patches come out frequently, this helps you stay up to date. + are running the latest version of GitLab, and as new versions with new features and + security patches come out frequently, this helps you stay up to date. - If your environment is isolated or one where your organizational requirements -restrict data gathering and statistics reporting to a software vendor, you may have -to disable the **Enable service ping** feature. For more information on what data is collected to -help you make an informed decision, see -[service ping](../development/internal_analytics/service_ping/index.md). + restrict data gathering and statistics reporting to a software vendor, you may have + to disable the **Enable service ping** feature. For more information on what data is collected to + help you make an informed decision, see + [service ping](../development/internal_analytics/service_ping/index.md). ## Network @@ -215,12 +215,12 @@ and user needs, which may require disabling and adjusting rate limits or enablin accesses. Here are a few notables to keep in mind: - In **Outbound requests**, if you need to open up access to a limited -number of systems, you can limit access to just those systems by specifying -IP address or hostname. Also in this section, make sure you've selected -**Enforce DNS rebinding attack protection** if you're allowing any access at all. + number of systems, you can limit access to just those systems by specifying + IP address or hostname. Also in this section, make sure you've selected + **Enforce DNS rebinding attack protection** if you're allowing any access at all. - Under **Notes rate limit** and **Users API rate limit** you can exclude specific users -from those limits if needed. + from those limits if needed. <!-- ## Troubleshooting diff --git a/doc/security/hardening_cicd_recommendations.md b/doc/security/hardening_cicd_recommendations.md index 4d0a85c362d..72f3bc8e7b8 100644 --- a/doc/security/hardening_cicd_recommendations.md +++ b/doc/security/hardening_cicd_recommendations.md @@ -22,18 +22,18 @@ individual scenarios themselves are numerous, we have summarized some basic information to help harden the CI/CD process. - **Secrets Management**. Passwords, tokens, keys, and other secrets that require any -level of protection should never be stored in plaintext. Some type of encrypted -container technology should be used, such as GCP Secret Manager, AWS KMS, or -HashiCorp Vault. For self-managed and standalone instances, HashiCorp Vault is -recommended, and many GitLab features can take advantage of Vault and are well -documented in the main [Documentation](../index.md). For detailed CI/CD examples, see [using external secrets in CI](../ci/secrets/index.md). + level of protection should never be stored in plaintext. Some type of encrypted + container technology should be used, such as GCP Secret Manager, AWS KMS, or + HashiCorp Vault. For self-managed and standalone instances, HashiCorp Vault is + recommended, and many GitLab features can take advantage of Vault and are well + documented in the main [Documentation](../index.md). For detailed CI/CD examples, see [using external secrets in CI](../ci/secrets/index.md). - **External Communications**. If your CI/CD process requires connectivity to other -hosts, ensure that these communication channels are encrypted. You should use TLS 1.2 or 1.3, and where possible implement mutual TLS. + hosts, ensure that these communication channels are encrypted. You should use TLS 1.2 or 1.3, and where possible implement mutual TLS. - **Logging**. Logging can be very important for auditing and troubleshooting, so it -is important that you enable any logging features to ensure you are getting -the information in logs you need. Make sure through periodic testing that -plaintext secrets or other sensitive information is not inadvertently added to log -files. + is important that you enable any logging features to ensure you are getting + the information in logs you need. Make sure through periodic testing that + plaintext secrets or other sensitive information is not inadvertently added to log + files. ## Specific Recommendations diff --git a/doc/security/hardening_general_concepts.md b/doc/security/hardening_general_concepts.md index 0ba8822dc5f..cb0dcb4eba7 100644 --- a/doc/security/hardening_general_concepts.md +++ b/doc/security/hardening_general_concepts.md @@ -19,10 +19,9 @@ just one. A quick example is account security: - Use a long, complex, and unique password for the account. - Implement a second factor to the authentication process for added security. - Use a hardware token as a second factor. -- Lock out an account (for at least a fixed amount of time) for failed authentication -attempts. +- Lock out an account (for at least a fixed amount of time) for failed authentication attempts. - An account that is unused for a specific time frame should be disabled, enforce this -with either automation or regular audits. + with either automation or regular audits. Instead of using only one or two items on the list, use as many as possible. This philosophy can apply to other areas besides account security - it should be applied to diff --git a/doc/security/index.md b/doc/security/index.md index 8fd55fd08ff..ffc436e4286 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -1,6 +1,7 @@ --- stage: Govern group: Authentication +description: SSH key limits, 2FA, tokens, hardening. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -27,6 +28,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Maximum decompressed file size for imported archives](../administration/settings/import_and_export_settings.md#maximum-decompressed-file-size-for-imported-archives) - [Responding to security incidents](responding_to_security_incidents.md) -To harden your GitLab instance and minimize the risk of unwanted user account creation, consider access control features like [Sign up restrictions](../administration/settings/sign_up_restrictions.md) and [Authentication options](../topics/authentication/index.md). For more detailed information, refer to [Hardening](hardening.md). +To harden your GitLab instance and minimize the risk of unwanted user account creation, consider access control features like [Sign up restrictions](../administration/settings/sign_up_restrictions.md) and [Authentication options](../administration/auth/index.md). For more detailed information, refer to [Hardening](hardening.md). Self-managed GitLab customers and administrators are responsible for the security of their underlying hosts, and for keeping GitLab itself up to date. It is important to [regularly patch GitLab](../policy/maintenance.md), patch your operating system and its software, and harden your hosts in accordance with vendor guidance. diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index 4555459e7c5..9cd445ed47b 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -238,13 +238,14 @@ The following tables show the prefixes for each type of token where applicable. | Deploy key | Not applicable. | | Runner registration token | Not applicable. | | Runner authentication token | `glrt-` | -| Job token | Not applicable. | +| CI/CD Job token | `glcbt-` ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/426137) in GitLab 16.8 behind a feature flag named `prefix_ci_build_tokens`. Disabled by default.) | | Trigger token | `glptt-` | | Legacy runner registration token | GR1348941 | | Feed token | `glft-` | | Incoming mail token | `glimt-` | | GitLab Agent for Kubernetes token | `glagent-` | | GitLab session cookies | `_gitlab_session=` | +| SCIM Tokens | `glsoat-` ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/435096) in GitLab 16.8 behind a feature flag named `prefix_scim_tokens`. Disabled by default.) | ### External system tokens diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index c7d6796a212..80b6988b1b1 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -49,6 +49,21 @@ Use the [application settings API](../api/settings.md) to modify the following s For more information, see the [list of settings that can be accessed through API calls](../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). +## Enforce 2FA for Administrator users **(FREE SELF)** + +Administrators can enforce 2FA for administrator users in a self-managed instance. + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Sign-in restrictions** section: + - Select **Require administrators to enable 2FA**. + - In **Two-factor grace period**, enter a number of hours. If you want to + enforce 2FA on the next sign-in attempt, enter `0`. +1. Select **Save changes**. + +NOTE: +If you are using an external provider to sign in into GitLab, this setting will **not** enforce 2FA for users. 2FA should be enabled on that external provider. + ## Enforce 2FA for all users in a group **(FREE ALL)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24965) in GitLab 12.0, 2FA settings for a group are also applied to subgroups. diff --git a/doc/solutions/cloud/aws/gitlab_aws_integration.md b/doc/solutions/cloud/aws/gitlab_aws_integration.md index 11add88f7c0..fddb5ccfa38 100644 --- a/doc/solutions/cloud/aws/gitlab_aws_integration.md +++ b/doc/solutions/cloud/aws/gitlab_aws_integration.md @@ -15,19 +15,15 @@ When content that is badged for GitLab SaaS ( **(SAAS)** ) or Self-Managed ( **( This page attempts to index the ways in which GitLab can integrate with AWS. It does so whether the integration is the result of configuring general functionality, was built in to AWS or GitLab or is provided as a solution. -| Text Tag | Configuration / Built / Solution | Support/Maintenance | -| -------------------- | ------------------------------------------------------------ | ------------------- | -| `[AWS Configuration]` | Integration via Configuring Existing AWS Functionality | AWS | -| `[GitLab Configuration]` | Integration via Configuring Existing GitLab Functionality | GitLab | -| `[AWS Built]` | Built into AWS by Product Team to Address AWS Integration | AWS | -| `[GitLab Built]` | Built into GitLab by Product Team to Address AWS Integration | GitLab | -| `[AWS Solution]` | Built as Solution Example by AWS or AWS Partners | Community/Example | -| `[GitLab Solution]` | Built as Solution Example by GitLab or GitLab Partners | Community/Example | -| `[CI Solution]` | Built, at least in part, using GitLab CI and therefore <br />more customer customizable. | Items tagged `[CI Solution]` will <br />also carry one of the other tags <br />that indicate the maintenance status. | - -## Table of Contents - -[TOC] +| Text Tag | Configuration / Built / Solution | Support/Maintenance | +| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| `[AWS Configuration]` | Integration via Configuring Existing AWS Functionality | AWS | +| `[GitLab Configuration]` | Integration via Configuring Existing GitLab Functionality | GitLab | +| `[AWS Built]` | Built into AWS by Product Team to Address AWS Integration | AWS | +| `[GitLab Built]` | Built into GitLab by Product Team to Address AWS Integration | GitLab | +| `[AWS Solution]` | Built as Solution Example by AWS or AWS Partners | Community/Example | +| `[GitLab Solution]` | Built as Solution Example by GitLab or GitLab Partners | Community/Example | +| `[CI Solution]` | Built, at least in part, using GitLab CI and therefore <br />more customer customizable. | Items tagged `[CI Solution]` will <br />also carry one of the other tags <br />that indicate the maintenance status. | ## Integrations For Development Activities @@ -35,47 +31,90 @@ These integrations have to do with using GitLab to build application workloads a ### SCM Integrations -- **AWS CodeStar Connections** - enables SCM connections to multiple AWS Services. **Currently for GitLab.com SaaS only**. [Configure GitLab](https://docs.aws.amazon.com/dtconsole/latest/userguide/connections-create-gitlab.html). [Supported Providers](https://docs.aws.amazon.com/dtconsole/latest/userguide/supported-versions-connections.html). [Supported AWS Services](https://docs.aws.amazon.com/dtconsole/latest/userguide/integrations-connections.html) - each one may have to make updates to support GitLab, so here is the subset that currently support GitLab `[AWS Built]` - - [AWS CodePipeline Integration](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab.html) - use GitLab as source for CodePipeline. `[AWS Built]` - - **AWS CodeBuild Integration** - indirectly through CodePipeline support. `[AWS Built]` - - **Amazon CodeWhisperer Customization Capability** [can connect to a GitLab repo](https://aws.amazon.com/blogs/aws/new-customization-capability-in-amazon-codewhisperer-generates-even-better-suggestions-preview/). `[AWS Built]` - - **AWS Service Catalog** directly inherits CodeStar Connections, there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. `[AWS Built]` - - **AWS Proton** directly inherits CodeStar Connections, there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. `[AWS Built]` - - **AWS Glue Notebook Jobs** directly inherit CodeStar Connections, there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. `[AWS Built]` - - **Amazon SageMaker MLOps Projects** are done in CodePipeline and so directly inherit CodeStar Connections ([as noted here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects-walkthrough-3rdgit.html#sagemaker-proejcts-walkthrough-connect-3rdgit)), there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. `[AWS Built]` - - **Amazon SageMaker Notebooks** [allow Git repositories to be specified by the Git clone URL](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-git-resource.html) and configuration of a secret - so GitLab is configurable. `[AWS Configuration]` - - **AWS CloudFormation** publishing of public extensions - **not yet supported**. `[AWS Built]` - - **Amazon CodeGuru Reviewer Repositories** - **not yet supported**. `[AWS Built]` -- [GitLab Push Mirroring to CodeCommit](../../../user/project/repository/mirror/push.md#set-up-a-push-mirror-from-gitlab-to-aws-codecommit) Workaround enables GitLab repositories to leverage CodePipeline SCM Triggers. GitLab can already leverage S3 and Container Triggers for CodePipeline. **Still required for Self-Managed and Dedicated for the time being.** `[GitLab Configuration]` +#### AWS CodeStar Connection Integrations + +[8/14/2023 AWS Release Announcement for GitLab.com SaaS](https://aws.amazon.com/about-aws/whats-new/2023/08/aws-codepipeline-supports-gitlab/) + +[12/28/2023 AWS Release Announcement for Self-Managed / Dedicated](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/) + +**AWS CodeStar Connections** - enables SCM connections to multiple AWS Services. [Configure GitLab](https://docs.aws.amazon.com/dtconsole/latest/userguide/connections-create-gitlab.html). [Supported Providers](https://docs.aws.amazon.com/dtconsole/latest/userguide/supported-versions-connections.html). [Supported AWS Services](https://docs.aws.amazon.com/dtconsole/latest/userguide/integrations-connections.html) - each one may have to make updates to support GitLab, so here is the subset that support GitLab. This works with GitLab.com SaaS, GitLab Self-Managed and GitLab Dedicated. AWS CodeStar connections are not available in all AWS regions - the exclusion list is [documented here](https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodestarConnectionSource.html). ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` + +[Video Explanation of AWS CodeStar Connection Integration for AWS (1 min)](https://youtu.be/f7qTSa_bNig) + +AWS Services that are supported directly by a CodeStar Connection in an AWS account: + +- **Amazon CodeWhisperer Customization Capability** ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) [can connect to a GitLab repository](https://aws.amazon.com/blogs/aws/new-customization-capability-in-amazon-codewhisperer-generates-even-better-suggestions-preview/). `[AWS Built]` +- **AWS Service Catalog** directly inherits CodeStar Connections, there is not any specific documentation about GitLab because it just uses any GitLab CodeStar Connection that has been created in the account. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` +- **AWS Proton** directly inherits CodeStar Connections, there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` +- **AWS Glue Notebook Jobs** directly inherit CodeStar Connections, there is not any specific documentation about GitLab because it just uses any GitLab CodeStar Connection that has been created in the account. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` + +Documentation and References: + +- [Creating a GitLab CodeStar Connection to a GitLab.com Project](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab-managed.html) +- [Creating a AWS CodeStar Connection for a Self-Managed GitLab Instance or GitLab Dedicated Instance](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab-managed.html) (must allow Internet Ingress from AWS or use a VPC connection) + +#### AWS CodePipeline Integrations + +[AWS CodePipeline Integration](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab.html) - by using GitLab as CodeStar Connections source for CodePipeline, additional AWS service integrations are available. [[12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)] `[AWS Built]` + +AWS Services that are supported by an AWS CodePipeline integration: + +- **AWS CodeBuild Integration** - through CodePipeline support. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` +- **Amazon SageMaker MLOps Projects** are created via CodePipeline ([as noted here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects-walkthrough-3rdgit.html#sagemaker-proejcts-walkthrough-connect-3rdgit)), there is not any specific documentation about GitLab since it just uses any GitLab CodeStar Connection that has been created in the account. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` + +Documentation and References: + +- [Creating a GitLab CodePipeline Integration to a GitLab.com Project](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab-managed.html) +- [Creating a AWS CodePipeline Integration for a Self-Managed GitLab Instance or GitLab Dedicated Instance](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-gitlab-managed.html) (must allow Internet Ingress from AWS or use a VPC connection) + +#### CodeStar Connections enabled AWS services that are not yet supported for GitLab + +- **AWS CloudFormation** publishing of public extensions - not yet supported. `[AWS Built]` +- **Amazon CodeGuru Reviewer Repositories** - not yet supported. `[AWS Built]` +- **AWS App Runner** - not yet supported. `[AWS Built]` + +#### Custom GitLab Integration in AWS Services + +- **Amazon SageMaker Notebooks** [allow Git repositories to be specified by the Git clone URL](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-git-resource.html) and configuration of a secret - so GitLab is configurable. ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Configuration]` +- **AWS Amplify** - [uses a Git integration mechanism designed by the AWS Amplify team](https://docs.aws.amazon.com/amplify/latest/userguide/getting-started.html). `[AWS Built]` + +#### Other SCM Integration Options + +- [GitLab Push Mirroring to CodeCommit](../../../user/project/repository/mirror/push.md#set-up-a-push-mirror-from-gitlab-to-aws-codecommit) Workaround enables GitLab repositories to leverage CodePipeline SCM Triggers. GitLab can already leverage S3 and Container Triggers for CodePipeline. This work around enabled CodePipeline capabilities since it was documented. (06/06/2020) `[GitLab Configuration]` + +See [CD and Operations Integrations](#cd-and-operations-integrations) below for Continuous Deployment (CD) specific integrations that are also available. ### CI Integrations - **Direct CI Integrations That Use Keys, IAM or OIDC/JWT to Authenticate to AWS Services from GitLab Runners** - - **Amazon CodeGuru Reviewer CI workflows using GitLab CI** - can be done, not yet documented. `[AWS Solution]` `[CI Solution]` - - [Amazon CodeGuru Secure Scanning using GitLab CI](https://docs.aws.amazon.com/codeguru/latest/security-ug/get-started-gitlab.html) `[AWS Solution]` `[CI Solution]` +- **Amazon CodeGuru Reviewer CI workflows using GitLab CI** - can be done, not yet documented.`[AWS Solution]` `[CI Solution]` +- [Amazon CodeGuru Secure Scanning using GitLab CI](https://docs.aws.amazon.com/codeguru/latest/security-ug/get-started-gitlab.html) ([06/13/2022](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-codeguru-security-available-preview/)) `[AWS Solution]` `[CI Solution]` ### CD and Operations Integrations -- **AWS CodeDeploy Integration** - indirectly through CodePipeline support. `[AWS Built]` +- **AWS CodeDeploy Integration** - through CodePipeline support discussed above in SCM integrations. This capability allows GitLab to interface with [this list of advanced deployment subsystems in AWS](https://docs.aws.amazon.com/codepipeline/latest/userguide/integrations-action-type.html#integrations-deploy). ([12/28/2023](https://aws.amazon.com/about-aws/whats-new/2023/12/codepipeline-gitlab-self-managed/)) `[AWS Built]` +- **AWS SAM Pipelines** - [pipelines support for GitLab](https://aws.amazon.com/about-aws/whats-new/2021/07/simplify-ci-cd-configuration-serverless-applications-your-favorite-ci-cd-system-public-preview). (7/31/2021) - [Integrate EKS clusters for application deployment](../../../user/infrastructure/clusters/connect/new_eks_cluster.md). `[GitLab Built]` +- [GitLab pushing a build Artifact to a CodePipeline monitored S3 location](https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-about-starting.html#change-detection-methods) `[AWS Built]` +- [GitLab Pushing a container to a CodePipeline monitored AWS ECR](https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-about-starting.html#change-detection-methods) `[AWS Built]` -## End-to-End Solutions for development and deployment if specific development frameworks and ecosystems +## End-to-End Solutions for development and deployment of specific development frameworks or ecosystems Generally solutions demonstrate end-to-end capabilities for the development framework - leveraging all relevant integration techniques to show the art of maximum value for using GitLab and AWS together. ### Serverless -- [Serverless Framework Deployment to AWS with GitLab Serverless SAST Scanning and Lifecycle Managed DevOps Environments](https://gitlab.com/guided-explorations/aws/serverless/serverless-framework-aws) - working example code and tutorials. `[GitLab Solution]` `[CI Solution]` +- [Enterprise DevOps Blueprint: Serverless Framework Apps on AWS](https://gitlab.com/guided-explorations/aws/serverless/serverless-framework-aws) - working example code and tutorials. `[GitLab Solution]` `[CI Solution]` - [Tutorial: Serverless Framework Deployment to AWS with GitLab Serverless SAST Scanning](https://gitlab.com/guided-explorations/aws/serverless/serverless-framework-aws/-/blob/master/TUTORIAL.md) `[GitLab Solution]` `[CI Solution]` - [Tutorial: Secure Serverless Framework Development with GitLab Security Policy Approval Rules and Managed DevOps Environments](https://gitlab.com/guided-explorations/aws/serverless/serverless-framework-aws/-/blob/master/TUTORIAL2-SecurityAndManagedEnvs.md) `[GitLab Solution]` `[CI Solution]` ### Terraform -- [Terraform Deployment to AWS with GitLab Lifecycle Managed DevOps Environments](https://gitlab.com/guided-explorations/aws/terraform/terraform-web-server-cluster) +- [Enterprise DevOps Blueprint: Terraform Deployment to AWS](https://gitlab.com/guided-explorations/aws/terraform/terraform-web-server-cluster) - [Tutorial: Terraform Deployment to AWS with GitLab IaC SAST Scanning](https://gitlab.com/guided-explorations/aws/terraform/terraform-web-server-cluster/-/blob/prod/TUTORIAL.md) `[GitLab Solution]` `[CI Solution]` - [Terraform Deployment to AWS with GitLab Security Policy Approval Rules and Managed DevOps Environments](https://gitlab.com/guided-explorations/aws/terraform/terraform-web-server-cluster/-/blob/prod/TUTORIAL2-SecurityAndManagedEnvs.md) `[GitLab Solution]` `[CI Solution]` -#### CloudFormation +### CloudFormation [CloudFormation Development and Deployment With GitLab Lifecycle Managed DevOps Environments Working Code](https://gitlab.com/guided-explorations/aws/cloudformation-deploy) `[GitLab Solution]` `[CI Solution]` diff --git a/doc/subscriptions/choosing_subscription.md b/doc/subscriptions/choosing_subscription.md index aa7ba3a9042..04e05ef1be9 100644 --- a/doc/subscriptions/choosing_subscription.md +++ b/doc/subscriptions/choosing_subscription.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Options for accessing GitLab. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/subscriptions/community_programs.md b/doc/subscriptions/community_programs.md index 309a100de7a..a7b95c78491 100644 --- a/doc/subscriptions/community_programs.md +++ b/doc/subscriptions/community_programs.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Education, Open Source, Startups. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/subscriptions/customers_portal.md b/doc/subscriptions/customers_portal.md index 10de69242d9..865dcf99515 100644 --- a/doc/subscriptions/customers_portal.md +++ b/doc/subscriptions/customers_portal.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Payment and company details. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -15,7 +16,7 @@ If you made your purchase through an authorized reseller, you must contact them ## Sign in to Customers Portal -You can sign in to Customers Portal either with your GitLab.com account or your email and password (if you have not yet [linked your Customers Portal account to your GitLab.com account](#link-a-gitlabcom-account)). +You can sign in to Customers Portal either with your GitLab.com account or a one-time sign-in link sent to your email (if you have not yet [linked your Customers Portal account to your GitLab.com account](#link-a-gitlabcom-account)). NOTE: If you registered for Customers Portal with your GitLab.com account, sign in with this account. @@ -50,7 +51,7 @@ if required. The profile owner's personal details are used on invoices. The profile owner's email address is used for the [Customers Portal legacy sign-in](#sign-in-to-customers-portal) and license-related email. -To change profile details, including name, billing address, and email address: +To change profile details, including name and email address: 1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). 1. Select **My profile > Profile settings**. @@ -64,13 +65,70 @@ to another person, after you enter that person's personal details, you must also ## Change your company details -To change your company details, including company name and VAT number: +To change your company details, including company name and tax ID: 1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. Select **My profile > Profile settings**. -1. Edit **Your company details**. +1. Select **Billing account settings**. +1. Scroll down to the **Company information** section. +1. Edit the company details. +1. Select **Save changes**. + +## Subscription and billing contacts + +### Change your subscription contact + +The subscription contact is the primary contact for your billing account. They receive subscription event notifications and information about applying subscription. + +To change the subscription contact: + +1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. On the left sidebar, select **Billing account settings**. +1. Scroll to the **Company information** section, then to **Subscription contact**. +1. To select a different subscription contact, select from the **Billing account manager** dropdown list. +1. Edit the contact details. 1. Select **Save changes**. +### Change your billing contact + +The billing contact receives all invoices and subscription event notifications. + +To change the billing contact: + +1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. On the left sidebar, select **Billing account settings**. +1. Scroll to the **Company information** section, then to **Billing contact**. + + - To change your billing contact to your subscription contact: + + 1. Select **Billing contact is the same as subscription contact**. + 1. Select **Save changes**. + + - To change your billing contact to a different billing account manager: + + 1. Clear the **Billing contact is the same as subscription contact** checkbox. + 1. Select a different billing account manager from the **User** dropdown list. + 1. Edit the contact details. + 1. Select **Save changes**. + + - To change your billing contact to a custom contact: + + 1. Clear the **Billing contact is the same as subscription contact** checkbox. + 1. Select **Enter a custom contact** from the **User** dropdown list. + 1. Enter the contact details. + 1. Select **Save changes**. + +### Troubleshooting your billing or subscription contact's name + +If the billing account manager's email is linked to contacts with different first or last names, you will be prompted to update the name. + +If you are the billing account manager, follow the instructions to [update your personal profile](#change-profile-owner-information). + +If you are not the billing account manager, notify them to update their personal profile. + +### Troubleshooting your subscription contact + +If the subscription contact is no longer a billing account manager, you will be prompted to select a new contact. Follow the instructions to [change your subscription contact](#change-your-subscription-contact). + ## Change your payment method Purchases in the Customers Portal require a credit card on record as a payment method. You can add @@ -83,7 +141,7 @@ If you would like to use an alternative method to pay, To change your payment method: 1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. Select **My profile > Payment methods**. +1. On the left sidebar, select **Billing account settings**. 1. **Edit** an existing payment method's information or **Add new payment method**. 1. Select **Save Changes**. @@ -93,18 +151,19 @@ Automatic renewal of a subscription is charged to your default payment method. T method as the default: 1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. Select **My profile > Payment methods**. +1. On the left sidebar, select **Billing account settings**. 1. **Edit** the selected payment method and check the **Make default payment method** checkbox. 1. Select **Save Changes**. ## Link a GitLab.com account -Follow this guideline if you have a legacy Customers Portal profile and use an email and password to log in. +Follow this guideline if you have a legacy Customers Portal profile to log in. To link a GitLab.com account to your Customers Portal profile: -1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in?legacy=true) using email and password. -1. On the Customers Portal page, select **My profile > Profile settings**. +1. Trigger a one-time sign-in link to your email from the [Customers Portal](https://customers.gitlab.com/customers/sign_in?legacy=true). +1. Locate the email and click on the one-time sign-in link to log into your Customers Portal account. +1. Select **My profile > Profile settings**. 1. Under **Your GitLab.com account**, select **Link account**. 1. Sign in to the [GitLab.com](https://gitlab.com/users/sign_in) account you want to link to the Customers Portal profile. @@ -112,7 +171,7 @@ To link a GitLab.com account to your Customers Portal profile: Customers are required to use their GitLab.com account to register for a new Customers Portal profile. -If you have a legacy Customers Portal profile that is not linked to a GitLab.com account, you may still [sign in](https://customers.gitlab.com/customers/sign_in?legacy=true) using an email and password. However, you should [create](https://gitlab.com/users/sign_up) and [link a GitLab.com account](#change-the-linked-account) to ensure continued access to the Customers Portal. +If you have a legacy Customers Portal profile that is not linked to a GitLab.com account, you may still [sign in](https://customers.gitlab.com/customers/sign_in?legacy=true) using a one-time sign-in link sent to your email. However, you should [create](https://gitlab.com/users/sign_up) and [link a GitLab.com account](#change-the-linked-account) to ensure continued access to the Customers Portal. To change the GitLab.com account linked to your Customers Portal profile: diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index 9299028e56a..74ecf1701a4 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Seat usage, compute minutes, storage limits, renewal info. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -231,13 +232,13 @@ After you dismiss the alert, it doesn't display until another seat is used. The alert displays based on the following seat usage. You cannot configure the amounts at which the alert displays. -| Seats in subscription | Seat usage | -|-----------------------|------------------------------------------------------------------------| -| 0-15 | One seat remaining in the subscription. | -| 16-25 | Two seats remaining in the subscription. | -| 26-99 | 10% of seats have been used. | -| 100-999 | 8% of seats have been used. | -| 1000+ | 5% of seats have been used | +| Seats in subscription | Seat usage | +|-----------------------|------------| +| 0-15 | One seat remaining in the subscription. | +| 16-25 | Two seats remaining in the subscription. | +| 26-99 | 10% of seats have been used. | +| 100-999 | 8% of seats have been used. | +| 1000+ | 5% of seats have been used | ## Change the linked namespace @@ -272,10 +273,10 @@ Changing the linked namespace is not supported for all subscription types. You cannot transfer: -- A subscription with compute minutes. - An expired or trial subscription. -- A subscription to a namespace which already has a Premium or Ultimate plan. -- A subscription from a namespace with multiple subscriptions. +- A subscription with compute minutes which is already linked to a namespace. +- A subscription with a Premium or Ultimate plan to a namespace which already has a Premium or Ultimate plan. +- A subscription with code suggestions to a namespace which already has a subscriptions with code suggestions. ## Upgrade your GitLab SaaS subscription tier @@ -310,8 +311,9 @@ To renew your subscription: Before you renew your subscription: 1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. On the **Account details** page, verify or update the invoice contact details. -1. On the **Payment Methods** page, verify or update the credit card on file. +1. On the left sidebar, select **Billing account settings**. +1. Under **Payment methods**, verify or update the credit card on file. +1. Scroll down to the **Company information** section to verify or update the invoice contact details. 1. In GitLab, review your list of user accounts and [remove inactive or unwanted users](#remove-users-from-your-subscription). ### Renew or change a GitLab SaaS subscription @@ -322,8 +324,8 @@ You can only renew your subscription 15 days before it is due to expire. To renew your subscription: 1. Sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select **Renew**. -The **Renew** button displays only 15 days before a subscription expires. If there are more than 15 days before -the subscription expires, select **Subscription actions** (**{ellipsis_v}**), then select **Renew subscription** to view the date when you can renew. + The **Renew** button displays only 15 days before a subscription expires. If there are more than 15 days before + the subscription expires, select **Subscription actions** (**{ellipsis_v}**), then select **Renew subscription** to view the date when you can renew. 1. Review your renewal details and complete the payment process. 1. Select **Confirm purchase**. diff --git a/doc/subscriptions/gitlab_dedicated/index.md b/doc/subscriptions/gitlab_dedicated/index.md index 0f2fdb43512..0cf604f620f 100644 --- a/doc/subscriptions/gitlab_dedicated/index.md +++ b/doc/subscriptions/gitlab_dedicated/index.md @@ -1,12 +1,13 @@ --- stage: SaaS Platforms group: GitLab Dedicated +description: Available features and benefits. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # GitLab Dedicated -GitLab Dedicated is a fully isolated, single-tenant SaaS service that is: +GitLab Dedicated is a fully isolated, single-tenant SaaS solution that is: - Hosted and managed by GitLab, Inc. - Deployed on AWS in a cloud region of your choice. See [available AWS regions](#available-aws-regions). @@ -56,6 +57,10 @@ Data is encrypted at rest and in transit using the latest encryption standards. During onboarding, you can specify an AWS KMS encryption key stored in your own AWS account that GitLab uses to encrypt the data for your Dedicated instance. This gives you full control over the data you store in GitLab. +#### SMTP + +Email sent from GitLab Dedicated uses [Amazon Simple Email Service (Amazon SES)](https://aws.amazon.com/ses/). The connection to Amazon SES is encrypted. + ### Compliance #### Certifications @@ -69,7 +74,7 @@ GitLab Dedicated offers the following [compliance certifications](https://about. #### Isolation -As a single-tenant SaaS service, GitLab Dedicated provides infrastructure-level isolation of your GitLab environment. Your environment is placed into a separate AWS account from other tenants. This AWS account contains all of the underlying infrastructure necessary to host the GitLab application and your data stays within the account boundary. You administer the application while GitLab manages the underlying infrastructure. Tenant environments are also completely isolated from GitLab.com. +As a single-tenant SaaS solution, GitLab Dedicated provides infrastructure-level isolation of your GitLab environment. Your environment is placed into a separate AWS account from other tenants. This AWS account contains all of the underlying infrastructure necessary to host the GitLab application and your data stays within the account boundary. You administer the application while GitLab manages the underlying infrastructure. Tenant environments are also completely isolated from GitLab.com. #### Access controls @@ -109,7 +114,7 @@ To help you migrate your data to GitLab Dedicated, you can choose from the follo - Use the UI, including [group import](../../user/group/import/index.md) and [project import](../../user/project/settings/import_export.md). - Use APIs, including the [group import API](../../api/group_import_export.md) and [project import API](../../api/project_import_export.md). - Note: Import functionality behind a feature flag (such as `bulk_import_project`) is not supported in GitLab Dedicated. -1. When migrating from third-party services, you can use [the GitLab importers](../../user/project/import/index.md#available-project-importers). +1. When migrating from third-party services, you can use [the GitLab importers](../../user/project/import/index.md#supported-import-sources). 1. You can perform a fully-automated migration through the [Congregate Automation Tool](../../user/project/import/index.md#automate-group-and-project-import), which supports migrating from existing GitLab instances as well as third-party services. ## Features that are not available diff --git a/doc/subscriptions/quarterly_reconciliation.md b/doc/subscriptions/quarterly_reconciliation.md index 87a4b65833c..db822a3569e 100644 --- a/doc/subscriptions/quarterly_reconciliation.md +++ b/doc/subscriptions/quarterly_reconciliation.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Billing examples. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -84,9 +85,9 @@ sent and subject to your payment terms. If your credit card is declined during the reconciliation process, an email will be sent with the subject `Your GitLab subscription failed to reconcile`. Follow these instructions to update your payment information, and the reconciliation will be automatically retried: 1. Log in to your account at `https://customers.gitlab.com`. -1. Go to **Payment Methods**. -1. Select **Add New Payment Method**. -1. Make sure that the payment method is set as **Default**. +1. On the left sidebar, select **Billing account settings**. +1. Under **Payment methods**, select **Add new payment method**. +1. After the new payment method has been added, select **Edit**, then select **Default** to mark it as the default payment method. Reconciliation is retried automatically as soon as the payment method is updated. diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index e9a6f3720d2..9a232256de2 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -1,6 +1,7 @@ --- stage: Fulfillment group: Purchase +description: Billable users, renewal and upgrade info. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -284,10 +285,10 @@ You can renew your subscription starting from 15 days before your subscription e The [Customers Portal](https://customers.gitlab.com/customers/sign_in) is your tool for renewing and modifying your subscription. Before going ahead with renewal, -sign in and verify or update: +sign in and go to **Billing account settings**. Verify or update: -- The invoice contact details on the **Account details** page. -- The credit card on file on the **Payment Methods** page. +- The credit card on file under the **Payment methods** section. +- The invoice contact details in the **Company information** section. NOTE: Contact our [support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md deleted file mode 100644 index 0702e715d73..00000000000 --- a/doc/topics/authentication/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../administration/auth/index.md' -remove_date: '2023-12-26' ---- - -This document was moved to [another location](../../administration/auth/index.md). - -<!-- This redirect file can be deleted after <2023-12-26>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index 630123510a9..05535d233e0 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -253,7 +253,7 @@ For more information, see [Limit the environment scope of CI/CD variables](../.. ## Customize `.gitlab-ci.yml` Auto DevOps is highly customizable because the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) -is an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/index.md) file. +is an implementation of a [`.gitlab-ci.yml` file](../../ci/index.md#the-gitlab-ciyml-file). The template uses only features available to any implementation of `.gitlab-ci.yml`. To add custom behaviors to the CI/CD pipeline used by Auto DevOps: @@ -266,12 +266,12 @@ To add custom behaviors to the CI/CD pipeline used by Auto DevOps: ``` 1. Add your changes to the `.gitlab-ci.yml` file. Your changes are merged with the Auto DevOps template. For more information about -how `include` merges your changes, see [the `include` documentation](../../ci/yaml/index.md#include). + how `include` merges your changes, see [the `include` documentation](../../ci/yaml/index.md#include). To remove behaviors from the Auto DevOps pipeline: 1. Copy the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) -into your project. + into your project. 1. Edit your copy of the template as needed. ### Use individual components of Auto DevOps diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 0413ec32283..58203997698 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -174,7 +174,7 @@ When enabled, Auto DevOps attempts to run pipelines in every project. If the pipeline fails in a particular project, it disables itself. GitLab administrators can change this in the [Auto DevOps settings](../../administration/settings/continuous_integration.md#auto-devops). -If a [CI/CD configuration file](../../ci/yaml/index.md) is present, +If a [`.gitlab-ci.yml` file](../../ci/index.md#the-gitlab-ciyml-file) is present, it remains unchanged and Auto DevOps does not affect it. To disable Auto DevOps in the instance level, follow the same process diff --git a/doc/topics/build_your_application.md b/doc/topics/build_your_application.md index 430ef9ec57d..787e056a1c7 100644 --- a/doc/topics/build_your_application.md +++ b/doc/topics/build_your_application.md @@ -1,16 +1,16 @@ --- stage: none group: unassigned +description: Runners, jobs, pipelines, variables. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Use CI/CD to build your application **(FREE ALL)** -Add your source code to a repository, create merge requests to check in -code, and use CI/CD to generate your application. Include packages in your app and output it to a variety of environments. +Use CI/CD to generate your application. - [Getting started](../ci/index.md) -- [`.gitlab-ci.yml reference`](../ci/yaml/index.md) +- [CI/CD YAML syntax reference](../ci/yaml/index.md) - [Runners](https://docs.gitlab.com/runner/) - [Pipelines](../ci/pipelines/index.md) - [Jobs](../ci/jobs/index.md) diff --git a/doc/topics/cron/index.md b/doc/topics/cron/index.md index 057ae82e6ad..f25e01965fd 100644 --- a/doc/topics/cron/index.md +++ b/doc/topics/cron/index.md @@ -42,6 +42,7 @@ are valid: - Run once a month on the 2nd Monday: `0 0 * * 1#2` - Run once a year at midnight of 1 January: `0 0 1 1 *` - Run every other Sunday at 0900 hours: `0 9 * * sun%2` +- Run twice a month at 3 AM, on the 1st and 15th of the month: `0 3 1,15 * *` - This syntax is from the [fugit modulo extension](https://github.com/floraison/fugit#the-modulo-extension) For complete cron documentation, refer to the diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index c7b1832c8b8..17321591e87 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -1,10 +1,11 @@ --- stage: Create group: Source Code +description: Common commands and workflows. info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# Git **(FREE ALL)** +# Learn Git **(FREE ALL)** Git is a [free and open source](https://git-scm.com/about/free-and-open-source) distributed version control system. It handles projects of all sizes quickly and diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md index a397ec749d0..54a877bd974 100644 --- a/doc/topics/git/useful_git_commands.md +++ b/doc/topics/git/useful_git_commands.md @@ -1,101 +1,11 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" +redirect_to: 'index.md' +remove_date: '2024-03-19' --- -# Frequently used Git commands **(FREE ALL)** +This document was moved to [another location](index.md). -The following commands are frequently used. - -## Add another URL to a remote - -Add another URL to a remote, so both remotes get updated on each push: - -```shell -git remote set-url --add <remote_name> <remote_url> -``` - -## Refs and Log - -### Use reflog to show the log of reference changes to HEAD - -```shell -git reflog -``` - -### Check the Git history of a file - -The basic command to check the Git history of a file: - -```shell -git log <file> -``` - -If you get this error message: - -```plaintext -fatal: ambiguous argument <file_name>: unknown revision or path not in the working tree. -Use '--' to separate paths from revisions, like this: -``` - -Use this to check the Git history of the file: - -```shell -git log -- <file> -``` - -### Check the content of each change to a file - -```shell -gitk <file> -``` - -### Check the content of each change to a file, follows it past file renames - -```shell -gitk --follow <file> -``` - -## Rebasing - -### Rebase your branch onto the default - -The `-i` flag stands for 'interactive'. Replace `<default-branch>` with the name -of your [default branch](../../user/project/repository/branches/default.md): - -```shell -git rebase -i <default-branch> -``` - -### Continue the rebase if paused - -```shell -git rebase --continue -``` - -### Use `git rerere` - -To _reuse_ recorded solutions to the same problems when repeated: - -```shell -git rerere -``` - -To enable `rerere` functionality: - -```shell -git config --global rerere.enabled true -``` - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2024-03-19>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/manage_code.md b/doc/topics/manage_code.md index 5fbdbee7017..4cbd97f8898 100644 --- a/doc/topics/manage_code.md +++ b/doc/topics/manage_code.md @@ -1,12 +1,13 @@ --- stage: none group: unassigned +description: Repositories, merge requests, remote development. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Manage your code **(FREE ALL)** -Store your source files in a repository and create merge requests. Write, debug, and compile code hosted on GitLab. +Store your source files in a repository and create merge requests. Write, debug, and collaborate on code. - [Repositories](../user/project/repository/index.md) - [Merge requests](../user/project/merge_requests/index.md) diff --git a/doc/topics/offline/index.md b/doc/topics/offline/index.md index b29a06463ba..b463abc79d3 100644 --- a/doc/topics/offline/index.md +++ b/doc/topics/offline/index.md @@ -1,6 +1,7 @@ --- stage: Systems group: Distribution +description: Isolated installation. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index b6340d905d4..f5489d53ae0 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -258,8 +258,9 @@ Package metadata is stored in the following Google Cloud Provider (GCP) buckets: ```shell # To download the package metadata exports, an outbound connection to Google Cloud Storage bucket must be allowed. + # Skip v1 objects using -y "^v1\/" to only download v2 objects. v1 data is no longer used and deprecated since 16.3. mkdir -p "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" - gsutil -m rsync -r -d gs://$PKG_METADATA_BUCKET "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" + gsutil -m rsync -r -d -y "^v1\/" gs://$PKG_METADATA_BUCKET "$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/$DATA_DIR" # Alternatively, if the GitLab instance is not allowed to connect to the Google Cloud Storage bucket, the package metadata # exports can be downloaded using a machine with the allowed access, and then copied to the root of the GitLab Rails directory. @@ -293,7 +294,9 @@ PKG_METADATA_MANIFEST_OUTPUT_FILE="/tmp/package_metadata_${DATA_TYPE}_export_man PKG_METADATA_DOWNLOADS_OUTPUT_FILE="/tmp/package_metadata_${DATA_TYPE}_object_links.tsv" # Download the contents of the bucket -curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/$PKG_METADATA_BUCKET/o?maxResults=7500" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" +# Filter results using `prefix=v2` to only download v2 objects. v1 data is no longer used and deprecated since 16.3. +# Maximum number of objects returned by the API seems to be 5000 and there are currently (2023-12-21) 2650 objects for V2 dataset. +curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/$PKG_METADATA_BUCKET/o?prefix=v2%2f&maxResults=5000" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" # Parse the links and names for the bucket objects and output them into a tsv file jq -r '.items[] | [.name, .mediaLink] | @tsv' "$PKG_METADATA_MANIFEST_OUTPUT_FILE" > "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" @@ -331,7 +334,7 @@ To automatically update your local copy with the upstream changes, a cron job ca For License Scanning: ```plaintext -*/30 * * * * gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses +*/30 * * * * gsutil -m rsync -r -d -y "^v1\/" gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses ``` For Dependency Scanning: diff --git a/doc/topics/plan_and_track.md b/doc/topics/plan_and_track.md index 3712d73929c..61c359e63ba 100644 --- a/doc/topics/plan_and_track.md +++ b/doc/topics/plan_and_track.md @@ -1,6 +1,7 @@ --- stage: Plan group: Project Management +description: Epics, issues, milestones, labels. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md index 27c5cc50e5f..d46ae98d47c 100644 --- a/doc/topics/release_your_application.md +++ b/doc/topics/release_your_application.md @@ -1,6 +1,7 @@ --- stage: none group: unassigned +description: Environments, packages, review apps, GitLab Pages. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/set_up_organization.md b/doc/topics/set_up_organization.md index 84d7bb1add0..22a594b6117 100644 --- a/doc/topics/set_up_organization.md +++ b/doc/topics/set_up_organization.md @@ -1,6 +1,7 @@ --- stage: none group: unassigned +description: Users, groups, namespaces, SSH keys. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/tutorials/build_application.md b/doc/tutorials/build_application.md index 9c10a755292..ea899ee107e 100644 --- a/doc/tutorials/build_application.md +++ b/doc/tutorials/build_application.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: CI/CD fundamentals and examples. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/develop.md b/doc/tutorials/develop.md index bc0015b2d29..c893a4839d4 100644 --- a/doc/tutorials/develop.md +++ b/doc/tutorials/develop.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: Integrations with third-party services. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/gitlab_navigation.md b/doc/tutorials/gitlab_navigation.md index 738518e1aa7..b456b0ee18f 100644 --- a/doc/tutorials/gitlab_navigation.md +++ b/doc/tutorials/gitlab_navigation.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: Introduction to the product. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- @@ -11,7 +12,7 @@ and running quickly. | Topic | Description | Good for beginners | |-------|-------------|--------------------| -| [GitLab with Git Essentials](https://levelup.gitlab.com/courses/gitlab-with-git-essentials) | Learn the basics of Git and GitLab in this self-paced course. | **{star}** | +| [GitLab with Git Essentials](https://levelup.gitlab.com/courses/gitlab-with-git-essentials-s2) | Learn the basics of Git and GitLab in this self-paced course. | **{star}** | | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Use GitLab for DevOps](https://www.youtube.com/watch?v=7q9Y1Cv-ib0) (12m 34s) | Use GitLab through the entire DevOps lifecycle, from planning to monitoring. | **{star}** | | [Use the left sidebar to navigate GitLab](left_sidebar/index.md) | Start navigating the GitLab UI. | **{star}** | | [Use Markdown at GitLab](../user/markdown.md) | GitLab Flavored Markdown (GLFM) is used in many areas of GitLab, for example, in merge requests. | **{star}** | diff --git a/doc/tutorials/infrastructure.md b/doc/tutorials/infrastructure.md index 49cd0e95230..0097d06e316 100644 --- a/doc/tutorials/infrastructure.md +++ b/doc/tutorials/infrastructure.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: GitOps, Kubernetes deployments. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/learn_git.md b/doc/tutorials/learn_git.md index 2ebc1f3fd0e..a8333734982 100644 --- a/doc/tutorials/learn_git.md +++ b/doc/tutorials/learn_git.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: Git basics. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/make_first_git_commit/index.md b/doc/tutorials/make_first_git_commit/index.md index 1663fa08ad6..e80cd6770d0 100644 --- a/doc/tutorials/make_first_git_commit/index.md +++ b/doc/tutorials/make_first_git_commit/index.md @@ -93,7 +93,7 @@ To start, create a sample project in GitLab. Now you can clone the repository in your project. *Cloning* a repository means you're creating a copy on your computer, or wherever you want to store and work with the files. -1. On your project page, select **Clone**. Copy the URL for **Clone with SSH**. +1. On your project's overview page, in the upper-right corner, select **Code**, then copy the URL for **Clone with SSH**.  diff --git a/doc/tutorials/plan_and_track.md b/doc/tutorials/plan_and_track.md index b3ec1fb4755..f03aff933f6 100644 --- a/doc/tutorials/plan_and_track.md +++ b/doc/tutorials/plan_and_track.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: Planning, agile, issue boards. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/protected_workflow/index.md b/doc/tutorials/protected_workflow/index.md index 8e9ed3e952a..e9e3739d836 100644 --- a/doc/tutorials/protected_workflow/index.md +++ b/doc/tutorials/protected_workflow/index.md @@ -77,8 +77,8 @@ Next, add the subgroup as a member of the `engineering` group: 1. On the left sidebar, select **Search or go to** and search for `engineering`. Select the group named `Engineering`. -1. On the left sidebar, select **Manage > Members**. -1. On the top right, select **Invite a group**. +1. Select **Manage > Members**. +1. In the upper right, select **Invite a group**. 1. For **Select a group to invite**, select `Engineering / Managers`. 1. When adding the subgroups select the role **Maintainer**. This configures the highest role a member of the subgroup can inherit when accessing the `engineering` group and its projects. @@ -270,7 +270,7 @@ Your rules are now in place, even though no `1.*` branches exist yet: Now that all branch protections in place, you're ready to create your 1.0.0 release branch: 1. On the left sidebar, select **Code > Branches**. -1. On the top right, select **New branch**. Name it `1.0.0`. +1. In the upper-right corner, select **New branch**. Name it `1.0.0`. 1. Select **Create branch**. The branch protections are now visible in the UI: diff --git a/doc/tutorials/secure_application.md b/doc/tutorials/secure_application.md index 4868ec53ed3..ac139a88cf1 100644 --- a/doc/tutorials/secure_application.md +++ b/doc/tutorials/secure_application.md @@ -1,6 +1,7 @@ --- stage: none group: Tutorials +description: Dependency and compliance scanning. info: For assistance with this tutorials page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- diff --git a/doc/tutorials/update_commit_messages/index.md b/doc/tutorials/update_commit_messages/index.md index 0228b33e3de..36106dd4f98 100644 --- a/doc/tutorials/update_commit_messages/index.md +++ b/doc/tutorials/update_commit_messages/index.md @@ -53,7 +53,7 @@ disabled to authenticate from the CLI. Alternatively, you can [use an SSH key to The first step is to get a clone of the repository on your local machine: -1. In GitLab, on your project's overview page, on the top right, select **Clone**. +1. In GitLab, on your project's overview page, in the upper-right corner, select **Code**. 1. In the dropdown list, copy the URL for your repository by selecting **{copy-to-clipboard}** next to: - **Clone with HTTPS** if your GitLab account uses basic username and password authentication. - **Clone with SSH** if you use SSH to authenticate with GitLab. diff --git a/doc/update/background_migrations.md b/doc/update/background_migrations.md index 8f7e4cc7b1d..f1a3f7e7839 100644 --- a/doc/update/background_migrations.md +++ b/doc/update/background_migrations.md @@ -56,7 +56,7 @@ as 'finished', but it is 'active': ``` If you get this error, -[review the options](#database-migrations-failing-because-of-batched-background-migration-not-finished) for +[review the options](background_migrations_troubleshooting.md#database-migrations-failing-because-of-batched-background-migration-not-finished) for how to complete the batched background migrations needed for the GitLab upgrade. #### From the GitLab UI @@ -419,208 +419,3 @@ sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::Ba ``` ::EndTabs - -## Troubleshooting - -<!-- Linked from lib/gitlab/database/migrations/batched_background_migration_helpers.rb --> - -### Database migrations failing because of batched background migration not finished - -When updating to GitLab version 14.2 or later, database migrations might fail with a message like: - -```plaintext -StandardError: An error has occurred, all later migrations canceled: - -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", - :table_name=>"push_event_payloads", - :column_name=>"event_id", - :job_arguments=>[["event_id"], - ["event_id_convert_to_bigint"]] - } -``` - -First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/versions/gitlab_14_changes.md#1420). -If you have, you can [manually finish the batched background migration](#finish-a-failed-migration-manually)). -If you haven't, choose one of the following methods: - -1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required -versions before updating to 14.2+. -1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current -version and manually ensuring that the batched migrations complete successfully. - -#### Roll back and follow the required upgrade path - -1. [Rollback and restore the previously installed version](../administration/backup_restore/index.md) -1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ -1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations and -make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, -you can [manually finish them](#finish-a-failed-migration-manually). - -#### Roll forward and finish the migrations on the upgraded version - -##### For a deployment with downtime - -To run all the batched background migrations, it can take a significant amount of time -depending on the size of your GitLab installation. - -1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations in the -database, and [manually run them](#finish-a-failed-migration-manually) with the appropriate -arguments until the status query returns no rows. -1. When the status of all of all them is marked as complete, re-run migrations for your installation. -1. [Complete the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade: - - ```plaintext - sudo gitlab-rake db:migrate - ``` - -1. Run a reconfigure: - - ```plaintext - sudo gitlab-ctl reconfigure - ``` - -1. Finish the upgrade for your installation. - -##### For a no-downtime deployment - -As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded -version and wait for the batched background migrations to finish. - -1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migration from -the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, -or [manually finish it](#finish-a-failed-migration-manually). -1. Re-run migrations for your installation, so the remaining post-deployment migrations finish. - -### The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails - -In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job -may fail to complete. When retried, a `500 Server Error` is returned. This issue was -[resolved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82387) in GitLab 14.9. - -To resolve this issue, [upgrade GitLab](../update/index.md) from 14.8 to 14.9. -You can ignore the failed batch migration until after you update to GitLab 14.9. - -### Background migrations remain in the Sidekiq queue - -WARNING: -The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates. - -Run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section. - -```shell -# For Linux package installations: -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' - -# For self-compiled installations: -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -``` - -It is safe to re-execute the following commands, especially if you have 1000+ pending jobs which would likely overflow your runtime memory. - -::Tabs - -:::TabTitle Linux package (Omnibus) - -```shell -# Start the rails console -sudo gitlab-rails c - -# Execute the following in the rails console -scheduled_queue = Sidekiq::ScheduledSet.new -pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq -pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } -``` - -:::TabTitle Self-compiled (source) - -```shell -# Start the rails console -sudo -u git -H bundle exec rails RAILS_ENV=production - -# Execute the following in the rails console -scheduled_queue = Sidekiq::ScheduledSet.new -pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq -pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } -``` - -::EndTabs - -### Background migrations stuck in 'pending' state - -WARNING: -The following operations can disrupt your GitLab performance. They run a number -of Sidekiq jobs that perform various database or file updates. - -- GitLab 14.2 introduced an issue where a background migration named - `BackfillDraftStatusOnMergeRequests` can be permanently stuck in a - **pending** state across upgrades when the instance lacks records that match - the migration's target. To clean up this stuck migration, see the - [14.2.0 version-specific instructions](versions/gitlab_14_changes.md#1420). -- GitLab 14.4 introduced an issue where a background migration named - `PopulateTopicsTotalProjectsCountCache` can be permanently stuck in a - **pending** state across upgrades when the instance lacks records that match - the migration's target. To clean up this stuck migration, see the - [14.4.0 version-specific instructions](versions/gitlab_14_changes.md#1440). -- GitLab 14.5 introduced an issue where a background migration named - `UpdateVulnerabilityOccurrencesLocation` can be permanently stuck in a - **pending** state across upgrades when the instance lacks records that match - the migration's target. To clean up this stuck migration, see the - [14.5.0 version-specific instructions](versions/gitlab_14_changes.md#1450). -- GitLab 14.8 introduced an issue where a background migration named - `PopulateTopicsNonPrivateProjectsCount` can be permanently stuck in a - **pending** state across upgrades. To clean up this stuck migration, see the - [14.8.0 version-specific instructions](versions/gitlab_14_changes.md#1480). -- GitLab 14.9 introduced an issue where a background migration named - `ResetDuplicateCiRunnersTokenValuesOnProjects` can be permanently stuck in a - **pending** state across upgrades when the instance lacks records that match - the migration's target. To clean up this stuck migration, see the - [14.9.0 version-specific instructions](versions/gitlab_14_changes.md#1490). - -For other background migrations stuck in pending, run the following check. If -it returns non-zero and the count does not decrease over time, follow the rest -of the steps in this section. - -```shell -# For Linux package installations: -sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' - -# For self-compiled installations: -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' -``` - -It is safe to re-attempt these migrations to clear them out from a pending status: - -::Tabs - -:::TabTitle Linux package (Omnibus) - -```shell -# Start the rails console -sudo gitlab-rails c - -# Execute the following in the rails console -Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| - puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}" - result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments) - puts "Result: #{result}" -end -``` - -:::TabTitle Self-compiled (source) - -```shell -# Start the rails console -sudo -u git -H bundle exec rails RAILS_ENV=production - -# Execute the following in the rails console -Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| - puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}" - result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments) - puts "Result: #{result}" -end -``` - -::EndTabs diff --git a/doc/update/background_migrations_troubleshooting.md b/doc/update/background_migrations_troubleshooting.md new file mode 100644 index 00000000000..f4ea9c2a556 --- /dev/null +++ b/doc/update/background_migrations_troubleshooting.md @@ -0,0 +1,210 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting **(FREE SELF)** + +<!-- Linked from lib/gitlab/database/migrations/batched_background_migration_helpers.rb --> + +## Database migrations failing because of batched background migration not finished + +When updating to GitLab version 14.2 or later, database migrations might fail with a message like: + +```plaintext +StandardError: An error has occurred, all later migrations canceled: + +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': + {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", + :table_name=>"push_event_payloads", + :column_name=>"event_id", + :job_arguments=>[["event_id"], + ["event_id_convert_to_bigint"]] + } +``` + +First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/versions/gitlab_14_changes.md#1420). +If you have, you can [manually finish the batched background migration](background_migrations.md#finish-a-failed-migration-manually)). +If you haven't, choose one of the following methods: + +1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required +versions before updating to 14.2+. +1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current +version and manually ensuring that the batched migrations complete successfully. + +### Roll back and follow the required upgrade path + +1. [Rollback and restore the previously installed version](../administration/backup_restore/index.md) +1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ +1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migrations and +make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, +you can [manually finish them](background_migrations.md#finish-a-failed-migration-manually). + +### Roll forward and finish the migrations on the upgraded version + +#### For a deployment with downtime + +To run all the batched background migrations, it can take a significant amount of time +depending on the size of your GitLab installation. + +1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migrations in the +database, and [manually run them](background_migrations.md#finish-a-failed-migration-manually) with the appropriate +arguments until the status query returns no rows. +1. When the status of all of all them is marked as complete, re-run migrations for your installation. +1. [Complete the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade: + + ```plaintext + sudo gitlab-rake db:migrate + ``` + +1. Run a reconfigure: + + ```plaintext + sudo gitlab-ctl reconfigure + ``` + +1. Finish the upgrade for your installation. + +#### For a no-downtime deployment + +As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded +version and wait for the batched background migrations to finish. + +1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migration from +the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, +or [manually finish it](background_migrations.md#finish-a-failed-migration-manually). +1. Re-run migrations for your installation, so the remaining post-deployment migrations finish. + +## The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails + +In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job +may fail to complete. When retried, a `500 Server Error` is returned. This issue was +[resolved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82387) in GitLab 14.9. + +To resolve this issue, [upgrade GitLab](../update/index.md) from 14.8 to 14.9. +You can ignore the failed batch migration until after you update to GitLab 14.9. + +## Background migrations remain in the Sidekiq queue + +WARNING: +The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates. + +Run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section. + +```shell +# For Linux package installations: +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' + +# For self-compiled installations: +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +It is safe to re-execute the following commands, especially if you have 1000+ pending jobs which would likely overflow your runtime memory. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +# Start the rails console +sudo gitlab-rails c + +# Execute the following in the rails console +scheduled_queue = Sidekiq::ScheduledSet.new +pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq +pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } +``` + +:::TabTitle Self-compiled (source) + +```shell +# Start the rails console +sudo -u git -H bundle exec rails RAILS_ENV=production + +# Execute the following in the rails console +scheduled_queue = Sidekiq::ScheduledSet.new +pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq +pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } +``` + +::EndTabs + +## Background migrations stuck in 'pending' state + +WARNING: +The following operations can disrupt your GitLab performance. They run a number +of Sidekiq jobs that perform various database or file updates. + +- GitLab 14.2 introduced an issue where a background migration named + `BackfillDraftStatusOnMergeRequests` can be permanently stuck in a + **pending** state across upgrades when the instance lacks records that match + the migration's target. To clean up this stuck migration, see the + [14.2.0 version-specific instructions](versions/gitlab_14_changes.md#1420). +- GitLab 14.4 introduced an issue where a background migration named + `PopulateTopicsTotalProjectsCountCache` can be permanently stuck in a + **pending** state across upgrades when the instance lacks records that match + the migration's target. To clean up this stuck migration, see the + [14.4.0 version-specific instructions](versions/gitlab_14_changes.md#1440). +- GitLab 14.5 introduced an issue where a background migration named + `UpdateVulnerabilityOccurrencesLocation` can be permanently stuck in a + **pending** state across upgrades when the instance lacks records that match + the migration's target. To clean up this stuck migration, see the + [14.5.0 version-specific instructions](versions/gitlab_14_changes.md#1450). +- GitLab 14.8 introduced an issue where a background migration named + `PopulateTopicsNonPrivateProjectsCount` can be permanently stuck in a + **pending** state across upgrades. To clean up this stuck migration, see the + [14.8.0 version-specific instructions](versions/gitlab_14_changes.md#1480). +- GitLab 14.9 introduced an issue where a background migration named + `ResetDuplicateCiRunnersTokenValuesOnProjects` can be permanently stuck in a + **pending** state across upgrades when the instance lacks records that match + the migration's target. To clean up this stuck migration, see the + [14.9.0 version-specific instructions](versions/gitlab_14_changes.md#1490). + +For other background migrations stuck in pending, run the following check. If +it returns non-zero and the count does not decrease over time, follow the rest +of the steps in this section. + +```shell +# For Linux package installations: +sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' + +# For self-compiled installations: +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' +``` + +It is safe to re-attempt these migrations to clear them out from a pending status: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +# Start the rails console +sudo gitlab-rails c + +# Execute the following in the rails console +Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| + puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}" + result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments) + puts "Result: #{result}" +end +``` + +:::TabTitle Self-compiled (source) + +```shell +# Start the rails console +sudo -u git -H bundle exec rails RAILS_ENV=production + +# Execute the following in the rails console +Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| + puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}" + result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments) + puts "Result: #{result}" +end +``` + +::EndTabs diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index cba85b1acd9..a7eb061fa47 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -52,6 +52,23 @@ For deprecation reviewers (Technical Writers only): <div class="deprecation breaking-change" data-milestone="18.0"> +### Atlassian Crowd OmniAuth provider + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.3</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/369117). +</div> + +The `omniauth_crowd` gem that provides GitLab with the Atlassian Crowd OmniAuth provider will be removed in our +next major release, GitLab 18.0. This gem sees very little use and its +[lack of compatibility](https://github.com/robdimarco/omniauth_crowd/issues/37) with OmniAuth 2.0 is +[blocking our upgrade](https://gitlab.com/gitlab-org/gitlab/-/issues/30073). + +</div> + +<div class="deprecation breaking-change" data-milestone="18.0"> + ### GitLab Runner registration token in Runner Operator <div class="deprecation-notes"> @@ -124,6 +141,24 @@ Before upgrading to GitLab 18.0, please ensure you have [migrated](https://docs. <div class="deprecation breaking-change" data-milestone="18.0"> +### Slack notifications integration + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">15.9</span> +- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/435909). +</div> + +As we're consolidating all Slack capabilities into the +GitLab for Slack app, we've deprecated the Slack notifications +integration. +Use the GitLab for Slack app to manage notifications +to your Slack workspace. + +</div> + +<div class="deprecation breaking-change" data-milestone="18.0"> + ### Support for REST API endpoints that reset runner registration tokens <div class="deprecation-notes"> @@ -176,34 +211,31 @@ From GitLab 18.0 and later, the methods to register runners introduced by the ne <div class="deprecation breaking-change" data-milestone="17.0"> -### Atlassian Crowd OmniAuth provider +### Auto DevOps support for Herokuish is deprecated <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.3</span> +- Announced in GitLab <span class="milestone">15.8</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/369117). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/211643). </div> -The `omniauth_crowd` gem that provides GitLab with the Atlassian Crowd OmniAuth provider will be removed in our -next major release, GitLab 16.0. This gem sees very little use and its -[lack of compatibility](https://github.com/robdimarco/omniauth_crowd/issues/37) with OmniAuth 2.0 is -[blocking our upgrade](https://gitlab.com/gitlab-org/gitlab/-/issues/30073). +Auto DevOps support for Herokuish is deprecated in favor of [Cloud Native Buildpacks](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-build-using-cloud-native-buildpacks). You should [migrate your builds from Herokuish to Cloud Native Buildpacks](https://docs.gitlab.com/ee/topics/autodevops/stages.html#moving-from-herokuish-to-cloud-native-buildpacks). From GitLab 14.0, Auto Build uses Cloud Native Buildpacks by default. + +Because Cloud Native Buildpacks do not support automatic testing, the Auto Test feature of Auto DevOps is also deprecated. </div> <div class="deprecation breaking-change" data-milestone="17.0"> -### Auto DevOps support for Herokuish is deprecated +### Block usage of ref and sha together in `GET /projects/:id/ci/lint` <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.8</span> +- Announced in GitLab <span class="milestone">16.8</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/211643). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/430322). </div> -Auto DevOps support for Herokuish is deprecated in favor of [Cloud Native Buildpacks](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-build-using-cloud-native-buildpacks). You should [migrate your builds from Herokuish to Cloud Native Buildpacks](https://docs.gitlab.com/ee/topics/autodevops/stages.html#moving-from-herokuish-to-cloud-native-buildpacks). From GitLab 14.0, Auto Build uses Cloud Native Buildpacks by default. - -Because Cloud Native Buildpacks do not support automatic testing, the Auto Test feature of Auto DevOps is also deprecated. +Due to a problem with ambiguity, we've deprecated the use of both `ref` and `sha` in the same API call to `GET /projects/:id/ci/lint`. Make sure your API calls to this endpoint use only `ref` or `sha`, but not both. In GitLab 17.0, using them in the same call will no longer be possible to ensure the correct ref or SHA is linted. </div> @@ -333,6 +365,22 @@ To help avoid being impacted by this breaking change, create new access tokens w </div> +<div class="deprecation breaking-change" data-milestone="17.0"> + +### Dependency Scanning support for sbt 1.0.X + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/415835). +</div> + +Supporting very old versions of sbt is preventing us from improving our support for additional use cases with this package manager without increasing our maintenance cost. + +Version 1.1.0 of sbt was released 6 years ago, and users are advised to upgrade from 1.0.x as Dependency Scanning will no longer work. + +</div> + <div class="deprecation " data-milestone="17.0"> ### Deprecate GraphQL fields related to the temporary storage increase @@ -579,6 +627,20 @@ are deprecated and will be removed from the GraphQL API. For installation instru <div class="deprecation breaking-change" data-milestone="17.0"> +### GitLab Runner provenance metadata SLSA v0.2 statement + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/36869). +</div> + +Runners generate provenance metadata and currently defaults to generating statements that adhere to SLSA v0.2. Because SLSA v1.0 has been released and is now supported by GitLab, the v0.2 statement is now deprecated and removal is planned in GitLab 17.0. The SLSA v1.0 statement is planned to become the new default statement format in GitLab 17.0. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### GraphQL deprecation of `dependencyProxyTotalSizeInBytes` field <div class="deprecation-notes"> @@ -784,6 +846,41 @@ The table below lists the deprecated metrics and their respective replacements. <div class="deprecation breaking-change" data-milestone="17.0"> +### License List is deprecated + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/436100). +</div> + +Today in GitLab you can see a list of all of the licenses your project and the components that +use that license on the License List. As of 16.8, the License List +is deprecated and scheduled to be removed in 17.0 as a breaking change. +With the release of the [Group Dependency List](https://docs.gitlab.com/ee/user/application_security/dependency_list/) +and the ability to filter by license on the project and group Dependency List, you can now +access all of the licenses your project or group is using on the Dependency List. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + +### License Scanning support for sbt 1.0.X + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/437591). +</div> + +GitLab 17.0 removes License Scanning support for sbt 1.0.x. + +Users are advised to upgrade from sbt 1.0.x. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### List repository directories Rake task <div class="deprecation-notes"> @@ -1083,20 +1180,20 @@ automatically from GitLab 16.0 onwards. <div class="deprecation breaking-change" data-milestone="17.0"> -### Slack notifications integration +### Support for setting custom schema for backup is deprecated <div class="deprecation-notes"> -- Announced in GitLab <span class="milestone">15.9</span> +- Announced in GitLab <span class="milestone">16.8</span> - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) -- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/372411). +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/435210). </div> -As we're consolidating all Slack capabilities into the -GitLab for Slack app, we're [deprecating the Slack notifications -integration](https://gitlab.com/gitlab-org/gitlab/-/issues/372411). -GitLab.com users can now use the GitLab for Slack app to manage notifications -to their Slack workspace. For self-managed users of the Slack notifications integration, -we'll be introducing support in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/1211). +You could configure GitLab to use a custom schema for backup, by setting +`gitlab_rails['backup_pg_schema'] = '<schema_name>'` in `/etc/gitlab/gitlab.rb` for Linux package installations, +or by editing `config/gitlab.yml` for self-compiled installations. + +While the configuration setting was available, it had no effect and did not serve the purpose it was intended. +This configuration setting will be removed in GitLab 17.0. </div> @@ -1259,6 +1356,34 @@ removed in 17.0. <div class="deprecation breaking-change" data-milestone="17.0"> +### `after_script` keyword will run for cancelled jobs + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/437789). +</div> + +The [`after_script`](https://docs.gitlab.com/ee/ci/yaml/#after_script) CI/CD keyword is used to run additional commands after the main `script` section of a job. This is often used for cleaning up environments or other resources that were used by the job. For many users, the fact that the `after_script` commands do not run if a job is cancelled was unexpected and undesired. In 17.0, the keyword will be updated to also run commands after job cancellation. Make sure that your CI/CD configuration that uses the `after_script` keyword is able to handle running for cancelled jobs as well. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + +### `metric` filter and `value` field for DORA API + +<div class="deprecation-notes"> +- Announced in GitLab <span class="milestone">16.8</span> +- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change)) +- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393172). +</div> + +Multiple DORA metrics can now be queried simultaneously using a new metrics field. The `metric` filter and `value` field for Graphql DORA API will be removed in GitLab 17.0. + +</div> + +<div class="deprecation breaking-change" data-milestone="17.0"> + ### `postgres_exporter['per_table_stats']` configuration setting <div class="deprecation-notes"> diff --git a/doc/update/index.md b/doc/update/index.md index c0a6b64a1ac..9192d409d55 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -1,10 +1,11 @@ --- stage: Systems group: Distribution +description: Latest version instructions. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Upgrading GitLab **(FREE SELF)** +# Upgrade GitLab **(FREE SELF)** Upgrading GitLab is a relatively straightforward process, but the complexity can increase based on the installation method you have used, how old your @@ -88,10 +89,10 @@ As for the artifacts, the GitLab Runner attempts to upload them three times, aft To address the above two scenarios, it is advised to do the following prior to upgrading: 1. Plan your maintenance. -1. Pause your runners or block new jobs from starting by adding following to your `/etc/gitlab/gitlab.rb`: +1. Pause your runners, or block new jobs from starting by adding the following to your `/etc/gitlab/gitlab.rb`: ```ruby - nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + nginx['custom_gitlab_server_config'] = "location ^~ /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" ``` And reconfigure GitLab with: @@ -192,12 +193,13 @@ When upgrading: - GitLab 15: [`15.0.5`](versions/gitlab_15_changes.md#1500) > [`15.1.6`](versions/gitlab_15_changes.md#1510) (for GitLab instances with multiple web nodes) > [`15.4.6`](versions/gitlab_15_changes.md#1540) > [`15.11.13`](versions/gitlab_15_changes.md#15110). - - GitLab 16: [`16.0.x`](versions/gitlab_16_changes.md#1600) (only + - GitLab 16: [`16.0.8`](versions/gitlab_16_changes.md#1600) (only instances with [lots of users](versions/gitlab_16_changes.md#long-running-user-type-data-change) or [large pipeline variables history](versions/gitlab_16_changes.md#1610)) > - [`16.1`](versions/gitlab_16_changes.md#1610) (instances with NPM packages in their package registry) > - [`16.2.x`](versions/gitlab_16_changes.md#1620) (only instances with [large pipeline variables history](versions/gitlab_16_changes.md#1630)) > - [`16.3`](versions/gitlab_16_changes.md#1630) > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases). + [`16.1.6`](versions/gitlab_16_changes.md#1610) (instances with NPM packages in their package registry) > + [`16.2.9`](versions/gitlab_16_changes.md#1620) (only instances with [large pipeline variables history](versions/gitlab_16_changes.md#1630)) > + [`16.3.7`](versions/gitlab_16_changes.md#1630) > [`16.7.z`](versions/gitlab_16_changes.md#1670) + > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases). 1. Check for [required upgrade stops](#required-upgrade-stops). 1. Consult the [version-specific upgrade instructions](#version-specific-upgrading-instructions). diff --git a/doc/update/package/convert_to_ee.md b/doc/update/package/convert_to_ee.md index 5a76c85e915..ae89891d4f4 100644 --- a/doc/update/package/convert_to_ee.md +++ b/doc/update/package/convert_to_ee.md @@ -17,7 +17,7 @@ you are upgrading the same version (for example, CE 12.1 to EE 12.1), which is * WARNING: When updating to EE from CE, avoid reverting back to CE if you plan to go to EE again in the future. Reverting back to CE can cause -[database issues](index.md#500-error-when-accessing-project--settings--repository) +[database issues](package_troubleshooting.md#500-error-when-accessing-project--settings--repository) that may require Support intervention. The steps can be summed up to: diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 93967a30660..662590e7f78 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -39,9 +39,9 @@ GitLab package. Upgrading versions might need some manual intervention. For more information, check the version your are upgrading to: -- [GitLab 16](https://docs.gitlab.com/omnibus/update/gitlab_16_changes.html) -- [GitLab 15](https://docs.gitlab.com/omnibus/update/gitlab_15_changes.html) -- [GitLab 14](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html) +- [GitLab 16](../versions/gitlab_16_changes.md) +- [GitLab 15](../versions/gitlab_15_changes.md) +- [GitLab 14](../versions/gitlab_14_changes.md) ### Earlier GitLab versions @@ -93,10 +93,10 @@ To upgrade to the latest GitLab version: # Ubuntu/Debian sudo apt update && sudo apt install gitlab-ee -# RHEL/CentOS 6 and 7 +# RHEL/CentOS 7 and Amazon Linux 2 sudo yum install gitlab-ee -# RHEL/CentOS 8 +# RHEL/Almalinux 8/9 and Amazon Linux 2023 sudo dnf install gitlab-ee # SUSE @@ -124,10 +124,10 @@ or upgrade command: # Ubuntu/Debian sudo apt-cache madison gitlab-ee - # RHEL/CentOS 6 and 7 + # RHEL/CentOS 7 and Amazon Linux 2 yum --showduplicates list gitlab-ee - # RHEL/CentOS 8 + # RHEL/Almalinux 8/9 and Amazon Linux 2023 dnf --showduplicates list gitlab-ee # SUSE @@ -143,10 +143,10 @@ or upgrade command: # Ubuntu/Debian sudo apt install gitlab-ee=<version> - # RHEL/CentOS 6 and 7 + # RHEL/CentOS 7 and Amazon Linux 2 yum install gitlab-ee-<version> - # RHEL/CentOS 8 + # RHEL/Almalinux 8/9 and Amazon Linux 2023 dnf install gitlab-ee-<version> # SUSE @@ -184,10 +184,10 @@ To download and install GitLab: # Debian/Ubuntu dpkg -i <package_name> - # RHEL/CentOS 6 and 7 + # RHEL/CentOS 7 and Amazon Linux 2 rpm -Uvh <package_name> - # RHEL/CentOS 8 + # RHEL/Almalinux 8/9 and Amazon Linux 2023 dnf install <package_name> # SUSE @@ -205,288 +205,4 @@ see how to [upgrade to a later version](../../administration/docs_self_host.md#u ## Troubleshooting -### Get the status of a GitLab installation - -```shell -sudo gitlab-ctl status -sudo gitlab-rake gitlab:check SANITIZE=true -``` - -- Information on using `gitlab-ctl` to perform [maintenance tasks](https://docs.gitlab.com/omnibus/maintenance/index.html). -- Information on using `gitlab-rake` to [check the configuration](../../administration/raketasks/maintenance.md#check-gitlab-configuration). - -### RPM 'package is already installed' error - -If you are using RPM and you are upgrading from GitLab Community Edition to GitLab Enterprise Edition you may get an error like this: - -```shell -package gitlab-7.5.2_omnibus.5.2.1.ci-1.el7.x86_64 (which is newer than gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64) is already installed -``` - -You can override this version check with the `--oldpackage` option: - -```shell -sudo rpm -Uvh --oldpackage gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64.rpm -``` - -### Package obsoleted by installed package - -CE and EE packages are marked as obsoleting and replacing each other so that both aren't installed and running at the same time. - -If you are using local RPM files to switch from CE to EE or vice versa, use `rpm` for installing the package rather than `yum`. If you try to use yum, then you may get an error like this: - -```plaintext -Cannot install package gitlab-ee-11.8.3-ee.0.el6.x86_64. It is obsoleted by installed package gitlab-ce-11.8.3-ce.0.el6.x86_64 -``` - -To avoid this issue, either: - -- Use the same instructions provided in the - [Upgrade using a manually-downloaded package](#upgrade-using-a-manually-downloaded-package) section. -- Temporarily disable this checking in yum by adding `--setopt=obsoletes=0` to the options given to the command. - -### 500 error when accessing Project > Settings > Repository - -This error occurs when GitLab is converted from CE > EE > CE, and then back to EE. -When viewing a project's repository settings, you can view this error in the logs: - -```shell -Processing by Projects::Settings::RepositoryController#show as HTML - Parameters: {"namespace_id"=>"<namespace_id>", "project_id"=>"<project_id>"} -Completed 500 Internal Server Error in 62ms (ActiveRecord: 4.7ms | Elasticsearch: 0.0ms | Allocations: 14583) - -NoMethodError (undefined method `commit_message_negative_regex' for #<PushRule:0x00007fbddf4229b8> -Did you mean? commit_message_regex_change): -``` - -This error is caused by an EE feature being added to a CE instance on the initial move to EE. -After the instance is moved back to CE and then is upgraded to EE again, the -`push_rules` table already exists in the database. Therefore, a migration is -unable to add the `commit_message_regex_change` column. - -This results in the [backport migration of EE tables](https://gitlab.com/gitlab-org/gitlab/-/blob/cf00e431024018ddd82158f8a9210f113d0f4dbc/db/migrate/20190402150158_backport_enterprise_schema.rb#L1619) not working correctly. -The backport migration assumes that certain tables in the database do not exist when running CE. - -To fix this issue: - -1. Start a database console: - - In GitLab 14.2 and later: - - ```shell - sudo gitlab-rails dbconsole --database main - ``` - - In GitLab 14.1 and earlier: - - ```shell - sudo gitlab-rails dbconsole - ``` - -1. Manually add the missing `commit_message_negative_regex` column: - - ```sql - ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; - - # Exit psql - \q - ``` - -1. Restart GitLab: - - ```shell - sudo gitlab-ctl restart - ``` - -### Error `Failed to connect to the internal GitLab API` on a separate GitLab Pages server - -See [GitLab Pages administration troubleshooting](../../administration/pages/troubleshooting.md#failed-to-connect-to-the-internal-gitlab-api). - -### Error `An error occurred during the signature verification` when running `apt-get update` - -To update the GPG key of the GitLab packages server run: - -```shell -curl --silent "https://packages.gitlab.com/gpg.key" | apt-key add - -apt-get update -``` - -### `Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] [..] Command timed out after 3600s` - -If database schema and data changes (database migrations) must take more than one hour to run, -upgrades fail with a `timed out` error: - -```plaintext -FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) -had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database] -(/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) -had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: -``` - -To fix this error: - -1. Run the remaining database migrations: - - ```shell - sudo gitlab-rake db:migrate - ``` - - This command may take a very long time to complete. Use `screen` or some other mechanism to ensure - the program is not interrupted if your SSH session drops. - -1. Complete the upgrade: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Hot reload `puma` and `sidekiq` services: - - ```shell - sudo gitlab-ctl hup puma - sudo gitlab-ctl restart sidekiq - ``` - -### Missing asset files - -Following an upgrade, GitLab might not be correctly serving up assets such as images, JavaScript, and style sheets. -It might be generating 500 errors, or the web UI may be failing to render properly. - -In a scaled out GitLab environment, if one web server behind the load balancer is demonstrating -this issue, the problem occurs intermittently. - -The [Rake task to recompile](../../administration/raketasks/maintenance.md#precompile-the-assets) the -assets doesn't apply to an Omnibus installation which serves -pre-compiled assets from `/opt/gitlab/embedded/service/gitlab-rails/public/assets`. - -Potential causes and fixes: - -- [Ensure no old processes are running](#old-processes). -- [Remove duplicate sprockets files](#duplicate-sprockets-files) -- [The installation is incomplete](#incomplete-installation) -- [NGINX Gzip support is disabled](#nginx-gzip-support) - -#### Old processes - -The most likely cause is that an old Puma process is running, instructing clients -to request asset files from a previous release of GitLab. As the files no longer exist, -HTTP 404 errors are returned. - -A reboot is the best way to ensure these old Puma processes are no longer running. - -Alternatively: - -1. Stop Puma: - - ```shell - gitlab-ctl stop puma - ``` - -1. Check for any remaining Puma processes, and kill them: - - ```shell - ps -ef | egrep 'puma[: ]' - kill <processid> - ``` - -1. Verify with `ps` that the Puma processes have stopped running. - -1. Start Puma - - ```shell - gitlab-ctl start puma - ``` - -#### Duplicate sprockets files - -The compiled asset files have unique file names in each release. The sprockets files -provide a mapping from the filenames in the application code to the unique filenames. - -```plaintext -/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest*.json -``` - -Make sure there's only one sprockets file. [Rails uses the first one](https://github.com/rails/sprockets-rails/blob/118ce60b1ffeb7a85640661b014cd2ee3c4e3e56/lib/sprockets/railtie.rb#L201). - -A check for duplicate sprockets files runs during Omnibus GitLab upgrades: - -```plaintext -GitLab discovered stale file(s) from the previous install that need to be cleaned up. -The following files need to be removed: - -/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-e16fdb7dd73cfdd64ed9c2cc0e35718a.json -``` - -Options for resolving this include: - -- If you have the output from the package upgrade, remove the specified files. Then restart Puma: - - ```shell - gitlab-ctl restart puma - ``` - -- If you don't have the message, perform a reinstall - (see [incomplete installation](#incomplete-installation) below for more details) - to generate it again. - -- Remove all the sprockets files and then follow the instructions for an [incomplete installation](#incomplete-installation). - -#### Incomplete installation - -An incomplete installation could be the cause of this issue. - -Verify the package to determine if this is the problem: - -- For Debian distributions: - - ```shell - apt-get install debsums - debsums -c gitlab-ee - ``` - -- For Red Hat/SUSE (RPM) distributions: - - ```shell - rpm -V gitlab-ee - ``` - -To reinstall the package to fix an incomplete installation: - -1. Check the installed version - - - For Debian distributions: - - ```shell - apt --installed list gitlab-ee - ``` - - - For Red Hat/SUSE (RPM) distributions: - - ```shell - rpm -qa gitlab-ee - ``` - -1. Reinstall the package, specifying the installed version. For example 14.4.0 Enterprise Edition: - - - For Debian distributions: - - ```shell - apt-get install --reinstall gitlab-ee=14.4.0-ee.0 - ``` - - - For Red Hat/SUSE (RPM) distributions: - - ```shell - yum reinstall gitlab-ee-14.4.0 - ``` - -#### NGINX Gzip support - -Check whether `nginx['gzip_enabled']` has been disabled: - -```shell -grep gzip /etc/gitlab/gitlab.rb -``` - -This might prevent some assets from being served. -[Read more](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6087#note_558194395) in one of the related issues. +See [troubleshooting](package_troubleshooting.md) for more information. diff --git a/doc/update/package/package_troubleshooting.md b/doc/update/package/package_troubleshooting.md new file mode 100644 index 00000000000..6d83db1fac1 --- /dev/null +++ b/doc/update/package/package_troubleshooting.md @@ -0,0 +1,293 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting **(FREE SELF)** + +## Get the status of a GitLab installation + +```shell +sudo gitlab-ctl status +sudo gitlab-rake gitlab:check SANITIZE=true +``` + +- Information on using `gitlab-ctl` to perform [maintenance tasks](https://docs.gitlab.com/omnibus/maintenance/index.html). +- Information on using `gitlab-rake` to [check the configuration](../../administration/raketasks/maintenance.md#check-gitlab-configuration). + +## RPM 'package is already installed' error + +If you are using RPM and you are upgrading from GitLab Community Edition to GitLab Enterprise Edition you may get an error like this: + +```shell +package gitlab-7.5.2_omnibus.5.2.1.ci-1.el7.x86_64 (which is newer than gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64) is already installed +``` + +You can override this version check with the `--oldpackage` option: + +```shell +sudo rpm -Uvh --oldpackage gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64.rpm +``` + +## Package obsoleted by installed package + +CE and EE packages are marked as obsoleting and replacing each other so that both aren't installed and running at the same time. + +If you are using local RPM files to switch from CE to EE or vice versa, use `rpm` for installing the package rather than `yum`. If you try to use yum, then you may get an error like this: + +```plaintext +Cannot install package gitlab-ee-11.8.3-ee.0.el6.x86_64. It is obsoleted by installed package gitlab-ce-11.8.3-ce.0.el6.x86_64 +``` + +To avoid this issue, either: + +- Use the same instructions provided in the + [Upgrade using a manually-downloaded package](index.md#upgrade-using-a-manually-downloaded-package) section. +- Temporarily disable this checking in yum by adding `--setopt=obsoletes=0` to the options given to the command. + +## 500 error when accessing Project > Settings > Repository + +This error occurs when GitLab is converted from CE > EE > CE, and then back to EE. +When viewing a project's repository settings, you can view this error in the logs: + +```shell +Processing by Projects::Settings::RepositoryController#show as HTML + Parameters: {"namespace_id"=>"<namespace_id>", "project_id"=>"<project_id>"} +Completed 500 Internal Server Error in 62ms (ActiveRecord: 4.7ms | Elasticsearch: 0.0ms | Allocations: 14583) + +NoMethodError (undefined method `commit_message_negative_regex' for #<PushRule:0x00007fbddf4229b8> +Did you mean? commit_message_regex_change): +``` + +This error is caused by an EE feature being added to a CE instance on the initial move to EE. +After the instance is moved back to CE and then is upgraded to EE again, the +`push_rules` table already exists in the database. Therefore, a migration is +unable to add the `commit_message_regex_change` column. + +This results in the [backport migration of EE tables](https://gitlab.com/gitlab-org/gitlab/-/blob/cf00e431024018ddd82158f8a9210f113d0f4dbc/db/migrate/20190402150158_backport_enterprise_schema.rb#L1619) not working correctly. +The backport migration assumes that certain tables in the database do not exist when running CE. + +To fix this issue: + +1. Start a database console: + + In GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + +1. Manually add the missing `commit_message_negative_regex` column: + + ```sql + ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; + + # Exit psql + \q + ``` + +1. Restart GitLab: + + ```shell + sudo gitlab-ctl restart + ``` + +## Error `Failed to connect to the internal GitLab API` on a separate GitLab Pages server + +See [GitLab Pages administration troubleshooting](../../administration/pages/troubleshooting.md#failed-to-connect-to-the-internal-gitlab-api). + +## Error `An error occurred during the signature verification` when running `apt-get update` + +To update the GPG key of the GitLab packages server run: + +```shell +curl --silent "https://packages.gitlab.com/gpg.key" | apt-key add - +apt-get update +``` + +## `Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] [..] Command timed out after 3600s` + +If database schema and data changes (database migrations) must take more than one hour to run, +upgrades fail with a `timed out` error: + +```plaintext +FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) +had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database] +(/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) +had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: +``` + +To fix this error: + +1. Run the remaining database migrations: + + ```shell + sudo gitlab-rake db:migrate + ``` + + This command may take a very long time to complete. Use `screen` or some other mechanism to ensure + the program is not interrupted if your SSH session drops. + +1. Complete the upgrade: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Hot reload `puma` and `sidekiq` services: + + ```shell + sudo gitlab-ctl hup puma + sudo gitlab-ctl restart sidekiq + ``` + +## Missing asset files + +Following an upgrade, GitLab might not be correctly serving up assets such as images, JavaScript, and style sheets. +It might be generating 500 errors, or the web UI may be failing to render properly. + +In a scaled out GitLab environment, if one web server behind the load balancer is demonstrating +this issue, the problem occurs intermittently. + +The [Rake task to recompile](../../administration/raketasks/maintenance.md#precompile-the-assets) the +assets doesn't apply to an Omnibus installation which serves +pre-compiled assets from `/opt/gitlab/embedded/service/gitlab-rails/public/assets`. + +Potential causes and fixes: + +- [Ensure no old processes are running](#old-processes). +- [Remove duplicate sprockets files](#duplicate-sprockets-files) +- [The installation is incomplete](#incomplete-installation) +- [NGINX Gzip support is disabled](#nginx-gzip-support) + +### Old processes + +The most likely cause is that an old Puma process is running, instructing clients +to request asset files from a previous release of GitLab. As the files no longer exist, +HTTP 404 errors are returned. + +A reboot is the best way to ensure these old Puma processes are no longer running. + +Alternatively: + +1. Stop Puma: + + ```shell + gitlab-ctl stop puma + ``` + +1. Check for any remaining Puma processes, and kill them: + + ```shell + ps -ef | egrep 'puma[: ]' + kill <processid> + ``` + +1. Verify with `ps` that the Puma processes have stopped running. + +1. Start Puma + + ```shell + gitlab-ctl start puma + ``` + +### Duplicate sprockets files + +The compiled asset files have unique file names in each release. The sprockets files +provide a mapping from the filenames in the application code to the unique filenames. + +```plaintext +/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest*.json +``` + +Make sure there's only one sprockets file. [Rails uses the first one](https://github.com/rails/sprockets-rails/blob/118ce60b1ffeb7a85640661b014cd2ee3c4e3e56/lib/sprockets/railtie.rb#L201). + +A check for duplicate sprockets files runs during Omnibus GitLab upgrades: + +```plaintext +GitLab discovered stale file(s) from the previous install that need to be cleaned up. +The following files need to be removed: + +/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-e16fdb7dd73cfdd64ed9c2cc0e35718a.json +``` + +Options for resolving this include: + +- If you have the output from the package upgrade, remove the specified files. Then restart Puma: + + ```shell + gitlab-ctl restart puma + ``` + +- If you don't have the message, perform a reinstall + (see [incomplete installation](#incomplete-installation) below for more details) + to generate it again. + +- Remove all the sprockets files and then follow the instructions for an [incomplete installation](#incomplete-installation). + +### Incomplete installation + +An incomplete installation could be the cause of this issue. + +Verify the package to determine if this is the problem: + +- For Debian distributions: + + ```shell + apt-get install debsums + debsums -c gitlab-ee + ``` + +- For Red Hat/SUSE (RPM) distributions: + + ```shell + rpm -V gitlab-ee + ``` + +To reinstall the package to fix an incomplete installation: + +1. Check the installed version + + - For Debian distributions: + + ```shell + apt --installed list gitlab-ee + ``` + + - For Red Hat/SUSE (RPM) distributions: + + ```shell + rpm -qa gitlab-ee + ``` + +1. Reinstall the package, specifying the installed version. For example 14.4.0 Enterprise Edition: + + - For Debian distributions: + + ```shell + apt-get install --reinstall gitlab-ee=14.4.0-ee.0 + ``` + + - For Red Hat/SUSE (RPM) distributions: + + ```shell + yum reinstall gitlab-ee-14.4.0 + ``` + +### NGINX Gzip support + +Check whether `nginx['gzip_enabled']` has been disabled: + +```shell +grep gzip /etc/gitlab/gitlab.rb +``` + +This might prevent some assets from being served. +[Read more](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6087#note_558194395) in one of the related issues. diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index 00ee87ac96c..eda78402c7a 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -143,7 +143,7 @@ version prior to upgrading the application server. If you're using Geo: - Review [Geo upgrade documentation](../administration/geo/replication/upgrading_the_geo_sites.md). -- Read about the [Geo version-specific update instructions](../administration/geo/replication/version_specific_upgrades.md). +- Read about the [Geo version-specific update instructions](index.md#version-specific-upgrading-instructions). - Review Geo-specific steps when [upgrading the database](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). - Create an upgrade and rollback plan for _each_ Geo site (primary and each secondary). diff --git a/doc/update/versions/gitlab_14_changes.md b/doc/update/versions/gitlab_14_changes.md index 3244e63df2d..68784a27e6b 100644 --- a/doc/update/versions/gitlab_14_changes.md +++ b/doc/update/versions/gitlab_14_changes.md @@ -44,7 +44,7 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: ``` - A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). + A workaround exists to [complete the data change and the upgrade manually](../package/package_troubleshooting.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). ### Linux package installations @@ -144,9 +144,10 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f 1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`. 1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`. + - GitLab 14.8.0 includes a -[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) -that may remain stuck permanently in a **pending** state. + [background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) + that may remain stuck permanently in a **pending** state. To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): @@ -287,7 +288,7 @@ that may remain stuck permanently in a **pending** state. Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: ``` - [There is a workaround to complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s) + [There is a workaround to complete the data change and the upgrade manually](../package/package_troubleshooting.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s) - As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default. For **self-compiled (source) installations**, `config/cable.yml` is required to be present. @@ -305,8 +306,8 @@ that may remain stuck permanently in a **pending** state. ### Self-compiled installations - When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you -are using a self-compiled installation, update paths to these binaries in your [systemd unit files](../upgrading_from_source.md#configure-systemd-units) -or [init scripts](../upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](../upgrading_from_source.md). + are using a self-compiled installation, update paths to these binaries in your [systemd unit files](../upgrading_from_source.md#configure-systemd-units) + or [init scripts](../upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](../upgrading_from_source.md). ### Geo installations **(PREMIUM SELF)** @@ -363,8 +364,8 @@ or [init scripts](../upgrading_from_source.md#configure-sysv-init-script) by [fo We recommend moving your databases from Aurora to RDS for PostgreSQL before upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../../administration/postgresql/moving.md). - GitLab 14.4.0 includes a -[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) -that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. + [background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) + that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md): @@ -804,7 +805,7 @@ Long running batched background database migrations: Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` - See how to [resolve this error](../background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). + See how to [resolve this error](../background_migrations_troubleshooting.md#database-migrations-failing-because-of-batched-background-migration-not-finished). Other issues: diff --git a/doc/update/versions/gitlab_15_changes.md b/doc/update/versions/gitlab_15_changes.md index df00ca1b46c..203d13a20e6 100644 --- a/doc/update/versions/gitlab_15_changes.md +++ b/doc/update/versions/gitlab_15_changes.md @@ -303,7 +303,7 @@ if you can't upgrade to 15.11.12 and later. Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: ``` - A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). + A workaround exists to [complete the data change and the upgrade manually](../package/package_troubleshooting.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). - The default Sidekiq `max_concurrency` has been changed to 20. This is now consistent in our documentation and product defaults. @@ -597,7 +597,7 @@ A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) ## 15.3.3 - In GitLab 15.3.3, [SAML Group Links](../../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See -[the API documentation](../../api/members.md). + [the API documentation](../../api/members.md). - A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue: - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly. diff --git a/doc/update/versions/gitlab_16_changes.md b/doc/update/versions/gitlab_16_changes.md index 57f613dd073..4948db7bd29 100644 --- a/doc/update/versions/gitlab_16_changes.md +++ b/doc/update/versions/gitlab_16_changes.md @@ -17,22 +17,55 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f ## Issues to be aware of when upgrading from 15.11 - [PostgreSQL 12 is not supported starting from GitLab 16](../../update/deprecations.md#postgresql-12-deprecated). Upgrade PostgreSQL to at least version 13.6 before upgrading to GitLab 16.0 or later. -- Some GitLab installations must upgrade to GitLab 16.0 before upgrading to any other version. For more information, see - [Long-running user type data change](#long-running-user-type-data-change). -- Other installations can skip 16.0, 16.1, and 16.2 as the first required stop on the upgrade path is 16.3. Review the notes for those intermediate - versions. - If your GitLab instance upgraded first to 15.11.0, 15.11.1, or 15.11.2 the database schema is incorrect. - Recommended: perform the workaround before upgrading to 16.x. - See [the details and workaround](#undefined-column-error-upgrading-to-162-or-later). -- Linux package installations must change Gitaly and Praefect configuration structure before upgrading to GitLab 16. + Perform the [workaround](#undefined-column-error-upgrading-to-162-or-later) before upgrading to 16.x. +- Most installations can skip 16.0, 16.1, and 16.2, as the first required stop on the upgrade path is 16.3. + In all cases, you should review the notes for those intermediate versions. + + Some GitLab installations must stop at those intermediate versions depending on which features are used + and the size of the environment: + + - 16.0.8: Instances with lots of records in the users table. + For more information, see [long-running user type data change](#long-running-user-type-data-change). + - [16.1.5](#1610): Instances that use the NPM package registry. + - [16.2.8](#1620): Instances with lots of pipeline variables (including historical pipelines). + + If your instance is affected and you skip these stops: + + - The upgrade can take hours to complete. + - The instance generates 500 errors until all the database changes are finished, after which + Puma and Sidekiq must restarted. + - For Linux package installations, a timeout occurs and a + [manual workaround to complete the migrations](../package/package_troubleshooting.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s) + is necessary. + +- GitLab 16.0 introduced changes around enforcing limits on project sizes. On self-managed, if you use + these limits, projects that have reached their limit causes error messages when pushing to unaffected Git + repositories in the same group. The errors often refer to exceeding a limit of zero bytes (`limit of 0 B`). + + The pushes succeed, but the errors imply otherwise and might cause issues with automation. + [Read more in the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/416646). + The [bug is fixed in GitLab 16.5 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131122). + +### Linux package installations + +- Gitaly and Praefect configuration structure must be changed before upgrading to GitLab 16. **To avoid data loss** reconfigure Praefect first, and as part of the new configuration, disable metadata verification. Read more: - [Praefect configuration structure change](#praefect-configuration-structure-change). - [Gitaly configuration structure change](#gitaly-configuration-structure-change). +- If you reconfigure Gitaly to store Git data in a location other than `/var/opt/gitlab/git-data/repositories`, + packaged GitLab 16.0 and later does not automatically create the directory structure. + [Read the issue for more details and the workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8320). + ## 16.7.0 +- GitLab 16.7 is a required upgrade stop. This ensures that all database changes introduced + in GitLab 16.7 and earlier have been implemented on all self-managed instances. Dependent changes can then be released + in GitLab 16.8 and later. [Issue 429611](https://gitlab.com/gitlab-org/gitlab/-/issues/429611) provides more details. + ### Linux package installations Specific information applies to Linux package installations: @@ -41,15 +74,45 @@ Specific information applies to Linux package installations: During a package upgrade, the database isn't upgraded to PostgreSQL 14. If you want to upgrade to PostgreSQL 14, [you must do it manually](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). - PostgreSQL 14 isn't supported on Geo deployments and is [planned](https://gitlab.com/groups/gitlab-org/-/epics/9065) - for future releases. - If you want to use PostgreSQL 13, you must set `postgresql['version'] = 13` in `/etc/gitlab/gitlab.rb`. +### Geo installations + +- PostgreSQL version 14 is the default for fresh installations of GitLab 16.7 and later. However, due to an [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7768#note_1652076255), existing Geo secondary sites cannot be upgraded to PostgreSQL version 14. All Geo sites must run the same version of PostgreSQL. If you are adding a new Geo secondary site based on GitLab 16.7 you must take one of the following actions based on your configuration: + + - You are adding your first Geo secondary site: [Upgrade the Primary site to PostgreSQL 14](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) before setting up the new Geo secondary site. No special action is required if your primary site is already running PostgreSQL 14. + - You are adding a new Geo secondary site to a deployment that already has one or more Geo secondaries: + - All sites are running PostgreSQL 13: Install the new Geo secondary site with [pinned PostgreSQL version 13](https://docs.gitlab.com/omnibus/settings/database.html#pin-the-packaged-postgresql-version-fresh-installs-only). + - All sites are running PostgreSQL 14: No special action is required. + +- You might experience verification failures on a subset of projects due to checksum mismatch between the primary site and the secondary site. The details are tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/427493). There is no risk of data loss as the data is being correctly replicated to the secondary sites. Users cloning impacted projects from a Geo secondary site will always be redirected to the primary site. There are no known workarounds at this time. We are actively working on a fix. + + **Affected releases**: + + | Affected minor releases | Affected patch releases | Fixed in | + | ----------------------- | ----------------------- | -------- | + | 16.4 | All | None | + | 16.5 | All | None | + | 16.6 | All | None | + | 16.7 | All | None | + ## 16.6.0 - Old [CI Environment destroy jobs may be spawned](https://gitlab.com/gitlab-org/gitlab/-/issues/433264#) after upgrading to GitLab 16.6. +### Geo installations + +- You might experience verification failures on a subset of projects due to checksum mismatch between the primary site and the secondary site. The details are tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/427493). There is no risk of data loss as the data is being correctly replicated to the secondary sites. Users cloning impacted projects from a Geo secondary site will always be redirected to the primary site. There are no known workarounds at this time. We are actively working on a fix. + + **Affected releases**: + + | Affected minor releases | Affected patch releases | Fixed in | + | ----------------------- | ----------------------- | -------- | + | 16.4 | All | None | + | 16.5 | All | None | + | 16.6 | All | None | + | 16.7 | All | None | + ## 16.5.0 - Git 2.42.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git). @@ -137,6 +200,17 @@ Specific information applies to installations using Geo: | 16.4 | All | None | | 16.5 | 16.5.0 - 16.5.1 | 16.5.2 | +- You might experience verification failures on a subset of projects due to checksum mismatch between the primary site and the secondary site. The details are tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/427493). There is no risk of data loss as the data is being correctly replicated to the secondary sites. Users cloning impacted projects from a Geo secondary site will always be redirected to the primary site. There are no known workarounds at this time. We are actively working on a fix. + + **Affected releases**: + + | Affected minor releases | Affected patch releases | Fixed in | + | ----------------------- | ----------------------- | -------- | + | 16.4 | All | None | + | 16.5 | All | None | + | 16.6 | All | None | + | 16.7 | All | None | + ## 16.4.0 - Updating a group path [received a bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/419289) that uses a database index introduced in 16.3. @@ -208,8 +282,8 @@ Specific information applies to installations using Geo: ### Self-compiled installations - A new method of configuring paths for the GitLab secret and custom hooks is preferred in GitLab 16.4 and later: - 1. Update your configuration `[gitlab] secret_file` to [configure the path](../../administration/gitaly/reference.md#gitlab) to the GitLab secret token. - 1. If you have custom hooks, update your configuration `[hooks] custom_hooks_dir` to [configure the path](../../administration/gitaly/reference.md#custom-hooks) to + 1. Update your configuration `[gitlab] secret_file` to [configure the path](../../administration/gitaly/reference.md) to the GitLab secret token. + 1. If you have custom hooks, update your configuration `[hooks] custom_hooks_dir` to [configure the path](../../administration/gitaly/reference.md) to server-side custom hooks. 1. Remove the `[gitlab-shell] dir` configuration. @@ -276,6 +350,17 @@ Specific information applies to installations using Geo: | 16.4 | All | None | | 16.5 | 16.5.0 - 16.5.1 | 16.5.2 | +- You might experience verification failures on a subset of projects due to checksum mismatch between the primary site and the secondary site. The details are tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/427493). There is no risk of data loss as the data is being correctly replicated to the secondary sites. Users cloning impacted projects from a Geo secondary site will always be redirected to the primary site. There are no known workarounds at this time. We are actively working on a fix. + + **Affected releases**: + + | Affected minor releases | Affected patch releases | Fixed in | + | ----------------------- | ----------------------- | -------- | + | 16.4 | All | None | + | 16.5 | All | None | + | 16.6 | All | None | + | 16.7 | All | None | + ## 16.3.0 - **Update to GitLab 16.3.5 or later**. This avoids [issue 425971](https://gitlab.com/gitlab-org/gitlab/-/issues/425971) that causes an excessive use of database disk space for GitLab 16.3.3 and 16.3.4. @@ -979,7 +1064,7 @@ FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s: ``` -[There is a fix-forward workaround for this issue](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). +[There is a fix-forward workaround for this issue](../package/package_troubleshooting.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s). While the workaround is completing the database changes, GitLab is likely to be in an unusable state, generating `500` errors. The errors are caused by Sidekiq and Puma running diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index 6ecd23a0dc8..c40c7a0524e 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -343,7 +343,7 @@ version), and then upgrade the original primary. For this, we must know the address of the current Redis primary. - If your application node is running GitLab 12.7.0 or later, you can use the -following command to get address of current Redis primary + following command to get address of current Redis primary ```shell sudo gitlab-ctl get-redis-master @@ -514,27 +514,27 @@ the update on the **primary** node: - Run post-deployment database migrations - ```shell - sudo gitlab-rake db:migrate - ``` + ```shell + sudo gitlab-rake db:migrate + ``` - After the update is finalized on the primary node, hot reload `puma` and -restart `sidekiq` and `geo-logcursor` services on **all primary and secondary** -nodes: + restart `sidekiq` and `geo-logcursor` services on **all primary and secondary** + nodes: - ```shell - sudo gitlab-ctl hup puma - sudo gitlab-ctl restart sidekiq - sudo gitlab-ctl restart geo-logcursor - ``` + ```shell + sudo gitlab-ctl hup puma + sudo gitlab-ctl restart sidekiq + sudo gitlab-ctl restart geo-logcursor + ``` After updating all nodes (both **primary** and all **secondaries**), check their status: - Verify Geo configuration and dependencies - ```shell - sudo gitlab-rake gitlab:geo:check - ``` + ```shell + sudo gitlab-rake gitlab:geo:check + ``` If you do not want to run zero downtime upgrades in the future, make sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert diff --git a/doc/user/ai_features.md b/doc/user/ai_features.md index 71b5617a4a9..931876682e0 100644 --- a/doc/user/ai_features.md +++ b/doc/user/ai_features.md @@ -17,7 +17,7 @@ GitLab is creating AI-assisted features across our DevSecOps platform. These fea | Helps you discover or recall Git commands when and where you need them. | [Git suggestions](https://gitlab.com/gitlab-org/gitlab/-/issues/409636) | **(ULTIMATE SAAS EXPERIMENT)** | | Assists with quickly getting everyone up to speed on lengthy conversations to help ensure you are all on the same page. | [Discussion summary](#summarize-issue-discussions-with-discussion-summary) | **(ULTIMATE SAAS EXPERIMENT)** | | Generates issue descriptions. | [Issue description generation](#summarize-an-issue-with-issue-description-generation) | **(ULTIMATE SAAS EXPERIMENT)** | -| Helps you write code more efficiently by viewing code suggestions as you type. <br><br><i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Watch overview](https://www.youtube.com/watch?v=hCAyCTacdAQ) | [Code Suggestions](project/repository/code_suggestions/index.md) | For SaaS: **(FREE BETA)**<br><br> For self-managed: **(ULTIMATE BETA)** | +| Helps you write code more efficiently by viewing code suggestions as you type. <br><br><i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Watch overview](https://www.youtube.com/watch?v=hCAyCTacdAQ) | [Code Suggestions](project/repository/code_suggestions/index.md) | For SaaS: **(FREE)**<br><br> For self-managed: **(PREMIUM)** | | Automates repetitive tasks and helps catch bugs early. | [Test generation](project/merge_requests/ai_in_merge_requests.md#generate-suggested-tests-in-merge-requests) | **(ULTIMATE SAAS EXPERIMENT)** | | Generates a description for the merge request based on the contents of the template. | [Merge request template population](project/merge_requests/ai_in_merge_requests.md#fill-in-merge-request-templates) | **(ULTIMATE SAAS EXPERIMENT)** | | Assists in creating faster and higher-quality reviews by automatically suggesting reviewers for your merge request. <br><br><i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Watch overview](https://www.youtube.com/watch?v=ivwZQgh4Rxw) | [Suggested Reviewers](project/merge_requests/reviews/index.md#gitlab-duo-suggested-reviewers) | **(ULTIMATE SAAS)** | @@ -26,7 +26,7 @@ GitLab is creating AI-assisted features across our DevSecOps platform. These fea | Helps you remediate vulnerabilities more efficiently, boost your skills, and write more secure code. <br><br><i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Watch overview](https://www.youtube.com/watch?v=6sDf73QOav8) | [Vulnerability summary](application_security/vulnerabilities/index.md#explaining-a-vulnerability) | **(ULTIMATE SAAS BETA)** | | Generates a merge request containing the changes required to mitigate a vulnerability. | [Vulnerability resolution](application_security/vulnerabilities/index.md#explaining-a-vulnerability) | **(ULTIMATE SAAS EXPERIMENT)** | | Helps you understand code by explaining it in English language. <br><br><i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Watch overview](https://www.youtube.com/watch?v=1izKaLmmaCA) | [Code explanation](#explain-code-in-the-web-ui-with-code-explanation) | **(ULTIMATE SAAS EXPERIMENT)** | -| Processes and generates text and code in a conversational manner. Helps you quickly identify useful information in large volumes of text in issues, epics, code, and GitLab documentation. | [GitLab Duo Chat](gitlab_duo_chat.md) | **(ULTIMATE SAAS BETA)** | +| Processes and generates text and code in a conversational manner. Helps you quickly identify useful information in large volumes of text in issues, epics, code, and GitLab documentation. | [GitLab Duo Chat](gitlab_duo_chat.md) | **(ULTIMATE BETA)** | | Assists you in determining the root cause for a pipeline failure and failed CI/CD build. | [Root cause analysis](#root-cause-analysis) | **(ULTIMATE SAAS EXPERIMENT)** | | Assists you with predicting productivity metrics and identifying anomalies across your software development lifecycle. | [Value stream forecasting](#forecast-deployment-frequency-with-value-stream-forecasting) | **(ULTIMATE ALL EXPERIMENT)** | diff --git a/doc/user/analytics/analytics_dashboards.md b/doc/user/analytics/analytics_dashboards.md index 0699be4e0ad..e647dcf170a 100644 --- a/doc/user/analytics/analytics_dashboards.md +++ b/doc/user/analytics/analytics_dashboards.md @@ -55,7 +55,7 @@ With custom dashboards, you can design and create visualizations for the metrics You can create custom dashboards with the dashboard designer. - Each project can have an unlimited number of dashboards. -The only limitation might be the [repository size limit](../project/repository/reducing_the_repo_size_using_git.md#storage-limits). + The only limitation might be the [repository size limit](../project/repository/reducing_the_repo_size_using_git.md#storage-limits). - Each dashboard can reference one or more [visualizations](#define-a-chart-visualization). - Visualizations are shared across dashboards. @@ -122,12 +122,11 @@ To view the Value Streams Dashboard as an analytics dashboard for a project: ## View group dashboards -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390542) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `group_analytics_dashboards`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390542) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `group_analytics_dashboards`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/416970) in GitLab 16.8. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `group_analytics_dashboards`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `group_analytics_dashboards`. On GitLab.com, this feature is available. Prerequisites: @@ -226,7 +225,7 @@ You can define different charts, and add visualization options to some of them: - Line chart, with the options listed in the [ECharts documentation](https://echarts.apache.org/en/option.html). - Column chart, with the options listed in the [ECharts documentation](https://echarts.apache.org/en/option.html). -- Data table, with the only option to render `links` (array of objects, each with `text` and `href` properties to specify the dimensions to be used in links). See [example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/analytics_visualization.json?ref_type=heads#L112)). +- Data table. - Single stat, with the only option to set `decimalPlaces` (number, default value is 0). To define a chart for your dashboards: @@ -304,5 +303,5 @@ If a dashboard panel displays a message that the visualization configuration is If a dashboard panel displays an error message: - Check your [Cube query](../product_analytics/index.md#product-analytics-dashboards) and [visualization](../analytics/analytics_dashboards.md#define-a-chart-visualization) -configurations, and make sure they are set up correctly. + configurations, and make sure they are set up correctly. - For [product analytics](../product_analytics/index.md), also check that your visualization's Cube query is valid. diff --git a/doc/user/analytics/contributor_statistics.md b/doc/user/analytics/contributor_statistics.md deleted file mode 100644 index b6f195e22ad..00000000000 --- a/doc/user/analytics/contributor_statistics.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'contributor_analytics.md' -remove_date: '2023-03-06' ---- - -This document was moved to [another location](contributor_analytics.md). - -<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/analytics/dora_metrics.md b/doc/user/analytics/dora_metrics.md index 53a25acbca5..372ea0a5807 100644 --- a/doc/user/analytics/dora_metrics.md +++ b/doc/user/analytics/dora_metrics.md @@ -121,7 +121,7 @@ GitLab calculates this as the number of incidents divided by the number of deplo - [GitLab incidents](../../operations/incident_management/incidents.md) are tracked. - All incidents are related to a production environment. - Incidents and deployments have a strictly one-to-one relationship. An incident is related to only one production deployment, and any production deployment is related to no -more than one incident. + more than one incident. ### How to improve change failure rate diff --git a/doc/user/analytics/img/enhanced_issue_analytics_v16_7.png b/doc/user/analytics/img/enhanced_issue_analytics_v16_7.png Binary files differindex 519e56acaa5..b253fe336ee 100644 --- a/doc/user/analytics/img/enhanced_issue_analytics_v16_7.png +++ b/doc/user/analytics/img/enhanced_issue_analytics_v16_7.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index d58426bd76b..eef34214c23 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -1,6 +1,7 @@ --- stage: Plan group: Optimize +description: Instance, group, and project analytics. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/analytics/issue_analytics.md b/doc/user/analytics/issue_analytics.md index b8aa23a0af2..089a5636a52 100644 --- a/doc/user/analytics/issue_analytics.md +++ b/doc/user/analytics/issue_analytics.md @@ -1,63 +1,11 @@ --- -stage: Plan -group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../group/issues_analytics/index.md' +remove_date: '2024-04-05' --- -# Issue analytics for projects **(PREMIUM ALL)** +This document was moved to [another location](../group/issues_analytics/index.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196561) in GitLab 12.9. - -Issue analytics is a bar graph which illustrates the number of issues created each month. -The default time span is 13 months, which includes the current month, and the 12 months -prior. - -To access the chart: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Analyze > Issue analytics**. - -You can also access the chart from the [Value Streams Dashboard](value_streams_dashboard.md#dashboard-metrics-and-drill-down-reports) through the **New issues** drill-down report. - -Hover over each bar to see the total number of issues. - -To narrow the scope of issues included in the graph, enter your criteria in the -**Search or filter results...** field. Criteria from the following list can be typed in or selected from a menu: - -- Author -- Assignee -- Milestone -- Label -- My reaction -- Weight - -You can change the total number of months displayed by setting a URL parameter. -For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` -shows a total of 15 months for the chart in the GitLab.org group. - - - -## Enhanced issue analytics **(ULTIMATE ALL)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233905/) in GitLab 16.4 [with a flag](../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, an administrator can -[enable the feature flag](../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. On GitLab.com, this feature is not -available. This feature is not ready for production use. - -Enhanced issue analytics display the additional metric "Issues closed", which represents the total number of resolved issues in your project over a selected period. -You can use this metric to improve the overall turn-around time and value delivered to your customers. - - - -## Drill into the information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196547) in GitLab 13.1. - -You can examine details of individual issues by browsing the table -located below the chart. - -The chart displays the top 100 issues based on the global page filters. - - +<!-- This redirect file can be deleted after <2024-04-05>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/analytics/merge_request_analytics.md b/doc/user/analytics/merge_request_analytics.md index 5b5b1ec002d..0d2c375f7ae 100644 --- a/doc/user/analytics/merge_request_analytics.md +++ b/doc/user/analytics/merge_request_analytics.md @@ -46,8 +46,8 @@ To view the number of merge requests merged during a specific date range: 1. Select a parameter. 1. Select a value or enter text to refine the results. 1. To adjust the date range: - - In the **From** field, select a start date. - - In the **To** field, select an end date. + - In the **From** field, select a start date. + - In the **To** field, select an end date. The **Throughput** chart shows issues closed or merge requests merged (not closed) over a period of time. @@ -75,4 +75,4 @@ To view **Mean time to merge**: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Analyze > Merge request analytics**. The **Mean time to merge** number -is displayed on the dashboard. + is displayed on the dashboard. diff --git a/doc/user/analytics/repository_analytics.md b/doc/user/analytics/repository_analytics.md index 93171dc3136..8bc163f6f3f 100644 --- a/doc/user/analytics/repository_analytics.md +++ b/doc/user/analytics/repository_analytics.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Use repository analytics to view information about a project's Git repository: -- Programming languages used in the repository. +- Programming languages used in the repository's default branch. - Code coverage history from last 3 months ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1). - Commit statistics (last month). - Commits per day of month. diff --git a/doc/user/analytics/value_streams_dashboard.md b/doc/user/analytics/value_streams_dashboard.md index a50eab42a2d..c093fdf8cb3 100644 --- a/doc/user/analytics/value_streams_dashboard.md +++ b/doc/user/analytics/value_streams_dashboard.md @@ -16,7 +16,7 @@ For more information, see also the [Value Stream Management category direction p The Value Streams Dashboard is a customizable dashboard you can use to identify trends, patterns, and opportunities for digital transformation improvements. The centralized UI in Value Streams Dashboard acts as the single source of truth (SSOT), where all stakeholders can access and view the same set of metrics that are relevant to the organization. -The Value Streams Dashboard includes the following metrics: +The Value Streams Dashboard includes two panels (DevSecOps metrics comparison and DORA Performers score) that visualize the following metrics: - [DORA metrics](dora_metrics.md) - [Value Stream Analytics (VSA) - flow metrics](../group/value_stream_analytics/index.md) @@ -29,9 +29,11 @@ With the Value Streams Dashboard, you can: - Understand security exposure. - Drill down into individual projects or metrics to take actions for improvement. -The Value Streams Dashboard has a default configuration, but you can also [customize the dashboard panels](#customize-the-dashboard-panels). +## Value Streams Dashboard panels -## DevSecOps metrics comparison panel +The Value Streams Dashboard panels has a default configuration, but you can also [customize the dashboard panels](#customize-the-dashboard-panels). + +### DevSecOps metrics comparison panel The DevSecOps metrics comparison displays DORA4, vulnerability, and flow metrics for a group or project in the month-to-date, last month, the month before, and the past 180 days. @@ -46,7 +48,7 @@ that are the largest value contributors, overperforming, or underperforming. You can also drill down the metrics for further analysis. When you hover over a metric, a tooltip displays an explanation of the metric and a link to the related documentation page. -## DORA Performers score panel +### DORA Performers score panel > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386843) in GitLab 16.2 [with a flag](../../administration/feature_flags.md) named `dora_performers_score_panel`. Disabled by default. @@ -73,7 +75,7 @@ For example, if a project has a high score for Deployment Frequency (Velocity), These scoring are based on Google's classifications in the [DORA 2022 Accelerate State of DevOps Report](https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out). -### Filter by project topics +#### Filter the DORA Performers score by project topics When used in combination with a [YAML configuration](#using-yaml-configuration), you can filter the projects shown based on their assigned [topics](../project/settings/project_features_permissions.md#project-topics). @@ -90,17 +92,9 @@ If multiple topics are provided, all topics will need to match for the project t ## Enable or disable overview background aggregation **(ULTIMATE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120610) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `modify_value_stream_dashboard_settings`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120610) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `value_stream_dashboard_on_off_setting`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130704) in GitLab 16.4. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `modify_value_stream_dashboard_settings`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. - -Prerequisites: - -- You must have administrator access to the instance. +> - [Feature flag `value_stream_dashboard_on_off_setting` removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134619) in GitLab 16.6. To enable or disable the overview count aggregation for the Value Streams Dashboard: @@ -221,9 +215,19 @@ panels: <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview of editing label filters in the configuration file, see [GitLab Value Streams Dashboard - Label filters demo](https://www.youtube.com/watch?v=4qDAHCxCfik). +### Filter the DevSecOps metrics comparison panel by labels + Label filters are appended as query parameters to the URL of the drill-down report of each eligible metric and automatically applied. If the comparison panel from the configuration file is enabled with `filter_labels`, the drill-down links inherit the labels from the panel filter. +```yaml + - data: + namespace: group/another-project + filter_labels: + - in_development + - in_review +``` + ## Dashboard metrics and drill-down reports | Metric | Description | Drill-down report | Documentation page | ID | diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index 9c16c70c78f..01515a90653 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -105,7 +105,7 @@ responses in HAR format. #### Create a HAR file with Fiddler 1. Go to the [Fiddler home page](https://www.telerik.com/fiddler) and sign in. If you don't already -have an account, first create an account. + have an account, first create an account. 1. Browse pages that call an API. Fiddler automatically captures the requests. 1. Select one or more requests, then from the context menu, select **Export > Selected Sessions**. 1. In the **Choose Format** dropdown list select **HTTPArchive v1.2**. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 735b2356780..cab8c926def 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -498,15 +498,15 @@ The following is a summary of the variable scopes supported by the Postman Clien - **Global Environment (Global) scope** is a special pre-defined environment that is available throughout a workspace. We can also refer to the _global environment_ scope as the _global_ scope. The Postman Client allows exporting the global environment into a JSON file, which can be used with API Fuzzing. - **Environment scope** is a named group of variables created by a user in the Postman Client. -The Postman Client supports a single active environment along with the global environment. The variables defined in an active user-created environment take precedence over variables defined in the global environment. The Postman Client allows exporting your environment into a JSON file, which can be used with API Fuzzing. + The Postman Client supports a single active environment along with the global environment. The variables defined in an active user-created environment take precedence over variables defined in the global environment. The Postman Client allows exporting your environment into a JSON file, which can be used with API Fuzzing. - **Collection scope** is a group of variables declared in a given collection. The collection variables are available to the collection where they have been declared and the nested requests or collections. Variables defined in the collection scope take precedence over the _global environment_ scope and also the _environment_ scope. -The Postman Client can export one or more collections into a JSON file, this JSON file contains selected collections, requests, and collection variables. + The Postman Client can export one or more collections into a JSON file, this JSON file contains selected collections, requests, and collection variables. - **API Fuzzing Scope** is a new scope added by API Fuzzing to allow users to provide extra variables, or override variables defined in other supported scopes. This scope is not supported by Postman. The _API Fuzzing Scope_ variables are provided using a [custom JSON file format](#api-fuzzing-scope-custom-json-file-format). - Override values defined in the environment or collection - Defining variables from scripts - Define a single row of data from the unsupported _data scope_ - **Data scope** is a group of variables in which their name and values come from JSON or CSV files. A Postman collection runner like [Newman](https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/) or [Postman Collection Runner](https://learning.postman.com/docs/running-collections/intro-to-collection-runs/) executes the requests in a collection as many times as entries have the JSON or CSV file. A good use case for these variables is to automate tests using scripts in Postman. -API Fuzzing does **not** support reading data from a CSV or JSON file. + API Fuzzing does **not** support reading data from a CSV or JSON file. - **Local scope** are variables that are defined in Postman scripts. API Fuzzing does **not** support Postman scripts and by extension, variables defined in scripts. You can still provide values for the script-defined variables by defining them in one of the supported scopes, or our custom JSON format. Not all scopes are supported by API Fuzzing and variables defined in scripts are not supported. The following table is sorted by broadest scope to narrowest scope. diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 008b5b54cca..c367d647c6c 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -50,7 +50,7 @@ You can configure the following security controls: - [Dynamic Application Security Testing](../dast/index.md) (DAST) - Select **Enable DAST** to configure DAST for the current project. - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. - For more details, read [DAST on-demand scans](../dast/proxy-based.md#on-demand-scans). + For more details, read [DAST on-demand scans](../dast/on-demand_scan.md). - [Dependency Scanning](../dependency_scanning/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to enable Dependency Scanning. For more information, see [Use a preconfigured merge request](../dependency_scanning/index.md#use-a-preconfigured-merge-request). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 8af262e564b..5be9e169078 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -17,7 +17,7 @@ vulnerabilities and displays them in a merge request, you can use GitLab to audi apps. - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Container Scanning](https://www.youtube.com/watch?v=C0jn2eN5MAs). + For an overview, see [Container Scanning](https://www.youtube.com/watch?v=C0jn2eN5MAs). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video walkthrough, see [How to set up Container Scanning using GitLab](https://youtu.be/h__mcXpil_4?si=w_BVG68qnkL9x4l1). Container Scanning is often considered part of Software Composition Analysis (SCA). SCA can contain @@ -128,10 +128,6 @@ Setting `CS_DEFAULT_BRANCH_IMAGE` avoids duplicate vulnerability findings when a The value of `CS_DEFAULT_BRANCH_IMAGE` indicates the name of the scanned image as it appears on the default branch. For more details on how this deduplication is achieved, see [Setting the default branch image](#setting-the-default-branch-image). -## Running jobs in merge request pipelines - -See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines) - ### Customizing the container scanning settings There may be cases where you want to customize how GitLab scans your containers. For example, you @@ -243,6 +239,10 @@ if [Dependency Scanning](../dependency_scanning/index.md) is enabled for your project. This happens because GitLab can't automatically deduplicate findings across different types of scanning tools. To understand which types of dependencies are likely to be duplicated, see [Dependency Scanning compared to Container Scanning](../comparison_dependency_and_container_scanning.md). +#### Running jobs in merge request pipelines + +See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines). + #### Available CI/CD variables You can [configure](#customizing-the-container-scanning-settings) analyzers by using the following CI/CD variables. @@ -263,7 +263,7 @@ including a large number of false positives. | `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` | `"true"` | Disable scanning for language-specific packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | | `CS_DOCKERFILE_PATH` | `Dockerfile` | The path to the `Dockerfile` to use for generating remediations. By default, the scanner looks for a file named `Dockerfile` in the root directory of the project. You should configure this variable only if your `Dockerfile` is in a non-standard location, such as a subdirectory. See [Solutions for vulnerabilities](#solutions-for-vulnerabilities-auto-remediation) for more details. | All | -| `CS_IGNORE_STATUSES` | `""` | Force the analyzer to ignore vulnerability findings with specified statuses in a comma-delimited list. For `trivy`, the following values are allowed: `unknown,not_affected,affected,fixed,under_investigation,will_not_fix,fix_deferred,end_of_life`. For `grype`, the following values are allowed: `fixed,not-fixed,unknown,wont-fix` | All | +| `CS_IGNORE_STATUSES`<sup><b><a href="#notes-regarding-cs-ignore-statuses">1</a></b></sup> | `""` | Force the analyzer to ignore vulnerability findings with specified statuses in a comma-delimited list. For `trivy`, the following values are allowed: `unknown,not_affected,affected,fixed,under_investigation,will_not_fix,fix_deferred,end_of_life`. For `grype`, the following values are allowed: `fixed,not-fixed,unknown,wont-fix` | All | | `CS_IGNORE_UNFIXED` | `"false"` | Ignore vulnerabilities that are not fixed. | All | | `CS_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | | `CS_IMAGE_SUFFIX` | `""` | Suffix added to `CS_ANALYZER_IMAGE`. If set to `-fips`, `FIPS-enabled` image is used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7630) in GitLab 14.10. | All | @@ -275,6 +275,15 @@ including a large number of false positives. | `CS_TRIVY_JAVA_DB` | `"ghcr.io/aquasecurity/trivy-java-db"` | Specify an alternate location for the [trivy-java-db](https://github.com/aquasecurity/trivy-java-db) vulnerability database. | Trivy | | `SECURE_LOG_LEVEL` | `info` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. | All | +<ol> + <li> + <a id="notes-regarding-cs-ignore-statuses"></a> + <p> + Fix status information is highly dependent on accurate fix availability data from the software vendor and container image operating system package metadata. It is also subject to interpretation by individual container scanners. In cases where a container scanner misreports the availability of a fixed package for a vulnerability, using `CS_IGNORE_STATUSES` can lead to false positive or false negative filtering of findings when this setting is enabled. + </p> + </li> +</ol> + ### Supported distributions Support depends on which scanner is used: @@ -370,6 +379,9 @@ The following options are available: | [Grype](https://github.com/anchore/grype) | `registry.gitlab.com/security-products/container-scanning/grype:6` | | Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:6` | +WARNING: +Do not use the `:latest` tag when selecting the scanner image. + ### Setting the default branch image > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. @@ -766,8 +778,7 @@ The images use data from upstream advisory databases depending on which scanner In addition to the sources provided by these scanners, GitLab maintains the following vulnerability databases: -- The proprietary -[GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db). +- The proprietary [GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db). - The open source [GitLab Advisory Database (Open Source Edition)](https://gitlab.com/gitlab-org/advisories-community). In the GitLab Ultimate tier, the data from the [GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is merged in to augment the data from the external sources. In the GitLab Premium and Free tiers, the data from the [GitLab Advisory Database (Open Source Edition)](https://gitlab.com/gitlab-org/advisories-community) is merged in to augment the data from the external sources. This augmentation currently only applies to the analyzer images for the Trivy scanner. diff --git a/doc/user/application_security/continuous_vulnerability_scanning/index.md b/doc/user/application_security/continuous_vulnerability_scanning/index.md index 4d6d48012ae..7d083ea1846 100644 --- a/doc/user/application_security/continuous_vulnerability_scanning/index.md +++ b/doc/user/application_security/continuous_vulnerability_scanning/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/427424) in GitLab 16.7 with an additional feature flag named `global_dependency_scanning_on_advisory_ingestion`. Enabled by default. FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flags](../../feature_flags.md) named `dependency_scanning_on_advisory_ingestion` and `package_metadata_advisory_sync`. +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flags](../../feature_flags.md) named `dependency_scanning_on_advisory_ingestion`. On GitLab.com, this feature is available. Continuous Vulnerability Scanning detects new vulnerabilities outside a pipeline. diff --git a/doc/user/application_security/dast/checks/78.1.md b/doc/user/application_security/dast/checks/78.1.md index bcb655f37ae..ae0af7b1552 100644 --- a/doc/user/application_security/dast/checks/78.1.md +++ b/doc/user/application_security/dast/checks/78.1.md @@ -22,7 +22,7 @@ Ensure your application does not: - Use user-supplied information in the process name to execute. - Use user-supplied information in an OS command execution function which does -not escape shell meta-characters. + not escape shell meta-characters. - Use user-supplied information in arguments to OS commands. The application should have a hardcoded set of arguments that are to be passed diff --git a/doc/user/application_security/dast/on-demand_scan.md b/doc/user/application_security/dast/on-demand_scan.md new file mode 100644 index 00000000000..e43057aea54 --- /dev/null +++ b/doc/user/application_security/dast/on-demand_scan.md @@ -0,0 +1,409 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# DAST On Demand Scan **(ULTIMATE ALL)** + +WARNING: +Do not run DAST scans against a production server. Not only can it perform *any* function that a user can, such +as clicking buttons or submitting forms, but it may also trigger bugs, leading to modification or loss of production data. +Only run DAST scans against a test server. + +## On-demand scans + +> - Auditing for DAST profile management [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. +> - Scheduled on-demand DAST scans [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3 [with a flag](../../../administration/feature_flags.md) named `dast_on_demand_scans_scheduler`. Disabled by default. +> - Scheduled on-demand DAST scans [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. Feature flag `dast_on_demand_scans_scheduler` removed. +> - Runner tags selection [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111499) in GitLab 16.3. +> - Browser based on-demand DAST scans [deployed behind the feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430212) in GitLab 16.8. + +An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger +the scan. You must either start it manually, or schedule it to run. For on-demand DAST scans, +a [site profile](#site-profile) defines **what** is to be scanned, and a +[scanner profile](#scanner-profile) defines **how** the application is to be scanned. + +An on-demand scan can be run in active or passive mode: + +- **Passive mode**: The default mode, which runs a [Passive Browser based scan](/ee/user/application_security/dast/browser_based.md#passive-scans). +- **Active mode**: Runs an [Active Browser based scan](/ee/user/application_security/dast/browser_based.md#active-scans) which is potentially harmful to the site being scanned. To + minimize the risk of accidental damage, running an active scan requires a + [validated site profile](#site-profile-validation). + +### View on-demand DAST scans + +To view on-demand scans: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Secure > On-demand scans**. + +On-demand scans are grouped by their status. The scan library contains all available on-demand +scans. + +### Run an on-demand DAST scan + +Prerequisites: + +- You must have permission to run an on-demand DAST scan against a protected branch. The default + branch is automatically protected. For more information, see + [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). + +To run an existing on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the scan's row, select **Run scan**. + + If the branch saved in the scan no longer exists, you must: + + 1. [Edit the scan](#edit-an-on-demand-scan). + 1. Select a new branch. + 1. Save the edited scan. + +The on-demand DAST scan runs, and the project's dashboard shows the results. + +#### Create an on-demand scan + +Create an on-demand scan to: + +- Run it immediately. +- Save it to be run in the future. +- Schedule it to be run at a specified schedule. + +To create an on-demand DAST scan: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Secure > On-demand scans**. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** fields. +1. In the **Branch** dropdown list, select the desired branch. +1. Optional. Select the runner tags. +1. Select **Select scanner profile** or **Change scanner profile** to open the drawer, and either: + - Select a scanner profile from the drawer, **or** + - Select **New profile**, create a [scanner profile](#scanner-profile), then select **Save profile**. +1. Select **Select site profile** or **Change site profile** to open the drawer, and either: + - Select a site profile from the **Site profile library** drawer, or + - Select **New profile**, create a [site profile](#site-profile), then select **Save profile**. +1. To run the on-demand scan: + + - Immediately, select **Save and run scan**. + - In the future, select **Save scan**. + - On a schedule: + + - Turn on the **Enable scan schedule** toggle. + - Complete the schedule fields. + - Select **Save scan**. + +The on-demand DAST scan runs as specified and the project's dashboard shows the results. + +### View details of an on-demand scan + +To view details of an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. + +### Edit an on-demand scan + +To edit an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. +1. Edit the saved scan's details. +1. Select **Save scan**. + +### Delete an on-demand scan + +To delete an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. +1. On the confirmation dialog, select **Delete**. + +## Site profile + +> - Site profile features, scan method and file URL, were [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - GraphQL endpoint path feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378692) in GitLab 15.7. + +A site profile defines the attributes and configuration details of the deployed application, +website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and +on-demand scans. + +A site profile contains: + +- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced + in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Site type**: The type of target to be scanned, either website or API scan. +- **Target URL**: The URL that DAST runs against. +- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. +- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. +- **Authentication**: + - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. + - **Username**: The username used to authenticate to the website. + - **Password**: The password used to authenticate to the website. + - **Username form field**: The name of username field at the sign-in HTML form. + - **Password form field**: The name of password field at the sign-in HTML form. + - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. + +- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, HTTP Archive (HAR), or GraphQL. + - **GraphQL endpoint path**: The path to the GraphQL endpoint. This path is concatenated with the target URL to provide the URI for the scan to test. The GraphQL endpoint must support introspection queries. + - **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. + +When an API site type is selected, a host override is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. + +When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. +This data can only be read and decrypted with a valid secrets file. + +### Site profile validation + +> Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. + +Site profile validation reduces the risk of running an active scan against the wrong website. A site +must be validated before an active scan can run against it. Each of the site validation methods are +equivalent in functionality, so use whichever is most suitable: + +- **Text file validation**: Requires a text file be uploaded to the target site. The text file is + allocated a name and content that is unique to the project. The validation process checks the + file's content. +- **Header validation**: Requires the header `Gitlab-On-Demand-DAST` be added to the target site, + with a value unique to the project. The validation process checks that the header is present, and + checks its value. +- **Meta tag validation**: Requires the meta tag named `gitlab-dast-validation` be added to the + target site, with a value unique to the project. Make sure it's added to the `<head>` section of + the page. The validation process checks that the meta tag is present, and checks its value. + +### Create a site profile + +To create a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select **New > Site profile**. +1. Complete the fields then select **Save profile**. + +The site profile is saved, for use in an on-demand scan. + +### Edit a site profile + +NOTE: +If a site profile is linked to a security policy, you cannot edit the profile from this page. See +[Scan execution policies](../policies/scan-execution-policies.md) for more information. + +NOTE: +If a site profile's Target URL or Authenticated URL is updated, the request headers and password fields associated with that profile are cleared. + +When a validated site profile's file, header, or meta tag is edited, the site's +[validation status](#site-profile-validation) is revoked. + +To edit a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the fields then select **Save profile**. + +### Delete a site profile + +NOTE: +If a site profile is linked to a security policy, a user cannot delete the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) for more information. + +To delete a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete** to confirm the deletion. + +### Validate a site profile + +Validating a site is required to run an active scan. + +Prerequisites: + +- A runner must be available in the project to run a validation job. +- The GitLab server's certificate must be trusted and must not use a self-signed certificate. + +To validate a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Validate**. +1. Select the validation method. + 1. For **Text file validation**: + 1. Download the validation file listed in **Step 2**. + 1. Upload the validation file to the host, to the location in **Step 3** or any location you + prefer. + 1. If required, edit the file location in **Step 3**. + 1. Select **Validate**. + 1. For **Header validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the header of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the header. + 1. Select **Validate**. + 1. For **Meta tag validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the content of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the meta tag. + 1. Select **Validate**. + +The site is validated and an active scan can run against it. A site profile's validation status is +revoked only when it's revoked manually, or its file, header, or meta tag is edited. + +### Retry a failed validation + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. +> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. +> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. + +Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** +page. + +To retry a site profile's failed validation: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Retry validation**. + +### Revoke a site profile's validation status + +WARNING: +When a site profile's validation status is revoked, all site profiles that share the same URL also +have their validation status revoked. + +To revoke a site profile's validation status: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Beside the validated profile, select **Revoke validation**. + +The site profile's validation status is revoked. + +### Validated site profile headers + +The following are code samples of how you can provide the required site profile header in your +application. + +#### Ruby on Rails example for on-demand scan + +Here's how you can add a custom header in a Ruby on Rails application: + +```ruby +class DastWebsiteTargetController < ActionController::Base + def dast_website_target + response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + head :ok + end +end +``` + +#### Django example for on-demand scan + +Here's how you can add a +[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): + +```python +class DastWebsiteTargetView(View): + def head(self, *args, **kwargs): + response = HttpResponse() + response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + + return response +``` + +#### Node (with Express) example for on-demand scan + +Here's how you can add a +[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): + +```javascript +app.get('/dast-website-target', function(req, res) { + res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') + res.send('Respond to DAST ping') +}) +``` + +## Scanner profile + +> - Deprecated AJAX Spider option with the [introduction of Browser based on-demand DAST scans behind feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430210). +> - Renamed spider timeout to crawl timeout with the [introduction of Browser based on-demand DAST scans behind feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430210). + +A scanner profile defines the configuration details of a security scanner. A scanner profile can be +referenced in `.gitlab-ci.yml` and on-demand scans. + +A scanner profile contains: + +- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner + profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. +- **Crawl timeout:** The maximum number of minutes allowed for the crawler to traverse the site. +- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before + starting the scan. +- **Debug messages:** Include debug messages in the DAST console output. + +### Create a scanner profile + +To create a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select **New > Scanner profile**. +1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). +1. Select **Save profile**. + +### Edit a scanner profile + +NOTE: +If a scanner profile is linked to a security policy, you cannot edit the profile from this page. +For more information, see [Scan execution policies](../policies/scan-execution-policies.md). + +To edit a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Scanner profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the form. +1. Select **Save profile**. + +### Delete a scanner profile + +NOTE: +If a scanner profile is linked to a security policy, a user cannot delete the profile from this +page. For more information, see [Scan execution policies](../policies/scan-execution-policies.md). + +To delete a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Scanner profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete**. + +## Auditing + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +The creation, updating, and deletion of DAST profiles, DAST scanner profiles, +and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md index 6127866b0a9..447babb0ad4 100644 --- a/doc/user/application_security/dast/proxy-based.md +++ b/doc/user/application_security/dast/proxy-based.md @@ -168,9 +168,9 @@ To configure DAST using the UI: 1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or **Configure DAST**. 1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a - scanner profile. For more details, see [scanner profiles](#scanner-profile). + scanner profile. For more details, see [scanner profiles](../dast/on-demand_scan.md#scanner-profile). 1. Select the desired **Site profile**, or select **Create site profile** and save a site - profile. For more details, see [site profiles](#site-profile). + profile. For more details, see [site profiles](../dast/on-demand_scan.md#site-profile). 1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the options you selected. 1. Do one of the following: @@ -466,403 +466,6 @@ variables: The DAST job does not require the project's repository to be present when running, so by default [`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. -## On-demand scans - -> - Auditing for DAST profile management [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. -> - Scheduled on-demand DAST scans [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3 [with a flag](../../../administration/feature_flags.md) named `dast_on_demand_scans_scheduler`. Disabled by default. -> - Scheduled on-demand DAST scans [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. Feature flag `dast_on_demand_scans_scheduler` removed. -> - Runner tags selection [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111499) in GitLab 16.3. - -WARNING: -On-demand scans are not available when GitLab is running in FIPS mode. - -An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger -the scan. You must either start it manually, or schedule it to run. For on-demand DAST scans, -a [site profile](#site-profile) defines **what** is to be scanned, and a -[scanner profile](#scanner-profile) defines **how** the application is to be scanned. - -An on-demand scan can be run in active or passive mode: - -- **Passive mode**: The default mode, which runs a ZAP Baseline Scan. -- **Active mode**: Runs a ZAP Full Scan which is potentially harmful to the site being scanned. To - minimize the risk of accidental damage, running an active scan requires a - [validated site profile](#site-profile-validation). - -### View on-demand DAST scans - -To view on-demand scans: - -1. On the left sidebar, select **Search or go to** and find your project or group. -1. Select **Secure > On-demand scans**. - -On-demand scans are grouped by their status. The scan library contains all available on-demand -scans. - -### Run an on-demand DAST scan - -Prerequisites: - -- You must have permission to run an on-demand DAST scan against a protected branch. The default - branch is automatically protected. For more information, see - [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). - -To run an existing on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the scan's row, select **Run scan**. - - If the branch saved in the scan no longer exists, you must: - - 1. [Edit the scan](#edit-an-on-demand-scan). - 1. Select a new branch. - 1. Save the edited scan. - -The on-demand DAST scan runs, and the project's dashboard shows the results. - -#### Create an on-demand scan - -Create an on-demand scan to: - -- Run it immediately. -- Save it to be run in the future. -- Schedule it to be run at a specified schedule. - -To create an on-demand DAST scan: - -1. On the left sidebar, select **Search or go to** and find your project or group. -1. Select **Secure > On-demand scans**. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** fields. -1. In the **Branch** dropdown list, select the desired branch. -1. Optional. Select the runner tags. -1. Select **Select scanner profile** or **Change scanner profile** to open the drawer, and either: - - Select a scanner profile from the drawer, **or** - - Select **New profile**, create a [scanner profile](#scanner-profile), then select **Save profile**. -1. Select **Select site profile** or **Change site profile** to open the drawer, and either: - - Select a site profile from the **Site profile library** drawer, or - - Select **New profile**, create a [site profile](#site-profile), then select **Save profile**. -1. To run the on-demand scan: - - - Immediately, select **Save and run scan**. - - In the future, select **Save scan**. - - On a schedule: - - - Turn on the **Enable scan schedule** toggle. - - Complete the schedule fields. - - Select **Save scan**. - -The on-demand DAST scan runs as specified and the project's dashboard shows the results. - -### View details of an on-demand scan - -To view details of an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. - -### Edit an on-demand scan - -To edit an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. -1. Edit the saved scan's details. -1. Select **Save scan**. - -### Delete an on-demand scan - -To delete an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. -1. On the confirmation dialog, select **Delete**. - -## Site profile - -> - Site profile features, scan method and file URL, were [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. -> - GraphQL endpoint path feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378692) in GitLab 15.7. - -A site profile defines the attributes and configuration details of the deployed application, -website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and -on-demand scans. - -A site profile contains: - -- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced - in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Site type**: The type of target to be scanned, either website or API scan. -- **Target URL**: The URL that DAST runs against. -- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. -- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. -- **Authentication**: - - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. - - **Username**: The username used to authenticate to the website. - - **Password**: The password used to authenticate to the website. - - **Username form field**: The name of username field at the sign-in HTML form. - - **Password form field**: The name of password field at the sign-in HTML form. - - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. - -- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, HTTP Archive (HAR), or GraphQL. - - **GraphQL endpoint path**: The path to the GraphQL endpoint. This path is concatenated with the target URL to provide the URI for the scan to test. The GraphQL endpoint must support introspection queries. - - **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. - -When an API site type is selected, a host override is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. - -When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. -This data can only be read and decrypted with a valid secrets file. - -### Site profile validation - -> Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. - -Site profile validation reduces the risk of running an active scan against the wrong website. A site -must be validated before an active scan can run against it. Each of the site validation methods are -equivalent in functionality, so use whichever is most suitable: - -- **Text file validation**: Requires a text file be uploaded to the target site. The text file is - allocated a name and content that is unique to the project. The validation process checks the - file's content. -- **Header validation**: Requires the header `Gitlab-On-Demand-DAST` be added to the target site, - with a value unique to the project. The validation process checks that the header is present, and - checks its value. -- **Meta tag validation**: Requires the meta tag named `gitlab-dast-validation` be added to the - target site, with a value unique to the project. Make sure it's added to the `<head>` section of - the page. The validation process checks that the meta tag is present, and checks its value. - -### Create a site profile - -To create a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select **New > Site profile**. -1. Complete the fields then select **Save profile**. - -The site profile is saved, for use in an on-demand scan. - -### Edit a site profile - -NOTE: -If a site profile is linked to a security policy, you cannot edit the profile from this page. See -[Scan execution policies](../policies/scan-execution-policies.md) for more information. - -NOTE: -If a site profile's Target URL or Authenticated URL is updated, the request headers and password fields associated with that profile are cleared. - -When a validated site profile's file, header, or meta tag is edited, the site's -[validation status](#site-profile-validation) is revoked. - -To edit a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the fields then select **Save profile**. - -### Delete a site profile - -NOTE: -If a site profile is linked to a security policy, a user cannot delete the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) for more information. - -To delete a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete** to confirm the deletion. - -### Validate a site profile - -Validating a site is required to run an active scan. - -Prerequisites: - -- A runner must be available in the project to run a validation job. -- The GitLab server's certificate must be trusted and must not use a self-signed certificate. - -To validate a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Validate**. -1. Select the validation method. - 1. For **Text file validation**: - 1. Download the validation file listed in **Step 2**. - 1. Upload the validation file to the host, to the location in **Step 3** or any location you - prefer. - 1. If required, edit the file location in **Step 3**. - 1. Select **Validate**. - 1. For **Header validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the header of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the header. - 1. Select **Validate**. - 1. For **Meta tag validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the content of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the meta tag. - 1. Select **Validate**. - -The site is validated and an active scan can run against it. A site profile's validation status is -revoked only when it's revoked manually, or its file, header, or meta tag is edited. - -### Retry a failed validation - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. -> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. -> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. - -Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** -page. - -To retry a site profile's failed validation: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Retry validation**. - -### Revoke a site profile's validation status - -WARNING: -When a site profile's validation status is revoked, all site profiles that share the same URL also -have their validation status revoked. - -To revoke a site profile's validation status: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Beside the validated profile, select **Revoke validation**. - -The site profile's validation status is revoked. - -### Validated site profile headers - -The following are code samples of how you can provide the required site profile header in your -application. - -#### Ruby on Rails example for on-demand scan - -Here's how you can add a custom header in a Ruby on Rails application: - -```ruby -class DastWebsiteTargetController < ActionController::Base - def dast_website_target - response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - head :ok - end -end -``` - -#### Django example for on-demand scan - -Here's how you can add a -[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): - -```python -class DastWebsiteTargetView(View): - def head(self, *args, **kwargs): - response = HttpResponse() - response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - - return response -``` - -#### Node (with Express) example for on-demand scan - -Here's how you can add a -[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): - -```javascript -app.get('/dast-website-target', function(req, res) { - res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') - res.send('Respond to DAST ping') -}) -``` - -## Scanner profile - -A scanner profile defines the configuration details of a security scanner. A scanner profile can be -referenced in `.gitlab-ci.yml` and on-demand scans. - -A scanner profile contains: - -- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner - profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. -- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. -- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before - starting the scan. -- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. -- **Debug messages:** Include debug messages in the DAST console output. - -### Create a scanner profile - -To create a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select **New > Scanner profile**. -1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). -1. Select **Save profile**. - -### Edit a scanner profile - -NOTE: -If a scanner profile is linked to a security policy, you cannot edit the profile from this page. -For more information, see [Scan execution policies](../policies/scan-execution-policies.md). - -To edit a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Scanner profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the form. -1. Select **Save profile**. - -### Delete a scanner profile - -NOTE: -If a scanner profile is linked to a security policy, a user cannot delete the profile from this -page. For more information, see [Scan execution policies](../policies/scan-execution-policies.md). - -To delete a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Scanner profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete**. - -## Auditing - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -The creation, updating, and deletion of DAST profiles, DAST scanner profiles, -and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). - ## Reports The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index df8fbff720b..e69734403ea 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Perform Dynamic Application Security Testing (DAST) of web APIs to help discover bugs and potential security issues that other QA processes may miss. Use DAST API tests in addition to other [GitLab Secure](../index.md) security scanners and your own test processes. You can run DAST -API tests either as part your CI/CD workflow, [on-demand](../dast/proxy-based.md#on-demand-scans), or both. +API tests either as part your CI/CD workflow, [on-demand](../dast/on-demand_scan.md), or both. WARNING: Do not run DAST API testing against a production server. Not only can it perform _any_ function that @@ -417,15 +417,15 @@ The following is a summary of the variable scopes supported by the Postman Clien - **Global Environment (Global) scope** is a special pre-defined environment that is available throughout a workspace. We can also refer to the _global environment_ scope as the _global_ scope. The Postman Client allows exporting the global environment into a JSON file, which can be used with DAST API. - **Environment scope** is a named group of variables created by a user in the Postman Client. -The Postman Client supports a single active environment along with the global environment. The variables defined in an active user-created environment take precedence over variables defined in the global environment. The Postman Client allows exporting your environment into a JSON file, which can be used with DAST API. + The Postman Client supports a single active environment along with the global environment. The variables defined in an active user-created environment take precedence over variables defined in the global environment. The Postman Client allows exporting your environment into a JSON file, which can be used with DAST API. - **Collection scope** is a group of variables declared in a given collection. The collection variables are available to the collection where they have been declared and the nested requests or collections. Variables defined in the collection scope take precedence over the _global environment_ scope and also the _environment_ scope. -The Postman Client can export one or more collections into a JSON file, this JSON file contains selected collections, requests, and collection variables. + The Postman Client can export one or more collections into a JSON file, this JSON file contains selected collections, requests, and collection variables. - **DAST API Scope** is a new scope added by DAST API to allow users to provide extra variables, or override variables defined in other supported scopes. This scope is not supported by Postman. The _DAST API Scope_ variables are provided using a [custom JSON file format](#dast-api-scope-custom-json-file-format). - Override values defined in the environment or collection - Defining variables from scripts - Define a single row of data from the unsupported _data scope_ - **Data scope** is a group of variables in which their name and values come from JSON or CSV files. A Postman collection runner like [Newman](https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/) or [Postman Collection Runner](https://learning.postman.com/docs/running-collections/intro-to-collection-runs/) executes the requests in a collection as many times as entries have the JSON or CSV file. A good use case for these variables is to automate tests using scripts in Postman. -DAST API does **not** support reading data from a CSV or JSON file. + DAST API does **not** support reading data from a CSV or JSON file. - **Local scope** are variables that are defined in Postman scripts. DAST API does **not** support Postman scripts and by extension, variables defined in scripts. You can still provide values for the script-defined variables by defining them in one of the supported scopes, or our custom JSON format. Not all scopes are supported by DAST API and variables defined in scripts are not supported. The following table is sorted by broadest scope to narrowest scope. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 9d898ec0266..2570ce03005 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -69,11 +69,11 @@ WARNING: Dependency Scanning does not support runtime installation of compilers and interpreters. - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Dependency Scanning](https://www.youtube.com/watch?v=TBnfbGk4c4o) + For an overview, see [Dependency Scanning](https://www.youtube.com/watch?v=TBnfbGk4c4o) - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an interactive reading and how-to demo of this Dependency Scanning documentation, see [How to use dependency scanning tutorial hands-on GitLab Application Security part 3](https://youtu.be/ii05cMbJ4xQ?feature=shared) + For an interactive reading and how-to demo of this Dependency Scanning documentation, see [How to use dependency scanning tutorial hands-on GitLab Application Security part 3](https://youtu.be/ii05cMbJ4xQ?feature=shared) - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For other interactive reading and how-to demos, see [Get Started With GitLab Application Security Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KrUrjDoefSkgZLx5aJYFaF9) + For other interactive reading and how-to demos, see [Get Started With GitLab Application Security Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KrUrjDoefSkgZLx5aJYFaF9) ## Supported languages and package managers @@ -277,7 +277,10 @@ The following languages and dependency managers are supported: <li> <a id="notes-regarding-supported-languages-and-package-managers-7"></a> <p> - Support for <a href="https://www.scala-sbt.org/">sbt</a> 1.3 and above was added in GitLab 13.9. + <ul> + <li>Support for <a href="https://www.scala-sbt.org/">sbt</a> 1.3 and above was added in GitLab 13.9.</li> + <li>Support for sbt 1.0.x was <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/415835">deprecated in GitLab 16.8</a>.</li> + </ul> </p> </li> </ol> @@ -305,22 +308,22 @@ the project first. When a supported dependency file is detected, all dependencies, including transitive dependencies are analyzed. There is no limit to the depth of nested or transitive dependencies that are analyzed. -### Dependency analyzers +### Analyzers -Dependency Scanning supports the following official analyzers: +Dependency Scanning supports the following official +[Gemnasium-based](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) analyzers: - `gemnasium` - `gemnasium-maven` - `gemnasium-python` -Each of these supported Gemnasium-based Dependency Scanning analyzers exist in the following project: - -- [`gemnasium`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) - -The analyzers are published as Docker images, which Dependency Scanning uses -to launch dedicated containers for each analysis. You can also integrate a custom +The analyzers are published as Docker images, which Dependency Scanning uses to launch dedicated +containers for each analysis. You can also integrate a custom [security scanner](../../../development/integrations/secure.md). +Each analyzer is updated as new versions of Gemnasium are released. For more information, see the +analyzer [Release Process documentation](../../../development/sec/analyzer_development_guide.md#versioning-and-release-process). + ### How analyzers obtain dependency information GitLab analyzers obtain dependency information using one of the following two methods: @@ -675,10 +678,6 @@ Support for additional languages, dependency managers, and dependency files are | ------------------- | --------- | --------------- | ---------- | ----- | | [Poetry](https://python-poetry.org/) | Python | `pyproject.toml` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) | [GitLab#32774](https://gitlab.com/gitlab-org/gitlab/-/issues/32774) | -## Contribute your scanner - -The [Security Scanner Integration](../../../development/integrations/secure.md) documentation explains how to integrate other security scanners into GitLab. - ## Configuration Enable the dependency scanning analyzer to ensure it scans your application's dependencies for known @@ -712,7 +711,10 @@ To enable dependency scanning: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Build > Pipeline editor**. -1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file: +1. If no `.gitlab-ci.yml` file exists, select **Configure pipeline**, then delete the example + content. +1. Copy and paste the following to the bottom of the `.gitlab-ci.yml` file. If an `include` line + already exists, add only the `template` line below it. ```yaml include: @@ -721,14 +723,14 @@ To enable dependency scanning: 1. Select the **Validate** tab, then select **Validate pipeline**. - Continue if you see the message **Simulation completed successfully**. That indicates the file is - valid. + The message **Simulation completed successfully** confirms the file is valid. 1. Select the **Edit** tab. 1. Complete the fields. Do not use the default branch for the **Branch** field. -1. Select **Commit changes**. -1. Select **Code > Merge requests**. -1. Select the merge request just created. -1. Review the merge request, then select **Merge**. +1. Select the **Start a new merge request with these changes** checkbox, then select **Commit + changes**. +1. Complete the fields according to your standard workflow, then select **Create + merge request**. +1. Review and edit the merge request according to your standard workflow, then select **Merge**. Pipelines now include a dependency scanning job. @@ -804,10 +806,10 @@ The following variables allow configuration of global dependency scanning settin | CI/CD variables | Description | | ----------------------------|------------ | -| `ADDITIONAL_CA_CERT_BUNDLE` | Bundle of CA certs to trust. The bundle of certificates provided here is also used by other tools during the scanning process, such as `git`, `yarn`, or `npm`. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | -| `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](#dependency-analyzers). | +| `ADDITIONAL_CA_CERT_BUNDLE` | Bundle of CA certificates to trust. The bundle of certificates provided here is also used by other tools during the scanning process, such as `git`, `yarn`, or `npm`. For more details, see [Custom TLS certificate authority](#custom-tls-certificate-authority). | +| `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Analyzers](#analyzers). | | `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | -| `DS_IMAGE_SUFFIX` | Suffix added to the image name. (Introduced in GitLab 14.10. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/354796`). Automatically set to `"-fips"` when FIPS mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357922) in GitLab 15.0.) | +| `DS_IMAGE_SUFFIX` | Suffix added to the image name. (GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/354796`). Automatically set to `"-fips"` when FIPS mode is enabled. | | `DS_MAX_DEPTH` | Defines how many directory levels deep that the analyzer should search for supported files to scan. A value of `-1` scans all directories regardless of depth. Default: `2`. | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). | @@ -816,29 +818,29 @@ The following variables allow configuration of global dependency scanning settin The following variables configure the behavior of specific dependency scanning analyzers. | CI/CD variable | Analyzer | Default | Description | -|--------------------------------------| ------------------ | ---------------------------- |------------ | +|--------------------------------------|--------------------|------------------------------|-------------| | `GEMNASIUM_DB_LOCAL_PATH` | `gemnasium` | `/gemnasium-db` | Path to local Gemnasium database. | -| `GEMNASIUM_DB_UPDATE_DISABLED` | `gemnasium` | `"false"` | Disable automatic updates for the `gemnasium-db` advisory database (For usage see: [examples](#hosting-a-copy-of-the-gemnasium_db-advisory-database))| +| `GEMNASIUM_DB_UPDATE_DISABLED` | `gemnasium` | `"false"` | Disable automatic updates for the `gemnasium-db` advisory database. For usage see [Hosting a copy of the Gemnasium advisory database](#hosting-a-copy-of-the-gemnasium_db-advisory-database). | | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"`, `"false"` in FIPS mode | Enable automatic remediation of vulnerable dependencies. Not supported in FIPS mode. | -| `DS_REMEDIATE_TIMEOUT` | `gemnasium` | `5m` | Timeout for auto-remediation. | +| `DS_REMEDIATE_TIMEOUT` | `gemnasium` | `5m` | Timeout for auto-remediation. | | `GEMNASIUM_LIBRARY_SCAN_ENABLED` | `gemnasium` | `"true"` | Enable detecting vulnerabilities in vendored JavaScript libraries. For now, `gemnasium` leverages [`Retire.js`](https://github.com/RetireJS/retire.js) to do this job. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350512) in GitLab 14.8. | -| `DS_JAVA_VERSION` | `gemnasium-maven` | `17` | Version of Java. Available versions: `8`, `11`, `17`, `21` | +| `DS_INCLUDE_DEV_DEPENDENCIES` | `gemnasium` | `"true"` | When set to `"false"`, development dependencies and their vulnerabilities are not reported. Only projects using Composer, npm, pnpm, Pipenv or Poetry are supported. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227861) in GitLab 15.1. | +| `GOOS` | `gemnasium` | `"linux"` | The operating system for which to compile Go code. | +| `GOARCH` | `gemnasium` | `"amd64"` | The architecture of the processor for which to compile Go code. | +| `GOFLAGS` | `gemnasium` | | The flags passed to the `go build` tool. | +| `GOPRIVATE` | `gemnasium` | | A list of glob patterns and prefixes to be fetched from source. For more information, see the Go private modules [documentation](https://go.dev/ref/mod#private-modules). | +| `DS_JAVA_VERSION` | `gemnasium-maven` | `17` | Version of Java. Available versions: `8`, `11`, `17`, `21`. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | | `GRADLE_CLI_OPTS` | `gemnasium-maven` | | List of command line arguments that are passed to `gradle` by the analyzer. | | `SBT_CLI_OPTS` | `gemnasium-maven` | | List of command-line arguments that the analyzer passes to `sbt`. | | `PIP_INDEX_URL` | `gemnasium-python` | `https://pypi.org/simple` | Base URL of Python Package Index. | | `PIP_EXTRA_INDEX_URL` | `gemnasium-python` | | Array of [extra URLs](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-extra-index-url) of package indexes to use in addition to `PIP_INDEX_URL`. Comma-separated. **Warning:** Read [the following security consideration](#python-projects) when using this environment variable. | -| `PIP_REQUIREMENTS_FILE` | `gemnasium-python` | | Pip requirements file to be scanned. | +| `PIP_REQUIREMENTS_FILE` | `gemnasium-python` | | Pip requirements file to be scanned. This is a filename and not a path. When this environment variable is set only the specified file is scanned. | | `PIPENV_PYPI_MIRROR` | `gemnasium-python` | | If set, overrides the PyPi index used by Pipenv with a [mirror](https://github.com/pypa/pipenv/blob/v2022.1.8/pipenv/environments.py#L263). | -| `DS_PIP_VERSION` | `gemnasium-python` | | Force the install of a specific pip version (example: `"19.3"`), otherwise the pip installed in the Docker image is used. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12811) in GitLab 12.7) | -| `DS_PIP_DEPENDENCY_PATH` | `gemnasium-python` | | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12412) in GitLab 12.2) | -| `DS_INCLUDE_DEV_DEPENDENCIES` | `gemnasium` | `"true"` | When set to `"false"`, development dependencies and their vulnerabilities are not reported. Only Composer, NPM, and Poetry projects are supported. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227861) in GitLab 15.1. | -| `GOOS` | `gemnasium` | `"linux"` | The operating system for which to compile Go code. | -| `GOARCH` | `gemnasium` | `"amd64"` | The architecture of the processor for which to compile Go code. | -| `GOFLAGS` | `gemnasium` | | The flags passed to the `go build` tool. | -| `GOPRIVATE` | `gemnasium` | | A list of glob patterns and prefixes to be fetched from source. Read the Go private modules [documentation](https://go.dev/ref/mod#private-modules) for more information. | +| `DS_PIP_VERSION` | `gemnasium-python` | | Force the install of a specific pip version (example: `"19.3"`), otherwise the pip installed in the Docker image is used. | +| `DS_PIP_DEPENDENCY_PATH` | `gemnasium-python` | | Path to load Python pip dependencies from. | #### Other variables @@ -869,9 +871,26 @@ If one does not work and you need it we suggest [submitting a feature request](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal%20-%20detailed&issue[title]=Docs%20feedback%20-%20feature%20proposal:%20Write%20your%20title) or [contributing to the code](../../../development/index.md) to enable it to be used. -### Using a custom SSL CA certificate authority +### Custom TLS certificate authority -You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://www.rfc-editor.org/rfc/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following: +Dependency Scanning allows for use of custom TLS certificates for SSL/TLS connections instead of the +default shipped with the analyzer container image. + +Support for custom certificate authorities was introduced in the following versions. + +| Analyzer | Version | +|--------------------|--------------------------------------------------------------------------------------------------------| +| `gemnasium` | [v2.8.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/releases/v2.8.0) | +| `gemnasium-maven` | [v2.9.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/releases/v2.9.0) | +| `gemnasium-python` | [v2.7.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/releases/v2.7.0) | + +#### Using a custom TLS certificate authority + +To use a custom TLS certificate authority, assign the +[text representation of the X.509 PEM public-key certificate](https://www.rfc-editor.org/rfc/rfc7468#section-5.1) +to the CI/CD variable `ADDITIONAL_CA_CERT_BUNDLE`. + +For example, to configure the certificate in the `.gitlab-ci.yml` file: ```yaml variables: @@ -883,8 +902,6 @@ variables: -----END CERTIFICATE----- ``` -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/index.md#for-a-project), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. - ### Using private Maven repositories If your private Maven repository requires login credentials, @@ -892,144 +909,50 @@ you can use the `MAVEN_CLI_OPTS` CI/CD variable. Read more on [how to use private Maven repositories](../index.md#using-private-maven-repositories). -#### FIPS-enabled images +### FIPS-enabled images -> Introduced in GitLab 14.10. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/354796` +> - Introduced in GitLab 14.10. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/354796` +> - Introduced in GitLab 15.0 - Gemnasium uses FIPS-enabled images when FIPS mode is enabled. GitLab also offers [FIPS-enabled Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) -versions of the Gemnasium images. You can therefore replace standard images with FIPS-enabled images. +versions of the Gemnasium images. When FIPS mode is enabled in the GitLab instance, Gemnasium +scanning jobs automatically use the FIPS-enabled images. To manually switch to FIPS-enabled images, +set the variable `DS_IMAGE_SUFFIX` to `"-fips"`. -Gemnasium scanning jobs automatically use FIPS-enabled image when FIPS mode is enabled in the GitLab instance. -([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357922) in GitLab 15.0.) +Dependency scanning for Gradle projects and auto-remediation for Yarn projects are not supported in FIPS mode. -To manually switch to FIPS-enabled images, set the variable `DS_IMAGE_SUFFIX` to `"-fips"`. +## Output -Dependency scanning for Gradle projects and auto-remediation for Yarn projects are not supported in FIPS mode. +Dependency Scanning produces the following output: -## Reports JSON format +- **Dependency scanning report**: Contains details of all vulnerabilities detected in dependencies. +- **CycloneDX Software Bill of Materials**: Software Bill of Materials (SBOM) for each supported + lock or build file detected. -The dependency scanning tool emits a JSON report file. For more information, see the -[schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json). +### Dependency scanning report -Here's an example dependency scanning report: +Dependency scanning outputs a report containing details of all vulnerabilities. The report is +processed internally and the results are shown in the UI. The report is also output as an artifact +of the dependency scanning job, named `gl-dependency-scanning-report.json`. -```json -{ - "version": "2.0", - "vulnerabilities": [ - { - "id": "51e83874-0ff6-4677-a4c5-249060554eae", - "category": "dependency_scanning", - "name": "Regular Expression Denial of Service", - "message": "Regular Expression Denial of Service in debug", - "description": "The debug module is vulnerable to regular expression denial of service when untrusted user input is passed into the `o` formatter. It takes around 50k characters to block for 2 seconds making this a low severity issue.", - "severity": "Unknown", - "solution": "Upgrade to latest versions.", - "scanner": { - "id": "gemnasium", - "name": "Gemnasium" - }, - "location": { - "file": "yarn.lock", - "dependency": { - "package": { - "name": "debug" - }, - "version": "1.0.5" - } - }, - "identifiers": [ - { - "type": "gemnasium", - "name": "Gemnasium-37283ed4-0380-40d7-ada7-2d994afcc62a", - "value": "37283ed4-0380-40d7-ada7-2d994afcc62a", - "url": "https://deps.sec.gitlab.com/packages/npm/debug/versions/1.0.5/advisories" - } - ], - "links": [ - { - "url": "https://nodesecurity.io/advisories/534" - }, - { - "url": "https://github.com/visionmedia/debug/issues/501" - }, - { - "url": "https://github.com/visionmedia/debug/pull/504" - } - ] - }, - { - "id": "5d681b13-e8fa-4668-957e-8d88f932ddc7", - "category": "dependency_scanning", - "name": "Authentication bypass via incorrect DOM traversal and canonicalization", - "message": "Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js", - "description": "Some XML DOM traversal and canonicalization APIs may be inconsistent in handling of comments within XML nodes. Incorrect use of these APIs by some SAML libraries results in incorrect parsing of the inner text of XML nodes such that any inner text after the comment is lost prior to cryptographically signing the SAML message. Text after the comment, therefore, has no impact on the signature on the SAML message.\r\n\r\nA remote attacker can modify SAML content for a SAML service provider without invalidating the cryptographic signature, which may allow attackers to bypass primary authentication for the affected SAML service provider.", - "severity": "Unknown", - "solution": "Upgrade to fixed version.\r\n", - "scanner": { - "id": "gemnasium", - "name": "Gemnasium" - }, - "location": { - "file": "yarn.lock", - "dependency": { - "package": { - "name": "saml2-js" - }, - "version": "1.5.0" - } - }, - "identifiers": [ - { - "type": "gemnasium", - "name": "Gemnasium-9952e574-7b5b-46fa-a270-aeb694198a98", - "value": "9952e574-7b5b-46fa-a270-aeb694198a98", - "url": "https://deps.sec.gitlab.com/packages/npm/saml2-js/versions/1.5.0/advisories" - }, - { - "type": "cve", - "name": "CVE-2017-11429", - "value": "CVE-2017-11429", - "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11429" - } - ], - "links": [ - { - "url": "https://github.com/Clever/saml2/commit/3546cb61fd541f219abda364c5b919633609ef3d#diff-af730f9f738de1c9ad87596df3f6de84R279" - }, - { - "url": "https://github.com/Clever/saml2/issues/127" - }, - { - "url": "https://www.kb.cert.org/vuls/id/475445" - } - ] - } - ], - "remediations": [ - { - "fixes": [ - { - "id": "5d681b13-e8fa-4668-957e-8d88f932ddc7", - } - ], - "summary": "Upgrade saml2-js", - "diff": "ZGlmZiAtLWdpdCBhL...OR0d1ZUc2THh3UT09Cg==" // some content is omitted for brevity - } - ] -} -``` +For more details of the dependency scanning report, see: + +- [Example dependency scanning report](#example-vulnerability-report). +- [Dependency scanning report schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json). ### CycloneDX Software Bill of Materials > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350509) in GitLab 14.8 in [Beta](../../../policy/experiment-beta-support.md#beta). > - Generally available in GitLab 15.7. -In addition to the [JSON report file](#reports-json-format), the [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) -Dependency Scanning tool outputs a [CycloneDX](https://cyclonedx.org/) Software Bill of Materials (SBOM) for -each supported lock or build file it detects. These CycloneDX SBOMs are named -`gl-sbom-<package-type>-<package-manager>.cdx.json`, and are saved in the same directory -as the detected lock or build files. +Dependency Scanning outputs a [CycloneDX](https://cyclonedx.org/) Software Bill of Materials (SBOM) +for each supported lock or build file it detects. + +The CycloneDX SBOMs are: + +- Named `gl-sbom-<package-type>-<package-manager>.cdx.json`. +- Available as job artifacts of the dependency scanning job. +- Saved in the same directory as the detected lock or build files. For example, if your project has the following structure: @@ -1063,12 +986,16 @@ Then the Gemnasium scanner generates the following CycloneDX SBOMs: └── gl-sbom-go-go.cdx.json ``` -You can download CycloneDX SBOMs [the same way as other job artifacts](../../../ci/jobs/job_artifacts.md#download-job-artifacts). +#### Merging multiple CycloneDX SBOMs -### Merging multiple CycloneDX SBOMs +You can use a CI/CD job to merge the multiple CycloneDX SBOMs into a single SBOM. GitLab uses +[CycloneDX Properties](https://cyclonedx.org/use-cases/#properties--name-value-store) to store +implementation-specific details in the metadata of each CycloneDX SBOM, such as the location of +build and lock files. If multiple CycloneDX SBOMs are merged together, this information is removed +from the resulting merged file. -You can use a CI/CD job to merge multiple CycloneDX SBOMs into a single SBOM. -For example: +For example, the following `.gitlab-ci.yml` extract demonstrates how the Cyclone SBOM files can be +merged, and the resulting file validated. ```yaml stages: @@ -1110,15 +1037,6 @@ merge cyclonedx sboms: - gl-sbom-all.cdx.json ``` -GitLab uses [CycloneDX Properties](https://cyclonedx.org/use-cases/#properties--name-value-store) -to store implementation-specific details in the metadata of each CycloneDX SBOM, -such as the location of build and lock files. If multiple CycloneDX SBOMs are merged together, -this information is removed from the resulting merged file. - -## Versioning and release process - -Check the [Release Process documentation](../../../development/sec/analyzer_development_guide.md#versioning-and-release-process). - ## Contributing to the vulnerability database To find a vulnerability, you can search the [`GitLab Advisory Database`](https://advisories.gitlab.com/). @@ -1170,16 +1088,6 @@ For details on saving and transporting Docker images as a file, see the Docker d [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). -#### Support for Custom Certificate Authorities - -Support for custom certificate authorities was introduced in the following versions. - -| Analyzer | Version | -| -------- | ------- | -| `gemnasium` | [v2.8.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/releases/v2.8.0) | -| `gemnasium-maven` | [v2.9.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/releases/v2.9.0) | -| `gemnasium-python` | [v2.7.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/releases/v2.7.0) | - ### Set dependency scanning CI/CD job variables to use local dependency scanning analyzers Add the following configuration to your `.gitlab-ci.yml` file. You must change the value of @@ -1332,3 +1240,114 @@ environment variable due to a possible exploit documented by [CVE-2018-20225](ht intended to obtain a private package from a private index. This only affects use of the `PIP_EXTRA_INDEX_URL` option, and exploitation requires that the package does not already exist in the public index (and thus the attacker can put the package there with an arbitrary version number). + +## Example vulnerability report + +The following is an example vulnerability report output by dependency scanning: + +```json +{ + "version": "2.0", + "vulnerabilities": [ + { + "id": "51e83874-0ff6-4677-a4c5-249060554eae", + "category": "dependency_scanning", + "name": "Regular Expression Denial of Service", + "message": "Regular Expression Denial of Service in debug", + "description": "The debug module is vulnerable to regular expression denial of service when untrusted user input is passed into the `o` formatter. It takes around 50k characters to block for 2 seconds making this a low severity issue.", + "severity": "Unknown", + "solution": "Upgrade to latest versions.", + "scanner": { + "id": "gemnasium", + "name": "Gemnasium" + }, + "location": { + "file": "yarn.lock", + "dependency": { + "package": { + "name": "debug" + }, + "version": "1.0.5" + } + }, + "identifiers": [ + { + "type": "gemnasium", + "name": "Gemnasium-37283ed4-0380-40d7-ada7-2d994afcc62a", + "value": "37283ed4-0380-40d7-ada7-2d994afcc62a", + "url": "https://deps.sec.gitlab.com/packages/npm/debug/versions/1.0.5/advisories" + } + ], + "links": [ + { + "url": "https://nodesecurity.io/advisories/534" + }, + { + "url": "https://github.com/visionmedia/debug/issues/501" + }, + { + "url": "https://github.com/visionmedia/debug/pull/504" + } + ] + }, + { + "id": "5d681b13-e8fa-4668-957e-8d88f932ddc7", + "category": "dependency_scanning", + "name": "Authentication bypass via incorrect DOM traversal and canonicalization", + "message": "Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js", + "description": "Some XML DOM traversal and canonicalization APIs may be inconsistent in handling of comments within XML nodes. Incorrect use of these APIs by some SAML libraries results in incorrect parsing of the inner text of XML nodes such that any inner text after the comment is lost prior to cryptographically signing the SAML message. Text after the comment, therefore, has no impact on the signature on the SAML message.\r\n\r\nA remote attacker can modify SAML content for a SAML service provider without invalidating the cryptographic signature, which may allow attackers to bypass primary authentication for the affected SAML service provider.", + "severity": "Unknown", + "solution": "Upgrade to fixed version.\r\n", + "scanner": { + "id": "gemnasium", + "name": "Gemnasium" + }, + "location": { + "file": "yarn.lock", + "dependency": { + "package": { + "name": "saml2-js" + }, + "version": "1.5.0" + } + }, + "identifiers": [ + { + "type": "gemnasium", + "name": "Gemnasium-9952e574-7b5b-46fa-a270-aeb694198a98", + "value": "9952e574-7b5b-46fa-a270-aeb694198a98", + "url": "https://deps.sec.gitlab.com/packages/npm/saml2-js/versions/1.5.0/advisories" + }, + { + "type": "cve", + "name": "CVE-2017-11429", + "value": "CVE-2017-11429", + "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11429" + } + ], + "links": [ + { + "url": "https://github.com/Clever/saml2/commit/3546cb61fd541f219abda364c5b919633609ef3d#diff-af730f9f738de1c9ad87596df3f6de84R279" + }, + { + "url": "https://github.com/Clever/saml2/issues/127" + }, + { + "url": "https://www.kb.cert.org/vuls/id/475445" + } + ] + } + ], + "remediations": [ + { + "fixes": [ + { + "id": "5d681b13-e8fa-4668-957e-8d88f932ddc7", + } + ], + "summary": "Upgrade saml2-js", + "diff": "ZGlmZiAtLWdpdCBhL...OR0d1ZUc2THh3UT09Cg==" // some content is omitted for brevity + } + ] +} +``` diff --git a/doc/user/application_security/dependency_scanning/troubleshooting_dependency_scanning.md b/doc/user/application_security/dependency_scanning/troubleshooting_dependency_scanning.md index 77579a04c7e..83004459051 100644 --- a/doc/user/application_security/dependency_scanning/troubleshooting_dependency_scanning.md +++ b/doc/user/application_security/dependency_scanning/troubleshooting_dependency_scanning.md @@ -65,10 +65,6 @@ Consider updating to Docker `19.03.1` or greater. Older versions are not affected. Read more in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/13830#note_211354992 "Current SAST container fails"). -## Getting warning message `gl-dependency-scanning-report.json: no matching files` - -For information, see the [general Application Security troubleshooting section](../../../ci/jobs/job_artifacts_troubleshooting.md#error-message-no-files-to-upload). - ## Limitation when using rules:exists The [dependency scanning CI template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) @@ -129,7 +125,7 @@ The lock file is cached during the build phase and passed to the dependency scan scan occurs. Because the cache is downloaded before the analyzer run occurs, the existence of a lock file in the `CI_BUILDS_DIR` directory triggers the dependency scanning job. -To prevent this warning, lock files should be committed. +To prevent this warning, lock files should be committed. ## You no longer get the latest Docker image after setting `DS_MAJOR_VERSION` or `DS_ANALYZER_IMAGE` diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index e31877d195a..6441f74a41b 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -247,10 +247,8 @@ Security scan information appears in multiple locations and formats: ### Merge request **(FREE ALL)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4393) in GitLab Free 13.5. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/273205) in 13.6. -> - Report download dropdown list [added](https://gitlab.com/gitlab-org/gitlab/-/issues/273418) in 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249550) in GitLab 13.9. +Output of all enabled application security tools is shown in a merge request widget. You can use +this information to manage the risk of any issues identified in the source branch. #### All tiers @@ -284,17 +282,25 @@ The merge request security widget displays only a subset of the vulnerabilities From the merge request security widget, select **Expand** to unfold the widget, displaying any new and no longer detected (removed) findings by scan type. -For each security report type, the widget displays the first 25 added and 25 fixed findings, sorted by severity. To see all -findings, select **View full report** to go directly to the **Security** tab in the latest branch pipeline. +For each security report type, the widget displays the first 25 added and 25 fixed findings, sorted by severity. +This is determined by comparing the security reports from the source branch and target branch pipelines. + +As an example, consider two pipelines with these scan results: + +- The source branch pipeline detects two vulnerabilities identified as `V1` and `V2`. +- The target branch pipeline detects two vulnerabilities identified as `V1` and `V3`. +- `V2` will show on the merge request widget as "added". +- `V3` will show on the merge request widget as "fixed". +- `V1` exists on both branches and is not shown on the merge request widget. + +To see all findings on the source branch of the merge request, select **View full report** to go directly to the **Security** tab in the latest source branch pipeline.  ### Pipeline security tab -A pipeline's security tab lists all findings in the current branch. It includes findings introduced -by this branch and vulnerabilities already present in the base branch. These results likely do not -match the findings displayed in the Merge Request security widget, as those do not include the -existing vulnerabilities. For more information see +A pipeline's security tab lists all findings from the security reports in the pipeline's +job artifacts. For more information see [Vulnerabilities in a pipeline](vulnerability_report/pipeline.md). ### Security dashboard diff --git a/doc/user/application_security/policies/img/scan_results_evaluation_white-bg.png b/doc/user/application_security/policies/img/scan_results_evaluation_white-bg.png Binary files differindex d2f5466e383..ac3164842a4 100644 --- a/doc/user/application_security/policies/img/scan_results_evaluation_white-bg.png +++ b/doc/user/application_security/policies/img/scan_results_evaluation_white-bg.png diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index f299a38dff1..9a6f7581876 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -46,6 +46,10 @@ to remove the `test` stage, jobs will run in the `scan-policies` stage instead. - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video walkthrough, see [How to set up Security Scan Policies in GitLab](https://youtu.be/ZBcqGmEwORA?si=aeT4EXtmHjosgjBY). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Enforcing scan execution policies on projects with no GitLab CI/CD configuration](https://www.youtube.com/watch?v=sUfwQQ4-qHs). +## Requirements and limitations + +- The maximum number of scan execution policies is five per security policy project. + ## Scan execution policy editor NOTE: @@ -201,27 +205,22 @@ The keys for a schedule rule are: ## `scan` action type -> - Scan Execution Policies variable precedence was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/424028) in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `security_policies_variables_precedence`. Enabled by default. -> - The `custom` scan action type was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126457) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `compliance_pipeline_in_policies`. On GitLab.com, this feature is not available. On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `compliance_pipeline_in_policies`. +> Scan Execution Policies variable precedence was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/424028) in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `security_policies_variables_precedence`. Enabled by default. [Feature flag removed in GitLab 16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/435727). This action executes the selected `scan` with additional parameters when conditions for at least one rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `scan` | `string` | `sast`, `sast_iac`, `dast`, `secret_detection`, `container_scanning`, `dependency_scanning`, `custom` | The action's type. | -| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | -| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `scan` | `string` | `sast`, `sast_iac`, `dast`, `secret_detection`, `container_scanning`, `dependency_scanning` | The action's type. | +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/on-demand_scan.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/on-demand_scan.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | | `tags` | `array` of `string` | | A list of runner tags for the policy. The policy jobs are run by runner with the specified tags. | -| `ci_configuration` <sup>1</sup> | `string` | | GitLab CI YAML as formatted as string. | -| `ci_configuration_path` <sup>1</sup> | object | Object with project path and file name pointing to a CI configuration. | - -1. For `custom` scans, you must specify one of `ci_configuration` or `ci_configuration_path`. Note the following: -- You must create the [site profile](../dast/proxy-based.md#site-profile) and [scanner profile](../dast/proxy-based.md#scanner-profile) +- You must create the [site profile](../dast/on-demand_scan.md#site-profile) and [scanner profile](../dast/on-demand_scan.md#scanner-profile) with selected names for each project that is assigned to the selected Security Policy Project. Otherwise, the policy is not applied and a job with an error message is created instead. - Once you associate the site profile and scanner profile by name in the policy, it is not possible @@ -237,16 +236,6 @@ Note the following: - A container scanning scan that is configured for the `pipeline` rule type ignores the agent defined in the `agents` object. The `agents` object is only considered for `schedule` rule types. An agent with a name provided in the `agents` object must be created and configured for the project. - Variables defined in a Scan Execution Policy follow the standard [CI/CD variable precedence](../../../ci/variables/index.md#cicd-variable-precedence). -- `custom` scans are be executed for scheduled rules. -- Jobs variables and stages definitions from `custom` scans take precedence over the project's CI/CD configuration. - -### `ci_configuration_path` object - -| Field | Type | Description | -|-------|------|-------------| -| `project` | `string` | A project namespace path. | -| `file` | `string` | The filename of the CI/CD YAML file. | -| `ref` | `string` (optional) | The branch name, tag name, or commit SHA. | ## Example security policies project @@ -267,12 +256,6 @@ scan_execution_policy: - scan: dast scanner_profile: Scanner Profile A site_profile: Site Profile B - - scan: custom - ci_configuration: |- - test job: - stage: test - script: - - echo "Hello World" - name: Enforce DAST and secret detection scans every 10 minutes description: This policy enforces DAST and secret detection scans to run every 10 minutes enabled: true @@ -306,7 +289,6 @@ In this example: - For every pipeline executed on branches that match the `release/*` wildcard (for example, branch `release/v1.2.1`) - DAST scans run with `Scanner Profile A` and `Site Profile B`. - - A `test job` is injected into the `test` stage of the pipeline, printing `Hello World`. - DAST and secret detection scans run every 10 minutes. The DAST scan runs with `Scanner Profile C` and `Site Profile D`. - Secret detection, container scanning, and SAST scans run for every pipeline executed on the `main` @@ -340,3 +322,155 @@ this case, two SAST jobs run in the pipeline, one with the developer's variables If you want to avoid running duplicate scans, you can either remove the scans from the project's `.gitlab-ci.yml` file or disable your local jobs by setting `SAST_DISABLED: "true"`. Disabling jobs this way does not prevent the security jobs defined by scan execution policies from running. + +## Experimental features **(EXPERIMENT)** + +These experimental features have limitations: + +1. Enforcing pipeline execution using the pipeline execution action in projects + without a `.gitlab-ci.yml` is not supported. +1. The pipeline execution action cannot be used with a scheduled trigger type. + +### Pipeline execution policy action + +> The `custom` scan action type was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126457) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `compliance_pipeline_in_policies`. + +FLAG: +On self-managed GitLab, by default this feature is available. +To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `compliance_pipeline_in_policies`. +On GitLab.com, this feature is available. + +The pipeline execution policy action introduces a new scan action type into +scan execution policies for creating and enforcing custom CI in your target +development projects. + +This custom scan type uses a remote CI configuration file to define the custom +CI you want enforced. Scan execution policies then merge this file with the +project's `.gitlab-ci.yml` to execute the compliance jobs for each project +enforced by the policy. + +#### `ci_configuration_path` object + +| Field | Type | Required | Description | +|-----------|---------------------|----------|-------------| +| `project` | `string` | true | A project namespace path. | +| `file` | `string` | true | The file name of the CI/CD YAML file. | +| `ref` | `string` | false | The branch name, tag name, or commit SHA. If not specified, uses the default branch. | + +#### `scan` action type + +This action executes the selected `scan` with additional parameters when +conditions for at least one rule in the defined policy are met. + +| Field | Type | Possible values | Description | +|-------------------------|----------|-----------------|-------------| +| `scan` | `string` | `custom` | The action's type. | +| `ci_configuration` | `string` | | GitLab CI YAML as formatted as string. | +| `ci_configuration_path` | object | | Object with project path and file name pointing to a CI configuration. | + +Note the following: + +- For `custom` scans, you must specify one of `ci_configuration` or `ci_configuration_path`. +- `custom` scans are being executed for triggered rules only. +- Jobs variables and stages definitions from `custom` scans take precedence over the project's CI/CD configuration. + +#### Example security policies project + +You can use this example in a `.gitlab/security-policies/policy.yml` file stored in a +[security policy project](index.md#security-policy-project): + +```yaml +--- +scan_execution_policy: +- name: Create a custom scan that injects test job + description: This policy enforces pipeline configuration to have a job with DAST scan for release branches + enabled: true + rules: + - type: pipeline + branches: + - release/* + actions: + - scan: custom + ci_configuration: |- + test job: + stage: test + script: + - echo "Hello World" +``` + +In this example a `test job` is injected into the `test` stage of the pipeline, printing `Hello World`. + +### Security policy scopes + +> The `policy_scope` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135398) in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `security_policies_policy_scope`. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +an administrator can [disable the feature flag](../../../administration/feature_flags.md) +named `security_policies_policy_scope`. +On GitLab.com, this feature is available. + +Security policy enforcement depends first on establishing a link between the group, subgroup, or +project on which you want to enforce policies, and the security policy project that contains the +policies. For example, if you are linking policies to a group, a group owner must create the link to +the security policy project. Then, all policies in the security policy project are inherited by all +projects in the group. + +You can refine a security policy's scope to: + +- _Include_ only projects containing a compliance framework label. +- _Include_ or _exclude_ selected projects from enforcement. + +#### Policy scope schema + +| Field | Type | Required | Possible values | Description | +|-------|------|----------|-----------------|-------------| +| `policy_scope` | `object` | false | `compliance_frameworks`, `projects` | Scopes the policy based on compliance framework labels or projects you define. | + +#### `policy_scope` scope type + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `compliance_frameworks` | `object` | `ids` | List of IDs of the compliance frameworks in scope of enforcement, in an `ids` array. | +| `projects` | `object` | `including`, `excluding` | Use `excluding:` or `including:` then list the IDs of the projects you wish to include or exclude, in an `ids` array. | + +#### Example `policy.yml` with security policy scopes + +```yaml +--- +scan_execution_policy: +- name: Enforce DAST in every release pipeline + description: This policy enforces pipeline configuration to have a job with DAST scan for release branches + enabled: true + rules: + - type: pipeline + branches: + - release/* + actions: + - scan: dast + scanner_profile: Scanner Profile A + site_profile: Site Profile B + policy_scope: + compliance_frameworks: + ids: + - 2 + - 11 +- name: Enforce Secret Detection and Container Scanning in every default branch pipeline + description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch + enabled: true + rules: + - type: pipeline + branches: + - main + actions: + - scan: secret_detection + - scan: sast + variables: + SAST_EXCLUDED_ANALYZERS: brakeman + policy_scope: + projects: + excluding: + ids: + - 24 + - 27 +``` diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 0fadd761fe4..e2ec6b8ae56 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -31,7 +31,7 @@ The following video gives you an overview of GitLab scan result policies: - You must add the respective [security scanning tools](../index.md#application-coverage). Otherwise, scan result policies do not have any effect. -- The maximum number of policies is five. +- The maximum number of scan result policies is five per security policy project. - Each policy can have a maximum of five rules. - All configured scanners must be present in the merge request's latest pipeline. If not, approvals are required even if some vulnerability criteria have not been met. - Scan result policies evaluate findings and determine approval requirements based on the job artifact reports published in a completed pipeline. However, scan result policies do not check the integrity or authenticity of the scan results generated in the artifact reports. @@ -45,10 +45,10 @@ A project can have multiple pipeline types configured. A single commit can initi pipelines, each of which may contain a security scan. - In GitLab 16.3 and later, the results of all completed pipelines for the latest commit in -the merge request's source and target branch are evaluated and used to enforce the scan result policy. -Parent-child pipelines and on-demand DAST pipelines are not considered. + the merge request's source and target branch are evaluated and used to enforce the scan result policy. + Parent-child pipelines and on-demand DAST pipelines are not considered. - In GitLab 16.2 and earlier, only the results of the latest completed pipeline were evaluated -when enforcing scan result policies. + when enforcing scan result policies. ## Scan result policy editor @@ -92,9 +92,6 @@ the following sections and tables provide an alternative. > The `approval_settings` fields were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418752) in GitLab 16.4 [with flags](../../../administration/feature_flags.md) named `scan_result_policies_block_unprotecting_branches`, `scan_result_any_merge_request`, or `scan_result_policies_block_force_push`. See the `approval_settings` section below for more information. -FLAG: -On self-managed GitLab, by default the `approval_settings` field is available. To hide the feature, an administrator can [disable the feature flags](../../../administration/feature_flags.md) named `scan_result_policies_block_unprotecting_branches`, `scan_result_any_merge_request` and `scan_result_policies_block_force_push`. See the `approval_settings` section below for more information. On GitLab.com, the `approval_settings` field is available. - | Field | Type | Required | Possible values | Description | |---------------------|--------------------|----------|-----------------|----------------------------------------------------------| | `name` | `string` | true | | Name of the policy. Maximum of 255 characters. | @@ -121,7 +118,7 @@ This rule enforces the defined actions based on security scan findings. | `scanners` | `array` of `string` | true | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. `sast` includes results from both SAST and SAST IaC scanners. | | `vulnerabilities_allowed` | `integer` | true | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | | `severity_levels` | `array` of `string` | true | `info`, `unknown`, `low`, `medium`, `high`, `critical` | The severity levels for this rule to consider. | -| `vulnerability_states` | `array` of `string` | true | `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed`, `new_needs_triage`, `new_dismissed` | All vulnerabilities fall into two categories:<br><br>**Newly Detected Vulnerabilities** - the `newly_detected` policy option covers vulnerabilities identified in the merge request branch itself but that do not currently exist on the default branch. This policy option requires a pipeline to complete before the rule is evaluated so that it knows whether vulnerabilities are newly detected or not. Merge requests are blocked until the pipeline and necessary security scans are complete. The `newly_detected` option considers both of the following statuses:<br><br> • Detected<br> • Dismissed<br><br> The `new_needs_triage` option considers the status<br><br> • Detected<br><br> The `new_dismissed` option considers the status<br><br> • Dismissed<br><br>**Pre-Existing Vulnerabilities** - these policy options are evaluated immediately and do not require a pipeline complete as they consider only vulnerabilities previously detected in the default branch.<br><br> • `Detected` - the policy looks for vulnerabilities in the detected state.<br> • `Confirmed` - the policy looks for vulnerabilities in the confirmed state.<br> • `Dismissed` - the policy looks for vulnerabilities in the dismissed state.<br> • `Resolved` - the policy looks for vulnerabilities in the resolved state. | +| `vulnerability_states` | `array` of `string` | true | `[]` or `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed`, `new_needs_triage`, `new_dismissed` | All vulnerabilities fall into two categories:<br><br>**Newly Detected Vulnerabilities** - the `newly_detected` policy option covers vulnerabilities identified in the merge request branch itself but that do not currently exist on the default branch. This policy option requires a pipeline to complete before the rule is evaluated so that it knows whether vulnerabilities are newly detected or not. Merge requests are blocked until the pipeline and necessary security scans are complete. The `newly_detected` option considers both of the following statuses:<br><br> • Detected<br> • Dismissed<br><br> The `new_needs_triage` option considers the status<br><br> • Detected<br><br> The `new_dismissed` option considers the status<br><br> • Dismissed<br><br>**Pre-Existing Vulnerabilities** - these policy options are evaluated immediately and do not require a pipeline complete as they consider only vulnerabilities previously detected in the default branch.<br><br> • `Detected` - the policy looks for vulnerabilities in the detected state.<br> • `Confirmed` - the policy looks for vulnerabilities in the confirmed state.<br> • `Dismissed` - the policy looks for vulnerabilities in the dismissed state.<br> • `Resolved` - the policy looks for vulnerabilities in the resolved state. <br><br>An empty array, `[]`, covers the same statuses as `newly_detected`. It is equivalent to specifying `['new_needs_triage', 'new_dismissed']`. | | `vulnerability_attributes` | `object` | false | `{false_positive: boolean, fix_available: boolean}` | All vulnerability findings are considered by default. But filters can be applied for attributes to consider only vulnerability findings: <br><br> • With a fix available (`fix_available: true`)<br><br> • With no fix available (`fix_available: false`)<br> • That are false positive (`false_positive: true`)<br> • That are not false positive (`false_positive: false`)<br> • Or a combination of both. For example (`fix_available: true, false_positive: false`) | | `vulnerability_age` | `object` | false | N/A | Filter pre-existing vulnerability findings by age. A vulnerability's age is calculated as the time since it was detected in the project. The criteria are `operator`, `value`, and `interval`.<br>- The `operator` criterion specifies if the age comparison used is older than (`greater_than`) or younger than (`less_than`).<br>- The `value` criterion specifies the numeric value representing the vulnerability's age.<br>- The `interval` criterion specifies the unit of measure of the vulnerability's age: `day`, `week`, `month`, or `year`.<br><br>Example: `operator: greater_than`, `value: 30`, `interval: day`. | @@ -179,6 +176,7 @@ the defined policy. ## `approval_settings` +> - The `block_group_branch_modification` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/420724) in GitLab 16.8 [with flag](../../../administration/feature_flags.md) named `scan_result_policy_block_group_branch_modification`. Disabled by default. > - The `block_unprotecting_branches` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/423101) in GitLab 16.4 [with flag](../../../administration/feature_flags.md) named `scan_result_policy_settings`. Disabled by default. > - The `scan_result_policy_settings` feature flag was replaced by the `scan_result_policies_block_unprotecting_branches` feature flag in 16.4. > - The `block_unprotecting_branches` field was [replaced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137153) by `block_branch_modification` field in GitLab 16.7. @@ -187,25 +185,26 @@ the defined policy. > - The `prevent_approval_by_author`, `prevent_approval_by_commit_author`, `remove_approvals_with_new_commit`, and `require_password_to_approve` fields were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/418752) in GitLab 16.4 [with flag](../../../administration/feature_flags.md) named `scan_result_any_merge_request`. Disabled by default. > - The above fields were [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/423988) in GitLab 16.6. > - The above fields were [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/423988) in GitLab 16.7. +> - Feature flag `scan_result_any_merge_request` [was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/432127) in GitLab 16.8. > - The `prevent_pushing_and_force_pushing` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/420629) in GitLab 16.4 [with flag](../../../administration/feature_flags.md) named `scan_result_policies_block_force_push`. Disabled by default. > - The above field was [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/427260) in GitLab 16.6. > - The above field was [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/427260) in GitLab 16.7. FLAG: On self-managed GitLab, by default the `block_branch_modification` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `scan_result_policies_block_unprotecting_branches`. On GitLab.com, this feature is available. -On self-managed GitLab, by default the `prevent_approval_by_author`, `prevent_approval_by_commit_author`, `remove_approvals_with_new_commit`, and `require_password_to_approve` fields are available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `scan_result_any_merge_request`. On GitLab.com, this feature is available. On self-managed GitLab, by default the `prevent_pushing_and_force_pushing` field is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `scan_result_policies_block_force_push`. On GitLab.com, this feature is available. The settings set in the policy overwrite settings in the project. -| Field | Type | Required | Possible values | Applicable rule type | Description | -|-------------------------------------|-----------|----------|-----------------|----------------------|-------------| -| `block_branch_modification` | `boolean` | false | `true`, `false` | All | When enabled, prevents a user from removing a branch from the protected branches list, deleting a protected branch, or changing the default branch if that branch is included in the security policy. This ensures users cannot remove protection status from a branch to merge vulnerable code. | -| `prevent_approval_by_author` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, merge request authors cannot approve their own MRs. This ensures code authors cannot introduce vulnerabilities and approve code to merge. | -| `prevent_approval_by_commit_author` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, users who have contributed code to the MR are ineligible for approval. This ensures code committers cannot introduce vulnerabilities and approve code to merge. | -| `remove_approvals_with_new_commit` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, if an MR receives all necessary approvals to merge, but then a new commit is added, new approvals are required. This ensures new commits that may include vulnerabilities cannot be introduced. | -| `require_password_to_approve` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, there will be password confirmation on approvals. Password confirmation adds an extra layer of security. | -| `prevent_pushing_and_force_pushing` | `boolean` | false | `true`, `false` | All | When enabled, prevents users from pushing and force pushing to a protected branch if that branch is included in the security policy. This ensures users do not bypass the merge request process to add vulnerable code to a branch. | +| Field | Type | Required | Possible values | Applicable rule type | Description | +|-------------------------------------|-----------------------|----------|---------------------------------------------------------------|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `block_branch_modification` | `boolean` | false | `true`, `false` | All | When enabled, prevents a user from removing a branch from the protected branches list, deleting a protected branch, or changing the default branch if that branch is included in the security policy. This ensures users cannot remove protection status from a branch to merge vulnerable code. | +| `block_group_branch_modification` | `boolean` or `object` | false | `true`, `false`, `{ enabled: boolean, exceptions: [string] }` | All | When enabled, prevents a user from removing group-level protected branches on every group the policy applies to. If `block_branch_modification` is `true`, implicitly defaults to `true`. | +| `prevent_approval_by_author` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, merge request authors cannot approve their own MRs. This ensures code authors cannot introduce vulnerabilities and approve code to merge. | +| `prevent_approval_by_commit_author` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, users who have contributed code to the MR are ineligible for approval. This ensures code committers cannot introduce vulnerabilities and approve code to merge. | +| `remove_approvals_with_new_commit` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, if an MR receives all necessary approvals to merge, but then a new commit is added, new approvals are required. This ensures new commits that may include vulnerabilities cannot be introduced. | +| `require_password_to_approve` | `boolean` | false | `true`, `false` | `Any merge request` | When enabled, there will be password confirmation on approvals. Password confirmation adds an extra layer of security. | +| `prevent_pushing_and_force_pushing` | `boolean` | false | `true`, `false` | All | When enabled, prevents users from pushing and force pushing to a protected branch if that branch is included in the security policy. This ensures users do not bypass the merge request process to add vulnerable code to a branch. | ## Example security scan result policies project @@ -300,10 +299,20 @@ actions: ## Understanding scan result policy approvals +> The branch comparison logic for `scan_finding` was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/428518) in GitLab 16.8 [with a flag](../../../administration/feature_flags.md) named `scan_result_policy_merge_base_pipeline`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `scan_result_policy_merge_base_pipeline`. +On GitLab.com, this feature is not available. + ### Scope of scan result policy comparison -- To determine when approval is required on a merge request, we compare the latest completed pipelines for each supported pipeline source for the source and target branch (for example, `feature`/`main`). This ensures the most comprehensive evaluation of scan results. -- We compare findings from the latest completed pipelines that ran on `HEAD` of the source and target branch. +- To determine when approval is required on a merge request, we compare completed pipelines for each supported pipeline source for the source and target branch (for example, `feature`/`main`). This ensures the most comprehensive evaluation of scan results. +- For the source branch, the comparison pipeline is its latest completed `HEAD` pipeline. +- For `license_finding` rules, we compare to a common ancestor's latest completed pipeline. +- For `scan_finding` rules, the comparison pipeline may differ: + - If the `scan_result_policy_merge_base_pipeline` feature flag is enabled, we compare to a common ancestor's latest completed pipeline. + - Otherwise, we compare to the target branch's latest completed `HEAD` pipeline. - Scan result policies considers all supported pipeline sources (based on the [`CI_PIPELINE_SOURCE` variable](../../../ci/variables/predefined_variables.md)) when comparing results from both the source and target branches when determining if a merge request requires approval. Pipeline sources `webide` and `parent_pipeline` are not supported. ### Accepting risk and ignoring vulnerabilities in future merge requests @@ -352,6 +361,78 @@ We have identified in [epic 11020](https://gitlab.com/groups/gitlab-org/-/epics/ - Findings or errors that cause approval to be required on a scan result policy may not be evident in the Security MR Widget. By using `merge base` in [issue 428518](https://gitlab.com/gitlab-org/gitlab/-/issues/428518) some cases will be addressed. We will additionally be [displaying more granular details](https://gitlab.com/groups/gitlab-org/-/epics/11185) about what caused security policy violations. - Security policy violations are distinct compared to findings displayed in the MR widgets. Some violations may not be present in the MR widget. We are working to harmonize our features in [epic 11020](https://gitlab.com/groups/gitlab-org/-/epics/11020) and to display policy violations explicitly in merge requests in [epic 11185](https://gitlab.com/groups/gitlab-org/-/epics/11185). +## Experimental features **(EXPERIMENT)** + +### Security policy scopes + +> The `policy_scope` field was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135398) in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `security_policies_policy_scope`. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +an administrator can [disable the feature flag](../../../administration/feature_flags.md) +named `security_policies_policy_scope`. +On GitLab.com, this feature is available. + +Security policy enforcement depends first on establishing a link between the group, subgroup, or +project on which you want to enforce policies, and the security policy project that contains the +policies. For example, if you are linking policies to a group, a group owner must create the link to +the security policy project. Then, all policies in the security policy project are inherited by all +projects in the group. + +You can refine a security policy's scope to: + +- _Include_ only projects containing a compliance framework label. +- _Include_ or _exclude_ selected projects from enforcement. + +#### Policy scope schema + +| Field | Type | Required | Possible values | Description | +|-------|------|----------|-----------------|-------------| +| `policy_scope` | `object` | false | `compliance_frameworks`, `projects` | Scopes the policy based on compliance framework labels or projects you define. | + +#### `policy_scope` scope type + +| Field | Type | Possible values | Description | +|-------|------|-----------------|-------------| +| `compliance_frameworks` | `object` | `ids` | List of IDs of the compliance frameworks in scope of enforcement, in an `ids` array. | +| `projects` | `object` | `including`, `excluding` | Use `excluding:` or `including:` then list the IDs of the projects you wish to include or exclude, in an `ids` array. | + +#### Example `policy.yml` with security policy scopes + +```yaml +--- +scan_result_policy: +- name: critical vulnerability CS approvals + description: critical severity level only for container scanning + enabled: true + rules: + - type: scan_finding + branches: + - main + scanners: + - container_scanning + vulnerabilities_allowed: 1 + severity_levels: + - critical + vulnerability_states: + - newly_detected + actions: + - type: require_approval + approvals_required: 1 + user_approvers: + - adalberto.dare + policy_scope: + compliance_frameworks: + ids: + - 2 + - 11 + projects: + including: + ids: + - 24 + - 27 +``` + ## Troubleshooting ### Merge request rules widget shows a scan result policy is invalid or duplicated **(ULTIMATE SELF)** diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index a813ac9888d..1f5340758c6 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -54,7 +54,7 @@ support the following features: - [Scan projects](index.md#supported-languages-and-frameworks) - [Multi-project support](index.md#multi-project-support) - [Offline support](index.md#running-sast-in-an-offline-environment) -- [Emits JSON report format](index.md#reports-json-format) +- [Output results in JSON report format](index.md#output) - [SELinux support](index.md#running-sast-in-selinux) ## Post analyzers diff --git a/doc/user/application_security/sast/customize_rulesets.md b/doc/user/application_security/sast/customize_rulesets.md index 992e99f1cc7..a9ef89077ca 100644 --- a/doc/user/application_security/sast/customize_rulesets.md +++ b/doc/user/application_security/sast/customize_rulesets.md @@ -28,7 +28,7 @@ You can disable predefined rules for any SAST analyzer. When you disable a rule: -- Most analyzers still scan for the vulnerability. The results are removed as a processing step after the scan completes, and they don't appear in the [`gl-sast-report.json` artifact](index.md#reports-json-format). +- Most analyzers still scan for the vulnerability. The results are removed as a processing step after the scan completes, and they don't appear in the [`gl-sast-report.json` artifact](index.md#output). - Findings for the disabled rule no longer appear in the [pipeline security tab](../index.md#pipeline-security-tab). - Existing findings for the disabled rule on the default branch are marked as [`No longer detected`](../vulnerability_report/index.md#activity-filter) in the [vulnerability report](../index.md#vulnerability-report). @@ -196,7 +196,7 @@ rule that you wish to modify. | `value` | The value of the identifier used by the predefined rule. | You can look up the correct values for `type` and `value` by viewing the -[`gl-sast-report.json`](index.md#reports-json-format) produced by the analyzer. +[`gl-sast-report.json`](index.md#output) produced by the analyzer. You can download this file as a job artifact from the analyzer's CI job. For example, the snippet below shows a finding from a `semgrep` rule with three diff --git a/doc/user/application_security/sast/img/sast_inline_indicator_v16_7.png b/doc/user/application_security/sast/img/sast_inline_indicator_v16_7.png Binary files differindex c86f536afc4..582e55f1d0a 100644 --- a/doc/user/application_security/sast/img/sast_inline_indicator_v16_7.png +++ b/doc/user/application_security/sast/img/sast_inline_indicator_v16_7.png diff --git a/doc/user/application_security/sast/img/sast_mr_widget_v16_7.png b/doc/user/application_security/sast/img/sast_mr_widget_v16_7.png Binary files differindex 199f8b6d322..b293989ae21 100644 --- a/doc/user/application_security/sast/img/sast_mr_widget_v16_7.png +++ b/doc/user/application_security/sast/img/sast_mr_widget_v16_7.png diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index ab6d5212227..cddd6a1f14d 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -54,36 +54,39 @@ For more information about our plans for language support in SAST, see the [cate | Language / framework | [Analyzer](analyzers.md) used for scanning | Minimum supported GitLab version | |------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| .NET (all versions, C# only) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 15.4 | +| .NET (all versions, C# only) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 15.4 | | Apex (Salesforce) | [PMD](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) | 12.1 | -| C | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.2 | +| C | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 14.2 | | C/C++ | [Flawfinder](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) | 10.7 | | Elixir (Phoenix) | [Sobelow](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) | 11.1 | -| Go | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.4 | +| Go | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 14.4 | | Groovy<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.3 (Gradle) & 11.9 (Maven, SBT) | | Helm Charts | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 13.1 | -| Java (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 14.10 | +| Java (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 14.10 | | Java (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | -| JavaScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | +| JavaScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 13.10 | | Kotlin (Android) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | | Kotlin (General)<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 13.11 | | Kubernetes manifests | [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) | 12.6 | | Node.js | [NodeJsScan](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) | 11.1 | | Objective-C (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | | PHP | [phpcs-security-audit](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) | 10.8 | -| Python | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.9 | -| React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | +| Python | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 13.9 | +| React | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 13.10 | | Ruby | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 13.9 | | Ruby on Rails | [brakeman](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) | 10.3 | -| Scala (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 16.0 | -| Scala<sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | +| Scala (any build system) | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 16.0 | +| Scala <sup>1</sup> | [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) with the find-sec-bugs plugin | 11.0 (SBT) & 11.9 (Gradle, Maven) | | Swift (iOS) | [MobSF (beta)](https://gitlab.com/gitlab-org/security-products/analyzers/mobsf) | 13.5 | -| TypeScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/RULES.md) | 13.10 | +| TypeScript | [Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) with [GitLab-managed rules](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/#sast-rules) | 13.10 | -1. The SpotBugs-based analyzer supports [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the -[Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), -[Grails](https://grails.org/), -and the [Maven wrapper](https://github.com/takari/maven-wrapper). However, SpotBugs has [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/350801) when used against [Ant](https://ant.apache.org/)-based projects. We recommend using the Semgrep-based analyzer for Ant-based Java or Scala projects. +<html> +<small>Footnotes: + <ol> + <li>The SpotBugs-based analyzer supports [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), [Grails](https://grails.org/), and the [Maven wrapper](https://github.com/takari/maven-wrapper). However, SpotBugs has [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/350801) when used against [Ant](https://ant.apache.org/)-based projects. We recommend using the Semgrep-based analyzer for Ant-based Java or Scala projects.</li> + </ol> +</small> +</html> ## End of supported analyzers @@ -220,7 +223,7 @@ as shown in the following table: | Automatically scan code with [appropriate analyzers](#supported-languages-and-frameworks) | **{check-circle}** | **{check-circle}** | | [Configure SAST scanners](#configuration) | **{check-circle}** | **{check-circle}** | | [Customize SAST settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | -| Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | +| Download [SAST output](#output) | **{check-circle}** | **{check-circle}** | | See new findings in merge request widget | **{dotted-circle}** | **{check-circle}** | | See new findings in merge request changes | **{dotted-circle}** | **{check-circle}** | | [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | @@ -230,25 +233,38 @@ as shown in the following table: | [Detect False Positives](#false-positive-detection) | **{dotted-circle}** | **{check-circle}** | | [Track moved vulnerabilities](#advanced-vulnerability-tracking) | **{dotted-circle}** | **{check-circle}** | +## Output + +SAST outputs the file `gl-sast-report.json` as a job artifact. The file contains details of all +detected vulnerabilities. You can +[download](../../../ci/jobs/job_artifacts.md#download-job-artifacts) the file for processing +outside GitLab. + +For more information, see: + +- [SAST report file schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json) +- [Example SAST report file](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/qa/expect/js/default/gl-sast-report.json) + ## View SAST results -SAST results are shown in the: +The [SAST report file](#output) is processed by GitLab and the details are shown in the UI: - Merge request widget - Merge request changes view -- Vulnerability Report +- Vulnerability report ### Merge request widget **(ULTIMATE ALL)** SAST results display in the merge request widget area if a report from the target -branch is available for comparison. The merge request widget displays SAST findings and resolutions that +branch is available for comparison. The merge request widget displays SAST results and resolutions that were introduced by the changes made in the merge request.  ### Merge request changes view **(ULTIMATE ALL)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10959) in GitLab 16.6 with a [flag](../../../administration/feature_flags.md) named `sast_reports_in_inline_diff`. Disabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10959) in GitLab 16.6 with a [flag](../../../administration/feature_flags.md) named `sast_reports_in_inline_diff`. Disabled by default. +> - Enabled by default in GitLab 16.8. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `sast_reports_in_inline_diff`. @@ -303,10 +319,6 @@ When downloading, you always receive the most recent SAST artifact available. You can enable and configure SAST by using the UI, either with the default settings or with customizations. The method you can use depends on your GitLab license tier. -### Running jobs in merge request pipelines - -See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines) - #### Configure SAST with customizations **(ULTIMATE ALL)** > [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/410013) individual SAST analyzers configuration options from the UI in GitLab 16.2. @@ -505,6 +517,10 @@ spotbugs-sast: sast: gl-sast-report.json ``` +### Running jobs in merge request pipelines + +See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines). + ### Available CI/CD variables SAST can be configured using the [`variables`](../../../ci/yaml/index.md#variables) parameter in @@ -593,6 +609,7 @@ Some analyzers can be customized with CI/CD variables. | `PHPCS_SECURITY_AUDIT_PHP_EXTENSIONS` | phpcs-security-audit | Comma separated list of additional PHP Extensions. | | `SAST_SEMGREP_METRICS` | Semgrep | Set to `"false"` to disable sending anonymized scan metrics to [r2c](https://semgrep.dev). Default: `true`. Introduced in GitLab 14.0. GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/330565`. | | `SAST_SCANNER_ALLOWED_CLI_OPTS` | Semgrep | CLI options (arguments with value, or flags) that are passed to the underlying security scanner when running scan operation. Only a limited set of [options](#security-scanner-configuration) are accepted. Separate a CLI option and its value using either a blank space or equals (`=`) character. For example: `name1 value1` or `name1=value1`. Multiple options must be separated by blank spaces. For example: `name1 value1 name2 value2`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368565) in GitLab 15.3. | +| `SAST_RULESET_GIT_REFERENCE` | Semgrep and nodejs-scan | Defines a path to a custom ruleset configuration. If a project has a `.gitlab/sast-ruleset.toml` file committed, that local configuration takes precedence and the file from `SAST_RULESET_GIT_REFERENCE` isn’t used. This variable is available for the Ultimate tier only. | #### Security scanner configuration @@ -646,21 +663,6 @@ variables: SAST_EXPERIMENTAL_FEATURES: "true" ``` -## Reports JSON format - -SAST outputs a report file in JSON format. The report file contains details of all found vulnerabilities. -To download the report file, you can either: - -- Download the file from the CI/CD pipelines page. -- In the pipelines tab on merge requests, set [`artifacts: paths`](../../../ci/yaml/index.md#artifactspaths) to `gl-sast-report.json`. - -For information, see [Download job artifacts](../../../ci/jobs/job_artifacts.md#download-job-artifacts). - -For details of the report file's schema, see -[SAST report file schema](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). - -For an example SAST report file, see [`gl-sast-report.json`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/blob/main/qa/expect/js/default/gl-sast-report.json) example. - ## Running SAST in an offline environment For self-managed GitLab instances in an environment with limited, restricted, or intermittent access diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index bf7375a58d7..9e2d67237d3 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -56,17 +56,17 @@ If you operate a cloud or SaaS product and you're interested in partnering with Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/). -| Capability | In Free & Premium | In Ultimate | -|:---------------------------------------------------------------- |:-----------------------|:-----------------------| -| [Configure Secret Detection scanner](#enable-secret-detection) | **{check-circle}** Yes | **{check-circle}** Yes | -| [Customize Secret Detection settings](#configure-scan-settings) | **{check-circle}** Yes | **{check-circle}** Yes | -| Download [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** Yes | **{check-circle}** Yes | +| Capability | In Free & Premium | In Ultimate | +|:-----------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------| +| [Configure Secret Detection scanner](#enable-secret-detection) | **{check-circle}** Yes | **{check-circle}** Yes | +| [Customize Secret Detection settings](#configure-scan-settings) | **{check-circle}** Yes | **{check-circle}** Yes | +| Download [SAST output](../sast/index.md#output) | **{check-circle}** Yes | **{check-circle}** Yes | | [Check text for potential secrets](#warnings-for-potential-leaks-in-text-content) before it's posted | **{check-circle}** Yes | **{check-circle}** Yes | -| See new findings in the merge request widget | **{dotted-circle}** No | **{check-circle}** Yes | -| View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** No | **{check-circle}** Yes | -| [Manage vulnerabilities](../vulnerability_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Customize Secret Detection rulesets](#custom-rulesets) | **{dotted-circle}** No | **{check-circle}** Yes | +| See new findings in the merge request widget | **{dotted-circle}** No | **{check-circle}** Yes | +| View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** No | **{check-circle}** Yes | +| [Manage vulnerabilities](../vulnerability_report/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Customize Secret Detection rulesets](#custom-rulesets) | **{dotted-circle}** No | **{check-circle}** Yes | ## Coverage @@ -110,17 +110,13 @@ Secret Detection can detect if a secret was added in one commit and removed in a [merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md). Secret Detection's results are only available after the pipeline is completed. -## Running jobs in merge request pipelines - -See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines) - ## Enable Secret Detection Prerequisites: - Linux-based GitLab Runner with the [`docker`](https://docs.gitlab.com/runner/executors/docker.html) or -[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. If you're using the -shared runners on GitLab.com, this is enabled by default. + [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. If you're using the + shared runners on GitLab.com, this is enabled by default. - Windows Runners are not supported. - CPU architectures other than amd64 are not supported. - If you use your own runners, make sure the Docker version installed is **not** `19.03.0`. See @@ -265,6 +261,10 @@ For example: "A personal token for GitLab will look like glpat-JUST20LETTERSANDNUMB" #gitleaks:allow ``` +### Running jobs in merge request pipelines + +See [Use security scanning tools with merge request pipelines](../index.md#use-security-scanning-tools-with-merge-request-pipelines). + ### Available CI/CD variables Secret Detection can be customized by defining available CI/CD variables: diff --git a/doc/user/application_security/secret_detection/pre_receive.md b/doc/user/application_security/secret_detection/pre_receive.md index 1e7ea4aaaeb..8bb56644926 100644 --- a/doc/user/application_security/secret_detection/pre_receive.md +++ b/doc/user/application_security/secret_detection/pre_receive.md @@ -29,7 +29,9 @@ Prerequisites: ## Limitations -This feature only scans non-binary blobs under 1 MiB in size. Binary blobs and blobs larger than 1 MiB are not scanned. +- This feature only scans non-binary blobs under 1 MiB in size. Binary blobs and blobs larger than 1 MiB are not scanned. +- The scan does not analyze the content of a commit if it is identical to the content of another file already present in the source code. +- The scan skips analyzing files that are renamed, deleted, or moved, unless their content is modified in the same commit. ## Resolve a blocked push @@ -53,15 +55,7 @@ If the blocked secret appears earlier in your Git history: ## Skip secret detection -In some cases, it may be necessary to skip pre-receive secret detection. For example, a developer may need to commit a placeholder secret for testing, or a user may want to bypass secret detection due to a Git operation timeout. To skip secret detection for a particular secret, add `# gitleaks:allow` to the end of the line. To skip secret detection for all commits in a push, add `[skip secret detection]` to one of the commit messages. For example: - -```ruby -# This secret will be skipped due to gitleaks:allow. -FAKE_TOKEN = allowfaketoken123 # gitleaks:allow - -# This secret will be scanned, and the push will be rejected. -REAL_TOKEN = rejectrealtoken123 -``` +In some cases, it may be necessary to skip pre-receive secret detection. For example, a developer may need to commit a placeholder secret for testing, or a user may want to bypass secret detection due to a Git operation timeout. To skip secret detection for all commits in a push, add `[skip secret detection]` to one of the commit messages. For example: ```shell # These commits are in the same push. Both will not be scanned. diff --git a/doc/user/application_security/secure_your_application.md b/doc/user/application_security/secure_your_application.md index b35de7827e8..095796f3dc4 100644 --- a/doc/user/application_security/secure_your_application.md +++ b/doc/user/application_security/secure_your_application.md @@ -1,6 +1,7 @@ --- stage: Secure group: Static Analysis +description: Container, dependency, and vulnerability scans. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 405017ab023..e9f3a3a2c0b 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -20,6 +20,12 @@ The data provided by the Security Dashboards can be used supply to insight on wh <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Security Dashboard](https://www.youtube.com/watch?v=Uo-pDns1OpQ). +## Vulnerability metrics in the Value Streams Dashboard + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383697) in GitLab 16.0. + +You can view vulnerability metrics also in the [Value Streams Dashboard](../../../user/analytics/value_streams_dashboard.md) comparison panel, which helps you understand security exposure in the context of your organization's software delivery workflows. + ## Prerequisites To view the Security Dashboards, the following is required: diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 23454bf387a..620d8c75e52 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -133,6 +133,10 @@ The content of the Project filter depends on the current level: ### Activity filter +> Introduced in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `activity_filter_has_remediations`. Disabled by default. + +FLAG: +On self-managed GitLab, by default the Solution Available filter is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `activity_filter_has_remediations`. On GitLab.com, this feature is not available. This feature is not ready for production use. The activity filter behaves differently from the other filters. You can select only one value in each category. To remove a filter, from the activity filter dropdown list select the filter you want to remove. @@ -141,7 +145,7 @@ Selection behavior when using the activity filter: - **Activity** - **All activity**: Vulnerabilities with any activity status (same as ignoring this filter). Selecting this deselects all other activity filter options. - **Detection** - - **Still detected**: Vulnerabilities that are still detected in the latest pipeline scan of the `default` branch. + - **Still detected** (default): Vulnerabilities that are still detected in the latest pipeline scan of the `default` branch. - **No longer detected**: Vulnerabilities that are no longer detected in the latest pipeline scan of the `default` branch. - **Issue** - **Has issues**: Vulnerabilities with one or more associated issues. @@ -149,6 +153,9 @@ Selection behavior when using the activity filter: - **Merge request** - **Has merge request**: Vulnerabilities with one or more associated merge requests. - **Does not have merge request**: Vulnerabilities without an associated merge request. +- **Solution available** + - **Has a solution**: Vulnerabilities with an available solution. + - **Does not have a solution**: Vulnerabilities without an available solution. ## View details of a vulnerability @@ -181,10 +188,13 @@ To change the status of vulnerabilities: - One or more vulnerabilities, select the checkbox beside each vulnerability. - All vulnerabilities on the page, select the checkbox in the table header. 1. In the **Set status** dropdown list, select the desired status. -1. If the **Dismissed** status is chosen, select the desired reason in the **Set dismissal reason** dropdown list. -1. In the **Add a comment** input, you can provide a comment. For the **Dismissed** status, a comment is required. +1. If the **Dismiss** status is chosen, select the desired reason in the **Set dismissal reason** dropdown list. +1. In the **Add a comment** input, you can provide a comment. For the **Dismiss** status, a comment is required. 1. Select **Change status**. +The status of the selected vulnerabilities is updated and the content of the vulnerability report is +refreshed. +  ## Sort vulnerabilities by date detected @@ -195,6 +205,8 @@ To sort vulnerabilities by the date each vulnerability was detected, select the ## Export vulnerability details +> Added "Dismissal Reason" as a column in the CSV export [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/434076) in GitLab 16.8. + You can export details of the vulnerabilities listed in the Vulnerability Report. The export format is CSV (comma separated values). All vulnerabilities are included because filters do not apply to the export. @@ -219,6 +231,7 @@ Fields included are: - Comments - Full Path - CVSS Vectors +- [Dismissal Reason](../vulnerabilities/index.md#vulnerability-dismissal-reasons) NOTE: Full details are available through our diff --git a/doc/user/application_security/vulnerability_report/pipeline.md b/doc/user/application_security/vulnerability_report/pipeline.md index e60ac7d4c21..41cc323f3e1 100644 --- a/doc/user/application_security/vulnerability_report/pipeline.md +++ b/doc/user/application_security/vulnerability_report/pipeline.md @@ -101,6 +101,29 @@ To view findings, either: NOTE: This does not apply for the vulnerabilities existing on the default branch. +## Change status of findings + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331408) in GitLab 16.7 [with a flag](../../../administration/feature_flags.md) named `pipeline_security_dashboard_graphql`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) named `pipeline_security_dashboard_graphql`. +On GitLab.com, this feature is not available. + +To change the status of findings to **Dismiss** or **Needs triage**: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Build > Pipelines**. +1. Select a pipeline and select the **Security** tab. +1. To select: + - One or more findings, select the checkbox beside each finding. + - All findings on the page, select the checkbox in the table header. +1. In the **Set status** dropdown list, select the desired status. +1. If the **Dismiss** status is chosen, select the desired reason in the **Set dismissal reason** dropdown list. +1. In the **Add a comment** input, you can provide a comment. For the **Dismiss** status, a comment is required. +1. Select **Change status**. + +The status of the selected findings is updated and the content of the security tab is refreshed. + ## Deduplication process When a pipeline contains jobs that produce multiple security reports of the same type, it is possible that the same @@ -159,7 +182,7 @@ appear in a report. - Deduplication result: duplicates because all criteria match, and type identifiers are ignored. Only one identifier needs to match, in this case CVE-2022-25510. -The examples above don't include the raw location values. Each scan type defines its own -`fingerprint_data`, which is used to generate a `SHA1` hash that is used as the `location_fingerprint`. -You can find definitions for each scan type [`gitlab/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/01c69e97340b7c1c7e30c0caec8506910b6503c8/lib/gitlab/ci/reports/security/locations) -and [`gitlab/ee/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/01c69e97340b7c1c7e30c0caec8506910b6503c8/ee/lib/gitlab/ci/reports/security/locations). +You can find definitions for each scan type [`gitlab/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/reports/security/locations) +and [`gitlab/ee/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/ci/reports/security/locations). + +For instance, for `container_scanning` type the location is defined by Docker image name without tag. However if the image tag contains at least one letter and/or is longer than 8 characters, it isn't considered a duplicate. So, locations `registry.gitlab.com/group-name/project-name/image1:12345019:libcrypto3` and `registry.gitlab.com/group-name/project-name/image1:libcrypto3` are treated as identical while `registry.gitlab.com/group-name/project-name/image1:v19202021:libcrypto3` and `registry.gitlab.com/group-name/project-name/image1:libcrypto3` are considered different. diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md deleted file mode 100644 index 09f7b4c77fa..00000000000 --- a/doc/user/award_emojis.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'emoji_reactions.md' -remove_date: '2023-12-20' ---- - -This document was moved to [another location](emoji_reactions.md). - -<!-- This redirect file can be deleted after <2023-12-20>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 66e67f56172..a764d0006a1 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -63,9 +63,9 @@ GitLab in a Kubernetes cluster, you might need a different version of Kubernetes You can upgrade your Kubernetes version to a supported version at any time: -- 1.27 (support ends on July 18, 2024 or when 1.30 becomes supported) -- 1.26 (support ends on March 21, 2024 or when 1.29 becomes supported) -- 1.25 (support ends on October 22, 2023 or when 1.28 becomes supported) +- 1.28 (support ends when GitLab version 17.5 is released or when 1.31 becomes supported) +- 1.27 (support ends when GitLab version 17.2 is released or when 1.30 becomes supported) +- 1.26 (support ends when GitLab version 16.10 is released or when 1.29 becomes supported) GitLab aims to support a new minor Kubernetes version three months after its initial release. GitLab supports at least three production-ready Kubernetes minor versions at any given time. diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index 5b07dbd56d9..b8caf6d0837 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -137,7 +137,7 @@ By default, the Helm installation command generated by GitLab: - Creates a `Secret` resource for the agent's access token. To instead bring your own secret with a token, omit the token (`--set token=...`) and instead use `--set config.secretName=<your secret name>`. - Creates a `Deployment` resource for the `agentk` pod. -To see the full list of customizations available, see the Helm chart's [default values file](https://gitlab.com/gitlab-org/charts/gitlab-agent/-/blob/main/values.yaml). +To see the full list of customizations available, see the Helm chart's [README](https://gitlab.com/gitlab-org/charts/gitlab-agent/-/blob/main/README.md#values). ##### Use the agent when KAS is behind a self-signed certificate @@ -224,13 +224,19 @@ For the best experience, the version of the agent installed in your cluster shou ### Update the agent version +NOTE: +Instead of using `--reuse-values`, you should specify all needed values. +If you use `--reuse-values`, you might miss new defaults or use deprecated values. +To retrieve previous `--set` arguments, use `helm get values <release name>`. +You can save the values to a file with `helm get values gitlab-agent > agent.yaml`, and pass the file to Helm with `-f`: +`helm upgrade gitlab-agent gitlab/gitlab-agent -f agent.yaml`. This safely replaces the behavior of `--reuse-values`. + To update the agent to the latest version, you can run: ```shell helm repo update helm upgrade --install gitlab-agent gitlab/gitlab-agent \ --namespace gitlab-agent \ - --reuse-values ``` To set a specific version, you can override the `image.tag` value. For example, to install version `v14.9.1`, run: @@ -238,7 +244,6 @@ To set a specific version, you can override the `image.tag` value. For example, ```shell helm upgrade gitlab-agent gitlab/gitlab-agent \ --namespace gitlab-agent \ - --reuse-values \ --set image.tag=v14.9.1 ``` diff --git a/doc/user/clusters/agent/kas_glossary.md b/doc/user/clusters/agent/kas_glossary.md new file mode 100644 index 00000000000..d2d4065f3fb --- /dev/null +++ b/doc/user/clusters/agent/kas_glossary.md @@ -0,0 +1,20 @@ +--- +stage: Deploy +group: Environments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Kubernetes integration glossary **(FREE ALL)** + +This glossary provides definitions for terms related to the GitLab Kubernetes integration. + +| Term | Definition | Scope | +| --- | --- | --- | +| GitLab agent for Kubernetes | The overall offering, including related features and the underlying components `agentk` and `kas`. | GitLab, Kubernetes, Flux | +| `agentk` | The cluster-side component that maintains a secure connection to GitLab for Kubernetes management and deployment automation. | GitLab | +| GitLab agent server for Kubernetes (`kas`) | The GitLab-side component of GitLab that handles operations and logic for the Kubernetes agent integration. Manages the connection and communication between GitLab and Kubernetes clusters. | GitLab | +| Pull-based deployment | A deployment method where Flux checks for changes in a Git repository and automatically applies these changes to the cluster. | GitLab, Kubernetes | +| Push-based deployment | A deployment method where updates are sent from GitLab CI/CD pipelines to the Kubernetes cluster. | GitLab | +| Flux | An open-source GitOps tool that integrates with the agent for pull-based deployments. | GitOps, Kubernetes | +| GitOps | A set of practices that involve using Git for version control and collaboration in the management and automation of cloud and Kubernetes resources. | DevOps, Kubernetes | +| Kubernetes namespace | A logical partition in a Kubernetes cluster that divides cluster resources between multiple users or environments. | Kubernetes | diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index 178f1bd7705..2c748712fe5 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -47,7 +47,7 @@ Management projects are restricted to the following: To use a cluster management project to manage your cluster: 1. Create a new project to serve as the cluster management project -for your cluster. + for your cluster. 1. [Associate the cluster with the management project](#associate-the-cluster-management-project-with-the-cluster). 1. [Configure your cluster's pipelines](#configuring-your-pipeline). 1. [Set the environment scope](#setting-the-environment-scope). @@ -66,12 +66,12 @@ To associate a cluster management project with your cluster: 1. Select **Kubernetes**. 1. Expand **Advanced settings**. 1. From the **Cluster management project** dropdown list, select the cluster management project -you created in the previous step. + you created in the previous step. ### Configuring your pipeline After designating a project as the management project for the cluster, -write a [`.gitlab-ci.yml`](../../ci/yaml/index.md) in that project. For example: +add a [`.gitlab-ci.yml` file](../../ci/index.md#the-gitlab-ciyml-file) in that project. For example: ```yaml configure cluster: @@ -99,7 +99,7 @@ to a management project: | Production | `production` | The environments set in the -[`.gitlab-ci.yml`](../../ci/yaml/index.md) file deploy to the Development, Staging, and Production cluster. +[`.gitlab-ci.yml` file](../../ci/index.md#the-gitlab-ciyml-file) deploy to the Development, Staging, and Production cluster. ```yaml stages: diff --git a/doc/user/compliance/compliance_center/index.md b/doc/user/compliance/compliance_center/index.md index 63b4560a498..da65cc9ef6e 100644 --- a/doc/user/compliance/compliance_center/index.md +++ b/doc/user/compliance/compliance_center/index.md @@ -71,6 +71,29 @@ To update the adherence status for these projects, the group-level or the projec To comply with GitLab standard, you must have at least two users approve a merge request to get it merged. For more information, see [Merge request approval rules](../../project/merge_requests/approvals/rules.md). +### Export standards adherence report for projects in a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/413736) in GitLab 16.8 [with a flag](../../../administration/feature_flags.md) named `compliance_standards_adherence_csv_export`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) +named `compliance_standards_adherence_csv_export`. On GitLab.com, this feature is not available. The feature is not ready for production use. + +Exports the contents of a standards adherence report for projects in a group. Reports are truncated at 15 MB to avoid a large email attachment. + +Prerequisites: + +- You must be an administrator or have the Owner role for the group. + +To export the standards adherence report for projects in a group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Secure > Compliance center**. +1. In the top-right corner, select **Export**. +1. Select **Export standards adherence report**. + +A report is compiled and delivered to your email inbox as an attachment. + ## Compliance violations report > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36524) in GitLab 12.8 as Compliance Dashboard. @@ -391,10 +414,11 @@ Repeat this process to filter by multiple attributes. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/422973) in GitLab 16.5 [with a flag](../../../administration/feature_flags.md) named `compliance_framework_report_ui`. Disabled by default. > - In GitLab 16.4 and earlier, **Compliance frameworks report** referred to what is now called **Compliance projects report**. The formally-named **Compliance frameworks report** was [renamed to **Compliance projects report**](https://gitlab.com/gitlab-org/gitlab/-/issues/422963) in GitLab 16.5. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140825) in GitLab 16.8. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named -`compliance_framework_report_ui`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature an administrator to [disable the feature flag](../../../administration/feature_flags.md) named +`compliance_framework_report_ui`. On GitLab.com, this feature is available. With compliance frameworks report, you can see all the compliance frameworks in a group. Each row of the report shows: diff --git a/doc/user/compliance/license_list.md b/doc/user/compliance/license_list.md index 43d76845ffa..3bfc5612db9 100644 --- a/doc/user/compliance/license_list.md +++ b/doc/user/compliance/license_list.md @@ -4,9 +4,13 @@ group: Threat Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# License list **(ULTIMATE ALL)** +<!--- start_remove The following content will be removed on remove_date: '2024-08-15' --> -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13582) in GitLab 12.7. +# License list (deprecated) **(ULTIMATE ALL)** + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/436100) in GitLab 16.8 +and is planned for removal in 17.0. Use the [Dependency List](../application_security/dependency_list/index.md) instead. The License list allows you to see your project's licenses and key details about them. @@ -32,3 +36,5 @@ The licenses are displayed, where: - **Policy Violation:** The license has a [license policy](license_approval_policies.md) marked as **Deny**.  + +<!--- end_remove --> diff --git a/doc/user/custom_roles.md b/doc/user/custom_roles.md index 1f3628efa39..07e14494ada 100644 --- a/doc/user/custom_roles.md +++ b/doc/user/custom_roles.md @@ -1,21 +1,15 @@ --- stage: Govern -group: Authentication +group: Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Custom roles **(ULTIMATE ALL)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `customizable_roles`. +> - [Custom roles feature introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `customizable_roles`. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110810) in GitLab 15.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114524) in GitLab 15.10. -> - The ability for a custom role to view a vulnerability report [introduced](https://gitlab.com/groups/gitlab-org/-/epics/10160) in GitLab 16.1 [with a flag](../administration/feature_flags.md) named `custom_roles_vulnerability`. -> - Ability to view a vulnerability report [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123835) in GitLab 16.1. -> - [Feature flag `custom_roles_vulnerability` removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124049) in GitLab 16.2. > - Ability to create and remove a custom role with the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393235) in GitLab 16.4. -> - Ability to manage group members [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17364) in GitLab 16.5. -> - Ability to manage project access tokens [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/421778) in GitLab 16.5 [with a flag](../administration/feature_flags.md) named `manage_project_access_tokens`. -> - Ability to archive projects [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/425957) in GitLab 16.7. > - Ability to use the UI to add a user to your group with a custom role, change a user's custom role, or remove a custom role from a group member [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393239) in GitLab 16.7. Custom roles allow group Owners or instance administrators to create roles @@ -26,6 +20,10 @@ For a demo of the custom roles feature, see [[Demo] Ultimate Guest can view code You can discuss individual custom role and permission requests in [issue 391760](https://gitlab.com/gitlab-org/gitlab/-/issues/391760). +## Available permissions + +For more information on available permissions, see [custom abilities](custom_roles/abilities.md). + ## Create a custom role Prerequisites: @@ -96,26 +94,6 @@ In **Settings > Roles and Permissions**, the list of all custom roles displays t To create a custom role, you can also [use the API](../api/member_roles.md#add-a-member-role-to-a-group). -### Available permissions - -The following permissions are available. You can add these permissions in any combination -to a base role to create a custom role. - -Some permissions require having other permissions enabled first. For example, administration of vulnerabilities (`admin_vulnerability`) can only be enabled if reading vulnerabilities (`read_vulnerability`) is also enabled. - -These requirements are documented in the `Required permission` column in the following table. - -| Permission | Version | Required permission | Description | -| ------------------------------- | -----------------------| -------------------- | ----------- | -| `read_code` | GitLab 15.7 and later | Not applicable | View project code. Does not include the ability to pull code. | -| `read_vulnerability` | GitLab 16.1 and later | Not applicable | View [vulnerability reports](application_security/vulnerability_report/index.md). | -| `admin_vulnerability` | GitLab 16.1 and later | `read_vulnerability` | Change the [status of vulnerabilities](application_security/vulnerabilities/index.md#vulnerability-status-values). | -| `read_dependency` | GitLab 16.3 and later | Not applicable | View [project dependencies](application_security/dependency_list/index.md). | -| `admin_merge_request` | GitLab 16.4 and later | Not applicable | View and approve [merge requests](project/merge_requests/index.md), revoke merge request approval, and view the associated merge request code. <br> Does not allow users to view or change merge request approval rules. | -| `manage_project_access_tokens` | GitLab 16.5 and later | Not applicable | Create, delete, and list [project access tokens](project/settings/project_access_tokens.md). | -| `admin_group_member` | GitLab 16.5 and later | Not applicable | Add or remove [group members](group/manage.md). | -| `archive_project` | GitLab 16.6 and later | Not applicable | Archive and unarchive [projects](project/settings/migrate_projects.md#archive-a-project). | - ## Billing and seat usage When you enable a custom role for a user with the Guest role, that user has @@ -219,8 +197,8 @@ To remove a custom role from a group member: 1. Select the **Max role** dropdown list for the member you want to remove a custom role from. 1. On the **Change role** dialog, select a static role. -You can update or remove a custom role from a group member also with the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project). -and pass an empty `member_role_id` value: +You can also use the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project) +to update or remove a custom role from a group member by passing an empty `member_role_id` value: ```shell # to update a project membership diff --git a/doc/user/custom_roles/abilities.md b/doc/user/custom_roles/abilities.md new file mode 100644 index 00000000000..d117a495798 --- /dev/null +++ b/doc/user/custom_roles/abilities.md @@ -0,0 +1,65 @@ +--- +stage: Govern +group: Authorization +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +<!--- + This documentation is auto generated by a Rake task. + + Please do not edit this file directly. To update this file, run: + bundle exec rake gitlab:custom_roles:compile_docs + + To make changes to the output of the Rake task, + edit `tooling/custom_roles/docs/templates/custom_abilities.md.erb`. +---> + +# Available custom abilities + +The following abilities are available. You can add these abilities in any combination +to a base role to create a custom role. + +Some abilities require having other abilities enabled first. For example, administration of vulnerabilities (`admin_vulnerability`) can only be enabled if reading vulnerabilities (`read_vulnerability`) is also enabled. + +These requirements are documented in the `Required ability` column in the following table. + +## Code review workflow + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`admin_merge_request`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128302) | | Allows approval of merge requests. | GitLab [16.4](https://gitlab.com/gitlab-org/gitlab/-/issues/412708) | | | +| [`read_code`](https://gitlab.com/gitlab-org/gitlab/-/issues/376180) | | Allows read-only access to the source code. | GitLab [15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/20277) | `customizable_roles` | GitLab [15.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110810) | + +## Group and projects + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`admin_group_member`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131914) | | Allows admin of group members. | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/17364) | `admin_group_member` | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136247) | + +## Groups and projects + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`archive_project`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134998) | | Allows archiving of projects. | GitLab [16.6](https://gitlab.com/gitlab-org/gitlab/-/issues/425957) | `archive_project` | GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139260) | +| [`remove_project`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139696) | | Allows deletion of projects. | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/425959) | | | + +## Infrastructure as code + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`admin_terraform_state`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140759) | | Allows to admin terraform state | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/421789) | | | + +## System access + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`manage_group_access_tokens`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140115) | | Allows manage access to the group access tokens. | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/428353) | | | +| [`manage_project_access_tokens`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132342) | | Allows manage access to the project access tokens. | GitLab [16.5](https://gitlab.com/gitlab-org/gitlab/-/issues/421778) | `manage_project_access_tokens` | GitLab [16.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141294) | + +## Vulnerability management + +| Name | Required permission | Description | Introduced in | Feature flag | Enabled in | +|:-----|:------------|:------------------|:---------|:--------------|:---------| +| [`admin_vulnerability`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121534) | read_vulnerability | Allows admin access to the vulnerability reports. | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/412536) | | | +| [`read_dependency`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126247) | | Allows read-only access to the dependencies. | GitLab [16.3](https://gitlab.com/gitlab-org/gitlab/-/issues/415255) | | | +| [`read_vulnerability`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120704) | | Allows read-only access to the vulnerability reports. | GitLab [16.1](https://gitlab.com/gitlab-org/gitlab/-/issues/399119) | | | diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 034e2e45127..dfcbc8a171d 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -163,7 +163,7 @@ To lock an issue or merge request: 1. On the left sidebar, select **Search or go to** and find your project. 1. For merge requests, select **Code > Merge requests**, and find your merge request. 1. For issues, select **Plan > Issues**, and find your issue. -1. On the top right, select **Merge request actions** or **Issue actions** (**{ellipsis_v}**), then select **Lock discussion**. +1. In the upper-right corner, select **Merge request actions** or **Issue actions** (**{ellipsis_v}**), then select **Lock discussion**. A system note is added to the page details. diff --git a/doc/user/emoji_reactions.md b/doc/user/emoji_reactions.md index 10385da7cdc..a72c15bb229 100644 --- a/doc/user/emoji_reactions.md +++ b/doc/user/emoji_reactions.md @@ -15,8 +15,7 @@ and thumbs-ups. React with emoji on: - [Issues](project/issues/index.md). - [Tasks](tasks.md). -- [Merge requests](project/merge_requests/index.md), -[snippets](snippets.md). +- [Merge requests](project/merge_requests/index.md), [snippets](snippets.md). - [Epics](../user/group/epics/index.md). - [Objectives and key results](okrs.md). - Anywhere else you can have a comment thread. diff --git a/doc/user/feature_flags.md b/doc/user/feature_flags.md index 83e2926e8e3..ccce9e9f9b4 100644 --- a/doc/user/feature_flags.md +++ b/doc/user/feature_flags.md @@ -1,8 +1,8 @@ --- stage: none group: unassigned -info: "See the Technical Writers assigned to Development Guidelines: https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" -description: "View a list of all the flags available in the GitLab application." +description: Complete list of flags. +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments layout: 'feature_flags' --- diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 70ac1f737e6..75d0c8fdd12 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -232,9 +232,8 @@ this limit. Repository limits apply to both public and private projects. ## Default import sources -> Disabling all importers by default for new GitLab self-managed installations [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118970) in GitLab 16.0. - -The import sources that are available by default depend on which GitLab you use: +The [import sources](../project/import/index.md#supported-import-sources) that are available to you by default depend on +which GitLab you use: - GitLab.com: all available import sources are enabled by default. - GitLab self-managed: no import sources are enabled by default and must be @@ -246,14 +245,12 @@ The import sources that are available by default depend on which GitLab you use: | [Bitbucket Server](../project/import/bitbucket_server.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [FogBugz](../project/import/fogbugz.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [Gitea](../project/import/gitea.md) | **{check-circle}** Yes | **{dotted-circle}** No | -| [GitLab by direct transfer](../group/import/index.md#migrate-groups-by-direct-transfer-recommended) | **{check-circle}** Yes | **{dotted-circle}** No | +| [GitLab by direct transfer](../group/import/index.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [GitLab using file exports](../project/settings/import_export.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [GitHub](../project/import/github.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [Manifest file](../project/import/manifest.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [Repository by URL](../project/import/repo_by_url.md) | **{check-circle}** Yes | **{dotted-circle}** No | -[Other importers](../project/import/index.md#available-project-importers) are available. - ## IP range GitLab.com uses the IP ranges `34.74.90.64/28` and `34.74.226.0/24` for traffic from its Web/API @@ -457,7 +454,7 @@ To help avoid abuse, the following are rate limited: For more information, see: - [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits). -- [Group import/export rate limits](../../user/group/import/index.md#rate-limits). +- [Group import/export rate limits](../../user/project/settings/import_export.md#rate-limits-1). ### Non-configurable limits diff --git a/doc/user/gitlab_duo_chat.md b/doc/user/gitlab_duo_chat.md index ebcb4c69e64..deb908444c3 100644 --- a/doc/user/gitlab_duo_chat.md +++ b/doc/user/gitlab_duo_chat.md @@ -4,37 +4,78 @@ group: Duo Chat info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Answer questions with GitLab Duo Chat **(ULTIMATE SAAS BETA)** +# GitLab Duo Chat **(ULTIMATE ALL BETA)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117695) as an [Experiment](../policy/experiment-beta-support.md#experiment) for SaaS in GitLab 16.0. +> - Changed to [Beta](../policy/experiment-beta-support.md#beta) for SaaS in GitLab 16.6. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11251) as a [Beta](../policy/experiment-beta-support.md#beta) for self-managed in GitLab 16.8. + +GitLab Duo Chat is your personal AI-powered assistant for boosting productivity. +It can assist various tasks of your daily work with the AI-generated content. +Here are the examples of use cases: + +| Feature | Use case example | Supported interfaces | Supported deployments | +| ------------------------------------- | ---------------- | -------------------------- | --------------------- | +| [Ask about GitLab](#ask-about-gitlab) | I want to know how to create an issue in GitLab. | GitLab, VS Code, and Web IDE | GitLab.com | +| [Ask about a specific issue](#ask-about-a-specific-issue) | I want to summarize this issue. | GitLab, VS Code, and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Ask about a specific epic](#ask-about-a-specific-epic) | I want to summarize this epic. | GitLab, VS Code, and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Ask about code](#ask-about-code) | I want to understand how this code works. | GitLab, VS Code, and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Ask about CI/CD](#ask-about-cicd) | I want to create a new CI/CD pipeline configuration. | GitLab, VS Code, and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Explain code in the IDE](#explain-code-in-the-ide) | I want to understand how this code works. | VS Code and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Refactor code in the IDE](#explain-code-in-the-ide) | I want to write a test for this code. | VS Code and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | +| [Write tests in the IDE](#write-tests-in-the-ide) | I want to refactor this code. | VS Code and Web IDE | GitLab.com, self-managed, and GitLab Dedicated | + +<div class="video-fallback"> + <a href="https://youtu.be/l6vsd1HMaYA?si=etXpFbj1cBvWyj3_">View how to setup and use GitLab Duo Chat</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/l6vsd1HMaYA?si=etXpFbj1cBvWyj3_" frameborder="0" allowfullscreen> </iframe> +</figure> -> Introduced in GitLab 16.6 as a [Beta](../policy/experiment-beta-support.md#beta). - -You can get AI-generated support from GitLab Duo Chat about: - -- How to use GitLab. -- The contents of an issue or issue. -- The contents of a code or CI/CD configuration file. - -You can also use GitLab Duo Chat to create code and CI/CD files. +NOTE: +This is a Beta feature. We're continuously extending the capabilities and reliability of the responses. -When you get an answer, you can ask follow-up questions to learn more. +## Features -This is a Beta feature. We're continuously extending the capabilities and reliability of the responses. +### Ask about GitLab -## Ask about GitLab +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117695) for SaaS in GitLab 16.0. You can ask questions about how GitLab works. Things like: - `Explain the concept of a 'fork' in a concise manner.` - `Provide step-by-step instructions on how to reset a user's password.` -## Ask about your work +NOTE: +This feature is not currently supported in self-managed instances. +See [this epic](https://gitlab.com/groups/gitlab-org/-/epics/11600) for more infomation. -You can ask about GitLab issues and epics. For example: +### Ask about a specific issue + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122235) for SaaS in GitLab 16.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122235) for self-managed in GitLab 16.8. + +You can ask about a specific GitLab issue. For example: - `Generate a summary for the issue identified via this link: <link to your issue>` -- `Generate a concise summary of the current issue.` +- When you are viewing an issue in GitLab, you can ask `Generate a concise summary of the current issue.` +- `How can I improve the description of <link to your issue> so that readers understand the value and problems to be solved?` + +### Ask about a specific epic + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128487) for SaaS in GitLab 16.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128487) for self-managed in GitLab 16.8. + +You can ask about a specific GitLab epic. For example: + +- `Generate a summary for the epic identified via this link: <link to your epic>` +- `Generate a concise summary of the opened epic.` +- `What are the unique use cases raised by commenters in <link to your epic>?` -## Ask about code +### Ask about code + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122235) for SaaS in GitLab 16.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122235) for self-managed in GitLab 16.8. You can also ask GitLab Duo Chat to generate code: @@ -45,13 +86,52 @@ And you can ask GitLab Duo Chat to explain code: - `Provide a clear explanation of the given Ruby code: def sum(a, b) a + b end. Describe what this code does and how it works.` -## Ask about CI/CD +### Ask about CI/CD + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/423524) for SaaS in GitLab 16.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/423524) for self-managed in GitLab 16.8. You can ask GitLab Duo Chat to create a CI/CD configuration: - `Create a .gitlab-ci.yml configuration file for testing and building a Ruby on Rails application in a GitLab CI/CD pipeline.` -## Ask follow up questions +### Explain code in the IDE + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for SaaS in GitLab 16.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for self-managed in GitLab 16.8. + +NOTE: +This feature is available in VS Code and the Web IDE only. + +`/explain` is a special command to explain the selected code in your editor. +You can also add additional instructions to be considered, for example: `/explain the performance` +See [Use GitLab Duo Chat in VS Code](#use-gitlab-duo-chat-in-vs-code) for more information. + +### Refactor code in the IDE + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for SaaS in GitLab 16.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for self-managed in GitLab 16.8. + +NOTE: +This feature is available in VS Code and the Web IDE only. + +`/refactor` is a special command to generate a refactoring suggestion for the selected code in your editor. +You can also add additional instructions to be considered, for example: `/refactor with ActiveRecord` +See [Use GitLab Duo Chat in the VS Code](#use-gitlab-duo-chat-in-vs-code) for more information. + +### Write tests in the IDE + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for SaaS in GitLab 16.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429915) for self-managed in GitLab 16.8. + +NOTE: +This feature is available in VS Code and the Web IDE only. + +`/tests` is a special command to generate a testing suggestion for the selected code in your editor. +You can also add additional instructions to be considered, for example: `/tests using the Boost.Test framework` +See [Use GitLab Duo Chat in the VS Code](#use-gitlab-duo-chat-in-vs-code) for more information. + +### Ask follow up questions You can ask follow-up questions to delve deeper into the topic or task at hand. This helps you get more detailed and precise responses tailored to your specific needs, @@ -63,10 +143,43 @@ A follow-up to the question `Write a Ruby function that prints 'Hello, World!' w ## Enable GitLab Duo Chat +### For SaaS users + To use this feature, at least one group you're a member of must have the [experiment and beta features setting](group/manage.md#enable-experiment-and-beta-features) enabled. -## Use GitLab Duo Chat +### For self-managed users + +NOTE: +Usage of GitLab Duo Chat is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using GitLab Duo Chat](ai_features.md#data-usage). + +Prerequisites: + +- You are using GitLab version 16.8 or later. +- The Ultimate license is activated in your GitLab instance by using [cloud Licensing](https://about.gitlab.com/pricing/licensing-faq/cloud-licensing/). +- All of the users in your instance have the latest version of their IDE extension. +- You are an administrator. + +To enable GitLab Duo Chat for your self-managed GitLab instance: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > General**. +1. Expand **AI-powered features** and select **Enable Experiment and Beta AI-powered features**. +1. Select **Save changes**. +1. To make sure GitLab Duo Chat works immediately, you must + [manually synchronize your subscription](#manually-synchronize-your-subscription). + +#### Manually synchronize your subscription + +You must [manually synchronize your subscription](../subscriptions/self_managed/index.md#manually-synchronize-your-subscription-details) if either: + +- You have just purchased a subscription for the Ultimate tier and have upgraded to GitLab 16.8. +- You already have a subscription for the Ultimate tier and have upgraded to GitLab 16.8. + +Without the manual synchronization, it might take up to 24 hours to activate GitLab Duo Chat on your instance. + +## Use GitLab Duo Chat in the GitLab UI 1. In the lower-left corner, select the **Help** icon. The [new left sidebar must be enabled](../tutorials/left_sidebar/index.md). @@ -84,13 +197,11 @@ To delete all previous conversations: 1. In the text box, type `/clean` and select **Send**. -## Use GitLab Duo Chat in the Web IDE and VS Code **(ULTIMATE SAAS EXPERIMENT)** +## Use GitLab Duo Chat in the Web IDE **(ULTIMATE EXPERIMENT)** > Introduced in GitLab 16.6 as an [EXPERIMENT](../policy/experiment-beta-support.md#experiment). -### Web IDE - -To use GitLab Duo Chat in the Web IDE on GitLab.com: +To use GitLab Duo Chat in the Web IDE on GitLab: 1. Open the Web IDE: 1. On the left sidebar, select **Search or go to** and find your project. @@ -107,9 +218,11 @@ To use GitLab Duo Chat in the Web IDE on GitLab.com: If you have selected code in the editor, this selection is sent along with your question to the AI. This way you can ask questions about this code selection. For instance, `Could you simplify this?`. -### GitLab Workflow extension for VS Code +## Use GitLab Duo Chat in VS Code **(ULTIMATE EXPERIMENT)** + +> Introduced in GitLab 16.6 as an [EXPERIMENT](../policy/experiment-beta-support.md#experiment). -To use GitLab Duo Chat in VS Code: +To use GitLab Duo Chat in GitLab Workflow extension for VS Code: 1. Install and set up the Workflow extension for VS Code: 1. In VS Code, download and Install the [GitLab Workflow extension for VS Code](../editor_extensions/visual_studio_code/index.md#download-the-extension). @@ -125,18 +238,26 @@ To use GitLab Duo Chat in VS Code: If you have selected code in the editor, this selection is sent along with your question to the AI. This way you can ask questions about this code selection. For instance, `Could you simplify this?`. -### Disable Chat in Web IDE and VS Code +### Perform standard task in the IDE from the context menu or by using slash commands + +Get code explained, code refactored or get tests generated for code. To do so: + +1. Select code in your editor in VS Code or in the Web IDE. +1. Type one the following slash commands into the chat field: [`/explain`](#explain-code-in-the-ide), [`/refactor`](#refactor-code-in-the-ide) or [`/tests`](#write-tests-in-the-ide). Alternatively, use the context menu to perform these tasks. -To disable GitLab Duo Chat in the Web IDE and VS Code: +When you use one of the slash commands you can also add additional instructions to be considered, for example: `/tests using the Boost.Test framework` + +### Disable Chat in VS Code + +To disable GitLab Duo Chat in VS Code: 1. Go to **Settings > Extensions > GitLab Workflow (GitLab VSCode Extension)**. 1. Clear the **Enable GitLab Duo Chat assistant** checkbox. ## Give feedback -Your feedback is important to us as we continually enhance your GitLab Duo Chat experience: - -- **Enhance Your Experience**: Leaving feedback helps us customize the Chat for your needs and improve its performance for everyone. +Your feedback is important to us as we continually enhance your GitLab Duo Chat experience. +Leaving feedback helps us customize the Chat for your needs and improve its performance for everyone. To give feedback about a specific response, use the feedback buttons in the response message. Or, you can add a comment in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/430124). diff --git a/doc/user/group/access_and_permissions.md b/doc/user/group/access_and_permissions.md index e08cfea7095..4628b7be9ce 100644 --- a/doc/user/group/access_and_permissions.md +++ b/doc/user/group/access_and_permissions.md @@ -63,8 +63,8 @@ address. This top-level group setting applies to: - The GitLab UI, including subgroups, projects, and issues. It does not apply to GitLab Pages. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/12874), the API. - In self-managed installations of GitLab 15.1 and later, you can also configure -[globally-allowed IP address ranges](../../administration/settings/visibility_and_access_controls.md#configure-globally-allowed-ip-address-ranges) -at the group level. + [globally-allowed IP address ranges](../../administration/settings/visibility_and_access_controls.md#configure-globally-allowed-ip-address-ranges) + at the group level. Administrators can combine restricted access by IP address with [globally-allowed IP addresses](../../administration/settings/visibility_and_access_controls.md#configure-globally-allowed-ip-address-ranges). @@ -118,7 +118,7 @@ To allow runner downloading, add the [outbound runner CIDR ranges](../gitlab_com > - Support for restricting access to projects in the group [added](https://gitlab.com/gitlab-org/gitlab/-/issues/14004) in GitLab 14.1.2. > - Support for restricting group memberships to groups with a subset of the allowed email domains [added](https://gitlab.com/gitlab-org/gitlab/-/issues/354791) in GitLab 15.1.1 -You can prevent users with email addresses in specific domains from being added to a group and its projects. You can define an email domain allowlist at the top-level namespace only. Subgroups do not offer the ability to define an alternative allowlist. +To ensure only users with email addresses in specific domains are added to a group and its projects, define an email domain allowlist at the top-level namespace. Subgroups do not offer the ability to define an alternative allowlist. To restrict group access by domain: diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index ded99f7c936..bbd53c09352 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -129,7 +129,7 @@ For example, if your project has the following Kubernetes clusters: | Test | `test` | Group | | Development| `*` | Group | -And the following environments are set in [`.gitlab-ci.yml`](../../../ci/yaml/index.md): +And the following environments are set in the [`.gitlab-ci.yml` file](../../../ci/index.md#the-gitlab-ciyml-file): ```yaml stages: diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index cdb11bb0548..978c893a0ec 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -21,9 +21,9 @@ DevOps Adoption shows you how groups in your organization adopt and use the most You can use Group DevOps Adoption to: - Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on -their DevOps journey. + their DevOps journey. - Find subgroups that have adopted certain features, and provide guidance to other subgroups on -how to use those features. + how to use those features. - Verify if you are getting the return on investment that you expected from GitLab.  @@ -43,11 +43,11 @@ To view DevOps Adoption: DevOps Adoption shows feature adoption for development, security, and operations. -| Category | Feature | -| --- | --- | -| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | -| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | -| Operations | Deployments<br>Pipelines<br>Runners | +| Category | Feature | +|-------------|---------| +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | ## Feature adoption diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 9cdebdd7d9d..286a1c474da 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -11,10 +11,6 @@ to them. ## Create an epic -> - The New Epic form [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211533) in GitLab 13.2. -> - In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/229621) and later, the New Epic button on the Epics list opens the New Epic form. -> - In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45948) and later, you can create a new epic from an empty roadmap. - Prerequisites: - You must have at least the Reporter role for the epic's group. @@ -175,6 +171,9 @@ To do so, either: - In the upper-right corner, select **epic actions** (**{ellipsis_v}**) and then **Reopen epic** - Use the `/reopen` [quick action](../../project/quick_actions.md). +You can also create an epic by +[promoting an issue](../../project/issues/managing_issues.md#promote-an-issue-to-an-epic). + ## Go to an epic from an issue If an issue belongs to an epic, you can go to the parent epic with the @@ -453,40 +452,6 @@ To move an issue to another epic: 1. Go to the **Child issues and epics** section. 1. Drag issues into the desired parent epic in the visible hierarchy. -### Promote an issue to an epic - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. - -Prerequisites: - -- The project to which the issue belongs must be in a group. -- You must have at least the Reporter role the project's immediate parent group. -- You must either: - - Have at least the Reporter role for the project. - - Be the author of the issue. - - Be assigned to the issue. - -You can promote an issue to an epic with the `/promote` -[quick action](../../project/quick_actions.md#issues-merge-requests-and-epics). - -NOTE: -Promoting a confidential issue to an epic makes all information -related to the issue public as epics are public to group members. - -When an issue is promoted to an epic: - -- If the issue was confidential, an additional warning is displayed first. -- An epic is created in the same group as the project of the issue. -- Subscribers of the issue are notified that the epic was created. - -The following issue metadata is copied to the epic: - -- Title, description, activity/comment thread. -- Upvotes and downvotes. -- Participants. -- Group labels that the issue already has. -- Parent epic. - ### Use an epic template for repeating issues You can create a spreadsheet template to manage a pattern of consistently repeating issues. diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 28878855098..28e77a942a6 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -4,7 +4,14 @@ group: Import and Integrate info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrating GitLab groups **(FREE ALL)** +# Migrate GitLab groups and projects by using direct transfer **(FREE ALL)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. +> - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 for project resources [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. +> - New application setting `bulk_import_enabled` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. `bulk_import` feature flag removed. +> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. You can migrate GitLab groups: @@ -16,19 +23,10 @@ You can migrate GitLab groups: You can migrate groups in two ways: - By direct transfer (recommended). -- By uploading an export file. +- By [uploading an export file](../../project/settings/import_export.md). If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance. -## Migrate groups by direct transfer (recommended) - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. -> - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 for project resources [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. -> - New application setting `bulk_import_enabled` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. `bulk_import` feature flag removed. -> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. - On self-managed GitLab, by default [migrating group items](#migrated-group-items) is not available. To show the feature, an administrator can [enable it in application settings](../../../administration/settings/import_and_export_settings.md#enable-migration-of-groups-and-projects-by-direct-transfer). @@ -59,12 +57,12 @@ We invite you to leave your feedback about migrating by direct transfer in If you want to move groups instead of copying groups, you can [transfer groups](../manage.md#transfer-a-group) if the groups are in the same GitLab instance. Transferring groups is a faster and more complete option. -### Known issues +## Known issues See [epic 6629](https://gitlab.com/groups/gitlab-org/-/epics/6629) for a list of known issues for migrating by direct transfer. -### Estimating migration duration +## Estimating migration duration Estimating the duration of migration by direct transfer is difficult. The following factors affect migration duration: @@ -111,7 +109,7 @@ There's no exact formula to reliably estimate a migration. However, the average If you are migrating large projects and encounter problems with timeouts or duration of the migration, see [Reducing migration duration](#reducing-migration-duration). -### Limits +## Limits > Eight hour time limit on migrations [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/429867) in GitLab 16.7. @@ -138,7 +136,7 @@ You can test the maximum relation size limit using these APIs: If either API produces files larger than the maximum relation size limit, group migration by direct transfer fails. -### Visibility rules +## Visibility rules After migration: @@ -154,7 +152,7 @@ After migration: If you used a private network on your source instance to hide content from the general public, make sure to have a similar setup on the destination instance, or to import into a private group. -### Prerequisites +## Prerequisites > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. @@ -181,7 +179,7 @@ To migrate groups by direct transfer: - [Configure `proxy_download`](../../../administration/object_storage.md#configure-the-common-parameters). - Ensure that the destination GitLab instance has access to the object storage of the source GitLab instance. -### Prepare user accounts +## Prepare user accounts To ensure GitLab maps users and their contributions correctly: @@ -196,7 +194,7 @@ To ensure GitLab maps users and their contributions correctly: 1. If users already exist on the destination instance and you use [SAML SSO for GitLab.com groups](../../group/saml_sso/index.md), all users must [link their SAML identity to their GitLab.com account](../../group/saml_sso/index.md#link-saml-to-your-existing-gitlabcom-account). -### Connect the source GitLab instance +## Connect the source GitLab instance Create the group you want to import to and connect the source GitLab instance: @@ -209,7 +207,7 @@ Create the group you want to import to and connect the source GitLab instance: 1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your source GitLab instance. 1. Select **Connect instance**. -### Select the groups and projects to import +## Select the groups and projects to import > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385689) in GitLab 15.8, option to import groups with or without projects. @@ -228,7 +226,7 @@ WARNING: Importing groups with projects is in [Beta](../../../policy/experiment-beta-support.md#beta). This feature is not ready for production use. -### Group import history +## Group import history > **Partially completed** status [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/394727) in GitLab 16.7. @@ -248,16 +246,17 @@ To view group import history: 1. In the upper-right corner, select **History**. 1. If there are any errors for a particular import, select **See failures** to see their details. -### Review results of the import +## Review results of the import -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429109) in GitLab 16.6 [with a flag](../../feature_flags.md) named `bulk_import_details_page`. Enabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429109) in GitLab 16.6 [with a flag](../../feature_flags.md) named `bulk_import_details_page`. Enabled by default. +> - Feature flag `bulk_import_details_page` removed in GitLab 16.8. To review the results of an import: 1. Go to the [Group import history page](#group-import-history). 1. To see the details of a failed import, select the **See failures** link on any import with a **Failed** or **Partially completed** status. -### Migrated group items +## Migrated group items The group items that are migrated depend on the version of GitLab you use on the destination. To determine if a specific group item is migrated: @@ -304,14 +303,14 @@ Group items that are migrated to the destination GitLab instance include: - Already exists in the destination GitLab instance. - Has a public email in the source GitLab instance that matches a confirmed email in the destination GitLab instance. -#### Excluded items +### Excluded items Some group items are excluded from migration because they either: - May contain sensitive information: CI/CD variables, webhooks, and deploy tokens. - Are not supported: push rules. -### Migrated project items **(BETA)** +## Migrated project items **(BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. @@ -384,7 +383,7 @@ Project items that are migrated to the destination GitLab instance include: </small> </html> -#### Issue-related items +### Issue-related items Issue-related project items that are migrated to the destination GitLab instance include: @@ -397,7 +396,7 @@ Issue-related project items that are migrated to the destination GitLab instance | Merge request URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | | Time tracking | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) | -#### Merge request-related items +### Merge request-related items Merge request-related project items that are migrated to the destination GitLab instance include: @@ -411,7 +410,7 @@ Merge request-related project items that are migrated to the destination GitLab | Issue URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | | Time tracking | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) | -#### Setting-related items +### Setting-related items Setting-related project items that are migrated to the destination GitLab instance include: @@ -422,7 +421,7 @@ Setting-related project items that are migrated to the destination GitLab instan | Project properties | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75898) | | Service Desk | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) | -#### Excluded items +### Excluded items Some project items are excluded from migration because they either: @@ -443,7 +442,7 @@ Some project items are excluded from migration because they either: - Pages domains - Remote mirrors -### Troubleshooting +## Troubleshooting In a [rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session), you can find the failure or error messages for the group import attempt using: @@ -468,7 +467,7 @@ entities.where(status: [-1]).pluck(:destination_name, :destination_namespace, :s You can also see all migrated entities with any failures related to them using an [API endpoint](../../../api/bulk_imports.md#list-all-group-or-project-migrations-entities). -#### Stale imports +### Stale imports > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352985) in GitLab 14.10. @@ -483,7 +482,7 @@ import = BulkImports::Entity.where(namespace_id: Group.id).map(&:bulk_import) import.status #=> 3 means that the import timed out. ``` -#### Error: `404 Group Not Found` +### Error: `404 Group Not Found` If you attempt to import a group that has a path comprised of only numbers (for example, `5000`), GitLab attempts to find the group by ID instead of the path. This causes a `404 Group Not Found` error in GitLab 15.4 and earlier. @@ -499,7 +498,7 @@ To solve this, you must change the source group path to include a non-numerical - The [Groups API](../../../api/groups.md#update-group). -#### Other `404` errors +### Other `404` errors You can receive other `404` errors when importing a group, for example: @@ -511,7 +510,7 @@ You can receive other `404` errors when importing a group, for example: This error indicates a problem transferring from the _source_ instance. To solve this, check that you have met the [prerequisites](#prerequisites) on the source instance. -#### Reducing migration duration +### Reducing migration duration A single direct transfer migration runs 5 entities (groups or projects) per import at a time, independent of the number of workers available on the destination instance. That said, having more workers on the destination instance speeds up migration by decreasing the time it takes to import each entity. @@ -529,157 +528,6 @@ Distributing projects in different groups helps to avoid timeouts. If several la The GitLab UI can only migrate top-level groups. Using the API, you can also migrate subgroups. -## Migrate groups by uploading an export file (deprecated) - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. -> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. - -WARNING: -This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by -[migrating groups by direct transfer](#migrate-groups-by-direct-transfer-recommended). However, this feature is still recommended for migrating groups between -offline systems. To follow progress on an alternative solution for [offline environments](../../application_security/offline_deployments/index.md), see -[the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). - -Prerequisites: - -- Owner role on the group to migrate. - -Using file exports, you can: - -- Export any group to a file and upload that file to another GitLab instance or to another location on the same instance. -- Use either the GitLab UI or the [API](../../../api/group_import_export.md). -- Migrate groups one by one, then export and import each project for the groups one by one. - -GitLab maps user contributions correctly when an admin access token is used to perform the import. GitLab does not map -user contributions correctly when you are importing from a self-managed instance to GitLab.com. Correct mapping of user -contributions when importing from a self-managed instance to GitLab.com can be preserved with paid involvement of -Professional Services team. - -Note the following: - -- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. -- To preserve group-level relationships from imported projects, export and import groups first so that projects can - be imported into the desired group structure. -- Imported groups are given a `private` visibility level, unless imported into a parent group. -- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. -- You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) - and vice versa. The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're - exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, - see [downgrading from EE to CE](../../../index.md). - -### Compatibility - -> Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/383682) in GitLab 15.8. - -Group file exports are in NDJSON format. - -You can import group file exports that were exported from a version of GitLab up to two -[minor](../../../policy/maintenance.md#versioning) versions behind, which is similar to our process for -[security releases](../../../policy/maintenance.md#security-releases). - -For example: - -| Destination version | Compatible source versions | -|:--------------------|:---------------------------| -| 13.0 | 13.0, 12.10, 12.9 | -| 13.1 | 13.1, 13.0, 12.10 | - -### Exported contents - -The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) -file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch -for your version of GitLab to check which items can be imported to the destination GitLab instance. For example, -[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). - -Group items that are exported include: - -- Milestones -- Group Labels (_without_ associated label priorities) -- Boards and Board Lists -- Badges -- Subgroups (including all the aforementioned data) -- Epics - - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) -- Events -- [Wikis](../../project/wiki/group.md) - ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) -- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) - -Items that are **not** exported include: - -- Projects -- Runner tokens -- SAML discovery tokens - -### Preparation - -- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make -sure these users exist before importing the desired groups. -- Users must set a public email in the source GitLab instance that matches their confirmed primary email in the destination GitLab instance. Most users receive an email asking them to confirm their email address. - -### Enable export for a group - -Prerequisites: - -- You must have the Owner role for the group. - -To enable import and export for a group: - -1. On the left sidebar, at the bottom, select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Visibility and access controls**. -1. In the **Import sources** section, select the checkboxes for the sources you want. - -### Export a group - -Prerequisites: - -- You must have the Owner role for the group. - -To export the contents of a group: - -1. On the left sidebar, select **Search or go to** and find your group. -1. Select **Settings > General**. -1. In the **Advanced** section, select **Export group**. -1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) - in a compressed tar archive, with contents in NDJSON format. -1. Alternatively, you can download the export from the UI: - - 1. Return to your group's **Settings > General** page. - 1. In the **Advanced** section, select **Download export**. - You can also generate a new file by selecting **Regenerate export**. - -You can also export a group [using the API](../../../api/group_import_export.md). - -### Import the group - -1. On the left sidebar, at the top, select **Create new** (**{plus}**) and **New subgroup**. -1. Select the **import an existing group** link. -1. Enter your group name. -1. Accept or modify the associated group URL. -1. Select **Choose file...**. -1. Select the file that you exported in the [Export a group](#export-a-group) section. -1. To begin importing, select **Import**. - -Your newly imported group page appears after the operation completes. - -NOTE: -The maximum import file size can be set by the administrator, default is `0` (unlimited). -As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the -[Application settings API](../../../api/settings.md#change-application-settings) or the -[Admin Area](../../../administration/settings/account_and_limit_settings.md). -Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. - -### Rate limits - -To help avoid abuse, by default, users are rate limited to: - -| Request Type | Limit | -| ---------------- | ---------------------------------------- | -| Export | 6 groups per minute | -| Download export | 1 download per group per minute | -| Import | 6 groups per minute | - ## Automate group and project import **(PREMIUM ALL)** For information on automating user, group, and project import API calls, see diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart_v15_4.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart_v15_4.png Binary files differdeleted file mode 100644 index f7963c170e1..00000000000 --- a/doc/user/group/insights/img/insights_example_stacked_bar_chart_v15_4.png +++ /dev/null diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 6ca37cb9a2c..0cb1ad093a5 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -1,94 +1,11 @@ --- -stage: Plan -group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../project/insights/index.md' +remove_date: '2024-04-20' --- -# Insights for groups **(ULTIMATE ALL)** +This document was moved to [another location](../../project/insights/index.md). -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in GitLab 12.0. - -Configure insights to explore data about you group's activity, such as -triage hygiene, issues created or closed in a given period, and average time for merge -requests to be merged. -You can also create custom insights reports that are relevant for your group. - -## View group insights - -Prerequisites: - -- You must have [permission](../../permissions.md#group-members-permissions) to view the group. -- You must have access to a project to view information about its merge requests and issues, - and permission to view them if they are confidential. - -To access your group's insights: - -1. On the left sidebar, select **Search or go to** and find your group. -1. Select **Analyze > Insights**. - -## Interact with insights charts - -You can interact with the insights charts to view details about your group's activity. - - - -### Display different reports - -To display one of the available reports on the insights page, from the **Select report** dropdown list, -select the report you want to display. - -### View bar chart annotations - -To view annotations, hover over each bar in the chart. - -### Zoom in on chart - -Insights display data from the last 90 days. You can zoom in to display data only from a subset of the 90-day range. - -To do this, select the pause icons (**{status-paused}**) and slide them along the horizontal axis: - -- To change the start date, slide the left pause icon to the left or right. -- To change the end date, slide the right pause icon to the left or right. - -### Exclude dimensions from charts - -By default, insights display all available dimensions on the chart. - -To exclude a dimension, from the legend below the chart, select the name of the dimension. - -### Drill down on charts - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372215/) in GitLab 16.7. - -You can drill down into the data of the **Bugs created per month by priority** and **Bugs created per month by severity** charts from the [default configuration file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml). - -To view a drill-down report of the data for a specific priority or severity in a month: - -- On the chart, select the bar stack you want to drill down on. - -## Configure group insights - -GitLab reads insights from the -[default configuration file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml). - -To configure group insights: - -1. Create a new file [`.gitlab/insights.yml`](../../project/insights/index.md#configure-project-insights) -in a project that belongs to your group. -1. On the left sidebar, select **Search or go to** and find your group. -1. Select **Settings > General**. -1. Expand **Analytics** and find the **Insights** section. -1. Select the project that contains your `.gitlab/insights.yml` configuration file. -1. Select **Save changes**. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/group/issues_analytics/img/enhanced_issue_analytics_v16_7.png b/doc/user/group/issues_analytics/img/enhanced_issue_analytics_v16_7.png Binary files differindex 519e56acaa5..b253fe336ee 100644 --- a/doc/user/group/issues_analytics/img/enhanced_issue_analytics_v16_7.png +++ b/doc/user/group/issues_analytics/img/enhanced_issue_analytics_v16_7.png diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md index 28f4026b3e3..efd4a46c710 100644 --- a/doc/user/group/issues_analytics/index.md +++ b/doc/user/group/issues_analytics/index.md @@ -4,17 +4,18 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Issue analytics for groups **(PREMIUM ALL)** +# Issue analytics **(PREMIUM ALL)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7478) in GitLab 11.5. +> - Issue analytics for groups [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7478) in GitLab 11.5. +> - Issue analytics for projects [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196561) in GitLab 12.9. Issue analytics is a bar graph which illustrates the number of issues created each month. -The default time span is 13 months, which includes the current month, and the 12 months -prior. +The default time span is 13 months, which includes the current month, and the 12 months prior. +Issue analytics is available for projects and groups. To access the chart: -1. On the left sidebar, select **Search or go to** and find your group. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Issue analytics**. You can also access the chart from the [Value Streams Dashboard](../../analytics/value_streams_dashboard.md) through the **New issues** drill-down report. @@ -39,12 +40,11 @@ shows a total of 15 months for the chart in the GitLab.org group. ## Enhanced issue analytics **(ULTIMATE ALL)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233905/) in GitLab 16.4 [with a flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233905/) in GitLab 16.3 [with a flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/437542) in GitLab 16.8. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, an administrator can -[enable the feature flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. On GitLab.com, this feature is not -available. This feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `issues_completed_analytics_feature_flag`. On GitLab.com, this feature is available. Enhanced issue analytics display the additional metric "Issues closed", which represents the total number of resolved issues in your group over a selected period. You can use this metric to improve the overall turn-around time and value delivered to your customers. diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index 877db58b716..58c3f837e26 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -165,7 +165,7 @@ Transferring groups moves them from one place to another in the same GitLab inst - Convert a subgroup into a top-level group by transferring it out of its current group. If you need to copy a group to a different GitLab instance, -[migrate the group by direct transfer](import/index.md#migrate-groups-by-direct-transfer-recommended). +[migrate the group by direct transfer](import/index.md). When transferring groups, note: @@ -449,11 +449,6 @@ for the ability to set merge request approval rules for groups is tracked in > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. -WARNING: -This feature is in [Beta](../../policy/experiment-beta-support.md#beta). -Beta users should read about the [known limitations](../project/repository/code_suggestions/index.md#known-limitations). -We look forward to hearing your [feedback](../project/repository/code_suggestions/index.md#feedback). - You can give all users in a group and its subgroups access to [Code Suggestions](../project/repository/code_suggestions/index.md). This setting [cascades to all projects](../project/merge_requests/approvals/settings.md#settings-cascading) in the group. diff --git a/doc/user/group/saml_sso/troubleshooting.md b/doc/user/group/saml_sso/troubleshooting.md index a2576f37ac9..1e7c749a705 100644 --- a/doc/user/group/saml_sso/troubleshooting.md +++ b/doc/user/group/saml_sso/troubleshooting.md @@ -248,6 +248,11 @@ For GitLab.com, alternatively, when users need to [link SAML to their existing G ### Users receive a 404 **(PREMIUM SAAS)** +If the user receives a `404` after signing in successfully, check if you have IP restrictions configured. IP restriction settings are configured: + +- On GitLab.com, [at the group level](../../../user/group/access_and_permissions.md#restrict-group-access-by-ip-address). +- For GitLab self-managed, [at the instance level](../../../administration/reporting/ip_addr_restrictions.md). + Because SAML SSO for groups is a paid feature, your subscription expiring can result in a `404` error when you're signing in using SAML SSO on GitLab.com. If all users are receiving a `404` when attempting to sign in using SAML, confirm [there is an active subscription](../../../subscriptions/gitlab_com/index.md#view-your-gitlab-saas-subscription) being used in this SAML SSO namespace. diff --git a/doc/user/group/saml_sso/troubleshooting_scim.md b/doc/user/group/saml_sso/troubleshooting_scim.md index 3dcb2d93096..47b2144c7ff 100644 --- a/doc/user/group/saml_sso/troubleshooting_scim.md +++ b/doc/user/group/saml_sso/troubleshooting_scim.md @@ -111,6 +111,26 @@ Changing the SAML or SCIM configuration or provider can cause the following prob the SCIM app. 1. Use the same SCIM API to update the SCIM `extern_uid` for the user on GitLab.com. +## The member's email address is not allowed for this group + +SCIM provisioning may fail with HTTP status `412` and the following error message: + +```plaintext +The member's email address is not allowed for this group. Check with your administrator. +``` + +This error occurs when both of the following are true: + +- [Restrict group access by domain](../access_and_permissions.md) is configured + for the group. +- The user account being provisioned has an email domain that is not allowed. + +To resolve this issue, you can do either of the following: + +- Add the user account's email domain to the list of allowed domains. +- Disable the [Restrict group access by domain](../access_and_permissions.md) + feature by removing all domains. + ## Search Rails logs for SCIM requests **(PREMIUM SAAS)** GitLab.com administrators can search for SCIM requests in the `api_json.log` using the `pubsub-rails-inf-gprd-*` index in @@ -149,9 +169,9 @@ The first workaround is: 1. Have the end user [link SAML to their existing GitLab.com account](index.md#link-saml-to-your-existing-gitlabcom-account). 1. After the user has done this, initiate a SCIM sync from your identity provider. -If the SCIM sync completes without the same error, GitLab has -successfully linked the SCIM identity to the existing user account, and the user -should now be able to sign in using SAML SSO. + If the SCIM sync completes without the same error, GitLab has + successfully linked the SCIM identity to the existing user account, and the user + should now be able to sign in using SAML SSO. If the error persists, the user most likely already exists, has both a SAML and SCIM identity, and a SCIM identity that is set to `active: false`. To resolve @@ -166,7 +186,7 @@ this: If any of this information does not match, [contact GitLab Support](https://support.gitlab.com/). 1. Use the API to [update the SCIM provisioned user's `active` value to `true`](/ee/development/internal_api/index.md#update-a-single-scim-provisioned-user). 1. If the update returns a status code `204`, have the user attempt to sign in -using SAML SSO. + using SAML SSO. ## Azure Active Directory diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index a63d4a98fa2..a43de3ef73b 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -10,12 +10,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can organize GitLab [groups](../index.md) into subgroups. You can use subgroups to: -- Separate internal and external organizations. Because every subgroup can have its own +- Separate internal and external content. Because every subgroup can have its own [visibility level](../../public_access.md), you can host groups for different purposes under the same parent group. -- Organize large projects. You can use subgroups to give different access to parts of +- Organize large projects. You can use subgroups to manage who can access parts of the source code. -- Manage people and control visibility. Give a user a different +- Manage permissions. Give a user a different [role](../../permissions.md#group-members-permissions) for each group they're [a member of](#subgroup-membership). Subgroups can: @@ -25,7 +25,7 @@ Subgroups can: - Be nested up to 20 levels. - Use [runners](../../../ci/runners/index.md) registered to parent groups: - Secrets configured for the parent group are available to subgroup jobs. - - Users with the Maintainer role in projects that belong to subgroups can see the details of runners registered to + - Users with at least the Maintainer role in projects that belong to subgroups can see the details of runners registered to parent groups. For example: @@ -52,7 +52,7 @@ graph TD Prerequisites: - To view private nested subgroups, you must be a direct or inherited member of -the private subgroup. + the private subgroup. To view the subgroups of a group: @@ -117,7 +117,7 @@ For more information, view the [permissions table](../../permissions.md#group-me ## Subgroup membership When you add a member to a group, that member is also added to all subgroups of that group. -The member's permissions are inherited from the group's parent. +The member's permissions are inherited from the group into all subgroups. Subgroup members can be: @@ -189,8 +189,8 @@ Members can be [filtered by inherited or direct membership](../index.md#filter-a Users with the Owner role in a subgroup can add members to it. -You can't give a user a role in a subgroup that is lower than the roles the user has in ancestor groups. -To override a user's role in an ancestor group, add the user to the subgroup again with a higher role. +You can't give a user a role in a subgroup that is lower than the roles the user has in parent groups. +To override a user's role in a parent group, add the user to the subgroup again with a higher role. For example: - If User 1 is added to group _Two_ with the Developer role, User 1 inherits that role in every subgroup of group _Two_. @@ -201,7 +201,7 @@ For example: ## Mention subgroups -Mentioning subgroups ([`@<subgroup_name>`](../../discussions/index.md#mentions)) in issues, commits, and merge requests +Mentioning subgroups ([`@<subgroup_name>`](../../discussions/index.md#mentions)) in epics, issues, commits, and merge requests notifies all direct members of that group. Inherited members of a subgroup are not notified by mentions. Mentioning works the same as for projects and groups, and you can choose the group of members to be notified. diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 0fdd572ed7c..7e077c7065c 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -166,7 +166,7 @@ In this example, milestones have been created and CI/CD for testing and setting - 14:00: Push branch and create a merge request that contains the [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically). **Code** stage stops and **Test** and **Review** stages start. -- GitLab CI/CD takes 5 minutes to run scripts defined in [`.gitlab-ci.yml`](../../../ci/yaml/index.md). +- GitLab CI/CD takes 5 minutes to run scripts defined in the [`.gitlab-ci.yml` file](../../../ci/index.md#the-gitlab-ciyml-file). - 19:00: Merge the merge request. **Review** stage stops and **Staging** stage starts. - 19:30: Deployment to the `production` environment finishes. **Staging** stops. @@ -305,7 +305,7 @@ In GitLab 13.8 and earlier, deployment frequency metrics are calculated based on Prerequisites: - To view deployment metrics, you must have a -[production environment configured](#how-value-stream-analytics-identifies-the-production-environment). + [production environment configured](#how-value-stream-analytics-identifies-the-production-environment). To view lifecycle metrics: @@ -442,11 +442,11 @@ After you create a value stream, you can customize it to suit your purposes. To 1. In the upper-right corner, select the dropdown list, then select a value stream. 1. Next to the value stream dropdown list, select **Edit**. 1. Optional: - - Rename the value stream. - - Hide or re-order default stages. - - Remove existing custom stages. - - To add new stages, select **Add another stage**. - - Select the start and end events for the stage. + - Rename the value stream. + - Hide or re-order default stages. + - Remove existing custom stages. + - To add new stages, select **Add another stage**. + - Select the start and end events for the stage. 1. Optional. To undo any modifications, select **Restore value stream defaults**. 1. Select **Save Value Stream**. diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 3b20125ff03..839bce217b7 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -52,7 +52,7 @@ To import the project: This project provides you with: - A [cluster on Google Cloud Platform (GCP)](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke/-/blob/master/gke.tf) -with defaults for name, location, node count, and Kubernetes version. + with defaults for name, location, node count, and Kubernetes version. - The [GitLab agent for Kubernetes](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke/-/blob/master/agent.tf) installed in the cluster. ## Register the agent @@ -73,10 +73,10 @@ To create a GitLab agent for Kubernetes: To set up your project to communicate to GCP and the GitLab API: 1. To authenticate GCP with GitLab, create a [GCP service account](https://cloud.google.com/docs/authentication#service-accounts) -with following roles: `Compute Network Viewer`, `Kubernetes Engine Admin`, `Service Account User`, and `Service Account Admin`. Both User and Admin -service accounts are necessary. The User role impersonates the [default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) -when [creating the node pool](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/using_gke_with_terraform#node-pool-management). -The Admin role creates a service account in the `kube-system` namespace. + with following roles: `Compute Network Viewer`, `Kubernetes Engine Admin`, `Service Account User`, and `Service Account Admin`. Both User and Admin + service accounts are necessary. The User role impersonates the [default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) + when [creating the node pool](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/using_gke_with_terraform#node-pool-management). + The Admin role creates a service account in the `kube-system` namespace. 1. Download the JSON file with the service account key you created in the previous step. 1. On your computer, encode the JSON file to `base64` (replace `/path/to/sa-key.json` to the path to your key): diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index d76f5dd736a..616e15dc230 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -84,13 +84,13 @@ To use a Terraform template: ```yaml variables: - TF_STATE_NAME: default - # If your terraform files are in a subdirectory, set TF_ROOT accordingly. For example: - # TF_ROOT: terraform/production + TF_STATE_NAME: default + # If your terraform files are in a subdirectory, set TF_ROOT accordingly. For example: + # TF_ROOT: terraform/production ``` 1. Optional. Override in your `.gitlab-ci.yml` file the attributes present -in the template you fetched to customize your configuration. + in the template you fetched to customize your configuration. ### Terraform template recipes diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index 04d6caff0ba..6662f6a9dcb 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -1,10 +1,11 @@ --- stage: Deploy group: Environments +description: Terraform and Kubernetes deployments. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Infrastructure management **(FREE ALL)** +# Manage your infrastructure **(FREE ALL)** With the rise of DevOps and SRE approaches, infrastructure management becomes codified, automatable, and software development best practices gain their place around infrastructure diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 01bd7ce0ba6..372442bb53f 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -25,13 +25,12 @@ When this list is rendered, it looks like this: - Dog - Turtle -These styles are **valid for GitLab only**. The [GitLab documentation website](https://docs.gitlab.com) -and the [main GitLab website](https://about.gitlab.com) use [Kramdown](https://kramdown.gettalong.org) instead. - -You should not view this page in the documentation, but instead [view these styles as they appear on GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md). +NOTE: +As this Markdown specification is **valid for GitLab only**, you should +[view these styles as they appear on GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md). -GitLab Flavored Markdown extends the [CommonMark specification](https://spec.commonmark.org/current/). -It was inspired by [GitHub Flavored Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax). +We do our best to render the Markdown faithfully here, however the [GitLab documentation website](https://docs.gitlab.com) +and the [GitLab handbook](https://handbook.gitlab.com) use a different Markdown processor. ## Where you can use GitLab Flavored Markdown @@ -39,44 +38,58 @@ You can use GitLab Flavored Markdown in the following areas: - Comments - Issues +- Epics - Merge requests - Milestones - Snippets (the snippet must be named with a `.md` extension) - Wiki pages - Markdown documents inside repositories -- Epics You can also use other rich text files in GitLab. You might have to install a dependency to do so. For more information, see the [`gitlab-markup` gem project](https://gitlab.com/gitlab-org/gitlab-markup). ### Differences between GitLab Flavored Markdown and standard Markdown -GitLab uses standard CommonMark formatting. However, GitLab Flavored Markdown -extends standard Markdown with features made specifically for GitLab. +<!-- +Use this topic to list features that are not present in standard Markdown. +Don't repeat this information in each individual topic, unless there's a specific +reason, like in "Newlines". +--> -Features not found in standard Markdown: +GitLab Flavored Markdown consists of the following: + +- Core Markdown features, based on the [CommonMark specification](https://spec.commonmark.org/current/). +- Extensions from [GitHub Flavored Markdown](https://github.github.com/gfm/). +- Extensions made specifically for GitLab. + +All standard Markdown formatting should work as expected in GitLab. Some standard +functionality is extended with additional features, without affecting the standard usage. + +The following features are not found in standard Markdown: - [Color chips written in `HEX`, `RGB` or `HSL`](#colors) - [Diagrams and flowcharts](#diagrams-and-flowcharts) - [Emoji](#emoji) +- [Footnotes](#footnotes) - [Front matter](#front-matter) +- [GitLab-specific references](#gitlab-specific-references) - [Inline diffs](#inline-diff) - [Math equations and symbols written in LaTeX](#math) -- [Task Lists](#task-lists) +- [Strikethrough](#emphasis) - [Table of Contents](#table-of-contents) -- [Wiki specific Markdown](#wiki-specific-markdown) +- [Tables](#tables) +- [Task lists](#task-lists) +- [Wiki-specific Markdown](#wiki-specific-markdown) -Features [extended from standard Markdown](#features-extended-from-standard-markdown): +The following features are extended from standard Markdown: | Standard Markdown | Extended Markdown in GitLab | |---------------------------------------|---------------------------------------------------------------------------------------| -| [blockquotes](#blockquotes) | [multi-line blockquotes](#multiline-blockquote) | -| [code blocks](#code-spans-and-blocks) | [colored code and syntax highlighting](#colored-code-and-syntax-highlighting) | -| [emphasis](#emphasis) | [multiple underscores in words](#multiple-underscores-in-words-and-mid-word-emphasis) | -| [headers](#headers) | [linkable Header IDs](#header-ids-and-links) | -| [images](#images) | [embedded videos](#videos) and [audio](#audio) | -| [line breaks](#line-breaks) | [more line break control](#newlines) | -| [links](#links) | [automatically linking URLs](#url-auto-linking) | +| [Blockquotes](#blockquotes) | [Multiline blockquotes](#multiline-blockquote) | +| [Code blocks](#code-spans-and-blocks) | [Colored code and syntax highlighting](#colored-code-and-syntax-highlighting) | +| [Headings](#headings) | [Linkable heading IDs](#heading-ids-and-links) | +| [Images](#images) | [Embedded videos](#videos) and [audio](#audio) | +| [Links](#links) | [Automatically linking URLs](#url-auto-linking) | ## Markdown and accessibility @@ -85,27 +98,25 @@ This content should be as accessible as possible to your audience. The following list is not exhaustive, but it provides guidance for some of the GLFM styles to pay particular attention to: -### Headings +### Accessible headings Use heading formatting to create a logical heading structure. The structure of headings on a page should make sense, like a good table of contents. Ensure that there is only one `h1` element on a page, that heading levels are not skipped, and that they are nested correctly. -### Tables +### Accessible tables To keep tables accessible and scannable, tables should not have any empty cells. If there is no otherwise meaningful value for a cell, consider entering **N/A** for "not applicable" or **None**. -### Images and videos +### Accessible images and videos Describe the image or video in the `[alt text]`. Make the description accurate, succinct, and unique. Don't use `image of` or `video of` in the description. For more information, see [WebAim Alternative Text](https://webaim.org/techniques/alttext/). -## Features not found in standard Markdown - -The following features are not found in standard Markdown. +## Colors -### Colors +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colors). Markdown does not support changing text color. @@ -132,9 +143,6 @@ display a color chip next to the color code. For example: - `HSLA(540,70%,50%,0.3)` ``` -[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colors) -to see the color chips next to the color code: - - `#F00` - `#F00A` - `#FF0000` @@ -145,7 +153,7 @@ to see the color chips next to the color code: - `HSL(540,70%,50%)` - `HSLA(540,70%,50%,0.3)` -### Diagrams and flowcharts +## Diagrams and flowcharts You can generate diagrams from text by using: @@ -155,7 +163,9 @@ You can generate diagrams from text by using: In wikis, you can also add and edit diagrams created with the [diagrams.net editor](#diagramsnet-editor). -#### Mermaid +### Mermaid + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#mermaid). Visit the [official page](https://mermaidjs.github.io/) for more details. The [Mermaid Live Editor](https://mermaid-js.github.io/mermaid-live-editor/) helps you @@ -220,24 +230,20 @@ graph TB end ``` -#### PlantUML +### PlantUML PlantUML integration is enabled on GitLab.com. To make PlantUML available in self-managed installation of GitLab, a GitLab administrator [must enable it](../administration/integration/plantuml.md). -#### Kroki +### Kroki To make Kroki available in GitLab, a GitLab administrator must enable it. For more information, see the [Kroki integration](../administration/integration/kroki.md) page. -### Emoji +## Emoji [View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#emoji). -::Tabs - -:::TabTitle Rendered Markdown - Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/monkey.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":monkey:" alt=":monkey:"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/star2.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":star2:" alt=":star2:"> to your <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":speech_balloon:" alt=":speech_balloon:">. Well we have a gift for you: <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/zap.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":zap:" alt=":zap:">You can use emoji anywhere GitLab Flavored Markdown is supported. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/v.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":v:" alt=":v:"> @@ -248,7 +254,7 @@ If you're new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-f Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0;border:0;padding:0;" title=":thumbsup:" alt=":thumbsup:"> -:::TabTitle Code +The above paragraphs in raw Markdown: ```markdown Sometimes you want to :monkey: around a bit and add some :star2: to your @@ -267,12 +273,10 @@ Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. :thumbsup: ``` -::EndTabs - -#### Emoji and your operating system +### Emoji and your operating system The previous emoji example uses hard-coded images. Rendered emoji -in GitLab may be different depending on the OS and browser used. +in GitLab might look different depending on the OS and browser used. Most emoji are natively supported on macOS, Windows, iOS, Android, and fall back on image-based emoji where there is no support. @@ -280,14 +284,14 @@ emoji where there is no support. <!-- vale gitlab.Spelling = NO --> On Linux, you can download [Noto Color Emoji](https://github.com/googlefonts/noto-emoji) -to get full native emoji support. Ubuntu 18.04 (like many modern Linux distributions) has +to get full native emoji support. Ubuntu 22.04 (like many modern Linux distributions) has this font installed by default. <!-- vale gitlab.Spelling = YES --> To learn more about adding custom emoji, see [Custom emoji](emoji_reactions.md#custom-emoji). -### Front matter +## Front matter Front matter is metadata included at the beginning of a Markdown document, preceding the content. This data can be used by static site generators like [Jekyll](https://jekyllrb.com/docs/front-matter/), @@ -349,7 +353,7 @@ $example = array( --- ``` -### Inline diff +## Inline diff [View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-diff). @@ -377,8 +381,8 @@ However, you cannot mix the wrapping tags: - [- deletion -} ``` -If your diff includes words in `` `code` `` font, make sure to escape each backtick `` ` `` with a -backslash <code>\</code>. Otherwise the diff highlight does not render correctly: +Diff highlighting doesn't work with `` `inline code` ``. If your text includes backticks (`` ` ``), escape +each backtick with a backslash <code>\</code>: ```markdown - {+ Just regular text +} @@ -388,7 +392,7 @@ backslash <code>\</code>. Otherwise the diff highlight does not render corre  -### Math +## Math > - LaTeX-compatible fencing [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21757) in GitLab 15.4 [with a flag](../administration/feature_flags.md) named `markdown_dollar_math`. Disabled by default. Enabled on GitLab.com. > - LaTeX-compatible fencing [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/371180) in GitLab 15.8. Feature flag `markdown_dollar_math` removed. @@ -397,7 +401,7 @@ backslash <code>\</code>. Otherwise the diff highlight does not render corre Math written in LaTeX syntax is rendered with [KaTeX](https://github.com/KaTeX/KaTeX). _KaTeX only supports a [subset](https://katex.org/docs/supported.html) of LaTeX._ -This syntax also works for the Asciidoctor `:stem: latexmath`. For details, see +This syntax also works in AsciiDoc wikis and files using `:stem: latexmath`. For details, see the [Asciidoctor user manual](https://asciidoctor.org/docs/user-manual/#activating-stem-support). To prevent malicious activity, GitLab renders only the first 50 inline math instances. @@ -445,7 +449,7 @@ $$ a^2+b^2=c^2 $$ -### Task lists +## Task lists > Inapplicable checkboxes [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85982) in GitLab 15.3. @@ -453,12 +457,12 @@ $$ You can add task lists anywhere Markdown is supported. -- In issues, merge requests, and comments, you can select the boxes. +- In issues, merge requests, epics, and comments, you can select the boxes. - In all other places, you cannot select the boxes. You must edit the Markdown manually by adding or removing an `x` in the brackets. Besides complete and incomplete, tasks can also be **inapplicable**. Selecting an inapplicable checkbox -in an issue, merge request, or comment has no effect. +in an issue, merge request, epic, or comment has no effect. To create a task list, follow the format of an ordered or unordered list: @@ -480,28 +484,40 @@ To create a task list, follow the format of an ordered or unordered list:  -### Table of contents +To include task lists in tables, [use HTML list tags or HTML tables](#task-lists-in-tables). + +## Table of contents + +A table of contents is an unordered list that links to subheadings in the document. +You can add a table of contents to issues, merge requests, and epics, but you can't add one +to notes or comments. + +Add one of these tags on their own line to the **description** field of any of the supported +content types: <!-- -The following paragraphs use HTML to work around a Markdown bug. -Do not change it back to a Markdown backticks. +Tags for the table of contents are presented in a code block to work around a Markdown bug. +Do not change the code block back to single backticks. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/359077. --> -<!-- vale gitlab.Uppercase = NO --> -A table of contents is an unordered list that links to subheadings in the document. -You can add a table of contents to issues and merge requests, but you can't add one -to notes or comments. Add either the `[[_TOC_]]` or <code>[TOC]</code> tag on its own line -to the **Description** field of any of the supported content types: -<!-- vale gitlab.Uppercase = YES --> -NOTE: -A table of contents renders also when you use <code>`[TOC]`</code>, regardless of being on its own line or not. -This behavior is unintended. For more information, see [issue 359077](https://gitlab.com/gitlab-org/gitlab/-/issues/359077). +```markdown +[[_TOC_]] +or +[TOC] +``` - Markdown files. - Wiki pages. - Issues. - Merge requests. +- Epics. + +NOTE: +A table of contents renders also when you use the TOC code in single square brackets, regardless of +being on its own line or not. +This behavior is unintended. +For more information, see [issue 359077](https://gitlab.com/gitlab-org/gitlab/-/issues/359077). ```markdown This sentence introduces my wiki page. @@ -519,11 +535,13 @@ Second section content.  -### Wiki-specific Markdown +## Wiki-specific Markdown The following topics show how links inside wikis behave. -#### Wiki - direct page link +When linking to wiki pages, you should use the **page slug** rather than the page name. + +### Wiki - direct page link A direct page link includes the slug for a page that points to that page, at the base level of the wiki. @@ -534,7 +552,7 @@ This example links to a `documentation` page at the root of your wiki: [Link to Documentation](documentation) ``` -#### Wiki - direct file link +### Wiki - direct file link A direct file link points to a file extension for a file, relative to the current page. @@ -545,7 +563,7 @@ it links to `<your_wiki>/documentation/file.md`: [Link to File](file.md) ``` -#### Wiki - hierarchical link +### Wiki - hierarchical link A hierarchical link can be constructed relative to the current wiki page by using `./<page>`, `../<page>`, and so on. @@ -578,7 +596,7 @@ it links to `<your_wiki>/documentation/main.md`: [Link to Related Page](../main.md) ``` -#### Wiki - root link +### Wiki - root link A root link starts with a `/` and is relative to the wiki root. @@ -594,7 +612,7 @@ This example links to `<wiki_root>/miscellaneous.md`: [Link to Related Page](/miscellaneous.md) ``` -#### diagrams.net editor +### diagrams.net editor > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322174) in GitLab 15.10. @@ -604,43 +622,54 @@ the plain text editor and the rich text editor. For more information, see [Diagrams.net](../administration/integration/diagrams_net.md). -##### Plain text editor +#### Plain text editor To create a diagram in the plain text editor: +1. On the wiki page you want to edit, select **Edit**. +1. In the text box, make sure you're using the plain text editor + (the button on the bottom left says **Switch to rich text editing**). 1. In the editor's toolbar, select **Insert or edit diagram** (**{diagram}**). -1. Use the diagrams.net editor to create the diagram. +1. Create the diagram in the [app.diagrams.net](https://app.diagrams.net/) editor. 1. Select **Save & exit**. A Markdown image reference to the diagram is inserted in the wiki content. To edit a diagram in the plain text editor: -1. Place the plain text editor's text field cursor in a Markdown image reference -that contains the diagram. -1. Select **Insert or edit diagram** (**{diagram}**) in the plain text editor. -1. Use the diagrams.net editor to edit the diagram. +1. On the wiki page you want to edit, select **Edit**. +1. In the text box, make sure you're using the plain text editor + (the button on the bottom left says **Switch to rich text editing**). +1. Position your cursor in the Markdown image reference that contains the diagram. +1. Select **Insert or edit diagram** (**{diagram}**). +1. Edit the diagram in the [app.diagrams.net](https://app.diagrams.net/) editor. 1. Select **Save & exit**. A Markdown image reference to the diagram is inserted in the wiki content, replacing the previous diagram. -##### Rich text editor +#### Rich text editor To create a diagram in the rich text editor: +1. On the wiki page you want to edit, select **Edit**. +1. In the text box, make sure you're using the rich text editor + (the button on the bottom left says **Switch to plain text editing**). 1. In the editor's toolbar, select **More options** (**{plus}**). 1. In the dropdown list, select **Create or edit diagram**. -1. Use the diagrams.net editor to create the diagram. +1. Create the diagram in the [app.diagrams.net](https://app.diagrams.net/) editor. 1. Select **Save & exit**. The diagram as visualized in the diagrams.net editor is inserted in the wiki content. To edit a diagram in the rich text editor: +1. On the wiki page you want to edit, select **Edit**. +1. In the text box, make sure you're using the rich text editor + (the button on the bottom left says **Switch to plain text editing**). 1. Select the diagram that you want to edit. 1. In the floating toolbar, select **Edit diagram** (**{diagram}**). -1. Use the diagrams.net editor to edit the diagram. +1. Edit the diagram in the [app.diagrams.net](https://app.diagrams.net/) editor. 1. Select **Save & exit**. The selected diagram is replaced with an updated version. @@ -657,54 +686,55 @@ version to reference other projects from the same namespace. GitLab Flavored Markdown recognizes the following: -| references | input | cross-project reference | shortcut inside same namespace | -|:----------------------------------------------------------------------------|:------------------------------|:----------------------------------------|:-------------------------------| -| specific user | `@user_name` | | | -| specific group | `@group_name` | | | -| entire team | [`@all`](discussions/index.md#mentioning-all-members) | | | -| project | `namespace/project>` | | | -| issue | ``#123`` | `namespace/project#123` | `project#123` | -| merge request | `!123` | `namespace/project!123` | `project!123` | -| snippet | `$123` | `namespace/project$123` | `project$123` | -| [epic](group/epics/index.md) | `&123` | `group1/subgroup&123` | | -| [iteration](group/iterations/index.md) | `*iteration:"iteration title"`| | | -| [vulnerability](application_security/vulnerabilities/index.md) <sup>1</sup> | `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | -| feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | -| label by ID | `~123` | `namespace/project~123` | `project~123` | -| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | -| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | -| scoped label by name | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | -| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | -| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | -| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | -| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | -| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | -| repository file references | `[README](doc/README.md)` | | | -| repository file line references | `[README](doc/README.md#L13)` | | | -| [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | -| contact | `[contact:test@example.com]` | | | - -1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222483) in GitLab 13.7. +| References | Input | Cross-project reference | Shortcut inside the same namespace | +| -------------------------------------------------------------- | ------------------------------ | --------------------------------------- | ---------------------------------- | +| Specific user | `@user_name` | | | +| Specific group | `@group_name` | | | +| Entire team | [`@all`](discussions/index.md#mentioning-all-members) | | | +| Project | `namespace/project>` | | | +| Issue | ``#123`` | `namespace/project#123` | `project#123` | +| Merge request | `!123` | `namespace/project!123` | `project!123` | +| Snippet | `$123` | `namespace/project$123` | `project$123` | +| [Epic](group/epics/index.md) | `&123` | `group1/subgroup&123` | | +| [Iteration](group/iterations/index.md) | `*iteration:"iteration title"` | | | +| [Vulnerability](application_security/vulnerabilities/index.md) | `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | +| Feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | +| Label by ID | `~123` | `namespace/project~123` | `project~123` | +| Label by name (one word) | `~bug` | `namespace/project~bug` | `project~bug` | +| Label by name (multiple words) | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | +| Label by name (scoped) | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | +| Project milestone by ID | `%123` | `namespace/project%123` | `project%123` | +| Milestone by name (one word) | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | +| Milestone by name (multiple words) | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | +| Commit (specific) | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | +| Commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | +| Repository file reference | `[README](doc/README.md)` | | | +| Repository file reference (specific line) | `[README](doc/README.md#L13)` | | | +| [Alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | +| [Contact](crm/index.md#contacts) | `[contact:test@example.com]` | | | For example, referencing an issue by using `#123` formats the output as a link to issue number 123 with text `#123`. Likewise, a link to issue number 123 is recognized and formatted with text `#123`. If you don't want `#123` to link to an issue, add a leading backslash `\#123`. -In addition to this, links to some objects are also recognized and formatted. Some examples of these are: +In addition to this, links to some objects are also recognized and formatted. +For example: -- Comments on issues: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234#note_101075757"`, which are rendered as `#1234 (comment 101075757)` -- The issues designs tab: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs"`, which are rendered as `#1234 (designs)`. -- Links to individual designs: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs/layout.png"`, which are rendered as `#1234[layout.png]`. +- Comments on issues: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234#note_101075757"`, rendered as `#1234 (comment 101075757)` +- The issues designs tab: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs"`, rendered as `#1234 (designs)`. +- Links to individual designs: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs/layout.png"`, rendered as `#1234[layout.png]`. ### Show the issue, merge request, or epic title in the reference > - Support for issues, merge requests, and epics [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15694) in GitLab 14.6. > - Support for work items [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390854) in GitLab 16.0. -To include the title in the rendered link of an issue, work item, merge request, or epic, add a plus (`+`) -at the end of the reference. For example, a reference like `#123+` is rendered as -`The issue title (#123)`. +To include the title in the rendered link of an issue, work item, merge request, or epic: + +- Add a plus (`+`) at the end of the reference. + +For example, a reference like `#123+` is rendered as `The issue title (#123)`. URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+` are also expanded. @@ -713,18 +743,23 @@ URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+` are al > - Support for issues and merge requests [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386937) in GitLab 15.10. > - Support for work items [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390854) in GitLab 16.0. -To include an extended summary in the rendered link of an issue, work item, or merge request, add a `+s` -at the end of the reference. Summary includes information about **assignees**, **milestone** -and **health status** of referenced item. +To include an extended summary in the rendered link of an issue, work item, or merge request: + +- Add a `+s` at the end of the reference. + +Summary includes information about **assignees**, **milestone** and **health status** of referenced item. For example, a reference like `#123+s` is rendered as `The issue title (#123) • First Assignee, Second Assignee+ • v15.10 • Needs attention`. URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+s` are also expanded. -To update the rendered references if the assignee, milestone, or health status changed, -edit the comment or description and save it. -For more information, see issue [420807](https://gitlab.com/gitlab-org/gitlab/-/issues/420807). +To update the rendered references if the assignee, milestone, or health status changed: + +- Edit the comment or description and save it. + +Issue [420807](https://gitlab.com/gitlab-org/gitlab/-/issues/420807) tracks improving how these +references refresh. ### Embedding Observability dashboards @@ -733,16 +768,11 @@ You can embed GitLab Observability UI dashboards descriptions and comments, for To embed an Observability dashboard URL: 1. In GitLab Observability UI, copy the URL in the address bar. +1. Paste your link in a comment or description. GitLab Flavored Markdown recognizes the URL and displays the source. -1. Paste your link wherever you want to embed your dashboard. GitLab Flavored Markdown recognizes the URL and displays the source. +## Blockquotes -## Features extended from standard Markdown - -All standard Markdown formatting should work as expected in GitLab. Some standard -functionality is extended with additional features, without affecting the standard usage. -If a functionality is extended, the new option is listed as a sub-section. - -### Blockquotes +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#blockquotes). Use a blockquote to highlight information, such as a side note. It's generated by starting the lines of the blockquote with `>`: @@ -753,7 +783,7 @@ by starting the lines of the blockquote with `>`: Quote break. -> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. +> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *use* **Markdown** in a blockquote. ``` > Blockquotes help you emulate reply text. @@ -761,14 +791,13 @@ Quote break. Quote break. -> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. +> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *use* **Markdown** in a blockquote. -#### Multiline blockquote +### Multiline blockquote -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiline-blockquote). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiline-blockquote). -GitLab Flavored Markdown extends the standard Markdown by also supporting multi-line blockquotes -fenced by `>>>`, with a blank line before and after the block: +Create multi-line blockquotes fenced by `>>>`, with a blank line before and after the block: ```markdown @@ -794,11 +823,13 @@ trigger this problem. > > you can quote that without having to manually prepend `>` to every line! -### Code spans and blocks +## Code spans and blocks -You can highlight anything that should be viewed as code and not standard text. +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#code-spans-and-blocks). -Inline code is highlighted with single backticks `` ` ``: +Highlight anything that should be viewed as code and not standard text. + +Inline code is formatted with single backticks `` ` ``: ```markdown Inline `code` has `back-ticks around` it. @@ -811,6 +842,9 @@ Inline `code` has `back-ticks around` it. To achieve a similar effect for a larger code example, you can: - Fence an entire block of code with triple backticks (```` ``` ````). + - You can use more than three backticks, as long as both the opening and closing set have the same number. + Use multiple backticks for example when you want to include [suggestions](project/merge_requests/reviews/suggestions.md#nest-code-blocks-in-suggestions) + in your code blocks, or the other way around. - Fence an entire block of code with triple tildes (`~~~`). - Indent it four or more spaces. @@ -852,10 +886,9 @@ is like using Tildes are OK too. ``` -#### Colored code and syntax highlighting +### Colored code and syntax highlighting -If this section isn't rendered correctly, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). GitLab uses the [Rouge Ruby library](https://github.com/rouge-ruby/rouge) for more colorful syntax highlighting in code blocks. For a list of supported languages visit the @@ -917,44 +950,45 @@ s = "No highlighting is shown for this line." But let's throw in a <b>tag</b>. ``` -### Emphasis +## Emphasis + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#emphasis). -In Markdown, you can emphasize text in multiple ways. You can italicize, bold, strikethrough, -and combine these emphasis styles together. -Strikethrough is not part of the core Markdown standard, but is part of GitLab Flavored Markdown. +You can emphasize text in multiple ways. Use italics, bold, strikethrough, +or combine these emphasis styles together. Examples: ```markdown -Emphasis, aka italics, with *asterisks* or _underscores_. +Emphasis, or italics, with *asterisks* or _underscores_. -Strong emphasis, aka bold, with double **asterisks** or __underscores__. +Strong emphasis, or bold, with double **asterisks** or __underscores__. Combined emphasis with **asterisks and _underscores_**. -Strikethrough uses two tildes. ~~Scratch this.~~ +Strikethrough with double tildes. ~~Scratch this.~~ ``` <!-- markdownlint-disable MD050 --> -Emphasis, aka italics, with *asterisks* or _underscores_. +Emphasis, or italics, with *asterisks* or _underscores_. -Strong emphasis, aka bold, with double **asterisks** or __underscores__. +Strong emphasis, or bold, with double **asterisks** or __underscores__. Combined emphasis with **asterisks and _underscores_**. -Strikethrough uses two tildes. ~~Scratch this.~~ +Strikethrough with double tildes. ~~Scratch this.~~ <!-- markdownlint-enable MD050 --> -#### Multiple underscores in words and mid-word emphasis +### Multiple underscores in words and mid-word emphasis -If this section isn't rendered correctly, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiple-underscores-in-words). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiple-underscores-in-words). Avoid italicizing a portion of a word, especially when you're dealing with code and names that often appear with multiple underscores. -GitLab Flavored Markdown extends the standard Markdown standard by ignoring multiple underlines in words, + +GitLab Flavored Markdown ignores multiple underlines in words, to allow better rendering of Markdown documents discussing code: ```markdown @@ -989,9 +1023,11 @@ perform*complicated*task do*this*and*do*that*and*another thing -### Footnotes +## Footnotes + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#footnotes). -Footnotes add a link to a note that are rendered at the end of a Markdown file. +Footnotes add a link to a note rendered at the end of a Markdown file. To make a footnote, you need both a reference tag and a separate line (anywhere in the file) with the note content. @@ -1026,7 +1062,9 @@ These are used to force the Vale ReferenceLinks check to skip these examples. [^footnote-42]: This text is another footnote. -### Headers +## Headings + +Create headings from 1 to 6 by using `#`. ```markdown # H1 @@ -1035,9 +1073,11 @@ These are used to force the Vale ReferenceLinks check to skip these examples. #### H4 ##### H5 ###### H6 +``` -Alternatively, for H1 and H2, an underline-ish style: +Alternatively, for H1 and H2, use an underline style: +```markdown Alt-H1 ====== @@ -1045,111 +1085,121 @@ Alt-H2 ------ ``` -#### Header IDs and links +### Heading IDs and links -GitLab Flavored Markdown extends the standard Markdown standard so that all Markdown-rendered headers automatically -get IDs, which can be linked to, except in comments. +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#heading-ids-and-links). + +All Markdown-rendered headings automatically +get IDs that can be linked to, except in comments. On hover, a link to those IDs becomes visible to make it easier to copy the link to -the header to use it somewhere else. +the heading to use it somewhere else. -The IDs are generated from the content of the header according to the following rules: +The IDs are generated from the content of the heading according to the following rules: 1. All text is converted to lowercase. 1. All non-word text (such as punctuation or HTML) is removed. 1. All spaces are converted to hyphens. 1. Two or more hyphens in a row are converted to one. -1. If a header with the same ID has already been generated, a unique +1. If a heading with the same ID has already been generated, a unique incrementing number is appended, starting at 1. Example: ```markdown -# This header has spaces in it -## This header has a :thumbsup: in it -# This header has Unicode in it: 한글 -## This header has spaces in it -### This header has spaces in it -## This header has 3.5 in it (and parentheses) +# This heading has spaces in it +## This heading has a :thumbsup: in it +# This heading has Unicode in it: 한글 +## This heading has spaces in it +### This heading has spaces in it +## This heading has 3.5 in it (and parentheses) ``` Would generate the following link IDs: -1. `this-header-has-spaces-in-it` -1. `this-header-has-a-in-it` -1. `this-header-has-unicode-in-it-한글` -1. `this-header-has-spaces-in-it-1` -1. `this-header-has-spaces-in-it-2` -1. `this-header-has-3-5-in-it-and-parentheses` +1. `this-heading-has-spaces-in-it` +1. `this-heading-has-a-in-it` +1. `this-heading-has-unicode-in-it-한글` +1. `this-heading-has-spaces-in-it-1` +1. `this-heading-has-spaces-in-it-2` +1. `this-heading-has-3-5-in-it-and-parentheses` -Emoji processing happens before the header IDs are generated. The +Emoji processing happens before the heading IDs are generated. The emoji is converted to an image, which is then removed from the ID. -### Horizontal Rule +## Horizontal rule + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#horizontal-rule). Create a horizontal rule by using three or more hyphens, asterisks, or underscores: ```markdown -Three or more hyphens, - --- -asterisks, - *** -or underscores - ___ ``` -### Images +--- -Examples: +--- + +--- + +## Images + +Embed images using inline or reference links. +To see title text, hover over the image. <!-- -The following codeblock uses HTML to skip the Vale ReferenceLinks test. -Do not change it back to a markdown codeblock. +The following examples use HTML to skip the Vale ReferenceLinks test. +Do not change it back to a markdown codeblocks. --> -<!-- markdownlint-disable proper-names --> +<!-- +DO NOT change the name of markdown_logo.png. This file is used for a test in +spec/controllers/help_controller_spec.rb. +--> -<pre class="highlight"><code>Inline-style (hover to see title text): +<!-- +The examples below use an in-line link to pass the Vale ReferenceLinks test. +Do not change to a reference style link. +--> - +Inline-style: -Reference-style (hover to see title text): +<!-- markdownlint-disable proper-names --> -![alt text1][logo] +<pre class="highlight"><code> + + -[logo]: img/markdown_logo.png "Title Text" </code></pre> -<!-- markdownlint-enable proper-names --> + -<!-- -DO NOT change the name of markdown_logo.png. This file is used for a test in -spec/controllers/help_controller_spec.rb. ---> +Reference-style: -Inline-style (hover to see title text): +<pre class="highlight"><code> - +![alt text1][logo] -Reference-style (hover to see title text): +[logo]: img/markdown_logo.png "Title Text" -<!-- -The example below uses an in-line link to pass the Vale ReferenceLinks test. -Do not change to a reference style link. ---> +</code></pre>  -#### Change the image or video dimensions +<!-- markdownlint-enable proper-names --> + +### Change the image or video dimensions > - Support for images [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28118) in GitLab 15.7. > - Support for videos [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17139) in GitLab 15.9. +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#change-the-image-or-video-dimensions). + You can control the width and height of an image or video by following the image with an attribute list. The value must an integer with a unit of either `px` (default) or `%`. @@ -1167,50 +1217,45 @@ For example You can also use the `img` HTML tag instead of Markdown and set its `height` and `width` parameters. -#### Videos +### Videos -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#videos). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#videos). Image tags that link to files with a video extension are automatically converted to a video player. The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`: -```markdown -Here's a sample video: +Here's an example video: +```markdown  ``` -Here's a sample video: -  -#### Audio +### Audio -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#audio). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#audio). Similar to videos, link tags for files with an audio extension are automatically converted to an audio player. The valid audio extensions are `.mp3`, `.oga`, `.ogg`, `.spx`, and `.wav`: -```markdown -Here's a sample audio clip: +Here's an example audio clip: +```markdown  ``` -Here's a sample audio clip: -  -### Inline HTML +## Inline HTML > Allowing `rel="license"` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20857) in GitLab 14.6. -To see the second example of Markdown rendered in HTML, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-html). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-html). You can also use raw HTML in your Markdown, and it usually works pretty well. -See the documentation for HTML::Pipeline's [SanitizationFilter](https://github.com/jch/html-pipeline/blob/v2.12.3/lib/html/pipeline/sanitization_filter.rb#L42) +See the documentation for `HTML::Pipeline`'s [SanitizationFilter](https://github.com/jch/html-pipeline/blob/v2.12.3/lib/html/pipeline/sanitization_filter.rb#L42) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` allowlist, GitLab allows `span`, `abbr`, `details` and `summary` elements. `rel="license"` is allowed on links to support the [Rel-License microformat](https://microformats.org/wiki/rel-license) and license attribution. @@ -1243,7 +1288,7 @@ are separated into their own lines: <dt>Markdown in HTML</dt> <dd>Does *not* work **very** well. HTML tags work, in most cases.</dd> - <dt>Markdown in HTML</dt> + <dt>Markdown in HTML with proper spacing</dt> <dd> Does *not* work **very** well. HTML tags work, in most cases. @@ -1261,7 +1306,7 @@ Markdown is fine in GitLab. <dt>Markdown in HTML</dt> <dd>Does *not* work **very** well. HTML tags work, in most cases.</dd> - <dt>Markdown in HTML</dt> + <dt>Markdown in HTML with proper spacing</dt> <dd> Does <em>not</em> work <b>very</b> well. HTML tags work, in most cases. @@ -1269,38 +1314,33 @@ Markdown is fine in GitLab. </dd> </dl> -#### Collapsible section +### Collapsible section -To see the second Markdown example rendered in HTML, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#details-and-summary). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#details-and-summary). Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) tags. For example, collapse a long log file so it takes up less screen space. ```html -<p> <details> -<summary>Click this to collapse/fold.</summary> +<summary>Click to expand</summary> These details <em>remain</em> <strong>hidden</strong> until expanded. <pre><code>PASTE LOGS HERE</code></pre> </details> -</p> ``` -<p> <details> -<summary>Click this to collapse/fold.</summary> +<summary>Click to expand</summary> These details <em>remain</em> <strong>hidden</strong> until expanded. <pre><code>PASTE LOGS HERE</code></pre> </details> -</p> --- @@ -1312,7 +1352,7 @@ Remember to leave a blank line before and after any Markdown sections, as shown <details> <summary> -Click this to _collapse/fold._ +Click to _expand._ </summary> @@ -1331,7 +1371,7 @@ works correctly in GitLab. --> <details> -<summary>Click this to <em>collapse/fold.</em></summary> +<summary>Click to <em>expand.</em></summary> These details <em>remain</em> <b>hidden</b> until expanded. @@ -1339,11 +1379,13 @@ These details <em>remain</em> <b>hidden</b> until expanded. </details> -### Line breaks +## Line breaks + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#line-breaks). A line break is inserted (a new paragraph starts) if the previous text is ended with two newlines, like when you press <kbd>Enter</kbd> twice in a row. If you only -use one newline (select <kbd>Enter</kbd> once), the next sentence remains part of the +use one newline (press <kbd>Enter</kbd> once), the next sentence remains part of the same paragraph. Use this approach if you want to keep long lines from wrapping, and keep them editable: @@ -1367,7 +1409,7 @@ These lines are only separated by single newlines, so they *do not break* and just follow the previous lines in the *same paragraph*. -#### Newlines +### Newlines GitLab Flavored Markdown adheres to the Markdown specification for handling [paragraphs and line breaks](https://spec.commonmark.org/current/). @@ -1382,7 +1424,7 @@ paragraph, with a blank line in between: ```markdown First paragraph. Another line in the same paragraph. -A third line in the same paragraph, but this time ending with two spaces.{space}{space} +A third line in the same paragraph, but this time ending with two spaces.<space><space> A new line directly under the first paragraph. Second paragraph. @@ -1390,7 +1432,9 @@ Another line, this time ending with a backslash.\ A new line due to the previous backslash. ``` -### Links +## Links + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#links). You can create links two ways: inline-style and reference-style. For example: @@ -1404,10 +1448,10 @@ Do not change it back to a markdown codeblock. - This line shows a [relative link to a file one directory higher](../index.md) - This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") -Using header ID anchors: +Using heading ID anchors: -- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-features-permissions) -- This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) +- This line links to [a section on a different Markdown page, using a "#" and the heading ID](permissions.md#project-features-permissions) +- This line links to [a different section on the same page, using a "#" and the heading ID](#heading-ids-and-links) Using references: @@ -1427,10 +1471,10 @@ Some text to show that the reference links can follow later. - This line shows a [relative link to a file one directory higher](../index.md) - This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") -Using header ID anchors: +Using heading ID anchors: -- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-members-permissions) -- This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) +- This line links to [a section on a different Markdown page, using a "#" and the heading ID](permissions.md#project-members-permissions) +- This line links to [a different section on the same page, using a "#" and the heading ID](#heading-ids-and-links) Using references: @@ -1451,9 +1495,11 @@ page, or a wiki page in a project file. The reason: a wiki is always in a separate Git repository in GitLab. For example, `[I'm a reference-style link](style)` points the link to `wikis/style` only when the link is inside of a wiki Markdown file. -#### URL auto-linking +### URL auto-linking + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#url-auto-linking). -GitLab Flavored Markdown auto-links almost any URL you put into your text: +Almost any URL you put into your text is auto-linked: ```markdown - https://www.google.com @@ -1469,12 +1515,15 @@ GitLab Flavored Markdown auto-links almost any URL you put into your text: - <https://www.google.com> - <https://www.google.com> - <ftp://ftp.us.debian.org/debian/> -- <smb://foo/bar/baz> -- <irc://irc.freenode.net/> +- <a href="smb://foo/bar/baz/">smb://foo/bar/baz</a> +- <a href="irc://irc.freenode.net">irc://irc.freenode.net</a> - <http://localhost:3000> <!-- vale gitlab.Spelling = YES --> -### Lists + +## Lists + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#lists). You can create ordered and unordered lists. @@ -1629,10 +1678,11 @@ For example: CommonMark ignores the blank line and renders this as one list with paragraph spacing. -### Superscripts / Subscripts +## Superscripts / Subscripts + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#superscripts-subscripts). -CommonMark and GitLab Flavored Markdown don't support the Redcarpet superscript syntax ( `x^2` ). -Use the standard HTML syntax for superscripts and subscripts: +For superscripts and subscripts, use the standard HTML syntax: ```html The formula for water is H<sub>2</sub>O @@ -1646,7 +1696,11 @@ while the equation for the theory of relativity is E = mc<sup>2</sup>. <!-- vale gitlab.Spelling = YES --> -### Keyboard HTML tag +GitLab Flavored Markdown doesn't support the Redcarpet superscript syntax ( `x^2` ). + +## Keyboard HTML tag + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#keyboard-html-tag). The `<kbd>` element is used to identify text that represents user keyboard input. Text surrounded by `<kbd>` tags is typically displayed in the browser's default monospace font. @@ -1656,27 +1710,27 @@ Press <kbd>Enter</kbd> to go to the next page. Press <kbd>Enter</kbd> to go to the next page. -### Tables +## Tables -Tables are not part of the core Markdown specification, but are part of GitLab Flavored Markdown. +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#tables-1). -#### Markdown +When creating tables: -1. The first line contains the headers, separated by "pipes" (`|`). -1. The second line separates the headers from the cells. - - The cells can contain only empty spaces, hyphens, and - (optionally) colons for horizontal alignment. - - Each cell must contain at least one hyphen, but adding more hyphens to a - cell does not change the cell's rendering. - - Any content other than hyphens, whitespace, or colons is not allowed -1. The third, and any following lines, contain the cell values. - - You **can't** have cells separated over many lines in the Markdown, they must be kept to single lines, - but they can be very long. You can also include HTML `<br>` tags to force newlines if needed. - - The cell sizes **don't** have to match each other. They are flexible, but must be separated - by pipes (`|`). - - You **can** have blank cells. -1. Column widths are calculated dynamically based on the content of the cells. -1. To use the pipe character (`|`) in the text and not as table delimiter, you must escape it with a backslash (`\|`). +- The first line contains the headers, separated by "pipes" (`|`). +- The second line separates the headers from the cells. + - The cells can contain only empty spaces, hyphens, and + (optionally) colons for horizontal alignment. + - Each cell must contain at least one hyphen, but adding more hyphens to a + cell does not change the cell's rendering. + - Any content other than hyphens, whitespace, or colons is not allowed +- The third, and any following lines, contain the cell values. + - You **can't** have cells separated over many lines in the Markdown, they must be kept to single lines, + but they can be very long. You can also include HTML `<br>` tags to force newlines if needed. + - The cell sizes **don't** have to match each other. They are flexible, but must be separated + by pipes (`|`). + - You **can** have blank cells. +- Column widths are calculated dynamically based on the content of the cells. +- To use the pipe character (`|`) in the text and not as table delimiter, you must escape it with a backslash (`\|`). Example: @@ -1694,6 +1748,10 @@ Example: | cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It eventually wraps the text when the cell is too large for the display size. | | cell 7 | | cell 9 | +### Alignment + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#alignment). + Additionally, you can choose the alignment of text in columns by adding colons (`:`) to the sides of the "dash" lines in the second row. This affects every cell in the column: @@ -1712,6 +1770,10 @@ to the sides of the "dash" lines in the second row. This affects every cell in t [In GitLab itself](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#tables), the headers are always left-aligned in Chrome and Firefox, and centered in Safari. +### Cells with multiple lines + +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#cells-with-multiple-lines). + You can use HTML formatting to adjust the rendering of tables. For example, you can use `<br>` tags to force a cell to have multiple lines: @@ -1727,45 +1789,49 @@ use `<br>` tags to force a cell to have multiple lines: | Item1 | This text is on one line | | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | -You can use HTML formatting in GitLab itself to add [task lists](#task-lists) with checkboxes, -but they do not render properly on `docs.gitlab.com`. These tasks will not save their -state when selected, like regular GitLab task lists. - -```markdown -| header 1 | header 2 | -| --- | --- | -| cell 1 | cell 2 | -| cell 3 | <ul><li> - [ ] Task one </li><li> - [ ] Task two </li></ul> | -``` - -To have fully functioning task lists in a table, create an HTML table with Markdown in the cells: - -```html -<table> -<thead> -<tr><th>header 1</th><th>header 2</th></tr> -</thead> -<tbody> -<tr> -<td>cell 1</td> -<td>cell 2</td> -</tr> -<tr> -<td>cell 3</td> -<td> +### Task lists in tables + +To add [task lists](#task-lists) with checkboxes, use HTML formatting. Using either: + +- **An HTML table with Markdown in the cells.** Tables formatted this way result in fully functioning + task lists. + + ```html + <table> + <thead> + <tr><th>header 1</th><th>header 2</th></tr> + </thead> + <tbody> + <tr> + <td>cell 1</td> + <td>cell 2</td> + </tr> + <tr> + <td>cell 3</td> + <td> + + - [ ] Task one + - [ ] Task two + + </td> + </tr> + </tbody> + </table> + ``` -- [ ] Task one -- [ ] Task two +- **A Markdown table with HTML list tags.** These tasks don't save their state when selected. + Tables formatted this way do not render properly on `docs.gitlab.com`. -</td> -</tr> -</tbody> -</table> -``` + ```markdown + | header 1 | header 2 | + | --- | --- | + | cell 1 | cell 2 | + | cell 3 | <ul><li> - [ ] Task one </li><li> - [ ] Task two </li></ul> | + ``` -##### Copy and paste from a spreadsheet +You can also [create a table in the rich text editor](rich_text_editor.md#tables) and insert a task list then. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27205) in GitLab 12.7. +### Copy and paste from a spreadsheet If you're working in spreadsheet software (for example, Microsoft Excel, Google Sheets, or Apple Numbers), GitLab creates a Markdown table when you copy and paste @@ -1779,7 +1845,7 @@ entry and paste the spreadsheet:  -#### JSON +### JSON > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86353) in GitLab 15.3. diff --git a/doc/user/okrs.md b/doc/user/okrs.md index 14e887fe297..1bed94a302b 100644 --- a/doc/user/okrs.md +++ b/doc/user/okrs.md @@ -72,7 +72,7 @@ To view an objective: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. [Filter the list of issues](project/issues/managing_issues.md#filter-the-list-of-issues) -for `Type = objective`. + for `Type = objective`. 1. Select the title of an objective from the list. ## View a key result @@ -82,7 +82,7 @@ To view a key result: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**. 1. [Filter the list of issues](project/issues/managing_issues.md#filter-the-list-of-issues) -for `Type = key_result`. + for `Type = key_result`. 1. Select the title of a key result from the list. Alternatively, you can access a key result from the **Child objectives and key results** section in @@ -220,7 +220,7 @@ Prerequisites: To promote a key result: 1. [Open the key result](#view-a-key-result). -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**).. +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Promote to objective**. Alternatively, use the `/promote_to objective` [quick action](../user/project/quick_actions.md). @@ -236,7 +236,7 @@ To copy the objective or key result reference to your clipboard: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your objective or key result to view it. -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. You can now paste the reference into another description or comment. @@ -256,7 +256,7 @@ To copy the objective's or key result's email address: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy objective email address** or **Copy key result email address**. +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy objective email address** or **Copy key result email address**. ## Close an OKR @@ -443,7 +443,7 @@ Prerequisites: To change the confidentiality of an existing OKR: 1. [Open the objective](#view-an-objective) or [key result](#view-a-key-result). -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**). +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Turn on confidentiality** or **Turn off confidentiality**. ### Who can see confidential OKRs @@ -498,6 +498,7 @@ or assignees, on the right. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/416558) in GitLab 16.5 [with a flag](../administration/feature_flags.md) named `linked_work_items`. Enabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139394) in GitLab 16.7. +> - Adding related items by entering their URLs and IDs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/427594) in GitLab 16.8. FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../administration/feature_flags.md) named `linked_work_items`. @@ -522,7 +523,7 @@ To link an item to an objective or key result: - **Relates to** - **Blocks** - **Is blocked by** -1. Enter the search text of the item. +1. Enter the search text of the item, URL, or its reference ID. 1. When you have added all the items to be linked, select **Add** below the search box. When you have finished adding all linked items, you can see diff --git a/doc/user/organization/index.md b/doc/user/organization/index.md index c4fff4178f1..ecc62b0d510 100644 --- a/doc/user/organization/index.md +++ b/doc/user/organization/index.md @@ -44,6 +44,8 @@ To view the organizations you have access to: 1. On the left sidebar, at the top, select **Create new** (**{plus}**) and **New organization**. 1. In the **Organization name** text box, enter a name for the organization. 1. In the **Organization URL** text box, enter a path for the organization. +1. In the **Organization description** text box, enter a description for the organization. Supports a [limited subset of Markdown](#supported-markdown-for-organization-description). +1. In the **Organization avatar** field, select **Upload** or drag and drop an avatar. 1. Select **Create organization**. ## Edit an organization's name @@ -51,6 +53,10 @@ To view the organizations you have access to: 1. On the left sidebar, select **Organizations** (**{organization}**) and find the organization you want to edit. 1. Select **Settings > General**. 1. In the **Organization name** text box, edit the name. +1. In the **Organization description** text box, edit the description. Supports a [limited subset of Markdown](#supported-markdown-for-organization-description). +1. In the **Organization avatar** field, if an avatar is: + - Selected, select **Remove avatar** to remove. + - Not selected, select **Upload** or drag and drop an avatar. 1. Select **Save changes**. ## Change an organization's URL @@ -72,6 +78,14 @@ To view the organizations you have access to: 1. On the left sidebar, select **Organizations** (**{organization}**) and find the organization you want to manage. 1. Select **Manage > Users**. +## Supported Markdown for Organization description + +The Organization description field supports a limited subset of [GitLab Flavored Markdown](../markdown.md), including: + +- [Emphasis](../markdown.md#emphasis) +- [Links](../markdown.md#links) +- [Superscripts / Subscripts](../markdown.md#superscripts--subscripts) + ## Related topics - [Organization developer documentation](../../development/organization/index.md) diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index db3d31d3c18..c116a43293b 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -128,6 +128,7 @@ Install a package from the package registry so you can use it as a dependency. Prerequisites: - A package in the package registry. +- The package registry is enabled in the project responsible for publishing the package. - The group ID, which is on the group's home page. - One of the following token types: - A [personal access token](../../../user/profile/personal_access_tokens.md) diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 56ea4fe74b4..72f36ca4e80 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -318,11 +318,11 @@ To search by full or partial package name, or by exact recipe, run the The scope of your search depends on your Conan remote configuration: - If you have a remote configured for your [instance](#add-a-remote-for-your-instance), your search includes -all projects you have permission to access. This includes your private projects - as well as all public projects. + all projects you have permission to access. This includes your private projects + as well as all public projects. - If you have a remote configured for a [project](#add-a-remote-for-your-project), your search includes all -packages in the target project, as long as you have permission to access it. + packages in the target project, as long as you have permission to access it. NOTE: The limit of the search results is 500 packages, and the results are sorted by the most recently published packages. diff --git a/doc/user/packages/container_registry/authenticate_with_container_registry.md b/doc/user/packages/container_registry/authenticate_with_container_registry.md index ae19a891fc9..6e1c0ded758 100644 --- a/doc/user/packages/container_registry/authenticate_with_container_registry.md +++ b/doc/user/packages/container_registry/authenticate_with_container_registry.md @@ -20,9 +20,9 @@ All of these authentication methods require the minimum scope: To authenticate, run the `docker login` command. For example: - ```shell - docker login registry.example.com -u <username> -p <token> - ``` +```shell +docker login registry.example.com -u <username> -p <token> +``` ## Use GitLab CI/CD to authenticate diff --git a/doc/user/packages/container_registry/build_and_push_images.md b/doc/user/packages/container_registry/build_and_push_images.md index 680aab42544..7187f5ef1e9 100644 --- a/doc/user/packages/container_registry/build_and_push_images.md +++ b/doc/user/packages/container_registry/build_and_push_images.md @@ -46,7 +46,7 @@ You can configure your `.gitlab-ci.yml` file to build and push container images ## Use GitLab CI/CD -You can use [GitLab CI/CD](../../../ci/yaml/index.md) to build and push container images to the +You can use [GitLab CI/CD](../../../ci/index.md) to build and push container images to the Container Registry. You can use CI/CD to test, build, and deploy your project from the container image you created. diff --git a/doc/user/packages/container_registry/delete_container_registry_images.md b/doc/user/packages/container_registry/delete_container_registry_images.md index 852c20a80f5..73ac0082058 100644 --- a/doc/user/packages/container_registry/delete_container_registry_images.md +++ b/doc/user/packages/container_registry/delete_container_registry_images.md @@ -101,7 +101,7 @@ delete_image: - ./reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG stage: clean variables: - IMAGE_TAG: $CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG + IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG REG_SHA256: ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228 REG_VERSION: 0.16.1 only: diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 614639c705f..de24f2618d5 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -191,14 +191,14 @@ To create a cleanup policy in the UI: 1. In the **Cleanup policies** section, select **Set cleanup rules**. 1. Complete the fields: - | Field | Description | - |----------------------------|-------------------------------------------------| - | **Toggle** | Turn the policy on or off. | - | **Run cleanup** | How often the policy should run. | - | **Keep the most recent** | How many tags to _always_ keep for each image. | + | Field | Description | + |----------------------------|-------------| + | **Toggle** | Turn the policy on or off. | + | **Run cleanup** | How often the policy should run. | + | **Keep the most recent** | How many tags to _always_ keep for each image. | | **Keep tags matching** | A regex pattern that determines which tags to preserve. The `latest` tag is always preserved. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | - | **Remove tags older than** | Remove only tags older than X days. | - | **Remove tags matching** | A regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | + | **Remove tags older than** | Remove only tags older than X days. | + | **Remove tags matching** | A regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | 1. Select **Save**. @@ -275,9 +275,9 @@ You can use the following application settings to prevent server resource starva For self-managed instances, those settings can be updated in the [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session): - ```ruby - ApplicationSetting.last.update(container_registry_expiration_policies_worker_capacity: 3) - ``` +```ruby +ApplicationSetting.last.update(container_registry_expiration_policies_worker_capacity: 3) +``` They are also available in the [administrator area](../../../administration/admin_area.md): @@ -292,7 +292,7 @@ You can set, update, and disable the cleanup policies using the GitLab API. Examples: - Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve -any images with the name `main`, and the policy is enabled: + any images with the name `main`, and the policy is enabled: ```shell curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" \ diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 8f702183adc..bd5311276c6 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -17,8 +17,6 @@ Supported clients: - `mvn`. Learn how to build a [Maven](../workflows/build_packages.md#maven) package. - `gradle`. Learn how to build a [Gradle](../workflows/build_packages.md#gradle) package. - `sbt`. - - `sbt` can only be used to [pull dependencies](#install-a-package). - See this [issue 408479](https://gitlab.com/gitlab-org/gitlab/-/issues/408479) for more details. ## Publish to the GitLab package registry @@ -255,9 +253,9 @@ credentials += Credentials("GitLab Packages Registry", "<host>", "<name>", "<tok In this example: - `<endpoint url>` is the [endpoint URL](#endpoint-urls). -Example: `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven`. + Example: `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven`. - `<host>` is the host present in the `<endpoint url>` without the protocol -scheme or the port. Example: `gitlab.example.com`. + scheme or the port. Example: `gitlab.example.com`. - `<name>` and `<token>` are explained in the table above. ::EndTabs @@ -289,11 +287,11 @@ For the instance-level endpoint, ensure the relevant section of your `pom.xml` i #### Endpoint URLs -| Endpoint | Endpoint URL for `pom.xml` | Additional information | -| -------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Endpoint | Endpoint URL for `pom.xml` | Additional information | +|----------|--------------------------------------------------------------------------|------------------------| | Project | `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<project_id>` with your project ID, found on your project's homepage. | -| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID, found on your group's homepage. | -| Instance | `https://gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | +| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID, found on your group's homepage. | +| Instance | `https://gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | ### Edit the configuration file for publishing @@ -454,6 +452,35 @@ gradle publish Go to your project's **Packages and registries** page and view the published packages. +:::TabTitle `sbt` + +Configure the `publishTo` setting in your `build.sbt` file: + +```scala +publishTo := Some("gitlab" at "<endpoint url>") +``` + +Ensure the credentials are referenced correctly. See the [`sbt` documentation](https://www.scala-sbt.org/1.x/docs/Publishing.html#Credentials) for more information. + +To publish a package using `sbt`: + +```shell +sbt publish +``` + +If the deploy is successful, the build success message is displayed: + +```shell +[success] Total time: 1 s, completed Jan 28, 2020 12:08:57 PM +``` + +Check the success message to ensure the package was published to the +correct location: + +```shell +[info] published my-project_2.12 to https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/my-project_2.12/0.1.1-SNAPSHOT/my-project_2.12-0.1.1-SNAPSHOT.pom +``` + ::EndTabs ## Install a package diff --git a/doc/user/packages/package_registry/dependency_proxy/index.md b/doc/user/packages/package_registry/dependency_proxy/index.md index 7b5e7a4c624..88e424ed3ac 100644 --- a/doc/user/packages/package_registry/dependency_proxy/index.md +++ b/doc/user/packages/package_registry/dependency_proxy/index.md @@ -7,12 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Dependency proxy for packages **(PREMIUM ALL BETA)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3610) in GitLab 16.6 [with a flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`. Disabled by default. -> - This feature is in [Beta](../../../../policy/experiment-beta-support.md). - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/415218) in GitLab 16.8. Feature flag `packages_dependency_proxy_maven` removed. The GitLab dependency proxy for packages is a local proxy for frequently pulled packages. It is implemented as a pull-through cache that works at the project level. diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 2c9576bf9f7..b4a0597bf60 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -48,10 +48,10 @@ You can publish Terraform modules by using the [Terraform Module Registry API](. Prerequisites: -- The package name and version [must be unique in the top-level namespace](#how-module-resolution-works). +- Unless [duplicates are allowed](#allow-duplicate-terraform-modules), the package name and version [must be unique in the top-level namespace](#how-module-resolution-works). - Your project and group names must not include a dot (`.`). For example, `source = "gitlab.example.com/my.group/project.name"`. - You must [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. -- The name of a module [must be unique in the scope of its group](#how-module-resolution-works), otherwise an +- Unless [duplicates are allowed](#allow-duplicate-terraform-modules), the name of a module [must be unique in the scope of its group](#how-module-resolution-works), otherwise an [error occurs](#troubleshooting). ```plaintext @@ -155,7 +155,23 @@ upload: ``` To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic versioning specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repository trigger the module upload job. -For other ways to control jobs in your CI/CD pipeline, refer to the [`.gitlab-ci.yml`](../../../ci/yaml/index.md) keyword reference. +For other ways to control jobs in your CI/CD pipeline, refer to the [CI/CD YAML syntax reference](../../../ci/yaml/index.md). + +### Allow duplicate Terraform modules + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368040) in GitLab 16.8. + +By default, the Terraform Module Registry enforces uniqueness for module names in the same namespace. To allow publishing duplicate module names: + +- Enable `terraform_module_duplicates_allowed` for the namespace with the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings). + +To allow duplicates with specific names: + +1. Ensure `terraform_module_duplicates_allowed` is disabled. +1. Use `terraform_module_duplicate_exception_regex` to define a regex pattern for the module names you want to allow duplicates for. + +The top-level namespace setting takes precedence over the child namespace settings. +For example, if you enable `terraform_module_duplicates_allowed` for a group, and disable it for a subgroup, duplicates are allowed for all projects in the group and its subgroups. ## Reference a Terraform module @@ -163,7 +179,9 @@ Prerequisites: - You need to [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a personal access token, it must be configured with the `read_api` scope. -Authentication tokens (Job Token or Personal Access Token) can be provided for `terraform` in your `~/.terraformrc` or `%APPDATA%/terraform.rc` file: +### From a namespace + +You can provide authentication tokens (job tokens, personal access tokens, or deploy tokens) for `terraform` in your `~/.terraformrc` or `%APPDATA%/terraform.rc` file: ```terraform credentials "gitlab.com" { @@ -183,6 +201,32 @@ module "<module>" { Where `<namespace>` is the [namespace](../../../user/namespace/index.md) of the Terraform Module Registry. +### From a project + +To reference a Terraform module using a project-level source, use the [fetching archives over HTTP](https://developer.hashicorp.com/terraform/language/modules/sources#fetching-archives-over-http) source type provided by Terraform. + +You can provide authentication tokens (job tokens, personal access tokens, or deploy tokens) for `terraform` in your `~/.netrc` file: + +```netrc +machine gitlab.com +login <USERNAME> +password <TOKEN> +``` + +Where `gitlab.com` can be replaced with the hostname of your self-managed GitLab instance, and `<USERNAME>` is your token username. + +You can refer to your Terraform module from a downstream Terraform project: + +```terraform +module "<module>" { + source = "https://gitlab.com/api/v4/projects/<project-id>/packages/terraform/modules/<module-name>/<module-system>/<module-version>" +} +``` + +If you need to reference the latest version of a module, you can omit the `<module-version>` from the source URL. To prevent future issues, you should reference a specific version if possible. + +If there are [duplicate module names](#allow-duplicate-terraform-modules) in the same namespace, referencing the module from the namespace level installs the recently published module. To reference a specific version of a duplicate module, use the [project-level](#from-a-project) source type. + ## Download a Terraform module To download a Terraform module: @@ -249,4 +293,4 @@ For examples of the Terraform Module Registry, check the projects below: ## Troubleshooting -- Publishing a module with a duplicate name results in a `{"message":"Access Denied"}` error. There's an ongoing discussion about allowing duplicate module names [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/368040). +- Publishing a module with a duplicate name results in a `{"message":"A package with the same name already exists in the namespace"}` error. diff --git a/doc/user/packages/workflows/build_packages.md b/doc/user/packages/workflows/build_packages.md index 59508b3e9e2..9d8db546566 100644 --- a/doc/user/packages/workflows/build_packages.md +++ b/doc/user/packages/workflows/build_packages.md @@ -276,6 +276,51 @@ OS: Windows 10 10.0 amd64 1. Enter a project name or press <kbd>Enter</kbd> to use the directory name as project name. +## sbt + +### Install sbt + +Install sbt to create new sbt projects. + +To install sbt for your development environment: + +1. Follow the instructions at [scala-sbt.org](https://www.scala-sbt.org/1.x/docs/Setup.html). + +1. From your terminal, verify you can use sbt: + + ```shell + sbt --version + ``` + +The output is similar to: + +```plaintext +[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r) +sbt script version: 1.9.8 +``` + +### Create a Scala project + +1. Open your terminal and create a directory to store the project. +1. From the new directory, initialize a new project: + + ```shell + sbt new scala/scala-seed.g8 + ``` + + The output is: + + ```plaintext + Minimum Scala build. + + name [My Something Project]: hello + + Template applied in ./hello + ``` + +1. Enter a project name or press <kbd>Enter</kbd> to use the directory name as project name. +1. Open the `build.sbt` file and edit it as described in the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Publishing.html) to publish your project to the package registry. + ## npm ### Install npm diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 8c15d696760..05633cac3b0 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -66,7 +66,7 @@ The following table lists project permissions available for each role: | [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/proxy-based.md#on-demand-scans) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/on-demand_scan.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | diff --git a/doc/user/product_analytics/index.md b/doc/user/product_analytics/index.md index 75e44471f92..54120ff2330 100644 --- a/doc/user/product_analytics/index.md +++ b/doc/user/product_analytics/index.md @@ -4,7 +4,7 @@ group: Product Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Product analytics **(ULTIMATE ALL EXPERIMENT)** +# Product analytics **(ULTIMATE SAAS BETA)** > - Introduced in GitLab 15.4 as an [Experiment](../../policy/experiment-beta-support.md#experiment) feature [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. > - `cube_api_proxy` revised to only reference the [Product Analytics API](../../api/product_analytics.md) in GitLab 15.6. @@ -12,11 +12,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - `product_analytics_internal_preview` replaced with `product_analytics_dashboards` in GitLab 15.11. > - Snowplow integration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398253) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `product_analytics_snowplow_support`. Disabled by default. > - Snowplow integration feature flag `product_analytics_snowplow_support` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130228) in GitLab 16.4. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_dashboards`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/414865) from GitLab self-managed to GitLab.com in 16.7. +> - Enabled in GitLab 16.7 as a [Beta](../../policy/experiment-beta-support.md#beta) feature. This page is a work in progress, and we're updating the information as we add more features. For more information, see the [group direction page](https://about.gitlab.com/direction/monitor/product-analytics/). @@ -30,7 +27,7 @@ To leave feedback about Product Analytics bugs or functionality: Product analytics uses several tools: - [**Snowplow**](https://docs.snowplow.io/docs) - A developer-first engine for collecting behavioral data, and passing it through to ClickHouse. -- [**ClickHouse**](https://clickhouse.com/docs) - A database suited to store, query, and retrieve analytical data. +- [**ClickHouse**](../../integration/clickhouse.md) - A database suited to store, query, and retrieve analytical data. - [**Cube**](https://cube.dev/docs/) - A universal semantic layer that provides an API to run queries against the data stored in ClickHouse. The following diagram illustrates the product analytics flow: @@ -65,22 +62,23 @@ flowchart TB > - `cube_api_proxy` removed and replaced with `product_analytics_internal_preview` in GitLab 15.10. > - `product_analytics_internal_preview` replaced with `product_analytics_dashboards` in GitLab 15.11. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flags](../../administration/feature_flags.md) named `product_analytics_dashboards`, `product_analytics_admin_settings`, and `combined_analytics_dashboards`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. - -To track events in your project applications on a self-managed instance, -you must enable and configure product analytics. +To track events in your project's applications on GitLab.com, +you must enable and configure Product Analytics. Prerequisites: -- You must be an administrator of a self-managed GitLab instance. +- You must be an Owner of the group you wish to enable Product Analytics for. + +### Group-level settings -1. On the left sidebar, at the bottom, select **Admin Area**. +NOTE: +These group-level settings are available for top-level groups and cascade to all projects that belong to the group. + +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. -1. Expand the **Analytics** tab and find the **Product analytics** section. -1. Select **Enable product analytics** and enter the configuration values. +1. Expand the **Permissions and group features** section. +1. Check **Use Experiment and Beta features** checkbox. +1. Check **Use product analytics** checkbox. 1. Select **Save changes**. ### Project-level settings @@ -90,7 +88,6 @@ have a different configured product analytics instance for your project. Prerequisites: -- Product analytics must be enabled at the instance-level. - You must have at least the Maintainer role for the project or group the project belongs to. - The project must be in a group namespace. @@ -109,19 +106,7 @@ To onboard a project: 1. Select **Analyze > Analytics dashboards**. 1. Under **Product analytics**, select **Set up**. 1. Select **Set up product analytics**. -Your instance is being created, and the project onboarded. - -### Onboard an internal project - -GitLab team members can enable Product Analytics on their internal projects on GitLab.com (Ultimate) during the experiment phase. - -1. Send a message to the Product Analytics team (`#g_analyze_product_analytics`) informing them of the repository to be enabled. -1. Using ChatOps, enable both the `product_analytics_dashboards` and `combined_analytics_dashboards`: - - ```plaintext - /chatops run feature set product_analytics_dashboards true --project=FULLPATH_TO_PROJECT - /chatops run feature set combined_analytics_dashboards true --project=FULLPATH_TO_PROJECT - ``` + Your instance is being created, and the project onboarded. ## Instrument your application @@ -138,11 +123,6 @@ To instrument code to collect data, use one or more of the existing SDKs: > - Introduced in GitLab 15.5 behind the [feature flag](../../administration/feature_flags.md) named `product_analytics_internal_preview`. Disabled by default. > - `product_analytics_internal_preview` replaced with `product_analytics_dashboards` in GitLab 15.11. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_dashboards`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. - Product analytics dashboards are a subset of dashboards under [Analytics dashboards](../analytics/analytics_dashboards.md). Specifically product analytics dashboards and visualizations make use of the `cube_analytics` data type. @@ -150,6 +130,8 @@ The `cube_analytics` data type connects to the Cube instance defined when [produ All filters and queries are sent to the Cube instance and the returned data is processed by the product analytics data source to be rendered by the appropriate visualizations. +Data table visualizations from `cube_analytics` have an additional configuration option for rendering `links` (array of objects, each with `text` and `href` properties to specify the dimensions to be used in links. If `href` contains multiple dimensions, values are joined into a single URL). See [example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/analytics_visualization.json?ref_type=heads#L112)). + ### Filling missing data - Introduced in GitLab 16.3 behind the [feature flag](../../administration/feature_flags.md) named `product_analytics_dashboards`. Disabled by default. diff --git a/doc/user/product_analytics/instrumentation/browser_sdk.md b/doc/user/product_analytics/instrumentation/browser_sdk.md index 6bc9a9ef234..b9cfbc5b2df 100644 --- a/doc/user/product_analytics/instrumentation/browser_sdk.md +++ b/doc/user/product_analytics/instrumentation/browser_sdk.md @@ -92,7 +92,7 @@ interface GitLabClientSDKOptions { ### Plugins - `Client Hints`: An alternative to tracking the User Agent, which is particularly useful in browsers that are freezing the User Agent string. -Enabling this plugin will automatically capture the following context: + Enabling this plugin will automatically capture the following context: For example, [iglu:org.ietf/http_client_hints/jsonschema/1-0-0](https://github.com/snowplow/iglu-central/blob/master/schemas/org.ietf/http_client_hints/jsonschema/1-0-0) @@ -163,12 +163,12 @@ glClient.page(eventAttributes); The `eventAttributes` object supports the following optional properties: -| Property | Type | Description | -| :--------------- | :-------------------------- | :---------------------------------------------------------------------------- | -| `title` | `String` | Override the default page title. | -| `contextCallback` | `Function` | A callback that fires on the page view. | -| `context` | `Object` | Add context (additional information) on the page view. | -| `timestamp` | `timestamp` | Set the true timestamp or overwrite the device-sent timestamp on an event. | +| Property | Type | Description | +|:------------------|:------------|:------------| +| `title` | `String` | Override the default page title. | +| `contextCallback` | `Function` | A callback that fires on the page view. | +| `context` | `Object` | Add context (additional information) on the page view. | +| `timestamp` | `timestamp` | Set the true timestamp or overwrite the device-sent timestamp on an event. | ### `track` diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 6c78152fa70..4611b3a6a40 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -1,6 +1,7 @@ --- stage: Govern group: Authentication +description: Passwords, user moderation, broadcast messages. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 81e2f3d7a55..b903e422a59 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -425,7 +425,7 @@ a session if the browser is closed or the existing session expires. ## Related topics - [Create users](account/create_accounts.md) -- [Sign in to your GitLab account](../../topics/authentication/index.md) +- [Sign in to your GitLab account](../../administration/auth/index.md) - [Change your password](user_passwords.md) - Receive emails for: - [Sign-ins from unknown IP addresses or devices](notifications.md#notifications-for-unknown-sign-ins) diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 9d54381ef87..dec42e74a58 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -123,8 +123,9 @@ A personal access token can perform actions based on the assigned scopes. | `sudo` | Grants permission to perform API actions as any user in the system, when authenticated as an administrator. | | `admin_mode` | Grants permission to perform API actions as an administrator, when Admin Mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107875) in GitLab 15.8.) | | `create_runner` | Grants permission to create runners. | -| `ai_features` | Grants permission to perform API actions for GitLab Duo. This scope is designed to work with the GitLab Duo Plugin for JetBrains. For all other extensions, see scope requirements. | +| `ai_features` | Grants permission to perform API actions for GitLab Duo. This scope is designed to work with the GitLab Duo Plugin for JetBrains. For all other extensions, see scope requirements. | | `k8s_proxy` | Grants permission to perform Kubernetes API calls using the agent for Kubernetes. | +| `read_service_ping`| Grant access to download Service Ping payload via API when authenticated as an admin use. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107875) in GitLab 16.8. | WARNING: If you enabled [external authorization](../../administration/settings/external_authorization.md), personal access tokens cannot access container or package registries. If you use personal access tokens to access these registries, this measure breaks this use of these tokens. Disable external authorization to use personal access tokens with container or package registries. diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 6cc5f7c5039..de8ab4b25e9 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -321,20 +321,14 @@ To access your **Followers** and **Following** tabs: ## Enable Code Suggestions **(FREE SAAS)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../policy/experiment-beta-support.md#beta). -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. Available to a percentage of users. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. UI user setting removed. -A percentage of users can use Code Suggestions without any additional configuration. +If Code Suggestions is +[enabled for your top-level group](../group/manage.md#enable-code-suggestions-for-a-group), +you can use Code Suggestions after you +[configure a supported IDE extension](../project/repository/code_suggestions/index.md#supported-editor-extensions). -If the following options are available to you, it means you are **not** part of the percentage of users -and you must manually enable Code Suggestions for your account: - -1. On the left sidebar, select your avatar. -1. Select **Preferences**. -1. Select the **Enable Code Suggestions** checkbox. -1. Select **Save changes**. - -NOTE: -If Code Suggestions are disabled [for any groups that you belong to](../../user/group/manage.md#enable-code-suggestions-for-a-group), then you cannot enable them for yourself. (Your setting has no effect.) +If Code Suggestions are disabled [for all the groups that you belong to](../../user/group/manage.md#enable-code-suggestions-for-a-group), then you cannot enable them for yourself. ## Integrate your GitLab instance with third-party services diff --git a/doc/user/project/changelogs.md b/doc/user/project/changelogs.md index a15bd39f1b7..df6df1653ac 100644 --- a/doc/user/project/changelogs.md +++ b/doc/user/project/changelogs.md @@ -15,7 +15,7 @@ commit author. Changelog formats [can be customized](#customize-the-changelog-ou Each section in the default changelog has a title containing the version number and release date, like this: -````markdown +```markdown ## 1.0.0 (2021-01-05) ### Features (4 changes) @@ -24,7 +24,7 @@ number and release date, like this: - [Feature 2](gitlab-org/gitlab@456abc) ([merge request](gitlab-org/gitlab!456)) - [Feature 3](gitlab-org/gitlab@234abc) by @steve - [Feature 4](gitlab-org/gitlab@456) -```` +``` The date format for sections can be customized, but the rest of the title cannot. When adding new sections, GitLab parses these titles to determine where to place @@ -121,11 +121,11 @@ these variables: `### Features`, `### Bug fixes`, and `### Performance improvements`: ```yaml - --- - categories: - feature: Features - bug: Bug fixes - performance: Performance improvements + --- + categories: + feature: Features + bug: Bug fixes + performance: Performance improvements ``` ### Custom templates diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 161a698a48c..ea4c345a592 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -25,7 +25,7 @@ See the prerequisites below to add existing clusters to GitLab. To add any cluster to GitLab, you need: - Either a GitLab.com account or an account for a self-managed installation -running GitLab 12.5 or later. + running GitLab 12.5 or later. - The Maintainer role for group-level and project-level clusters. - Access to the Admin Area for instance-level clusters. - A Kubernetes cluster. @@ -48,7 +48,7 @@ To add an existing **EKS** cluster, you need: - An Amazon EKS cluster with worker nodes properly configured. - `kubectl` [installed and configured](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#get-started-kubectl) -for access to the EKS cluster. + for access to the EKS cluster. - Ensure the token of the account has administrator privileges for the cluster. ### GKE clusters @@ -56,8 +56,8 @@ for access to the EKS cluster. To add an existing **GKE** cluster, you need: - The `container.clusterRoleBindings.create` permission to create a cluster -role binding. You can follow the [Google Cloud documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) -to grant access. + role binding. You can follow the [Google Cloud documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) + to grant access. ## How to add an existing cluster diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 940561d70d0..7d18ef0d1e4 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -39,7 +39,7 @@ When removing a cluster integration, you have two options: - **Remove integration**: remove only the Kubernetes integration. - **Remove integration and resources**: remove the cluster integration and -all GitLab cluster-related resources such as namespaces, roles, and bindings. + all GitLab cluster-related resources such as namespaces, roles, and bindings. To remove the Kubernetes cluster integration: diff --git a/doc/user/project/clusters/multiple_kubernetes_clusters.md b/doc/user/project/clusters/multiple_kubernetes_clusters.md index 4fb6ecb1336..9e96438393e 100644 --- a/doc/user/project/clusters/multiple_kubernetes_clusters.md +++ b/doc/user/project/clusters/multiple_kubernetes_clusters.md @@ -40,8 +40,8 @@ For example, let's say the following Kubernetes clusters exist in a project: | Development | `*` | | Production | `production` | -And the following environments are set in -[`.gitlab-ci.yml`](../../../ci/yaml/index.md): +And the following environments are set in the +[`.gitlab-ci.yml` file](../../../ci/index.md#the-gitlab-ciyml-file): ```yaml stages: diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 87467cbc56c..b861fe9d154 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -480,6 +480,23 @@ repository to be imported manually. Administrators can manually import the repos project.create_import_state if project.import_state.blank? # Set state to start project.import_state.force_start + + # Optional: If your import had certain optional stages selected or a timeout strategy + # set, you can reset them here. Below is an example. + # The params follow the format documented in the API: + # https://docs.gitlab.com/ee/api/import.html#import-repository-from-github + Gitlab::GithubImport::Settings + .new(project) + .write( + timeout_strategy: "optimistic", + optional_stages: { + single_endpoint_issue_events_import: true, + single_endpoint_notes_import: true, + attachments_import: true, + collaborators_import: true + } + ) + # Trigger import from second step Gitlab::GithubImport::Stage::ImportRepositoryWorker.perform_async(project.id) ``` diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index e604d9d871b..135b51bf81a 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The GitLab.com importer was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108502) in GitLab 15.8 and removed in GitLab 16.0. To import GitLab projects from GitLab.com to a self-managed GitLab instance use -[migrating groups and projects by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +[migrating groups and projects by direct transfer](../../group/import/index.md). ## Related topics diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 681174400a2..8c9ba408799 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -4,34 +4,61 @@ group: Import and Integrate info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Import and migrate projects **(FREE ALL)** +# Import and migrate groups and projects **(FREE ALL)** -If you want to bring existing projects to GitLab or copy GitLab projects to a different location, you can: +To bring existing projects to GitLab, or copy GitLab groups and projects to a different location, you can: -- Import projects from external systems using one of the [available importers](#available-project-importers). -- Migrate GitLab projects: - - Between two GitLab self-managed instances. - - Between a self-managed instance and GitLab.com in both directions. - - In the same GitLab instance. +- Migrate GitLab groups and projects by using direct transfer. +- Import from supported import sources. +- Import from other import sources. -For any type of source and target, you can migrate GitLab projects: +## Migrate from GitLab to GitLab by using direct transfer -- When [migrating groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended), - which allows you to migrate all projects in a group simultaneously. Migrating projects by direct transfer is in - [Beta](../../../policy/experiment-beta-support.md#beta). The feature is not ready for production use. -- Using [file exports](../settings/import_export.md). With this method you can migrate projects one by one. No network - connection between instances is required. +The best way to migrate GitLab groups and projects between GitLab instances, or in the same GitLab instance, is +[by using direct transfer](../../group/import/index.md). -If you only need to migrate Git repositories, you can [import each project by URL](repo_by_url.md). However, you can't -import issues and merge requests this way. To retain metadata like issues and merge requests, either: +You can also migrate GitLab projects by using a GitLab file export, which is a supported import source. -- [Migrate projects with groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). - This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). It is not ready for production use. -- Use [file exports](../settings/import_export.md) to import projects. +## Supported import sources -Keep in mind the limitations of [migrating using file exports](../settings/import_export.md#items-that-are-exported). -When migrating from self-managed to GitLab.com, user associations (such as comment author) -are changed to the user who is importing the projects. +> All importers default to disabled for GitLab self-managed installations. This change was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118970) in GitLab 16.0. + +The import sources that are available to you by default depend on which GitLab you use: + +- GitLab.com: all available import sources are [enabled by default](../../gitlab_com/index.md#default-import-sources). +- GitLab self-managed: no import sources are enabled by default and must be + [enabled](../../../administration/settings/import_and_export_settings.md#configure-allowed-import-sources). + +GitLab can import projects from these supported import sources. + +| Import source | Description | +|:----------------------------------------------|:------------| +| [Bitbucket Cloud](bitbucket.md) | Using [Bitbucket.org as an OmniAuth provider](../../../integration/bitbucket.md), import Bitbucket repositories. | +| [Bitbucket Server](bitbucket_server.md) | Import repositories from Bitbucket Server (also known as Stash). | +| [FogBugz](fogbugz.md) | Import FogBuz projects. | +| [Gitea](gitea.md) | Import Gitea projects. | +| [GitHub](github.md) | Import from either GitHub.com or GitHub Enterprise. | +| [GitLab export](../settings/import_export.md) | Migrate projects one by one by using a GitLab export file. | +| [Manifest file](manifest.md) | Upload a manifest file. | +| [Repository by URL](repo_by_url.md) | Provide a Git repository URL to create a new project from. | + +## Other import sources + +You can also read information on importing from these other import sources: + +- [ClearCase](clearcase.md) +- [Concurrent Versions System (CVS)](cvs.md) +- [Jira (issues only)](jira.md) +- [Perforce Helix](perforce.md) +- [Team Foundation Version Control (TFVC)](tfvc.md) + +### Import repositories from Subversion + +GitLab can not automatically migrate Subversion repositories to Git. Converting Subversion repositories to Git can be +difficult, but several tools exist including: + +- [`git svn`](https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git), for very small and basic repositories. +- [`reposurgeon`](http://www.catb.org/~esr/reposurgeon/repository-editing.html), for larger and more complex repositories. ## Security @@ -49,35 +76,6 @@ GitLab self-managed administrators can reduce their attack surface by disabling In GitLab 16.1 and earlier, you should **not** use direct transfer with [scheduled scan execution policies](../../../user/application_security/policies/scan-execution-policies.md). -## Available project importers - -You can import projects from: - -- [Bitbucket Cloud](bitbucket.md) -- [Bitbucket Server (also known as Stash)](bitbucket_server.md) -- [ClearCase](clearcase.md) -- [CVS](cvs.md) -- [FogBugz](fogbugz.md) -- [GitHub.com or GitHub Enterprise](github.md) -- [Gitea](gitea.md) -- [Perforce](perforce.md) -- [TFVC](tfvc.md) -- [Repository by URL](repo_by_url.md) -- [Uploading a manifest file (AOSP)](manifest.md) -- [Jira (issues only)](jira.md) - -You can also import any Git repository through HTTP from the **New Project** page. If the repository -is too large, the import can timeout. - -You can then [connect your external repository to get CI/CD benefits](../../../ci/ci_cd_for_external_repos/index.md). - -## Import from Subversion - -GitLab can not automatically migrate Subversion repositories to Git. Converting Subversion repositories to Git can be difficult, but several tools exist including: - -- [`git svn`](https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git), for very small and basic repositories. -- [`reposurgeon`](http://www.catb.org/~esr/reposurgeon/repository-editing.html), for larger and more complex repositories. - ## Migrate using the API To migrate all data from self-managed to GitLab.com, you can leverage the [API](../../../api/rest/index.md). diff --git a/doc/user/project/import/repo_by_url.md b/doc/user/project/import/repo_by_url.md index 5d67d10582d..3c5a40b8d27 100644 --- a/doc/user/project/import/repo_by_url.md +++ b/doc/user/project/import/repo_by_url.md @@ -6,7 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Import project from repository by URL **(FREE ALL)** -You can import your existing repositories by providing the Git URL. +You can import your existing repositories by providing the Git URL. You can't import GitLab issues and merge requests +this way. Other methods provide more complete import methods. + +If the repository is too large, the import can timeout. ## Prerequisites diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 8a91a0c4621..b7addb5131f 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -4,27 +4,28 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Insights for projects **(ULTIMATE ALL)** +# Insights **(ULTIMATE ALL)** -Configure project insights to explore data such as: +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in GitLab 12.0. + +Configure insights for your projects and groups to explore data such as: - Issues created and closed during a specified period. - Average time for merge requests to be merged. - Triage hygiene. -Insights are also available for [groups](../../group/insights/index.md). +You can also create custom Insights reports that are relevant for your group. ## View project insights Prerequisites: -- You must have: - - Access to a project to view information about its merge requests and issues. - - Permission to view confidential merge requests and issues in the project. +- For project insights, you must have access to the project and permission to view information about its merge requests and issues. +- For group insights, you must have permission to view the group. -To view project insights: +To view insights for a project or group: -1. On the left sidebar, select **Search or go to** and find your project. +1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Analyze > Insights**. 1. To view a report, select the **Select report** dropdown list. @@ -35,23 +36,61 @@ You can direct users to a specific report in Insights by using the deep-linked U To create a deep link, append the report key to the end of the Insights report URL. For example, a GitLab report with the key `bugsCharts` has the deep link URL `https://gitlab.com/gitlab-org/gitlab/insights/#/bugsCharts`. +## Interact with Insights charts + +You can interact with the insights charts to view details about your group's activity. + +### Display different reports + +To display one of the available reports on the insights page, from the **Select report** dropdown list, +select the report you want to display. + +### View bar chart annotations + +To view annotations, hover over each bar in the chart. + +### Zoom in on chart + +Insights display data from the last 90 days. You can zoom in to display data only from a subset of the 90-day range. + +To do this, select the pause icons (**{status-paused}**) and slide them along the horizontal axis: + +- To change the start date, slide the left pause icon to the left or right. +- To change the end date, slide the right pause icon to the left or right. + +### Exclude dimensions from charts + +By default, insights display all available dimensions on the chart. + +To exclude a dimension, from the legend below the chart, select the name of the dimension. + +### Drill down on charts + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/372215/) in GitLab 16.7. + +You can drill down into the data of the **Bugs created per month by priority** and **Bugs created per month by severity** charts from the [default configuration file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml). + +To view a drill-down report of the data for a specific priority or severity in a month: + +- On the chart, select the bar stack you want to drill down on. + ## Configure project insights Prerequisites: - Depending on your project configuration, you must have at least the Developer role. -Project insights are configured with the [`.gitlab/insights.yml`](#insights-configuration-file) file in the project. If a project doesn't have a configuration file, it uses the [group configuration](../../group/insights/index.md#configure-group-insights). +Project insights are configured with the [`.gitlab/insights.yml`](#insights-configuration-file) file in the project. If a project doesn't have a configuration file, it uses the [group configuration](#configure-group-insights). The `.gitlab/insights.yml` file is a YAML file where you define: - The structure and order of charts in a report. - The style of charts displayed in the report of your project or group. -To configure project insights, either: +To configure project insights, create a file `.gitlab/insights.yml` either: -- Create a `.gitlab/insights.yml` file locally in the root directory of your project, and push your changes. -- Create a `.gitlab/insights.yml` file in the UI: +- Locally, in the root directory of your project, and push your changes. +- From the UI: 1. On the left sidebar, select **Search or go to** and find your project. 1. Above the file list, select the branch you want to commit to, select the plus icon, then select **New file**. 1. In the **File name** text box, enter `.gitlab/insights.yml`. @@ -59,7 +98,21 @@ To configure project insights, either: 1. Select **Commit changes**. After you create the configuration file, you can also -[use it for the project's group](../../group/insights/index.md#configure-group-insights). +use it for the project's group. + +## Configure group insights + +GitLab reads insights from the +[default configuration file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml). + +To configure group insights: + +1. In a project that belongs to your group, [create a `.gitlab/insights.yml` file](#configure-project-insights). +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > General**. +1. Expand **Analytics** and find the **Insights** section. +1. Select the project that contains your `.gitlab/insights.yml` configuration file. +1. Select **Save changes**. ## Insights configuration file @@ -403,7 +456,7 @@ Use `query.environment_tiers` to define an array of environments to include the Use `projects` to limit where issuables are queried from: -- If `.gitlab/insights.yml` is used for a [group's insights](../../group/insights/index.md#configure-group-insights), use `projects` to define the projects from which to query issuables. By default, all projects under the group are used. +- If `.gitlab/insights.yml` is used for a group's insights, use `projects` to define the projects from which to query issuables. By default, all projects under the group are used. - If `.gitlab/insights.yml` is used for a project's insights, specifying other projects does not yield results. By default, the project is used. #### `projects.only` diff --git a/doc/user/project/integrations/apple_app_store.md b/doc/user/project/integrations/apple_app_store.md index 3031ae42e4d..18022fbaeb8 100644 --- a/doc/user/project/integrations/apple_app_store.md +++ b/doc/user/project/integrations/apple_app_store.md @@ -16,19 +16,20 @@ The feature is still in development, but you can: - [Report a bug](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/feedback/-/issues/new?issuable_template=report_bug). - [Share feedback](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/feedback/-/issues/new?issuable_template=general_feedback). -With the Apple App Store Connect integration, you can configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com) to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. +Use the Apple App Store Connect integration to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com). +With this integration, you can build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. The Apple App Store Connect integration works out of the box with [fastlane](https://fastlane.tools/). You can also use this integration with other build tools. -## Prerequisites +## Enable the integration in GitLab -An Apple ID enrolled in the [Apple Developer Program](https://developer.apple.com/programs/enroll/) is required to enable this integration. +Prerequisites: -## Configure GitLab +- You must have an Apple ID enrolled in the [Apple Developer Program](https://developer.apple.com/programs/enroll/). +- You must [generate a new private key](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api) for your project in the Apple App Store Connect portal. -GitLab supports enabling the Apple App Store Connect integration at the project level. Complete these steps in GitLab: +To enable the Apple App Store Connect integration in GitLab: -1. In the Apple App Store Connect portal, generate a new private key for your project by following [these instructions](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api). 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Integrations**. 1. Select **Apple App Store Connect**. @@ -38,7 +39,6 @@ GitLab supports enabling the Apple App Store Connect integration at the project - **Key ID**: The key ID of the generated private key. - **Private key**: The generated private key. You can download this key only once. - **Protected branches and tags only**: Enable to set variables on protected branches and tags only. - 1. Select **Save changes**. After you enable the integration: diff --git a/doc/user/project/integrations/gitlab_slack_app_troubleshooting.md b/doc/user/project/integrations/gitlab_slack_app_troubleshooting.md index 363e7c2c364..3a1bc746d5b 100644 --- a/doc/user/project/integrations/gitlab_slack_app_troubleshooting.md +++ b/doc/user/project/integrations/gitlab_slack_app_troubleshooting.md @@ -10,13 +10,13 @@ When configuring the GitLab for Slack app on GitLab.com, you might encounter the For self-managed GitLab, see [GitLab for Slack app administration](../../../administration/settings/slack_app.md#troubleshooting). -## The app does not appear in the list of integrations +## App does not appear in the list of integrations The GitLab for Slack app might not appear in the list of integrations. To have the GitLab for Slack app on your self-managed instance, an administrator must [enable the integration](../../../administration/settings/slack_app.md). On GitLab.com, the GitLab for Slack app is available by default. The GitLab for Slack app is enabled at the project level only. Support for the app at the group and instance levels is proposed in [issue 391526](https://gitlab.com/gitlab-org/gitlab/-/issues/391526). -## Project or alias not found +## `Project or alias not found` Some Slack commands must have a project full path or alias and fail with the following error if the project cannot be found: @@ -31,18 +31,19 @@ To resolve this issue, ensure: - If using a [project alias](gitlab_slack_application.md#create-a-project-alias-for-slash-commands), the alias is correct. - The GitLab for Slack app is [enabled for the project](gitlab_slack_application.md#from-project-integration-settings). -## Slash commands return an error in Slack +## Slash commands return `dispatch_failed` in Slack Slash commands might return `/gitlab failed with the error "dispatch_failed"` in Slack. + To resolve this issue, ensure an administrator has properly configured the [GitLab for Slack app settings](../../../administration/settings/slack_app.md) on your self-managed instance. -## Notifications are not received to a channel +## Notifications not received to a channel If you're not receiving notifications to a Slack channel, ensure: - The channel name you configured is correct. - If the channel is private, you've [added the GitLab for Slack app to the channel](gitlab_slack_application.md#receive-notifications-to-a-private-channel). -## The App Home does not display properly +## App Home does not display properly If the [App Home](https://api.slack.com/start/overview#app_home) does not display properly, ensure your [app is up to date](gitlab_slack_application.md#update-the-gitlab-for-slack-app). diff --git a/doc/user/project/integrations/google_play.md b/doc/user/project/integrations/google_play.md index 70a1fe5b054..cfe48d11bcb 100644 --- a/doc/user/project/integrations/google_play.md +++ b/doc/user/project/integrations/google_play.md @@ -34,7 +34,7 @@ To enable the Google Play integration in GitLab: 1. Select **Google Play**. 1. In **Enable integration**, select the **Active** checkbox. 1. In **Package name**, enter the package name of the app (for example, `com.gitlab.app_name`). -1. Optional. Under **Protected branches and tags only**, select the **Only set variables on protected branches and tags** checkbox. +1. Optional. Under **Protected branches and tags only**, select the **Set variables on protected branches and tags only** checkbox. 1. In **Service account key (.JSON)**, drag or upload your key file. 1. Optional. Select **Test settings**. 1. Select **Save changes**. diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 0a82a94d998..91c8e9ce2f1 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -19,8 +19,8 @@ To use the Mattermost integration you must create an incoming webhook integratio in Mattermost: 1. Sign in to your Mattermost instance. -1. [Enable incoming webhooks](https://docs.mattermost.com/developer/webhooks-incoming.html#enabling-incoming-webhooks). -1. [Add an incoming webhook](https://docs.mattermost.com/developer/webhooks-incoming.html#creating-integrations-using-incoming-webhooks). +1. [Enable incoming webhooks](https://docs.mattermost.com/configure/integrations-configuration-settings.html#enable-incoming-webhooks). +1. [Add an incoming webhook](https://developers.mattermost.com/integrate/webhooks/incoming/#create-an-incoming-webhook). 1. Choose a display name, description and channel, those can be overridden on GitLab. 1. Save it and copy the **Webhook URL** because we need this later for GitLab. diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index 9b92fa35e24..fee6de2af6d 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/372411) in GitLab 15.9 -and is planned for removal in 17.0. Use the [GitLab for Slack app](gitlab_slack_application.md) instead. +and is planned for removal in 18.0. Use the [GitLab for Slack app](gitlab_slack_application.md) instead. This change is a breaking change. The Slack notifications integration enables your GitLab project to send events diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 33da78191c0..ccf416fc8c6 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -1,6 +1,7 @@ --- stage: Manage group: Import and Integrate +description: Custom HTTP callbacks, used to send events. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index e6fd302e4f0..39353480908 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -101,7 +101,7 @@ For examples of using issue boards along with [epics](../group/epics/index.md), - [How to use GitLab for Agile portfolio planning and project management](https://about.gitlab.com/blog/2020/11/11/gitlab-for-agile-portfolio-planning-project-management/) blog post (November 2020) - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -[Cross-project Agile work management with GitLab](https://www.youtube.com/watch?v=5J0bonGoECs) (15 min, July 2020) + [Cross-project Agile work management with GitLab](https://www.youtube.com/watch?v=5J0bonGoECs) (15 min, July 2020) ### Use cases for a single issue board diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 6f2d0083ae8..b80db3887bf 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -355,7 +355,7 @@ To delete an issue: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. -1. On the top right corner, select **Issue actions** (**{ellipsis_v}**). +1. In the upper-right corner, select **Issue actions** (**{ellipsis_v}**). 1. Select **Delete issue**. Alternatively: @@ -367,23 +367,44 @@ Alternatively: ## Promote an issue to an epic **(PREMIUM ALL)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab 11.6. -> - Moved from GitLab Ultimate to GitLab Premium in 12.8. -> - Promoting issues to epics via the UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233974) in GitLab 13.6. - You can promote an issue to an [epic](../../group/epics/index.md) in the immediate parent group. +NOTE: +Promoting a confidential issue to an epic makes all information +related to the issue public, as epics are public to group members. + +When an issue is promoted to an epic: + +- If the issue was confidential, an additional warning is displayed first. +- An epic is created in the same group as the project of the issue. +- Subscribers of the issue are notified that the epic was created. + +The following issue metadata is copied to the epic: + +- Title, description, activity, and comment threads. +- Upvotes and downvotes. +- Participants. +- Group labels that the issue had. +- Parent epic. + +Prerequisites: + +- The project to which the issue belongs must be in a group. +- You must have at least the Reporter role the project's immediate parent group. +- You must either: + - Have at least the Reporter role for the project. + - Be the author of the issue. + - Be assigned to the issue. + To promote an issue to an epic: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. -1. On the top right corner, select **Issue actions** (**{ellipsis_v}**). +1. In the upper-right corner, select **Issue actions** (**{ellipsis_v}**). 1. Select **Promote to epic**. Alternatively, you can use the `/promote` [quick action](../quick_actions.md#issues-merge-requests-and-epics). -Read more about [promoting an issues to epics](../../group/epics/manage_epics.md#promote-an-issue-to-an-epic). - ## Promote an issue to an incident > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296787) in GitLab 14.5. diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 2cc38e6a31c..f064d867e0f 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -14,7 +14,7 @@ you're interested in. Labels are a key part of [issue boards](issue_board.md). With labels you can: - Categorize [epics](../group/epics/index.md), issues, and merge requests using colors and descriptive titles like -`bug`, `feature request`, or `docs`. + `bug`, `feature request`, or `docs`. - Dynamically filter and manage [epics](../group/epics/index.md), issues, and merge requests. - Search lists of issues, merge requests, and epics, as well as issue boards. diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 92aaee1ae54..66258c3873e 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -59,6 +59,7 @@ Prerequisites: - You must have the Owner or Maintainer role. - [Group membership lock](../../group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) must be disabled. +- If [sign-up is disabled](../../../administration/settings/sign_up_restrictions.md#disable-new-sign-ups), an administrator must add the user by email first. To add a user to a project: @@ -156,6 +157,7 @@ To add a group to a project: The invited group is displayed on the **Groups** tab. Private groups are masked from unauthorized users. +With the feature flag `allow_members_to_see_invited_groups_in_access_dropdowns` enabled, private groups are displayed in project settings for protected branches, protected tags, and protected environments. The members of the invited group are not displayed on the **Members** tab. The **Members** tab shows: diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 3881220ec7a..bf8a7468199 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -16,7 +16,7 @@ For a project that was created by `Group 1`: - The members of `Group 1` have access to the project. - The owner of `Group 1` can invite `Group 2` to the project. -This way, members of both `Group 1` and `Group 2` have access to the shared project. + This way, members of both `Group 1` and `Group 2` have access to the shared project. ## Prerequisites @@ -31,7 +31,7 @@ In addition: - You must be a member of the group or the subgroup being invited. - The [visibility level](../../public_access.md) of the group you're inviting -must be at least as restrictive as that of the project. For example, you can invite: + must be at least as restrictive as that of the project. For example, you can invite: - A _private_ group to a _private_ project - A _private_ group to an _internal_ project. - A _private_ group to a _public_ project. @@ -46,12 +46,9 @@ must be at least as restrictive as that of the project. For example, you can inv ## Share a project with a group -> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal - window [with a flag](../../feature_flags.md). Disabled by default. -> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) - in GitLab 14.8. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. - [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 13.11 from a form to a modal window [with a flag](../../feature_flags.md). Disabled by default. +> - Modal window [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/247208) in GitLab 14.8. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. Similar to how you [share a group with another group](../../group/manage.md#share-a-group-with-another-group), you can share a project with a group by inviting that group to the project. @@ -101,6 +98,15 @@ NOTE: The Max role does not elevate the privileges of users. For example, if a group member has the role of Developer, and the group is invited to a project with a Max role of Maintainer, the member's role is not elevated to Maintainer. +### Which roles you can assign + +In GitLab [16.7](https://gitlab.com/gitlab-org/gitlab/-/issues/233408) and later, the maximum role you can assign depends on whether you have the Owner or Maintainer role for the project. The maximum role you can set is: + +- Owner (`50`), if you have the Owner role for the project. +- Maintainer (`40`), if you have the Maintainer role for the project. + +In GitLab 16.6 and earlier, the maximum role you can assign to an invited group is Maintainer (`40`). + ### View the member's Max role To view the maximum role assigned to a member: diff --git a/doc/user/project/merge_requests/approvals/img/group_access_example_01_v16_8.png b/doc/user/project/merge_requests/approvals/img/group_access_example_01_v16_8.png Binary files differnew file mode 100644 index 00000000000..0b5fbf7e075 --- /dev/null +++ b/doc/user/project/merge_requests/approvals/img/group_access_example_01_v16_8.png diff --git a/doc/user/project/merge_requests/approvals/img/group_access_example_02_v16_8.png b/doc/user/project/merge_requests/approvals/img/group_access_example_02_v16_8.png Binary files differnew file mode 100644 index 00000000000..49df19f2c46 --- /dev/null +++ b/doc/user/project/merge_requests/approvals/img/group_access_example_02_v16_8.png diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index 5436edf9f8d..bf4e2e8334e 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -17,15 +17,13 @@ flexibility: - Create required [rules](rules.md) about the number and type of approvers before work can merge. - Specify a list of users who act as [code owners](../../codeowners/index.md) for specific files, and require their approval before work can merge. +- For GitLab Premium and GitLab Ultimate, configure approvals + [for the entire instance](../../../../administration/merge_requests_approvals.md). You can configure merge request approvals on a per-project basis, and some approvals can be configured [on the group level](../../../group/manage.md#group-merge-request-approval-settings). Support for group-level settings for merge request approval rules is tracked in this -[epic](https://gitlab.com/groups/gitlab-org/-/epics/4367). Administrators of -[GitLab Premium](https://about.gitlab.com/pricing/) and -[GitLab Ultimate](https://about.gitlab.com/pricing/) self-managed GitLab instances -can also configure approvals -[for the entire instance](../../../../administration/admin_area.md). +[epic](https://gitlab.com/groups/gitlab-org/-/epics/4367). ## How approvals work @@ -48,6 +46,8 @@ You can also configure: - Merge request approval rules and settings through the GitLab UI or with the [Merge request approvals API](../../../../api/merge_request_approvals.md). +Approvals cannot be added after a merge request is merged. + ## Approve a merge request When an [eligible approver](rules.md#eligible-approvers) visits an open merge request, @@ -60,9 +60,8 @@ GitLab displays one of these buttons after the body of the merge request: Eligible approvers can also use the `/approve` [quick action](../../../project/quick_actions.md) when adding a comment to -a merge request. [In GitLab 13.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/292936), -if a user approves a merge request and is shown in the reviewer list, a green check mark -(**{check-circle-filled}**) displays next to their name. +a merge request. Users in the reviewer list who have approved a merge request display +a green check mark (**{check-circle-filled}**) next to their name. After a merge request receives the [number and type of approvals](rules.md) you configure, it can merge unless it's blocked for another reason. Merge requests can be blocked by other problems, @@ -80,8 +79,6 @@ of the rule. ## Optional approvals -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27426) in GitLab 13.2. - GitLab allows all users with Developer or greater [permissions](../../../permissions.md) to approve merge requests. Approvals in GitLab Free are optional, and don't prevent a merge request from merging without approval. @@ -128,7 +125,7 @@ Invalid approval rules created through a scan result policy are presented with ## Related topics - [Merge request approvals API](../../../../api/merge_request_approvals.md) -- [Instance-level approval rules](../../../../administration/admin_area.md) for self-managed installations +- [Instance-level approval rules](../../../../administration/merge_requests_approvals.md) for self-managed installations <!-- ## Troubleshooting diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index c284df8d9aa..2f1e2a96295 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -16,7 +16,10 @@ You can define approval rules: - [As project defaults](#add-an-approval-rule). - [Per merge request](#edit-or-override-merge-request-approval-rules). -- [At the instance level](../../../../administration/admin_area.md) + +You can configure approval rules: + +- [At the instance level](../../../../administration/merge_requests_approvals.md). If you don't define a [default approval rule](#add-an-approval-rule), any user can approve a merge request. Even if you don't define a rule, you can still @@ -311,8 +314,18 @@ For more information about this validation error, read ### Groups need explicit or inherited Developer role on a project A group created to handle approvals may be created in a different area of the -project hierarchy than the project requiring review. If this happens, the approvals group -isn't recognized as a valid Code Owner for the project, nor does it display in the -project's **Approvals** list. To fix this problem, add the approval group as a shared group -high enough in the shared hierarchy so the project requiring review inherits this -group of users. +project hierarchy than the project requiring review. If this happens, members of the +group may not be able to approve the merge request as they do not have access to it. + +For example: + +In the group structure below, project 1 belongs to subgroup 1 and subgroup 4 has users. + + + +Project 1 has a project level approval rule which assigns subgroup 4 as approvers. +When a merge request is created approvers from subgroup 4 appear in the eligible approvers list. +However, as users from subgroup 4 do not have permission to view the merge request, the `404` error is returned. +To grant membership, the group must be invited as a project member. It is now possible for users from subgroup 4 to approve. + + diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index f9e40a6714c..0120be0cf17 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -57,7 +57,7 @@ this setting, unless you configure one of these options: - [Prevent overrides of default approvals](#prevent-editing-approval-rules-in-merge-requests) at the project level. - *(Self-managed instances only)* Prevent overrides of default approvals - [at the instance level](../../../../administration/admin_area.md). When configured + [at the instance level](../../../../administration/merge_requests_approvals.md). When configured at the instance level, you can't edit this setting at the project or individual merge request levels. @@ -68,7 +68,7 @@ this setting, unless you configure one of these options: > - [Feature flag `keep_merge_commits_for_approvals`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131778) removed in GitLab 16.5. This check now includes merge commits. By default, users who commit to a merge request can still approve it. At both -the project level or [instance level](../../../../administration/admin_area.md), +the project level or instance level, you can prevent committers from approving merge requests that are partially their own. To do this: @@ -76,7 +76,7 @@ their own. To do this: 1. In the **Merge request approvals** section, scroll to **Approval settings** and select **Prevent approvals by users who add commits**. If this checkbox is cleared, an administrator has disabled it - [at the instance level](../../../../administration/admin_area.md), and + [at the instance level](../../../../administration/merge_requests_approvals.md), and it can't be changed at the project level. 1. Select **Save changes**. @@ -116,10 +116,11 @@ When this field is changed, it can affect all open merge requests depending on t > - Requiring re-authentication by using SAML authentication for GitLab.com groups [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5981) in GitLab 16.6 [with a flag](../../../../administration/feature_flags.md) named `ff_require_saml_auth_to_approve`. Disabled by default. > - Requiring re-authentication by using SAML authentication for self-managed instances [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/431415) in GitLab 16.7 [with a flag](../../../../administration/feature_flags.md) named `ff_require_saml_auth_to_approve`. Disabled by default. +> - [Enabled `ff_require_saml_auth_to_approve` by default](https://gitlab.com/gitlab-org/gitlab/-/issues/431714) in GitLab 16.8 for GitLab.com and self-managed instances. FLAG: -On self-managed GitLab, by default requiring re-authentication by using SAML authentication is not available. To make it available, an administrator can -[enable the feature flag](../../../../administration/feature_flags.md) named `ff_require_saml_auth_to_approve`. On GitLab.com, this feature is not available. +On self-managed GitLab, by default requiring re-authentication by using SAML authentication is available. To hide the feature, an administrator can +[disable the feature flag](../../../../administration/feature_flags.md) named `ff_require_saml_auth_to_approve`. On GitLab.com, this feature is available. You can force potential approvers to first authenticate with either: @@ -184,7 +185,7 @@ To do this: You can also enforce merge request approval settings: -- At the [instance level](../../../../administration/admin_area.md), which apply to all groups +- At the [instance level](../../../../administration/merge_requests_approvals.md), which apply to all groups on an instance and, therefore, all projects. - On a [top-level group](../../../group/manage.md#group-merge-request-approval-settings), which apply to all subgroups and projects. @@ -194,6 +195,6 @@ that inherited them. ## Related topics -- [Instance-level merge request approval settings](../../../../administration/admin_area.md) +- [Instance-level merge request approval settings](../../../../administration/merge_requests_approvals.md) - [Compliance center](../../../compliance/compliance_center/index.md) - [Merge request approvals API](../../../../api/merge_request_approvals.md) diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index 780041ac411..094d2cf5730 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -162,6 +162,13 @@ per conflicted file on the merge request diff:  +## Show scanner findings in diff **(ULTIMATE ALL)** + +You can show scanner findings in the diff. For details, see: + +- [Code Quality findings](../../../ci/testing/code_quality.md#merge-request-changes-view) +- [Static Analysis findings](../../application_security/sast/index.md#merge-request-changes-view) + ## Add a comment to a merge request file > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123515) in GitLab 16.1 [with a flag](../../../administration/feature_flags.md) named `comment_on_files`. Enabled by default. diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 951c848dee1..d2c5b0af339 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -155,23 +155,50 @@ You can create a merge request by running Git commands on your local machine. You can create a merge request from your fork to contribute back to the main project. -1. On the left sidebar, select **Search or go to** and find your project. -1. Select your fork of the repository. -1. On the left sidebar, select **Code > Merge requests**, and select **New merge request**. -1. In the **Source branch** dropdown list box, select the branch in your forked repository as the source branch. -1. In the **Target branch** dropdown list box, select the branch from the upstream repository as the target branch. - You can set a [default target project](#set-the-default-target-project) to - change the default target branch (which can be useful if you are working in a - forked project). +1. On the left sidebar, select **Search or go to** and find your fork. +1. Select **Code > Merge requests**, and select **New merge request**. +1. For **Source branch**, select the branch in your fork that contains your changes. +1. For **Target branch**: + + 1. Select the target project. (Make sure to select the upstream project, rather than your fork.) + 1. Select a branch from the upstream repository. + + NOTE: + If you contribute changes upstream frequently, consider setting a + [default target project](#set-the-default-target-project) for your fork. + 1. Select **Compare branches and continue**. -1. Select **Create merge request**. +1. Select **Create merge request**. The merge request is created in the target project, + not your fork. -After your work is merged, if you don't intend to -make any other contributions to the upstream project, you can -[unlink your fork](../repository/forking_workflow.md#unlink-a-fork) from its upstream project. +After your work merges, [unlink your fork](../repository/forking_workflow.md#unlink-a-fork) +from its upstream project if you don't intend to make more contributions. For more information, [see the forking workflow documentation](../repository/forking_workflow.md). +### Set the default target project + +By default, merge requests originating from a fork target your fork, not the upstream project. +If you frequently contribute back to the upstream project, and want to target it +by default, change the default target for your fork. + +Prerequisites: + +- You're working in a fork. +- You must have at least the Developer role, or be allowed to create merge requests in the project. +- The upstream project allows merge requests to be created. +- The [visibility settings](../../public_access.md#change-project-visibility) for + the fork must match, or be less strict than, the upstream repository. For example: + this setting isn't shown if your fork is private, but the upstream is public. + +To do this: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > Merge requests**. +1. In the **Target project** section, select the option you want to use for + your default target project. +1. Select **Save changes**. + ## By sending an email You can create a merge request by sending an email message to GitLab. @@ -179,60 +206,41 @@ The merge request target branch is the project's default branch. Prerequisites: +- The merge request must target the current project, not an upstream project. - A GitLab administrator must configure [incoming email](../../../administration/incoming_email.md). - A GitLab administrator must configure [Reply by email](../../../administration/reply_by_email.md). +- You must have at least the Developer role, or be allowed to create merge requests in the project. To create a merge request by sending an email: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. -1. In the upper-right corner, select **Email a new merge request to this project**. - An email address is displayed. Copy this address. - Ensure you keep this address private. +1. If the project contains any merge requests, select **Email a new merge request to this project**. +1. In the dialog, copy the email address shown. Keep this address private. Anyone who + has it can create issues or merge requests as if they were you. 1. Open an email and compose a message with the following information: - The **To** line is the email address you copied. - - The subject line is the source branch name. - - The message body is the merge request description. + - The **Subject** is the source branch name. + - The body of the email is the merge request description. -1. Send the email message. +1. To add commits, attach `.patch` files to the message. +1. Send the email. A merge request is created. ### Add attachments when creating a merge request by email -You can add commits to a merge request by adding -patches as attachments to the email. All attachments with a file name ending in `.patch` are considered patches and are processed -ordered by name. +Add commits to a merge request by adding patches as attachments to the email. -The combined size of the patches can be 2 MB. - -If the source branch from the subject does not exist, it is -created from the repository's HEAD or the specified target branch. -You can specify the target branch by using the -[`/target_branch` quick action](../quick_actions.md). If the source -branch already exists, the patches are applied on top of it. - -## Set the default target project - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58093) in GitLab 13.11. - -Merge requests have a source and a target project that are the same, unless -forking is involved. Creating a fork of the project can cause either of these -scenarios when you create a new merge request: - -- You target an upstream project (the project you forked, and the default - option). -- You target your own fork. - -To have merge requests from a fork by default target your own fork -(instead of the upstream project), you can change the default. - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Settings > Merge requests**. -1. In the **Target project** section, select the option you want to use for - your default target project. -1. Select **Save changes**. +- The combined size of the patches must be 2 MB or less. +- To be considered a patch, the attachment's file name must end in `.patch`. +- Patches are processed in order by name. +- If the source branch from the subject does not exist, it is + created from the repository's `HEAD`, or the default target branch. + To change the target branch manually, use the + [`/target_branch` quick action](../quick_actions.md). +- If the source branch already exists, patches are applied on top of it. ## Troubleshooting @@ -249,3 +257,14 @@ To make this button appear, one possible workaround is to [remove your project's fork relationship](../repository/forking_workflow.md#unlink-a-fork). After removal, the fork relationship cannot be restored. This project can no longer be able to receive or send merge requests to the source project, or other forks. + +### Email message could not be processed + +When sending an email to create a merge request, and you attempt to target an +upstream project, GitLab responds with this error: + +```plaintext +Unfortunately, your email message to GitLab could not be processed. + +You are not allowed to perform this action. If you believe this is in error, contact a staff member. +``` diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index a9cad78449b..3a2729bd64b 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -23,6 +23,9 @@ author can either retry any failed jobs, or push new commits to fix the failure: - If a retried job succeeds on the second try, the merge request is merged. - If new commits are added to the merge request, GitLab cancels the request to ensure the new changes are reviewed before merge. +- If new commits are added to the target branch of the merge request and + fast-forward only merge request is configured, GitLab cancels the request + to prevent merge conflicts. ## Auto-merge a merge request diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 23b1207619e..78e4c19dd57 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -19,6 +19,7 @@ review merge requests in Visual Studio Code. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Merge request review](https://www.youtube.com/watch?v=2MayfXKpU08&list=PLFGfElNsQthYDx0A_FaNNfUm9NHsK6zED&index=183). +<!-- Video published on 2023-04-29 --> ## GitLab Duo Suggested Reviewers **(ULTIMATE SAAS)** @@ -31,6 +32,7 @@ GitLab uses machine learning to suggest reviewers for your merge request. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [GitLab Duo Suggested Reviewers](https://www.youtube.com/embed/ivwZQgh4Rxw). +<!-- Video published on 2023-11-03 --> To suggest reviewers, GitLab uses: diff --git a/doc/user/project/merge_requests/widgets.md b/doc/user/project/merge_requests/widgets.md index d60bd660b53..a6680e5e4f4 100644 --- a/doc/user/project/merge_requests/widgets.md +++ b/doc/user/project/merge_requests/widgets.md @@ -73,3 +73,9 @@ If you have configured [License Compliance](../../compliance/license_scanning_of If you have configured [external status checks](status_checks.md) you can see the status of these checks in merge requests [in a specific widget](status_checks.md#status-checks-widget). + +## Application security scanning + +If you have enabled any application security scanning tools, the results are shown in the security +scanning widget. For more information, see +[security scanning output in merge request widget](../../application_security/index.md#merge-request). diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index a959507b338..f4df178794d 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -142,7 +142,7 @@ To edit a milestone: 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. 1. Select a milestone's title. -1. In the top right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Edit**. +1. In the upper-right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Edit**. 1. Edit the title, start date, due date, or description. 1. Select **Save changes**. @@ -159,7 +159,7 @@ To edit a milestone: 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Milestones**. 1. Select a milestone's title. -1. In the top right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Delete**. +1. In the upper-right corner, select **Milestone actions** (**{ellipsis_v}**) and then select **Delete**. 1. Select **Delete milestone**. ## Promote a project milestone to a group milestone diff --git a/doc/user/project/ml/experiment_tracking/mlflow_client.md b/doc/user/project/ml/experiment_tracking/mlflow_client.md index 35972f0ad7f..1522fd8e4fc 100644 --- a/doc/user/project/ml/experiment_tracking/mlflow_client.md +++ b/doc/user/project/ml/experiment_tracking/mlflow_client.md @@ -39,6 +39,9 @@ To use MLflow client compatibility from a local environment: 1. If the training code contains the call to `mlflow.set_tracking_uri()`, remove it. +In the model registry, you can copy the tracking URI from the overflow menu in the top right +by selecting the vertical ellipsis (**{ellipsis_v}**). + ## Model experiments When running the training code, MLflow client can be used to create experiments, runs, @@ -141,11 +144,22 @@ description = 'Model version description' model_version = client.create_model_version(model_name, source="", description=description) ``` +If the version parameter is not passed, it will be auto-incremented from the latest uploaded +version. You can set the version by passing a tag during model version creation. The version +must follow [SemVer](https://semver.org/) format. + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +tags = { "gitlab.version" = version } +client.create_model)version(model_name, version, description=description, tags=tags) +``` + **Notes** - Argument `run_id` is ignored. Every model version behaves as a Candidate/Run. Creating a mode version from a run is not yet supported. - Argument `source` is ignored. GitLab will create a package location for the model version files. -- Argument `tags` is ignored. - Argument `run_link` is ignored. - Argument `await_creation_for` is ignored. diff --git a/doc/user/project/ml/model_registry/index.md b/doc/user/project/ml/model_registry/index.md index 492ec9940ab..026afc01f22 100644 --- a/doc/user/project/ml/model_registry/index.md +++ b/doc/user/project/ml/model_registry/index.md @@ -29,13 +29,14 @@ at least the [Reporter role](../../../permissions.md#roles) to modify or delete ## Exploring models, model versions and model candidates -Model registry can be accessed on `https/<your-project>-/ml/models`. +To access the model registry, from the left sidebar, select **Deploy > Model registry**. ## Creating machine learning models and model versions Models and model versions can be created using the [MLflow](https://www.mlflow.org/docs/latest/tracking.html) client compatibility. -See [MLflow client compatibility](../experiment_tracking/mlflow_client.md#model-registry) on how to -create and manage models and model versions. +For more information about how to create and manage models and model versions, see [MLflow client compatibility](../experiment_tracking/mlflow_client.md#model-registry). +You can also create models directly on GitLab by selecting **Create Model** +on the Model registry page. ## Upload files, log metrics, log parameters to a model version diff --git a/doc/user/project/organize_work_with_projects.md b/doc/user/project/organize_work_with_projects.md index d41825af613..1371f5e77d0 100644 --- a/doc/user/project/organize_work_with_projects.md +++ b/doc/user/project/organize_work_with_projects.md @@ -1,6 +1,7 @@ --- stage: Data Stores group: Tenant Scale +description: Project visibility, search, badges, layout. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 345a30da198..b34369cb3b7 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -47,7 +47,7 @@ this document for an [overview on DNS records](dns_concepts.md). To add your custom domain to GitLab Pages: 1. On the left sidebar, select **Search or go to** and find your project. -1. On the left sidebar, select **Deploy > Pages**. +1. Select **Deploy > Pages**. 1. In the upper-right corner, select **New Domain**. 1. In **Domain**, enter the domain name. 1. Optional. In **Certificate**, turn off the **Automatic certificate management using Let's Encrypt** toggle to add an [SSL/TLS certificate](#adding-an-ssltls-certificate-to-pages). You can also add the certificate and key later. @@ -158,7 +158,7 @@ If you're using Cloudflare, check After you have added all the DNS records: 1. On the left sidebar, select **Search or go to** and find your project. -1. On the left sidebar, select **Deploy > Pages**. +1. Select **Deploy > Pages**. 1. Next to the domain name, select **Edit**. 1. In **Verification status**, select **Retry verification** (**{retry}**). @@ -287,7 +287,7 @@ domain (as long as you've set a valid certificate for it). To enable this setting: 1. On the left sidebar, select **Search or go to** and find your project. -1. On the left sidebar, select **Deploy > Pages**. +1. Select **Deploy > Pages**. 1. Select the **Force HTTPS (requires valid certificates)** checkbox. 1. Select **Save changes**. diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index 9de2703b82b..73583eefdda 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -70,7 +70,7 @@ This specific Ruby image is maintained on [DockerHub](https://hub.docker.com/_/r Edit your `.gitlab-ci.yml` file and add this text as the first line: ```yaml -image: ruby:2.7 +image: ruby:3.2 ``` If your SSG needs [NodeJS](https://nodejs.org/) to build, you must specify an @@ -156,7 +156,7 @@ pages: Your `.gitlab-ci.yml` file should now look like this: ```yaml -image: ruby:2.7 +image: ruby:3.2 pages: script: @@ -185,7 +185,7 @@ GitLab Pages daemon. GitLab runs it in the background and doesn't use a runner. ## Other options for your CI/CD file If you want to do more advanced tasks, you can update your `.gitlab-ci.yml` file -with [any of the available settings](../../../../ci/yaml/index.md). You can validate +with [other CI/CD YAML keywords](../../../../ci/yaml/index.md). You can validate your `.gitlab-ci.yml` file with the [CI Lint](../../../../ci/lint.md) tool that's included with GitLab. The following topics show other examples of other options you can add to your CI/CD file. @@ -198,7 +198,7 @@ First, add a `workflow` section to force the pipeline to run only when changes a pushed to branches: ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: @@ -218,7 +218,7 @@ Then configure the pipeline to run the job for the [default branch](../../repository/branches/default.md) (here, `main`) only. ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: @@ -249,7 +249,7 @@ To specify a stage for your job to run in, add a `stage` line to your CI file: ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: @@ -273,7 +273,7 @@ Now add another job to the CI file, telling it to test every push to every branch **except** the `main` branch: ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: @@ -325,7 +325,7 @@ for both jobs, `pages` and `test`. Move these commands to a `before_script` section: ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: @@ -366,7 +366,7 @@ This example caches Jekyll dependencies in a `vendor` directory when you run `bundle install`: ```yaml -image: ruby:2.7 +image: ruby:3.2 workflow: rules: diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index d658a09e760..00dfe0c66d9 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -226,7 +226,7 @@ Some other examples of mixing [variables](../../../ci/variables/index.md) with s ### Use multiple deployments to create pages environments -You can use multiple GitLap Pages deployments to create a new [environment](../../../ci/environments/index.md). +You can use multiple GitLab Pages deployments to create a new [environment](../../../ci/environments/index.md). For example: ```yaml diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 42a05e0b1bb..9f40b4d64af 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -133,8 +133,8 @@ pages: artifacts: paths: - public - only: - - main + rules: + - if: $CI_COMMIT_BRANCH == "main" ``` ### `.gitlab-ci.yml` for a static site generator @@ -145,7 +145,7 @@ See this document for a [step-by-step guide](getting_started/pages_from_scratch. Remember that GitLab Pages are by default branch/tag agnostic and their deployment relies solely on what you specify in `.gitlab-ci.yml`. You can limit -the `pages` job with the [`only` parameter](../../../ci/yaml/index.md#only--except), +the `pages` job with [`rules:if`](../../../ci/yaml/index.md#rulesif), whenever a new commit is pushed to a branch used specifically for your pages. @@ -175,8 +175,8 @@ pages: artifacts: paths: - public - only: - - pages + rules: + - if: '$CI_COMMIT_REF_NAME == "pages"' ``` See an example that has different files in the [`main` branch](https://gitlab.com/pages/jekyll-branched/tree/main) diff --git a/doc/user/project/pages/pages_access_control.md b/doc/user/project/pages/pages_access_control.md index 1c7aa0f182c..07b41f391ba 100644 --- a/doc/user/project/pages/pages_access_control.md +++ b/doc/user/project/pages/pages_access_control.md @@ -48,7 +48,8 @@ can access the website. To sign out of your GitLab Pages website, revoke the application access token for GitLab Pages: -1. In the top menu, select your profile, and then select **Settings**. -1. On the left sidebar, select **Applications**. -1. Scroll to the **Authorized applications** section, find the **GitLab Pages** - entry, and select its **Revoke** button. +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **Applications**. +1. In the **Authorized applications** section, find the **GitLab Pages** + entry, and select **Revoke**. diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index d13f30060e7..b88e7d2d376 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -26,7 +26,7 @@ are supported. | Rewrites (other than `200`) | **{dotted-circle}** No | `/en/* /en/404.html 404` | | Query parameters | **{dotted-circle}** No | `/store id=:id /blog/:id 301` | | Force ([shadowing](https://docs.netlify.com/routing/redirects/rewrites-proxies/#shadowing)) | **{dotted-circle}** No | `/app/ /app/index.html 200!` | -| Domain-level redirects | **{dotted-circle}** No | `http://blog.example.com/* https://www.example.com/blog/:splat 301` | +| [Domain-level redirects](#domain-level-redirects) | **{check-circle}** Yes | `http://blog.example.com/* https://www.example.com/blog/:splat 301` | | Redirect by country or language | **{dotted-circle}** No | `/ /anz 302 Country=au,nz` | | Redirect by role | **{dotted-circle}** No | `/admin/* 200! Role=admin` | @@ -119,6 +119,31 @@ request matches the `from`: This status code can be used in combination with [splat rules](#splats) to dynamically rewrite the URL. +## Domain-level redirects + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/936) in GitLab 16.8 [with a flag](../../../administration/feature_flags.md) named `FF_ENABLE_DOMAIN_REDIRECT`. Disabled by default. + +To create a domain-level redirect, add a domain-level path (beginning with `http://` +or `https://`) to either: + +- The `to` path only. +- The `from` and `to` paths. + +The supported [HTTP status codes](#http-status-codes) are `301` and `302`: + +```plaintext +# 301 permanent redirect +http://blog.example.com/file_1.html https://www.example.com/blog/file_1.html 301 +/file_2.html https://www.example.com/blog/file_2.html 301 + +# 302 temporary redirect +http://blog.example.com/file_3.html https://www.example.com/blog/file_3.html 302 +/file_4.html https://www.example.com/blog/file_4.html 302 +``` + +Domain-level redirects can be used in combination with [splat rules](#splats) (including splat placeholders) +to dynamically rewrite the URL path. + ## Splats > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/458) in GitLab 14.3. diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 0377ab389a5..60b862a4d3b 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -395,6 +395,6 @@ third-party Git clients. ### Branch names are case-sensitive -Branch names in `git` are case-sensitive. When configuring your protected branch -or [target branch rule](repository/branches/index.md#configure-rules-for-target-branches), +Branch names in `git` are case-sensitive. When configuring your protected branch, +or your [target branch workflow](repository/branches/index.md#configure-workflows-for-target-branches), `dev` is not the same `DEV` or `Dev`. diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 6c31b2ad5d3..1d721d71444 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -70,7 +70,7 @@ You should create a release as one of the last steps in your CI/CD pipeline. Prerequisites: - You must have at least the Developer role for a project. For more information, read -[Release permissions](#release-permissions). + [Release permissions](#release-permissions). To create a release in the Releases page: diff --git a/doc/user/project/releases/release_fields.md b/doc/user/project/releases/release_fields.md index c74ebaab89d..7c45a510877 100644 --- a/doc/user/project/releases/release_fields.md +++ b/doc/user/project/releases/release_fields.md @@ -50,7 +50,8 @@ A release contains the following types of assets: ### Source code GitLab automatically generates `zip`, `tar.gz`, `tar.bz2`, and `tar` -archived source code from the given Git tag. These are read-only assets. +archived source code from the given Git tag. These assets are read-only, +and [can be downloaded](../repository/index.md#download-the-code-in-a-repository). ### Links diff --git a/doc/user/project/remote_development/connect_machine.md b/doc/user/project/remote_development/connect_machine.md index cf46774741c..b37a2c5fc0f 100644 --- a/doc/user/project/remote_development/connect_machine.md +++ b/doc/user/project/remote_development/connect_machine.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Tutorial: Connect a remote machine to the Web IDE **(FREE ALL BETA)** +# Tutorial: Connect a remote machine to the Web IDE **(FREE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/371084) in GitLab 15.7. @@ -13,9 +13,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `vscode_web_ide`. On GitLab.com, this feature is available. The feature is not ready for production use. -WARNING: -This feature is in [Beta](../../../policy/experiment-beta-support.md#beta) and subject to change without notice. - This tutorial shows you how to: - Create a development environment outside of GitLab. diff --git a/doc/user/project/remote_development/index.md b/doc/user/project/remote_development/index.md index ac8d7102e40..65445e54949 100644 --- a/doc/user/project/remote_development/index.md +++ b/doc/user/project/remote_development/index.md @@ -4,7 +4,7 @@ group: IDE info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Remote development **(FREE ALL BETA)** +# Remote development **(FREE ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95169) in GitLab 15.4 [with a flag](../../../administration/feature_flags.md) named `vscode_web_ide`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/371084) in GitLab 15.7. @@ -13,9 +13,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `vscode_web_ide`. On GitLab.com, this feature is available. The feature is not ready for production use. -WARNING: -This feature is in [Beta](../../../policy/experiment-beta-support.md#beta) and subject to change without notice. - You can use remote development to write and compile code hosted on GitLab. With remote development, you can: diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index 0a0cbcbb9c8..d6bcae57322 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -35,7 +35,7 @@ To create a new branch from the GitLab UI: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. -1. On the top right, select **New branch**. +1. In the upper-right corner, select **New branch**. 1. Enter a **Branch name**. 1. In **Create from**, select the base of your branch: an existing branch, an existing tag, or a commit SHA. @@ -283,62 +283,64 @@ To do this: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. -1. On the upper right corner of the page, select **More** **{ellipsis_v}**. +1. In the upper right corner of the page, select **More** **{ellipsis_v}**. 1. Select **Delete merged branches**. 1. In the dialog, enter the word `delete` to confirm, then select **Delete merged branches**. -## Configure rules for target branches **(PREMIUM ALL)** +## Configure workflows for target branches **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127115) in GitLab 16.4 [with a flag](../../../../administration/feature_flags.md) named `target_branch_rules_flag`. Enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136431) in GitLab 16.7. Some projects use multiple long-term branches for development, like `develop` and `qa`. In these projects, you might want to keep `main` as the default branch, but expect -merge requests to target `develop` or `qa` instead. Target branch rules help ensure +merge requests to target `develop` or `qa` instead. Target branch workflows help ensure merge requests target the appropriate development branch for your project. -When you create a merge request, the rule checks the name of the branch. If the -branch name matches the rule, the merge request targets the branch you specify -in the rule. If the branch name does not match, the merge request targets the +When you create a merge request, the workflow checks the name of the branch. If the +branch name matches the workflow, the merge request targets the branch you specify. If the branch name does not match, the merge request targets the default branch of the project. +Rules are processed on a "first-match" basis - if two rules match the same branch name, the top-most rule is applied. + Prerequisites: - You must have at least the Maintainer role. -To create a target branch rule: +To create a target branch workflow: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. -1. Select **Add target branch rule**. -1. For **Rule name**, provide a string or wild card to compare against branch names. -1. Select the **Target branch** to use when the branch name matches the **Rule name**. +1. Scroll down to **Merge request branch workflow** +1. Select **Add branch target**. +1. For **Branch name pattern**, provide a string or wild card to compare against branch names. +1. Select the **Target branch** to use when the branch name matches the **Branch name pattern**. 1. Select **Save**. ### Example -You could configure your project to have the following target branch rules: +You could configure your project to have the following target branch workflows: -| Rule name | Target branch | +| Branch name pattern | Target branch | |-------------|---------------| | `feature/*` | `develop` | | `bug/*` | `develop` | | `release/*` | `main` | -These rules simplify the process of creating merge requests for a project that: +These target branches simplify the process of creating merge requests for a project that: - Uses `main` to represent the deployed state of your application. - Tracks current, unreleased development work in another long-running branch, like `develop`. -If your workflow initially places new features in `develop` instead of `main`, these rules +If your workflow initially places new features in `develop` instead of `main`, these target branches ensure all branches matching either `feature/*` or `bug/*` do not target `main` by mistake. -When you're ready to release to `main`, create a branch named `release/*`, and the rules +When you're ready to release to `main`, create a branch named `release/*`, and ensure this branch targets `main`. -## Delete a target branch rule +## Delete a target branch workflow -When you remove a target branch rule, existing merge requests remain unchanged. +When you remove a target branch workflow, existing merge requests remain unchanged. Prerequisites: @@ -348,7 +350,7 @@ To do this: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Merge requests**. -1. Select **Delete** on the rule you want to delete. +1. Select **Delete** on the branch target you want to delete. ## Related topics diff --git a/doc/user/project/repository/code_suggestions/index.md b/doc/user/project/repository/code_suggestions/index.md index 9e52da6068c..ed9eeac1c2c 100644 --- a/doc/user/project/repository/code_suggestions/index.md +++ b/doc/user/project/repository/code_suggestions/index.md @@ -4,40 +4,45 @@ group: Code Creation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Code Suggestions **(FREE ALL BETA)** +# Code Suggestions **(FREE ALL)** > - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. > - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. > - [Introduced support for Code Generation](https://gitlab.com/gitlab-org/gitlab/-/issues/415583) in GitLab 16.3. - -WARNING: -This feature is in [Beta](../../../../policy/experiment-beta-support.md#beta). -Beta users should read about the [known limitations](#known-limitations). We look forward to hearing your [feedback](#feedback). +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/435271) in GitLab 16.7. Write code more efficiently by using generative AI to suggest code while you're developing. -With Code Suggestions, you get: +With [GitLab Duo Code Suggestions](https://about.gitlab.com/solutions/code-suggestions/), you get: -- Code Completion, which suggests completions the current line you are typing. These suggestions are usually low latency. -- Code Generation, which generates code based on a natural language code comment block. Generating code can exceed multiple seconds. +- Code Completion, which suggests completions to the current line you are typing. These suggestions are usually low latency. +- Code Generation, which generates code based on a natural language code + comment block. Write a comment like `# Type more here` to generate the + appropriate code, based on the context of your comment and the rest of your code. + - Algorithms or large code blocks may take more than 10 seconds to generate. + - Streaming of code generation responses is supported in VS Code, leading to faster average response times. Other supported IDEs offer slower response times and will return the generated code in a single block. ## Start using Code Suggestions GitLab Duo Code Suggestions are available: -- On [self-managed](self_managed.md) and [SaaS](saas.md). View these pages to get started. +- In the Premium and Ultimate tier for [self-managed](self_managed.md), and across all tiers for [SaaS](saas.md). View these pages to get started. - In VS Code, Microsoft Visual Studio, JetBrains IDEs, and Neovim. You must have the corresponding GitLab extension installed. - In the GitLab Web IDE. <div class="video-fallback"> - <a href="https://youtu.be/wAYiy05fjF0">View how to setup and use GitLab Duo Code Suggestions</a>. + <a href="https://youtu.be/xQUlrbIWo8o">View how to setup and use GitLab Duo Code Suggestions</a>. </div> <figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/wAYiy05fjF0" frameborder="0" allowfullscreen> </iframe> + <iframe src="https://www.youtube-nocookie.com/embed/xQUlrbIWo8o" frameborder="0" allowfullscreen> </iframe> </figure> -During Beta, usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). -Learn about [data usage when using Code Suggestions](#code-suggestions-data-usage). As Code Suggestions matures to General Availability it will be governed by our [AI Functionality Terms](https://about.gitlab.com/handbook/legal/ai-functionality-terms/). +Code Suggestions is available and free to use until February 15, 2024: + +- Before February 15, 2024, usage of Code Suggestions is governed by the + [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +- On February 15, 2024, Code Suggestions becomes a paid add-on and will be governed by our + [AI Functionality Terms](https://about.gitlab.com/handbook/legal/ai-functionality-terms/). ## Supported languages @@ -51,7 +56,11 @@ For languages not listed in the following table, Code Suggestions might not func ### Supported languages in IDEs -Editor support for languages is documented in the following table. +Code Suggestions is aware of common popular programming concepts and +infrastructure-as-code interfaces, like Kubernetes Resource Model (KRM), +Google Cloud CLI, and Terraform. + +The editor supports these languages: | Language | VS Code | JetBrains IDEs | Visual Studio | Neovim | |------------------|------------------------|------------------------|------------------------|--------| @@ -61,16 +70,14 @@ Editor support for languages is documented in the following table. | Google SQL | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | Java | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | JavaScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | -| Kotlin | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Kotlin | **{check-circle}** Yes (Requires third-party extension providing Kotlin support) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | PHP | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | Python | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | Ruby | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | Rust | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | -| Scala | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Scala | **{check-circle}** Yes (Requires third-party extension providing Scala support) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | Swift | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | | TypeScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | -| Google Cloud | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | -| Kubernetes Resource Model (KRM) | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | | Terraform | **{check-circle}** Yes (Requires third-party extension providing Terraform support) | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes (Requires third-party extension providing the `terraform` file type) | NOTE: @@ -81,18 +88,13 @@ plugin support. Refer to the JetBrains documentation for specifics on your IDE. Code Suggestions supports a variety of popular editors including: -- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). +- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). Supports streaming responses for code generation. - [GitLab WebIDE (VS Code in the Cloud)](../../../project/web_ide/index.md), with no additional configuration. - Microsoft Visual Studio, using the [Visual Studio GitLab extension](https://marketplace.visualstudio.com/items?itemName=GitLab.GitLabExtensionForVisualStudio). - JetBrains IDEs, using the [GitLab plugin](https://plugins.jetbrains.com/plugin/22325-gitlab). - Neovim, using the [`gitlab.vim` plugin](https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim). -A [GitLab Language Server for Code Suggestions](https://gitlab.com/gitlab-org/editor-extensions/gitlab-language-server-for-code-suggestions) -is also in process. -This improvement should result in: - -- Faster iteration and standardization of the IDE extensions. -- The ability to use Code Suggestions even when an official editor extension isn't available. +A [GitLab Language Server](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp) is used in VS Code, Visual Studio, and Neovim. The Language Server supports faster iteration across more platforms. Users can also configure it to support Code Suggestions in IDEs where GitLab doesn't provide official support. ## Code Suggestions data usage @@ -120,7 +122,7 @@ For self-managed instances that have enabled Code Suggestions and SaaS accounts, ### Inference window context -Code Suggestions inferences against the currently opened file, the content before and after the cursor, the filename, and the extension type. For more information on possible future context expansion to improve the quality of suggestions, see [epic 11669](https://gitlab.com/groups/gitlab-org/-/epics/11669). +Code Suggestions inferences against the currently opened file, the content before and after the cursor, the file name, and the extension type. For more information on possible future context expansion to improve the quality of suggestions, see [epic 11669](https://gitlab.com/groups/gitlab-org/-/epics/11669). ### Training data @@ -133,14 +135,18 @@ For more information on GitLab Code Suggestions data [sub-processors](https://ab ## Known limitations -While in Beta, we are working on improving the accuracy of overall generated content. +We are continuing to work on the accuracy of overall generated content. However, Code Suggestions may generate suggestions that are: -- Low-quality -- Incomplete -- Produce failed pipelines -- Insecure code -- Offensive or insensitive +- Irrelevant. +- Incomplete. +- Results in failed pipelines. +- Potentially insecure. +- Offensive or insensitive. + +When using Code Suggestions, [code review best practice](../../../../development/code_review.md) still applies. + +Let us know if you have [feedback](#feedback). ## Progressive enhancement @@ -151,4 +157,11 @@ Code Suggestions do not prevent you from writing code in your IDE. ## Feedback -Report issues in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152). +Let us know about your Code Suggestions experience in [issue 435783](https://gitlab.com/gitlab-org/gitlab/-/issues/435783). + +## Troubleshooting + +### Disable Code Suggestions + +Individual users can disable Code Suggestions by disabling the feature in their +[installed IDE editor extension](index.md#supported-editor-extensions). diff --git a/doc/user/project/repository/code_suggestions/repository_xray.md b/doc/user/project/repository/code_suggestions/repository_xray.md new file mode 100644 index 00000000000..d851ee94e34 --- /dev/null +++ b/doc/user/project/repository/code_suggestions/repository_xray.md @@ -0,0 +1,60 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Repository X-Ray **(PREMIUM)** + +Repository X-Ray enhances GitLab Duo Code Suggestions by providing additional context to improve the accuracy and relevance of code recommendations. + +Repository X-Ray gives the code assistant more insight into the project's codebase and dependencies to generate better code suggestions. It does this by analyzing key project configuration files such as `Gemfile.lock`, `package.json`, and `go.mod` to build additional context. + +By understanding the frameworks, libraries and other dependencies in use, Repository X-Ray helps the code assistant tailor suggestions to match the coding patterns, styles and technologies used in the project. This results in code recommendations that integrate more seamlessly and follow best practices for that stack. + +## Supported languages and package managers + +| Language | Package Manager | Configuration File | +| ---------- |-----------------| -------------------- | +| Go | Go Modules | `go.mod` | +| JavaScript | NPM, Yarn | `package.json` | +| Ruby | RubyGems | `Gemfile.lock` | +| Python | Poetry | `pyproject.toml` | +| Python | Pip | `requirements.txt` | +| Python | Conda | `environment.yml` | + +## Enable Repository X-Ray + +Prerequisites: + +- You must have access to [GitLab Duo Code Suggestions](index.md) in the project. +- GitLab Runner must be set up and enabled for the project, because Repository X-Ray runs analysis pipelines using GitLab runners. + +To enable Repository X-Ray, add the following definition job to the project's `.gitlab-ci.yml`. + +```yaml +xray: + stage: build + image: registry.gitlab.com/gitlab-org/code-creation/repository-x-ray:latest + allow_failure: true + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + variables: + OUTPUT_DIR: reports + script: + - x-ray-scan -p "$CI_PROJECT_DIR" -o "$OUTPUT_DIR" + artifacts: + reports: + repository_xray: "$OUTPUT_DIR/*/*.json" +``` + +- The `$OUTPUT_DIR` environment variable defines the: + - Output directory for reports. + - Path that artifacts are uploaded from. +- The added rules restrict the job to the default branch only. Restricting the job this way ensures development changes do not impact the baseline X-Ray data used for production code suggestions. + +After the initial x-ray job completes and uploads the repository analysis reports, no further action is required. Repository X-Ray automatically enriches all code generation requests from that point forward. + +The X-Ray data for your project updates each time a CI/CD pipeline containing the `xray` +job is run. To learn more about pipeline configuration and triggers, see the +[pipelines documentation](../../../../ci/pipelines/merge_request_pipelines.md). diff --git a/doc/user/project/repository/code_suggestions/saas.md b/doc/user/project/repository/code_suggestions/saas.md index 1af5eef585c..4b1cc762406 100644 --- a/doc/user/project/repository/code_suggestions/saas.md +++ b/doc/user/project/repository/code_suggestions/saas.md @@ -4,13 +4,14 @@ group: Code Creation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Code Suggestions on GitLab SaaS **(FREE SAAS BETA)** +# Code Suggestions on GitLab SaaS **(FREE SAAS)** > - [Introduced](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#code-suggestions-available-in-closed-beta) in GitLab 15.9 as [Beta](../../../../policy/experiment-beta-support.md#beta) for early access Ultimate customers on GitLab.com. > - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/408104) as opt-in with GitLab 15.11 as [Beta](../../../../policy/experiment-beta-support.md#beta). > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/435271) in GitLab 16.7. Write code more efficiently by using generative AI to suggest code while you're developing. @@ -20,14 +21,14 @@ Learn about [data usage when using Code Suggestions](index.md#code-suggestions-d ## Enable Code Suggestions > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. Available to a percentage of users. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. UI user setting removed. A group owner must [enable Code Suggestions for your top-level group](../../../group/manage.md#enable-code-suggestions-for-a-group). NOTE: If you are having issues enabling Code Suggestions, view the -[troubleshooting guide](troubleshooting.md#code-suggestions-arent-displayed). +[troubleshooting guide](troubleshooting.md#code-suggestions-are-not-displayed). ## Use Code Suggestions @@ -35,19 +36,23 @@ Prerequisites: - You must have configured Code Suggestions in a [supported IDE editor extension](index.md#supported-editor-extensions). -- Code Suggestions must be enabled for: - - [The top-level group](../../../group/manage.md#enable-code-suggestions-for-a-group). - - [Your own account](../../../profile/preferences.md#enable-code-suggestions), if your - account is not part of the percentage rollout. +- Code Suggestions must be enabled for [the top-level group](../../../group/manage.md#enable-code-suggestions-for-a-group). To use Code Suggestions: -1. Determine if your user account is part of the percentage rollout. See - [Enable Code Suggestions](../../../profile/preferences.md#enable-code-suggestions) - for more information. 1. Author your code. As you type, suggestions are displayed. Code Suggestions provide code snippets or complete the current line, depending on the cursor position. -1. Describe the requirements in natural language. Be concise and specific. Code Suggestions generates functions and code snippets as appropriate. +1. Describe the requirements in natural language. Code Suggestions generates functions and code snippets based on the context provided. To get the best results from code generation: + - Be as specific as possible while remaining concise. State the outcome you want to generate (for example, a function) and provide details on what you want to achieve. Add additional information, such as the framework or library you want to use when applicable. + For example, to create a Python web service with some specific requirements, you might write something similar to the following: + + ```plaintext + # Create a web service using Tornado that allows a user to log in, run a security scan, and review the scan results. + # Each action (log in, run a scan, and review results) should be its own resource in the web service + ... + ``` + + - Add a space or go to a new line after each comment. This tells the code generator that you have completed your instructions. 1. To accept a suggestion, press <kbd>Tab</kbd>. 1. To ignore a suggestion, keep typing as you usually would. 1. To explicitly reject a suggestion, press <kbd>Esc</kbd>. diff --git a/doc/user/project/repository/code_suggestions/self_managed.md b/doc/user/project/repository/code_suggestions/self_managed.md index 26850bc8b5f..8cd499c13d0 100644 --- a/doc/user/project/repository/code_suggestions/self_managed.md +++ b/doc/user/project/repository/code_suggestions/self_managed.md @@ -4,12 +4,13 @@ group: Code Creation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Code Suggestions on self-managed GitLab **(SELF BETA)** +# Code Suggestions on self-managed GitLab **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta) on self-managed GitLab. > - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. > - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. > - Code Suggestions in the GitLab WebIDE enabled for all GitLab-hosted customers. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/435271) in GitLab 16.7. Write code more efficiently by using generative AI to suggest code while you're developing. @@ -28,16 +29,19 @@ Learn about [data usage when using Code Suggestions](index.md#code-suggestions-d ## Enable Code Suggestions on self-managed GitLab > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. Available to a percentage of users. +> - [Enabled self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139916) in GitLab 16.8. When you enable Code Suggestions for your self-managed instance, you: - Agree to the [GitLab testing agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). - Acknowledge that GitLab sends data from the instance, including personal data, to GitLab.com infrastructure. -How you enable Code Suggestions for your instance differs depending on your version of GitLab. +How you enable Code Suggestions for your instance differs depending on your +version of GitLab. This setting is visible only in self-managed GitLab instances. -### GitLab 16.3 and later **(PREMIUM)** +::Tabs + +:::TabTitle GitLab 16.3 and later Prerequisites: @@ -53,17 +57,12 @@ To enable Code Suggestions for your self-managed GitLab instance: In GitLab 16.3, you do not need to enter anything into the **Personal access token** field. In GitLab 16.4 and later, there is no **Personal access token** field. 1. Select **Save changes**. - -This setting is visible only in self-managed GitLab instances. - -WARNING: -In GitLab 16.2 and earlier, if you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. - -To make sure Code Suggestions works immediately, you must [manually synchronize your subscription](#manually-synchronize-your-subscription). +1. To make sure Code Suggestions works immediately, you must + [manually synchronize your subscription](#manually-synchronize-your-subscription). The users in your instance can now use Code Suggestions. -### GitLab 16.2 and earlier +:::TabTitle GitLab 16.2 and earlier FLAG: On self-managed GitLab 16.0 and earlier, GitLab Duo Code Suggestions is not available. To use this feature, you must have GitLab 16.1 or later. For optimal performance and full feature access, you should upgrade to GitLab 16.3 or later, which supports cloud licensing. @@ -75,7 +74,7 @@ Prerequisites: - You have a [GitLab SaaS account](https://gitlab.com/users/sign_up). You do not need to have a GitLab SaaS subscription. NOTE: -If you do not have a customer success manager, you cannot participate in the free trial of Code Suggestions on self-managed GitLab. Upgrade to GitLab 16.3 to [perform self-service onboarding](#gitlab-163-and-later). +If you do not have a customer success manager, you cannot participate in the free trial of Code Suggestions on self-managed GitLab. Upgrade to GitLab 16.3 or later to perform self-service onboarding. Then, you will: @@ -83,7 +82,7 @@ Then, you will: 1. Enable Code Suggestions for the instance. 1. [Request early access](#request-access-to-code-suggestions) to the Code Suggestions Beta. -#### Enable Code Suggestions for your SaaS account +### Enable Code Suggestions for your SaaS account To enable Code Suggestions for your GitLab SaaS account: @@ -94,7 +93,7 @@ To enable Code Suggestions for your GitLab SaaS account: 1. In the **Code Suggestions** section, select **Enable Code Suggestions**. 1. Select **Save changes**. -#### Enable Code Suggestions for the instance +### Enable Code Suggestions for the instance To enable Code Suggestions for your self-managed GitLab instance: @@ -110,7 +109,9 @@ This setting is visible only in self-managed GitLab instances. WARNING: If you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. -#### Request access to Code Suggestions +::EndTabs + +### Request access to Code Suggestions GitLab provisions access on a customer-by-customer basis for Code Suggestions on self-managed instances. To request access, contact your customer success manager. @@ -120,7 +121,7 @@ Your customer success manager then provisions access by commenting on [issue 415 After GitLab has provisioned access to Code Suggestions for your instance, the users in your instance can now enable Code Suggestions. -### Configure network and proxy settings +## Configure network and proxy settings Configure any firewalls to allow outbound connections to `https://codesuggestions.gitlab.com/`. @@ -128,7 +129,7 @@ If your GitLab instance uses an HTTP proxy server to access the internet, ensure the server is configured to allow outbound connections, including the [`gitlab_workhorse` environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html). -### Upgrade GitLab +## Upgrade GitLab In GitLab 16.3 and later, GitLab is enforcing the cloud licensing requirement for Code Suggestions: @@ -138,11 +139,11 @@ In GitLab 16.3 and later, GitLab is enforcing the cloud licensing requirement fo If you have a GitLab Free subscription and upgrade to GitLab 16.3 or later, to continue having early access to Code Suggestions, you must: -1. Have a [subscription that supports cloud licensing](https://about.gitlab.com/pricing/). +1. Have a [subscription that supports cloud licensing](https://about.gitlab.com/pricing/licensing-faq/cloud-licensing/). 1. Make sure you have the latest version of your [IDE extension](index.md#supported-editor-extensions). 1. [Manually synchronize your subscription](#manually-synchronize-your-subscription). -#### Manually synchronize your subscription +### Manually synchronize your subscription You must [manually synchronize your subscription](../../../../subscriptions/self_managed/index.md#manually-synchronize-your-subscription-details) if either: diff --git a/doc/user/project/repository/code_suggestions/troubleshooting.md b/doc/user/project/repository/code_suggestions/troubleshooting.md index c18ea2dd26b..47327d7a28f 100644 --- a/doc/user/project/repository/code_suggestions/troubleshooting.md +++ b/doc/user/project/repository/code_suggestions/troubleshooting.md @@ -4,35 +4,34 @@ group: Code Creation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Troubleshooting Code Suggestions **(FREE ALL BETA)** +# Troubleshooting Code Suggestions **(FREE ALL)** When working with GitLab Duo Code Suggestions, you might encounter the following issues. -## Code Suggestions aren't displayed +## Code Suggestions are not displayed If Code Suggestions are not displayed, and you have [installed a supported IDE extension](index.md#supported-editor-extensions), try the following troubleshooting steps. -In GitLab, ensure Code Suggestions is enabled: - -- [For your user account](../../../profile/preferences.md#enable-code-suggestions). -- [For **all** top-level groups your account belongs to](../../../group/manage.md#enable-code-suggestions-for-a-group). If you don't have a role that lets you view the top-level group's settings, contact a group owner. +In GitLab, ensure Code Suggestions is enabled for **at least one** +[top-level group your account belongs to](../../../group/manage.md#enable-code-suggestions-for-a-group). +If you don't have a role that lets you view the top-level group's settings, contact a group owner. ### Code Suggestions not displayed in VS Code or GitLab WebIDE -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. +Check all the steps in [Code Suggestions are not displayed](#code-suggestions-are-not-displayed) first. If you are a self-managed user, ensure that Code Suggestions for the [GitLab WebIDE](../../../project/web_ide/index.md) are enabled. The same settings apply to VS Code as local IDE. 1. On the left sidebar, select **Extensions > GitLab Workflow**. 1. Select **Settings** (**{settings}**), and then select **Extension Settings**. -1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion (Beta)** +1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion** checkbox. If the settings are enabled, but Code Suggestions are still not displayed, try the following steps: 1. Enable the `Debug` checkbox in the GitLab Workflow **Extension Settings**. 1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Workflow** as the log filter. The command palette command is `GitLab: Show Extension Logs`. -1. Disable and re-enable the **Enable code completion (Beta)** checkbox. +1. Disable and re-enable the **Enable code completion** checkbox. 1. Verify that the debug log contains similar output: ```shell @@ -43,7 +42,7 @@ If the settings are enabled, but Code Suggestions are still not displayed, try t ### Code Suggestions not displayed in Microsoft Visual Studio -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. +Check all the steps in [Code Suggestions are not displayed](#code-suggestions-are-not-displayed) first. 1. Ensure you have properly [set up the extension](https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension#setup). 1. From the **Tools > Options** menu, find the **GitLab** option. Ensure **Log Level** is set to **Debug**. diff --git a/doc/user/project/repository/git_blame.md b/doc/user/project/repository/git_blame.md index a602638d244..f1fb118b8d5 100644 --- a/doc/user/project/repository/git_blame.md +++ b/doc/user/project/repository/git_blame.md @@ -34,6 +34,28 @@ to the left of the user avatar shows the general age of the commit. The newest commits have a dark blue bar. As the age of the commit increases, the bar color changes to light gray. +### View blame directly in the file view +<!-- +When feature flags `graphql_git_blame`, `blob_blame_info` and `highlight_js_worker` are removed, +delete this section and update the steps in "View blame for a file". +--> + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/430950) in GitLab 16.7 [with flags](../../../administration/feature_flags.md) named `graphql_git_blame`, `blob_blame_info` and `highlight_js_worker`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available. +To hide the feature, an administrator can [disable the feature flag](../../../administration/feature_flags.md) named `blob_blame_info`. +On GitLab.com, this feature is available. + +When this feature is enabled, you can additionally view blame for a file directly from the file page. + +To do so: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Code > Repository**. +1. Select the file you want to review. +1. In the file header, select **Blame**, and go to the line you want to see. + ### Blame previous commit To see earlier revisions of a specific line: diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md deleted file mode 100644 index 592041ef4e2..00000000000 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../signed_commits/gpg.md' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](../signed_commits/gpg.md). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index dd8ee61f6ae..550ff25e0b1 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -39,7 +39,7 @@ You can upload a file from the GitLab UI. 1. Go to the directory where you want to upload the file. 1. Next to the directory name, select the plus icon (**{plus}**) > **Upload file**. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Upload file**. @@ -85,7 +85,7 @@ Projects that contain a `.xcodeproj` or `.xcworkspace` directory can be cloned into Xcode on macOS. 1. From the GitLab UI, go to the project's overview page. -1. Select **Clone**. +1. In the upper-right corner, select **Code**. 1. Select **Xcode**. The project is cloned onto your computer and you are @@ -101,7 +101,7 @@ Visual Studio Code: - From the GitLab interface: 1. Go to the project's overview page. - 1. Select **Clone**. + 1. In the upper-right corner, select **Code**. 1. Under **Open in your IDE**, select **Visual Studio Code (SSH)** or **Visual Studio Code (HTTPS)**. 1. Select a folder to clone the project into. @@ -121,16 +121,15 @@ Prerequisites: To do this: 1. Go to the project's overview page. -1. Select **Clone**. +1. In the upper-right corner, select **Code**. 1. Under **Open in your IDE**, select **IntelliJ IDEA (SSH)** or **IntelliJ IDEA (HTTPS)**. ## Download the code in a repository -> Support for [including Git LFS blobs](../../../topics/git/lfs#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. - You can download the source code that's stored in a repository. -1. Above the file list, select the download icon (**{download}**). +1. On the left sidebar, select **Search or go to** and find your project. +1. Above the file list, select **Code**. 1. From the options, select the files you want to download. - **Source code:** diff --git a/doc/user/project/repository/managing_large_repositories.md b/doc/user/project/repository/managing_large_repositories.md deleted file mode 100644 index 1fedd8da20c..00000000000 --- a/doc/user/project/repository/managing_large_repositories.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'monorepos/index.md' -remove_date: '2023-12-17' ---- - -This document was moved to [another location](monorepos/index.md). - -<!-- This redirect file can be deleted after <2023-12-17>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md index d4ab550cb8a..dc789d28a4f 100644 --- a/doc/user/project/repository/mirror/bidirectional.md +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -39,7 +39,7 @@ instance can help reduce race conditions by syncing changes more frequently. Prerequisites: - You have configured the [push](push.md#set-up-a-push-mirror-to-another-gitlab-instance-with-2fa-activated) -and [pull](pull.md#pull-from-a-remote-repository) mirrors in the upstream GitLab instance. + and [pull](pull.md#pull-from-a-remote-repository) mirrors in the upstream GitLab instance. To create the webhook in the downstream instance: diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index 34a2757bb67..9d5048a4fed 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -208,4 +208,4 @@ Older versions of SSH may require you to remove `-E md5` from the command. - [Troubleshooting](troubleshooting.md) for repository mirroring. - Configure a [Pull Mirroring Interval](../../../../administration/instance_limits.md#pull-mirroring-interval) - [Disable mirrors for a project](../../../../administration/settings/visibility_and_access_controls.md#enable-project-mirroring) -- [Secrets file and mirroring](../../../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost) +- [Secrets file and mirroring](../../../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost) diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md index 3aa4c768ebe..babe99441ef 100644 --- a/doc/user/project/repository/mirror/push.md +++ b/doc/user/project/repository/mirror/push.md @@ -161,7 +161,7 @@ To set up a mirror from GitLab to AWS CodeCommit: 1. Copy or download the special Git HTTPS user ID and password. 1. In the AWS CodeCommit console, create a new repository to mirror from your GitLab repository. -1. Open your new repository, and then select **Clone URL > Clone HTTPS** (not **Clone HTTPS (GRC)**). +1. Open your new repository, in the upper-right corner, select **Code > Clone HTTPS** (not **Clone HTTPS (GRC)**). 1. In GitLab, open the repository to be push-mirrored. 1. Select **Settings > Repository**, and then expand **Mirroring repositories**. 1. Fill in the **Git repository URL** field using this format, replacing diff --git a/doc/user/project/repository/mirror/troubleshooting.md b/doc/user/project/repository/mirror/troubleshooting.md index 57a9351e85d..f252c047072 100644 --- a/doc/user/project/repository/mirror/troubleshooting.md +++ b/doc/user/project/repository/mirror/troubleshooting.md @@ -215,3 +215,13 @@ Project.where(mirror: true).each do |project| project.save end ``` + +## `The requested URL returned error: 301` + +When mirroring using the `http://` or `https://` protocols, be sure to specify the exact URL to the repository: `https://gitlab.example.com/group/project.git` + +HTTP redirects are not followed and omitting `.git` can result in a 301 error: + +```plaintext +13:fetch remote: "fatal: unable to access 'https://gitlab.com/group/project': The requested URL returned error: 301\n": exit status 128. +``` diff --git a/doc/user/project/repository/push_rules.md b/doc/user/project/repository/push_rules.md index fc36748a8dd..b15d66e27fb 100644 --- a/doc/user/project/repository/push_rules.md +++ b/doc/user/project/repository/push_rules.md @@ -118,20 +118,20 @@ Some validation examples: - Branches must start with `JIRA-`. ```plaintext - `^JIRA-` + ^JIRA- ``` - Branches must end with `-JIRA`. ```plaintext - `-JIRA$` + -JIRA$ ``` - Branches must be between `4` and `15` characters long, accepting only lowercase letters, numbers and dashes. ```plaintext - `^[a-z0-9\\-]{4,15}$` + ^[a-z0-9\\-]{4,15}$ ``` ## Prevent unintended consequences diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index a060973d89f..528e9eefa44 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -190,7 +190,7 @@ no longer being available. To clean up a repository: -1. Go to the project for the repository. +1. On the left sidebar, select **Search or go to** and find your project. 1. Go to **Settings > Repository**. 1. Upload a list of objects. For example, a `commit-map` file created by `git filter-repo` which is located in the `filter-repo` directory. diff --git a/doc/user/project/repository/signed_commits/ssh.md b/doc/user/project/repository/signed_commits/ssh.md index e1c2a73be3e..1d3fca6d681 100644 --- a/doc/user/project/repository/signed_commits/ssh.md +++ b/doc/user/project/repository/signed_commits/ssh.md @@ -156,7 +156,7 @@ To revoke an SSH key: 1. On the left sidebar, select your avatar. 1. Select **Edit profile**. -1. On the left sidebar, select (**{key}**) **SSH Keys**. +1. On the left sidebar, select **SSH Keys** (**{key}**). 1. Select **Revoke** next to the SSH key you want to delete. ## Related topics diff --git a/doc/user/project/repository/ssh_signed_commits/index.md b/doc/user/project/repository/ssh_signed_commits/index.md deleted file mode 100644 index 89e3d811dba..00000000000 --- a/doc/user/project/repository/ssh_signed_commits/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../signed_commits/ssh.md' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](../signed_commits/ssh.md). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 3899890ea7e..d2df9cc18ae 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -20,7 +20,7 @@ To create a text file in the Web Editor: 1. Go to the directory where you want to create the new file. 1. Next to the directory name, select the plus icon (**{plus}**) > **New file**. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Commit changes**. @@ -31,14 +31,14 @@ To create a text file from a template in the Web Editor: 1. On the left sidebar, select **Search or go to** and find your project. 1. Go to the directory where you want to create the new file. 1. Next to the directory name, select the plus icon (**{plus}**) > **New file**. -1. In **Filename**, enter a filename that GitLab provides a template for: +1. In **Filename**, enter a name that GitLab provides a template for: - `.gitignore` - `.gitlab-ci.yml` - `LICENSE` - `Dockerfile` 1. From the **Apply a template** dropdown list, select a template. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Commit changes**. @@ -50,7 +50,7 @@ To edit a text file in the Web Editor: 1. Go to the file you want to edit. 1. Select **Edit > Edit single file**. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Commit changes**. @@ -72,7 +72,7 @@ To close the preview panel, select the **Write** tab. ### Link to specific lines To link to single or multiple lines in the Web Editor, add hash -information to the filename segment of the URL. For example: +information to the file name segment of the URL. For example: - `MY_FILE.js#L3` highlights line 3 in `MY_FILE.js`. - `MY_FILE.js#L3-10` highlights lines 3 to 10 in `MY_FILE.js`. @@ -90,7 +90,7 @@ To upload a file in the Web Editor: 1. Go to the directory where you want to upload the file. 1. Next to the directory name, select the plus icon (**{plus}**) > **Upload file**. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Upload file**. @@ -102,7 +102,7 @@ To create a directory in the Web Editor: 1. Go to the directory where you want to create the new directory. 1. Next to the directory name, select the plus icon (**{plus}**) > **New directory**. 1. Complete the fields. - - To create a merge request with your changes, enter a branch name + To create a merge request with your changes, enter a branch name that's not your repository's [default branch](branches/default.md). 1. Select **Create directory**. diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md deleted file mode 100644 index ae418581820..00000000000 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../signed_commits/x509.md' -remove_date: '2023-12-01' ---- - -This document was moved to [another location](../signed_commits/x509.md). - -<!-- This redirect file can be deleted after <2023-12-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 0594f3fe2ee..e489f19585c 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -50,7 +50,7 @@ To create a requirement: 1. In a project, go to **Plan > Requirements**. 1. Select **New requirement**. -1. Enter a title and description and select **Create requirement**. +1. Enter a title and description and select **New requirement**.  @@ -240,7 +240,8 @@ To import requirements: 1. In a project, go to **Plan > Requirements**. - For a project with requirements, in the - upper-right corner, select the import icon (**{import}**). + upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), + then select **Import requirements** (**{import}**). - For a project without requirements, in the middle of the page, select **Import CSV**. 1. Select the file and select **Import requirements**. @@ -300,7 +301,8 @@ Prerequisites: To export requirements: 1. In a project, go to **Plan > Requirements**. -1. In the upper-right corner, select **Export as CSV** (**{export}**). +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), + then select **Export as CSV** (**{export}**). A confirmation modal appears. diff --git a/doc/user/project/service_desk/configure.md b/doc/user/project/service_desk/configure.md index 91dbe7a38dd..95c15ef42b7 100644 --- a/doc/user/project/service_desk/configure.md +++ b/doc/user/project/service_desk/configure.md @@ -63,14 +63,14 @@ For example, you can format the emails to include a header and footer in accorda organization's brand guidelines. You can also include the following placeholders to display dynamic content specific to the Service Desk ticket or your GitLab instance. -| Placeholder | `thank_you.md` | `new_note.md` | Description -| ---------------------- | ---------------------- | ---------------------- | ----------- -| `%{ISSUE_ID}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket IID. -| `%{ISSUE_PATH}` | **{check-circle}** Yes | **{check-circle}** Yes | Project path appended with the ticket IID. -| `%{ISSUE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | URL of the ticket. External participants can only view the ticket if the project is public and ticket is not confidential (Service Desk tickets are confidential by default). -| `%{ISSUE_DESCRIPTION}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket description. If a user has edited the description, it may contain sensitive information that is not intended to be delivered to external participants. Use this placeholder with care and ideally only if you never modify descriptions or your team is aware of the template design. -| `%{UNSUBSCRIBE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe URL. -| `%{NOTE_TEXT}` | **{dotted-circle}** No | **{check-circle}** Yes | The new comment added to the ticket by a user. Take care to include this placeholder in `new_note.md`. Otherwise, the requesters may never see the updates on their Service Desk ticket. +| Placeholder | `thank_you.md` | `new_note.md` | Description | +|------------------------|------------------------|------------------------|-------------| +| `%{ISSUE_ID}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket IID. | +| `%{ISSUE_PATH}` | **{check-circle}** Yes | **{check-circle}** Yes | Project path appended with the ticket IID. | +| `%{ISSUE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | URL of the ticket. External participants can only view the ticket if the project is public and ticket is not confidential (Service Desk tickets are confidential by default). | +| `%{ISSUE_DESCRIPTION}` | **{check-circle}** Yes | **{check-circle}** Yes | Ticket description. If a user has edited the description, it may contain sensitive information that is not intended to be delivered to external participants. Use this placeholder with care and ideally only if you never modify descriptions or your team is aware of the template design. | +| `%{UNSUBSCRIBE_URL}` | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe URL. | +| `%{NOTE_TEXT}` | **{dotted-circle}** No | **{check-circle}** Yes | The new comment added to the ticket by a user. Take care to include this placeholder in `new_note.md`. Otherwise, the requesters may never see the updates on their Service Desk ticket. | ### Thank you email @@ -105,7 +105,7 @@ Instance administrators can add a header, footer or additional text to the GitLa them to all emails sent from GitLab. If you're using a custom `thank_you.md` or `new_note.md`, to include this content, add `%{SYSTEM_HEADER}`, `%{SYSTEM_FOOTER}`, or `%{ADDITIONAL_TEXT}` to your templates. -For more information, see [System header and footer messages](../../../administration/appearance.md#system-header-and-footer-messages) and [custom additional text](../../../administration/settings/email.md#custom-additional-text). +For more information, see [System header and footer messages](../../../administration/appearance.md#add-system-header-and-footer-messages) and [custom additional text](../../../administration/settings/email.md#custom-additional-text). ## Use a custom template for Service Desk tickets @@ -165,6 +165,7 @@ the assignees of the issue and creates to-do items for them. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a walkthrough, see [a short showcase video](https://youtu.be/163wDM1e43o). +<!-- Video published on 2023-12-12 --> Prerequisites: @@ -191,6 +192,7 @@ Maintain brand identity and instill confidence among support requesters with a d <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [a short showcase video](https://youtu.be/_moD5U3xcQs). +<!-- Video published on 2023-09-12 --> This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). A Beta feature is not production-ready, but is unlikely to change drastically @@ -225,12 +227,12 @@ Configure and verify a custom email address when you want to send Service Desk e 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. -1. Expand **Service Desk** and find the **Custom email** settings. +1. Expand **Service Desk** and find the **Configure a custom email address** section. 1. Note the presented Service Desk address of this project, and with your email provider (for example, Gmail), set up email forwarding from the custom email address to the Service Desk address. 1. Back in GitLab, complete the fields. -1. Select **Save & test settings**. +1. Select **Save & test connection**. The configuration has been saved and the verification of the custom email address is triggered. @@ -945,7 +947,7 @@ or completely separately. ::EndTabs 1. GitLab offers two methods to transport emails from `mail_room` to the GitLab -application. You can configure the `delivery_method` for each email setting individually: + application. You can configure the `delivery_method` for each email setting individually: 1. Recommended: `webhook` (default in GitLab 15.3 and later) sends the email payload via an API POST request to your GitLab application. It uses a shared token to authenticate. If you choose this method, make sure the `mail_room` process can access the API endpoint and distribute the shared diff --git a/doc/user/project/service_desk/index.md b/doc/user/project/service_desk/index.md index 9ab69c4bdb8..6a15b9798f7 100644 --- a/doc/user/project/service_desk/index.md +++ b/doc/user/project/service_desk/index.md @@ -14,6 +14,10 @@ Service Desk emails are created in your GitLab project as new issues. Your team can respond directly from the project, while customers interact with the thread only through email. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video overview, see [Introducing GitLab Service Desk (GitLab 16.7)](https://www.youtube.com/watch?v=LDVQXv3I5rI). +<!-- Video published on 2023-12-19 --> + ## Service Desk workflow For example, let's assume you develop a game for iOS or Android. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index fc9b24362e0..f2faa0676b5 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -4,27 +4,31 @@ group: Import and Integrate info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# Migrating projects using file exports **(FREE ALL)** +# Migrate projects and groups by using file exports **(FREE ALL)** + +You can migrate projects and groups by using file exports. However, using +[direct transfer](../../group/import/index.md) is recommended if possible. + +## Migrate projects by uploading an export file Existing projects on any self-managed GitLab instance or GitLab.com can be exported to a file and -then imported into another GitLab instance. You can also copy GitLab projects to another location with more automation by -[migrating groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +then imported into another GitLab instance. -## Preserving user contributions +### Preserving user contributions Preserving user contribution depends on meeting the following requirements: -### Migrating from GitLab self-managed to GitLab.com +#### Migrating from GitLab self-managed to GitLab.com When migrating projects by using file exports, an administrator's access token is required for user contributions to map correctly. Therefore, user contributions never map correctly when importing file exports from a self-managed instance to GitLab.com. Instead, all GitLab user associations (such as comment author) are changed to the user importing the project. To preserve contribution history, do one of the following: -- [Migrate by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +- [Migrate by direct transfer](../../group/import/index.md). - Consider paid GitLab [migration services](https://about.gitlab.com/services/migration/). -### Migrating to GitLab self-managed +#### Migrating to GitLab self-managed To ensure GitLab maps users and their contributions correctly: @@ -43,7 +47,7 @@ That user becomes an author of merge requests created by other users. Supplement - Added for comments, merge request approvals, linked tasks, and items. - Not added for the merge request or issue creator, added or removed labels, and merged-by information. -## Edit project export files +### Edit project export files You can add or remove data from export files. For example, you can: @@ -58,7 +62,7 @@ To edit a project export file: You can also make sure that all members were exported by checking the `project_members.ndjson` file. -## Compatibility +### Compatibility > Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/389888) in GitLab 15.11. @@ -75,7 +79,7 @@ For example: | 13.0 | 13.0, 12.10, 12.9 | | 13.1 | 13.1, 13.0, 12.10 | -## Configure file exports as an import source **(FREE SELF)** +### Configure file exports as an import source **(FREE SELF)** Before you can migrate projects on a self-managed GitLab instance using file exports, GitLab administrators must: @@ -92,7 +96,7 @@ To enable file exports as an import source for the destination instance: 1. Scroll to **Import sources**. 1. Select the **GitLab export** checkbox. -## Between CE and EE +### Between CE and EE You can export projects from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) and vice versa, assuming [compatibility](#compatibility) is met. @@ -101,7 +105,7 @@ If you're exporting a project from the Enterprise Edition to the Community Editi data that is retained only in the Enterprise Edition. For more information, see [downgrading from EE to CE](../../../index.md). -## Export a project and its data +### Export a project and its data Before you can import a project, you must export it. @@ -124,7 +128,7 @@ To export a project and its data, follow these steps: The export is generated in your configured `shared_path`, a temporary shared directory, and then moved to your configured `uploads_directory`. Every 24 hours, a worker deletes these export files. -### Items that are exported +#### Items that are exported Exported project items depend on the version of GitLab you use. To determine if a specific project item is exported: @@ -166,7 +170,7 @@ For a quick overview, items that are exported include: - Project and inherited group members, as long as the user has the Maintainer role in the exported project's group or is an administrator -### Items that are not exported +#### Items that are not exported Items that are **not** exported include: @@ -189,7 +193,7 @@ Items that are **not** exported include: Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../../group/custom_project_templates.md) and [instance](../../../administration/custom_project_templates.md) levels. Therefore, the list of exported items is the same. -## Import a project and its data +### Import a project and its data > Default maximum import file size [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. Administrators of self-managed instances can [set maximum import file size](#set-maximum-import-file-size). On GitLab.com, the value is [set to 5 GB](../../gitlab_com/index.md#account-and-limit-settings). @@ -199,7 +203,7 @@ WARNING: Only import projects from sources you trust. If you import a project from an untrusted source, it may be possible for an attacker to steal your sensitive data. -### Prerequisites +#### Prerequisites > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. @@ -209,7 +213,7 @@ may be possible for an attacker to steal your sensitive data. - Review [compatibility](#compatibility) for any issues. - At least the Maintainer role on the destination group to migrate to. -### Import a project +#### Import a project To import a project: @@ -222,7 +226,7 @@ To import a project: To get the status of an import, you can query it through the [API](../../../api/project_import_export.md#import-status). As described in the API documentation, the query may return an import error or exceptions. -### Changes to imported items +#### Changes to imported items Exported items are imported with the following changes: @@ -235,11 +239,11 @@ Exported items are imported with the following changes: Deploy keys aren't imported. To use deploy keys, you must enable them in your imported project and update protected branches. -### Import large projects **(FREE SELF)** +#### Import large projects **(FREE SELF)** If you have a larger project, consider [using a Rake task](../../../administration/raketasks/project_import_export.md#import-large-projects). -## Set maximum import file size **(FREE SELF)** +### Set maximum import file size **(FREE SELF)** Administrators can set the maximum import file size one of two ways: @@ -248,7 +252,7 @@ Administrators can set the maximum import file size one of two ways: The default is `0` (unlimited). -## Rate limits +### Rate limits To help avoid abuse, by default, users are rate limited to: @@ -258,11 +262,161 @@ To help avoid abuse, by default, users are rate limited to: | Download export | 1 download per group per minute | | Import | 6 projects per minute | +## Migrate groups by uploading an export file (deprecated) + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by +[migrating groups by direct transfer](../../group/import/index.md). However, this feature is still recommended for migrating groups between +offline systems. To follow progress on an alternative solution for [offline environments](../../application_security/offline_deployments/index.md), see +[the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). + +Prerequisites: + +- Owner role on the group to migrate. + +Using file exports, you can: + +- Export any group to a file and upload that file to another GitLab instance or to another location on the same instance. +- Use either the GitLab UI or the [API](../../../api/group_import_export.md). +- Migrate groups one by one, then export and import each project for the groups one by one. + +GitLab maps user contributions correctly when an admin access token is used to perform the import. GitLab does not map +user contributions correctly when you are importing from a self-managed instance to GitLab.com. Correct mapping of user +contributions when importing from a self-managed instance to GitLab.com can be preserved with paid involvement of +Professional Services team. + +Note the following: + +- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. +- To preserve group-level relationships from imported projects, export and import groups first so that projects can + be imported into the desired group structure. +- Imported groups are given a `private` visibility level, unless imported into a parent group. +- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. +- You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) + and vice versa. The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're + exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, + see [downgrading from EE to CE](../../../index.md). + +### Compatibility + +> Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/383682) in GitLab 15.8. + +Group file exports are in NDJSON format. + +You can import group file exports that were exported from a version of GitLab up to two +[minor](../../../policy/maintenance.md#versioning) versions behind, which is similar to our process for +[security releases](../../../policy/maintenance.md#security-releases). + +For example: + +| Destination version | Compatible source versions | +|:--------------------|:---------------------------| +| 13.0 | 13.0, 12.10, 12.9 | +| 13.1 | 13.1, 13.0, 12.10 | + +### Exported contents + +The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) +file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch +for your version of GitLab to check which items can be imported to the destination GitLab instance. For example, +[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). + +Group items that are exported include: + +- Milestones +- Group Labels (_without_ associated label priorities) +- Boards and Board Lists +- Badges +- Subgroups (including all the aforementioned data) +- Epics + - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) +- Events +- [Wikis](../../project/wiki/group.md) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) +- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) + +Items that are **not** exported include: + +- Projects +- Runner tokens +- SAML discovery tokens + +### Preparation + +- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make + sure these users exist before importing the desired groups. +- Users must set a public email in the source GitLab instance that matches their confirmed primary email in the destination GitLab instance. Most users receive an email asking them to confirm their email address. + +### Enable export for a group + +Prerequisites: + +- You must have the Owner role for the group. + +To enable import and export for a group: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Visibility and access controls**. +1. In the **Import sources** section, select the checkboxes for the sources you want. + +### Export a group + +Prerequisites: + +- You must have the Owner role for the group. + +To export the contents of a group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > General**. +1. In the **Advanced** section, select **Export group**. +1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) + in a compressed tar archive, with contents in NDJSON format. +1. Alternatively, you can download the export from the UI: + + 1. Return to your group's **Settings > General** page. + 1. In the **Advanced** section, select **Download export**. + You can also generate a new file by selecting **Regenerate export**. + +You can also export a group [using the API](../../../api/group_import_export.md). + +### Import the group + +1. On the left sidebar, at the top, select **Create new** (**{plus}**) and **New subgroup**. +1. Select the **import an existing group** link. +1. Enter your group name. +1. Accept or modify the associated group URL. +1. Select **Choose file...**. +1. Select the file that you exported in the [Export a group](#export-a-group) section. +1. To begin importing, select **Import**. + +Your newly imported group page appears after the operation completes. + +NOTE: +The maximum import file size can be set by the administrator, default is `0` (unlimited). +As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the +[Application settings API](../../../api/settings.md#change-application-settings) or the +[Admin Area](../../../administration/settings/account_and_limit_settings.md). +Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. + +### Rate limits + +To help avoid abuse, by default, users are rate limited to: + +| Request Type | Limit | +|-----------------|-------| +| Export | 6 groups per minute | +| Download export | 1 download per group per minute | +| Import | 6 groups per minute | + ## Related topics - [Project import and export API](../../../api/project_import_export.md) - [Project import and export administration Rake tasks](../../../administration/raketasks/project_import_export.md) - [Migrating GitLab groups](../../group/import/index.md) - [Group import and export API](../../../api/group_import_export.md) -- [Migrate groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). -- [Migrate groups by using file exports](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). +- [Migrate groups by direct transfer](../../group/import/index.md). diff --git a/doc/user/project/settings/project_features_permissions.md b/doc/user/project/settings/project_features_permissions.md index 27c0668079c..c1fe9045ea5 100644 --- a/doc/user/project/settings/project_features_permissions.md +++ b/doc/user/project/settings/project_features_permissions.md @@ -173,7 +173,7 @@ To subscribe to a topic: - From the **Explore topics** page: - 1. On the left sidebar, expand the top-most chevron ({**chevron-down**}). + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). 1. Select **Explore**. 1. Select **Topics**. 1. Select the topic you want to subscribe to. diff --git a/doc/user/project/use_project_as_go_package.md b/doc/user/project/use_project_as_go_package.md index 54e9eac7756..bf11cd784cb 100644 --- a/doc/user/project/use_project_as_go_package.md +++ b/doc/user/project/use_project_as_go_package.md @@ -10,7 +10,7 @@ Prerequisites: - Contact your administrator to enable the [GitLab Go Proxy](../packages/go_proxy/index.md). - To use a private project in a subgroup as a Go package, you must [authenticate Go requests](#authenticate-go-requests-to-private-projects). Go requests that are not authenticated cause -`go get` to fail. You don't need to authenticate Go requests for projects that are not in subgroups. + `go get` to fail. You don't need to authenticate Go requests for projects that are not in subgroups. To use a project as a Go package, use the `go get` and `godoc.org` discovery requests. You can use the meta tags: diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 49efd463334..4aaf7f27229 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -17,71 +17,65 @@ The Web IDE is an advanced editor with commit staging. You can use the Web IDE to make changes to multiple files directly from the GitLab UI. For a more basic implementation, see [Web Editor](../repository/web_editor.md). -To pair the Web IDE with a remote development environment, see [remote development](../remote_development/index.md). +To pair the Web IDE with a remote development environment, see [Remote development](../remote_development/index.md). -## Use the Web IDE +## Open the Web IDE -To open the Web IDE from the GitLab UI: +To open the Web IDE: 1. On the left sidebar, select **Search or go to** and find your project. 1. Use the <kbd>.</kbd> keyboard shortcut. -You can also open the Web IDE from: +### From a file or directory -- A file -- The repository file list -- A merge request +To open the Web IDE from a file or directory: -### From a file or the repository file list - -To open the Web IDE from a file or the repository file list: - -- In the upper right, select **Edit > Open in Web IDE**. +1. On the left sidebar, select **Search or go to** and find your project. +1. Go to your file or directory. +1. Select **Edit > Open in Web IDE**. ### From a merge request To open the Web IDE from a merge request: +1. On the left sidebar, select **Search or go to** and find your project. 1. Go to your merge request. -1. In the upper-right corner, select **Code > Open in Web IDE**. +1. In the upper right, select **Code > Open in Web IDE**. -The Web IDE opens new and modified files in separate tabs and displays changes side by side with the original source. -To optimize loading time, only the top 10 files (by number of lines changed) are opened automatically. +The Web IDE opens new and modified files in separate tabs and displays changes side by side. +To reduce load time, only 10 files with the most lines changed are opened automatically. -In the file tree, any new or modified file in the merge request is indicated by an icon next to the filename. -To view changes to a file, right-click the filename and select **Compare with merge request base**. +On the left **Explorer** sidebar, any new or modified file is indicated +by the merge request icon (**{merge-request}**) next to the file name. +To view changes to a file, right-click the file and select **Compare with merge request base**. -## Open a file in the Web IDE +## Open a file -To open any file by its name: +To open a file by name in the Web IDE: 1. Press <kbd>Command</kbd>+<kbd>P</kbd>. -1. Enter the name of your file. +1. In the search box, enter the file name. -## Search across files +## Search open files -You can use the Web IDE to search all files in the opened folder. - -To search across files: +To search across open files in the Web IDE: 1. Press <kbd>Shift</kbd>+<kbd>Command</kbd>+<kbd>F</kbd>. -1. Enter your search term. - -In the Web IDE, only partial results from opened files are displayed. +1. In the search box, enter your search term. -## View a list of changed files +## View a list of modified files -To view a list of files you changed in the Web IDE: +To view a list of files you modified in the Web IDE: -- On the activity bar on the left, select **Source Control**, - or press <kbd>Control</kbd>+<kbd>Shift</kbd>+<kbd>G</kbd>. +- On the left activity bar, select **Source Control**, or + press <kbd>Control</kbd>+<kbd>Shift</kbd>+<kbd>G</kbd>. Your `CHANGES`, `STAGED CHANGES`, and `MERGE CHANGES` are displayed. For more information, see the [VS Code documentation](https://code.visualstudio.com/docs/sourcecontrol/overview#_commit). ## Restore uncommitted changes -You don't have to manually save any file you modify in the Web IDE. +You do not have to manually save any file you modify in the Web IDE. Modified files are automatically staged and can be [committed](#commit-changes). Uncommitted changes are saved in your browser's local storage and persist even if you close the browser tab or refresh the Web IDE. @@ -93,68 +87,67 @@ To restore uncommitted changes in the Web IDE: 1. In the search box, enter `Local History: Find Entry to Restore`. 1. Select the file that contains the uncommitted changes. -## Upload a new file +## Upload a file + +To upload a file in the Web IDE: + +1. On the left activity bar, select **Explorer**, or + press <kbd>Shift</kbd>+<kbd>Command</kbd>+<kbd>E</kbd>. +1. Go to the directory where you want to upload the file. + To create a new directory: -To upload a new file in the Web IDE: + - On the left **Explorer** sidebar, in the upper right, + select **New Folder** (**{folder-new}**). -1. On the activity bar on the left, select **Explorer** and go to the directory where you want to upload the file. -1. Optional. For a new directory, go to the path where you want to have the directory and do one of the following: - - Right-click the path, and select **New Folder...**. You can create a nested path with `/` (for example, `parentdir/subdir1/subdir2`). - - In the upper right of the **Explorer** panel, select **New Folder...** (**{folder-new}**). -1. Enter the name of the new directory, and press <kbd>Enter</kbd>. -1. Right-click the path, and select **Upload...**. -1. Select the file you want to upload, then select **Open**. You can upload multiple files at once. +1. Right-click the directory and select **Upload**. +1. Select the file you want to upload. -The new file is uploaded and automatically added to the repository. +You can upload multiple files at once. +The files are uploaded and automatically added to the repository. ## Switch branches -The Web IDE uses the currently selected branch by default. +The Web IDE uses the current branch by default. To switch branches in the Web IDE: -1. On the status bar, in the lower-left corner, select the current branch name. -1. In the search box, start typing the branch name. -1. From the dropdown list, select the branch. +1. On the bottom status bar, on the left, select the current branch name. +1. Enter or select an existing branch. ## Create a branch To create a branch from the current branch in the Web IDE: -1. On the status bar, in the lower-left corner, select the current branch name. -1. From the dropdown list, select **Create new branch...**. -1. Enter the branch name. -1. Press <kbd>Enter</kbd>. +1. On the bottom status bar, on the left, select the current branch name. +1. From the dropdown list, select **Create new branch**. +1. Enter the new branch name. -If you don't have write access to the repository, **Create new branch...** is not visible. +If you do not have write access to the repository, **Create new branch** is not visible. ## Commit changes To commit changes in the Web IDE: -1. On the activity bar on the left, select **Source Control**, - or press <kbd>Control</kbd>+<kbd>Shift</kbd>+<kbd>G</kbd>. +1. On the left activity bar, select **Source Control**, or + press <kbd>Control</kbd>+<kbd>Shift</kbd>+<kbd>G</kbd>. 1. Enter your commit message. -1. Select **Commit & Push**. -1. Commit to the current branch, or [create a new branch](#create-a-branch). +1. Commit to the current branch or [create a new branch](#create-a-branch). ## Create a merge request -To create a merge request in the Web IDE: +To create a [merge request](../merge_requests/index.md) in the Web IDE: 1. [Commit the changes](#commit-changes). -1. In the pop-up notification in the lower-right corner, select **Create Merge Request**. - A new window opens for you to create the [merge request](../merge_requests/index.md). +1. In the notification that appears in the lower right, select **Create MR**. -To access missed notifications, see [Access notifications](#access-notifications). +For more information, see [View missed notifications](#view-missed-notifications). ## Use the command palette -In the Web IDE, you can access many commands through the command palette. +You can use the command palette to access many commands. To open the command palette and run a command in the Web IDE: 1. Press <kbd>Shift</kbd>+<kbd>Command</kbd>+<kbd>P</kbd>. -1. In the search box, start typing the command name. -1. From the dropdown list, select the command. +1. Enter or select the command. ## Edit settings @@ -168,7 +161,8 @@ In the settings editor, you can search for the settings you want to modify. ## Edit keyboard shortcuts -You can use the keyboard shortcuts editor to view and modify the default keybindings for all available commands. +You can use the keyboard shortcuts editor to view and modify +the default keybindings for all available commands. To open the keyboard shortcuts editor in the Web IDE: - On the top menu bar, select **File > Preferences > Keyboard Shortcuts**, @@ -179,13 +173,15 @@ In the keyboard shortcuts editor, you can search for: - The keybindings you want to change - The commands you want to add or remove keybindings for -Keybindings are based on your keyboard layout. If you change your keyboard layout, existing keybindings are updated automatically. +Keybindings are based on your keyboard layout. +If you change your keyboard layout, existing keybindings are updated automatically. -## Change themes +## Change the color theme -You can choose between different themes for the Web IDE. The default theme for the Web IDE is **GitLab Dark**. +You can choose between different color themes for the Web IDE. +The default theme is **GitLab Dark**. -To change the Web IDE theme: +To change the color theme in the Web IDE: 1. On the top menu bar, select **File > Preferences > Theme > Color Theme**, or press <kbd>Command</kbd>+<kbd>K</kbd> then <kbd>Command</kbd>+<kbd>T</kbd>. @@ -194,12 +190,13 @@ To change the Web IDE theme: The active color theme is stored in the [user settings](#edit-settings). -## Access notifications +## View missed notifications -When you perform actions in the Web IDE, notifications appear in the lower-right corner. To access missed notifications: +When you perform actions in the Web IDE, notifications appear in the lower right. +To view any notification you might have missed: -1. On the status bar, in the lower-right corner, select the bell (**{notifications}**) for a list of notifications. -1. Select the notification you want to access. +1. On the bottom status bar, on the right, select the bell icon (**{notifications}**) for a list of notifications. +1. Select the notification you want to view. <!-- ## Privacy and data collection for extensions @@ -215,7 +212,7 @@ To protect your privacy and data: - Carefully review the permissions requested by an extension before you install the extension. - Keep your extensions up to date to ensure that any security or privacy vulnerabilities are addressed promptly. --> -## Interactive web terminals for the Web IDE **(BETA)** +## Interactive web terminals **(BETA)** WARNING: This feature is in [Beta](../../../policy/experiment-beta-support.md#beta) and subject to change without notice. @@ -228,21 +225,25 @@ When you set up a remote development server in the Web IDE, you can use interact You cannot use interactive web terminals to interact with a runner. However, you can use a terminal to install dependencies and compile and debug code. -For more information about configuring a workspace that supports interactive web terminals, see [remote development](../remote_development/index.md). +For more information, see [Remote development](../remote_development/index.md). ## Related topics -- [GitLab Duo Chat in the Web IDE](../../gitlab_duo_chat.md#web-ide) +- [GitLab Duo Chat in the Web IDE](../../gitlab_duo_chat.md#use-gitlab-duo-chat-in-the-web-ide) ## Troubleshooting When working with the Web IDE, you might encounter the following issues. -### Character offset in the Web IDE +### Character offset when typing -When you type in the Web IDE, you might get a four-character offset. To resolve the issue, do one of the following: +When you type in the Web IDE, you might get a four-character offset. +As a workaround: -- Add `"editor.disableMonospaceOptimizations": true` to your settings. -- Modify your `"editor.font"` setting. +1. On the top menu bar, select **File > Preferences > Settings**, + or press <kbd>Command</kbd>+<kbd>,</kbd>. +1. In the upper-right corner, select **Open Settings (JSON)**. +1. In the `settings.json` file, add `"editor.disableMonospaceOptimizations": true` + or modify the `"editor.fontFamily"` setting. For more information, see [VS Code issue 80170](https://github.com/microsoft/vscode/issues/80170). diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 64f6fa2c75e..59e949c5218 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -37,7 +37,7 @@ To access a group wiki: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9. Users with the Owner role in a group can -[import or export a group wiki](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) when they +[import or export a group wiki](../../project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) when they import or export a group. Content created in a group wiki is not deleted when an account is downgraded or a @@ -47,7 +47,7 @@ the wiki is exported. To access the group wiki data from the export file if the feature is no longer available, you have to: -1. Extract the [export file tarball](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) +1. Extract the [export file tarball](../../project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) with this command, replacing `FILENAME` with your file's name: `tar -xvzf FILENAME.tar.gz` 1. Browse to the `repositories` directory. This directory contains a diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index f4946230360..07c5ce73470 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -94,8 +94,12 @@ Users with at least the Developer role can create new wiki pages: Wikis are based on Git repositories, so you can clone them locally and edit them like you would do with every other Git repository. To clone a wiki repository -locally, select **Clone repository** from the right-hand sidebar of any wiki page, -and follow the on-screen instructions. +locally: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Plan > Wiki**. +1. On the right sidebar, select **Clone repository**. +1. Follow the on-screen instructions. Files you add to your wiki locally must use one of the following supported extensions, depending on the markup language you wish to use. @@ -155,7 +159,9 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). ## Delete a wiki page -You need at least the Developer role to delete a wiki page: +Prerequisites: + +- You must have at least the Developer role. 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. @@ -166,7 +172,9 @@ You need at least the Developer role to delete a wiki page: ## Move a wiki page -You need at least the Developer role to move a wiki page: +Prerequisites: + +- You must have at least the Developer role. 1. On the left sidebar, select **Search or go to** and find your project or group. 1. Select **Plan > Wiki**. @@ -245,8 +253,11 @@ Commits to wikis are not counted in [repository analytics](../../analytics/repos > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23109) in GitLab 13.8, the sidebar can be customized by selecting the **Edit sidebar** button. -You need at least the Developer role to customize the wiki -navigation sidebar. This process creates a wiki page named `_sidebar` which fully +Prerequisites: + +- You must have at least the Developer role. + +This process creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation: 1. On the left sidebar, select **Search or go to** and find your project or group. @@ -312,7 +323,7 @@ To disable a project's internal wiki: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. -1. Scroll down to find **Wiki** and toggle it off (in gray). +1. Scroll down to find and turn off the **Wiki** toggle (in gray). 1. Select **Save changes**. The internal wiki is now disabled, and users and project members: diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 1c60f3bebf3..7d8305519e4 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -61,7 +61,7 @@ Prerequisites: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. In the **Project name** text box, enter your project name. See the [limitations on project names](../../user/reserved_names.md). -1. In the **Project description** text box, enter your project description. The description is limited to 500 characters. +1. In the **Project description** text box, enter your project description. The description is limited to 2,000 characters. 1. Under **Project avatar**, to change your project avatar, select **Choose file**. ## Star a project @@ -121,7 +121,7 @@ You can [view projects that are pending deletion](#view-projects-pending-deletio and use the Rails console to [find projects that are pending deletion](#find-projects-that-are-pending-deletion). -### Delete a project immediately +### Delete a project immediately **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191367) in GitLab 14.1. > - Option to delete projects immediately from the Admin Area and as a group setting removed [on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/393622) and [on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119606) in GitLab 16.0. @@ -144,7 +144,7 @@ To immediately delete a project marked for deletion: 1. In the **Delete this project** section, select **Delete project**. 1. On the confirmation dialog, enter the project name and select **Yes, delete project**. -### View projects pending deletion +### View projects pending deletion **(PREMIUM ALL)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3 for Administrators. > - [Tab renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/347468) from **Deleted projects** in GitLab 14.6. @@ -309,7 +309,7 @@ Prerequisites: 1. [Create a group](../group/index.md#create-a-group) to track membership of your project. 1. [Set up LDAP synchronization](../../administration/auth/ldap/ldap_synchronization.md) for that group. 1. To use LDAP groups to manage access to a project, -[add the LDAP-synchronized group as a member](../group/manage.md) to the project. + [add the LDAP-synchronized group as a member](../group/manage.md) to the project. ## Troubleshooting diff --git a/doc/user/public_access.md b/doc/user/public_access.md index b7ee354ed9a..826f9548982 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -8,11 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w Projects and groups in GitLab can be private, internal, or public. -The visibility level of the group or project has no influence on whether members within the group or project can see each other. -A group or project is an object to allow collaborative work. This is only possible if all members know about each other. +The visibility level of the project or group does not affect whether members of the project or group can see each other. +Projects and groups are intended for collaborative work. This work is only possible if all members know about each other. -Group or project members can see all members of the group or project they belong to. -Group or project owners can see the origin of membership (the original group or project) of all members. +Project or group members can see all members of the project or group they belong to. +Project or group members can see the origin of membership (the original project or group) of all members for the projects and groups they have access to. ## Private projects and groups @@ -38,15 +38,9 @@ Only internal members can view internal content. Internal groups can have internal or private subgroups. -NOTE: -From July 2019, the `Internal` visibility setting is disabled for new projects, groups, -and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` -visibility setting keep this setting. For more information, see -[issue 12388](https://gitlab.com/gitlab-org/gitlab/-/issues/12388). - ## Public projects and groups -For public projects, **users who are not authenticated**, including users with the Guest role, can: +For public projects, **unauthenticated users**, including users with the Guest role, can: - Clone the project. - View the public access directory (`/public`). @@ -56,7 +50,7 @@ Public groups can have public, internal, or private subgroups. NOTE: If an administrator restricts the [**Public** visibility level](../administration/settings/visibility_and_access_controls.md#restrict-visibility-levels), -then `/public` is visible only to authenticated users. +then the public access directory (`/public`) is visible only to authenticated users. ## Change project visibility @@ -85,7 +79,7 @@ Prerequisites: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Visibility, project features, permissions**. -1. To enable or disable a feature, turn on or off the feature toggle. +1. To enable or disable a feature, turn on or turn off the feature toggle. 1. Select **Save changes**. ## Change group visibility @@ -95,9 +89,9 @@ You can change the visibility of all projects in a group. Prerequisites: - You must have the Owner role for a group. -- Subgroups and projects must already have visibility settings that are at least as +- Projects and subgroups must already have visibility settings that are at least as restrictive as the new setting of the parent group. For example, you cannot set a group - to private if a subgroup or project in that group is public. + to private if a project or subgroup in that group is public. 1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > General**. diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 3b715fb13da..757231ffb80 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -30,10 +30,10 @@ You can use advanced search in: ## Enable advanced search -- On GitLab.com, advanced search is enabled for groups with paid subscriptions. -- For self-managed GitLab instances, an administrator must +- For [GitLab SaaS](../../subscriptions/gitlab_com/index.md) and [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md), + advanced search is enabled in paid subscriptions. +- For [GitLab self-managed](../../subscriptions/self_managed/index.md), an administrator must [enable advanced search](../../integration/advanced_search/elasticsearch.md#enable-advanced-search). -- For GitLab Dedicated, advanced search is enabled. ## Syntax diff --git a/doc/user/storage_management_automation.md b/doc/user/storage_management_automation.md index f25ae8ba92d..e04a7f1bdee 100644 --- a/doc/user/storage_management_automation.md +++ b/doc/user/storage_management_automation.md @@ -78,7 +78,7 @@ For more information, see the [GitLab CLI endpoint documentation](../editor_exte The storage management and cleanup automation methods described in this page use: - The [`python-gitlab`](https://python-gitlab.readthedocs.io/en/stable/) library, which provides -a feature-rich programming interface. + a feature-rich programming interface. - The `get_all_projects_top_level_namespace_storage_analysis_cleanup_example.py` script in the [GitLab API with Python](https://gitlab.com/gitlab-de/use-cases/gitlab-api/gitlab-api-python/) project. For more information about use cases for the `python-gitlab` library, diff --git a/doc/user/tasks.md b/doc/user/tasks.md index 1ec211dcab3..3f26329485b 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -366,7 +366,7 @@ To copy the task reference to your clipboard: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. 1. In the issue description, in the **Tasks** section, select your task. -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy Reference**. You can now paste the reference into another description or comment. @@ -386,7 +386,7 @@ To copy the task's email address: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues**, then select your issue to view it. -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy task email address**. +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**), then select **Copy task email address**. ## Set an issue as a parent @@ -442,7 +442,7 @@ Check that box and select **Create task**. To change the confidentiality of an existing task: 1. [Open the task](#view-tasks). -1. In the top right corner, select the vertical ellipsis (**{ellipsis_v}**). +1. In the upper-right corner, select the vertical ellipsis (**{ellipsis_v}**). 1. Select **Turn on confidentiality**. ### Who can see confidential tasks @@ -497,6 +497,7 @@ or assignees, on the right. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/416558) in GitLab 16.5 [with a flag](../administration/feature_flags.md) named `linked_work_items`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/139394) in GitLab 16.7. +> - Adding related items by entering their URLs and IDs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/427594) in GitLab 16.8. FLAG: On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../administration/feature_flags.md) named `linked_work_items`. @@ -524,7 +525,7 @@ To link an item to a task: - **relates to** - **blocks** - **is blocked by** -1. Enter the search text of the item. +1. Enter the search text of the item, URL, or its reference ID. 1. When you have added all the items to be linked, select **Add** below the search box. When you have finished adding all linked items, you can see diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 973ad9d0b07..2dc5c1ef819 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -163,11 +163,11 @@ Storage limits are included in GitLab subscription terms but do not apply. At le GitLab will notify you of namespaces that exceed, or are close to exceeding, the storage limit. - In the command-line interface, a notification displays after each `git push` -action when your namespace has reached between 95% and 100% of your namespace storage quota. + action when your namespace has reached between 95% and 100% of your namespace storage quota. - In the GitLab UI, a notification displays when your namespace has reached between -75% and 100% of your namespace storage quota. + 75% and 100% of your namespace storage quota. - GitLab sends an email to members with the Owner role to notify them when namespace -storage usage is at 70%, 85%, 95%, and 100%. + storage usage is at 70%, 85%, 95%, and 100%. ## Manage storage usage diff --git a/doc/user/version.md b/doc/user/version.md index d39c0394610..2dcaf16d53b 100644 --- a/doc/user/version.md +++ b/doc/user/version.md @@ -1,6 +1,7 @@ --- stage: none group: none +description: Version information. info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/workspace/gitlab_agent_configuration.md b/doc/user/workspace/gitlab_agent_configuration.md index 0e35c72c5ef..bef935f2426 100644 --- a/doc/user/workspace/gitlab_agent_configuration.md +++ b/doc/user/workspace/gitlab_agent_configuration.md @@ -20,12 +20,14 @@ provided that the agent is properly configured for remote development. ## Remote development settings -| Setting | Description | -|-------------------------------------------------------|:---------------------------------------------------------------------| -| [`enabled`](#enabled) | Indicates whether remote development is enabled for the GitLab agent | -| [`dns_zone`](#dns_zone) | DNS zone where workspaces are available | -| [`gitlab_workspaces_proxy`](#gitlab_workspaces_proxy) | Namespace where [`gitlab-workspaces-proxy`](https://gitlab.com/gitlab-org/remote-development/gitlab-workspaces-proxy) is installed | -| [`network_policy`](#network_policy) | Firewall rules for workspaces | +| Setting | Description | +|-------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------| +| [`enabled`](#enabled) | Indicates whether remote development is enabled for the GitLab agent. | +| [`dns_zone`](#dns_zone) | DNS zone where workspaces are available. | +| [`gitlab_workspaces_proxy`](#gitlab_workspaces_proxy) | Namespace where [`gitlab-workspaces-proxy`](https://gitlab.com/gitlab-org/remote-development/gitlab-workspaces-proxy) is installed. | +| [`network_policy`](#network_policy) | Firewall rules for workspaces. | +| [`default_resources_per_workspace_container`](#default_resources_per_workspace_container) | Default requests and limits for CPU and memory per workspace container. | +| [`max_resources_per_workspace`](#max_resources_per_workspace) | Maximum requests and limits for CPU and memory per workspace. | NOTE: If a setting has an invalid value, it's not possible to update any setting until you fix that value. @@ -85,6 +87,7 @@ The default value is: ```yaml remote_development: network_policy: + enabled: true egress: - allow: "0.0.0.0/0" except: @@ -141,6 +144,64 @@ In this example, traffic from the workspace is allowed if: - The destination IP is any range except `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`. - The destination IP is `172.16.123.1/32`. +### `default_resources_per_workspace_container` + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11625) in GitLab 16.8. + +Use this setting to define the default [requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) +for CPU and memory per workspace container. +Any resources you define in your [devfile](index.md#devfile) override this setting. + +For `default_resources_per_workspace_container`, `requests` and `limits` are required. +For more information about possible CPU and memory values, see [Resource units in Kubernetes](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes). + +When you change any of these values, existing workspaces restart immediately for the changes to take effect. + +**Example configuration:** + +```yaml +remote_development: + default_resources_per_workspace_container: + requests: + cpu: "0.5" + memory: "512Mi" + limits: + cpu: "1" + memory: "1Gi" +``` + +### `max_resources_per_workspace` + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/11625) in GitLab 16.8. + +Use this setting to define the maximum [requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) +for CPU and memory per workspace. + +For `max_resources_per_workspace`, `requests` and `limits` are required. +For more information about possible CPU and memory values, see: + +- [Resource units in Kubernetes](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes) +- [Resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) + +When you change any of these values, existing workspaces restart immediately for the changes to take effect. +Workspaces fail when they exceed the values you set for `requests` and `limits`. + +**Example configuration:** + +```yaml +remote_development: + max_resources_per_workspace: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "2" + memory: "2Gi" +``` + +The maximum resources you define must include any resources required for init containers +to perform bootstrapping operations such as cloning the project repository. + ## Configuring user access with remote development You can configure the `user_access` module to access the connected Kubernetes cluster with your GitLab credentials. diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index d24102afcf9..5fa6108de6f 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -67,32 +67,37 @@ The devfile is used to automatically configure the development environment with This way, you can create consistent and reproducible development environments regardless of the machine or platform you use. -### Relevant schema properties - -GitLab only supports the `container` and `volume` components in [devfile 2.2.0](https://devfile.io/docs/2.2.0/devfile-schema). -Use the `container` component to define a container image as the execution environment for a devfile workspace. +### Validation rules + +- `schemaVersion` must be [`2.2.0`](https://devfile.io/docs/2.2.0/devfile-schema). +- The devfile must have at least one component. +- For `components`: + - Names must not start with `gl-`. + - Only [`container`](#container-component-type) and `volume` are supported. +- For `commands`, IDs must not start with `gl-`. +- For `events`: + - Names must not start with `gl-`. + - Only `preStart` is supported. +- `parent`, `projects`, and `starterProjects` are not supported. +- For `variables`, keys must not start with `gl-`, `gl_`, `GL-`, or `GL_`. + +### `container` component type + +Use the `container` component type to define a container image as the execution environment for a workspace. You can specify the base image, dependencies, and other settings. -Only these properties are relevant to the GitLab implementation of the `container` component: - -| Properties | Definition | -|----------------| ----------------------------------------------------------------------------------| -| `image` | Name of the container image to use for the workspace. | -| `memoryRequest`| Minimum amount of memory the container can use. | -| `memoryLimit` | Maximum amount of memory the container can use. | -| `cpuRequest` | Minimum amount of CPU the container can use. | -| `cpuLimit` | Maximum amount of CPU the container can use. | -| `env` | Environment variables to use in the container. | -| `endpoints` | Port mappings to expose from the container. | -| `volumeMounts` | Storage volume to mount in the container. | - -### Using variables in a devfile - -You can define variables to use in your devfile. -The `variables` object is a map of name-value pairs that you can use for string replacement in the devfile. - -Variables cannot have names that start with `gl-`, `gl_`, `GL-`, or `GL_`. -For more information about how and where to use variables, see the [devfile documentation](https://devfile.io/docs/2.2.0/defining-variables). +The `container` component type supports the following schema properties only: + +| Property | Description | +|----------------| -------------------------------------------------------------------------------------------------------------------------------| +| `image` | Name of the container image to use for the workspace. | +| `memoryRequest`| Minimum amount of memory the container can use. | +| `memoryLimit` | Maximum amount of memory the container can use. | +| `cpuRequest` | Minimum amount of CPU the container can use. | +| `cpuLimit` | Maximum amount of CPU the container can use. | +| `env` | Environment variables to use in the container. Names must not start with `gl-`. | +| `endpoints` | Port mappings to expose from the container. Names must not start with `gl-`. | +| `volumeMounts` | Storage volume to mount in the container. | ### Example configurations |
