diff options
Diffstat (limited to 'doc')
867 files changed, 19250 insertions, 11817 deletions
diff --git a/doc/.vale/gitlab/CodeblockFences.yml b/doc/.vale/gitlab/CodeblockFences.yml index 9d5efe7f60a..27159f7e72e 100644 --- a/doc/.vale/gitlab/CodeblockFences.yml +++ b/doc/.vale/gitlab/CodeblockFences.yml @@ -5,9 +5,9 @@ # # For a list of all options, see https://vale.sh/docs/topics/styles/ extends: existence -message: "Instead of '%s' for the code block, use yaml, ruby, plaintext, markdown, javascript, shell, golang, python, dockerfile, or typescript." +message: "Instead of '%s' for the code block, use yaml, ruby, plaintext, markdown, javascript, shell, go, python, dockerfile, or typescript." link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#code-blocks level: error scope: raw raw: - - '\`\`\`(yml|rb|text|md|bash|sh\n|js\n|go\n|py\n|docker\n|ts)' + - '\`\`\`(yml|rb|text|md|bash|sh\n|js\n|golang\n|py\n|docker\n|ts)' diff --git a/doc/.vale/gitlab/HeadingDepth.yml b/doc/.vale/gitlab/HeadingDepth.yml index 5bbe667481c..f29ec0e4ac7 100644 --- a/doc/.vale/gitlab/HeadingDepth.yml +++ b/doc/.vale/gitlab/HeadingDepth.yml @@ -6,7 +6,7 @@ # For a list of all options, see https://vale.sh/docs/topics/styles/ extends: existence message: "Refactor the section or page to avoid headings greater than H5." -link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#headings-in-markdown +link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#heading-levels-in-markdown level: suggestion scope: raw raw: diff --git a/doc/.vale/gitlab/Substitutions.yml b/doc/.vale/gitlab/Substitutions.yml index 675abc6ef6d..225e4e74880 100644 --- a/doc/.vale/gitlab/Substitutions.yml +++ b/doc/.vale/gitlab/Substitutions.yml @@ -23,6 +23,7 @@ swap: params: parameters pg: PostgreSQL 'postgres$': PostgreSQL + golang: Go raketask: Rake task raketasks: Rake tasks rspec: RSpec diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index a2a23c03aa3..83e187fe1b5 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -223,6 +223,8 @@ cybersecurity CycloneDX Dangerfile DAST +Database Lab Engine +Database Lab Databricks Datadog datasource @@ -495,6 +497,7 @@ Kibana Kinesis Klar Knative +KPIs Kramdown Kroki kubeconfig @@ -615,6 +618,7 @@ Nurtch NVMe nyc OAuth +OCP Octokit offboarded offboarding @@ -623,6 +627,7 @@ OIDs OKRs OKRs Okta +OLM OmniAuth onboarding OpenID @@ -666,6 +671,7 @@ Pipfile Pipfiles Piwik plaintext +podman Poedit polyfill polyfills @@ -833,8 +839,8 @@ sanitization SBOMs sbt SBT -scalers scalar's +scalers scatterplot scatterplots schedulable @@ -1182,7 +1188,7 @@ ZAProxy Zeitwerk Zendesk ZenTao +Zoekt zsh Zstandard Zuora -Zoekt diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 92ccbe8b1a6..bbf4c0699ca 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -41,7 +41,7 @@ Streaming destinations receive **all** audit event data, which could include sen Users with the Owner role for a group can add streaming destinations for it: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **Add streaming destination** to show the section for adding destinations. 1. Enter the destination URL to add. @@ -119,7 +119,7 @@ Users with the Owner role for a group can list streaming destinations. To list the streaming destinations: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select **Streams** tab. 1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. @@ -164,7 +164,7 @@ Users with the Owner role for a group can update streaming destinations. To update a streaming destinations custom HTTP headers: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select **Streams** tab. 1. To the right of the item, select **Edit** (**{pencil}**). 1. Locate the **Custom HTTP headers** table. @@ -218,14 +218,14 @@ When the last destination is successfully deleted, streaming is disabled for the To delete a streaming destination: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select the **Streams** tab. 1. To the right of the item, select **Delete** (**{remove}**). To delete only the custom HTTP headers for a streaming destination: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select the **Streams** tab. 1. To the right of the item, **Edit** (**{pencil}**). 1. Locate the **Custom HTTP headers** table. @@ -282,7 +282,7 @@ the destination's value when [listing streaming destinations](#list-streaming-de Users with the Owner role for a group can list streaming destinations and see the verification tokens: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the left sidebar, select **Security and Compliance > Audit events**. 1. On the main area, select the **Streams**. 1. View the verification token on the right side of each item. @@ -365,6 +365,50 @@ Streamed audit events have a predictable schema in the body of the response. | `target_id` | ID of the audit event's target | | | `target_type` | String representation of the target's type | | +### JSON payload schema + +```json +{ + "properties": { + "id": { + "type": "string" + }, + "author_id": { + "type": "integer" + }, + "author_name": { + "type": "string" + }, + "details": {}, + "ip_address": { + "type": "string" + }, + "entity_id": { + "type": "integer" + }, + "entity_path": { + "type": "string" + }, + "entity_type": { + "type": "string" + }, + "event_type": { + "type": "string" + }, + "target_id": { + "type": "integer" + }, + "target_type": { + "type": "string" + }, + "target_details": { + "type": "string" + }, + }, + "type": "object" +} +``` + ## Audit event streaming on Git operations > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index c51b5fbb431..47833224c0a 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -49,7 +49,7 @@ You can view audit events scoped to a group or project. To view a group's audit events: 1. Go to the group. -1. On the left sidebar, select **Security & Compliance > Audit Events**. +1. On the left sidebar, select **Security and Compliance > Audit Events**. Group events do not include project audit events. Group events can also be accessed using the [Group Audit Events API](../api/audit_events.md#group-audit-events). Group event queries are limited to a maximum of 30 @@ -120,7 +120,7 @@ Successful sign-in events are the only audit events available at all tiers. To s 1. Select your avatar. 1. Select **Edit profile > Authentication log**. -After upgrading to a paid tier, you can see also see successful sign-in events on audit event pages. +After upgrading to a paid tier, you can also see successful sign-in events on audit event pages. ## Filter audit events @@ -314,6 +314,15 @@ The following actions on projects generate project audit events: - An environment is protected or unprotected. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216164) in GitLab 15.8. +### GitLab agent for Kubernetes events + +The following actions on projects generate agent audit events: + +- A cluster agent token is created. + Introduced in GitLab 15.9 +- A cluster agent token is revoked. + Introduced in GitLab 15.9 + ### Instance events **(PREMIUM SELF)** The following user actions on a GitLab instance generate instance audit events: diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md deleted file mode 100644 index a32d2a2cf94..00000000000 --- a/doc/administration/auth/authentiq.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Manage -group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-22' -redirect_to: '../../integration/omniauth.md' ---- - -# Authentiq OmniAuth Provider (removed) **(FREE SELF)** - -This feature was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/389452) in 15.9. -Use another [OmniAuth provider](../../integration/omniauth.md) instead. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 0c7bd33c2c1..1f7f096eb2f 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -270,7 +270,7 @@ These LDAP sync configuration settings are available: | Setting | Description | Required | Examples | |-------------------|-------------|----------|----------| -| `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` | +| `group_base` | Base used to search for groups. | **{dotted-circle}** No (required when `external_groups` is configured) | `'ou=groups,dc=gitlab,dc=example'` | | `admin_group` | The CN of a group containing GitLab administrators. Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | | `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 7e21d3c6655..9c925e0a40e 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -170,10 +170,10 @@ We have a workaround, based on toggling the access level of affected users: 1. As an administrator, on the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. -1. In the upper right, select **Edit**. +1. In the upper-right corner, select **Edit**. 1. Change the user's access level from `Regular` to `Administrator` (or vice versa). 1. At the bottom of the page, select **Save changes**. -1. In the upper right, select **Edit** again. +1. In the upper-right corner, select **Edit** again. 1. Restore the user's original access level (`Regular` or `Administrator`) and select **Save changes** again. diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index cc300941470..59ed3a4153d 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -33,9 +33,6 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/ <!-- vale gitlab.Spelling = YES --> -The user is set to an `ldap_blocked` state in GitLab if the previous conditions -fail. This means the user cannot sign in or push or pull code. - The process also updates the following user information: - Name. Because of a [sync issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342598), `name` is not synchronized if @@ -44,6 +41,26 @@ The process also updates the following user information: - SSH public keys if `sync_ssh_keys` is set. - Kerberos identity if Kerberos is enabled. +### Blocked users + +A user is blocked if either the: + +- [Access check fails](#user-sync) and that user is set to an `ldap_blocked` state in GitLab. +- LDAP server is not available when that user signs in. + +If a user is blocked, that user cannot sign in or push or pull code. + +A blocked user is unblocked when they sign in with LDAP if all of the following are true: + +- All the access check conditions are true. +- The LDAP server is available when the user signs in. + +**All users** are blocked if the LDAP server is unavailable when an LDAP user synchronization is run. + +NOTE: +If all users are blocked due to the LDAP server not being available when an LDAP user synchronization is run, +a subsequent LDAP user synchronization does not automatically unblock those users. + ### Adjust LDAP user sync schedule By default, GitLab runs a worker once per day at 01:30 a.m. server time to @@ -387,6 +404,23 @@ To enable global group memberships lock: 1. Expand the **Visibility and access controls** section. 1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. +### Change LDAP group synchronization settings management + +By default, group members with the Owner role can manage [LDAP group synchronization settings](../../../user/group/access_and_permissions.md#manage-group-memberships-via-ldap). + +GitLab administrators can remove this permission from group Owners: + +1. [Configure LDAP](index.md#configure-ldap). +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility and access controls**. +1. Ensure the **Allow group owners to manage LDAP-related settings** checkbox is not checked. + +When **Allow group owners to manage LDAP-related settings** is disabled: + +- Group Owners cannot change LDAP synchronization settings for either top-level groups and subgroups. +- Instance administrators can manage LDAP group synchronization settings on all groups on an instance. + ### Adjust LDAP group sync schedule By default, GitLab runs a group sync process every hour, on the hour. @@ -411,7 +445,7 @@ sync to run once every two hours at the top of the hour. 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * *" ``` 1. Save the file and reconfigure GitLab: @@ -454,7 +488,7 @@ sync to run once every two hours at the top of the hour. gitlab: environment: GITLAB_OMNIBUS_CONFIG: | - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * *" ``` 1. Save the file and restart GitLab: diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index ea6922629d8..4edc00b0d62 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -561,6 +561,254 @@ Example installations from source configuration (file path: `config/gitlab.yml`) } ``` +## Configure users based on OIDC group membership **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209898) in GitLab 15.10. + +You can configure OIDC group membership to: + +- Require users to be members of a certain group. +- Assign users [external roles](../../user/admin_area/external_users.md), or as + administrators based on group membership. + +GitLab checks these groups on each sign in and updates user attributes as necessary. +This feature **does not** allow you to automatically add users to GitLab +[groups](../../user/group/index.md). + +### Required groups + +Your identity provider (IdP) must pass group information to GitLab in the OIDC response. To use this +response to require users to be members of a certain group, configure GitLab to identify: + +- Where to look for the groups in the OIDC response, using the `groups_attribute` setting. +- Which group membership is required to sign in, using the `required_groups` setting. + +If you do not set `required_groups` or leave the setting empty, any user authenticated by the IdP through OIDC can use GitLab. + +For Omnibus GitLab: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "openid_connect", + label: "Provider name", + args: { + name: "openid_connect", + scope: ["openid","profile","email"], + response_type: "code", + issuer: "<your_oidc_url>", + discovery: true, + client_auth_method: "query", + uid_field: "<uid_field>", + client_options: { + identifier: "<your_oidc_client_id>", + secret: "<your_oidc_client_secret>", + redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback", + gitlab: { + groups_attribute: "groups", + required_groups: ["Developer"] + } + } + } + } + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installation from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + omniauth: + providers: + - { name: 'openid_connect', + label: 'Provider name', + args: { + name: 'openid_connect', + scope: ['openid','profile','email'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback', + gitlab: { + groups_attribute: "groups", + required_groups: ["Developer"] + } + } + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + +### External groups + +Your IdP must pass group information to GitLab in the OIDC response. To use this +response to identify users as [external users](../../user/admin_area/external_users.md) +based on group membership, configure GitLab to identify: + +- Where to look for the groups in the OIDC response, using the `groups_attribute` setting. +- Which group memberships should identify a user as an + [external user](../../user/admin_area/external_users.md), using the + `external_groups` setting. + +For Omnibus GitLab: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "openid_connect", + label: "Provider name", + args: { + name: "openid_connect", + scope: ["openid","profile","email"], + response_type: "code", + issuer: "<your_oidc_url>", + discovery: true, + client_auth_method: "query", + uid_field: "<uid_field>", + client_options: { + identifier: "<your_oidc_client_id>", + secret: "<your_oidc_client_secret>", + redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback", + gitlab: { + groups_attribute: "groups", + external_groups: ["Freelancer"] + } + } + } + } + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installation from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + omniauth: + providers: + - { name: 'openid_connect', + label: 'Provider name', + args: { + name: 'openid_connect', + scope: ['openid','profile','email'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback', + gitlab: { + groups_attribute: "groups", + external_groups: ["Freelancer"] + } + } + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + +### Administrator groups + +Your IdP must pass group information to GitLab in the OIDC response. To use this +response to assign users as administrator based on group membership, configure GitLab to identify: + +- Where to look for the groups in the OIDC response, using the `groups_attribute` setting. +- Which group memberships grant the user administrator access, using the + `admin_groups` setting. + +For Omnibus GitLab: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "openid_connect", + label: "Provider name", + args: { + name: "openid_connect", + scope: ["openid","profile","email"], + response_type: "code", + issuer: "<your_oidc_url>", + discovery: true, + client_auth_method: "query", + uid_field: "<uid_field>", + client_options: { + identifier: "<your_oidc_client_id>", + secret: "<your_oidc_client_secret>", + redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback", + gitlab: { + groups_attribute: "groups", + admin_groups: ["Admin"] + } + } + } + } + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installation from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + omniauth: + providers: + - { name: 'openid_connect', + label: 'Provider name', + args: { + name: 'openid_connect', + scope: ['openid','profile','email'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback', + gitlab: { + groups_attribute: "groups", + admin_groups: ["Admin"] + } + } + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + ## Troubleshooting 1. Ensure `discovery` is set to `true`. If you set it to `false`, you must diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index a7f8f8e712b..7a289b6a500 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -50,6 +50,8 @@ To enable the agent server on multiple nodes: 1. For each agent server node, edit `/etc/gitlab/gitlab.rb`: ```ruby + gitlab_kas_external_url 'wss://kas.gitlab.example.com/' + gitlab_kas['enable'] = true gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>' gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>' @@ -65,17 +67,23 @@ To enable the agent server on multiple nodes: gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/' ``` - In this configuration: +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - - `gitlab_kas['private_api_listen_address']` is the address the agent server listens on. You can set it to `0.0.0.0` or an IP address reachable by other nodes in the cluster. - - `OWN_PRIVATE_API_URL` is the environment variable used by the KAS process for service discovery. You can set it to a hostname or IP address of the node you're configuring. The node must be reachable by other nodes in the cluster. - - `gitlab_kas['api_secret_key']` is the shared secret used for authentication between KAS and GitLab. This value must be Base64-encoded and exactly 32 bytes long. - - `gitlab_kas['private_api_secret_key']` is the shared secret used for authentication between different KAS instances. This value must be Base64-encoded and exactly 32 bytes long. - - `gitlab_rails['gitlab_kas_external_url']` is the user-facing URL for the in-cluster `agentk`. - - `gitlab_rails['gitlab_kas_internal_url']` is the internal URL the GitLab backend uses to communicate with KAS. - - `gitlab_rails['gitlab_kas_external_k8s_proxy_url']` is the user-facing URL for Kubernetes API proxying. +##### Agent server node settings -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +| Setting | Description | +|---------|-------------| +| `gitlab_kas['private_api_listen_address']` | The address the agent server listens on. Set to `0.0.0.0` or to an IP address reachable by other nodes in the cluster. | +| `gitlab_kas['api_secret_key']` | The shared secret used for authentication between KAS and GitLab. The value must be Base64-encoded and exactly 32 bytes long. | +| `gitlab_kas['private_api_secret_key']` | The shared secret used for authentication between different KAS instances. The value must be Base64-encoded and exactly 32 bytes long. | +| `OWN_PRIVATE_API_URL` | The environment variable used by KAS for service discovery. Set to the hostname or IP address of the node you're configuring. The node must be reachable by other nodes in the cluster. | +| `gitlab_kas_external_url` | The user-facing URL for the in-cluster `agentk`. Can be a fully qualified domain or subdomain, <sup>**1**</sup> or a GitLab external URL. <sup>**2**</sup> If blank, defaults to a GitLab external URL. | +| `gitlab_rails['gitlab_kas_external_url']` | The user-facing URL for the in-cluster `agentk`. If blank, defaults to the `gitlab_kas_external_url`. | +| `gitlab_rails['gitlab_kas_external_k8s_proxy_url']` | The user-facing URL for Kubernetes API proxying. If blank, defaults to a URL based on `gitlab_kas_external_url`. | +| `gitlab_rails['gitlab_kas_internal_url']` | The internal URL the GitLab backend uses to communicate with KAS. | + +1. For example, `wss://kas.gitlab.example.com/`. +1. For example, `wss://gitlab.example.com/-/kubernetes-agent/`. ### For GitLab Helm Chart @@ -105,6 +113,24 @@ For GitLab [Helm Chart](https://docs.gitlab.com/charts/) installations: For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). +## Kubernetes API proxy cookie + +> Introduced in GitLab 15.10 [with feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. + +KAS proxies Kubernetes API requests to the GitLab agent with either: + +- A [CI/CD job](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kubernetes_ci_access.md). +- [GitLab user credentials](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kubernetes_user_access.md). + +To authenticate with user credentials, Rails sets a cookie for the GitLab frontend. +This cookie is called `_gitlab_kas` and it contains an encrypted +session ID, like the [`_gitlab_session` cookie](../../user/profile/index.md#cookies-used-for-sign-in). +The `_gitlab_kas` cookie must be sent to the KAS proxy endpoint with every request +to authenticate and authorize the user. + ## Troubleshooting If you have issues while using the agent server for Kubernetes, view the diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index 22b8cc13637..1991d0b64cc 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -27,6 +27,7 @@ To request the creation of a new GitLab Dedicated environment for your organizat - Desired instance subdomain: The main domain for GitLab Dedicated instances is `gitlab-dedicated.com`. You get to choose the subdomain name where your instance is accessible from (for example, `customer_name.gitlab-dedicated.com`). - Initial storage: Initial storage size for your repositories in GB. - Availability Zone IDs for PrivateLink: If you plan to later add a PrivateLink connection (either [inbound](#inbound-private-link) or [outbound](#outbound-private-link)) to your environment, and you require the connections to be available in specific Availability Zones, you must provide up to two [Availability Zone IDs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#az-ids) during onboarding. If not specified, GitLab will select two random Availability Zone IDs in which the connections will be available. +- [KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) for encrypted AWS services (if you are using that functionality). ### Maintenance window @@ -45,6 +46,111 @@ To change or update the configuration for your GitLab Dedicated instance, open a The turnaround time for processing configuration change requests is [documented in the GitLab handbook](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#handling-configuration-changes-for-tenant-environments). +### Encrypted Data At Rest (BYOK) + +If you want your GitLab data to be encrypted at rest, the KMS keys used must be accessible by GitLab services. KMS keys can be used in two modes for this purpose: + +1. Per-service KMS keys (Backup, EBS, RDS, S3), or +1. One KMS key for all services. + +If you use a key per service, all services must be encrypted at rest. Selective enablement of this feature is not supported. + +The keys provided have to reside in the same primary and secondary region specified during [onboarding](#onboarding). + +For instructions on how to create and manage KMS keys, visit [Managing keys](https://docs.aws.amazon.com/kms/latest/developerguide/getting-started.html) in the AWS KMS documentation. + +To create a KMS key using the AWS Console: + +1. In `Configure key`, select: + 1. Key type: **Symmetrical** + 1. Key usage: **Encrypt and decrypt** + 1. `Advanced options`: + 1. Key material origin: **KMS** + 1. Regionality: **Multi-Region key** +1. Enter your values for key alias, description, and tags. +1. Select Key administrators (optionally allow or deny key administrators to delete the key). +1. For Key usage permissions, add the GitLab AWS account using the **Other AWS accounts** dialog. + +The last page asks you to confirm the KMS key policy. It should look similar to the following example, populated with your account IDs and usernames: + +```json +{ + "Version": "2012-10-17", + "Id": "byok-key-policy", + "Statement": [ + { + "Sid": "Enable IAM User Permissions", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::<CUSTOMER-ACCOUNT-ID>:root" + }, + "Action": "kms:*", + "Resource": "*" + }, + { + "Sid": "Allow access for Key Administrators", + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::<CUSTOMER-ACCOUNT-ID>:user/<CUSTOMER-USER>" + ] + }, + "Action": [ + "kms:Create*", + "kms:Describe*", + "kms:Enable*", + "kms:List*", + "kms:Put*", + "kms:Update*", + "kms:Revoke*", + "kms:Disable*", + "kms:Get*", + "kms:Delete*", + "kms:TagResource", + "kms:UntagResource", + "kms:ScheduleKeyDeletion", + "kms:CancelKeyDeletion", + "kms:ReplicateKey", + "kms:UpdatePrimaryRegion" + ], + "Resource": "*" + }, + { + "Sid": "Allow use of the key", + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::<GITLAB-ACCOUNT-ID>:root" + ] + }, + "Action": [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ], + "Resource": "*" + }, + { + "Sid": "Allow attachment of persistent resources", + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::<GITLAB-ACCOUNT-ID>:root" + ] + }, + "Action": [ + "kms:CreateGrant", + "kms:ListGrants", + "kms:RevokeGrant" + ], + "Resource": "*" + } + ] +} +``` + ### Inbound Private Link [AWS Private Link](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) allows users and applications in your VPC on AWS to securely connect to the GitLab Dedicated endpoint without network traffic going over the public internet. @@ -97,7 +203,7 @@ Prerequisites: To activate SAML for your GitLab Dedicated instance: -1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#providers). +1. Read the [GitLab documentation about SAML](../../integration/saml.md#https://docs.gitlab.com/ee/integration/saml.html#configure-saml-on-your-idp) to gather all data your identity provider requires for configuration. You can also find some providers and their requirements in the [group SAML documentation](../../user/group/saml_sso/index.md#set-up-identity-provider). 1. To make the necessary changes, include in your [support ticket](#configuration-changes) the desired [SAML configuration block](../../integration/saml.md#configure-saml-support-in-gitlab) that will be set on the GitLab application. At a minimum, GitLab needs the following information to enable SAML for your instance: - Assertion consumer service URL - Certificate fingerprint or certificate diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 97eff35da91..932ffb87288 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -22,7 +22,7 @@ To host the GitLab product documentation, you can use: - GitLab Pages - Your own web server -After you create a website by using one of these methods, you redirect the UI links +After you create a website by using one of these methods, redirect the UI links in the product to point to your website. NOTE: @@ -41,7 +41,7 @@ In the following example, we expose this on the host under the same port. Make sure you either: - Allow port `4000` in your firewall. -- Use a different port. In following examples, replace the leftmost `4000` with the port different port. +- Use a different port. In following examples, replace the leftmost `4000` with a different port number. To run the GitLab product documentation website in a Docker container: @@ -74,7 +74,7 @@ To run the GitLab product documentation website in a Docker container: docker-compose up -d ``` -1. Visit `http://0.0.0.0:4000` to view the documentation website and verify +1. Visit `http://0.0.0.0:4000` to view the documentation website and verify that it works. 1. [Redirect the help links to the new Docs site](#redirect-the-help-links-to-the-new-docs-site). @@ -84,7 +84,7 @@ You can use GitLab Pages to host the GitLab product documentation. Prerequisite: -- Ensure the Pages site URL does not use a subfolder. Because of how the docs +- Ensure the Pages site URL does not use a subfolder. Because of the way the docs site is pre-compiled, the CSS and JavaScript files are relative to the main domain or subdomain. For example, URLs like `https://example.com/docs/` are not supported. @@ -177,7 +177,11 @@ documentation URL requests as needed. For example, if your GitLab version is - When you select the link, you are redirected to `http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`. -To test the setting, select a **Learn more** link in the GitLab application. +To test the setting, in GitLab, select a **Learn more** link. For example: + +1. On the top bar, in the upper-right corner, select your avatar. +1. Select **Preferences**. +1. In the **Syntax highlighting theme** section, select **Learn more**. ## Upgrade the product documentation to a later version @@ -187,7 +191,7 @@ Upgrading the Docs site to a later version requires downloading the newer Docker To upgrade to a later version [using Docker](#self-host-the-product-documentation-with-docker): -- If you use plain Docker: +- If you use Docker: 1. Stop the running container: @@ -207,7 +211,7 @@ To upgrade to a later version [using Docker](#self-host-the-product-documentatio docker run --detach --name gitlab_docs -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.6 ``` -- If you use Docker compose: +- If you use Docker Compose: 1. Change the version in `docker-compose.yaml`, for example 14.6: @@ -231,7 +235,7 @@ To upgrade to a later version [using Docker](#self-host-the-product-documentatio To upgrade to a later version [using GitLab Pages](#self-host-the-product-documentation-with-gitlab-pages): -1. Edit your existing `.gitlab-ci.yml` file, and replace the `image`'s version number: +1. Edit your existing `.gitlab-ci.yml` file, and replace the `image` version number: ```yaml image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index d2edc3085f1..b6a3a563ae1 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -35,7 +35,7 @@ You can use the following environment variables to override certain values: | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | -| `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | +| `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `false`). | | `GITLAB_RAILS_CACHE_DEFAULT_TTL_SECONDS` | integer | The default TTL used for entries stored in the Rails-cache. Default is `28800`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95042) in 15.3. | ## Adding more variables diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 3f98f1e12fe..731f16822fd 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -123,7 +123,8 @@ The following are required to run Geo: - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and is removed in GitLab 16.0. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS -- All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). +- All sites must run the same GitLab version. +- All sites must run [the same PostgreSQL versions](setup/database.md#postgresql-replication). - If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites to avoid silent corruption of database indexes. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 912de360e43..feae2d49e8d 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -205,15 +205,16 @@ keys must be manually replicated to the **secondary** site. 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**. -  - 1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in - `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character +  + 1. In **Name**, enter the value for `gitlab_rails['geo_node_name']` in + `/etc/gitlab/gitlab.rb`. These values must always match **exactly**, character for character. - 1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These + 1. In **External URL**, enter the value for `external_url` in `/etc/gitlab/gitlab.rb`. These values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. - 1. (Optional) Choose which groups or storage shards should be replicated by the - **secondary** site. Leave blank to replicate all. Read more in + 1. Optional. In **Internal URL (optional)**, enter an internal URL for the primary site. + 1. Optional. Select which groups or storage shards should be replicated by the + **secondary** site. Leave blank to replicate all. For more information, see [selective synchronization](#selective-synchronization). 1. Select **Save changes** to add the **secondary** site. 1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: @@ -366,6 +367,9 @@ former is ideal for replicating data belonging to a subset of users, while the latter is more suited to progressively rolling out Geo to a large GitLab instance. +NOTE: +Geo's synchronization logic is outlined in the [documentation](../index.md). Both the solution and the documentation is subject to change from time to time. You must independently determine your legal obligations in regard to privacy and cybersecurity laws, and applicable trade control law on an ongoing basis. + Selective synchronization: 1. Does not restrict permissions from **secondary** sites. diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index 88ca8781dc3..66c67f29c1c 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -7,7 +7,7 @@ type: howto # Container Registry for a secondary site **(PREMIUM SELF)** -You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. +You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. NOTE: The Container Registry replication is used only for disaster recovery purposes. We do not recommend @@ -76,7 +76,9 @@ the **primary** site before following the next steps. We need to make Container Registry send notification events to the **primary** site. -1. SSH into your GitLab **primary** server and login as root: +For each application and Sidekiq node on the **primary** site: + +1. SSH into the node and login as the `root` user: ```shell sudo -i @@ -105,17 +107,15 @@ We need to make Container Registry send notification events to the that starts with a letter. You can generate one with `< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo` NOTE: - If you use an external Registry (not the one integrated with GitLab), you must add - these settings to its configuration yourself. In this case, you also have to specify - notification secret in `registry.notification_secret` section of + If you use an external Registry (not the one integrated with GitLab), you also have to specify + the notification secret (`registry['notification_secret']`) in the `/etc/gitlab/gitlab.rb` file. NOTE: - If you use GitLab HA, you also have to specify - the notification secret in `registry.notification_secret` section of + If you use GitLab HA, you also have to specify the notification secret (`registry['notification_secret']`) in `/etc/gitlab/gitlab.rb` file for every web node. -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure each node: ```shell gitlab-ctl reconfigure diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 27e1b990b2b..95faafd073c 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -43,8 +43,8 @@ verification methods: | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | -| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | +| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _SHA256 checksum_ | +| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _SHA256 checksum_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Infrastructure registry _(file system)_ | Geo with API | SHA256 checksum | @@ -139,7 +139,7 @@ We use PostgreSQL's own replication functionality to replicate data from the **p We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. -Elasticsearch is an optional database that for advanced searching capabilities. It can improve search +Elasticsearch is an optional database for advanced search. It can improve search in both source-code level, and user generated content in issues, merge requests, and discussions. Elasticsearch is not supported in Geo. @@ -201,7 +201,7 @@ successfully, you must replicate their data using some other means. |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | See [instructions](container_registry.md) to set up the Container Registry replication. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | **Yes** (15.10) | **Yes** (12.3)* | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index a12dd8d9d68..cad3a396bfc 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -127,8 +127,7 @@ The following are PostgreSQL upgrade validation tests we performed. - Description: With PostgreSQL 12 available as an opt-in version in GitLab 13.3, we tested upgrading existing Geo installations from PostgreSQL 11 to 12. We also re-tested fresh installations of GitLab with Geo after fixes were made to support PostgreSQL 12. These tests were done using a - [nightly build](https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/bionic/gitlab-ee_13.3.6+rnightly.169516.d5209202-0_amd64.deb) - of GitLab 13.4. + nightly build of GitLab 13.4. - Outcome: Tests were successful for Geo deployments with a single database node on the primary and secondary. We encountered known issues with repmgr and Patroni managed PostgreSQL clusters on the Geo primary. Using PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. diff --git a/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png Binary files differnew file mode 100644 index 00000000000..d7c99e6551e --- /dev/null +++ b/doc/administration/geo/replication/img/adding_a_secondary_v15_8.png diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 59a67fecfcd..c18bb5b6351 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -42,7 +42,7 @@ to help identify if something is wrong:  -A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected site: +A site shows as "Unhealthy" if the site's status is more than 10 minutes old. In that case, try running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site: ```ruby Geo::MetricsUpdateWorker.new.perform @@ -52,7 +52,7 @@ If it raises an error, then the error is probably also preventing the jobs from If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data. -A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected site: +A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site: ```ruby Gitlab::Geo::HealthCheck.new.perform_checks @@ -1373,6 +1373,23 @@ The bug causes all wildcard domains (`.example.com`) to be ignored except for th You can have only one wildcard domain in the `no_proxy` list. +### Secondary site shows "Unhealthy" in UI after changing the value of `external_url` for the primary site + +If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for the primary site or changed the protocol from `http` to `https`, you may see that secondary sites are shown as `Unhealthy`. You may also find the following error in `geo.log`: + +```plaintext +"class": "Geo::NodeStatusRequestService", +... +"message": "Failed to Net::HTTP::Post to primary url: http://primary-site.gitlab.tld/api/v4/geo/status", + "error": "Failed to open TCP connection to <PRIMARY_IP_ADDRESS>:80 (Connection refused - connect(2) for \"<PRIMARY_ID_ADDRESS>\" port 80)" +``` + +In this case, make sure to update the changed URL on all your sites: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Admin > Geo > Sites**. +1. Change the URL and save the change. + ## Fixing non-PostgreSQL replication failures If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 55395a55857..644232a2c9e 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -33,6 +33,7 @@ and all **secondary** sites: 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. +1. Ensure that the secrets in the `/etc/gitlab/gitlab-secrets.json` file of both the primary site and the secondary site are the same. The file must be the same on all of a site’s nodes. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index addf894f0a5..35ab1d8252c 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -165,6 +165,7 @@ It does not cover all data types. | LFS objects (using Git) | **{check-circle}** Yes | | Pages | **{dotted-circle}** No <sup>2</sup> | | Advanced search (using the web UI) | **{dotted-circle}** No | +| Container registry | **{dotted-circle}** No | 1. Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 99f7b32be59..5f04326d49f 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -151,6 +151,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; ``` +1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_primary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)): + + ```ruby + ## Geo Primary role + roles(['geo_primary_role']) + ``` + 1. Configure PostgreSQL to listen on network interfaces: For security reasons, PostgreSQL does not listen on any network interfaces @@ -211,17 +218,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## - ## Geo Primary role - ## - Configures Postgres settings for replication - ## - Prevents automatic upgrade of Postgres since it requires downtime of - ## streaming replication to Geo secondary sites - ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, - ## or Sidekiq. If you are segregating services, then you will need to - ## explicitly disable unwanted services. - ## - roles(['geo_primary_role']) - - ## ## Primary address ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node ## @@ -239,11 +235,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # postgresql['max_replication_slots'] = 1 # Set this to be the number of Geo secondary nodes if you have more than one # postgresql['max_wal_senders'] = 10 # postgresql['wal_keep_segments'] = 10 + ``` - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## +1. Disable automatic database migrations temporarily until PostgreSQL is restarted and listening on the private address. + Edit `/etc/gitlab/gitlab.rb` and change the configuration to false: + + ```ruby + ## Disable automatic database migrations gitlab_rails['auto_migrate'] = false ``` @@ -402,6 +400,16 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** site match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** site. +1. Edit `/etc/gitlab/gitlab.rb` and set the role to `geo_secondary_role` (for more information, see [Geo roles](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)): + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles(['geo_secondary_role']) + ``` + 1. Configure PostgreSQL: This step is similar to how you configured the **primary** instance. @@ -412,12 +420,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles(['geo_secondary_role']) - - ## ## Secondary address ## - replace '<secondary_site_ip>' with the public or VPC address of your Geo secondary site ## diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 0fefc11f078..f93ef5f5d5e 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -217,14 +217,14 @@ the tracking database on port 5432. `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: - ```plaintext - ## - ## Geo Tracking Database Role - ## - pg_hba.conf - ## - host all all <trusted tracking IP>/32 md5 - host all all <trusted secondary IP>/32 md5 - ``` + ```plaintext + ## + ## Geo Tracking Database Role + ## - pg_hba.conf + ## + host all all <trusted tracking IP>/32 md5 + host all all <trusted secondary IP>/32 md5 + ``` 1. SSH into a GitLab **secondary** server and login as root: diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index c794b8ef219..022d9c00772 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -7,25 +7,22 @@ type: howto # Setting up Geo **(PREMIUM SELF)** -These instructions assume you have a working instance of GitLab. They guide you through: +## Prerequisites -1. Making your existing instance the **primary** site. -1. Adding **secondary** sites. +- Two (or more) independently working GitLab sites: + - One GitLab site serves as the Geo **primary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. You can use different reference architecture sizes for each Geo site. If you already have a working GitLab instance that is in-use, it can be used as a **primary** site. + - The second GitLab site serves as the Geo **secondary** site. Use the [GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. It's a good idea to sign in and test it. However, be aware that **all of the data on the secondary are lost** as part of the process of replicating from the **primary** site. -You must use a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher, -but you only need one license for all the sites. + NOTE: + Geo supports multiple secondaries. You can follow the same steps and make any changes accordingly. -WARNING: -The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites. Do not create an account or sign in to the new secondary.** +- Ensure the **primary** site has a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher to unlock Geo. You only need one license for all the sites. +- Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met by all sites. For example, sites must use the same GitLab version, and sites must be able to communicate with each other over certain ports. ## Using Omnibus GitLab If you installed GitLab using the Omnibus packages (highly recommended): -1. Confirm the [requirements for running Geo](../index.md#requirements-for-running-geo) are met. -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or sign in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. -1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. -1. [Confirm network connectivity](../index.md#firewall-rules) between the **primary** and **secondary** site. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. @@ -34,6 +31,10 @@ If you installed GitLab using the Omnibus packages (highly recommended): 1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. 1. Follow the [Using a Geo Site](../replication/usage.md) guide. +## Using GitLab Charts + +[Configure the GitLab chart with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/). + ## Post-installation documentation After installing GitLab on the **secondary** sites and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index d9191440a24..b11524083b1 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -1,7 +1,7 @@ --- -info: For assistance with this CSM Onboarding page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. +info: For assistance with this tutorial, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. stage: none -group: unassigned +group: Tutorials --- # Get started administering GitLab **(FREE)** @@ -151,7 +151,8 @@ Backups of GitLab databases and file systems are taken every 24 hours, and are k - You can use the project export option in: - [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data). - [The API](../api/project_import_export.md#schedule-an-export). -- [Group export](../user/group/settings/import_export.md) does *not* export the projects in it, but does export: +- [Group export by uploading a file export](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) + does **not** export the projects in it, but does export: - Epics - Milestones - Boards diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 349a92de51e..df9bdb0ee8d 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -111,4 +111,4 @@ URL to use SSH. ### Observe Git protocol version of connections For information on observing the Git protocol versions are being used in a production environment, -see the [relevant documentation](gitaly/monitoring.md#useful-queries). +see the [relevant documentation](gitaly/monitoring.md#queries). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 143f7dca7d3..543f5b3c119 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -143,10 +143,16 @@ Gitaly and GitLab use two shared secrets for authentication: To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`: ```ruby - gitaly['auth_token'] = 'abc123secret' + gitaly['configuration'] = { + # ... + auth: { + # ... + token: 'abc123secret', + }, + } ``` -There are two ways to configure the _GitLab Shell token_. +Configure the _GitLab Shell token_ in one of two ways. Method 1 (recommended): @@ -230,14 +236,21 @@ Updates to example must be made at: # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Authentication token to ensure only authorized servers can communicate with - # Gitaly server - gitaly['auth_token'] = 'AUTH_TOKEN' + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + auth: { + # ... + # + # Authentication token to ensure only authorized servers can communicate with + # Gitaly server + token: 'AUTH_TOKEN', + }, + } ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: @@ -247,24 +260,33 @@ Updates to example must be made at: On `gitaly1.internal`: ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'default', + path: '/var/opt/gitlab/git-data', + }, + { + name: 'storage1', + path: '/mnt/gitlab/git-data', + }, + ], + } ``` On `gitaly2.internal`: ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/srv/gitlab/git-data' - }, - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'storage2', + path: '/srv/gitlab/git-data', + }, + ], + } ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -461,17 +483,28 @@ example: git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, # Address of the GitLab server that also has Gitaly running on it - 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) -# Make Gitaly accept connections on all network interfaces -gitaly['listen_addr'] = "0.0.0.0:8075" - -# Or for TLS -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" -gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" +gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces + listen_addr: '0.0.0.0:8075', + # Or for TLS + tls_listen_addr: '0.0.0.0:9999', + tls: { + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + storage: [ + { + name: 'storage1', + path: '/mnt/gitlab/git-data', + }, + ], +} ``` `path` can be included only for storage shards on the local Gitaly server. @@ -523,9 +556,6 @@ To disable Gitaly on a GitLab server: ## Enable TLS support -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8. -> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured. - Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. @@ -543,6 +573,8 @@ Additionally, the certificate (or its certificate authority) must be installed o - Gitaly servers. - Gitaly clients that communicate with it. +If you use a load balancer, it must be able to negotiate HTTP/2 using the ALPN TLS extension. + ### Certificate requirements - The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. @@ -600,16 +632,21 @@ To configure Gitaly with TLS: <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + gitaly['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:9999', + tls: { + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). 1. Optional. Improve security by: - 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in + 1. Disabling non-TLS connections by commenting out or deleting `gitaly['configuration'][:listen_addr]` in `/etc/gitlab/gitlab.rb`. 1. Saving the file. 1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -693,7 +730,7 @@ To configure Gitaly with TLS: ### Observe type of Gitaly connections For information on observing the type of Gitaly connections being served, see the -[relevant documentation](monitoring.md#useful-queries). +[relevant documentation](monitoring.md#queries). ## `gitaly-ruby` @@ -708,14 +745,13 @@ Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` - RPCs that deal with wikis. - RPCs that create commits on behalf of a user, such as merge commits. -We recommend: +Recommended settings: - At least 300 MB memory per worker. - No more than one worker per core. NOTE: -`gitaly-ruby` is planned to be eventually removed. To track progress, see the -[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic. +[Epic 2862](https://gitlab.com/groups/gitlab-org/-/epics/2862) proposes to remove `gitaly-ruby`. ### Configure number of `gitaly-ruby` workers @@ -733,9 +769,16 @@ settings: 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - # Default is 2 workers. The minimum is 2; 1 worker is always reserved as - # a passive stand-by. - gitaly['ruby_num_workers'] = 4 + gitaly['configuration'] = { + # ... + 'gitaly-ruby': { + # ... + # + # Default is 2 workers. The minimum is 2; 1 worker is always reserved as + # a passive stand-by. + num_workers: 4 + }, + } ``` 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -775,21 +818,23 @@ example: ```ruby # in /etc/gitlab/gitlab.rb - -gitaly['concurrency'] = [ - { - 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel", - 'max_per_repo' => 20, - 'max_queue_time' => "1s", - 'max_queue_size' => 10 - }, - { - 'rpc' => "/gitaly.SSHService/SSHUploadPackWithSidechannel", - 'max_per_repo' => 20 - 'max_queue_time' => "1s", - 'max_queue_size' => 10 - } -] +gitaly['configuration'] = { + # ... + concurrency: [ + { + rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel', + max_per_repo: 20, + max_queue_time: '1s', + max_queue_size: 10, + }, + { + rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel', + max_per_repo: 20, + max_queue_time: '1s', + max_queue_size: 10, + }, + ], +} ``` - `rpc` is the name of the RPC to set a concurrency limit for per repository. @@ -834,7 +879,7 @@ performance. You can use control groups (cgroups) in Linux to impose limits on how much memory and CPU can be consumed by Gitaly processes. See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information. -cgroups can be useful for protecting the system against unexpected resource exhaustion because of over consumption of memory and CPU. +cgroups can help protect the system against unexpected resource exhaustion because of over consumption of memory and CPU. Some Git operations can consume notable resources up to the point of exhaustion in situations such as: @@ -861,43 +906,60 @@ When these limits are reached, performance may be reduced and users may be disco ### Configure repository cgroups (new method) -> This method of configuring repository cgroups was introduced in GitLab 15.1. +> - This method of configuring repository cgroups was introduced in GitLab 15.1. +> - `cpu_quota_us`[introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5422) in GitLab 15.10. To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method -to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`: +to `gitaly['configuration'][:cgroups]` in `/etc/gitlab/gitlab.rb`: -- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. -- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and +- `mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. +- `hierarchy_root` is the parent cgroup under which Gitaly creates groups, and is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` when Gitaly starts. -- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all +- `memory_bytes` is the total memory limit that is imposed collectively on all Git processes that Gitaly spawns. 0 implies no limit. -- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git +- `cpu_shares` is the CPU limit that is imposed collectively on all Git processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU. -- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git +- `cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management) + to throttle the cgroups' processes if they exceed this quota value. We set + `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit. +- `repositories.count` is the number of cgroups in the cgroups pool. Each time a new Git command is spawned, Gitaly assigns it to one of these cgroups based on the repository the command is for. A circular hashing algorithm assigns Git commands to these cgroups, so a Git command for a repository is always assigned to the same cgroup. -- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup. - 0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`. -- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup. +- `repositories.memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup. + 0 implies no limit. This value cannot exceed that of the top level `memory_bytes`. +- `repositories.cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup. 0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU. - This value cannot exceed that of the top level`cgroups_cpu_shares`. + This value cannot exceed that of the top level`cpu_shares`. +- `repositories.cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management) + that is imposed on all Git processes contained in a repository cgroup. A Git + process can't use more then the given quota. We set + `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit. For example: ```ruby # in /etc/gitlab/gitlab.rb -gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" -gitaly['cgroups_hierarchy_root'] => "gitaly" -gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb -gitaly['cgroups_cpu_shares'] = 1024 -gitaly['cgroups_repositories_count'] => 1000, -gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb -gitaly['cgroups_repositories_cpu_shares'] => 512 +gitaly['configuration'] = { + # ... + cgroups: { + mountpoint: '/sys/fs/cgroup', + hierarchy_root: 'gitaly', + memory_bytes: 64424509440, # 60gb + cpu_shares: 1024, + cpu_quota_us: 400000 # 4 cores + repositories: { + count: 1000, + memory_bytes: 32212254720, # 20gb + cpu_shares: 512, + cpu_quota_us: 200000 # 2 cores + }, + }, +} ``` ### Configure repository cgroups (legacy method) @@ -953,14 +1015,14 @@ This strategy has two main benefits: to 3 child cgroups can concurrently burst up to their max. In general, all 1000 cgroups would use much less than the 20 GB. -## Background Repository Optimization +## Background repository optimization Empty directories and unneeded configuration settings may accumulate in a repository and slow down Git operations. Gitaly can schedule a daily background task with a maximum duration to clean up these items and improve performance. WARNING: -This is an experimental feature and may place significant load on the host while running. +Background repository optimization is an experimental feature and may place significant load on the host while running. Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes). **For Omnibus GitLab** @@ -968,10 +1030,16 @@ Make sure to schedule this during off-peak hours and keep the duration short (fo Edit `/etc/gitlab/gitlab.rb` and add: ```ruby -gitaly['daily_maintenance_start_hour'] = 4 -gitaly['daily_maintenance_start_minute'] = 30 -gitaly['daily_maintenance_duration'] = '30m' -gitaly['daily_maintenance_storages'] = ["default"] +gitaly['configuration'] = { + # ... + daily_maintenance: { + # ... + start_hour: 4, + start_minute: 30, + duration: '30m', + storages: ['default'], + }, +} ``` **For installations from source** @@ -1006,7 +1074,7 @@ server" and "Gitaly client" refers to the same machine. ### Verify authentication monitoring Before rotating a Gitaly authentication token, verify that you can -[monitor the authentication behavior](monitoring.md#useful-queries) of your GitLab installation using +[monitor the authentication behavior](monitoring.md#queries) of your GitLab installation using Prometheus. You can then continue the rest of the procedure. @@ -1018,7 +1086,13 @@ transitioning" mode as follows: ```ruby # in /etc/gitlab/gitlab.rb -gitaly['auth_transitioning'] = true +gitaly['configuration'] = { + # ... + auth: { + # ... + transitioning: true, + }, +} ``` After you have made this change, your [Prometheus query](#verify-authentication-monitoring) @@ -1038,8 +1112,13 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi ```ruby # in /etc/gitlab/gitlab.rb - - gitaly['auth_token'] = '<new secret token>' + gitaly['configuration'] = { + # ... + auth: { + # ... + token: '<new secret token>', + }, + } ``` 1. Restart Gitaly: @@ -1069,7 +1148,13 @@ your Gitaly servers as follows: ```ruby # in /etc/gitlab/gitlab.rb -gitaly['auth_transitioning'] = false +gitaly['configuration'] = { + # ... + auth: { + # ... + transitioning: false, + }, +} ``` WARNING: @@ -1088,17 +1173,11 @@ result as you did at the start. For example: ## Pack-objects cache **(FREE SELF)** -> - [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372) in GitLab 13.11. -> - It's enabled on GitLab.com. -> - It's recommended for production use. - [Gitaly](index.md), the service that provides storage for Git repositories, can be configured to cache a short rolling window of Git fetch responses. This can reduce server load when your server receives lots of CI fetch traffic. -### Overview - The pack-objects cache wraps `git pack-objects`, an internal part of Git that gets invoked indirectly via the PostUploadPack and SSHUploadPack Gitaly RPCs. Gitaly runs PostUploadPack when a @@ -1135,27 +1214,33 @@ disk write IO, it is off by default. ### Configure the cache -These are the configuration settings for the pack-objects cache. Each -setting is discussed in greater detail below. +These configuration settings are available for the pack-objects cache. Each setting is discussed in greater detail +below. -|Setting|Default|Description| -|:---|:---|:---| -|`enabled`|`false`|Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. | -|`dir`|`<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache`|Local directory where cache files get stored.| -|`max_age`|`5m` (5 minutes)|Cache entries older than this get evicted and removed from disk.| +| Setting | Default | Description | +|:----------|:---------------------------------------------------|:---------------------------------------------------------------------------------------------------| +| `enabled` | `false` | Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. | +| `dir` | `<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache` | Local directory where cache files get stored. | +| `max_age` | `5m` (5 minutes) | Cache entries older than this get evicted and removed from disk. | In `/etc/gitlab/gitlab.rb`, set: ```ruby -gitaly['pack_objects_cache_enabled'] = true -## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache' -## gitaly['pack_objects_cache_max_age'] = '5m' +gitaly['configuration'] = { + # ... + pack_objects_cache: { + # ... + enabled: true, + # dir: '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache', + # max_age: '5m', + }, +} ``` #### `enabled` defaults to `false` -The cache is disabled by default. This is because in some cases, it -can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) +The cache is disabled by default because in some cases, it can create an +[extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) in the number of bytes written to disk. On GitLab.com, we have verified that our repository storage disks can handle this extra workload, but we felt we cannot assume this is true everywhere. @@ -1198,7 +1283,7 @@ The amount of space required depends on: - The size of the `max_age` cache eviction window. If your users pull 100 MB/s and you use a 5 minute window, then on average you have -`5*60*100MB = 30GB` of data in your cache directory. This is an expected average, not +`5*60*100MB = 30GB` of data in your cache directory. This average is an expected average, not a guarantee. Peak size may exceed this average. #### Cache eviction window `max_age` @@ -1208,11 +1293,9 @@ cache hit and the average amount of storage used by cache files. Entries older than `max_age` get evicted from the in-memory metadata store, and deleted from disk. -Eviction does not interfere with ongoing requests, so it is OK -for `max_age` to be less than the time it takes to do a fetch over a -slow connection. This is because Unix filesystems do not truly delete -a file until all processes that are reading the deleted file have -closed it. +Eviction does not interfere with ongoing requests. It is OK for `max_age` to be less than the time it takes to do a +fetch over a slow connection because Unix filesystems do not truly delete a file until all processes that are reading +the deleted file have closed it. ### Observe the cache @@ -1315,7 +1398,32 @@ process repositories that do not pass consistency checks. For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the following keys (in this example, to disable the `hasDotgit` consistency check): -- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later: +- In [GitLab 15.10](https://gitlab.com/gitlab-org/gitaly/-/issues/4754) and later: + + ```ruby + ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" + + gitaly['configuration'] = { + # ... + git: { + # ... + config: [ + # Populate a file with one unabbreviated SHA-1 per line. + # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList + { key: "fsck.skipList", value: ignored_blobs }, + { key: "fetch.fsck.skipList", value: ignored_blobs }, + { key: "receive.fsck.skipList", value: ignored_blobs }, + + { key: "fsck.hasDotgit", value: "ignore" }, + { key: "fetch.fsck.hasDotgit", value: "ignore" }, + { key: "receive.fsck.hasDotgit", value: "ignore" }, + { key: "fsck.missingSpaceBeforeEmail", value: "ignore" }, + ], + }, + } + ``` + +- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) to GitLab 15.9: ```ruby ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" @@ -1423,7 +1531,13 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185). 1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`: ```ruby - gitaly['gpg_signing_key_path'] = "/etc/gitlab/gitaly/signing_key.gpg" + gitaly['configuration'] = { + # ... + git: { + # ... + signing_key: '/etc/gitlab/gitaly/signing_key.gpg', + }, + } ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 405c56284cf..34e72f3eb5a 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -78,7 +78,7 @@ the current status of these issues, refer to the referenced issues and epics. Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information -during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). +during a restore, you should use the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups) can be used to speed up Gitaly Cluster backups. @@ -100,7 +100,7 @@ These SSDs should have a throughput of at least: - 2,000 IOPS for write operations. These IOPS values are initial recommendations, and may be adjusted to greater or lesser values -depending on the scale of your environment's workload. If you’re running the environment on a +depending on the scale of your environment's workload. If you're running the environment on a cloud provider, refer to their documentation about how to configure IOPS correctly. For repository data, only local storage is supported for Gitaly and Gitaly Cluster for performance and consistency reasons. @@ -363,10 +363,10 @@ For example, `@cluster/repositories/6f/96/54771`. The last component of the replica path, `54771`, is the repository ID. This can be used to identify the repository on the disk. -`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. This is used to balance -the repositories evenly into subdirectories to avoid overly large directories that might cause problems on some file -systems. In this case, `54771` hashes to `6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the -first four digits are `6f` and `96`. +`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. These +digits are used to balance the repositories evenly into subdirectories to avoid overly large directories that might +cause problems on some file systems. In this case, `54771` hashes to +`6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the first four digits are `6f` and `96`. #### Identify repositories on disk @@ -387,8 +387,9 @@ Follow the [instructions in hashed storage's documentation](../repository_storag Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ensure atomicity of repository creation, deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave -the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state -does not interfere with future operations but may use up disk space unnecessarily until a clean up is performed. +the metadata consistent. The disks may contain stale state even after successful operations. This situation is expected and +the leftover state does not interfere with future operations but may use up disk space unnecessarily until a clean up is +performed. There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover repositories from the storages. @@ -397,7 +398,7 @@ repositories from the storages. When creating repositories, Praefect: -1. Reserves a repository ID from PostgreSQL. This is atomic and no two creations receive the same ID. +1. Reserves a repository ID from PostgreSQL, which is atomic and no two creations receive the same ID. 1. Creates replicas on the Gitaly storages in the replica path derived from the repository ID. 1. Creates metadata records after the repository is successfully created on disk. @@ -470,7 +471,7 @@ _Up to date_ in this context means that: The primary node is chosen to serve the request if: -- There are no up to date nodes. +- No up-to-date nodes exist. - Any other error occurs during node selection. You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) using Prometheus. @@ -677,7 +678,7 @@ Direct Git access is: For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't remove it as long some GitLab installations require Git repositories on NFS. -There are two facets to our efforts to remove direct Git access in GitLab: +Two facets of our efforts to remove direct Git access in GitLab are: - Reduce the number of inefficient Gitaly queries made by GitLab. - Persuade administrators of fault-tolerant or horizontally-scaled GitLab instances to migrate off diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 0fd34d5c89f..d4f3d931b03 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -54,6 +54,9 @@ You can observe the status of [control groups (cgroups)](configure_gitaly.md#con - `gitaly_cgroups_cpu_usage`, a gauge that measures CPU usage per cgroup. - `gitaly_cgroup_procs_total`, a gauge that measures the total number of processes Gitaly has spawned under the control of cgroups. +- `gitaly_cgroup_cpu_cfs_periods_total`, a counter that for the value of [`nr_periods`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics). +- `gitaly_cgroup_cpu_cfs_throttled_periods_total`, a counter for the value of [`nr_throttled`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics). +- `gitaly_cgroup_cpu_cfs_throttled_seconds_total`, a counter for the value of [`throttled_time`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics) in seconds. ## `pack-objects` cache @@ -86,9 +89,9 @@ gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/reposit gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 ``` -## Useful queries +## Queries -The following are useful queries for monitoring Gitaly: +The following are some queries for monitoring Gitaly: - Use the following Prometheus query to observe the [type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production @@ -130,8 +133,8 @@ The following are useful queries for monitoring Gitaly: ## Monitor Gitaly Cluster -To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics -endpoints from which metrics can be scraped: +To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. Two separate metrics endpoints are +available from which metrics can be scraped: - The default `/metrics` endpoint. - `/db_metrics`, which contains metrics that require database queries. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 440bd7427ae..efaa8c1ee18 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -142,7 +142,7 @@ with secure tokens as you complete the setup process. 1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect cluster can only be accessed by Gitaly clients that carry this token. 1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication traffic inside - your Praefect cluster. This is distinct from `PRAEFECT_EXTERNAL_TOKEN` + your Praefect cluster. This token is distinct from `PRAEFECT_EXTERNAL_TOKEN` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. 1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to @@ -269,11 +269,16 @@ The database used by Praefect is now configured. You can now configure Praefect to use the database: ```ruby -praefect['database_host'] = POSTGRESQL_HOST -praefect['database_port'] = 5432 -praefect['database_user'] = 'praefect' -praefect['database_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_dbname'] = 'praefect_production' +praefect['configuration'] = { + # ... + database: { + # ... + host: POSTGRESQL_HOST, + port: 5432, + password: PRAEFECT_SQL_PASSWORD, + dbname: 'praefect_production', + } +} ``` If you see Praefect database errors after configuring PostgreSQL, see @@ -285,19 +290,27 @@ Praefect performance can be improved by additionally configuring the `database_d settings: ```ruby -praefect['database_direct_host'] = POSTGRESQL_HOST -praefect['database_direct_port'] = 5432 - -# Use the following to override parameters of direct database connection. -# Comment out where the parameters are the same for both connections. - -praefect['database_direct_user'] = 'praefect' -praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_direct_dbname'] = 'praefect_production' -#praefect['database_direct_sslmode'] = '...' -#praefect['database_direct_sslcert'] = '...' -#praefect['database_direct_sslkey'] = '...' -#praefect['database_direct_sslrootcert'] = '...' +praefect['configuration'] = { + # ... + database: { + # ... + session_pooled: { + # ... + host: POSTGRESQL_HOST, + port: 5432 + + # Use the following to override parameters of direct database connection. + # Comment out where the parameters are the same for both connections. + user: 'praefect', + password: PRAEFECT_SQL_PASSWORD, + dbname: 'praefect_production', + # sslmode: '...', + # sslcert: '...', + # sslkey: '...', + # sslrootcert: '...', + } + } +} ``` When configured, this connection is automatically used for the @@ -313,8 +326,8 @@ reads distribution caching is enabled by configuration #### Use PgBouncer -To reduce PostgreSQL resource consumption, we recommend setting up and configuring -[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. However, PgBouncer isn't required because +To reduce PostgreSQL resource consumption, you should set up and configure [PgBouncer](https://www.pgbouncer.org/) in +front of the PostgreSQL instance. However, PgBouncer isn't required because Praefect makes a low number of connections. If you choose to use PgBouncer, you can use the same PgBouncer instance for both the GitLab application database and the Praefect database. @@ -322,15 +335,21 @@ To configure PgBouncer in front of the PostgreSQL instance, you must point Praef parameters on Praefect configuration: ```ruby -praefect['database_host'] = PGBOUNCER_HOST -praefect['database_port'] = 6432 -praefect['database_user'] = 'praefect' -praefect['database_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_dbname'] = 'praefect_production' -#praefect['database_sslmode'] = '...' -#praefect['database_sslcert'] = '...' -#praefect['database_sslkey'] = '...' -#praefect['database_sslrootcert'] = '...' +praefect['configuration'] = { + # ... + database: { + # ... + host: PGBOUNCER_HOST, + port: 6432, + user: 'praefect', + password: PRAEFECT_SQL_PASSWORD, + dbname: 'praefect_production', + # sslmode: '...', + # sslcert: '...', + # sslkey: '...', + # sslrootcert: '...', + } +} ``` Praefect requires an additional connection to the PostgreSQL that supports the @@ -341,12 +360,12 @@ It is not supported in `transaction` pool mode (`pool_mode = transaction`). To configure the additional connection, you must either: - Configure a new PgBouncer database that uses to the same PostgreSQL database endpoint, - but with different pool mode. That is, `pool_mode = session`. + but with different pool mode (`pool_mode = session`). - Connect Praefect directly to PostgreSQL and bypass PgBouncer. #### Configure a new PgBouncer database with `pool_mode = session` -We recommend using PgBouncer with `session` pool mode. You can use the +You should use PgBouncer with `session` pool mode. You can use the [bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it manually](https://www.pgbouncer.org/config.html). @@ -399,23 +418,30 @@ praefect_production_direct = host=POSTGRESQL_HOST auth_user=pgbouncer dbname=pra Now you can configure Praefect to use PgBouncer for both connections: ```ruby -praefect['database_host'] = PGBOUNCER_HOST -praefect['database_port'] = 6432 -praefect['database_user'] = 'praefect' -# `PRAEFECT_SQL_PASSWORD` is the plain-text password of -# Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`. -praefect['database_password'] = PRAEFECT_SQL_PASSWORD - -praefect['database_dbname'] = 'praefect_production' -praefect['database_direct_dbname'] = 'praefect_production_direct' - -# There is no need to repeat the following. Parameters of direct -# database connection will fall back to the values above. - -#praefect['database_direct_host'] = PGBOUNCER_HOST -#praefect['database_direct_port'] = 6432 -#praefect['database_direct_user'] = 'praefect' -#praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD +praefect['configuration'] = { + # ... + database: { + # ... + host: PGBOUNCER_HOST, + port: 6432, + user: 'praefect', + # `PRAEFECT_SQL_PASSWORD` is the plain-text password of + # Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`. + password: PRAEFECT_SQL_PASSWORD, + dbname: 'praefect_production', + session_pooled: { + # ... + dbname: 'praefect_production_direct', + # There is no need to repeat the following. Parameters of direct + # database connection will fall back to the values above. + # + # host: PGBOUNCER_HOST, + # port: 6432, + # user: 'praefect', + # password: PRAEFECT_SQL_PASSWORD, + }, + }, +} ``` With this configuration, Praefect uses PgBouncer for both connection types. @@ -428,25 +454,34 @@ configuration option is set. For more details, consult the PgBouncer documentati #### Configure Praefect to connect directly to PostgreSQL -As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different connection parameters for direct access -to PostgreSQL. This is the connection that supports the `LISTEN` feature. +As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different +connection parameters for direct access to PostgreSQL. This connection supports the `LISTEN` feature. An example of Praefect configuration that bypasses PgBouncer and directly connects to PostgreSQL: ```ruby -praefect['database_direct_host'] = POSTGRESQL_HOST -praefect['database_direct_port'] = 5432 - -# Use the following to override parameters of direct database connection. -# Comment out where the parameters are the same for both connections. - -praefect['database_direct_user'] = 'praefect' -praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_direct_dbname'] = 'praefect_production' -#praefect['database_direct_sslmode'] = '...' -#praefect['database_direct_sslcert'] = '...' -#praefect['database_direct_sslkey'] = '...' -#praefect['database_direct_sslrootcert'] = '...' +praefect['configuration'] = { + # ... + database: { + # ... + session_pooled: { + # ... + host: POSTGRESQL_HOST, + port: 5432, + + # Use the following to override parameters of direct database connection. + # Comment out where the parameters are the same for both connections. + # + user: 'praefect', + password: PRAEFECT_SQL_PASSWORD, + dbname: 'praefect_production', + # sslmode: '...', + # sslcert: '...', + # sslkey: '...', + # sslrootcert: '...', + }, + }, +} ``` ### Praefect @@ -501,30 +536,42 @@ Updates to example must be made at: `/etc/gitlab/gitlab.rb`: ```ruby - praefect['listen_addr'] = '0.0.0.0:2305' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + } ``` 1. Configure Prometheus metrics by editing `/etc/gitlab/gitlab.rb`: ```ruby - # Enable Prometheus metrics access to Praefect. You must use firewalls - # to restrict access to this address/port. - # The default metrics endpoint is /metrics - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' - - # Some metrics run queries against the database. Enabling separate database metrics allows - # these metrics to be collected when the metrics are - # scraped on a separate /db_metrics endpoint. - praefect['separate_database_metrics'] = true + praefect['configuration'] = { + # ... + # + # Enable Prometheus metrics access to Praefect. You must use firewalls + # to restrict access to this address/port. + # The default metrics endpoint is /metrics + prometheus_listen_addr: '0.0.0.0:9652', + # Some metrics run queries against the database. Enabling separate database metrics allows + # these metrics to be collected when the metrics are + # scraped on a separate /db_metrics endpoint. + prometheus_exclude_database_from_default_metrics: true, + } ``` -1. Configure a strong `auth_token` for **Praefect** by editing - `/etc/gitlab/gitlab.rb`. This is needed by clients outside the cluster +1. Configure a strong authentication token for **Praefect** by editing + `/etc/gitlab/gitlab.rb`, which is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster: ```ruby - praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN' + praefect['configuration'] = { + # ... + auth: { + # ... + token: 'PRAEFECT_EXTERNAL_TOKEN', + }, + } ``` 1. Configure **Praefect** to [connect to the PostgreSQL database](#postgresql). We @@ -533,19 +580,32 @@ Updates to example must be made at: If you want to use a TLS client certificate, the options below can be used: ```ruby - # Connect to PostgreSQL using a TLS client certificate - # praefect['database_sslcert'] = '/path/to/client-cert' - # praefect['database_sslkey'] = '/path/to/client-key' - - # Trust a custom certificate authority - # praefect['database_sslrootcert'] = '/path/to/rootcert' + praefect['configuration'] = { + # ... + database: { + # ... + # + # Connect to PostgreSQL using a TLS client certificate + # sslcert: '/path/to/client-cert', + # sslkey: '/path/to/client-key', + # + # Trust a custom certificate authority + # sslrootcert: '/path/to/rootcert', + }, + } ``` By default, Praefect refuses to make an unencrypted connection to PostgreSQL. You can override this by uncommenting the following line: ```ruby - # praefect['database_sslmode'] = 'disable' + praefect['configuration'] = { + # ... + database: { + # ... + # sslmode: 'disable', + }, + } ``` 1. Configure the **Praefect** cluster to connect to each Gitaly node in the @@ -573,29 +633,37 @@ Updates to example must be made at: NOTE: When adding additional Gitaly nodes to a virtual storage, all storage names - within that virtual storage must be unique. Additionally, all Gitaly node + in that virtual storage must be unique. Additionally, all Gitaly node addresses referenced in the Praefect configuration must be unique. ```ruby # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('default') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://GITALY_HOST_1:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN', - }, - 'gitaly-2' => { - 'address' => 'tcp://GITALY_HOST_2:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN' + # server ('default') and in gitaly['configuration'][:storage][INDEX][:name] on Gitaly nodes ('gitaly-1') + praefect['configuration'] = { + # ... + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://GITALY_HOST_1:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-2', + address: 'tcp://GITALY_HOST_2:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-3', + address: 'tcp://GITALY_HOST_3:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + ], }, - 'gitaly-3' => { - 'address' => 'tcp://GITALY_HOST_3:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN' - } - } - } + ], } ``` @@ -681,7 +749,14 @@ Note the following: This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. - To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + To disable the unencrypted listener, set: + + ```ruby + praefect['configuration'] = { + # ... + listen_addr: nil, + } + ``` To configure Praefect with TLS: @@ -702,9 +777,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -790,6 +871,125 @@ To configure Praefect with TLS: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +#### Service discovery + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8971) in GitLab 15.10. + +Prerequisites: + +- A DNS server. + +GitLab uses service discovery to retrieve a list of Praefect hosts. Service +discovery involves periodic checks of a DNS A or AAAA record, with the IPs +retrieved from the record serving as the addresses of the target nodes. +Praefect does not support service discovery by SRV record. + +By default, the minimum time between checks is 5 minutes, regardless of the +records' TTLs. Praefect does not support customizing this interval. When clients +receive an update, they: + +- Establish new connections to the new IP addresses. +- Keep existing connections to intact IP addresses. +- Drop connections to removed IP addresses. + +In-flight requests on to-be-removed connections are still handled until they +finish. Workhorse has a 10-minute timeout, while other clients do not specify a +graceful timeout. + +The DNS server should return all IP addresses instead of load-balancing itself. +Clients can distribute requests to IP addresses in a round-robin fashion. + +Before updating client configuration, ensure that DNS service discovery works +correctly. It should return the list of IP addresses correctly. `dig` is a good +tool to use to verify. + +```console +❯ dig A praefect.service.consul @127.0.0.1 + +; <<>> DiG 9.10.6 <<>> A praefect.service.consul @127.0.0.1 +;; global options: +cmd +;; Got answer: +;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29210 +;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 + +;; OPT PSEUDOSECTION: +; EDNS: version: 0, flags:; udp: 4096 +;; QUESTION SECTION: +;praefect.service.consul. IN A + +;; ANSWER SECTION: +praefect.service.consul. 0 IN A 10.0.0.3 +praefect.service.consul. 0 IN A 10.0.0.2 +praefect.service.consul. 0 IN A 10.0.0.1 + +;; Query time: 0 msec +;; SERVER: ::1#53(::1) +;; WHEN: Wed Dec 14 12:53:58 +07 2022 +;; MSG SIZE rcvd: 86 +``` + +##### Configure service discovery + +By default, Praefect delegates DNS resolution to the operating system. In such +cases, the Gitaly address can be set in either of these formats: + +- `dns:[host]:[port]` +- `dns:///[host]:[port]` (note the three slashes) + +You can also appoint an authoritative name server by setting it in this format: + +- `dns://[authority_host]:[authority_port]/[host]:[port]` + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + praefect['consul_service_name'] = 'praefect' + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in +`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` +with Praefect service discovery address, such as `praefect.service.consul`. + + ```ruby + git_data_dirs({ + "default" => { + "gitaly_address" => 'dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +:::TabTitle Self-compiled (source) + +1. Install a DNS service discovery service. Register all Praefect nodes with the service. +1. On the Praefect clients (except Gitaly servers), edit `storages` in + `/home/git/gitlab/config/gitlab.yml` as follows: + + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305 + path: /some/local/path + ``` + + NOTE: + `/some/local/path` should be set to a local folder that exists, however no + data is stored in this folder. [Issue 375254](https://gitlab.com/gitlab-org/gitlab/-/issues/375254) + proposes to remove this requirement. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +::EndTabs + ### Gitaly NOTE: @@ -813,12 +1013,11 @@ because we rely on Praefect to route operations correctly. Particular attention should be shown to: -- The `gitaly['auth_token']` configured in this section must match the `token` - value under `praefect['virtual_storages']['nodes']` on the Praefect node. This was set - in the [previous section](#praefect). This document uses the placeholder - `PRAEFECT_INTERNAL_TOKEN` throughout. -- The storage names in `git_data_dirs` configured in this section must match the - storage names under `praefect['virtual_storages']` on the Praefect node. This +- The `gitaly['configuration'][:auth][:token]` configured in this section must match the `token` + value under `praefect['configuration'][:virtual_storage][<index>][:node][<index>][:token]` on the Praefect node. This value was + set in the [previous section](#praefect). This document uses the placeholder `PRAEFECT_INTERNAL_TOKEN` throughout. +- The storage names in `gitaly['configuration'][:storage]` configured in this section must match the + storage names under `praefect['configuration'][:virtual_storage]` on the Praefect node. This was set in the [previous section](#praefect). This document uses `gitaly-1`, `gitaly-2`, and `gitaly-3` as Gitaly storage names. @@ -859,22 +1058,31 @@ For more information on Gitaly server configuration, see our `/etc/gitlab/gitlab.rb`: ```ruby - # Make Gitaly accept connections on all network interfaces. - # Use firewalls to restrict access to this address/port. - gitaly['listen_addr'] = '0.0.0.0:8075' - - # Enable Prometheus metrics access to Gitaly. You must use firewalls - # to restrict access to this address/port. - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. + # Use firewalls to restrict access to this address/port. + listen_addr: '0.0.0.0:8075', + # Enable Prometheus metrics access to Gitaly. You must use firewalls + # to restrict access to this address/port. + prometheus_listen_addr: '0.0.0.0:9236', + } ``` 1. Configure a strong `auth_token` for **Gitaly** by editing - `/etc/gitlab/gitlab.rb`. This is needed by clients to communicate with + `/etc/gitlab/gitlab.rb`, which is needed by clients to communicate with this Gitaly nodes. Typically, this token is the same for all Gitaly nodes. ```ruby - gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN' + gitaly['configuration'] = { + # ... + auth: { + # ... + token: 'PRAEFECT_INTERNAL_TOKEN', + }, + } ``` 1. Configure the GitLab Shell secret token, which is needed for `git push` operations. Either: @@ -903,13 +1111,13 @@ For more information on Gitaly server configuration, see our gitlab_rails['internal_api_url'] = 'http://GITLAB_HOST' ``` -1. Configure the storage location for Git data by setting `git_data_dirs` in +1. Configure the storage location for Git data by setting `gitaly['configuration'][:storage]` in `/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name (such as `gitaly-1`). - Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is + Instead of configuring `gitaly['configuration'][:storage]` uniquely for each Gitaly node, it is often easier to have include the configuration for all Gitaly nodes on every - Gitaly node. This is supported because the Praefect `virtual_storages` + Gitaly node. You can do this because the Praefect `virtual_storage` configuration maps each storage name (such as `gitaly-1`) to a specific node, and requests are routed accordingly. This means every Gitaly node in your fleet can share the same configuration. @@ -918,17 +1126,23 @@ For more information on Gitaly server configuration, see our # You can include the data dirs for all nodes in the same config, because # Praefect will only route requests according to the addresses provided in the # prior step. - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - }, - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - }, - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Save the changes to `/etc/gitlab/gitlab.rb` and @@ -980,8 +1194,8 @@ Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports and protocols you need configure. NOTE: -We recommend the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for example, -clones) keep some connections open for extended periods. +You should use the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for +example, clones) keep some connections open for extended periods. | LB Port | Backend Port | Protocol | |:--------|:-------------|:---------| @@ -995,12 +1209,12 @@ To complete this section you need: - [Configured Gitaly nodes](#gitaly) The Praefect cluster needs to be exposed as a storage location to the GitLab -application. This is done by updating the `git_data_dirs`. +application, which is done by updating the `git_data_dirs`. Particular attention should be shown to: - the storage name added to `git_data_dirs` in this section must match the - storage name under `praefect['virtual_storages']` on the Praefect nodes. This + storage name under `praefect['configuration'][:virtual_storage]` on the Praefect nodes. This was set in the [Praefect](#praefect) section of this guide. This document uses `default` as the Praefect storage name. @@ -1219,12 +1433,16 @@ You can configure: The configuration is added to the `/etc/gitlab/gitlab.rb` file: ```ruby - praefect['virtual_storages'] = { - 'default' => { - 'default_replication_factor' => 1, + praefect['configuration'] = { # ... - } - } + virtual_storage: [ + { + # ... + name: 'default', + default_replication_factor: 1, + }, + ], + } ``` - A replication factor for an existing repository using the `set-replication-factor` sub-command. @@ -1313,29 +1531,50 @@ interval is configurable with any valid [Go duration string](https://pkg.go.dev/ To verify the metadata every three days: ```ruby -praefect['background_verification_verification_interval'] = '72h' +praefect['configuration'] = { + # ... + background_verification: { + # ... + verification_interval: '72h', + }, +} ``` Values of 0 and below disable the background verifier. ```ruby -praefect['background_verification_verification_interval'] = '0' +praefect['configuration'] = { + # ... + background_verification: { + # ... + verification_interval: '0', + }, +} ``` #### Enable deletions +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) and disabled by default in GitLab 15.0 +> - [Default enabled](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5321) in GitLab 15.9. + WARNING: -Deletions are disabled by default due to a race condition with repository renames that can cause incorrect -deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo. -You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. +Deletions were disabled by default prior to GitLab 15.9 due to a race condition with repository renames +that can cause incorrect deletions, which is especially prominent in Geo instances as Geo performs more renames +than instances without Geo. In GitLab 15.0 to 15.5, you should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. The feature flag was removed in GitLab 15.6 making deletions always safe to enable. -By default, the worker does not delete invalid metadata records but logs them and outputs Prometheus -metrics for them. +By default, the worker deletes invalid metadata records. It also logs the deleted records and outputs Prometheus +metrics. -You can enable deleting invalid metadata records with: +You can disable deleting invalid metadata records with: ```ruby -praefect['background_verification_delete_invalid_records'] = true +praefect['configuration'] = { + # ... + background_verification: { + # ... + delete_invalid_records: false, + }, +} ``` ### Prioritize verification manually @@ -1370,10 +1609,10 @@ The output includes the number of replicas that were marked unverified. ## Automatic failover and primary election strategies -Praefect regularly checks the health of each Gitaly node. This is used to automatically fail over +Praefect regularly checks the health of each Gitaly node, which is used to automatically fail over to a newly-elected primary Gitaly node if the current primary node is found to be unhealthy. -We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes). This is +You should use [repository-specific primary nodes](#repository-specific-primary-nodes). This is [the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) from GitLab 14.0. ### Repository-specific primary nodes diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 1207d7af3e7..331f9b5a956 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -74,7 +74,7 @@ Cluster: ### Unavailable repositories > - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. -> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes. +> - From GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. From GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes. A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are not accessible through Praefect to prevent serving stale data that may break automated tooling. @@ -277,15 +277,33 @@ The reconciliation frequency can be changed via the configuration. The value can Examples: ```ruby -praefect['reconciliation_scheduling_interval'] = '5m' # the default value +praefect['configuration'] = { + # ... + reconciliation: { + # ... + scheduling_interval: '5m', # the default value + }, +} ``` ```ruby -praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds +praefect['configuration'] = { + # ... + reconciliation: { + # ... + scheduling_interval: '30s', # reconcile every 30 seconds + }, +} ``` ```ruby -praefect['reconciliation_scheduling_interval'] = '0' # disable the feature +praefect['configuration'] = { + # ... + reconciliation: { + # ... + scheduling_interval: '0', # disable the feature + }, +} ``` ### Manual reconciliation @@ -334,16 +352,21 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply ``` -- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: +- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration']['virtual_storage]` and looks like the following: ```ruby - praefect['virtual_storages'] = { - 'default' => { - ... - }, - 'storage-1' => { - ... - } + praefect['configuration'] = { + # ... + virtual_storage: [ + { + # ... + name: 'default', + }, + { + # ... + name: 'storage-1', + }, + ], } ``` @@ -415,16 +438,21 @@ The `track-repository` Praefect sub-command adds repositories on disk to the Pra sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately ``` -- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: +- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration'][:virtual_storage]` and looks like the following: ```ruby - praefect['virtual_storages'] = { - 'default' => { - ... - }, - 'storage-1' => { - ... - } + praefect['configuration'] = { + # ... + virtual_storage: [ + { + # ... + name: 'default', + }, + { + # ... + name: 'storage-1', + }, + ], } ``` @@ -473,7 +501,7 @@ The command validates that all entries: - Are formatted correctly and contain required fields. - Correspond to a valid Git repository on disk. -- Are not currently tracked in the Praefect tracking database. +- Are not tracked in the Praefect tracking database. If any entry fails these checks, the command aborts prior to attempting to track a repository. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 1516b82a906..8b4fc51bcdf 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -10,9 +10,8 @@ Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) configuration file. Unlike installations from source, in Omnibus GitLab, you would not edit this file directly. -The configuration file is passed as an argument to the `gitaly` -executable. This is usually done by either Omnibus GitLab or your -[init](https://en.wikipedia.org/wiki/Init) script. +The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either Omnibus +GitLab or your [init](https://en.wikipedia.org/wiki/Init) script. An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) can be found in the Gitaly project. @@ -42,7 +41,7 @@ prometheus_listen_addr = "localhost:9236" ### Authentication Gitaly can be configured to reject requests that do not contain a -specific bearer token in their headers. This is a security measure to +specific bearer token in their headers, which is a security measure to be used when serving requests over TCP: ```toml @@ -70,7 +69,7 @@ Remember to disable `transitioning` when you are done changing your token settings. All authentication attempts are counted in Prometheus under -the [`gitaly_authentications_total` metric](monitoring.md#useful-queries). +the [`gitaly_authentications_total` metric](monitoring.md#queries). ### TLS diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 5f05b6322b0..a3db28b0d7b 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -66,7 +66,7 @@ for details. ### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when +client has its own log file which may contain helpful information when you are seeing Gitaly errors. You can control the log level of the gRPC client with the `GRPC_LOG_LEVEL` environment variable. The default level is `WARN`. @@ -345,7 +345,7 @@ You might see the following in Gitaly and Praefect logs: } ``` -This is a gRPC call +This information in the logs is a gRPC call [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). If this error occurs, even though @@ -358,7 +358,7 @@ server to keep them synchronized. ### Gitaly not listening on new address after reconfiguring -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may +When updating the `gitaly['configuration'][:listen_addr]` or `gitaly['configuration'][:prometheus_listen_addr]` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. When this occurs, run `sudo gitlab-ctl restart` to resolve the issue. This should no longer be @@ -514,9 +514,9 @@ Here are common errors and potential causes: - 500 response code - `ActionView::Template::Error (7:permission denied)` - - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. + - `praefect['configuration'][:auth][:token]` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. - `Unable to save project. Error: 7:permission denied` - - Secret token in `praefect['storage_nodes']` on GitLab server does not match the + - Secret token in `praefect['configuration'][:virtual_storage]` on GitLab server does not match the value in `gitaly['auth_token']` on one or more Gitaly servers. - 503 response code - `GRPC::Unavailable (14:failed to connect to all addresses)` @@ -530,7 +530,7 @@ Here are common errors and potential causes: Some common reasons for the Praefect database to experience elevated CPU usage include: - Prometheus metrics scrapes [running an expensive query](https://gitlab.com/gitlab-org/gitaly/-/issues/3796). If you have GitLab 14.2 - or above, set `praefect['separate_database_metrics'] = true` in `gitlab.rb`. + or above, set `praefect['configuration'][:prometheus_exclude_database_from_default_metrics] = true` in `gitlab.rb`. - [Read distribution caching](praefect.md#reads-distribution-caching) is disabled, increasing the number of queries made to the database when user traffic is high. Ensure read distribution caching is enabled. @@ -544,9 +544,8 @@ To determine the primary node of a repository: - With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage. To determine the current primary Gitaly node for a specific virtual storage: - - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the + - (Recommended) Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). - This is recommended. - If you do not have Grafana set up, use the following command on each host of each Praefect node: @@ -650,7 +649,7 @@ If the supplied value for `-virtual-storage` is incorrect, the command returns t get metadata: rpc error: code = NotFound desc = repository not found ``` -The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`. +The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['configuration'][:virtual_storage]` in `/etc/gitlab/gitlab.rb`. ### Check that repositories are in sync @@ -669,7 +668,7 @@ However, the Praefect database tables are not created on initial reconfigure and errors that relations do not exist if either: - The `gitlab-ctl reconfigure` command isn't executed. -- There are errors during the execution. +- Errors occur during the execution. For example: @@ -693,7 +692,7 @@ praefect sql-migrate: OK (applied 21 migrations) This indicates that the virtual storage name used in the [Praefect configuration](praefect.md#praefect) does not match the storage name used in -[`git_data_dirs` setting](praefect.md#gitaly) for GitLab. +[`gitaly['configuration'][:storage][<index>][:name]` setting](praefect.md#gitaly) for GitLab. Resolve this by matching the virtual storage names used in Praefect and GitLab configuration. @@ -715,9 +714,13 @@ Possible solutions: - Provision larger VMs to gain access to larger network traffic allowances. - Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances. +### `gitlab-ctl reconfigure` fails with error: `STDOUT: praefect: configuration error: error reading config file: toml: cannot store TOML string into a Go int` + +This error occurs when `praefect['database_port']` or `praefect['database_direct_port']` are configured as a string instead of an integer. + ## Profiling Gitaly -Gitaly exposes several of the Golang built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening +Gitaly exposes several of the Go built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening on port `9236` of the GitLab server: - Get a list of running `goroutines` and their backtraces: diff --git a/doc/administration/index.md b/doc/administration/index.md index 9f4477bcbf7..68e16b56624 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -66,7 +66,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. - [Elasticsearch](../integration/advanced_search/elasticsearch.md): Enable Elasticsearch to - empower Advanced Search. Use when you deal with a huge amount of data. + empower advanced search. Use when you deal with a huge amount of data. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) - [Add a license](../user/admin_area/license.md): Add a license to unlock features that are in paid tiers of GitLab. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index b1d97fd16a9..2b2eefdb17c 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -290,7 +290,7 @@ Plan.default.actual_limits.update!(group_hooks: 100) Set the limit to `0` to disable it. -The default maximum number of webhooks is `100` per project, `50` per group. +The default maximum number of webhooks is `100` per project and `50` per group. Webhooks in a child group do not count towards the webhook limit of their parent group. For GitLab.com, see the [webhook limits for GitLab.com](../user/gitlab_com/index.md#webhooks). @@ -723,22 +723,23 @@ GitLab checks these limits against runners that have been active in the last 3 m A runner's registration fails if it exceeds the limit for the scope determined by the runner registration token. If the limit value is set to zero, the limit is disabled. -- GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan. -- Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: +GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan. - | Runner scope | Default value | - |---------------------------------------------|---------------| - | `ci_registered_group_runners` | 1000 | - | `ci_registered_project_runners` | 1000 | +Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: - To update these limits, run the following in the - [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): +| Runner scope | Default value | +|---------------------------------------------|---------------| +| `ci_registered_group_runners` | 1000 | +| `ci_registered_project_runners` | 1000 | + +To update these limits, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): - ```ruby - # Use ci_registered_group_runners or ci_registered_project_runners - # depending on desired scope - Plan.default.actual_limits.update!(ci_registered_project_runners: 100) - ``` +```ruby +# Use ci_registered_group_runners or ci_registered_project_runners +# depending on desired scope +Plan.default.actual_limits.update!(ci_registered_project_runners: 100) +``` ### Maximum file size for job logs @@ -922,7 +923,7 @@ Reports that go over the 20 MB limit aren't loaded. Affected reports: - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/index.md#artifactsexpose_as) - [Unit test reports](../ci/testing/unit_test_reports.md) -## Advanced Search limits +## Advanced search limits ### Maximum file size indexed @@ -945,7 +946,7 @@ is pre-allocated during indexing. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8. -You can set a limit on the content of text fields indexed for Advanced Search. +You can set a limit on the content of text fields indexed for advanced search. Setting a maximum helps to reduce the load of the indexing processes. If any text field exceeds this limit, then the text is truncated to this number of characters. The rest of the text is not indexed, and not searchable. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 081c4d8a08c..f90458200b3 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -9,12 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. > - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12. -When [Kroki](https://kroki.io) integration is enabled and configured in -GitLab, you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. +With the [Kroki](https://kroki.io) integration, +you can create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile. -## Kroki Server +## Kroki server -When Kroki is enabled, GitLab sends diagrams to an instance of Kroki to display them as images. +When you enable Kroki, GitLab sends diagrams to an instance of Kroki to display them as images. You can use the free public cloud instance `https://kroki.io` or you can [install Kroki](https://docs.kroki.io/kroki/setup/install/) on your own infrastructure. After you've installed Kroki, make sure to update the server URL to point to your instance. @@ -29,11 +29,16 @@ docker run -d --name kroki -p 8080:8000 yuzutech/kroki The **Kroki URL** is the hostname of the server running the container. -The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains the following diagrams libraries out-of-the-box: +The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) Docker image contains several diagram +libraries out of the box. For a complete list, see the +[`asciidoctor-kroki` README](https://github.com/ggrossetie/asciidoctor-kroki/blob/master/README.md#supported-diagram-types). +Supported libraries include: <!-- vale gitlab.Spelling = NO --> - [Bytefield](https://bytefield-svg.deepsymmetry.org/) +- [D2](https://d2lang.com/tour/intro/) +- [DBML](https://www.dbml.org/home/) - [Ditaa](https://ditaa.sourceforge.net) - [Erd](https://github.com/BurntSushi/erd) - [GraphViz](https://www.graphviz.org/) diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 33434e15c4e..042bca1f6c9 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -7,9 +7,8 @@ type: reference, howto # PlantUML **(FREE)** -When the [PlantUML](https://plantuml.com) integration is enabled and configured in -GitLab, you can create diagrams in snippets, wikis, and repositories. This integration -is enabled on GitLab.com for all SaaS users and does not require any additional configuration. +With the [PlantUML](https://plantuml.com) integration, you can create diagrams in snippets, wikis, and repositories. +This integration is enabled on GitLab.com for all SaaS users and does not require any additional configuration. To set up the integration on a self-managed instance, you must: @@ -117,7 +116,7 @@ services: image: 'gitlab/gitlab-ee:12.2.5-ee.0' environment: GITLAB_OMNIBUS_CONFIG: | - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" plantuml: image: 'plantuml/plantuml-server:tomcat' @@ -148,7 +147,7 @@ using Tomcat: ``` The Tomcat service should restart. After the restart is complete, the -PlantUML service is ready and listening for requests on port 8080: +PlantUML integration is ready and listening for requests on port 8080: `http://localhost:8080/plantuml` To change these defaults, edit the `/etc/tomcat8/server.xml` file. @@ -180,10 +179,10 @@ To enable this redirection: ```ruby # Docker deployment - nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" # Built from source - nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/(plantuml.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" + nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/plantuml/(.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" ``` 1. To activate the changes, run the following command: diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index a1522ccac31..0ffb83886ed 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Web terminals (DEPRECATED) **(FREE)** +# Web terminals (deprecated) **(FREE)** > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. > - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/353410) in GitLab 15.0. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 4f2eb20877e..0a4213b7e2d 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -276,7 +276,7 @@ to clean up orphaned artifacts. ### Migrating from object storage to local storage To migrate back to local storage, you must -[selectively disable the artifacts storage](object_storage.md#selectively-disabling-object-storage). +[selectively disable the artifacts storage](object_storage.md#disable-object-storage-for-specific-features). ## Expiring artifacts diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 3638729a6b6..adfba28520e 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -272,7 +272,7 @@ To migrate back to local storage: ``` 1. Edit `/etc/gitlab/gitlab.rb` and - [disable object storage](../object_storage.md#selectively-disabling-object-storage) + [disable object storage](../object_storage.md#disable-object-storage-for-specific-features) for LFS objects: ```ruby @@ -421,6 +421,18 @@ To check an installed Git LFS client's version, run this command: git lfs version ``` +### `Connection refused` errors + +If you push or mirror LFS objects and receive errors like the following: + +- `dial tcp <IP>:443: connect: connection refused` +- `Connection refused - connect(2) for \"<target-or-proxy-IP>\" port 443` + +a firewall or proxy rule may be terminating the connection. + +If connection checks with standard Unix tools or manual Git pushes are successful, +the rule may be related to the size of the request. + ## Error viewing a PDF file When LFS has been configured with object storage and `proxy_download` set to diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index 298d22f1da5..a077558c7d2 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -37,7 +37,7 @@ for details on managing SSL certificates and configuring NGINX. ### Load Balancers terminate SSL without backend SSL Configure your load balancers to use the `HTTP(S)` protocol rather than `TCP`. -The load balancers is be responsible for managing SSL certificates and +The load balancers are responsible for managing SSL certificates and terminating SSL. Because communication between the load balancers and GitLab isn't secure, diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index eab4c9b7d83..1104412020a 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -180,7 +180,7 @@ In addition, the log contains the originating IP address, (`remote_ip`), the user's ID (`user_id`), and username (`username`). Some endpoints (such as `/search`) may make requests to Elasticsearch if using -[Advanced Search](../../user/search/advanced_search.md). These +[advanced search](../../user/search/advanced_search.md). These additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, which correspond to: @@ -426,7 +426,7 @@ like this example: } ``` -## `kubernetes.log` (DEPRECATED) +## `kubernetes.log` (deprecated) > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -771,6 +771,24 @@ are recorded in this file. For example: {"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"} ``` +## `ci_resource_groups_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384180) in GitLab 15.9. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/ci_resource_group_json.log` +- Installations from source: `/home/git/gitlab/log/ci_resource_group_json.log` + +It contains information about [resource group](../../ci/resource_groups/index.md) acquisition. For example: + +```json +{"severity":"INFO","time":"2023-02-10T23:02:06.095Z","correlation_id":"01GRYS10C2DZQ9J1G12ZVAD4YD","resource_group_id":1,"processable_id":288,"message":"attempted to assign resource to processable","success":true} +{"severity":"INFO","time":"2023-02-10T23:02:08.945Z","correlation_id":"01GRYS138MYEG32C0QEWMC4BDM","resource_group_id":1,"processable_id":288,"message":"attempted to release resource from processable","success":true} +``` + +The examples show the `resource_group_id`, `processable_id`, `message`, and `success` fields for each entry. + ## `auth.log` > Introduced in GitLab 12.0. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 566bc070347..53b578eb2f3 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Self-monitoring project (DEPRECATED) **(FREE SELF)** +# Self-monitoring project (deprecated) **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default. > - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index ed55e7d7ff3..c21c92a3d63 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -37,8 +37,8 @@ The following metrics are available: | `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | | `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | | `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | -| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | -| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | +| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | `operation`, `store` | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation`, `store` | | `gitlab_cache_read_multikey_count` | Histogram | 15.7 | Count of keys in multi-key cache read operations | `controller`, `action` | | `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | `gitlab` | @@ -163,6 +163,9 @@ The following metrics are available: | `gitlab_diffs_render_real_duration_seconds` | Histogram | 15.8 | Duration in seconds spent on serializing and rendering diffs on diffs batch request | `controller`, `action` | | `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Ruby process violated a memory threshold | | | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Ruby process memory violations were handled | | +| `gitlab_sli_rails_request_apdex_total` | Counter | 14.4 | Total number of request Apdex measurements. For more information, see [Rails request SLIs](../../../development/application_slis/rails_request.md) | `endpoint_id`, `feature_category`, `request_urgency` | +| `gitlab_sli_rails_request_apdex_success_total` | Counter | 14.4 | Total number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli_rails_requests_apdex_total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | +| `gitlab_sli_rails_request_error_total` | Counter | 15.7 | Total number of request error measurements. For more information, see [Rails request SLIs](../../../development/application_slis/rails_request.md) | `endpoint_id`, `feature_category`, `request_urgency`, `error` | ## Metrics controlled by a feature flag @@ -206,16 +209,16 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | `url` | | `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | `url` | | `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | `url` | -| `geo_lfs_objects` | Gauge | 10.2 | Number of LFS objects on primary | `url` | +| `geo_lfs_objects` | Gauge | 10.2 | Number of LFS objects on primary | `url` | | `geo_lfs_objects_checksummed` | Gauge | 14.6 | Number of LFS objects checksummed successfully on primary | `url` | | `geo_lfs_objects_checksum_failed` | Gauge | 14.6 | Number of LFS objects failed to calculate the checksum on primary | `url` | -| `geo_lfs_objects_checksum_total` | Gauge | 14.6 | Number of LFS objects tried to checksum on primary | `url` | +| `geo_lfs_objects_checksum_total` | Gauge | 14.6 | Number of LFS objects that need to be checksummed on primary | `url` | | `geo_lfs_objects_synced` | Gauge | 10.2 | Number of syncable LFS objects synced on secondary | `url` | | `geo_lfs_objects_failed` | Gauge | 10.2 | Number of syncable LFS objects failed to sync on secondary | `url` | | `geo_lfs_objects_registry` | Gauge | 14.6 | Number of LFS objects in the registry | `url` | -| `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects verified on secondary | `url` | -| `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects' verifications failed on secondary | `url` | -| `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects' verifications tried on secondary | `url` |LFS objects failed to sync on secondary | `url` | +| `geo_lfs_objects_verified` | Gauge | 14.6 | Number of LFS objects successfully verified on secondary | `url` | +| `geo_lfs_objects_verification_failed` | Gauge | 14.6 | Number of LFS objects that failed verifications on secondary | `url` | +| `geo_lfs_objects_verification_total` | Gauge | 14.6 | Number of LFS objects to attempt to verify on secondary | `url` | | `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` | | `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` | | `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` | @@ -230,11 +233,11 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | | `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | | `geo_wikis_checksum_failed` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | -| `geo_repositories_verified` | Gauge | 10.7 | Number of repositories verified on secondary | `url` | -| `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` | +| `geo_repositories_verified` | Gauge | 10.7 | Number of repositories successfully verified on secondary | `url` | +| `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories that failed verification on secondary | `url` | | `geo_repositories_checksum_mismatch` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | -| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis verified on secondary | `url` | -| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` | +| `geo_wikis_verified` | Gauge | 10.7 | Number of wikis successfully verified on secondary | `url` | +| `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis that failed verification on secondary | `url` | | `geo_wikis_checksum_mismatch` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | | `geo_repositories_checked` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | | `geo_repositories_checked_failed` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | @@ -249,25 +252,25 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_terraform_state_versions` | Gauge | 13.5 | Number of terraform state versions on primary | `url` | | `geo_terraform_state_versions_checksummed` | Gauge | 13.5 | Number of terraform state versions checksummed successfully on primary | `url` | | `geo_terraform_state_versions_checksum_failed` | Gauge | 13.5 | Number of terraform state versions failed to calculate the checksum on primary | `url` | -| `geo_terraform_state_versions_checksum_total` | Gauge | 13.12 | Number of terraform state versions tried to checksum on primary | `url` | +| `geo_terraform_state_versions_checksum_total` | Gauge | 13.12 | Number of terraform state versions that need to be checksummed on primary | `url` | | `geo_terraform_state_versions_synced` | Gauge | 13.5 | Number of syncable terraform state versions synced on secondary | `url` | | `geo_terraform_state_versions_failed` | Gauge | 13.5 | Number of syncable terraform state versions failed to sync on secondary | `url` | | `geo_terraform_state_versions_registry` | Gauge | 13.5 | Number of terraform state versions in the registry | `url` | -| `geo_terraform_state_versions_verified` | Gauge | 13.12 | Number of terraform state versions verified on secondary | `url` | -| `geo_terraform_state_versions_verification_failed` | Gauge | 13.12 | Number of terraform state versions verifications failed on secondary | `url` | -| `geo_terraform_state_versions_verification_total` | Gauge | 13.12 | Number of terraform state versions verifications tried on secondary | `url` | +| `geo_terraform_state_versions_verified` | Gauge | 13.12 | Number of terraform state versions successfully verified on secondary | `url` | +| `geo_terraform_state_versions_verification_failed` | Gauge | 13.12 | Number of terraform state versions that failed verification on secondary | `url` | +| `geo_terraform_state_versions_verification_total` | Gauge | 13.12 | Number of terraform state versions to attempt to verify on secondary | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | -| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs tried to checksum on primary | `url` | -| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs successfully checksummed on primary | `url` | +| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs to checksum on primary | `url` | +| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs that successfully calculated the checksum on primary | `url` | | `geo_merge_request_diffs_checksum_failed` | Gauge | 13.4 | Number of merge request diffs failed to calculate the checksum on primary | `url` | | `geo_merge_request_diffs_synced` | Gauge | 13.4 | Number of syncable merge request diffs synced on secondary | `url` | | `geo_merge_request_diffs_failed` | Gauge | 13.4 | Number of syncable merge request diffs failed to sync on secondary | `url` | | `geo_merge_request_diffs_registry` | Gauge | 13.4 | Number of merge request diffs in the registry | `url` | -| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs verifications tried on secondary | `url` | -| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs verified on secondary | `url` | -| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs verifications failed on secondary | `url` | +| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs to attempt to verify on secondary | `url` | +| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs successfully verified on secondary | `url` | +| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs that failed verification on secondary | `url` | | `geo_snippet_repositories` | Gauge | 13.4 | Number of snippets on primary | `url` | | `geo_snippet_repositories_checksummed` | Gauge | 13.4 | Number of snippets checksummed on primary | `url` | | `geo_snippet_repositories_checksum_failed` | Gauge | 13.4 | Number of snippets failed to calculate the checksum on primary | `url` | @@ -281,25 +284,25 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_group_wiki_repositories_failed` | Gauge | 13.10 | Number of syncable group wikis failed on secondary | `url` | | `geo_group_wiki_repositories_registry` | Gauge | 13.10 | Number of syncable group wikis in the registry | `url` | | `geo_pages_deployments` | Gauge | 14.3 | Number of pages deployments on primary | `url` | -| `geo_pages_deployments_checksum_total` | Gauge | 14.6 | Number of pages deployments tried to checksum on primary | `url` | -| `geo_pages_deployments_checksummed` | Gauge | 14.6 | Number of pages deployments successfully checksummed on primary | `url` | +| `geo_pages_deployments_checksum_total` | Gauge | 14.6 | Number of pages deployments to checksum on primary | `url` | +| `geo_pages_deployments_checksummed` | Gauge | 14.6 | Number of pages deployments that successfully calculated the checksum on primary | `url` | | `geo_pages_deployments_checksum_failed` | Gauge | 14.6 | Number of pages deployments failed to calculate the checksum on primary | `url` | | `geo_pages_deployments_synced` | Gauge | 14.3 | Number of syncable pages deployments synced on secondary | `url` | | `geo_pages_deployments_failed` | Gauge | 14.3 | Number of syncable pages deployments failed to sync on secondary | `url` | | `geo_pages_deployments_registry` | Gauge | 14.3 | Number of pages deployments in the registry | `url` | -| `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments verifications tried on secondary | `url` | -| `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments verified on secondary | `url` | +| `geo_pages_deployments_verification_total` | Gauge | 14.6 | Number of pages deployments to attempt to verify on secondary | `url` | +| `geo_pages_deployments_verified` | Gauge | 14.6 | Number of pages deployments successfully verified on secondary | `url` | | `geo_pages_deployments_verification_failed` | Gauge | 14.6 | Number of pages deployments verifications failed on secondary | `url` | | `geo_job_artifacts` | Gauge | 14.8 | Number of job artifacts on primary | `url` | -| `geo_job_artifacts_checksum_total` | Gauge | 14.8 | Number of job artifacts tried to checksum on primary | `url` | -| `geo_job_artifacts_checksummed` | Gauge | 14.8 | Number of job artifacts successfully checksummed on primary | `url` | +| `geo_job_artifacts_checksum_total` | Gauge | 14.8 | Number of job artifacts to checksum on primary | `url` | +| `geo_job_artifacts_checksummed` | Gauge | 14.8 | Number of job artifacts that successfully calculated the checksum on primary | `url` | | `geo_job_artifacts_checksum_failed` | Gauge | 14.8 | Number of job artifacts failed to calculate the checksum on primary | `url` | | `geo_job_artifacts_synced` | Gauge | 14.8 | Number of syncable job artifacts synced on secondary | `url` | | `geo_job_artifacts_failed` | Gauge | 14.8 | Number of syncable job artifacts failed to sync on secondary | `url` | | `geo_job_artifacts_registry` | Gauge | 14.8 | Number of job artifacts in the registry | `url` | -| `geo_job_artifacts_verification_total` | Gauge | 14.8 | Number of job artifacts verifications tried on secondary | `url` | -| `geo_job_artifacts_verified` | Gauge | 14.8 | Number of job artifacts verified on secondary | `url` | -| `geo_job_artifacts_verification_failed` | Gauge | 14.8 | Number of job artifacts verifications failed on secondary | `url` | +| `geo_job_artifacts_verification_total` | Gauge | 14.8 | Number of job artifacts to attempt to verify on secondary | `url` | +| `geo_job_artifacts_verified` | Gauge | 14.8 | Number of job artifacts successfully verified on secondary | `url` | +| `geo_job_artifacts_verification_failed` | Gauge | 14.8 | Number of job artifacts that failed verification on secondary | `url` | | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | @@ -310,48 +313,52 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` | | `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` | | `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` | -| `geo_uploads_checksum_total` | Gauge | 14.6 | Number of uploads tried to checksum on primary | `url` | -| `geo_uploads_checksummed` | Gauge | 14.6 | Number of uploads successfully checksummed on primary | `url` | +| `geo_uploads_checksum_total` | Gauge | 14.6 | Number of uploads to checksum on primary | `url` | +| `geo_uploads_checksummed` | Gauge | 14.6 | Number of uploads that successfully calculated the checksum on primary | `url` | | `geo_uploads_checksum_failed` | Gauge | 14.6 | Number of uploads failed to calculate the checksum on primary | `url` | -| `geo_uploads_verification_total` | Gauge | 14.6 | Number of uploads verifications tried on secondary | `url` | -| `geo_uploads_verified` | Gauge | 14.6 | Number of uploads verified on secondary | `url` | -| `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads verifications failed on secondary | `url` | +| `geo_uploads_verification_total` | Gauge | 14.6 | Number of uploads to attempt to verify on secondary | `url` | +| `geo_uploads_verified` | Gauge | 14.6 | Number of uploads successfully verified on secondary | `url` | +| `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads that failed verification on secondary | `url` | | `geo_container_repositories` | Gauge | 15.4 | Number of container repositories on primary | `url` | | `geo_container_repositories_synced` | Gauge | 15.4 | Number of container repositories synced on secondary | `url` | | `geo_container_repositories_failed` | Gauge | 15.4 | Number of syncable container repositories failed to sync on secondary | `url` | | `geo_container_repositories_registry` | Gauge | 15.4 | Number of container repositories in the registry | `url` | -| `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` | -| `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | +| `geo_container_repositories_checksum_total` | Gauge | 15.10 | Number of container repositories checksummed successfully on primary | `url` | +| `geo_container_repositories_checksummed` | Gauge | 15.10 | Number of container repositories tried to checksum on primary | `url` | +| `geo_container_repositories_checksum_failed` | Gauge | 15.10 | Number of container repositories failed to calculate the checksum on primary | `url` | +| `geo_container_repositories_verification_total` | Gauge | 15.10 | Number of container repositories' verifications tried on secondary | `url` | +| `geo_container_repositories_verified` | Gauge | 15.10 | Number of container repositories verified on secondary | `url` | +| `geo_container_repositories_verification_failed` | Gauge | 15.10 | Number of container repositories' failed verifications on secondary | `url` | | `geo_ci_secure_files` | Gauge | 15.3 | Number of secure files on primary | `url` | -| `geo_ci_secure_files_checksum_total` | Gauge | 15.3 | Number of secure files tried to checksum on primary | `url` | -| `geo_ci_secure_files_checksummed` | Gauge | 15.3 | Number of secure files successfully checksummed on primary | `url` | +| `geo_ci_secure_files_checksum_total` | Gauge | 15.3 | Number of secure files to checksum on primary | `url` | +| `geo_ci_secure_files_checksummed` | Gauge | 15.3 | Number of secure files that successfully calculated the checksum on primary | `url` | | `geo_ci_secure_files_checksum_failed` | Gauge | 15.3 | Number of secure files failed to calculate the checksum on primary | `url` | | `geo_ci_secure_files_synced` | Gauge | 15.3 | Number of syncable secure files synced on secondary | `url` | | `geo_ci_secure_files_failed` | Gauge | 15.3 | Number of syncable secure files failed to sync on secondary | `url` | | `geo_ci_secure_files_registry` | Gauge | 15.3 | Number of secure files in the registry | `url` | -| `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files verifications tried on secondary | `url` | -| `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files verified on secondary | `url` | -| `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files verifications failed on secondary | `url` | +| `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files to attempt to verify on secondary | `url` | +| `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files successfully verified on secondary | `url` | +| `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files that failed verification on secondary | `url` | | `geo_dependency_proxy_blob` | Gauge | 15.6 | Number of dependency proxy blobs on primary | | -| `geo_dependency_proxy_blob_checksum_total` | Gauge | 15.6 | Number of dependency proxy blobs tried to checksum on primary | | -| `geo_dependency_proxy_blob_checksummed` | Gauge | 15.6 | Number of dependency proxy blobs successfully checksummed on primary | | +| `geo_dependency_proxy_blob_checksum_total` | Gauge | 15.6 | Number of dependency proxy blobs to checksum on primary | | +| `geo_dependency_proxy_blob_checksummed` | Gauge | 15.6 | Number of dependency proxy blobs that successfully calculated the checksum on primary | | | `geo_dependency_proxy_blob_checksum_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to calculate the checksum on primary | | | `geo_dependency_proxy_blob_synced` | Gauge | 15.6 | Number of dependency proxy blobs synced on secondary | | | `geo_dependency_proxy_blob_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to sync on secondary | | | `geo_dependency_proxy_blob_registry` | Gauge | 15.6 | Number of dependency proxy blobs in the registry | | -| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs verifications tried on secondary | | -| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs verified on secondary | | -| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs verifications failed on secondary | | +| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs to attempt to verify on secondary | | +| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs successfully verified on secondary | | +| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs that failed verification on secondary | | | `geo_dependency_proxy_manifests` | Gauge | 15.6 | Number of dependency proxy manifests on primary | `url` | -| `geo_dependency_proxy_manifests_checksum_total` | Gauge | 15.6 | Number of dependency proxy manifests tried to checksum on primary | `url` | -| `geo_dependency_proxy_manifests_checksummed` | Gauge | 15.6 | Number of dependency proxy manifests successfully checksummed on primary | `url` | +| `geo_dependency_proxy_manifests_checksum_total` | Gauge | 15.6 | Number of dependency proxy manifests to checksum on primary | `url` | +| `geo_dependency_proxy_manifests_checksummed` | Gauge | 15.6 | Number of dependency proxy manifests that successfully calculated the checksum on primary | `url` | | `geo_dependency_proxy_manifests_checksum_failed` | Gauge | 15.6 | Number of dependency proxy manifests failed to calculate the checksum on primary | `url` | | `geo_dependency_proxy_manifests_synced` | Gauge | 15.6 | Number of syncable dependency proxy manifests synced on secondary | `url` | | `geo_dependency_proxy_manifests_failed` | Gauge | 15.6 | Number of syncable dependency proxy manifests failed to sync on secondary | `url` | | `geo_dependency_proxy_manifests_registry` | Gauge | 15.6 | Number of dependency proxy manifests in the registry | `url` | -| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests verifications tried on secondary | `url` | -| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests verified on secondary | `url` | -| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests verifications failed on secondary | `url` | +| `geo_dependency_proxy_manifests_verification_total` | Gauge | 15.6 | Number of dependency proxy manifests to attempt to verify on secondary | `url` | +| `geo_dependency_proxy_manifests_verified` | Gauge | 15.6 | Number of dependency proxy manifests successfully verified on secondary | `url` | +| `geo_dependency_proxy_manifests_verification_failed` | Gauge | 15.6 | Number of dependency proxy manifests that failed verification on secondary | `url` | | `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | | | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | | | `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` | @@ -445,6 +452,7 @@ instance. For example, `cache` or `shared_state`. | `gitlab_redis_client_exceptions_total` | Counter | 13.2 | Number of Redis client exceptions, broken down by exception class | | `gitlab_redis_client_requests_total` | Counter | 13.2 | Number of Redis client requests | | `gitlab_redis_client_requests_duration_seconds` | Histogram | 13.2 | Redis request latency, excluding blocking commands | +| `gitlab_redis_client_redirections_total` | Counter | 15.10 | Number of Redis Cluster MOVED/ASK redirections, broken down by redirection type | ## Metrics shared directory diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 56583deca89..37a7445c290 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -201,7 +201,10 @@ To use an external Prometheus server: postgres_exporter['listen_address'] = '0.0.0.0:9187' # Gitaly nodes - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + gitaly['configuration'] = { + # ... + prometheus_listen_addr: '0.0.0.0:9236', + } ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 4b3e251d407..587da749766 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -11,7 +11,7 @@ It's recommended over NFS and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -## Options +## Supported object storage providers GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](https://fog.io/about/provider_documentation.html) to check @@ -19,7 +19,9 @@ which storage services can be integrated with GitLab. Specifically, GitLab has been tested by vendors and customers on a number of object storage providers: -- [Amazon S3](https://aws.amazon.com/s3/) +- [Amazon S3](https://aws.amazon.com/s3/) ([Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) + is not supported, see [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775) + for more information) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) @@ -28,19 +30,7 @@ Specifically, GitLab has been tested by vendors and customers on a number of obj - On-premises hardware and appliances from various storage vendors, whose list is not officially established. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. -### Known compatibility issues - -- Dell EMC ECS: Prior to GitLab 13.3, there is a - [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). - Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware. - -- Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). - -- Amazon S3 [Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) - is not supported. Follow [issue #335775](https://gitlab.com/gitlab-org/gitlab/-/issues/335775) - for progress on enabling this option. - -## Configuration guides +## Configure the object storage There are two ways of specifying object storage configuration in GitLab: @@ -69,7 +59,7 @@ Using the consolidated object storage configuration has a number of advantages: Because [direct upload mode](../development/uploads/index.md#direct-upload) must be enabled, only the following providers can be used: -- [Amazon S3-compatible providers](#s3-compatible-connection-settings) +- [Amazon S3-compatible providers](#amazon-s3) - [Google Cloud Storage](#google-cloud-storage-gcs) - [Azure Blob storage](#azure-blob-storage) @@ -90,7 +80,7 @@ Object storage for <object type> must have a bucket specified ``` If you want to use local storage for specific object types, you can -[selectively disable object storages](#selectively-disabling-object-storage). +[disable object storage for specific features](#disable-object-storage-for-specific-features). Most types of objects, such as CI artifacts, LFS files, and upload attachments can be saved in object storage by specifying a single @@ -105,7 +95,7 @@ When the consolidated form is: See the section on [ETag mismatch errors](#etag-mismatch) for more details. -#### Use AWS S3 +#### Full example using Amazon S3 The following example uses AWS S3 to enable object storage for all supported services: @@ -116,42 +106,42 @@ The following example uses AWS S3 to enable object storage for all supported ser 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting the values you want: - ```ruby - # Consolidated object storage configuration - gitlab_rails['object_store']['enabled'] = true - gitlab_rails['object_store']['proxy_download'] = true - gitlab_rails['object_store']['connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>', - 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>' - } - # OPTIONAL: The following lines are only needed if server side encryption is required - gitlab_rails['object_store']['storage_options'] = { - 'server_side_encryption' => '<AES256 or aws:kms>', - 'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>' - } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts' - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs' - gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs' - gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads' - gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages' - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy' - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state' - gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files' - gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages' - ``` - - If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit - the AWS access key and secret access key/value pairs. For example: - - ```ruby - gitlab_rails['object_store']['connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` + ```ruby + # Consolidated object storage configuration + gitlab_rails['object_store']['enabled'] = true + gitlab_rails['object_store']['proxy_download'] = true + gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>', + 'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>' + } + # OPTIONAL: The following lines are only needed if server side encryption is required + gitlab_rails['object_store']['storage_options'] = { + 'server_side_encryption' => '<AES256 or aws:kms>', + 'server_side_encryption_kms_key_id' => '<arn:aws:kms:xxx>' + } + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts' + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs' + gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs' + gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads' + gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages' + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy' + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state' + gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files' + gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages' + ``` + + If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit + the AWS access key and secret access key/value pairs. For example: + + ```ruby + gitlab_rails['object_store']['connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` 1. Save the file and reconfigure GitLab: @@ -171,7 +161,7 @@ The following example uses AWS S3 to enable object storage for all supported ser aws_secret_access_key: <AWS_SECRET_ACCESS_KEY> ``` - If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit + If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit the AWS access key and secret access key/value pairs. For example: ```yaml @@ -293,7 +283,7 @@ The following example uses AWS S3 to enable object storage for all supported ser gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages' ``` - If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit + If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit the AWS access key and secret access key/value pairs. For example: ```ruby @@ -348,7 +338,7 @@ The following example uses AWS S3 to enable object storage for all supported ser bucket: gitlab-pages ``` - If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit + If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit the AWS access key and secret access key/value pairs. For example: ```yaml @@ -369,7 +359,7 @@ The following example uses AWS S3 to enable object storage for all supported ser aws_secret_access_key = "<AWS_SECRET_ACCESS_KEY>" ``` - If you’re using [AWS IAM profiles](#using-amazon-instance-profiles), omit + If you're using [AWS IAM profiles](#use-amazon-instance-profiles), omit the AWS access key and secret access key/value pairs. For example: ```yaml @@ -432,7 +422,7 @@ gitlab_rails['object_store']['connection'] = { Both consolidated configuration form and storage-specific configuration form must configure a connection. The following sections describe parameters that can be used in the `connection` setting. -#### S3-compatible connection settings +#### Amazon S3 The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): @@ -450,7 +440,7 @@ The connection settings match those provided by [fog-aws](https://github.com/fog | `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | | `aws_credentials_refresh_threshold_seconds` | Sets the [automatic refresh threshold](https://github.com/fog/fog-aws#controlling-credential-refresh-time-with-iam-authentication) when using temporary credentials in IAM. | `15` | -#### Oracle Cloud S3 connection settings +#### Oracle Cloud S3 Oracle Cloud S3 must be sure to use the following settings: @@ -487,9 +477,10 @@ see the [Cloud Storage authentication documentation](https://cloud.google.com/st NOTE: Bucket encryption with the [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in [ETag mismatch errors](#etag-mismatch). -##### Google example (consolidated form) +##### GCS example -For Omnibus installations, this is an example of the `connection` setting: +For Omnibus installations, this is an example of the `connection` setting +in the consolidated form: ```ruby gitlab_rails['object_store']['connection'] = { @@ -499,13 +490,13 @@ gitlab_rails['object_store']['connection'] = { } ``` -##### Google example with ADC (consolidated form) +##### GCS example with ADC > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6. Google Cloud Application Default Credentials (ADC) are typically used with GitLab to use the default service account. This eliminates the -need to supply credentials for the instance. For example: +need to supply credentials for the instance. For example, in the consolidated form: ```ruby gitlab_rails['object_store']['connection'] = { @@ -549,42 +540,39 @@ The following are the valid connection parameters for Azure. For more informatio | `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | | `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | -##### Azure example (consolidated form) - -For Omnibus installations, this is an example of the `connection` setting: - -```ruby -gitlab_rails['object_store']['connection'] = { - 'provider' => 'AzureRM', - 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', - 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', - 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>' -} -``` +- For Omnibus installations, this is an example of the `connection` setting + in the consolidated form: -###### Azure Workhorse settings (source installs only) + ```ruby + gitlab_rails['object_store']['connection'] = { + 'provider' => 'AzureRM', + 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', + 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', + 'azure_storage_domain' => '<AZURE STORAGE DOMAIN>' + } + ``` -For source installations, Workhorse also needs to be configured with Azure -credentials. This isn't needed in Omnibus installs, because the Workhorse -settings are populated from the previous settings. +- For source installations, Workhorse also needs to be configured with Azure + credentials. This isn't needed in Omnibus installs, because the Workhorse + settings are populated from the previous settings. -1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: + 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: - ```toml - [object_storage] - provider = "AzureRM" + ```toml + [object_storage] + provider = "AzureRM" - [object_storage.azurerm] - azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>" - azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" - ``` + [object_storage.azurerm] + azure_storage_account_name = "<AZURE STORAGE ACCOUNT NAME>" + azure_storage_access_key = "<AZURE STORAGE ACCESS KEY>" + ``` -If you are using a custom Azure storage domain, -`azure_storage_domain` does **not** have to be set in the Workhorse -configuration. This information is exchanged in an API call between -GitLab Rails and Workhorse. + If you are using a custom Azure storage domain, + `azure_storage_domain` does **not** have to be set in the Workhorse + configuration. This information is exchanged in an API call between + GitLab Rails and Workhorse. -#### Storj Gateway Configuration (SJ) +#### Storj Gateway (SJ) NOTE: The Storj Gateway [does not support](https://github.com/storj/gateway-st/blob/4b74c3b92c63b5de7409378b0d1ebd029db9337d/docs/s3-compatibility.md) multi-threaded copying (see `UploadPartCopy` in the table). @@ -676,7 +664,7 @@ Within each object type, three parameters can be defined: | `enabled` | **{dotted-circle}** No | Overrides the common parameter. | | `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. | -#### Selectively disabling object storage +#### Disable object storage for specific features As seen above, object storage can be disabled for specific types by setting the `enabled` flag to `false`. For example, to disable object @@ -779,11 +767,133 @@ See the following additional guides: 1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). 1. [Disable Pages local storage](pages/index.md#disable-pages-local-storage). -## Warnings, limitations, and known issues +## Use Amazon instance profiles + +Instead of supplying AWS access and secret keys in object storage +configuration, GitLab can be configured to use IAM roles to set up an +[Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). +When this is used, GitLab fetches temporary credentials each time an +S3 bucket is accessed, so no hard-coded values are needed in the +configuration. + +To use an Amazon instance profile, GitLab must be able to connect to the +[instance metadata endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html). +If GitLab is [configured to use an Internet proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html), the endpoint IP +address must be added to the `no_proxy` list. + +### Encrypted S3 buckets + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. + +When configured either with an instance profile or with the consolidated +object configuration, GitLab Workhorse properly uploads files to S3 +buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). +Customer master keys (CMKs) and SSE-C encryption are +[not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). + +#### Server-side encryption headers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3. + +Setting a default encryption on an S3 bucket is the easiest way to +enable encryption, but you may want to +[set a bucket policy to ensure only encrypted objects are uploaded](https://repost.aws/knowledge-center/s3-bucket-store-kms-encrypted-objects). +To do this, you must configure GitLab to send the proper encryption headers +in the `storage_options` configuration section: + +| Setting | Description | +|-------------------------------------|------------------------------------------| +| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). | +| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). | + +As with the case for default encryption, these options only work when +the Workhorse S3 client is enabled. One of the following two conditions +must be fulfilled: + +- `use_iam_profile` is `true` in the connection settings. +- Consolidated object storage settings are in use. + +[ETag mismatch errors](#etag-mismatch) occur if server side +encryption headers are used without enabling the Workhorse S3 client. + +#### IAM Permissions + +To set up an instance profile: + +1. Create an Amazon Identity Access and Management (IAM) role with the necessary permissions. The + following example is a role for an S3 bucket named `test-bucket`: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject" + ], + "Resource": "arn:aws:s3:::test-bucket/*" + } + ] + } + ``` + +1. [Attach this role](https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/) + to the EC2 instance hosting your GitLab instance. +1. Configure GitLab to use it via the `use_iam_profile` configuration option. + +## Migrate objects to a different object storage provider + +You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using [Rclone](https://rclone.org/). + +The steps assume you are moving the `uploads` bucket, but the same process works for other buckets. + +Prerequisites: + +- Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone. + +1. [Install](https://rclone.org/downloads/) Rclone. +1. Configure Rclone by running the following: + + ```shell + rclone config + ``` + + The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (`old`), and one for the provider you are moving to (`new`). + +1. Verify that you can read the old data. The following example refers to the `uploads` bucket , but your bucket may have a different name: + + ```shell + rclone ls old:uploads | head + ``` + + This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if + the list is empty, go back and update your Rclone configuration using `rclone config`. + +1. Perform an initial copy. You do not need to take your GitLab server offline for this step. + + ```shell + rclone sync -P old:uploads new:uploads + ``` + +1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to + verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone + sync`, check your Rclone configuration and try again. + +After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things: + +1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket. +1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`. + +## Troubleshooting ### Objects are not included in GitLab backups -As noted in [our backup documentation](../raketasks/backup_restore.md), +As noted in [the backup documentation](../raketasks/backup_restore.md), objects are not included in GitLab backups. You can enable backups with your object storage provider instead. @@ -883,7 +993,7 @@ it's likely this is due to [encryption settings on your bucket](https://docs.aws To fix this issue, you have two options: - [Use the consolidated object configuration](#consolidated-object-storage-configuration). -- [Use Amazon instance profiles](#using-amazon-instance-profiles). +- [Use Amazon instance profiles](#use-amazon-instance-profiles). The first option is recommended for MinIO. Otherwise, the [workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) @@ -903,85 +1013,6 @@ eliminates the need to compare ETag headers returned from the S3 server. Encrypting buckets with the GCS [Cloud Key Management Service (KMS)](https://cloud.google.com/kms/docs) is not supported and results in ETag mismatch errors. -### Using Amazon instance profiles - -Instead of supplying AWS access and secret keys in object storage -configuration, GitLab can be configured to use IAM roles to set up an -[Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). -When this is used, GitLab fetches temporary credentials each time an -S3 bucket is accessed, so no hard-coded values are needed in the -configuration. - -To use an Amazon instance profile, GitLab must be able to connect to the -[instance metadata endpoint](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html). -If GitLab is [configured to use an Internet proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html), the endpoint IP -address must be added to the `no_proxy` list. - -#### Encrypted S3 buckets - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) in GitLab 13.1 for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) in GitLab 13.2 for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. - -When configured either with an instance profile or with the consolidated -object configuration, GitLab Workhorse properly uploads files to S3 -buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). -Customer master keys (CMKs) and SSE-C encryption are -[not supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). - -##### Server-side encryption headers - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240) in GitLab 13.3. - -Setting a default encryption on an S3 bucket is the easiest way to -enable encryption, but you may want to -[set a bucket policy to ensure only encrypted objects are uploaded](https://repost.aws/knowledge-center/s3-bucket-store-kms-encrypted-objects). -To do this, you must configure GitLab to send the proper encryption headers -in the `storage_options` configuration section: - -| Setting | Description | -|-------------------------------------|------------------------------------------| -| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). | -| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). | - -As with the case for default encryption, these options only work when -the Workhorse S3 client is enabled. One of the following two conditions -must be fulfilled: - -- `use_iam_profile` is `true` in the connection settings. -- Consolidated object storage settings are in use. - -[ETag mismatch errors](#etag-mismatch) occur if server side -encryption headers are used without enabling the Workhorse S3 client. - -#### IAM Permissions - -To set up an instance profile: - -1. Create an Amazon Identity Access and Management (IAM) role with the necessary permissions. The - following example is a role for an S3 bucket named `test-bucket`: - - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject" - ], - "Resource": "arn:aws:s3:::test-bucket/*" - } - ] - } - ``` - -1. [Attach this role](https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/) - to the EC2 instance hosting your GitLab instance. -1. Configure GitLab to use it via the `use_iam_profile` configuration option. - ### Multi-threaded copying GitLab uses the [S3 Upload Part Copy API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) @@ -996,46 +1027,3 @@ to run the following command: ```ruby Feature.disable(:s3_multithreaded_uploads) ``` - -## Migrate objects to a different object storage provider - -You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using [Rclone](https://rclone.org/). - -The steps assume you are moving the `uploads` bucket, but the same process works for other buckets. - -Prerequisites: - -- Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone. - -1. [Install](https://rclone.org/downloads/) Rclone. -1. Configure Rclone by running the following: - - ```shell - rclone config - ``` - - The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (`old`), and one for the provider you are moving to (`new`). - -1. Verify that you can read the old data. The following example refers to the `uploads` bucket , but your bucket may have a different name: - - ```shell - rclone ls old:uploads | head - ``` - - This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if - the list is empty, go back and update your Rclone configuration using `rclone config`. - -1. Perform an initial copy. You do not need to take your GitLab server offline for this step. - - ```shell - rclone sync -P old:uploads new:uploads - ``` - -1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to - verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone - sync`, check your Rclone configuration and try again. - -After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things: - -1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you do not leave any behind in the old bucket. -1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`. diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 7b61631fe3a..1153321267d 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -110,11 +110,11 @@ To enable the PROXY protocol: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_sshd['proxy_protocol'] = true - # # Proxy protocol policy ("use", "require", "reject", "ignore"), "use" is the default value - gitlab_sshd['proxy_policy'] = "use" - ``` + ```ruby + gitlab_sshd['proxy_protocol'] = true + # # Proxy protocol policy ("use", "require", "reject", "ignore"), "use" is the default value + gitlab_sshd['proxy_policy'] = "use" + ``` 1. Save the file and reconfigure GitLab: diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index f6ab46b9fbf..b02aab738ea 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -8,6 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Keep your GitLab instance up and running smoothly. +- [Upgrading GitLab](../../update/index.md). - [Rake tasks](../../raketasks/index.md): Tasks for common administration and operational processes such as [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, and more. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index aa0477be788..e9d829f3f08 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -139,14 +139,14 @@ To move all snippets by using the API: To move all groups by using the API: 1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard). - For example: - - ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ - "https://gitlab.example.com/api/v4/group_repository_storage_moves" - ``` + For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" + ``` 1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves). The response indicates either: diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 652a4fa5497..ac550d30566 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -569,7 +569,7 @@ def disable_two_factor! end def two_factor_enabled? - two_factor_otp_enabled? || two_factor_u2f_enabled? + two_factor_otp_enabled? || two_factor_webauthn_enabled? end ``` diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 9d1c8dcde5a..f92d57c0035 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -32,6 +32,7 @@ The following lists the currently supported OSs and their possible EOL dates. | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 22.04 | GitLab CE / GitLab EE 15.5.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2027 | <https://wiki.ubuntu.com/Releases> | | Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Amazon Linux 2022 | GitLab CE / GitLab EE 15.9.0 | amd64, arm64 | [Amazon Linux 2022 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2022) | October 2027 | <https://aws.amazon.com/linux/amazon-linux-2022/faqs/> | | Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | | Raspberry Pi OS (Bullseye) | GitLab CE 15.5.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2026 | [Raspberry Pi Details](https://www.raspberrypi.com/news/raspberry-pi-os-debian-bullseye/) | @@ -55,6 +56,14 @@ although [new versions have been released](https://about.gitlab.com/releases/cat of the [Linux package install guide](https://about.gitlab.com/install/#content). Future GitLab upgrades are fetched according to your upgraded OS. +## Update both GitLab and the operating system + +To upgrade both the operating system (OS) and GitLab: + +1. Upgrade the OS. +1. Check if it's necessary to [update the GitLab package sources](#update-gitlab-package-sources-after-upgrading-the-os). +1. [Upgrade GitLab](../../update/index.md). + ## Packages for ARM64 > [Introduced](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues/27) in GitLab 13.4. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 34acf57ce70..619ec2490a7 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -607,10 +607,10 @@ Without this configuration, the Azure storage driver uses `//` instead of `/` as registry['storage'] = { 'azure' => { 'accountname' => 'accountname', - 'accesskey' => 'base64encodedaccountkey', + 'accountkey' => 'base64encodedaccountkey', 'container' => 'containername', 'rootdirectory' => '/azure/virtual/container', - 'trimlegacyrootprefix' => 'true' + 'trimlegacyrootprefix' => true } } ``` @@ -627,6 +627,8 @@ storage: trimlegacyrootprefix: true ``` +By default, Azure Storage Driver uses the `core.windows.net` realm. You can set another value for `realm` in the `azure` section (for example, `core.usgovcloudapi.net` for Azure Government Cloud). For more information, see the [Docker documentation](https://docs.docker.com/registry/storage-drivers/azure/). + ### Disable redirect for storage driver By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server. @@ -637,20 +639,20 @@ However, this behavior is undesirable for registries used by internal hosts that 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['storage'] = { - 's3' => { - 'accesskey' => 's3-access-key', - 'secretkey' => 's3-secret-key-for-access-key', - 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region', - 'regionendpoint' => 'your-s3-regionendpoint' - }, - 'redirect' => { - 'disable' => true - } - } - ``` + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint' + }, + 'redirect' => { + 'disable' => true + } + } + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -658,21 +660,21 @@ However, this behavior is undesirable for registries used by internal hosts that 1. Add the `redirect` flag to your registry configuration YAML file: - ```yaml - storage: - s3: - accesskey: 'AKIAKIAKI' - secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' - region: 'your-s3-region' - regionendpoint: 'your-s3-regionendpoint' - redirect: - disable: true - cache: - blobdescriptor: inmemory - delete: - enabled: true - ``` + ```yaml + storage: + s3: + accesskey: 'AKIAKIAKI' + secretkey: 'secret123' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' + region: 'your-s3-region' + regionendpoint: 'your-s3-regionendpoint' + redirect: + disable: true + cache: + blobdescriptor: inmemory + delete: + enabled: true + ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -690,18 +692,18 @@ For Omnibus GitLab installations: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - registry['storage'] = { - 's3' => { - 'accesskey' => 's3-access-key', - 'secretkey' => 's3-secret-key-for-access-key', - 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region', - 'regionendpoint' => 'your-s3-regionendpoint', - 'encrypt' => true - } - } - ``` + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint', + 'encrypt' => true + } + } + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -710,16 +712,16 @@ For installations from source: 1. Edit your registry configuration YAML file: - ```yaml - storage: - s3: - accesskey: 'AKIAKIAKI' - secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' - region: 'your-s3-region' - regionendpoint: 'your-s3-regionendpoint' - encrypt: true - ``` + ```yaml + storage: + s3: + accesskey: 'AKIAKIAKI' + secretkey: 'secret123' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' + region: 'your-s3-region' + regionendpoint: 'your-s3-regionendpoint' + encrypt: true + ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index ed08b10fe97..efeb19a8d5e 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -631,34 +631,6 @@ are stored. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Alternatively, if you have existing Pages deployed you can follow -the below steps to do a no downtime transfer to a new storage location. - -1. Pause Pages deployments by setting the following in `/etc/gitlab/gitlab.rb`: - - ```ruby - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category!=pages" - ] - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. `rsync` contents from the current storage location to the new storage location: `sudo rsync -avzh --progress /var/opt/gitlab/gitlab-rails/shared/pages/ /mnt/storage/pages` -1. Set the new storage location in `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['pages_path'] = "/mnt/storage/pages" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Verify Pages are still being served up as expected. -1. Resume Pages deployments by removing from `/etc/gitlab/gitlab.rb` the `sidekiq` setting set above. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Trigger a new Pages deployment and verify it's working as expected. -1. Remove the old Pages storage location: `sudo rm -rf /var/opt/gitlab/gitlab-rails/shared/pages` -1. Verify Pages are still being served up as expected. - ## Configure listener for reverse proxy requests Follow the steps below to configure the proxy listener of GitLab Pages. @@ -683,23 +655,23 @@ Follow the steps below to configure the proxy listener of GitLab Pages. ## Set global maximum size of each GitLab Pages site **(FREE SELF)** -Prerequisites: +Prerequisite: -- Only GitLab administrators can edit this setting. +- You must have administrator access to the instance. To set the global maximum pages size for a project: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Preferences**. 1. Expand **Pages**. -1. Enter a value under **Maximum size of pages**. +1. In **Maximum size of pages**, enter a value. The default is `100`. 1. Select **Save changes**. ## Set maximum size of each GitLab Pages site in a group **(PREMIUM SELF)** -Prerequisites: +Prerequisite: -- You must have at least the Maintainer role for the group. +- You must have administrator access to the instance. To set the maximum size of each GitLab Pages site in a group, overriding the inherited setting: @@ -711,9 +683,9 @@ To set the maximum size of each GitLab Pages site in a group, overriding the inh ## Set maximum size of GitLab Pages site in a project **(PREMIUM SELF)** -Prerequisites: +Prerequisite: -- You must have at least the Maintainer role for the project. +- You must have administrator access to the instance. To set the maximum size of GitLab Pages site in a project, overriding the inherited setting: @@ -729,7 +701,7 @@ To set the maximum size of GitLab Pages site in a project, overriding the inheri Prerequisite: -- You must be an administrator of a self-managed GitLab instance. +- You must have administrator access to the instance. To set the maximum number of GitLab Pages custom domains for a project: @@ -891,7 +863,7 @@ Incorrect configuration of these values may result in intermittent or persistent errors, or the Pages Daemon serving old content. NOTE: -Expiry, interval and timeout flags use [Golang's duration formatting](https://pkg.go.dev/time#ParseDuration). +Expiry, interval and timeout flags use [Go duration formatting](https://pkg.go.dev/time#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as `300ms`, `1.5h` or `2h45m`. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 8605ee94255..24548f690ff 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -30,18 +30,18 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. Configure the GitLab application servers with the appropriate connection details for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: - ```ruby - # Disable the bundled Omnibus provided PostgreSQL - postgresql['enable'] = false - - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - ``` - - For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). + ```ruby + # Disable the bundled Omnibus provided PostgreSQL + postgresql['enable'] = false + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + ``` + + For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). 1. Reconfigure for the changes to take effect: diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 836736fb73f..188578a739c 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Pods +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -17,21 +17,81 @@ To scale GitLab, you can configure GitLab to use multiple application databases. Due to [known issues](#known-issues), configuring GitLab with multiple databases is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features). +After you have set up multiple databases, GitLab uses a second application database for +[CI/CD features](../../ci/index.md), referred to as the `ci` database. + +All tables have exactly the same structure in both the `main`, and `ci` +databases. Some examples: + +- When multiple databases are configured, the `ci_pipelines` table exists in + both the `main` and `ci` databases, but GitLab reads and writes only to the + `ci_pipelines` table in the `ci` database. +- Similarly, the `projects` table exists in + both the `main` and `ci` databases, but GitLab reads and writes only to the + `projects` table in the `main` database. +- For some tables (such as `loose_foreign_keys_deleted_records`) GitLab reads and writes to both the `main` and `ci` databases. See the + [development documentation](../../development/database/multiple_databases.md#gitlab-schema) + ## Known issues -- Migrating data from the `main` database to the `ci` database is not supported or documented yet. - Once data is migrated to the `ci` database, you cannot migrate it back. -## Set up multiple databases +## Migrate existing installations -Use the following content to set up multiple databases with a new GitLab installation. +To migrate existing data from the `main` database to the `ci` database, you can +copy the database across. -There is no documentation for existing GitLab installations yet. +### Existing source installation -After you have set up multiple databases, GitLab uses a second application database for -[CI/CD features](../../ci/index.md), referred to as the `ci` database. For -example, GitLab reads and writes to the `ci_pipelines` table in the `ci` -database. +1. Stop GitLab, except for PostgreSQL: + + ```shell + sudo service gitlab stop + sudo service postgresql start + ``` + +1. Dump the `main` database: + + ```shell + sudo -u git pg_dump -f gitlabhq_production.sql gitlabhq_production + ``` + +1. Create the `ci` database, and copy the data from the previous dump: + + ```shell + sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER git;" + sudo -u git psql -f gitlabhq_production.sql gitlabhq_production_ci + ``` + +1. Configure GitLab to [use multiple databases](#set-up-multiple-databases). + +### Existing Omnibus installation + +1. Stop GitLab, except for PostgreSQL: + + ```shell + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` + +1. Dump the `main` database: + + ```shell + sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production + ``` + +1. Create the `ci` database, and copy the data from the previous dump: + + ```shell + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;" + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production_ci + ``` + +1. Configure GitLab to [use multiple databases](#set-up-multiple-databases). + +## Set up multiple databases + +To configure GitLab to use multiple application databases, follow the instructions below for your installation type. WARNING: You must stop GitLab before setting up multiple databases. This prevents @@ -40,6 +100,9 @@ the other way around. ### Installations from source +1. For existing installations, + [migrate the data](#migrate-existing-installations) first. + 1. [Back up GitLab](../../raketasks/backup_restore.md) in case of unforeseen issues. @@ -70,7 +133,7 @@ the other way around. 1. Update the service files to set the `GITLAB_ALLOW_SEPARATE_CI_DATABASE` environment variable to `true`. -1. Create the `gitlabhq_production_ci` database: +1. For new installations only. Create the `gitlabhq_production_ci` database: ```shell sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;" @@ -91,6 +154,9 @@ the other way around. ### Omnibus GitLab installations +1. For existing installations, + [migrate the data](#migrate-existing-installations) first. + 1. [Back up GitLab](../../raketasks/backup_restore.md) in case of unforeseen issues. @@ -116,7 +182,8 @@ the other way around. sudo gitlab-ctl reconfigure ``` -1. Optional. Reconfiguring GitLab should create the `gitlabhq_production_ci`. If it did not, manually create the `gitlabhq_production_ci`: +1. Optional, for new installations only. Reconfiguring GitLab should create the + `gitlabhq_production_ci` database if it does not exist. If the database is not created automatically, create it manually: ```shell sudo gitlab-ctl start postgresql diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 5dd0aad7162..59aac9141a4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -174,10 +174,12 @@ ote_pid | tls ## Procedure for bypassing PgBouncer +### Omnibus installations + Some database changes have to be done directly, and not through PgBouncer. -Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) -and [GitLab upgrades](../../update/zero_downtime.md#postgresql). +The main affected tasks are [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) +and [GitLab upgrades with database migrations](../../update/zero_downtime.md#postgresql). 1. To find the primary node, run the following on a database node: @@ -195,7 +197,7 @@ and [GitLab upgrades](../../update/zero_downtime.md#postgresql). sudo gitlab-ctl reconfigure ``` -Once you've performed the tasks or procedure, switch back to using PgBouncer: +After you've performed the tasks or procedure, switch back to using PgBouncer: 1. Change back `/etc/gitlab/gitlab.rb` to point to PgBouncer. 1. Run reconfigure: @@ -204,6 +206,19 @@ Once you've performed the tasks or procedure, switch back to using PgBouncer: sudo gitlab-ctl reconfigure ``` +### Helm chart installations + +High-availability deployments also need to bypass PgBouncer for the same reasons as Omnibus-based ones. +For this type of installation: + +- Database backup and restore tasks are performed by the toolbox container. +- Migration tasks are performed by the migrations container. + +You should override the PostgreSQL port on each subchart, so these tasks can execute and connect to PostgreSQL directly: + +- [Toolbox](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/toolbox/values.yaml#L40) +- [Migrations](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/migrations/values.yaml#L46) + ## Fine tuning PgBouncer's default settings suit the majority of installations. diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index d089682f78e..5e96813103f 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -22,7 +22,7 @@ Prerequisite: - At least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0. -## Caveats +## Rate limit If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while importing, the importing process waits (`sleep()`) until it can continue importing. diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 4694af18af2..17a0eb46a30 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -4,61 +4,42 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project import/export administration **(FREE SELF)** +# Project import and export Rake tasks **(FREE SELF)** -GitLab provides Rake tasks relating to project import and export. For more information, see: +GitLab provides Rake tasks for [project import and export](../../user/project/settings/import_export.md). -- [Project import/export documentation](../../user/project/settings/import_export.md). -- [Project import/export API](../../api/project_import_export.md). -- [Developer documentation: project import/export](../../development/import_export.md) +You can only import from a [compatible](../../user/project/settings/import_export.md#compatibility) GitLab instance. -## Project import status +## Import large projects -You can query an import through the [Project import/export API](../../api/project_import_export.md#import-status). -As described in the API documentation, the query may return an import error or exceptions. +> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. -## Import/export Rake tasks +This script was introduced in GitLab 12.6 for importing large GitLab project exports. -The GitLab import/export version can be checked by using the following command: +As part of this script, we also disable direct upload. This avoids uploading a huge archive to GCS, which can cause idle transaction timeouts. -```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:version +We can run this script from the terminal: -# Installations from source -bundle exec rake gitlab:import_export:version RAILS_ENV=production -``` +Parameters: -The current list of DB tables to export can be listed by using the following command: +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | User name | +| `namespace_path` | string | yes | Namespace path | +| `project_path` | string | yes | Project path | +| `archive_path` | string | yes | Path to the exported project tarball you want to import | ```shell -# Omnibus installations -sudo gitlab-rake gitlab:import_export:data - -# Installations from source -bundle exec rake gitlab:import_export:data RAILS_ENV=production +bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" ``` -Note the following: - -- Importing is only possible if the version of the import and export GitLab instances are - [compatible](../../user/project/settings/import_export.md#compatibility). -- The project import option must be enabled: - - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Settings > General**. - 1. Expand **Visibility and access controls**. - 1. Under **Import sources**, check the "Project export enabled" option. - 1. Select **Save changes**. - -- The exports are stored in a temporary directory and are deleted every - 24 hours by a specific worker. - -### Import large projects using a Rake task +If you're running Omnibus, run the following Rake task: -If you have a larger project, consider using a Rake task as described in our [developer documentation](../../development/import_project.md#importing-via-a-rake-task). +```shell +gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" +``` -### Export using a Rake task +## Export large projects > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25598) in GitLab 12.9. @@ -88,3 +69,86 @@ IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, # Export EXPORT_DEBUG=true gitlab-rake "gitlab:import_export:export[root, group/subgroup, projectnametoexport, /tmp/export_file.tar.gz]" ``` + +Check the common errors listed below, what they mean, and how to fix them. + +### `Exception: undefined method 'name' for nil:NilClass` + +The `username` is not valid. + +### `Exception: undefined method 'full_path' for nil:NilClass` + +The `namespace_path` does not exist. +For example, one of the groups or subgroups is mistyped or missing, +or you've specified the project name in the path. + +The task only creates the project. +If you want to import it to a new group or subgroup, create it first. + +### `Exception: No such file or directory @ rb_sysopen - (filename)` + +The specified project export file in `archive_path` is missing. + +### `Exception: Permission denied @ rb_sysopen - (filename)` + +The specified project export file cannot be accessed by the `git` user. + +To fix the issue: + +1. Set the file owner to `git:git`. +1. Change the file permissions to `0400`. +1. Move the file to a public folder (for example `/tmp/`). + +### `Name can contain only letters, digits, emojis ...` + +```plaintext +Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, +digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start +with '-', end in '.git', or end in '.atom'. +``` + +The project name specified in `project_path` is not valid for one of the specified reasons. + +Only put the project name in `project_path`. For example, if you provide a path of subgroups +it fails with this error as `/` is not a valid character in a project name. + +### `Name has already been taken and Path has already been taken` + +A project with that name already exists. + +### `Exception: Error importing repository into (namespace) - No space left on device` + +The disk has insufficient space to complete the import. + +During import, the tarball is cached in your configured `shared_path` directory. Verify the +disk has enough free space to accommodate both the cached tarball and the unpacked +project files on disk. + +### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import + +If you receive a `Total number of not imported relations: XX` message, and issues +aren't created during the import, check [exceptions_json.log](../logs/index.md#exceptions_jsonlog). +You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`, +where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you +are likely hitting the issue with rebalancing of `relative_position` field of the issues. + +```ruby +# Check the current maximum value of relative_position +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) + +# Run the rebalancing process and check if the maximum value of relative_position has changed +Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) +``` + +Repeat the import attempt and check if the issues are imported successfully. + +### Gitaly calls error when importing + +If you're attempting to import a large project into a development environment, Gitaly might throw an error about too many calls or invocations. For example: + +```plaintext +Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? +``` + +This error is due to a [n+1 calls limit for development setups](../../development/gitaly.md#toomanyinvocationserror-errors). To resolve this error, set `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable. Then restart your development environment and import again. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 3718741e2e9..3842cf0846b 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -70,28 +70,28 @@ the database is read-only: in case things don't go as expected. 1. Enter PostgreSQL on the console as an administrator user: - ```shell - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/postgresql gitlabhq_production - ``` + ```shell + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` 1. Create the `gitlab_read_only` user. The password is set to `mypassword`, change that to your liking: - ```sql - -- NOTE: Use the password defined earlier - CREATE USER gitlab_read_only WITH password 'mypassword'; - GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only; - GRANT USAGE ON SCHEMA public TO gitlab_read_only; - GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only; - GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only; - - -- Tables created by "gitlab" should be made read-only for "gitlab_read_only" - -- automatically. - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only; - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only; - ``` + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_read_only WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only; + GRANT USAGE ON SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only; + + -- Tables created by "gitlab" should be made read-only for "gitlab_read_only" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only; + ``` 1. Get the hashed password of the `gitlab_read_only` user and copy the result: @@ -101,10 +101,10 @@ the database is read-only: 1. Edit `/etc/gitlab/gitlab.rb` and add the password from the previous step: - ```ruby - postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239' - postgresql['sql_user'] = "gitlab_read_only" - ``` + ```ruby + postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239' + postgresql['sql_user'] = "gitlab_read_only" + ``` 1. Reconfigure GitLab and restart PostgreSQL: diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 2e56884e309..a37794ec44b 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -766,6 +766,49 @@ redis['master'] = false You can find the relevant attributes defined in [`gitlab_rails.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb). +### Control startup behavior + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6646) in GitLab 15.10. + +To prevent the bundled Redis service from starting at boot or restarting after changing its configuration: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + redis['start_down'] = true + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +If you need to test a new replica node, you may set `start_down` to +`true` and manually start the node. After the new replica node is confirmed +working in the Redis cluster, set `start_down` to `false` and reconfigure GitLab +to ensure the node starts and restarts as expected during operation. + +### Control replica configuration + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6646) in GitLab 15.10. + +To prevent the `replicaof` line from rendering in the Redis configuration file: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + redis['set_replicaof'] = false + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +This setting can be used to prevent replication of a Redis node independently of other Redis settings. + ## Troubleshooting See the [Redis troubleshooting guide](troubleshooting.md). diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 23c9ce33c2d..16383db6c2a 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -32,15 +32,15 @@ Note the Redis node's IP address or hostname, port, and password (if required). 1. Configure the GitLab application servers with the appropriate connection details for your external Redis service in your `/etc/gitlab/gitlab.rb` file: - ```ruby - redis['enable'] = false + ```ruby + redis['enable'] = false - gitlab_rails['redis_host'] = 'redis.example.com' - gitlab_rails['redis_port'] = 6379 + gitlab_rails['redis_host'] = 'redis.example.com' + gitlab_rails['redis_port'] = 6379 - # Required if Redis authentication is configured on the Redis node - gitlab_rails['redis_password'] = 'Redis Password' - ``` + # Required if Redis authentication is configured on the Redis node + gitlab_rails['redis_password'] = 'Redis Password' + ``` 1. Reconfigure for the changes to take effect: diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 529621813aa..1934ca6bff0 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -170,7 +170,7 @@ To set up GitLab and its components to accommodate up to 10,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -1389,7 +1389,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1404,45 +1403,63 @@ Updates to example must be made at: # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1505,7 +1522,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1554,20 +1571,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1582,9 +1585,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1593,31 +1616,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1646,7 +1681,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS pass-through. Refer to the load balancers documentation on how to configure this. @@ -1668,9 +1703,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1751,7 +1792,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1760,22 +1801,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1936,7 +1966,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1945,22 +1975,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2200,9 +2219,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index d3aa6fef51e..adcf8eeb4c6 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -78,9 +78,9 @@ You can also optionally configure GitLab to use an [external PostgreSQL service] or an [external object storage service](../object_storage.md) for added performance and reliability at an increased complexity cost. -## Configure Advanced Search **(PREMIUM SELF)** +## Configure advanced search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 71fe8b301e2..08cb6e2cdff 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -170,7 +170,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -1406,7 +1406,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1415,51 +1414,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1522,7 +1539,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1571,20 +1588,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1599,9 +1602,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1610,31 +1633,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1663,7 +1698,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1685,9 +1720,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1768,7 +1809,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1777,22 +1818,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1955,7 +1985,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1964,22 +1994,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2218,9 +2237,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index ee26504902c..d87e8270dcb 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -101,7 +101,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. ## Configure the external load balancer @@ -459,7 +459,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: storage paths, enable the network listener, and to configure the token: NOTE: - You can't remove the `default` entry from `git_data_dirs` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + You can't remove the `default` entry from `gitaly['configuration'][:storage]` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). <!-- Updates to example must be made at: @@ -493,30 +493,48 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' + # The secret token is used for authentication callbacks from Gitaly to the GitLab internal API. + # This must match the respective value in GitLab Rails application setup. gitlab_shell['secret_token'] = 'shellsecret' # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + prometheus_listen_addr: '0.0.0.0:9236', + # Gitaly Auth Token + # Should be the same as praefect_internal_token + auth: { + # ... + # + # Gitaly's authentication token is used to authenticate gRPC requests to Gitaly. This must match + # the respective value in GitLab Rails application setup. + token: 'gitalysecret', + }, + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + pack_objects_cache: { + # ... + enabled: true, + }, + storage: [ + { + name: 'default', + path: '/var/opt/gitlab/git-data', + }, + { + name: 'storage1', + path: '/mnt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -574,9 +592,14 @@ To configure Gitaly with TLS: <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + gitaly['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:9999', + tls: { + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Delete `gitaly['listen_addr']` to allow only encrypted connections. @@ -911,9 +934,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search **(PREMIUM SELF)** +## Configure advanced search **(PREMIUM SELF)** -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 1d5dad02b18..c28799ed459 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -176,7 +176,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -1341,7 +1341,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1350,51 +1349,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1457,7 +1474,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1506,20 +1523,6 @@ Updates to example must be made at: # balancer. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1534,9 +1537,33 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + # Gitaly Auth Token + # Should be the same as praefect_internal_token + auth: { + # ... + token: '<praefect_internal_token>', + }, + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + pack_objects_cache: { + # ... + enabled: true, + }, + } + # # END user configuration ``` @@ -1545,31 +1572,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1598,7 +1637,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1620,9 +1659,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2165,9 +2210,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 3bcffa8f606..3f2771dda29 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -170,7 +170,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -1402,7 +1402,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1411,51 +1410,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1518,7 +1535,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs will be used as an example: @@ -1567,20 +1584,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1595,9 +1598,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1606,31 +1629,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1659,7 +1694,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1681,9 +1716,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1764,7 +1805,7 @@ Updates to example must be made at: # Redis ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1773,22 +1814,11 @@ Updates to example must be made at: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actioncable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -1958,7 +1988,7 @@ On each node perform the following: gitlab_rails['auto_migrate'] = false ## Redis connection details - ## First cluster that will host the cache + ## First cluster that will host the cache data gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ @@ -1967,22 +1997,11 @@ On each node perform the following: {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the persistent queues, shared state, and actionable - gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' - gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' + ## Second cluster that hosts all other persistent data + redis['master_name'] = 'gitlab-redis-persistent' + redis['master_password'] = '<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>' - gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.61', port: 26379}, - {host: '10.6.0.62', port: 26379}, - {host: '10.6.0.63', port: 26379}, - ] - gitlab_rails['redis_actioncable_sentinels'] = [ + gitlab_rails['redis_sentinels'] = [ {host: '10.6.0.61', port: 26379}, {host: '10.6.0.62', port: 26379}, {host: '10.6.0.63', port: 26379}, @@ -2217,9 +2236,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 691f71289c3..43d63d4f08e 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -173,7 +173,7 @@ To set up GitLab and its components to accommodate up to 5,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, +1. [Configure advanced search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can @@ -1338,7 +1338,6 @@ Updates to example must be made at: # Praefect Configuration praefect['enable'] = true - praefect['listen_addr'] = '0.0.0.0:2305' # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false @@ -1347,51 +1346,69 @@ Updates to example must be made at: # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section # - # Praefect External Token - # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster - praefect['auth_token'] = '<praefect_external_token>' - - # Praefect Database Settings - praefect['database_host'] = '10.6.0.141' - praefect['database_port'] = 5432 - # `no_proxy` settings must always be a direct connection for caching - praefect['database_direct_host'] = '10.6.0.141' - praefect['database_direct_port'] = 5432 - praefect['database_dbname'] = 'praefect_production' - praefect['database_user'] = 'praefect' - praefect['database_password'] = '<praefect_postgresql_password>' - - # Praefect Virtual Storage config - # Name of storage hash must match storage name in git_data_dirs on GitLab - # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') - praefect['virtual_storages'] = { - 'default' => { - 'nodes' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' + praefect['configuration'] = { + # ... + listen_addr: '0.0.0.0:2305', + auth: { + # ... + # + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + token: '<praefect_external_token>', + }, + # Praefect Database Settings + database: { + # ... + host: '10.6.0.141', + port: 5432, + # `no_proxy` settings must always be a direct connection for caching + session_pooled: { + # ... + host: '10.6.0.141', + port: 5432, + dbname: 'praefect_production', + user: 'praefect', + password: '<praefect_postgresql_password>', + }, + }, + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in gitaly['configuration'][:storage] on Gitaly nodes ('gitaly-1') + virtual_storage: [ + { + # ... + name: 'default', + node: [ + { + storage: 'gitaly-1', + address: 'tcp://10.6.0.91:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-2', + address: 'tcp://10.6.0.92:8075', + token: '<praefect_internal_token>' + }, + { + storage: 'gitaly-3', + address: 'tcp://10.6.0.93:8075', + token: '<praefect_internal_token>' + }, + ], }, - } - } + ], + # Set the network address Praefect will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9652', } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs @@ -1454,7 +1471,7 @@ to restrict access to the Gitaly server. Another option is to For configuring Gitaly you should note the following: -- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `gitaly['configuration'][:storage]` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` The following IPs are used as an example: @@ -1503,20 +1520,6 @@ Updates to example must be made at: # Gitaly gitaly['enable'] = true - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - - # Gitaly Auth Token - # Should be the same as praefect_internal_token - gitaly['auth_token'] = '<praefect_internal_token>' - - # Gitaly Pack-objects cache - # Recommended to be enabled for improved performance but can notably increase disk I/O - # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info - gitaly['pack_objects_cache_enabled'] = true - # Configure the Consul agent consul['enable'] = true ## Enable service discovery for Prometheus @@ -1531,9 +1534,29 @@ Updates to example must be made at: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } - # Set the network addresses that the exporters will listen on for monitoring + # Set the network address that the node exporter will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + + gitaly['configuration'] = { + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + listen_addr: '0.0.0.0:8075', + # Set the network address that Gitaly will listen on for monitoring + prometheus_listen_addr: '0.0.0.0:9236', + auth: { + # Gitaly Auth Token + # Should be the same as praefect_internal_token + token: '<praefect_internal_token>', + }, + pack_objects_cache: { + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + enabled: true, + }, + } + # # END user configuration ``` @@ -1542,31 +1565,43 @@ Updates to example must be made at: - On Gitaly node 1: ```ruby - git_data_dirs({ - "gitaly-1" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-1', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 2: ```ruby - git_data_dirs({ - "gitaly-2" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-2', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` - On Gitaly node 3: ```ruby - git_data_dirs({ - "gitaly-3" => { - "path" => "/var/opt/gitlab/git-data" - } - }) + gitaly['configuration'] = { + # ... + storage: [ + { + name: 'gitaly-3', + path: '/var/opt/gitlab/git-data', + }, + ], + } ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace @@ -1595,7 +1630,7 @@ Note the following: - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if - necessary. To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + necessary. To disable the unencrypted listener, set `praefect['configuration'][:listen_addr] = nil`. - The Internal Load Balancer will also access to the certificates and need to be configured to allow for TLS passthrough. Refer to the load balancers documentation on how to configure this. @@ -1617,9 +1652,15 @@ To configure Praefect with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - praefect['tls_listen_addr'] = "0.0.0.0:3305" - praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - praefect['key_path'] = "/etc/gitlab/ssl/key.pem" + praefect['configuration'] = { + # ... + tls_listen_addr: '0.0.0.0:3305', + tls: { + # ... + certificate_path: '/etc/gitlab/ssl/cert.pem', + key_path: '/etc/gitlab/ssl/key.pem', + }, + } ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2164,9 +2205,9 @@ GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search +## Configure advanced search -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +You can leverage Elasticsearch and [enable advanced search](../../integration/advanced_search/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. Elasticsearch cluster design and requirements are dependent on your specific diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 7b01efa183b..e4d07558306 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -513,51 +513,51 @@ The following table details the cost to run the different reference architecture </tr> <tr> <th scope="row">2k</th> - <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=0d3aff1f-ea3d-43f9-aa59-df49d27c35ca">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=3b3e3b81953737132789591d3a5179521943f1c0">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=25f66c35ba454bb98fb4034a8a50bb8c">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=15fc2bd9-5b1c-479d-bc46-d5ce096b8107">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=7e94eb8712f6845fdeb05e61f459598a91dac3cb">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24ac11fd947a4985ae9c9a5142649ad3">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=9a798136-53f2-4c35-be43-8e1e975a6663">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=ad4c9db623a214c92d780cd9dff33f444d62cf02">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=bcf23017ddfd40649fdc885cacd08d0c">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> - <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=cbe61840-31a1-487f-88fa-631251c2fde5">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=3e2970f919915a6337acea76a9f07655a1ecda4a">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=5748068be4864af6a34efb1cde685fa1">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> - <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator#id=b4b8b587-508a-4433-adc8-dc506bbe924f">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=32acaeaa93366110cd5fbf98a66a8a141db7adcb">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=24f878f20ee64b5cb64de459d34c8128">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> - <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td> + <td><a href="https://cloud.google.com/products/calculator/#id=48b4d817-d6cd-44b8-b069-0ba9a5d123ea">Calculated cost</a></td> <td></td> - <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td> + <td><a href="https://calculator.aws/#/estimate?id=5a0bba1338e3577d627ec97833dbc80ac9615562">Calculated cost</a></td> <td></td> - <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=3f973040ebc14023933d35f576c89846">Calculated cost</a></td> + <td><a href="https://azure.microsoft.com/en-us/pricing/calculator/?shared-estimate=4dd065eea2194d70b44d6d897e81f460">Calculated cost</a></td> </tr> </table> diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 6e97ffc3b47..5172a9613ee 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -1,6 +1,6 @@ --- -stage: Plan -group: Certify +stage: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 08c1df3d5eb..3f715e451a8 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -9,9 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can use [`git fsck`](https://git-scm.com/docs/git-fsck) to verify the integrity of all data committed to a repository. GitLab administrators can: -- Manually trigger this check for a project, using the GitLab UI. -- Schedule this check to run automatically for all projects. -- Run this check from the command line. +- [Manually trigger this check for a project](#check-a-projects-repository-using-gitlab-ui). +- [Schedule this check](#enable-repository-checks-for-all-projects) to run automatically for all projects. +- [Run this check from the command line](#run-a-check-using-the-command-line). - Run a [Rake task](raketasks/check.md#repository-integrity) for checking Git repositories, which can be used to run `git fsck` against all repositories and generate repository checksums, as a way to compare repositories on different servers. @@ -68,9 +68,13 @@ You can run [`git fsck`](https://git-scm.com/docs/git-fsck) using the command li 1. Run the check. For example: ```shell - sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck + sudo -u git /opt/gitlab/embedded/bin/git \ + -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck --no-dangling ``` + The error `fatal: detected dubious ownership in repository` means you're running the command + using the wrong account. For example, `root`. + ## What to do if a check failed If a repository check fails, locate the error in the [`repocheck.log` file](logs/index.md#repochecklog) on disk at: @@ -93,3 +97,26 @@ of date. The `commit-graph` cache is an auxiliary cache and is not required for While the message can be safely ignored, see the issue [error: Could not read from object database for commit-graph](https://gitlab.com/gitlab-org/gitaly/-/issues/2359) for more details. + +### Dangling commit, tag, or blob messages + +Repository check output often includes tags, blobs, and commits that must be pruned: + +```plaintext +dangling tag 5c6886c774b713a43158aae35c4effdb03a3ceca +dangling blob 3e268c23fcd736db92e89b31d9f267dd4a50ac4b +dangling commit 919ff61d8d78c2e3ea9a32701dff70ecbefdd1d7 +``` + +This is common in Git repositories. They're generated by operations like +force pushing to branches, because this generates a commit in the repository +that is not longer referred to by a ref or by another commit. + +If a repository check fails, the output is likely to include these warnings. + +Ignore these messages, and identify the root cause of the repository check failure +from the other output. + +[GitLab 15.8 and later](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5230) no +longer includes these messages in the check output. Use the `--no-dangling` option +to suppress then when run from the command line. diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 3d4f39b5ff0..87ae6fdb003 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -84,7 +84,7 @@ To create a Git hook that applies to all repositories, set a global server hook. Before creating a global server hook, you must choose a directory for it. -For Omnibus GitLab installations, the directory is set in `gitlab.rb` under `gitaly['custom_hooks_dir']`. You can either: +For Omnibus GitLab installations, the directory is set in `gitlab.rb` under `gitaly['configuration'][:hooks][:custom_hooks_dir]`. You can either: - Use the default suggestion of the `/var/opt/gitlab/gitaly/custom_hooks` directory by uncommenting it. - Add your own setting. diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index bf858476c0c..8645df55a62 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -32,12 +32,20 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T ## Gitaly - # Make Gitaly accept connections on all network interfaces - gitaly['listen_addr'] = "0.0.0.0:8075" - ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network - ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token - gitaly['auth_token'] = 'abc123secret' - praefect['auth_token'] = 'abc123secret' + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces + listen_addr: '0.0.0.0:8075', + auth: { + ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + token: 'abc123secret', + }, + } + + gitaly['auth_token'] = '' + praefect['configuration'][:auth][:token] = 'abc123secret' gitlab_rails['gitaly_token'] = 'abc123secret' ## Redis configuration @@ -170,7 +178,7 @@ Updates to example must be made at: # Replace <database_host> and <database_password> gitlab_rails['db_host'] = '<database_host>' - gitlab_rails['db_port'] = '5432' + gitlab_rails['db_port'] = 5432 gitlab_rails['db_password'] = '<database_password>' ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -257,7 +265,7 @@ To configure the metrics server: ```ruby sidekiq['metrics_enabled'] = true sidekiq['listen_address'] = "localhost" - sidekiq['listen_port'] = "8082" + sidekiq['listen_port'] = 8082 # Optionally log all the metrics server logs to log/sidekiq_exporter.log sidekiq['exporter_log_enabled'] = true @@ -299,7 +307,7 @@ To make health checks available from `localhost:8092`: ```ruby sidekiq['health_checks_enabled'] = true sidekiq['health_checks_listen_address'] = "localhost" - sidekiq['health_checks_listen_port'] = "8092" + sidekiq['health_checks_listen_port'] = 8092 ``` 1. Reconfigure GitLab: @@ -325,7 +333,7 @@ To enable LDAP with the synchronization worker for Sidekiq: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby gitlab_rails['ldap_enabled'] = true gitlab_rails['prevent_ldap_sign_in'] = false gitlab_rails['ldap_servers'] = { diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md index 61a154c8db2..0e4a0662277 100644 --- a/doc/administration/sidekiq/processing_specific_job_classes.md +++ b/doc/administration/sidekiq/processing_specific_job_classes.md @@ -17,7 +17,7 @@ job classes: 1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs inside the application to queue names configured by administrators. This lowers the load on Redis, which is important on very large-scale deployments. -1. [Queue selectors](#queue-selectors) perform the job selection outside the +1. [Queue selectors](#queue-selectors-deprecated) perform the job selection outside the application, when starting the Sidekiq process. This was used on GitLab.com until September 2021, and is retained for compatibility reasons. @@ -106,11 +106,19 @@ not a recommendation. sudo gitlab-ctl reconfigure ``` -## Queue selectors +<!--- start_remove The following content will be removed on remove_date: '2024-08-22' --> + +## Queue selectors (deprecated) > - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8. > - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9 +and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes). +Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change. The `queue_selector` option allows queue groups to be selected in a more general way using a [worker matching query](#worker-matching-query). After @@ -141,7 +149,12 @@ syntax. sudo gitlab-ctl reconfigure ``` -### Negate settings +### Negate settings (deprecated) + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9 +and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes). +Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change. This allows you to have the Sidekiq process work on every queue **except** the ones you list. This is generally only used when there are multiple Sidekiq @@ -168,7 +181,7 @@ nodes. In this example, we exclude all import-related jobs from a Sidekiq node. We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the [Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend -[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors). We use routing rules on GitLab.com as +[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors-deprecated). We use routing rules on GitLab.com as it helps to lower the load on Redis. To migrate from queue selectors to routing rules: @@ -255,11 +268,13 @@ in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_con concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU bound tasks. +<!--- end_remove --> + ## Worker matching query GitLab provides a query syntax to match a worker based on its attributes. This query syntax is employed by both [routing rules](#routing-rules) and -[queue selectors](#queue-selectors). A query includes two components: +[queue selectors](#queue-selectors-deprecated). A query includes two components: - Attributes that can be selected. - Operators used to construct a query. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index d3b941bd129..12d88c7c74a 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -22,9 +22,23 @@ Use [external object storage](https://docs.gitlab.com/charts/advanced/external-o ## Disabling Terraform state -To disable terraform state site-wide, follow the steps below. -A GitLab administrator may want to disable Terraform state to reduce disk space or if Terraform is not used in your instance. -To do so, follow the steps below according to your installation's type. +You can disable Terraform state across the entire instance. You might want to disable Terraform to reduce disk space, +or because your instance doesn't use Terraform. + +When Terraform state administration is disabled: + +- On the left sidebar, you cannot select **Infrastructure > Terraform**. +- Any CI/CD jobs that access the Terraform state fail with this error: + + ```shell + Error refreshing state: HTTP remote state endpoint invalid auth + ``` + +To disable Terraform administration, follow the steps below according to your installation. + +Prerequisite: + +- You must be an administrator. **In Omnibus installations:** @@ -79,7 +93,7 @@ Terraform state files are stored locally, follow the steps below. ## Using object storage **(FREE SELF)** Instead of storing Terraform state files on disk, we recommend the use of -[one of the supported object storage options](object_storage.md#options). +[one of the supported object storage options](object_storage.md#supported-object-storage-providers). This configuration relies on valid credentials to be configured already. [Read more about using object storage with GitLab](object_storage.md). diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index ff0b8ecf178..c4e2c352e9b 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -8,6 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w Uploads represent all user data that may be sent to GitLab as a single file. For example, avatars and note attachments are uploads. Uploads are integral to GitLab functionality and therefore cannot be disabled. +NOTE: +Attachments added to comments or descriptions are deleted **only** when the parent project or group +is deleted. Attachments remain in file storage even when the comment or resource (like issue, merge +request, epic) where they were uploaded is deleted. + ## Using local storage This is the default configuration. To change the location where the uploads are diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index fec719b189c..f42ace8c1de 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -253,13 +253,16 @@ Example response: ## Project Audit Events -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219238) in GitLab 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219238) in GitLab 13.1. +> - [Support for keyset pagination added](https://gitlab.com/gitlab-org/gitlab/-/issues/367528) in GitLab 15.10. The Project Audit Events API allows you to retrieve [project audit events](../administration/audit_events.md#project-events). A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. +When requesting consecutive pages of results, you should use [keyset pagination](rest/index.md#keyset-based-pagination). + ### Retrieve all project audit events ```plaintext diff --git a/doc/api/commits.md b/doc/api/commits.md index 5a3481ee086..7c4d15e5d80 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -20,6 +20,8 @@ information: ## List repository commits +> Commits by author [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114417) in GitLab 15.10. + Get a list of repository commits in a project. ```plaintext @@ -33,6 +35,7 @@ GET /projects/:id/repository/commits | `since` | string | no | Only commits after or on this date are returned in ISO 8601 format `YYYY-MM-DDTHH:MM:SSZ` | | `until` | string | no | Only commits before or on this date are returned in ISO 8601 format `YYYY-MM-DDTHH:MM:SSZ` | | `path` | string | no | The file path | +| `author` | string | no | Search commits by commit author.| | `all` | boolean | no | Retrieve every commit from the repository | | `with_stats` | boolean | no | Stats about each commit are added to the response | | `first_parent` | boolean | no | Follow only the first parent commit upon seeing a merge commit | @@ -525,6 +528,11 @@ cases below is valid: In any of the above cases, the response of `line`, `line_type` and `path` is set to `null`. +For other approaches to commenting on a merge request, see +[Create new merge request note](notes.md#create-new-merge-request-note) in the Notes API, +and [Create a new thread in the merge request diff](discussions.md#create-a-new-thread-in-the-merge-request-diff) +in the Discussions API. + ```plaintext POST /projects/:id/repository/commits/:sha/comments ``` diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index 684df9fdfdc..9b02f30b7ee 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index 2cfb58c25e1..0b7ea13c341 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/deployments.md b/doc/api/deployments.md index c6537493eab..eb718d80b78 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/discussions.md b/doc/api/discussions.md index ccef57dab7f..925d3ae6c9b 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -65,6 +65,7 @@ GET /projects/:id/issues/:issue_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Issue", + "project_id": 5, "noteable_iid": null }, { @@ -85,6 +86,7 @@ GET /projects/:id/issues/:issue_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Issue", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -112,6 +114,7 @@ GET /projects/:id/issues/:issue_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Issue", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -279,6 +282,7 @@ GET /projects/:id/snippets/:snippet_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Snippet", + "project_id": 5, "noteable_iid": null }, { @@ -299,6 +303,7 @@ GET /projects/:id/snippets/:snippet_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Snippet", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -326,6 +331,7 @@ GET /projects/:id/snippets/:snippet_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Snippet", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -491,6 +497,7 @@ GET /groups/:id/epics/:epic_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Epic", + "project_id": 5, "noteable_iid": null, "resolvable": false }, @@ -512,6 +519,7 @@ GET /groups/:id/epics/:epic_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Epic", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -539,6 +547,7 @@ GET /groups/:id/epics/:epic_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Epic", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -705,6 +714,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Merge request", + "project_id": 5, "noteable_iid": null, "resolved": false, "resolvable": true, @@ -729,6 +739,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Merge request", + "project_id": 5, "noteable_iid": null, "resolved": false, "resolvable": true, @@ -758,6 +769,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/discussions "system": false, "noteable_id": 3, "noteable_type": "Merge request", + "project_id": 5, "noteable_iid": null, "resolved": false, "resolvable": true, @@ -794,6 +806,7 @@ Diff comments also contain position: "system": false, "noteable_id": 3, "noteable_type": "Merge request", + "project_id": 5, "noteable_iid": null, "commit_id": "4803c71e6b1833ca72b8b26ef2ecd5adc8a38031", "position": { @@ -856,7 +869,9 @@ curl --header "PRIVATE-TOKEN: <your_access_token>"\ > The `commit id` entry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47130) in GitLab 13.7. Creates a new thread to a single project merge request. This is similar to creating -a note but other comments (replies) can be added to it later. +a note but other comments (replies) can be added to it later. For other approaches, +see [Post comment to commit](commits.md#post-comment-to-commit) in the Commits API, +and [Create new merge request note](notes.md#create-new-merge-request-note) in the Notes API. ```plaintext POST /projects/:id/merge_requests/:merge_request_iid/discussions @@ -1135,6 +1150,7 @@ GET /projects/:id/repository/commits/:commit_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Commit", + "project_id": 5, "noteable_iid": null, "resolvable": false }, @@ -1156,6 +1172,7 @@ GET /projects/:id/repository/commits/:commit_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Commit", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -1183,6 +1200,7 @@ GET /projects/:id/repository/commits/:commit_id/discussions "system": false, "noteable_id": 3, "noteable_type": "Commit", + "project_id": 5, "noteable_iid": null, "resolvable": false } @@ -1217,6 +1235,7 @@ Diff comments contain also position: "system": false, "noteable_id": 3, "noteable_type": "Commit", + "project_id": 5, "noteable_iid": null, "position": { "base_sha": "b5d6e7b1613fca24d250fa8e5bc7bcc3dd6002ef", diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index f0e90234ff4..a3865213714 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -101,8 +101,8 @@ parameter: | `metric` query parameter | Description of `value` in response | |:---------------------------|:-----------------------------------| +| `deployment_frequency` | The API returns the total number of successful deployments during the time period. [Issue 371271](https://gitlab.com/gitlab-org/gitlab/-/issues/371271) proposes to update the API to return the daily average instead of the total number. | | `change_failure_rate` | The number of incidents divided by the number of deployments during the time period. Available only for production environment. | -| `deployment_frequency` | The number of successful deployments during the time period. | | `lead_time_for_changes` | The median number of seconds between the merge of the merge request (MR) and the deployment of the MR commits for all MRs deployed during the time period. | | `time_to_restore_service` | The median number of seconds an incident was open during the time period. Available only for production environment. | diff --git a/doc/api/draft_notes.md b/doc/api/draft_notes.md index a168c41092c..079b08781ae 100644 --- a/doc/api/draft_notes.md +++ b/doc/api/draft_notes.md @@ -94,6 +94,46 @@ GET /projects/:id/merge_requests/:merge_request_iid/draft_notes/:draft_note_id curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/14/merge_requests/11/draft_notes/5" ``` +## Create a draft note + +Create a draft note for a given merge request. + +```plaintext +POST /projects/:id/merge_requests/:merge_request_iid/draft_notes +``` + +| Attribute | Type | Required | Description | +| --------------------------- | ----------------- | ----------- | --------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). +| `merge_request_iid` | integer | yes | The IID of a project merge request. +| `note` | string | yes | The content of a note. +| `commit_id` | string | no | The SHA of a commit to associate the draft note to. +| `in_reply_to_discussion_id` | integer | no | The ID of a discussion the draft note replies to. +| `resolve_discussion` | boolean | no | The associated discussion should be resolved. + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/14/merge_requests/11/draft_notes?note=note +``` + +## Modify existing draft note + +Modify a draft note for a given merge request. + +```plaintext +PUT /projects/:id/merge_requests/:merge_request_iid/draft_notes/:draft_note_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ----------------- | ----------- | --------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). +| `draft_note_id` | integer | yes | The ID of a draft note. +| `merge_request_iid` | integer | yes | The IID of a project merge request. +| `note` | string | no | The content of a note. + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/14/merge_requests/11/draft_notes/5" +``` + ## Delete a draft note Deletes an existing draft note for a given merge request. diff --git a/doc/api/environments.md b/doc/api/environments.md index bbf6c5fee99..2bd7b28d0d0 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index 36bcbb30d4b..3515b080b12 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -39,6 +39,45 @@ Example response: } ``` +### Create Error Tracking settings + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/393035/) in GitLab 15.10. +The API allows you to create Error Tracking settings for a project. Only for users with Maintainer role for +the project. + +NOTE: +This API is only available when used with [integrated error tracking](../operations/error_tracking.md#integrated-error-tracking). + +```plaintext +PUT /projects/:id/error_tracking/settings +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| ------------ | ------- |----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `active` | boolean | yes | Pass `true` to enable the error tracking setting configuration or `false` to disable it. | +| `integrated` | boolean | yes | Pass `true` to enable the integrated error tracking backend. [Available in](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68260) GitLab 14.2 and later. | + +Example request: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/error_tracking/settings?active=true&integrated=true" +``` + +Example response: + +```json +{ + "active": true, + "project_name": null, + "sentry_external_url": null, + "api_url": null, + "integrated": true +} +``` + ### Enable or disable the Error Tracking project settings The API allows you to enable or disable the Error Tracking settings for a project. Only for users with the @@ -55,7 +94,7 @@ PATCH /projects/:id/error_tracking/settings | `integrated` | boolean | no | Pass `true` to enable the integrated error tracking backend. [Available in](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68260) GitLab 14.2 and later. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/error_tracking/settings?active=true" +curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/error_tracking/settings?active=true" ``` Example response: diff --git a/doc/api/feature_flag_specs.md b/doc/api/feature_flag_specs.md deleted file mode 100644 index 960d00278d6..00000000000 --- a/doc/api/feature_flag_specs.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -stage: Release -group: Release -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-14' -redirect_to: 'feature_flags.md' ---- - -# Feature Flag Specs API (removed) **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in GitLab 12.5. - -This API was removed in [GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). -Use [the new API](feature_flags.md) instead. diff --git a/doc/api/feature_flag_user_lists.md b/doc/api/feature_flag_user_lists.md index 27e2e925506..0d48f8daf40 100644 --- a/doc/api/feature_flag_user_lists.md +++ b/doc/api/feature_flag_user_lists.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/feature_flags.md b/doc/api/feature_flags.md index cd9907c8032..f62d51ea427 100644 --- a/doc/api/feature_flags.md +++ b/doc/api/feature_flags.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/features.md b/doc/api/features.md index c3db1e53f68..2b0bf656664 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/freeze_periods.md b/doc/api/freeze_periods.md index ce7377e1e35..cb9bf81e961 100644 --- a/doc/api/freeze_periods.md +++ b/doc/api/freeze_periods.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index 0b0dd543503..ed413c358de 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -520,6 +520,13 @@ Example response: "container_repositories_registry_count": 5, "container_repositories_synced_in_percentage": "100.00%", "container_repositories_synced_missing_on_primary_count": 0, + "container_repositories_checksum_total_count": 0, + "container_repositories_checksummed_count": 0, + "container_repositories_checksum_failed_count": 0, + "container_repositories_verification_total_count": 0, + "container_repositories_verified_count": 0, + "container_repositories_verification_failed_count": 0, + "container_repositories_verified_in_percentage": "100.00%", "dependency_proxy_manifests_count": 5, "dependency_proxy_manifests_checksum_total_count": 5, "dependency_proxy_manifests_checksummed_count": 5, @@ -716,6 +723,13 @@ Example response: "container_repositories_registry_count": 5, "container_repositories_synced_in_percentage": "100.00%", "container_repositories_synced_missing_on_primary_count": 0, + "container_repositories_checksum_total_count": 0, + "container_repositories_checksummed_count": 0, + "container_repositories_checksum_failed_count": 0, + "container_repositories_verification_total_count": 0, + "container_repositories_verified_count": 0, + "container_repositories_verification_failed_count": 0, + "container_repositories_verified_in_percentage": "100.00%", "dependency_proxy_manifests_count": 5, "dependency_proxy_manifests_checksum_total_count": 5, "dependency_proxy_manifests_checksummed_count": 5, @@ -922,6 +936,13 @@ Example response: "container_repositories_registry_count": 5, "container_repositories_synced_in_percentage": "100.00%", "container_repositories_synced_missing_on_primary_count": 0, + "container_repositories_checksum_total_count": 0, + "container_repositories_checksummed_count": 0, + "container_repositories_checksum_failed_count": 0, + "container_repositories_verification_total_count": 0, + "container_repositories_verified_count": 0, + "container_repositories_verification_failed_count": 0, + "container_repositories_verified_in_percentage": "100.00%", "dependency_proxy_manifests_count": 5, "dependency_proxy_manifests_checksum_total_count": 5, "dependency_proxy_manifests_checksummed_count": 5, diff --git a/doc/api/graphql/getting_started.md b/doc/api/graphql/getting_started.md index 57d7880988b..df1b1cdca7c 100644 --- a/doc/api/graphql/getting_started.md +++ b/doc/api/graphql/getting_started.md @@ -16,9 +16,9 @@ the API itself. The examples documented here can be run using: -- The command line. -- GraphiQL. -- Rails console. +- [Command line](#command-line). +- [GraphiQL](#graphiql). +- [Rails console](#rails-console). ### Command line @@ -79,6 +79,7 @@ If you are running GitLab 12.0, enable the `graphql` GraphQL queries can be run in a [Rails console session](../../administration/operations/rails_console.md#starting-a-rails-console-session). For example, to search projects: ```ruby +current_user = User.find_by_id(1) query = <<~EOQ query securityGetProjects($search: String!) { projects(search: $search) { diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 7dc5b557d33..27da9f2b653 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -745,8 +745,36 @@ mutation($id: NoteableID!, $body: String!) { } ``` +### `Mutation.achievementsAward` + +WARNING: +**Introduced** in 15.10. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `AchievementsAwardInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationachievementsawardachievementid"></a>`achievementId` | [`AchievementsAchievementID!`](#achievementsachievementid) | Global ID of the achievement being awarded. | +| <a id="mutationachievementsawardclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationachievementsawarduserid"></a>`userId` | [`UserID!`](#userid) | Global ID of the user being awarded the achievement. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationachievementsawardclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationachievementsawarderrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationachievementsawarduserachievement"></a>`userAchievement` | [`UserAchievement`](#userachievement) | Achievement award. | + ### `Mutation.achievementsCreate` +WARNING: +**Introduced** in 15.8. +This feature is in Alpha. It can be changed or removed at any time. + Input type: `AchievementsCreateInput` #### Arguments @@ -767,6 +795,29 @@ Input type: `AchievementsCreateInput` | <a id="mutationachievementscreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationachievementscreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.achievementsRevoke` + +WARNING: +**Introduced** in 15.10. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `AchievementsRevokeInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationachievementsrevokeclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationachievementsrevokeuserachievementid"></a>`userAchievementId` | [`AchievementsUserAchievementID!`](#achievementsuserachievementid) | Global ID of the user achievement being revoked. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationachievementsrevokeclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationachievementsrevokeerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationachievementsrevokeuserachievement"></a>`userAchievement` | [`UserAchievement`](#userachievement) | Achievement award. | + ### `Mutation.addProjectToSecurityDashboard` Input type: `AddProjectToSecurityDashboardInput` @@ -1168,6 +1219,31 @@ Input type: `BoardListUpdateLimitMetricsInput` | <a id="mutationboardlistupdatelimitmetricserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationboardlistupdatelimitmetricslist"></a>`list` | [`BoardList`](#boardlist) | Updated list. | +### `Mutation.bulkDestroyJobArtifacts` + +WARNING: +**Introduced** in 15.10. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `BulkDestroyJobArtifactsInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationbulkdestroyjobartifactsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbulkdestroyjobartifactsids"></a>`ids` | [`[CiJobArtifactID!]!`](#cijobartifactid) | Global IDs of the job artifacts to destroy. | +| <a id="mutationbulkdestroyjobartifactsprojectid"></a>`projectId` | [`ProjectID!`](#projectid) | Global Project ID of the job artifacts to destroy. Incompatible with projectPath. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationbulkdestroyjobartifactsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbulkdestroyjobartifactsdestroyedcount"></a>`destroyedCount` | [`Int`](#int) | Number of job artifacts deleted. | +| <a id="mutationbulkdestroyjobartifactsdestroyedids"></a>`destroyedIds` | [`[CiJobArtifactID!]`](#cijobartifactid) | IDs of job artifacts that were deleted. | +| <a id="mutationbulkdestroyjobartifactserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.bulkEnableDevopsAdoptionNamespaces` **BETA** This endpoint is subject to change without notice. @@ -2401,6 +2477,26 @@ Input type: `DesignManagementMoveInput` | <a id="mutationdesignmanagementmovedesigncollection"></a>`designCollection` | [`DesignCollection`](#designcollection) | Current state of the collection. | | <a id="mutationdesignmanagementmoveerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.designManagementUpdate` + +Input type: `DesignManagementUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationdesignmanagementupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdesignmanagementupdatedescription"></a>`description` | [`String`](#string) | Description of the design. | +| <a id="mutationdesignmanagementupdateid"></a>`id` | [`DesignManagementDesignID!`](#designmanagementdesignid) | ID of the design to update. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationdesignmanagementupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdesignmanagementupdatedesign"></a>`design` | [`Design!`](#design) | Updated design. | +| <a id="mutationdesignmanagementupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.designManagementUpload` Input type: `DesignManagementUploadInput` @@ -3136,7 +3232,7 @@ Input type: `GroupMemberBulkUpdateInput` | <a id="mutationgroupmemberbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationgroupmemberbulkupdateexpiresat"></a>`expiresAt` | [`Time`](#time) | Date and time the membership expires. | | <a id="mutationgroupmemberbulkupdategroupid"></a>`groupId` | [`GroupID!`](#groupid) | Global ID of the group. | -| <a id="mutationgroupmemberbulkupdateuserids"></a>`userIds` | [`[UserID!]!`](#userid) | Global IDs of the group members. | +| <a id="mutationgroupmemberbulkupdateuserids"></a>`userIds` | [`[UserID!]!`](#userid) | Global IDs of the members. | #### Fields @@ -3649,12 +3745,18 @@ Input type: `IssuesBulkUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="mutationissuesbulkupdateaddlabelids"></a>`addLabelIds` | [`[LabelID!]`](#labelid) | Global ID array of the labels that will be added to the issues. | | <a id="mutationissuesbulkupdateassigneeids"></a>`assigneeIds` | [`[UserID!]`](#userid) | Global ID array of the users that will be assigned to the given issues. Existing assignees will be replaced with the ones on this list. | | <a id="mutationissuesbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationissuesbulkupdateepicid"></a>`epicId` | [`EpicID`](#epicid) | Global ID of the epic that will be assigned to the issues. | +| <a id="mutationissuesbulkupdatehealthstatus"></a>`healthStatus` | [`HealthStatus`](#healthstatus) | Health status that will be assigned to the issues. | | <a id="mutationissuesbulkupdateids"></a>`ids` | [`[IssueID!]!`](#issueid) | Global ID array of the issues that will be updated. IDs that the user can't update will be ignored. A max of 100 can be provided. | | <a id="mutationissuesbulkupdateiterationid"></a>`iterationId` | [`IterationID`](#iterationid) | Global ID of the iteration that will be assigned to the issues. | | <a id="mutationissuesbulkupdatemilestoneid"></a>`milestoneId` | [`MilestoneID`](#milestoneid) | Global ID of the milestone that will be assigned to the issues. | -| <a id="mutationissuesbulkupdateparentid"></a>`parentId` | [`IssueParentID!`](#issueparentid) | Global ID of the parent that the bulk update will be scoped to . Example `IssueParentID` are `"gid://gitlab/Project/1"` and `"gid://gitlab/Group/1"`. | +| <a id="mutationissuesbulkupdateparentid"></a>`parentId` | [`IssueParentID!`](#issueparentid) | Global ID of the parent to which the bulk update will be scoped. The parent can be a project **(FREE)** or a group **(PREMIUM)**. Example `IssueParentID` are `"gid://gitlab/Project/1"` and `"gid://gitlab/Group/1"`. | +| <a id="mutationissuesbulkupdateremovelabelids"></a>`removeLabelIds` | [`[LabelID!]`](#labelid) | Global ID array of the labels that will be removed from the issues. | +| <a id="mutationissuesbulkupdatestateevent"></a>`stateEvent` | [`IssueStateEvent`](#issuestateevent) | Close or reopen an issue. | +| <a id="mutationissuesbulkupdatesubscriptionevent"></a>`subscriptionEvent` | [`IssuableSubscriptionEvent`](#issuablesubscriptionevent) | Subscribe to or unsubscribe from issue notifications. | #### Fields @@ -4215,6 +4317,32 @@ Input type: `MergeRequestUpdateInput` | <a id="mutationmergerequestupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationmergerequestupdatemergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | +### `Mutation.mergeRequestUpdateApprovalRule` + +Input type: `MergeRequestUpdateApprovalRuleInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationmergerequestupdateapprovalruleapprovalruleid"></a>`approvalRuleId` | [`Int!`](#int) | ID of an approval rule. | +| <a id="mutationmergerequestupdateapprovalruleapprovalsrequired"></a>`approvalsRequired` | [`Int!`](#int) | Number of required approvals for a given rule. | +| <a id="mutationmergerequestupdateapprovalruleclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmergerequestupdateapprovalrulegroupids"></a>`groupIds` | [`[String!]`](#string) | IDs of groups as approvers. | +| <a id="mutationmergerequestupdateapprovalruleiid"></a>`iid` | [`String!`](#string) | IID of the merge request to mutate. | +| <a id="mutationmergerequestupdateapprovalrulename"></a>`name` | [`String!`](#string) | Name of the approval rule. | +| <a id="mutationmergerequestupdateapprovalruleprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the merge request to mutate is in. | +| <a id="mutationmergerequestupdateapprovalruleremovehiddengroups"></a>`removeHiddenGroups` | [`[Boolean!]`](#boolean) | Whether hidden groups should be removed. | +| <a id="mutationmergerequestupdateapprovalruleuserids"></a>`userIds` | [`[String!]`](#string) | IDs of users as approvers. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationmergerequestupdateapprovalruleclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmergerequestupdateapprovalruleerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationmergerequestupdateapprovalrulemergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | + ### `Mutation.namespaceBanDestroy` Input type: `NamespaceBanDestroyInput` @@ -4634,6 +4762,28 @@ Input type: `ProjectInitializeProductAnalyticsInput` | <a id="mutationprojectinitializeproductanalyticserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationprojectinitializeproductanalyticsproject"></a>`project` | [`Project`](#project) | Project on which the initialization took place. | +### `Mutation.projectMemberBulkUpdate` + +Input type: `ProjectMemberBulkUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectmemberbulkupdateaccesslevel"></a>`accessLevel` | [`MemberAccessLevel!`](#memberaccesslevel) | Access level to update the members to. | +| <a id="mutationprojectmemberbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectmemberbulkupdateexpiresat"></a>`expiresAt` | [`Time`](#time) | Date and time the membership expires. | +| <a id="mutationprojectmemberbulkupdateprojectid"></a>`projectId` | [`ProjectID!`](#projectid) | Global ID of the project. | +| <a id="mutationprojectmemberbulkupdateuserids"></a>`userIds` | [`[UserID!]!`](#userid) | Global IDs of the members. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectmemberbulkupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectmemberbulkupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationprojectmemberbulkupdateprojectmembers"></a>`projectMembers` | [`[ProjectMember!]`](#projectmember) | Project members after mutation. | + ### `Mutation.projectSetComplianceFramework` Assign (or unset) a compliance framework to a project. @@ -4677,6 +4827,30 @@ Input type: `ProjectSetLockedInput` | <a id="mutationprojectsetlockederrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationprojectsetlockedproject"></a>`project` | [`Project`](#project) | Project after mutation. | +### `Mutation.projectSyncFork` + +WARNING: +**Introduced** in 15.9. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `ProjectSyncForkInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectsyncforkclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectsyncforkprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project to initialize. | +| <a id="mutationprojectsyncforktargetbranch"></a>`targetBranch` | [`String!`](#string) | Ref of the fork to fetch into. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationprojectsyncforkclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationprojectsyncforkdetails"></a>`details` | [`ForkDetails`](#forkdetails) | Updated fork details. | +| <a id="mutationprojectsyncforkerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.prometheusIntegrationCreate` Input type: `PrometheusIntegrationCreateInput` @@ -4937,6 +5111,37 @@ Input type: `RepositionImageDiffNoteInput` | <a id="mutationrepositionimagediffnoteerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationrepositionimagediffnotenote"></a>`note` | [`Note`](#note) | Note after mutation. | +### `Mutation.runnerCreate` + +WARNING: +**Introduced** in 15.10. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `RunnerCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationrunnercreateaccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel`](#cirunneraccesslevel) | Access level of the runner. | +| <a id="mutationrunnercreateassociatedprojects"></a>`associatedProjects` | [`[ProjectID!]`](#projectid) | Projects associated with the runner. Available only for project runners. | +| <a id="mutationrunnercreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnercreatedescription"></a>`description` | [`String`](#string) | Description of the runner. | +| <a id="mutationrunnercreatelocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | +| <a id="mutationrunnercreatemaintenancenote"></a>`maintenanceNote` | [`String`](#string) | Runner's maintenance notes. | +| <a id="mutationrunnercreatemaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | +| <a id="mutationrunnercreatepaused"></a>`paused` | [`Boolean`](#boolean) | Indicates the runner is not allowed to receive jobs. | +| <a id="mutationrunnercreaterununtagged"></a>`runUntagged` | [`Boolean`](#boolean) | Indicates the runner is able to run untagged jobs. | +| <a id="mutationrunnercreatetaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationrunnercreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnercreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationrunnercreaterunner"></a>`runner` | [`CiRunner`](#cirunner) | Runner after mutation. | + ### `Mutation.runnerDelete` Input type: `RunnerDeleteInput` @@ -5130,6 +5335,7 @@ Input type: `SecurityFindingDismissInput` | ---- | ---- | ----------- | | <a id="mutationsecurityfindingdismissclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationsecurityfindingdismisserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationsecurityfindingdismisssecurityfinding"></a>`securityFinding` | [`PipelineSecurityReportFinding`](#pipelinesecurityreportfinding) | Dismissed finding. | | <a id="mutationsecurityfindingdismissuuid"></a>`uuid` | [`String`](#string) | UUID of dismissed finding. | ### `Mutation.securityFindingRevertToDetected` @@ -6049,6 +6255,7 @@ Input type: `VulnerabilityConfirmInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationvulnerabilityconfirmclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationvulnerabilityconfirmcomment"></a>`comment` | [`String`](#string) | Comment why vulnerability was marked as confirmed (max. 50 000 characters). | | <a id="mutationvulnerabilityconfirmid"></a>`id` | [`VulnerabilityID!`](#vulnerabilityid) | ID of the vulnerability to be confirmed. | #### Fields @@ -6186,6 +6393,7 @@ Input type: `VulnerabilityResolveInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationvulnerabilityresolveclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationvulnerabilityresolvecomment"></a>`comment` | [`String`](#string) | Comment why vulnerability was reverted to detected (max. 50 000 characters). | | <a id="mutationvulnerabilityresolveid"></a>`id` | [`VulnerabilityID!`](#vulnerabilityid) | ID of the vulnerability to be resolved. | #### Fields @@ -6205,6 +6413,7 @@ Input type: `VulnerabilityRevertToDetectedInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationvulnerabilityreverttodetectedclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationvulnerabilityreverttodetectedcomment"></a>`comment` | [`String`](#string) | Comment why vulnerability was reverted to detected (max. 50 000 characters). | | <a id="mutationvulnerabilityreverttodetectedid"></a>`id` | [`VulnerabilityID!`](#vulnerabilityid) | ID of the vulnerability to be reverted. | #### Fields @@ -6326,6 +6535,35 @@ Input type: `WorkItemDeleteTaskInput` | <a id="mutationworkitemdeletetaskerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationworkitemdeletetaskworkitem"></a>`workItem` | [`WorkItem`](#workitem) | Updated work item. | +### `Mutation.workItemExport` + +WARNING: +**Introduced** in 15.10. +This feature is in Alpha. It can be changed or removed at any time. + +Input type: `WorkItemExportInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemexportauthorusername"></a>`authorUsername` **{warning-solid}** | [`String`](#string) | **Deprecated:** This feature is in Alpha. It can be changed or removed at any time. Introduced in 15.9. | +| <a id="mutationworkitemexportclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemexportiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of work items. For example, `["1", "2"]`. | +| <a id="mutationworkitemexportin"></a>`in` | [`[IssuableSearchableField!]`](#issuablesearchablefield) | Specify the fields to perform the search in. Defaults to `[TITLE, DESCRIPTION]`. Requires the `search` argument.'. | +| <a id="mutationworkitemexportprojectpath"></a>`projectPath` | [`ID!`](#id) | Full project path. | +| <a id="mutationworkitemexportsearch"></a>`search` | [`String`](#string) | Search query for title or description. | +| <a id="mutationworkitemexportselectedfields"></a>`selectedFields` | [`[AvailableExportFields!]`](#availableexportfields) | List of selected fields to be exported. Omit to export all available fields. | +| <a id="mutationworkitemexportstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of the work item. | +| <a id="mutationworkitemexporttypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter work items by the given work item types. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationworkitemexportclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemexporterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.workItemUpdate` Updates a work item by Global ID. @@ -6350,6 +6588,7 @@ Input type: `WorkItemUpdateInput` | <a id="mutationworkitemupdateiterationwidget"></a>`iterationWidget` | [`WorkItemWidgetIterationInput`](#workitemwidgetiterationinput) | Input for iteration widget. | | <a id="mutationworkitemupdatelabelswidget"></a>`labelsWidget` | [`WorkItemWidgetLabelsUpdateInput`](#workitemwidgetlabelsupdateinput) | Input for labels widget. | | <a id="mutationworkitemupdatemilestonewidget"></a>`milestoneWidget` | [`WorkItemWidgetMilestoneInput`](#workitemwidgetmilestoneinput) | Input for milestone widget. | +| <a id="mutationworkitemupdatenotificationswidget"></a>`notificationsWidget` | [`WorkItemWidgetNotificationsUpdateInput`](#workitemwidgetnotificationsupdateinput) | Input for notifications widget. | | <a id="mutationworkitemupdateprogresswidget"></a>`progressWidget` | [`WorkItemWidgetProgressInput`](#workitemwidgetprogressinput) | Input for progress widget. | | <a id="mutationworkitemupdatestartandduedatewidget"></a>`startAndDueDateWidget` | [`WorkItemWidgetStartAndDueDateUpdateInput`](#workitemwidgetstartandduedateupdateinput) | Input for start and due date widget. | | <a id="mutationworkitemupdatestateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | @@ -7096,6 +7335,30 @@ The edge type for [`CiRunner`](#cirunner). | <a id="cirunneredgenode"></a>`node` | [`CiRunner`](#cirunner) | The item at the end of the edge. | | <a id="cirunneredgeweburl"></a>`webUrl` | [`String`](#string) | Web URL of the runner. The value depends on where you put this field in the query. You can use it for projects or groups. | +#### `CiRunnerMachineConnection` + +The connection type for [`CiRunnerMachine`](#cirunnermachine). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="cirunnermachineconnectioncount"></a>`count` | [`Int!`](#int) | Total count of collection. | +| <a id="cirunnermachineconnectionedges"></a>`edges` | [`[CiRunnerMachineEdge]`](#cirunnermachineedge) | A list of edges. | +| <a id="cirunnermachineconnectionnodes"></a>`nodes` | [`[CiRunnerMachine]`](#cirunnermachine) | A list of nodes. | +| <a id="cirunnermachineconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CiRunnerMachineEdge` + +The edge type for [`CiRunnerMachine`](#cirunnermachine). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="cirunnermachineedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="cirunnermachineedgenode"></a>`node` | [`CiRunnerMachine`](#cirunnermachine) | The item at the end of the edge. | + #### `CiSecureFileRegistryConnection` The connection type for [`CiSecureFileRegistry`](#cisecurefileregistry). @@ -7630,6 +7893,29 @@ The edge type for [`DastSiteValidation`](#dastsitevalidation). | <a id="dastsitevalidationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="dastsitevalidationedgenode"></a>`node` | [`DastSiteValidation`](#dastsitevalidation) | The item at the end of the edge. | +#### `DependencyConnection` + +The connection type for [`Dependency`](#dependency). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="dependencyconnectionedges"></a>`edges` | [`[DependencyEdge]`](#dependencyedge) | A list of edges. | +| <a id="dependencyconnectionnodes"></a>`nodes` | [`[Dependency]`](#dependency) | A list of nodes. | +| <a id="dependencyconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `DependencyEdge` + +The edge type for [`Dependency`](#dependency). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="dependencyedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="dependencyedgenode"></a>`node` | [`Dependency`](#dependency) | The item at the end of the edge. | + #### `DependencyProxyBlobConnection` The connection type for [`DependencyProxyBlob`](#dependencyproxyblob). @@ -10273,6 +10559,29 @@ The edge type for [`UsageTrendsMeasurement`](#usagetrendsmeasurement). | <a id="usagetrendsmeasurementedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="usagetrendsmeasurementedgenode"></a>`node` | [`UsageTrendsMeasurement`](#usagetrendsmeasurement) | The item at the end of the edge. | +#### `UserAchievementConnection` + +The connection type for [`UserAchievement`](#userachievement). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="userachievementconnectionedges"></a>`edges` | [`[UserAchievementEdge]`](#userachievementedge) | A list of edges. | +| <a id="userachievementconnectionnodes"></a>`nodes` | [`[UserAchievement]`](#userachievement) | A list of nodes. | +| <a id="userachievementconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `UserAchievementEdge` + +The edge type for [`UserAchievement`](#userachievement). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="userachievementedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="userachievementedgenode"></a>`node` | [`UserAchievement`](#userachievement) | The item at the end of the edge. | + #### `UserCalloutConnection` The connection type for [`UserCallout`](#usercallout). @@ -10457,6 +10766,29 @@ The edge type for [`VulnerabilityScanner`](#vulnerabilityscanner). | <a id="vulnerabilityscanneredgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="vulnerabilityscanneredgenode"></a>`node` | [`VulnerabilityScanner`](#vulnerabilityscanner) | The item at the end of the edge. | +#### `VulnerabilityStateTransitionTypeConnection` + +The connection type for [`VulnerabilityStateTransitionType`](#vulnerabilitystatetransitiontype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitystatetransitiontypeconnectionedges"></a>`edges` | [`[VulnerabilityStateTransitionTypeEdge]`](#vulnerabilitystatetransitiontypeedge) | A list of edges. | +| <a id="vulnerabilitystatetransitiontypeconnectionnodes"></a>`nodes` | [`[VulnerabilityStateTransitionType]`](#vulnerabilitystatetransitiontype) | A list of nodes. | +| <a id="vulnerabilitystatetransitiontypeconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `VulnerabilityStateTransitionTypeEdge` + +The edge type for [`VulnerabilityStateTransitionType`](#vulnerabilitystatetransitiontype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitystatetransitiontypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="vulnerabilitystatetransitiontypeedgenode"></a>`node` | [`VulnerabilityStateTransitionType`](#vulnerabilitystatetransitiontype) | The item at the end of the edge. | + #### `WorkItemConnection` The connection type for [`WorkItem`](#workitem). @@ -10570,6 +10902,7 @@ Representation of a GitLab user. | <a id="achievementname"></a>`name` | [`String!`](#string) | Name of the achievement. | | <a id="achievementnamespace"></a>`namespace` | [`Namespace!`](#namespace) | Namespace of the achievement. | | <a id="achievementupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp the achievement was last updated. | +| <a id="achievementuserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Recipients for the achievement. | ### `AgentConfiguration` @@ -11390,6 +11723,7 @@ CI/CD variables for a GitLab instance. | <a id="cijoballowfailure"></a>`allowFailure` | [`Boolean!`](#boolean) | Whether the job is allowed to fail. | | <a id="cijobartifacts"></a>`artifacts` | [`CiJobArtifactConnection`](#cijobartifactconnection) | Artifacts generated by the job. (see [Connections](#connections)) | | <a id="cijobbrowseartifactspath"></a>`browseArtifactsPath` | [`String`](#string) | URL for browsing the artifact's archive. | +| <a id="cijobcanplayjob"></a>`canPlayJob` | [`Boolean!`](#boolean) | Indicates whether the current user can play the job. | | <a id="cijobcancelable"></a>`cancelable` | [`Boolean!`](#boolean) | Indicates the job can be canceled. | | <a id="cijobcommitpath"></a>`commitPath` | [`String`](#string) | Path to the commit that triggered the job. | | <a id="cijobcoverage"></a>`coverage` | [`Float`](#float) | Coverage level of the job. | @@ -11407,6 +11741,7 @@ CI/CD variables for a GitLab instance. | <a id="cijobname"></a>`name` | [`String`](#string) | Name of the job. | | <a id="cijobneeds"></a>`needs` | [`CiBuildNeedConnection`](#cibuildneedconnection) | References to builds that must complete before the jobs run. (see [Connections](#connections)) | | <a id="cijobpipeline"></a>`pipeline` | [`Pipeline`](#pipeline) | Pipeline the job belongs to. | +| <a id="cijobplaypath"></a>`playPath` | [`String`](#string) | Play path of the job. | | <a id="cijobplayable"></a>`playable` | [`Boolean!`](#boolean) | Indicates the job can be played. | | <a id="cijobpreviousstagejobsorneeds"></a>`previousStageJobsOrNeeds` | [`JobNeedUnionConnection`](#jobneedunionconnection) | Jobs that must complete before the job runs. Returns `BuildNeed`, which is the needed jobs if the job uses the `needs` keyword, or the previous stage jobs otherwise. (see [Connections](#connections)) | | <a id="cijobproject"></a>`project` | [`Project`](#project) | Project that the job belongs to. | @@ -11416,6 +11751,8 @@ CI/CD variables for a GitLab instance. | <a id="cijobrefpath"></a>`refPath` | [`String`](#string) | Path to the ref. | | <a id="cijobretried"></a>`retried` | [`Boolean`](#boolean) | Indicates that the job has been retried. | | <a id="cijobretryable"></a>`retryable` | [`Boolean!`](#boolean) | Indicates the job can be retried. | +| <a id="cijobrunnermachine"></a>`runnerMachine` **{warning-solid}** | [`CiRunnerMachine`](#cirunnermachine) | **Introduced** in 15.11. This feature is in Alpha. It can be changed or removed at any time. Runner machine assigned to the job. | +| <a id="cijobscheduled"></a>`scheduled` | [`Boolean!`](#boolean) | Indicates the job is scheduled. | | <a id="cijobscheduledat"></a>`scheduledAt` | [`Time`](#time) | Schedule for the build. | | <a id="cijobschedulingtype"></a>`schedulingType` | [`String`](#string) | Type of job scheduling. Value is `dag` if the job uses the `needs` keyword, and `stage` otherwise. | | <a id="cijobshortsha"></a>`shortSha` | [`String!`](#string) | Short SHA1 ID of the commit. | @@ -11542,9 +11879,10 @@ CI/CD variables for a project. | <a id="cirunnerarchitecturename"></a>`architectureName` | [`String`](#string) | Architecture provided by the the runner. | | <a id="cirunnercontactedat"></a>`contactedAt` | [`Time`](#time) | Timestamp of last contact from this runner. | | <a id="cirunnercreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of creation of this runner. | +| <a id="cirunnercreatedby"></a>`createdBy` | [`UserCore`](#usercore) | User that created this runner. | | <a id="cirunnerdescription"></a>`description` | [`String`](#string) | Description of the runner. | | <a id="cirunnereditadminurl"></a>`editAdminUrl` | [`String`](#string) | Admin form URL of the runner. Only available for administrators. | -| <a id="cirunnerephemeralauthenticationtoken"></a>`ephemeralAuthenticationToken` **{warning-solid}** | [`String`](#string) | **Introduced** in 15.9. This feature is in Alpha. It can be changed or removed at any time. Ephemeral authentication token used for runner machine registration. | +| <a id="cirunnerephemeralauthenticationtoken"></a>`ephemeralAuthenticationToken` **{warning-solid}** | [`String`](#string) | **Introduced** in 15.9. This feature is in Alpha. It can be changed or removed at any time. Ephemeral authentication token used for runner machine registration. Only available for the creator of the runner for a limited time during registration. | | <a id="cirunnerexecutorname"></a>`executorName` | [`String`](#string) | Executor last advertised by the runner. | | <a id="cirunnergroups"></a>`groups` | [`GroupConnection`](#groupconnection) | Groups the runner is associated with. For group runners only. (see [Connections](#connections)) | | <a id="cirunnerid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner. | @@ -11552,6 +11890,7 @@ CI/CD variables for a project. | <a id="cirunnerjobcount"></a>`jobCount` | [`Int`](#int) | Number of jobs processed by the runner (limited to 1000, plus one to indicate that more items exist). | | <a id="cirunnerjobexecutionstatus"></a>`jobExecutionStatus` **{warning-solid}** | [`CiRunnerJobExecutionStatus`](#cirunnerjobexecutionstatus) | **Introduced** in 15.7. This feature is in Alpha. It can be changed or removed at any time. Job execution status of the runner. | | <a id="cirunnerlocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | +| <a id="cirunnermachines"></a>`machines` **{warning-solid}** | [`CiRunnerMachineConnection`](#cirunnermachineconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Machines associated with the runner configuration. | | <a id="cirunnermaintenancenote"></a>`maintenanceNote` | [`String`](#string) | Runner's maintenance notes. | | <a id="cirunnermaintenancenotehtml"></a>`maintenanceNoteHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `maintenance_note`. | | <a id="cirunnermaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | @@ -11561,6 +11900,7 @@ CI/CD variables for a project. | <a id="cirunnerprivateprojectsminutescostfactor"></a>`privateProjectsMinutesCostFactor` | [`Float`](#float) | Private projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="cirunnerprojectcount"></a>`projectCount` | [`Int`](#int) | Number of projects that the runner is associated with. | | <a id="cirunnerpublicprojectsminutescostfactor"></a>`publicProjectsMinutesCostFactor` | [`Float`](#float) | Public projects' "minutes cost factor" associated with the runner (GitLab.com only). | +| <a id="cirunnerregisteradminurl"></a>`registerAdminUrl` | [`String`](#string) | URL of the temporary registration page of the runner. Only available before the runner is registered. Only available for administrators. | | <a id="cirunnerrevision"></a>`revision` | [`String`](#string) | Revision of the runner. | | <a id="cirunnerrununtagged"></a>`runUntagged` | [`Boolean!`](#boolean) | Indicates the runner is able to run untagged jobs. | | <a id="cirunnerrunnertype"></a>`runnerType` | [`CiRunnerType!`](#cirunnertype) | Type of the runner. | @@ -11621,6 +11961,25 @@ Returns [`CiRunnerStatus!`](#cirunnerstatus). | ---- | ---- | ----------- | | <a id="cirunnerstatuslegacymode"></a>`legacyMode` **{warning-solid}** | [`String`](#string) | **Deprecated** in 15.0. Will be removed in 17.0. In GitLab 16.0 and later, the field will act as if `legacyMode` is null. | +### `CiRunnerMachine` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="cirunnermachinearchitecturename"></a>`architectureName` | [`String`](#string) | Architecture provided by the runner machine. | +| <a id="cirunnermachinecontactedat"></a>`contactedAt` | [`Time`](#time) | Timestamp of last contact from the runner machine. | +| <a id="cirunnermachinecreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of creation of the runner machine. | +| <a id="cirunnermachineexecutorname"></a>`executorName` | [`String`](#string) | Executor last advertised by the runner. | +| <a id="cirunnermachineid"></a>`id` | [`CiRunnerMachineID!`](#cirunnermachineid) | ID of the runner machine. | +| <a id="cirunnermachineipaddress"></a>`ipAddress` | [`String`](#string) | IP address of the runner machine. | +| <a id="cirunnermachineplatformname"></a>`platformName` | [`String`](#string) | Platform provided by the runner machine. | +| <a id="cirunnermachinerevision"></a>`revision` | [`String`](#string) | Revision of the runner. | +| <a id="cirunnermachinerunner"></a>`runner` | [`CiRunner`](#cirunner) | Runner configuration for the runner machine. | +| <a id="cirunnermachinestatus"></a>`status` | [`CiRunnerStatus!`](#cirunnerstatus) | Status of the runner machine. | +| <a id="cirunnermachinesystemid"></a>`systemId` | [`String!`](#string) | System ID associated with the runner machine. | +| <a id="cirunnermachineversion"></a>`version` | [`String`](#string) | Version of the runner. | + ### `CiSecureFileRegistry` Represents the Geo replication and verification state of a ci_secure_file. @@ -12169,8 +12528,9 @@ Represents a DAST Pre Scan Verification Step. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="dastprescanverificationstepchecktype"></a>`checkType` | [`DastPreScanVerificationCheckType`](#dastprescanverificationchecktype) | Type of the pre scan verification check. | | <a id="dastprescanverificationsteperrors"></a>`errors` | [`[String!]`](#string) | Errors that occurred in the pre scan verification step. | -| <a id="dastprescanverificationstepname"></a>`name` | [`String`](#string) | Name of the pre scan verification step. | +| <a id="dastprescanverificationstepname"></a>`name` **{warning-solid}** | [`String`](#string) | **Deprecated** in 15.10. This was renamed. Use: [`DastPreScanVerificationStep.checkType`](#dastprescanverificationstepchecktype). | | <a id="dastprescanverificationstepsuccess"></a>`success` | [`Boolean!`](#boolean) | Whether or not the pre scan verification step has errors. | ### `DastProfile` @@ -12334,6 +12694,20 @@ The response from the AdminSidekiqQueuesDeleteJobs mutation. | <a id="deletednoteid"></a>`id` | [`NoteID!`](#noteid) | ID of the deleted note. | | <a id="deletednotelastdiscussionnote"></a>`lastDiscussionNote` | [`Boolean`](#boolean) | Whether deleted note is the last note in the discussion. | +### `Dependency` + +A software dependency used by a project. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="dependencyid"></a>`id` | [`GlobalID!`](#globalid) | ID of the dependency. | +| <a id="dependencylocation"></a>`location` | [`Location`](#location) | Information about where the dependency is located. | +| <a id="dependencyname"></a>`name` | [`String!`](#string) | Name of the dependency. | +| <a id="dependencypackager"></a>`packager` | [`String`](#string) | Description of the tool used to manage the dependency. | +| <a id="dependencyversion"></a>`version` | [`String`](#string) | Version of the dependency. | + ### `DependencyProxyBlob` Dependency proxy blob. @@ -12533,6 +12907,8 @@ A single design. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="designcommenters"></a>`commenters` | [`UserCoreConnection!`](#usercoreconnection) | All commenters on this noteable. (see [Connections](#connections)) | +| <a id="designdescription"></a>`description` | [`String`](#string) | Description of the design. | +| <a id="designdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="designdiffrefs"></a>`diffRefs` | [`DiffRefs!`](#diffrefs) | Diff refs for this design. | | <a id="designdiscussions"></a>`discussions` | [`DiscussionConnection!`](#discussionconnection) | All discussions on this noteable. (see [Connections](#connections)) | | <a id="designevent"></a>`event` | [`DesignVersionEvent!`](#designversionevent) | How this design was changed in the current version. | @@ -12925,10 +13301,10 @@ Returns [`[DoraMetric!]`](#dorametric). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="dorametricsenddate"></a>`endDate` | [`Date`](#date) | Date range to end at. Default is the current date. | -| <a id="dorametricsenvironmenttier"></a>`environmentTier` | [`DeploymentTier`](#deploymenttier) | Deployment tier of the environments to return. Deprecated, please update to `environment_tiers` param. | +| <a id="dorametricsenvironmenttier"></a>`environmentTier` **{warning-solid}** | [`DeploymentTier`](#deploymenttier) | **Deprecated** in 15.2. Superseded by `environment_tiers` param. | | <a id="dorametricsenvironmenttiers"></a>`environmentTiers` | [`[DeploymentTier!]`](#deploymenttier) | Deployment tiers of the environments to return. Defaults to `[PRODUCTION]`. | -| <a id="dorametricsinterval"></a>`interval` | [`DoraMetricBucketingInterval`](#dorametricbucketinginterval) | How the metric should be aggregrated. Defaults to `DAILY`. In the case of `ALL`, the `date` field in the response will be `null`. | -| <a id="dorametricsmetric"></a>`metric` | [`DoraMetricType!`](#dorametrictype) | Type of metric to return. | +| <a id="dorametricsinterval"></a>`interval` | [`DoraMetricBucketingInterval`](#dorametricbucketinginterval) | How the metric should be aggregated. Defaults to `DAILY`. In the case of `ALL`, the `date` field in the response will be `null`. | +| <a id="dorametricsmetric"></a>`metric` **{warning-solid}** | [`DoraMetricType`](#dorametrictype) | **Deprecated** in 15.10. Superseded by metrics fields. See `DoraMetric` type. | | <a id="dorametricsstartdate"></a>`startDate` | [`Date`](#date) | Date range to start from. Default is 3 months ago. | ### `DoraMetric` @@ -12937,8 +13313,12 @@ Returns [`[DoraMetric!]`](#dorametric). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="dorametricchangefailurerate"></a>`changeFailureRate` | [`Float`](#float) | Percentage of deployments that caused incidents in production. | | <a id="dorametricdate"></a>`date` | [`String`](#string) | Date of the data point. | -| <a id="dorametricvalue"></a>`value` | [`Float`](#float) | Value of the data point. | +| <a id="dorametricdeploymentfrequency"></a>`deploymentFrequency` | [`Float`](#float) | Number of deployments per day. | +| <a id="dorametricleadtimeforchanges"></a>`leadTimeForChanges` | [`Float`](#float) | Median time to deploy a merged merge request. | +| <a id="dorametrictimetorestoreservice"></a>`timeToRestoreService` | [`Float`](#float) | Median time to close an incident. | +| <a id="dorametricvalue"></a>`value` **{warning-solid}** | [`Float`](#float) | **Deprecated** in 15.10. Moved to corresponding metric field. | ### `EgressNode` @@ -13597,6 +13977,8 @@ Details of the fork project compared to its upstream project. | ---- | ---- | ----------- | | <a id="forkdetailsahead"></a>`ahead` | [`Int`](#int) | Number of commits ahead of upstream. | | <a id="forkdetailsbehind"></a>`behind` | [`Int`](#int) | Number of commits behind upstream. | +| <a id="forkdetailshasconflicts"></a>`hasConflicts` | [`Boolean`](#boolean) | Indicates if the fork conflicts with its upstream project. | +| <a id="forkdetailsissyncing"></a>`isSyncing` | [`Boolean`](#boolean) | Indicates if there is a synchronization in progress. | ### `GeoNode` @@ -13952,6 +14334,7 @@ GPG signature for a signed commit. | <a id="groupepicboards"></a>`epicBoards` | [`EpicBoardConnection`](#epicboardconnection) | Find epic boards. (see [Connections](#connections)) | | <a id="groupepicsenabled"></a>`epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace. | | <a id="groupexternalauditeventdestinations"></a>`externalAuditEventDestinations` | [`ExternalAuditEventDestinationConnection`](#externalauditeventdestinationconnection) | External locations that receive audit events belonging to the group. (see [Connections](#connections)) | +| <a id="groupflowmetrics"></a>`flowMetrics` **{warning-solid}** | [`GroupValueStreamAnalyticsFlowMetrics`](#groupvaluestreamanalyticsflowmetrics) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Flow metrics for value stream analytics. | | <a id="groupfullname"></a>`fullName` | [`String!`](#string) | Full name of the namespace. | | <a id="groupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. | | <a id="groupid"></a>`id` | [`ID!`](#id) | ID of the namespace. | @@ -14153,7 +14536,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="groupcontributionsfrom"></a>`from` | [`ISO8601Date!`](#iso8601date) | Start date of the reporting time range. | -| <a id="groupcontributionsto"></a>`to` | [`ISO8601Date!`](#iso8601date) | End date of the reporting time range. The end date must be within 31 days after the start date. | +| <a id="groupcontributionsto"></a>`to` | [`ISO8601Date!`](#iso8601date) | End date of the reporting time range. The end date must be within 93 days after the start date. | ##### `Group.dataTransfer` @@ -14578,6 +14961,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="groupprojectscomplianceframeworkfilters"></a>`complianceFrameworkFilters` | [`ComplianceFrameworkFilters`](#complianceframeworkfilters) | Filters applied when selecting a compliance framework. | | <a id="groupprojectshascodecoverage"></a>`hasCodeCoverage` | [`Boolean`](#boolean) | Returns only the projects which have code coverage. | | <a id="groupprojectshasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only the projects which have vulnerabilities. | | <a id="groupprojectsids"></a>`ids` | [`[ID!]`](#id) | Filter projects by IDs. | @@ -14641,7 +15025,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="groupscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `dependency_scanning`. | +| <a id="groupscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `sast_iac`, `dependency_scanning`. | | <a id="groupscanexecutionpoliciesrelationship"></a>`relationship` | [`SecurityPolicyRelationType`](#securitypolicyrelationtype) | Filter policies by the given policy relationship. | ##### `Group.scanResultPolicies` @@ -14859,6 +15243,80 @@ Contains statistics about a group. | ---- | ---- | ----------- | | <a id="groupstatsreleasestats"></a>`releaseStats` | [`GroupReleaseStats`](#groupreleasestats) | Statistics related to releases within the group. | +### `GroupValueStreamAnalyticsFlowMetrics` + +Exposes aggregated value stream flow metrics. + +#### Fields with arguments + +##### `GroupValueStreamAnalyticsFlowMetrics.cycleTime` + +Median time from first commit to issue closed. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupvaluestreamanalyticsflowmetricscycletimeassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimeauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimefrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimelabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimemilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimeprojectids"></a>`projectIds` | [`[ID!]`](#id) | Project IDs within the group hierarchy. | +| <a id="groupvaluestreamanalyticsflowmetricscycletimeto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + +##### `GroupValueStreamAnalyticsFlowMetrics.deploymentCount` + +Number of production deployments in the given period. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupvaluestreamanalyticsflowmetricsdeploymentcountfrom"></a>`from` | [`Time!`](#time) | Deployments finished after the date. | +| <a id="groupvaluestreamanalyticsflowmetricsdeploymentcountprojectids"></a>`projectIds` | [`[ID!]`](#id) | Project IDs within the group hierarchy. | +| <a id="groupvaluestreamanalyticsflowmetricsdeploymentcountto"></a>`to` | [`Time!`](#time) | Deployments finished before the date. | + +##### `GroupValueStreamAnalyticsFlowMetrics.issueCount` + +Number of issues opened in the given period. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountfrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountlabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountprojectids"></a>`projectIds` | [`[ID!]`](#id) | Project IDs within the group hierarchy. | +| <a id="groupvaluestreamanalyticsflowmetricsissuecountto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + +##### `GroupValueStreamAnalyticsFlowMetrics.leadTime` + +Median time from when the issue was created to when it was closed. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimeassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimeauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimefrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimelabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimemilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimeprojectids"></a>`projectIds` | [`[ID!]`](#id) | Project IDs within the group hierarchy. | +| <a id="groupvaluestreamanalyticsflowmetricsleadtimeto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + ### `GroupWikiRepositoryRegistry` Represents the Geo sync and verification state of a group wiki repository. @@ -15456,6 +15914,15 @@ Represents an entry from the Cloud License history. | <a id="licensehistoryentrytype"></a>`type` | [`String!`](#string) | Type of the license. | | <a id="licensehistoryentryusersinlicensecount"></a>`usersInLicenseCount` | [`Int`](#int) | Number of paid users in the license. | +### `Location` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="locationblobpath"></a>`blobPath` | [`String`](#string) | HTTP URI path to view the input file in GitLab. | +| <a id="locationpath"></a>`path` | [`String`](#string) | Path, relative to the root of the repository, of the filewhich was analyzed to detect the dependency. | + ### `MavenMetadata` Maven metadata. @@ -15690,6 +16157,7 @@ A user assigned to a merge request. | <a id="mergerequestassigneesavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestassigneestate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestassigneestatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="mergerequestassigneeuserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="mergerequestassigneeuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="mergerequestassigneeusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="mergerequestassigneewebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -15936,6 +16404,7 @@ The author of the merge request. | <a id="mergerequestauthorsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestauthorstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestauthorstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="mergerequestauthoruserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="mergerequestauthoruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="mergerequestauthorusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="mergerequestauthorwebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -16201,6 +16670,7 @@ A user participating in a merge request. | <a id="mergerequestparticipantsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestparticipantstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestparticipantstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="mergerequestparticipantuserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="mergerequestparticipantuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="mergerequestparticipantusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="mergerequestparticipantwebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -16466,6 +16936,7 @@ A user assigned to a merge request as a reviewer. | <a id="mergerequestreviewersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestreviewerstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestreviewerstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="mergerequestrevieweruserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="mergerequestrevieweruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="mergerequestreviewerusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="mergerequestreviewerwebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -16861,6 +17332,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="namespaceprojectscomplianceframeworkfilters"></a>`complianceFrameworkFilters` | [`ComplianceFrameworkFilters`](#complianceframeworkfilters) | Filters applied when selecting a compliance framework. | | <a id="namespaceprojectshascodecoverage"></a>`hasCodeCoverage` | [`Boolean`](#boolean) | Returns only the projects which have code coverage. | | <a id="namespaceprojectshasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only the projects which have vulnerabilities. | | <a id="namespaceprojectsids"></a>`ids` | [`[ID!]`](#id) | Filter projects by IDs. | @@ -16884,7 +17356,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="namespacescanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `dependency_scanning`. | +| <a id="namespacescanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `sast_iac`, `dependency_scanning`. | | <a id="namespacescanexecutionpoliciesrelationship"></a>`relationship` | [`SecurityPolicyRelationType`](#securitypolicyrelationtype) | Filter policies by the given policy relationship. | ##### `Namespace.scanResultPolicies` @@ -17615,7 +18087,7 @@ Represents a pipeline schedule. | ---- | ---- | ----------- | | <a id="pipelineschedulepermissionsadminpipelineschedule"></a>`adminPipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `admin_pipeline_schedule` on this resource. | | <a id="pipelineschedulepermissionsplaypipelineschedule"></a>`playPipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `play_pipeline_schedule` on this resource. | -| <a id="pipelineschedulepermissionstakeownershippipelineschedule"></a>`takeOwnershipPipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `take_ownership_pipeline_schedule` on this resource. | +| <a id="pipelineschedulepermissionstakeownershippipelineschedule"></a>`takeOwnershipPipelineSchedule` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 15.9. Use admin_pipeline_schedule permission to determine if the user can take ownership of a pipeline schedule. | | <a id="pipelineschedulepermissionsupdatepipelineschedule"></a>`updatePipelineSchedule` | [`Boolean!`](#boolean) | Indicates the user can perform `update_pipeline_schedule` on this resource. | ### `PipelineScheduleVariable` @@ -17739,9 +18211,11 @@ Represents a product analytics dashboard visualization. | <a id="projectcreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of the project creation. | | <a id="projectdastscannerprofiles"></a>`dastScannerProfiles` | [`DastScannerProfileConnection`](#dastscannerprofileconnection) | DAST scanner profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdastsiteprofiles"></a>`dastSiteProfiles` | [`DastSiteProfileConnection`](#dastsiteprofileconnection) | DAST Site Profiles associated with the project. (see [Connections](#connections)) | +| <a id="projectdependencies"></a>`dependencies` **{warning-solid}** | [`DependencyConnection`](#dependencyconnection) | **Introduced** in 15.9. This feature is in Alpha. It can be changed or removed at any time. Software dependencies used by the project. | | <a id="projectdescription"></a>`description` | [`String`](#string) | Short description of the project. | | <a id="projectdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | GitLab Flavored Markdown rendering of `description`. | | <a id="projectdora"></a>`dora` | [`Dora`](#dora) | Project's DORA metrics. | +| <a id="projectflowmetrics"></a>`flowMetrics` **{warning-solid}** | [`ProjectValueStreamAnalyticsFlowMetrics`](#projectvaluestreamanalyticsflowmetrics) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Flow metrics for value stream analytics. | | <a id="projectforkscount"></a>`forksCount` | [`Int!`](#int) | Number of times the project has been forked. | | <a id="projectfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the project. | | <a id="projectgrafanaintegration"></a>`grafanaIntegration` | [`GrafanaIntegration`](#grafanaintegration) | Grafana integration details for the project. | @@ -17759,6 +18233,7 @@ Represents a product analytics dashboard visualization. | <a id="projectlastactivityat"></a>`lastActivityAt` | [`Time`](#time) | Timestamp of the project last activity. | | <a id="projectlfsenabled"></a>`lfsEnabled` | [`Boolean`](#boolean) | Indicates if the project has Large File Storage (LFS) enabled. | | <a id="projectmergecommittemplate"></a>`mergeCommitTemplate` | [`String`](#string) | Template used to create merge commit message in merge requests. | +| <a id="projectmergerequestsdisablecommittersapproval"></a>`mergeRequestsDisableCommittersApproval` | [`Boolean!`](#boolean) | Indicates that committers of the given merge request cannot approve. | | <a id="projectmergerequestsenabled"></a>`mergeRequestsEnabled` | [`Boolean`](#boolean) | Indicates if Merge Requests are enabled for the current user. | | <a id="projectmergerequestsffonlyenabled"></a>`mergeRequestsFfOnlyEnabled` | [`Boolean`](#boolean) | Indicates if no merge commits should be created and all merges should instead be fast-forwarded, which means that merging is only allowed if the branch could be fast-forwarded. | | <a id="projectname"></a>`name` | [`String!`](#string) | Name of the project (without namespace). | @@ -17773,6 +18248,7 @@ Represents a product analytics dashboard visualization. | <a id="projectpathlocks"></a>`pathLocks` | [`PathLockConnection`](#pathlockconnection) | The project's path locks. (see [Connections](#connections)) | | <a id="projectpipelineanalytics"></a>`pipelineAnalytics` | [`PipelineAnalytics`](#pipelineanalytics) | Pipeline analytics. | | <a id="projectprintingmergerequestlinkenabled"></a>`printingMergeRequestLinkEnabled` | [`Boolean`](#boolean) | Indicates if a link to create or view a merge request should display after a push to Git repositories of the project from the command line. | +| <a id="projectproductanalyticsstate"></a>`productAnalyticsState` **{warning-solid}** | [`ProductAnalyticsState`](#productanalyticsstate) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Current state of the product analytics stack for this project.Can only be called for one project in a single request. | | <a id="projectpublicjobs"></a>`publicJobs` | [`Boolean`](#boolean) | Indicates if there is public access to pipelines and job details of the project, including output logs and artifacts. | | <a id="projectpushrules"></a>`pushRules` | [`PushRules`](#pushrules) | Project's push rules settings. | | <a id="projectrecentissueboards"></a>`recentIssueBoards` | [`BoardConnection`](#boardconnection) | List of recently visited boards of the project. Maximum size is 4. (see [Connections](#connections)) | @@ -18847,7 +19323,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="projectscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `dependency_scanning`. | +| <a id="projectscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`, `sast_iac`, `dependency_scanning`. | | <a id="projectscanexecutionpoliciesrelationship"></a>`relationship` | [`SecurityPolicyRelationType`](#securitypolicyrelationtype) | Filter policies by the given policy relationship. | ##### `Project.scanResultPolicies` @@ -19096,13 +19572,13 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectworkitemsauthorusername"></a>`authorUsername` **{warning-solid}** | [`String`](#string) | **Introduced** in 15.9. This feature is in Alpha. It can be changed or removed at any time. Filter work items by author username. | -| <a id="projectworkitemsiid"></a>`iid` | [`String`](#string) | IID of the issue. For example, "1". | +| <a id="projectworkitemsiid"></a>`iid` | [`String`](#string) | IID of the work item. For example, "1". | | <a id="projectworkitemsiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of work items. For example, `["1", "2"]`. | | <a id="projectworkitemsin"></a>`in` | [`[IssuableSearchableField!]`](#issuablesearchablefield) | Specify the fields to perform the search in. Defaults to `[TITLE, DESCRIPTION]`. Requires the `search` argument.'. | | <a id="projectworkitemsrequirementlegacywidget"></a>`requirementLegacyWidget` **{warning-solid}** | [`RequirementLegacyFilterInput`](#requirementlegacyfilterinput) | **Deprecated** in 15.9. Use work item IID filter instead. | | <a id="projectworkitemssearch"></a>`search` | [`String`](#string) | Search query for title or description. | -| <a id="projectworkitemssort"></a>`sort` | [`WorkItemSort`](#workitemsort) | Sort work items by this criteria. | -| <a id="projectworkitemsstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this work item. | +| <a id="projectworkitemssort"></a>`sort` | [`WorkItemSort`](#workitemsort) | Sort work items by criteria. | +| <a id="projectworkitemsstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of the work item. | | <a id="projectworkitemsstatuswidget"></a>`statusWidget` | [`StatusFilterInput`](#statusfilterinput) | Input for status widget filter. Ignored if `work_items_mvc_2` is disabled. | | <a id="projectworkitemstypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter work items by the given work item types. | @@ -19254,6 +19730,76 @@ Represents the source of a security policy belonging to a project. | <a id="projectstatisticsuploadssize"></a>`uploadsSize` | [`Float`](#float) | Uploads size of the project in bytes. | | <a id="projectstatisticswikisize"></a>`wikiSize` | [`Float`](#float) | Wiki size of the project in bytes. | +### `ProjectValueStreamAnalyticsFlowMetrics` + +Exposes aggregated value stream flow metrics. + +#### Fields with arguments + +##### `ProjectValueStreamAnalyticsFlowMetrics.cycleTime` + +Median time from first commit to issue closed. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectvaluestreamanalyticsflowmetricscycletimeassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricscycletimeauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="projectvaluestreamanalyticsflowmetricscycletimefrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="projectvaluestreamanalyticsflowmetricscycletimelabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricscycletimemilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricscycletimeto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + +##### `ProjectValueStreamAnalyticsFlowMetrics.deploymentCount` + +Number of production deployments in the given period. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectvaluestreamanalyticsflowmetricsdeploymentcountfrom"></a>`from` | [`Time!`](#time) | Deployments finished after the date. | +| <a id="projectvaluestreamanalyticsflowmetricsdeploymentcountto"></a>`to` | [`Time!`](#time) | Deployments finished before the date. | + +##### `ProjectValueStreamAnalyticsFlowMetrics.issueCount` + +Number of issues opened in the given period. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountfrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountlabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsissuecountto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + +##### `ProjectValueStreamAnalyticsFlowMetrics.leadTime` + +Median time from when the issue was created to when it was closed. + +Returns [`ValueStreamAnalyticsMetric`](#valuestreamanalyticsmetric). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimeassigneeusernames"></a>`assigneeUsernames` | [`[String!]`](#string) | Usernames of users assigned to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimeauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimefrom"></a>`from` | [`Time!`](#time) | Issues created after the date. | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimelabelnames"></a>`labelNames` | [`[String!]`](#string) | Labels applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimemilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Milestone applied to the issue. | +| <a id="projectvaluestreamanalyticsflowmetricsleadtimeto"></a>`to` | [`Time!`](#time) | Issues created before the date. | + ### `PrometheusAlert` The alert condition for Prometheus. @@ -19278,6 +19824,7 @@ Protected Environments of the environment. | <a id="protectedenvironmentgroup"></a>`group` | [`Group`](#group) | Group details. Present if it's group-level protected environment. | | <a id="protectedenvironmentname"></a>`name` | [`String`](#string) | Name of the environment if it's a project-level protected environment. Tier of the environment if it's a group-level protected environment. | | <a id="protectedenvironmentproject"></a>`project` | [`Project`](#project) | Project details. Present if it's project-level protected environment. | +| <a id="protectedenvironmentrequiredapprovalcount"></a>`requiredApprovalCount` | [`Int`](#int) | Required approval count for Unified Approval Setting. | ### `ProtectedEnvironmentApprovalRule` @@ -19678,6 +20225,7 @@ Counts of requirements by their state. | <a id="rootstoragestatisticslfsobjectssize"></a>`lfsObjectsSize` | [`Float!`](#float) | LFS objects size in bytes. | | <a id="rootstoragestatisticspackagessize"></a>`packagesSize` | [`Float!`](#float) | Packages size in bytes. | | <a id="rootstoragestatisticspipelineartifactssize"></a>`pipelineArtifactsSize` | [`Float!`](#float) | CI pipeline artifacts size in bytes. | +| <a id="rootstoragestatisticsregistrysizeestimated"></a>`registrySizeEstimated` | [`Boolean!`](#boolean) | Indicates whether the deduplicated Container Registry size for the namespace is an estimated value or not. | | <a id="rootstoragestatisticsrepositorysize"></a>`repositorySize` | [`Float!`](#float) | Git repository size in bytes. | | <a id="rootstoragestatisticssnippetssize"></a>`snippetsSize` | [`Float!`](#float) | Snippets size in bytes. | | <a id="rootstoragestatisticsstoragesize"></a>`storageSize` | [`Float!`](#float) | Total storage in bytes. | @@ -20199,6 +20747,7 @@ SSH signature for a signed commit. | ---- | ---- | ----------- | | <a id="sshsignaturecommitsha"></a>`commitSha` | [`String`](#string) | SHA of the associated commit. | | <a id="sshsignaturekey"></a>`key` | [`Key`](#key) | SSH key used for the signature. | +| <a id="sshsignaturekeyfingerprintsha256"></a>`keyFingerprintSha256` | [`String`](#string) | Fingerprint of the key. | | <a id="sshsignatureproject"></a>`project` | [`Project`](#project) | Project of the associated commit. | | <a id="sshsignatureuser"></a>`user` | [`UserCore`](#usercore) | User associated with the key. | | <a id="sshsignatureverificationstatus"></a>`verificationStatus` | [`VerificationStatus`](#verificationstatus) | Indicates verification status of the associated key or certificate. | @@ -20648,6 +21197,21 @@ Represents a recorded measurement (object count) for the Admins. | <a id="usagetrendsmeasurementidentifier"></a>`identifier` | [`MeasurementIdentifier!`](#measurementidentifier) | Type of objects being measured. | | <a id="usagetrendsmeasurementrecordedat"></a>`recordedAt` | [`Time`](#time) | Time the measurement was recorded. | +### `UserAchievement` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="userachievementachievement"></a>`achievement` | [`Achievement!`](#achievement) | Achievement awarded. | +| <a id="userachievementawardedbyuser"></a>`awardedByUser` | [`UserCore!`](#usercore) | Awarded by. | +| <a id="userachievementcreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp the achievement was created. | +| <a id="userachievementid"></a>`id` | [`AchievementsUserAchievementID!`](#achievementsuserachievementid) | ID of the user achievement. | +| <a id="userachievementrevokedat"></a>`revokedAt` | [`Time`](#time) | Timestamp the achievement was revoked. | +| <a id="userachievementrevokedbyuser"></a>`revokedByUser` | [`UserCore`](#usercore) | Revoked by. | +| <a id="userachievementupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp the achievement was last updated. | +| <a id="userachievementuser"></a>`user` | [`UserCore!`](#usercore) | Achievement recipient. | + ### `UserCallout` #### Fields @@ -20686,6 +21250,7 @@ Core represention of a GitLab user. | <a id="usercoresavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="usercorestate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="usercorestatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="usercoreuserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="usercoreuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="usercoreusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="usercorewebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -20947,6 +21512,29 @@ fields relate to interactions between the two entities. | <a id="userstatusmessage"></a>`message` | [`String`](#string) | User status message. | | <a id="userstatusmessagehtml"></a>`messageHtml` | [`String`](#string) | HTML of the user status message. | +### `ValueStreamAnalyticsMetric` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="valuestreamanalyticsmetricidentifier"></a>`identifier` | [`String!`](#string) | Identifier for the metric. | +| <a id="valuestreamanalyticsmetriclinks"></a>`links` | [`[ValueStreamMetricLinkType!]!`](#valuestreammetriclinktype) | Optional links for drilling down. | +| <a id="valuestreamanalyticsmetrictitle"></a>`title` | [`String!`](#string) | Title for the metric. | +| <a id="valuestreamanalyticsmetricunit"></a>`unit` | [`String`](#string) | Unit of measurement. | +| <a id="valuestreamanalyticsmetricvalue"></a>`value` | [`Float`](#float) | Value for the metric. | + +### `ValueStreamMetricLinkType` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="valuestreammetriclinktypedocslink"></a>`docsLink` | [`Boolean`](#boolean) | Link to the metric documentation. | +| <a id="valuestreammetriclinktypelabel"></a>`label` | [`String!`](#string) | Label for the link. | +| <a id="valuestreammetriclinktypename"></a>`name` | [`String!`](#string) | Name of the link group. | +| <a id="valuestreammetriclinktypeurl"></a>`url` | [`String!`](#string) | Drill-down URL. | + ### `VulnerabilitiesCountByDay` Represents the count of vulnerabilities by severity on a particular day. This data is retained for 365 days. @@ -21002,6 +21590,7 @@ Represents a vulnerability. | <a id="vulnerabilityseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (INFO, UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL). | | <a id="vulnerabilitystate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (DETECTED, CONFIRMED, RESOLVED, DISMISSED). | | <a id="vulnerabilitystatecomment"></a>`stateComment` | [`String`](#string) | Comment given for the vulnerability state change. | +| <a id="vulnerabilitystatetransitions"></a>`stateTransitions` | [`VulnerabilityStateTransitionTypeConnection`](#vulnerabilitystatetransitiontypeconnection) | List of state transitions related to the vulnerability. (see [Connections](#connections)) | | <a id="vulnerabilitytitle"></a>`title` | [`String`](#string) | Title of the vulnerability. | | <a id="vulnerabilityupdatedat"></a>`updatedAt` | [`Time`](#time) | Timestamp of when the vulnerability was last updated. | | <a id="vulnerabilityusernotescount"></a>`userNotesCount` | [`Int!`](#int) | Number of user notes attached to the vulnerability. | @@ -21171,6 +21760,19 @@ Represents the vulnerability details location within a file in the project. | <a id="vulnerabilitydetailmodulelocationname"></a>`name` | [`String`](#string) | Name of the field. | | <a id="vulnerabilitydetailmodulelocationoffset"></a>`offset` | [`Int!`](#int) | Offset of the module location. | +### `VulnerabilityDetailRow` + +Represents an individual row in a table. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitydetailrowdescription"></a>`description` | [`String`](#string) | Description of the field. | +| <a id="vulnerabilitydetailrowfieldname"></a>`fieldName` | [`String`](#string) | Name of the field. | +| <a id="vulnerabilitydetailrowname"></a>`name` | [`String`](#string) | Name of the field. | +| <a id="vulnerabilitydetailrowrow"></a>`row` | [`[VulnerabilityDetail!]!`](#vulnerabilitydetail) | Value of the field. | + ### `VulnerabilityDetailTable` Represents the vulnerability details table value. @@ -21183,7 +21785,7 @@ Represents the vulnerability details table value. | <a id="vulnerabilitydetailtablefieldname"></a>`fieldName` | [`String`](#string) | Name of the field. | | <a id="vulnerabilitydetailtableheaders"></a>`headers` | [`[VulnerabilityDetail!]!`](#vulnerabilitydetail) | Table headers. | | <a id="vulnerabilitydetailtablename"></a>`name` | [`String`](#string) | Name of the field. | -| <a id="vulnerabilitydetailtablerows"></a>`rows` | [`[VulnerabilityDetail!]!`](#vulnerabilitydetail) | Table rows. | +| <a id="vulnerabilitydetailtablerows"></a>`rows` | [`[VulnerabilityDetailRow!]!`](#vulnerabilitydetailrow) | Table rows. | ### `VulnerabilityDetailText` @@ -21502,6 +22104,21 @@ Represents vulnerability counts by severity. | <a id="vulnerabilityseveritiescountmedium"></a>`medium` | [`Int`](#int) | Number of vulnerabilities of MEDIUM severity of the project. | | <a id="vulnerabilityseveritiescountunknown"></a>`unknown` | [`Int`](#int) | Number of vulnerabilities of UNKNOWN severity of the project. | +### `VulnerabilityStateTransitionType` + +Represents a state transition of a vulnerability. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitystatetransitiontypeauthor"></a>`author` | [`UserCore!`](#usercore) | User who changed the state of the vulnerability. | +| <a id="vulnerabilitystatetransitiontypecomment"></a>`comment` | [`String`](#string) | Comment for the state change. | +| <a id="vulnerabilitystatetransitiontypecreatedat"></a>`createdAt` | [`Time!`](#time) | Time of the state change of the vulnerability. | +| <a id="vulnerabilitystatetransitiontypedismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason for the dismissal. | +| <a id="vulnerabilitystatetransitiontypefromstate"></a>`fromState` | [`VulnerabilityState!`](#vulnerabilitystate) | State of the vulnerability before transition. | +| <a id="vulnerabilitystatetransitiontypetostate"></a>`toState` | [`VulnerabilityState!`](#vulnerabilitystate) | State of the vulnerability after transition. | + ### `VulnerableDependency` Represents a vulnerable dependency. Used in vulnerability location data. @@ -21712,6 +22329,17 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="workitemwidgetnotesdiscussionsfilter"></a>`filter` | [`NotesFilterType`](#notesfiltertype) | Type of notes collection: ALL_NOTES, ONLY_COMMENTS, ONLY_ACTIVITY. | +### `WorkItemWidgetNotifications` + +Represents the notifications widget. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetnotificationssubscribed"></a>`subscribed` | [`Boolean!`](#boolean) | Whether the current user is subscribed to notifications on the work item. | +| <a id="workitemwidgetnotificationstype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | + ### `WorkItemWidgetProgress` Represents a progress widget. @@ -21996,6 +22624,19 @@ User availability status. | <a id="availabilityenumbusy"></a>`BUSY` | Busy. | | <a id="availabilityenumnot_set"></a>`NOT_SET` | Not Set. | +### `AvailableExportFields` + +Available fields to be exported as CSV. + +| Value | Description | +| ----- | ----------- | +| <a id="availableexportfieldsauthor"></a>`AUTHOR` | Author name. | +| <a id="availableexportfieldsauthor_username"></a>`AUTHOR_USERNAME` | Author username. | +| <a id="availableexportfieldscreated_at"></a>`CREATED_AT` | Date of creation. | +| <a id="availableexportfieldsid"></a>`ID` | Unique identifier. | +| <a id="availableexportfieldstitle"></a>`TITLE` | Title. | +| <a id="availableexportfieldstype"></a>`TYPE` | Type of the work item. | + ### `BlobViewersType` Types of blob viewers. @@ -22178,6 +22819,15 @@ Mode of a commit action. | <a id="commitencodingbase64"></a>`BASE64` | Base64 encoding. | | <a id="commitencodingtext"></a>`TEXT` | Text encoding. | +### `ComplianceFrameworkPresenceFilter` + +ComplianceFramework of a project for filtering. + +| Value | Description | +| ----- | ----------- | +| <a id="complianceframeworkpresencefilterany"></a>`ANY` | Any compliance framework is assigned. | +| <a id="complianceframeworkpresencefilternone"></a>`NONE` | No compliance framework is assigned. | + ### `ComplianceViolationReason` Reason for the compliance violation. @@ -22345,6 +22995,16 @@ Values for sorting tags. | <a id="customerrelationsorganizationstateall"></a>`all` | All available organizations. | | <a id="customerrelationsorganizationstateinactive"></a>`inactive` | Inactive organizations. | +### `DastPreScanVerificationCheckType` + +Check type of the pre scan verification step. + +| Value | Description | +| ----- | ----------- | +| <a id="dastprescanverificationchecktypeauthentication"></a>`AUTHENTICATION` | Authentication check. | +| <a id="dastprescanverificationchecktypeconnection"></a>`CONNECTION` | Connection check. | +| <a id="dastprescanverificationchecktypecrawling"></a>`CRAWLING` | Crawling check. | + ### `DastPreScanVerificationStatus` Status of DAST pre scan verification. @@ -22767,6 +23427,15 @@ State of a GitLab issue or merge request. | <a id="issuablestatelocked"></a>`locked` | Discussion has been locked. | | <a id="issuablestateopened"></a>`opened` | In open state. | +### `IssuableSubscriptionEvent` + +Values for subscribing and unsubscribing from issuables. + +| Value | Description | +| ----- | ----------- | +| <a id="issuablesubscriptioneventsubscribe"></a>`SUBSCRIBE` | Subscribe to an issuable. | +| <a id="issuablesubscriptioneventunsubscribe"></a>`UNSUBSCRIBE` | Unsubscribe from an issuable. | + ### `IssueCreationIterationWildcardId` Iteration ID wildcard values for issue creation. @@ -23139,9 +23808,11 @@ Values for sorting projects. | Value | Description | | ----- | ----------- | -| <a id="namespaceprojectsortactivity_desc"></a>`ACTIVITY_DESC` | Sort by latest activity, in descending order. | +| <a id="namespaceprojectsortactivity_desc"></a>`ACTIVITY_DESC` | Sort by latest activity, descending order. | | <a id="namespaceprojectsortsimilarity"></a>`SIMILARITY` | Most similar to the search query. | -| <a id="namespaceprojectsortstorage"></a>`STORAGE` | Sort by storage size. | +| <a id="namespaceprojectsortstorage"></a>`STORAGE` | Sort by excess repository storage size, descending order. | +| <a id="namespaceprojectsortstorage_size_asc"></a>`STORAGE_SIZE_ASC` | Sort by total storage size, ascending order. | +| <a id="namespaceprojectsortstorage_size_desc"></a>`STORAGE_SIZE_DESC` | Sort by total storage size, descending order. | ### `NegatedIterationWildcardId` @@ -23348,6 +24019,17 @@ Event type of the pipeline associated with a merge request. | <a id="pipelinestatusenumsuccess"></a>`SUCCESS` | Pipeline completed successfully. | | <a id="pipelinestatusenumwaiting_for_resource"></a>`WAITING_FOR_RESOURCE` | A resource (for example, a runner) that the pipeline requires to run is unavailable. | +### `ProductAnalyticsState` + +Current state of the product analytics stack. + +| Value | Description | +| ----- | ----------- | +| <a id="productanalyticsstatecomplete"></a>`COMPLETE` | Stack has been initialized and has data. | +| <a id="productanalyticsstatecreate_instance"></a>`CREATE_INSTANCE` | Stack has not been created yet. | +| <a id="productanalyticsstateloading_instance"></a>`LOADING_INSTANCE` | Stack is currently initializing. | +| <a id="productanalyticsstatewaiting_for_events"></a>`WAITING_FOR_EVENTS` | Stack is waiting for events from users. | + ### `ProjectMemberRelation` Project member relation. @@ -23523,6 +24205,7 @@ State of a Sentry error. | <a id="servicetypeexternal_wiki_service"></a>`EXTERNAL_WIKI_SERVICE` | ExternalWikiService type. | | <a id="servicetypegithub_service"></a>`GITHUB_SERVICE` | GithubService type. | | <a id="servicetypegitlab_slack_application_service"></a>`GITLAB_SLACK_APPLICATION_SERVICE` | GitlabSlackApplicationService type (Gitlab.com only). | +| <a id="servicetypegoogle_play_service"></a>`GOOGLE_PLAY_SERVICE` | GooglePlayService type. | | <a id="servicetypehangouts_chat_service"></a>`HANGOUTS_CHAT_SERVICE` | HangoutsChatService type. | | <a id="servicetypeharbor_service"></a>`HARBOR_SERVICE` | HarborService type. | | <a id="servicetypeirker_service"></a>`IRKER_SERVICE` | IrkerService type. | @@ -23541,6 +24224,7 @@ State of a Sentry error. | <a id="servicetypeshimo_service"></a>`SHIMO_SERVICE` | ShimoService type. | | <a id="servicetypeslack_service"></a>`SLACK_SERVICE` | SlackService type. | | <a id="servicetypeslack_slash_commands_service"></a>`SLACK_SLASH_COMMANDS_SERVICE` | SlackSlashCommandsService type. | +| <a id="servicetypesquash_tm_service"></a>`SQUASH_TM_SERVICE` | SquashTmService type. | | <a id="servicetypeteamcity_service"></a>`TEAMCITY_SERVICE` | TeamcityService type. | | <a id="servicetypeunify_circuit_service"></a>`UNIFY_CIRCUIT_SERVICE` | UnifyCircuitService type. | | <a id="servicetypewebex_teams_service"></a>`WEBEX_TEAMS_SERVICE` | WebexTeamsService type. | @@ -23719,6 +24403,7 @@ Name of the feature that the callout is for. | ----- | ----------- | | <a id="usercalloutfeaturenameenumactive_user_count_threshold"></a>`ACTIVE_USER_COUNT_THRESHOLD` | Callout feature name for active_user_count_threshold. | | <a id="usercalloutfeaturenameenumartifacts_management_page_feedback_banner"></a>`ARTIFACTS_MANAGEMENT_PAGE_FEEDBACK_BANNER` | Callout feature name for artifacts_management_page_feedback_banner. | +| <a id="usercalloutfeaturenameenumbranch_rules_info_callout"></a>`BRANCH_RULES_INFO_CALLOUT` | Callout feature name for branch_rules_info_callout. | | <a id="usercalloutfeaturenameenumbuy_pipeline_minutes_notification_dot"></a>`BUY_PIPELINE_MINUTES_NOTIFICATION_DOT` | Callout feature name for buy_pipeline_minutes_notification_dot. | | <a id="usercalloutfeaturenameenumcanary_deployment"></a>`CANARY_DEPLOYMENT` | Callout feature name for canary_deployment. | | <a id="usercalloutfeaturenameenumci_deprecation_warning_for_types_keyword"></a>`CI_DEPRECATION_WARNING_FOR_TYPES_KEYWORD` | Callout feature name for ci_deprecation_warning_for_types_keyword. | @@ -23996,6 +24681,7 @@ Type of a work item widget. | <a id="workitemwidgettypelabels"></a>`LABELS` | Labels widget. | | <a id="workitemwidgettypemilestone"></a>`MILESTONE` | Milestone widget. | | <a id="workitemwidgettypenotes"></a>`NOTES` | Notes widget. | +| <a id="workitemwidgettypenotifications"></a>`NOTIFICATIONS` | Notifications widget. | | <a id="workitemwidgettypeprogress"></a>`PROGRESS` | Progress widget. | | <a id="workitemwidgettyperequirement_legacy"></a>`REQUIREMENT_LEGACY` | Requirement Legacy widget. | | <a id="workitemwidgettypestart_and_due_date"></a>`START_AND_DUE_DATE` | Start And Due Date widget. | @@ -24020,6 +24706,12 @@ A `AchievementsAchievementID` is a global ID. It is encoded as a string. An example `AchievementsAchievementID` is: `"gid://gitlab/Achievements::Achievement/1"`. +### `AchievementsUserAchievementID` + +A `AchievementsUserAchievementID` is a global ID. It is encoded as a string. + +An example `AchievementsUserAchievementID` is: `"gid://gitlab/Achievements::UserAchievement/1"`. + ### `AlertManagementAlertID` A `AlertManagementAlertID` is a global ID. It is encoded as a string. @@ -24118,6 +24810,12 @@ A `CiRunnerID` is a global ID. It is encoded as a string. An example `CiRunnerID` is: `"gid://gitlab/Ci::Runner/1"`. +### `CiRunnerMachineID` + +A `CiRunnerMachineID` is a global ID. It is encoded as a string. + +An example `CiRunnerMachineID` is: `"gid://gitlab/Ci::RunnerMachine/1"`. + ### `ClustersAgentID` A `ClustersAgentID` is a global ID. It is encoded as a string. @@ -25122,6 +25820,7 @@ Implementations: | <a id="usersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="userstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="userstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | +| <a id="useruserachievements"></a>`userAchievements` **{warning-solid}** | [`UserAchievementConnection`](#userachievementconnection) | **Introduced** in 15.10. This feature is in Alpha. It can be changed or removed at any time. Achievements for the user. Only returns for namespaces where the `achievements` feature flag is enabled. | | <a id="useruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | | <a id="userusername"></a>`username` | [`String!`](#string) | Username of the user. Unique within this instance of GitLab. | | <a id="userwebpath"></a>`webPath` | [`String!`](#string) | Web path of the user. | @@ -25350,6 +26049,7 @@ Implementations: - [`WorkItemWidgetLabels`](#workitemwidgetlabels) - [`WorkItemWidgetMilestone`](#workitemwidgetmilestone) - [`WorkItemWidgetNotes`](#workitemwidgetnotes) +- [`WorkItemWidgetNotifications`](#workitemwidgetnotifications) - [`WorkItemWidgetProgress`](#workitemwidgetprogress) - [`WorkItemWidgetRequirementLegacy`](#workitemwidgetrequirementlegacy) - [`WorkItemWidgetStartAndDueDate`](#workitemwidgetstartandduedate) @@ -25439,6 +26139,16 @@ Attributes for defining a CI/CD variable. | <a id="commitactionlastcommitid"></a>`lastCommitId` | [`String`](#string) | Last known file commit ID. | | <a id="commitactionpreviouspath"></a>`previousPath` | [`String`](#string) | Original full path to the file being moved. | +### `ComplianceFrameworkFilters` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="complianceframeworkfiltersid"></a>`id` | [`ComplianceManagementFrameworkID`](#compliancemanagementframeworkid) | ID of the compliance framework. | +| <a id="complianceframeworkfiltersnot"></a>`not` | [`NegatedComplianceFrameworkFilters`](#negatedcomplianceframeworkfilters) | Negated compliance framework filter input. | +| <a id="complianceframeworkfilterspresencefilter"></a>`presenceFilter` | [`ComplianceFrameworkPresenceFilter`](#complianceframeworkpresencefilter) | Checks presence of compliance framework of the project, "none" and "any" values are supported. | + ### `ComplianceFrameworkInput` #### Arguments @@ -25629,6 +26339,14 @@ Represents an escalation rule. | <a id="negatedboardissueinputtypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter by the given issue types. | | <a id="negatedboardissueinputweight"></a>`weight` | [`String`](#string) | Filter by weight. | +### `NegatedComplianceFrameworkFilters` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="negatedcomplianceframeworkfiltersid"></a>`id` | [`ComplianceManagementFrameworkID`](#compliancemanagementframeworkid) | ID of the compliance framework. | + ### `NegatedEpicBoardIssueInput` #### Arguments @@ -25931,6 +26649,7 @@ A time-frame defined as a closed inclusive range of two dates. | <a id="workitemupdatedtaskinputid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | | <a id="workitemupdatedtaskinputlabelswidget"></a>`labelsWidget` | [`WorkItemWidgetLabelsUpdateInput`](#workitemwidgetlabelsupdateinput) | Input for labels widget. | | <a id="workitemupdatedtaskinputmilestonewidget"></a>`milestoneWidget` | [`WorkItemWidgetMilestoneInput`](#workitemwidgetmilestoneinput) | Input for milestone widget. | +| <a id="workitemupdatedtaskinputnotificationswidget"></a>`notificationsWidget` | [`WorkItemWidgetNotificationsUpdateInput`](#workitemwidgetnotificationsupdateinput) | Input for notifications widget. | | <a id="workitemupdatedtaskinputstartandduedatewidget"></a>`startAndDueDateWidget` | [`WorkItemWidgetStartAndDueDateUpdateInput`](#workitemwidgetstartandduedateupdateinput) | Input for start and due date widget. | | <a id="workitemupdatedtaskinputstateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | | <a id="workitemupdatedtaskinputtitle"></a>`title` | [`String`](#string) | Title of the work item. | @@ -26001,6 +26720,14 @@ A time-frame defined as a closed inclusive range of two dates. | ---- | ---- | ----------- | | <a id="workitemwidgetmilestoneinputmilestoneid"></a>`milestoneId` | [`MilestoneID`](#milestoneid) | Milestone to assign to the work item. | +### `WorkItemWidgetNotificationsUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetnotificationsupdateinputsubscribed"></a>`subscribed` | [`Boolean!`](#boolean) | Desired state of the subscription. | + ### `WorkItemWidgetProgressInput` #### Arguments diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index ede7591c6d1..4c225e8aacb 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Placeholder tokens -Badges support placeholders that are replaced in real time in both the link and image URL. The allowed placeholders are: +[Badges](../user/project/badges.md) support placeholders that are replaced in real time in both the link and image URL. The allowed placeholders are: <!-- vale gitlab.Spelling = NO --> diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index daffda8340c..a180c7a1f92 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Group clusters API (certificate-based) (DEPRECATED) **(FREE)** +# Group clusters API (certificate-based) (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30213) in GitLab 12.1. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index c648b6bad37..5dd0e4e3d52 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -4,12 +4,12 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Group import/export API **(FREE)** +# Group import and export API **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20353) in GitLab 12.8. -Group Import/Export allows you to export group structure and import it to a new location. -When used with [Project Import/Export](project_import_export.md), you can preserve connections with +Use the group import and export API to export a group structure and import it to a new location. +When you use the group import and export API with the [project import and export API](project_import_export.md), you can preserve connections with group-level relationships, such as connections between project issues and group epics. Group exports include the following: @@ -22,6 +22,13 @@ Group exports include the following: - Subgroups. Each subgroup includes all data above - Group wikis **(PREMIUM SELF)** +To preserve group-level relationships from imported projects, you should run group import and export first. This way, you can import project exports into the desired group structure. + +Imported groups have a `private` visibility level unless you import them into a parent group. +If you import groups into a parent group, the subgroups inherit by default a similar level of visibility. + +To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make sure these users exist before importing the desired groups. + ## Schedule new export Start a new group export. @@ -103,14 +110,3 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ NOTE: The maximum import file size can be set by the Administrator, default is `0` (unlimited). As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](settings.md#change-application-settings) or the [Admin Area](../user/admin_area/settings/account_and_limit_settings.md). Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. - -## Important notes - -Note the following: - -- To preserve group-level relationships from imported projects, run Group Import/Export first, - to allow project imports into the desired group structure. -- Imported groups are given a `private` visibility level, unless imported into a parent group. -- If imported into a parent group, subgroups inherit a similar level of visibility, unless otherwise restricted. -- To preserve the member list and their respective permissions on imported groups, - review the users in these groups. Make sure these users exist before importing the desired groups. diff --git a/doc/api/group_iterations.md b/doc/api/group_iterations.md index 3c445ee09dd..b2439861c2f 100644 --- a/doc/api/group_iterations.md +++ b/doc/api/group_iterations.md @@ -22,6 +22,8 @@ GET /groups/:id/iterations?state=opened GET /groups/:id/iterations?state=closed GET /groups/:id/iterations?search=version GET /groups/:id/iterations?include_ancestors=false +GET /groups/:id/iterations?updated_before=2013-10-02T09%3A24%3A18Z +GET /groups/:id/iterations?updated_after=2013-10-02T09%3A24%3A18Z ``` | Attribute | Type | Required | Description | @@ -29,6 +31,8 @@ GET /groups/:id/iterations?include_ancestors=false | `state` | string | no | 'Return `opened`, `upcoming`, `current (previously started)`, `closed`, or `all` iterations. Filtering by `started` state is deprecated starting with 14.1, use `current` instead.' | | `search` | string | no | Return only iterations with a title matching the provided string. | | `include_ancestors` | boolean | no | Include iterations from parent group and its ancestors. Defaults to `true`. | +| `updated_before` | datetime | no | Return only iterations updated before the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378662) in GitLab 15.10. | +| `updated_after` | datetime | no | Return only iterations updated after the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378662) in GitLab 15.10. | Example request: diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index 4037a778d7f..921a9922c9b 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md index fc95230cbba..52bce54119a 100644 --- a/doc/api/group_milestones.md +++ b/doc/api/group_milestones.md @@ -21,6 +21,8 @@ GET /groups/:id/milestones?state=active GET /groups/:id/milestones?state=closed GET /groups/:id/milestones?title=1.0 GET /groups/:id/milestones?search=version +GET /groups/:id/milestones?updated_before=2013-10-02T09%3A24%3A18Z +GET /groups/:id/milestones?updated_after=2013-10-02T09%3A24%3A18Z ``` Parameters: @@ -32,7 +34,9 @@ Parameters: | `state` | string | no | Return only `active` or `closed` milestones | | `title` | string | no | Return only the milestones having the given `title` | | `search` | string | no | Return only milestones with a title or description matching the provided string | -| `include_parent_milestones` | boolean | optional | Include milestones from parent group and its ancestors. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196066) in GitLab 13.4 | +| `include_parent_milestones` | boolean | no | Include milestones from parent group and its ancestors. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196066) in GitLab 13.4 | +| `updated_before` | datetime | no | Return only milestones updated before the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). Introduced in GitLab 15.10 | +| `updated_after` | datetime | no | Return only milestones updated after the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). Introduced in GitLab 15.10 | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/milestones" diff --git a/doc/api/group_protected_branches.md b/doc/api/group_protected_branches.md new file mode 100644 index 00000000000..db648f6870e --- /dev/null +++ b/doc/api/group_protected_branches.md @@ -0,0 +1,469 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Group-level protected branches API **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110603) in GitLab 15.9 [with a flag](../administration/feature_flags.md) named `group_protected_branches`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `group_protected_branches`. +On GitLab.com, this feature is not available. + +## Valid access levels + +The access levels are defined in the `ProtectedRefAccess.allowed_access_levels` method. +These levels are recognized: + +```plaintext +0 => No access +30 => Developer access +40 => Maintainer access +60 => Admin access +``` + +## List protected branches + +Gets a list of protected branches from a group. If a wildcard is set, it is returned instead +of the exact name of the branches that match that wildcard. + +```plaintext +GET /groups/:id/protected_branches +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `search` | string | no | Name or part of the name of protected branches to be searched for. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches" +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "master", + "push_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": 1234, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": 1234, + "access_level_description": "Maintainers" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false + }, + { + "id": 1, + "name": "release/*", + "push_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": 1234, + "access_level_description": "Maintainers" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false + }, + ... +] +``` + +## Get a single protected branch or wildcard protected branch + +Gets a single protected branch or wildcard protected branch. + +```plaintext +GET /groups/:id/protected_branches/:name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the branch or wildcard. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches/master" +``` + +Example response: + +```json +{ + "id": 1, + "name": "master", + "push_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": null, + "user_id": null, + "group_id": 1234, + "access_level_description": "Example Merge Group" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false +} +``` + +## Protect repository branches + +Protects a single repository branch using a wildcard protected branch. + +```plaintext +POST /groups/:id/protected_branches +``` + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30&unprotect_access_level=40" +``` + +| Attribute | Type | Required | Description | +| -------------------------------------------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the branch or wildcard. | +| `allow_force_push` | boolean | no | Allow all users with push access to force push. Default: `false`. | +| `allowed_to_merge` | array | no | Array of access levels allowed to merge, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. | +| `allowed_to_push` | array | no | Array of access levels allowed to push, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. | +| `allowed_to_unprotect` | array | no | Array of access levels allowed to unprotect, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. | +| `code_owner_approval_required` | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). Default: `false`. | +| `merge_access_level` | integer | no | Access levels allowed to merge. Defaults: `40`, Maintainer role. | +| `push_access_level` | integer | no | Access levels allowed to push. Defaults: `40`, Maintainer role. | +| `unprotect_access_level` | integer | no | Access levels allowed to unprotect. Defaults: `40`, Maintainer role. | + +Example response: + +```json +{ + "id": 1, + "name": "*-stable", + "push_access_levels": [ + { + "id": 1, + "access_level": 30, + "user_id": null, + "group_id": null, + "access_level_description": "Developers + Maintainers" + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": 30, + "user_id": null, + "group_id": null, + "access_level_description": "Developers + Maintainers" + } + ], + "unprotect_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false +} +``` + +### Example with user / group level access + +Elements in the `allowed_to_push` / `allowed_to_merge` / `allowed_to_unprotect` array should take the +form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. Each user must have +access to the project and each group must +[have this project shared](../user/project/members/share_project_with_groups.md). These access levels +allow [more granular control over protected branch access](../user/project/protected_branches.md). + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches?name=*-stable&allowed_to_push%5B%5D%5Buser_id%5D=1" +``` + +Example response: + +```json +{ + "id": 1, + "name": "*-stable", + "push_access_levels": [ + { + "id": 1, + "access_level": null, + "user_id": 1, + "group_id": null, + "access_level_description": "Administrator" + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "unprotect_access_levels": [ + { + "id": 1, + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false +} +``` + +### Example with allow to push and allow to merge access + +Example request: + +```shell +curl --request POST \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{ + "name": "master", + "allowed_to_push": [{"access_level": 30}], + "allowed_to_merge": [{ + "access_level": 30 + },{ + "access_level": 40 + } + ]}' + "https://gitlab.example.com/api/v4/groups/5/protected_branches" +``` + +Example response: + +```json +{ + "id": 5, + "name": "master", + "push_access_levels": [ + { + "id": 1, + "access_level": 30, + "access_level_description": "Developers + Maintainers", + "user_id": null, + "group_id": null + } + ], + "merge_access_levels": [ + { + "id": 1, + "access_level": 30, + "access_level_description": "Developers + Maintainers", + "user_id": null, + "group_id": null + }, + { + "id": 2, + "access_level": 40, + "access_level_description": "Maintainers", + "user_id": null, + "group_id": null + } + ], + "unprotect_access_levels": [ + { + "id": 1, + "access_level": 40, + "access_level_description": "Maintainers", + "user_id": null, + "group_id": null + } + ], + "allow_force_push":false, + "code_owner_approval_required": false +} +``` + +## Unprotect repository branches + +Unprotects the given protected branch or wildcard protected branch. + +```plaintext +DELETE /groups/:id/protected_branches/:name +``` + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches/*-stable" +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the branch. | + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [ + { + "id": 12, + "access_level": 40, + "access_level_description": "Maintainers", + "user_id": null, + "group_id": null + } + ] +} +``` + +## Update a protected branch + +Updates a protected branch. + +```plaintext +PATCH /groups/:id/protected_branches/:name +``` + +```shell +curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5/protected_branches/feature-branch?allow_force_push=true&code_owner_approval_required=true" +``` + +| Attribute | Type | Required | Description | +| -------------------------------------------- | ---- | -------- | ----------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the branch. | +| `allow_force_push` | boolean | no | When enabled, members who can push to this branch can also force push. | +| `allowed_to_push` | array | no | Array of push access levels, with each described by a hash. | +| `allowed_to_merge` | array | no | Array of merge access levels, with each described by a hash. | +| `allowed_to_unprotect` | array | no | Array of unprotect access levels, with each described by a hash. | +| `code_owner_approval_required` | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). Default: `false`. | + +Elements in the `allowed_to_push`, `allowed_to_merge` and `allowed_to_unprotect` arrays should: + +- Be one of `user_id`, `group_id`, or `access_level`. +- Take the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. + +To update: + +- `user_id`: Ensure the updated user has access to the project. You must also pass the + `id` of the `access_level` in the respective hash. +- `group_id`: Ensure the updated group [has this project shared](../user/project/members/share_project_with_groups.md). + You must also pass the `id` of the `access_level` in the respective hash. + +To delete: + +- You must pass `_destroy` set to `true`. See the following examples. + +### Example: create a `push_access_level` record + +```shell +curl --header 'Content-Type: application/json' --request PATCH \ + --data '{"allowed_to_push": [{access_level: 40}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" +``` + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [ + { + "id": 12, + "access_level": 40, + "access_level_description": "Maintainers", + "user_id": null, + "group_id": null + } + ] +} +``` + +### Example: update a `push_access_level` record + +```shell +curl --header 'Content-Type: application/json' --request PATCH \ + --data '{"allowed_to_push": [{"id": 12, "access_level": 0}]' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" +``` + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [ + { + "id": 12, + "access_level": 0, + "access_level_description": "No One", + "user_id": null, + "group_id": null + } + ] +} +``` + +### Example: delete a `push_access_level` record + +```shell +curl --header 'Content-Type: application/json' --request PATCH \ + --data '{"allowed_to_push": [{"id": 12, "_destroy": true}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_branches/master" +``` + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [] +} +``` diff --git a/doc/api/group_protected_environments.md b/doc/api/group_protected_environments.md index 3003b6b840f..bddfd774d43 100644 --- a/doc/api/group_protected_environments.md +++ b/doc/api/group_protected_environments.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/group_releases.md b/doc/api/group_releases.md index 9c395adbe04..978218041f2 100644 --- a/doc/api/group_releases.md +++ b/doc/api/group_releases.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/groups.md b/doc/api/groups.md index 8b4850fa6de..8c0e94ece28 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -110,11 +110,14 @@ GET /groups?statistics=true "packages_size": 0, "snippets_size": 50, "uploads_size": 0 - } + }, + "wiki_access_level": "private" } ] ``` +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `wiki_access_level` attribute. + You can search for groups by name or path, see below. You can filter by [custom attributes](custom_attributes.md) with: @@ -184,6 +187,8 @@ GET /groups/:id/subgroups ] ``` +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `wiki_access_level` attribute. + ## List a group's descendant groups > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217115) in GitLab 13.5 @@ -267,6 +272,8 @@ GET /groups/:id/descendant_groups ] ``` +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `wiki_access_level` attribute. + ## List a group's projects Get a list of projects in this group. When accessed without authentication, only public projects are returned. @@ -695,10 +702,15 @@ Example response: The `prevent_sharing_groups_outside_hierarchy` attribute is present only on top-level groups. -Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see -the `shared_runners_minutes_limit` and `extra_shared_runners_minutes_limit` parameters: +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the attributes: + +- `shared_runners_minutes_limit` +- `extra_shared_runners_minutes_limit` +- `marked_for_deletion_on` +- `membership_lock` +- `wiki_access_level` -Additional response parameters: +Additional response attributes: ```json { @@ -706,30 +718,9 @@ Additional response parameters: "description": "Aliquid qui quis dignissimos distinctio ut commodi voluptas est.", "shared_runners_minutes_limit": 133, "extra_shared_runners_minutes_limit": 133, - ... -} -``` - -Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see -the `marked_for_deletion_on` attribute: - -```json -{ - "id": 4, - "description": "Aliquid qui quis dignissimos distinctio ut commodi voluptas est.", "marked_for_deletion_on": "2020-04-03", - ... -} -``` - -Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see -the `membership_lock` attribute: - -```json -{ - "id": 4, - "description": "Aliquid qui quis dignissimos distinctio ut commodi voluptas est.", "membership_lock": false, + "wiki_access_level": "disabled", ... } ``` @@ -832,6 +823,7 @@ Parameters: | `membership_lock` **(PREMIUM)** | boolean | no | Users cannot be added to projects in this group. | | `extra_shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | | `shared_runners_minutes_limit` **(PREMIUM SELF)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | +| `wiki_access_level` **(PREMIUM)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | ### Options for `default_branch_protection` @@ -996,6 +988,7 @@ PUT /groups/:id | `unique_project_download_limit_alertlist` **(ULTIMATE)** | array of integers | no | List of user IDs that are emailed when the unique project download limit is exceeded. Available only on top-level groups. Default: `[]`, Maximum: 100 user IDs. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110201) in GitLab 15.9. | | `auto_ban_user_on_excessive_projects_download` **(ULTIMATE)** | boolean | no | When enabled, users are automatically banned from the group when they download more than the maximum number of unique projects specified by `unique_project_download_limit` and `unique_project_download_limit_interval_in_seconds`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/94159) in GitLab 15.4. | | `ip_restriction_ranges` **(PREMIUM)** | string | no | Comma-separated list of IP addresses or subnet masks to restrict group access. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351493) in GitLab 15.4. | +| `wiki_access_level` **(PREMIUM)** | string | no | The wiki access level. Can be `disabled`, `private`, or `enabled`. | NOTE: The `projects` and `shared_projects` attributes in the response are deprecated and [scheduled for removal in API v5](https://gitlab.com/gitlab-org/gitlab/-/issues/213797). @@ -1078,6 +1071,8 @@ Example response: The `prevent_sharing_groups_outside_hierarchy` attribute is present in the response only for top-level groups. +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `wiki_access_level` attribute. + ### Disable the results limit **(FREE SELF)** The 100 results limit can break integrations developed using GitLab 12.4 and earlier. diff --git a/doc/api/import.md b/doc/api/import.md index 5e9fbf30226..87bbb56869d 100644 --- a/doc/api/import.md +++ b/doc/api/import.md @@ -6,6 +6,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Import API **(FREE)** +Use the Import API to import repositories from GitHub or Bitbucket Server. + +Related APIs include: + +- [Group migration by direct transfer API](bulk_imports.md) +- [Group import and export API](group_import_export.md) +- [Project import and export API](project_import_export.md) + +## Prerequisites + +For information on prerequisites for using the Import API, see: + +- [Prerequisites for GitHub importer](../user/project/import/github.md#prerequisites) +- [Prerequisites for Bitbucket Server importer](../user/project/import/bitbucket_server.md#import-your-bitbucket-repositories) + ## Import repository from GitHub > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/381902) in GitLab 15.8, GitLab no longer automatically creates namespaces or groups if the namespace or group name specified in `target_namespace` doesn't exist. GitLab also no longer falls back to using the user's personal namespace if the namespace or group name is taken or `target_namespace` is blank. @@ -77,7 +92,7 @@ token: - The GitLab project inherits the original project's visibility settings. As a result, the project is publicly accessible if the original project is public. - If the `path` or `target_namespace` does not exist, the project import fails. -## Cancel GitHub project import +### Cancel GitHub project import > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364783) in GitLab 15.5. @@ -122,13 +137,14 @@ Returns the following status codes: - `400 Bad Request`: the project import cannot be canceled. - `404 Not Found`: the project associated with `project_id` does not exist. -## Import GitHub gists into GitLab snippets +### Import GitHub gists into GitLab snippets -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371099) in GitLab 15.8 [with a flag](../administration/feature_flags.md) named `github_import_gists`. Disabled by default. Enabled on GitLab.com. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371099) in GitLab 15.8 [with a flag](../administration/feature_flags.md) named `github_import_gists`. Disabled by default. Enabled on GitLab.com. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/386579) in GitLab 15.10. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `github_import_gists`. +On self-managed GitLab, this feature is available by default. To hide the feature, +ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `github_import_gists`. On GitLab.com, this feature is available. You can use the GitLab API to import personal GitHub gists (with up to 10 files) into personal GitLab snippets. diff --git a/doc/api/instance_clusters.md b/doc/api/instance_clusters.md index 45243003004..51296c2e96e 100644 --- a/doc/api/instance_clusters.md +++ b/doc/api/instance_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Instance clusters API (certificate-based) (DEPRECATED) **(FREE SELF)** +# Instance clusters API (certificate-based) (deprecated) **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36001) in GitLab 13.2. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/api/instance_level_ci_variables.md b/doc/api/instance_level_ci_variables.md index e71cf777b2c..840744bcae1 100644 --- a/doc/api/instance_level_ci_variables.md +++ b/doc/api/instance_level_ci_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/integrations.md b/doc/api/integrations.md index 24e0f189aad..e25753b892e 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -96,7 +96,7 @@ Parameters: ### Disable Apple App Store integration -Disable the Apple App Store integration for a project. Integration settings are preserved. +Disable the Apple App Store integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/apple_app_store @@ -133,7 +133,7 @@ Parameters: ### Disable Asana integration -Disable the Asana integration for a project. Integration settings are preserved. +Disable the Asana integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/asana @@ -168,7 +168,7 @@ Parameters: ### Disable Assembla integration -Disable the Assembla integration for a project. Integration settings are preserved. +Disable the Assembla integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/assembla @@ -208,7 +208,7 @@ Parameters: ### Disable Atlassian Bamboo CI integration -Disable the Atlassian Bamboo CI integration for a project. Integration settings are preserved. +Disable the Atlassian Bamboo CI integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/bamboo @@ -244,7 +244,7 @@ Parameters: ### Disable Bugzilla integration -Disable the Bugzilla integration for a project. Integration settings are preserved. +Disable the Bugzilla integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/bugzilla @@ -283,7 +283,7 @@ Parameters: ### Disable Buildkite integration -Disable the Buildkite integration for a project. Integration settings are preserved. +Disable the Buildkite integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/buildkite @@ -320,7 +320,7 @@ Parameters: ### Disable Campfire integration -Disable the Campfire integration for a project. Integration settings are preserved. +Disable the Campfire integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/campfire @@ -360,7 +360,7 @@ Parameters: ### Disable Datadog integration -Disable the Datadog integration for a project. Integration settings are preserved. +Disable the Datadog integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/datadog @@ -405,7 +405,7 @@ Parameters: ### Disable Unify Circuit integration -Disable the Unify Circuit integration for a project. Integration settings are preserved. +Disable the Unify Circuit integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/unify-circuit @@ -450,7 +450,7 @@ Parameters: ### Disable Pumble integration -Disable the Pumble integration for a project. Integration settings are preserved. +Disable the Pumble integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/pumble @@ -495,7 +495,7 @@ Parameters: ### Disable Webex Teams integration -Disable the Webex Teams integration for a project. Integration settings are preserved. +Disable the Webex Teams integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/webex-teams @@ -531,7 +531,7 @@ Parameters: ### Disable Custom Issue Tracker integration -Disable the Custom Issue Tracker integration for a project. Integration settings are preserved. +Disable the Custom Issue Tracker integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/custom-issue-tracker @@ -565,7 +565,7 @@ Parameters: ### Disable Discord integration -Disable the Discord integration for a project. Integration settings are preserved. +Disable the Discord integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/discord @@ -604,7 +604,7 @@ Parameters: ### Disable Drone CI integration -Disable the Drone CI integration for a project. Integration settings are preserved. +Disable the Drone CI integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/drone-ci @@ -643,7 +643,7 @@ Parameters: ### Disable Emails on Push integration -Disable the Emails on Push integration for a project. Integration settings are preserved. +Disable the Emails on Push integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/emails-on-push @@ -679,7 +679,7 @@ Parameters: ### Disable EWM integration -Disable the EWM integration for a project. Integration settings are preserved. +Disable the EWM integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/ewm @@ -715,7 +715,7 @@ Parameters: ### Disable Confluence integration -Disable the Confluence integration for a project. Integration settings are preserved. +Disable the Confluence integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/confluence @@ -753,7 +753,7 @@ Parameters: ### Disable Shimo integration -Disable the Shimo integration for a project. Integration settings are preserved. +Disable the Shimo integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/shimo @@ -779,7 +779,7 @@ Parameters: ### Disable External wiki integration -Disable the External wiki integration for a project. Integration settings are preserved. +Disable the External wiki integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/external-wiki @@ -815,7 +815,7 @@ Parameters: ### Disable GitHub integration -Disable the GitHub integration for a project. Integration settings are preserved. +Disable the GitHub integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/github @@ -861,7 +861,7 @@ Parameters: ### Disable Hangouts Chat integration -Disable the Hangouts Chat integration for a project. Integration settings are preserved. +Disable the Hangouts Chat integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/hangouts-chat @@ -901,7 +901,7 @@ Parameters: ### Disable Irker (IRC gateway) integration -Disable the Irker (IRC gateway) integration for a project. Integration settings are preserved. +Disable the Irker (IRC gateway) integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/irker @@ -952,7 +952,7 @@ Parameters: ### Disable Jira integration -Disable the Jira integration for a project. Integration settings are preserved. +Disable the Jira integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/jira @@ -1011,7 +1011,7 @@ Parameters: ### Disable Slack Slash Command integration -Disable the Slack Slash Command integration for a project. Integration settings are preserved. +Disable the Slack Slash Command integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/slack-slash-commands @@ -1045,7 +1045,7 @@ Parameters: ### Disable Mattermost Slash Command integration -Disable the Mattermost Slash Command integration for a project. Integration settings are preserved. +Disable the Mattermost Slash Command integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/mattermost-slash-commands @@ -1076,7 +1076,7 @@ Parameters: ### Disable Packagist integration -Disable the Packagist integration for a project. Integration settings are preserved. +Disable the Packagist integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/packagist @@ -1114,7 +1114,7 @@ Parameters: ### Disable Pipeline-Emails integration -Disable the Pipeline-Emails integration for a project. Integration settings are preserved. +Disable the Pipeline-Emails integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/pipelines-email @@ -1151,7 +1151,7 @@ Parameters: ### Disable Pivotal Tracker integration -Disable the Pivotal Tracker integration for a project. Integration settings are preserved. +Disable the Pivotal Tracker integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/pivotaltracker @@ -1187,7 +1187,7 @@ Parameters: ### Disable Prometheus integration -Disable the Prometheus integration for a project. Integration settings are preserved. +Disable the Prometheus integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/prometheus @@ -1225,7 +1225,7 @@ Parameters: ### Disable Pushover integration -Disable the Pushover integration for a project. Integration settings are preserved. +Disable the Pushover integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/pushover @@ -1261,7 +1261,7 @@ Parameters: ### Disable Redmine integration -Disable the Redmine integration for a project. Integration settings are preserved. +Disable the Redmine integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/redmine @@ -1322,7 +1322,7 @@ Parameters: ### Disable Slack integration -Disable the Slack integration for a project. Integration settings are preserved. +Disable the Slack integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/slack @@ -1368,7 +1368,7 @@ Parameters: ### Disable Microsoft Teams integration -Disable the Microsoft Teams integration for a project. Integration settings are preserved. +Disable the Microsoft Teams integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/microsoft-teams @@ -1425,7 +1425,7 @@ Parameters: ### Disable Mattermost notifications integration -Disable the Mattermost notifications integration for a project. Integration settings are preserved. +Disable the Mattermost notifications integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/mattermost @@ -1467,7 +1467,7 @@ Parameters: ### Disable JetBrains TeamCity CI integration -Disable the JetBrains TeamCity CI integration for a project. Integration settings are preserved. +Disable the JetBrains TeamCity CI integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/teamcity @@ -1508,7 +1508,7 @@ Parameters: ### Disable Jenkins CI integration -Disable the Jenkins CI integration for a project. Integration settings are preserved. +Disable the Jenkins CI integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/jenkins @@ -1545,7 +1545,7 @@ Parameters: ### Disable Jenkins CI (Deprecated) integration -Disable the Jenkins CI (Deprecated) integration for a project. Integration settings are preserved. +Disable the Jenkins CI (Deprecated) integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/jenkins-deprecated @@ -1582,7 +1582,7 @@ Parameters: ### Disable MockCI integration -Disable the MockCI integration for a project. Integration settings are preserved. +Disable the MockCI integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/mock-ci @@ -1596,6 +1596,43 @@ Get MockCI integration settings for a project. GET /projects/:id/integrations/mock-ci ``` +## Squash TM + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337855) in GitLab 15.10. + +Update [Squash TM](https://www.squashtest.com/product-squash-tm?lang=en) requirements when GitLab issues are modified. + +### Create/Edit Squash TM integration + +Set Squash TM integration settings for a project. + +```plaintext +PUT /projects/:id/integrations/squash-tm +``` + +Parameters: + +| Parameter | Type | Required | Description | +|-------------------------|--------|----------|-------------------------------| +| `url` | string | yes | URL of the Squash TM webhook. | +| `token` | string | no | Optional token | + +### Disable Squash TM integration + +Disable the Squash TM integration for a project. Integration settings are preserved. + +```plaintext +DELETE /projects/:id/integrations/squash-tm +``` + +### Get Squash TM integration settings + +Get Squash TM integration settings for a project. + +```plaintext +GET /projects/:id/integrations/squash-tm +``` + ## YouTrack YouTrack issue tracker @@ -1617,7 +1654,7 @@ Parameters: ### Disable YouTrack integration -Disable the YouTrack integration for a project. Integration settings are preserved. +Disable the YouTrack integration for a project. Integration settings are reset. ```plaintext DELETE /projects/:id/integrations/youtrack diff --git a/doc/api/iterations.md b/doc/api/iterations.md index 4997a917a5a..ca280437eaf 100644 --- a/doc/api/iterations.md +++ b/doc/api/iterations.md @@ -24,6 +24,8 @@ GET /projects/:id/iterations?state=opened GET /projects/:id/iterations?state=closed GET /projects/:id/iterations?search=version GET /projects/:id/iterations?include_ancestors=false +GET /projects/:id/iterations?updated_before=2013-10-02T09%3A24%3A18Z +GET /projects/:id/iterations?updated_after=2013-10-02T09%3A24%3A18Z ``` | Attribute | Type | Required | Description | @@ -31,6 +33,8 @@ GET /projects/:id/iterations?include_ancestors=false | `state` | string | no | 'Return `opened`, `upcoming`, `current (previously started)`, `closed`, or `all` iterations. Filtering by `started` state is deprecated starting with 14.1, please use `current` instead.' | | `search` | string | no | Return only iterations with a title matching the provided string. | | `include_ancestors` | boolean | no | Include iterations from parent group and its ancestors. Defaults to `true`. | +| `updated_before` | datetime | no | Return only iterations updated before the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378662) in GitLab 15.10. | +| `updated_after` | datetime | no | Return only iterations updated after the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378662) in GitLab 15.10. | Example request: diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index b73eafea3c4..715b0614cc4 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/keys.md b/doc/api/keys.md index e7bdc70017c..90144310238 100644 --- a/doc/api/keys.md +++ b/doc/api/keys.md @@ -7,6 +7,8 @@ type: reference, api # Keys API **(FREE)** +If using a SHA256 fingerprint in an API call, you should URL-encode the fingerprint. + ## Get SSH key with user by ID of an SSH key Get SSH key with user by ID of an SSH key. Note only administrators can lookup SSH key with user by ID of an SSH key. @@ -22,7 +24,8 @@ GET /keys/:id Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/keys/1" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/keys/1" ``` ```json @@ -82,15 +85,8 @@ GET /keys Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/keys?fingerprint=ba:81:59:68:d7:6c:cd:02:02:bf:6a:9b:55:4e:af:d1" -``` - -If using sha256 fingerprint API calls, make sure that the fingerprint is URL-encoded. - -For example, `/` is represented by `%2F` and `:` is represented by`%3A`: - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/keys?fingerprint=SHA256%3AnUhzNyftwADy8AH3wFY31tAKs7HufskYTte2aXo%2FlCg" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/keys?fingerprint=ba:81:59:68:d7:6c:cd:02:02:bf:6a:9b:55:4e:af:d1" ``` Example response: @@ -142,15 +138,21 @@ Example response: ## Get user by deploy key fingerprint -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119209) in GitLab 12.7. +Deploy keys are bound to the creating user. If you query with a deploy key +fingerprint, you get additional information about the projects using that key. -Deploy keys are bound to the creating user, so if you query with a deploy key -fingerprint you get additional information about the projects using that key. +Example request with an MD5 fingerprint: -Example request: +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/keys?fingerprint=ba:81:59:68:d7:6c:cd:02:02:bf:6a:9b:55:4e:af:d1" +``` + +In this SHA256 example, `/` is represented by `%2F` and `:` is represented by`%3A`: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/keys?fingerprint=SHA256%3AnUhzNyftwADy8AH3wFY31tAKs7HufskYTte2aXo%2FlCg" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/keys?fingerprint=SHA256%3AnUhzNyftwADy8AH3wFY31tAKs7HufskYTte2aXo%2FlCg" ``` Example response: diff --git a/doc/api/license.md b/doc/api/license.md index ca9d9cf386d..39da6af30d4 100644 --- a/doc/api/license.md +++ b/doc/api/license.md @@ -233,3 +233,35 @@ Returns: - `204 No Content` if the license is successfully deleted. - `403 Forbidden` if the current user in not permitted to delete the license. - `404 Not Found` if the license to delete could not be found. + +## Trigger recalculation of billable users + +```plaintext +PUT /license/:id/refresh_billable_users +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | ID of the GitLab license. | + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license/:id/refresh_billable_users" +``` + +Example response: + +```json +{ + "success": true +} +``` + +Returns: + +- `202 Accepted` if the request to refresh billable users is successfully initiated. +- `403 Forbidden` if the current user in not permitted to refresh billable users for the license. +- `404 Not Found` if the license could not be found. + +| Attribute | Type | Description | +|:-----------------------------|:--------------|:------------------------------------------| +| `success` | boolean | Whether the request succeeded or not. | diff --git a/doc/api/lint.md b/doc/api/lint.md index e50832a9528..9de9aa23560 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -6,7 +6,113 @@ info: To determine the technical writer assigned to the Stage/Group associated w # CI Lint API **(FREE)** -## Validate the CI YAML configuration +## Validate a CI YAML configuration with a namespace + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231352) in GitLab 13.6. + +Checks if CI/CD YAML configuration is valid. This endpoint has namespace +specific context. + +```plaintext +POST /projects/:id/ci/lint +``` + +| Attribute | Type | Required | Description | +| ---------- | ------- | -------- | -------- | +| `content` | string | yes | The CI/CD configuration content. | +| `dry_run` | boolean | no | Run [pipeline creation simulation](../ci/lint.md#simulate-a-pipeline), or only do static check. This is false by default. | +| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | +| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | + +Example request: + +```shell +curl --header "Content-Type: application/json" "https://gitlab.example.com/api/v4/projects/:id/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' +``` + +Example responses: + +- Valid configuration: + + ```json + { + "valid": true, + "merged_yaml": "---\n:test_job:\n :script: echo 1\n", + "errors": [], + "warnings": [] + } + ``` + +- Invalid configuration: + + ```json + { + "valid": false, + "merged_yaml": "---\n:test_job:\n :script: echo 1\n", + "errors": [ + "jobs config should contain at least one visible job" + ], + "warnings": [] + } + ``` + +## Validate a project's CI configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231352) in GitLab 13.5. + +Checks if a project's latest (`HEAD` of the project's default branch) +`.gitlab-ci.yml` configuration is valid. This endpoint uses all namespace +specific data available, including variables and local includes. + +```plaintext +GET /projects/:id/ci/lint +``` + +| Attribute | Type | Required | Description | +| ---------- | ------- | -------- | -------- | +| `dry_run` | boolean | no | Run pipeline creation simulation, or only do static check. | +| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | +| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | + +Example request: + +```shell +curl "https://gitlab.example.com/api/v4/projects/:id/ci/lint" +``` + +Example responses: + +- Valid configuration: + +```json +{ + "valid": true, + "merged_yaml": "---\n:test_job:\n :script: echo 1\n", + "errors": [], + "warnings": [] +} +``` + +- Invalid configuration: + +```json +{ + "valid": false, + "merged_yaml": "---\n:test_job:\n :script: echo 1\n", + "errors": [ + "jobs config should contain at least one visible job" + ], + "warnings": [] +} +``` + +<!--- start_remove The following content will be removed on remove_date: '2023-08-22' --> + +## Validate the CI YAML configuration (deprecated) + +WARNING: +This endpoint was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/381669) in GitLab 15.7 +and is planned for removal in 16.0. Use [`POST /projects/:id/ci/lint`](#validate-a-ci-yaml-configuration-with-a-namespace) instead. Checks if CI/CD YAML configuration is valid. This endpoint validates basic CI/CD configuration syntax. It doesn't have any namespace-specific context. @@ -154,105 +260,7 @@ Example response: } ``` -## Validate a CI YAML configuration with a namespace - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231352) in GitLab 13.6. - -Checks if CI/CD YAML configuration is valid. This endpoint has namespace -specific context. - -```plaintext -POST /projects/:id/ci/lint -``` - -| Attribute | Type | Required | Description | -| ---------- | ------- | -------- | -------- | -| `content` | string | yes | The CI/CD configuration content. | -| `dry_run` | boolean | no | Run [pipeline creation simulation](../ci/lint.md#simulate-a-pipeline), or only do static check. This is false by default. | -| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | -| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | - -Example request: - -```shell -curl --header "Content-Type: application/json" "https://gitlab.example.com/api/v4/projects/:id/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' -``` - -Example responses: - -- Valid configuration: - - ```json - { - "valid": true, - "merged_yaml": "---\n:test_job:\n :script: echo 1\n", - "errors": [], - "warnings": [] - } - ``` - -- Invalid configuration: - - ```json - { - "valid": false, - "merged_yaml": "---\n:test_job:\n :script: echo 1\n", - "errors": [ - "jobs config should contain at least one visible job" - ], - "warnings": [] - } - ``` - -## Validate a project's CI configuration - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231352) in GitLab 13.5. - -Checks if a project's latest (`HEAD` of the project's default branch) -`.gitlab-ci.yml` configuration is valid. This endpoint uses all namespace -specific data available, including variables and local includes. - -```plaintext -GET /projects/:id/ci/lint -``` - -| Attribute | Type | Required | Description | -| ---------- | ------- | -------- | -------- | -| `dry_run` | boolean | no | Run pipeline creation simulation, or only do static check. | -| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | -| `ref` | string | no | When `dry_run` is `true`, sets the branch or tag to use. Defaults to the project's default branch when not set. | - -Example request: - -```shell -curl "https://gitlab.example.com/api/v4/projects/:id/ci/lint" -``` - -Example responses: - -- Valid configuration: - -```json -{ - "valid": true, - "merged_yaml": "---\n:test_job:\n :script: echo 1\n", - "errors": [], - "warnings": [] -} -``` - -- Invalid configuration: - -```json -{ - "valid": false, - "merged_yaml": "---\n:test_job:\n :script: echo 1\n", - "errors": [ - "jobs config should contain at least one visible job" - ], - "warnings": [] -} -``` +<!--- end_remove --> ## Use jq to create and process YAML & JSON payloads diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md index 6aee60c57e0..b98be629e90 100644 --- a/doc/api/managed_licenses.md +++ b/doc/api/managed_licenses.md @@ -4,7 +4,10 @@ group: Utilization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Managed Licenses API **(ULTIMATE)** +# Managed Licenses API (deprecated) **(ULTIMATE)** + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390417) in GitLab 15.9. WARNING: "approval" and "blacklisted" approval statuses are changed to "allowed" and "denied" in GitLab 15.0. diff --git a/doc/api/member_roles.md b/doc/api/member_roles.md index a7fc93e0df5..3ef6e287418 100644 --- a/doc/api/member_roles.md +++ b/doc/api/member_roles.md @@ -63,6 +63,8 @@ Adds a member role to a group. POST /groups/:id/member_roles ``` +To add a member role to a group, the group must be at root-level (have no parent group). + | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index 0df2b90e64d..79a70bddfee 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -312,7 +312,7 @@ Supported attributes: | `applies_to_all_protected_branches` | boolean | **{dotted-circle}** No | Whether the rule is applied to all protected branches. If set to `true`, the value of `protected_branch_ids` is ignored. Default is `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335316) in GitLab 15.3. | | `group_ids` | Array | **{dotted-circle}** No | The IDs of groups as approvers. | | `protected_branch_ids` | Array | **{dotted-circle}** No | The IDs of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | -| `report_type` | string | **{dotted-circle}** No | The report type required when the rule type is `report_approver`. The supported report types are `license_scanning` and `code_coverage`. | +| `report_type` | string | **{dotted-circle}** No | The report type required when the rule type is `report_approver`. The supported report types are `license_scanning` [(Deprecated in GitLab 15.9)](../update/deprecations.md#license-check-and-the-policies-tab-on-the-license-compliance-page) and `code_coverage`. | | `rule_type` | string | **{dotted-circle}** No | The type of rule. `any_approver` is a pre-configured default rule with `approvals_required` at `0`. Other rules are `regular`. | | `user_ids` | Array | **{dotted-circle}** No | The IDs of users as approvers. | | `usernames` | string array | **{dotted-circle}** No | The usernames for this rule. | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 024593b2c6b..3e5982faee8 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1174,7 +1174,8 @@ Example response: ## List merge request pipelines -Get a list of merge request pipelines. +Get a list of merge request pipelines. The pagination parameters `page` and +`per_page` can be used to restrict the list of merge request pipelines. ```plaintext GET /projects/:id/merge_requests/:merge_request_iid/pipelines diff --git a/doc/api/merge_trains.md b/doc/api/merge_trains.md index 6d5d12a618c..afa5b38b9c3 100644 --- a/doc/api/merge_trains.md +++ b/doc/api/merge_trains.md @@ -220,3 +220,74 @@ Example response: "duration":null } ``` + +## Add a merge request to a merge train + +Add a merge request to the merge train targeting the merge request's target branch. + +```plaintext +POST /projects/:id/merge_trains/merge_requests/:merge_request_iid +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| ------------------------------ | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | +| `merge_request_iid` | integer | yes | The internal ID of the merge request. | +| `when_pipeline_succeeds` | boolean | no | If true, the merge request is added to the merge train when the pipeline succeeds. When false or unspecified, the merge request is added directly to the merge train. | +| `sha` | string | no | If present, the SHA must match the `HEAD` of the source branch, otherwise the merge fails. | +| `squash` | boolean | no | If true, the commits are squashed into a single commit on merge. | + +Example request: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/597/merge_trains/merge_requests/1" +``` + +Example response: + +```json +[ + { + "id": 267, + "merge_request": { + "id": 273, + "iid": 1, + "project_id": 597, + "title": "My title 9", + "description": null, + "state": "opened", + "created_at": "2022-10-31T19:06:05.725Z", + "updated_at": "2022-10-31T19:06:05.725Z", + "web_url": "http://localhost/namespace18/project21/-/merge_requests/1" + }, + "user": { + "id": 933, + "username": "user12", + "name": "Sidney Jones31", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/6c8365de387cb3db10ecc7b1880203c4?s=80\u0026d=identicon", + "web_url": "http://localhost/user12" + }, + "pipeline": { + "id": 273, + "iid": 1, + "project_id": 598, + "sha": "b83d6e391c22777fca1ed3012fce84f633d7fed0", + "ref": "main", + "status": "pending", + "source": "push", + "created_at": "2022-10-31T19:06:06.231Z", + "updated_at": "2022-10-31T19:06:06.231Z", + "web_url": "http://localhost/namespace19/project22/-/pipelines/273" + }, + "created_at": "2022-10-31T19:06:06.237Z", + "updated_at":"2022-10-31T19:06:06.237Z", + "target_branch":"main", + "status":"idle", + "merged_at":null, + "duration":null + } +] +``` diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 998beeb9b3b..e1acf4c14bb 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -21,6 +21,8 @@ GET /projects/:id/milestones?state=active GET /projects/:id/milestones?state=closed GET /projects/:id/milestones?title=1.0 GET /projects/:id/milestones?search=version +GET /projects/:id/milestones?updated_before=2013-10-02T09%3A24%3A18Z +GET /projects/:id/milestones?updated_after=2013-10-02T09%3A24%3A18Z ``` Parameters: @@ -28,11 +30,13 @@ Parameters: | Attribute | Type | Required | Description | | ---------------------------- | ------ | -------- | ----------- | | `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `iids[]` | integer array | optional | Return only the milestones having the given `iid` (Note: ignored if `include_parent_milestones` is set as `true`) | -| `state` | string | optional | Return only `active` or `closed` milestones | -| `title` | string | optional | Return only the milestones having the given `title` | -| `search` | string | optional | Return only milestones with a title or description matching the provided string | -| `include_parent_milestones` | boolean | optional | Include group milestones from parent group and its ancestors. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196066) in GitLab 13.4 | +| `iids[]` | integer array | no | Return only the milestones having the given `iid` (Note: ignored if `include_parent_milestones` is set as `true`) | +| `state` | string | no | Return only `active` or `closed` milestones | +| `title` | string | no | Return only the milestones having the given `title` | +| `search` | string | no | Return only milestones with a title or description matching the provided string | +| `include_parent_milestones` | boolean | no | Include group milestones from parent group and its ancestors. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196066) in GitLab 13.4 | +| `updated_before` | datetime | no | Return only milestones updated before the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). Introduced in GitLab 15.10 | +| `updated_after` | datetime | no | Return only milestones updated after the given datetime. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). Introduced in GitLab 15.10 | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/milestones" diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index cbf7ea079bc..25c26ee9339 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -100,9 +100,9 @@ Owners also see the `plan` property associated with a namespace: ] ``` -Users on GitLab.com also see `max_seats_used` and `seats_in_use` parameters. +Users on GitLab.com also see `max_seats_used`, `seats_in_use` and `max_seats_used_changed_at` parameters. `max_seats_used` is the highest number of users the group had. `seats_in_use` is -the number of license seats currently being used. Both values are updated +the number of license seats currently being used. `max_seats_used_changed_at` shows the date when the `max_seats_used` value changed. All the values are updated once a day. `max_seats_used` and `seats_in_use` are non-zero only for namespaces on paid plans. @@ -114,6 +114,7 @@ once a day. "name": "user1", "billable_members_count": 2, "max_seats_used": 3, + "max_seats_used_changed_at":"2023-02-13T12:00:02.000Z", "seats_in_use": 2, ... } diff --git a/doc/api/notes.md b/doc/api/notes.md index 9c453c6390f..305bdd294c5 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -78,6 +78,7 @@ GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at "system": true, "noteable_id": 377, "noteable_type": "Issue", + "project_id": 5, "noteable_iid": 377, "resolvable": false, "confidential": false, @@ -100,6 +101,7 @@ GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at "system": true, "noteable_id": 121, "noteable_type": "Issue", + "project_id": 5, "noteable_iid": 121, "resolvable": false, "confidential": true, @@ -239,9 +241,9 @@ Parameters: ```json { - "id": 52, - "title": "Snippet", - "file_name": "snippet.rb", + "id": 302, + "body": "closed", + "attachment": null, "author": { "id": 1, "username": "pipin", @@ -250,9 +252,16 @@ Parameters: "state": "active", "created_at": "2013-09-30T13:46:01Z" }, - "expires_at": null, - "updated_at": "2013-10-02T07:34:20Z", - "created_at": "2013-10-02T07:34:20Z" + "created_at": "2013-10-02T09:22:45Z", + "updated_at": "2013-10-02T10:22:45Z", + "system": true, + "noteable_id": 377, + "noteable_type": "Issue", + "project_id": 5, + "noteable_iid": 377, + "resolvable": false, + "confidential": false, + "internal": false } ``` @@ -379,6 +388,7 @@ Parameters: "system": false, "noteable_id": 2, "noteable_type": "MergeRequest", + "project_id": 5, "noteable_iid": 2, "resolvable": false, "confidential": false, @@ -392,8 +402,13 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ### Create new merge request note -Creates a new note for a single merge request. -If you create a note where the body only contains an Award Emoji, GitLab returns this object. +Creates a new note for a single merge request. Notes are not attached to specific +lines in a merge request. For other approaches with more granular control, see +[Post comment to commit](commits.md#post-comment-to-commit) in the Commits API, +and [Create a new thread in the merge request diff](discussions.md#create-a-new-thread-in-the-merge-request-diff) +in the Discussions API. + +If you create a note where the body only contains an award emoji, GitLab returns this object. ```plaintext POST /projects/:id/merge_requests/:merge_request_iid/notes @@ -496,9 +511,9 @@ Parameters: ```json { - "id": 52, - "title": "Epic", - "file_name": "epic.rb", + "id": 302, + "body": "Epic note", + "attachment": null, "author": { "id": 1, "username": "pipin", @@ -507,9 +522,14 @@ Parameters: "state": "active", "created_at": "2013-09-30T13:46:01Z" }, - "expires_at": null, - "updated_at": "2013-10-02T07:34:20Z", - "created_at": "2013-10-02T07:34:20Z", + "created_at": "2013-10-02T09:22:45Z", + "updated_at": "2013-10-02T10:22:45Z", + "system": true, + "noteable_id": 11, + "noteable_type": "Epic", + "project_id": 5, + "noteable_iid": 11, + "resolvable": false, "confidential": false, "internal": false } diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 3e470c5cb91..eb9d1d3bc8a 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # OAuth 2.0 identity provider API **(FREE)** GitLab provides an API to allow third-party services to access GitLab resources on a user's behalf -with the [OAuth2](https://oauth.net/2/) protocol. +with the [OAuth 2.0](https://oauth.net/2/) protocol. To configure GitLab for this, see [Configure GitLab as an OAuth 2.0 authentication identity provider](../integration/oauth_provider.md). @@ -45,6 +45,8 @@ GitLab supports the following authorization flows: - **Resource owner password credentials:** To be used **only** for securely hosted, first-party services. GitLab recommends against use of this flow. +Device Authorization Grant is not supported. [Issue 332682](https://gitlab.com/gitlab-org/gitlab/-/issues/332682) proposes to add support. + The draft specification for [OAuth 2.1](https://oauth.net/2.1/) specifically omits both the Implicit grant and Resource Owner Password Credentials flows. diff --git a/doc/api/openapi/openapi.yaml b/doc/api/openapi/openapi.yaml index 9ee2b8119be..e090eab1e5d 100644 --- a/doc/api/openapi/openapi.yaml +++ b/doc/api/openapi/openapi.yaml @@ -20,7 +20,7 @@ info: The feature uses the current [GitLab session cookie](https://docs.gitlab.com/ee/api/index.html#session-cookie), so each request is made using your account. - Instructions for using this tool can be found in [Interactive API Documentation](openapi_interactive.md). + Instructions for using this tool can be found in [Interactive API Documentation](https://docs.gitlab.com/ee/api/openapi/openapi_interactive.html). version: v4 title: GitLab API diff --git a/doc/api/packages.md b/doc/api/packages.md index 32b21ce354d..86eaf3028cf 100644 --- a/doc/api/packages.md +++ b/doc/api/packages.md @@ -352,6 +352,9 @@ Can return the following status codes: - `204 No Content`, if the package was deleted successfully. - `404 Not Found`, if the package was not found. +If [request forwarding](../user/packages/package_registry/supported_functionality.md#forwarding-requests) is enabled, +deleting a package can introduce a [dependency confusion risk](../user/packages/package_registry/supported_functionality.md#deleting-packages). + ## Delete a package file > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32107) in GitLab 13.12. diff --git a/doc/api/pages.md b/doc/api/pages.md index dbe7416b939..ed63090b9be 100644 --- a/doc/api/pages.md +++ b/doc/api/pages.md @@ -10,9 +10,13 @@ Endpoints for managing [GitLab Pages](https://about.gitlab.com/stages-devops-lif The GitLab Pages feature must be enabled to use these endpoints. Find out more about [administering](../administration/pages/index.md) and [using](../user/project/pages/index.md) the feature. -## Unpublish pages +## Unpublish Pages -Remove pages. The user must have administrator access. +Prerequisite: + +- You must have administrator access to the instance. + +Remove Pages. ```plaintext DELETE /projects/:id/pages diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index 815cea7cc63..e83811d0415 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -12,7 +12,11 @@ The GitLab Pages feature must be enabled to use these endpoints. Find out more a ## List all Pages domains -Get a list of all Pages domains. The user must have administrator access. +Prerequisite: + +- You must have administrator access to the instance. + +Get a list of all Pages domains. ```plaintext GET /pages/domains diff --git a/doc/api/pipeline_triggers.md b/doc/api/pipeline_triggers.md index 1ffda1c0d79..50acac6bc2a 100644 --- a/doc/api/pipeline_triggers.md +++ b/doc/api/pipeline_triggers.md @@ -155,7 +155,7 @@ or a [CI/CD job token](../ci/jobs/ci_job_token.md) for authentication. With a CI/CD job token, the [triggered pipeline is a multi-project pipeline](../ci/pipelines/downstream_pipelines.md#trigger-a-multi-project-pipeline-by-using-the-api). The job that authenticates the request becomes associated with the upstream pipeline, -which is visible on the [pipeline graph](../ci/pipelines/downstream_pipelines.md#view-multi-project-pipelines-in-pipeline-graphs). +which is visible on the pipeline graph. If you use a trigger token in a job, the job is not associated with the upstream pipeline. diff --git a/doc/api/product_analytics.md b/doc/api/product_analytics.md index c687acdb5db..8eda24d1c65 100644 --- a/doc/api/product_analytics.md +++ b/doc/api/product_analytics.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Product analytics API **(ULTIMATE)** -> Introduced in GitLab 15.4 [with a flag](../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. +> - Introduced in GitLab 15.4 [with a flag](../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. +> - `cube_api_proxy` removed and replaced with `product_analytics_internal_preview` in GitLab 15.10. FLAG: On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `cube_api_proxy`. diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md index 3dad40a3f96..a8940a7875c 100644 --- a/doc/api/project_badges.md +++ b/doc/api/project_badges.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Placeholder tokens -Badges support placeholders that are replaced in real-time in both the link and image URL. The allowed placeholders are: +[Badges](../user/project/badges.md) support placeholders that are replaced in real-time in both the link and image URL. The allowed placeholders are: <!-- vale gitlab.Spelling = NO --> diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 01081bdd6d9..f6d572f9a7d 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project clusters API (certificate-based) (DEPRECATED) **(FREE)** +# Project clusters API (certificate-based) (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23922) in GitLab 11.7. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index 00e73d41b46..758c91dd3e7 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -4,12 +4,17 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project import/export API **(FREE)** +# Project import and export API **(FREE)** -See also: +Use the project import and export API to import and export projects using file transfers. -- [Project import/export documentation](../user/project/settings/import_export.md). -- [Project import/export administration Rake tasks](../administration/raketasks/project_import_export.md). **(FREE SELF)** +For more information, see: + +- [Migrating projects using file exports](../user/project/settings/import_export.md) +- [Project import and export Rake tasks](../administration/raketasks/project_import_export.md) + +Before using the project import and export API, you might want to use the +[group import and export API](group_import_export.md). ## Schedule an export @@ -135,6 +140,7 @@ POST /projects/import | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ---------------------------------------- | +<!-- markdownlint-disable-next-line gitlab.SentenceSpacing --> | `namespace` | integer/string | no | The ID or path of the namespace to import the project to. Defaults to the current user's namespace.<br/><br/>Requires at least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0. | | `name` | string | no | The name of the project to be imported. Defaults to the path of the project if not provided | | `file` | string | yes | The file to be uploaded | diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 92ca2849e8e..fa699c34a8a 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, api --- diff --git a/doc/api/project_vulnerabilities.md b/doc/api/project_vulnerabilities.md index 9effe54b8e2..446c629c3bf 100644 --- a/doc/api/project_vulnerabilities.md +++ b/doc/api/project_vulnerabilities.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Govern +group: Threat Insights info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, api --- diff --git a/doc/api/projects.md b/doc/api/projects.md index b502d547ddc..3cd7cdd811d 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -1433,6 +1433,7 @@ Supported attributes: |-------------------------------------------------------------|----------------|------------------------|-------------| | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding). | | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | +| `allow_pipeline_trigger_approve_deployment` **(PREMIUM)** | boolean | **{dotted-circle}** No | Set whether or not a pipeline triggerer is allowed to approve deployments. | | `only_allow_merge_if_all_status_checks_passed` **(ULTIMATE)** | boolean | **{dotted-circle}** No | Indicates that merges of merge requests should be blocked unless all status checks have passed. Defaults to false.<br/><br/>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369859) in GitLab 15.5 with feature flag `only_allow_merge_if_all_status_checks_passed` disabled by default. The feature flag was enabled by default in GitLab 15.9. | | `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | | `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge request by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | @@ -2244,9 +2245,12 @@ POST /projects/:id/restore ## Upload a file +> - Maximum attachment size enforcement [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57250) in GitLab 13.11 [with a flag](../administration/feature_flags.md) named `enforce_max_attachment_size_upload_api`. Disabled by default. +> - Maximum attachment size [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62542) in GitLab 13.11. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112450) in GitLab 15.10. Feature flag `enforce_max_attachment_size_upload_api` removed. + Uploads a file to the specified project to be used in an issue or merge request -description, or a comment. GitLab versions 14.0 and later -[enforce](#max-attachment-size-enforcement) this limit. +description, or a comment. ```plaintext POST /projects/:id/uploads @@ -2282,42 +2286,6 @@ The returned `url` is relative to the project path. The returned `full_path` is the absolute path to the file. In Markdown contexts, the link is expanded when the format in `markdown` is used. -### Max attachment size enforcement - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57250) in GitLab 13.11. - -GitLab 13.11 added enforcement of the [maximum attachment size limit](../user/admin_area/settings/account_and_limit_settings.md#max-attachment-size) behind the `enforce_max_attachment_size_upload_api` feature flag. GitLab 14.0 enables this by default. -To disable this enforcement: - -**In Omnibus installations:** - -1. Enter the Rails console: - - ```shell - sudo gitlab-rails console - ``` - -1. Disable the feature flag: - - ```ruby - Feature.disable(:enforce_max_attachment_size_upload_api) - ``` - -**In installations from source:** - -1. Enter the Rails console: - - ```shell - cd /home/git/gitlab - sudo -u git -H bundle exec rails console -e production - ``` - -1. Disable the feature flag: - - ```ruby - Feature.disable(:enforce_max_attachment_size_upload_api) - ``` - ## Upload a project avatar Uploads an avatar to the specified project. diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index 5f6e03fde4d..6c3bd63bbad 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -208,16 +208,16 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla | Attribute | Type | Required | Description | | -------------------------------------------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the branch or wildcard | -| `push_access_level` | integer | no | Access levels allowed to push (defaults: `40`, Maintainer role) | -| `merge_access_level` | integer | no | Access levels allowed to merge (defaults: `40`, Maintainer role) | -| `unprotect_access_level` | integer | no | Access levels allowed to unprotect (defaults: `40`, Maintainer role) | -| `allow_force_push` | boolean | no | Allow all users with push access to force push. (default: `false`) | -| `allowed_to_push` **(PREMIUM)** | array | no | Array of access levels allowed to push, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}` | -| `allowed_to_merge` **(PREMIUM)** | array | no | Array of access levels allowed to merge, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}` | -| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of access levels allowed to unprotect, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}` | -| `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false) | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. +| `name` | string | yes | The name of the branch or wildcard. +| `allow_force_push` | boolean | no | Allow all users with push access to force push. (default: `false`) +| `allowed_to_merge` **(PREMIUM)** | array | no | Array of access levels allowed to merge, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_push` **(PREMIUM)** | array | no | Array of access levels allowed to push, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. +| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of access levels allowed to unprotect, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. The access level `No access` is not available for this field. | +| `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). (defaults: false) +| `merge_access_level` | integer | no | Access levels allowed to merge. (defaults: `40`, Maintainer role) +| `push_access_level` | integer | no | Access levels allowed to push. (defaults: `40`, Maintainer role) +| `unprotect_access_level` | integer | no | Access levels allowed to unprotect. (defaults: `40`, Maintainer role) Example response: @@ -369,7 +369,7 @@ Example response: "name": "master", "push_access_levels": [ { - "id": 1, + "id": 1, "access_level": 30, "access_level_description": "Developers + Maintainers", "user_id": null, @@ -378,14 +378,14 @@ Example response: ], "merge_access_levels": [ { - "id": 1, + "id": 1, "access_level": 30, "access_level_description": "Developers + Maintainers", "user_id": null, "group_id": null }, { - "id": 2, + "id": 2, "access_level": 40, "access_level_description": "Maintainers", "user_id": null, @@ -394,7 +394,7 @@ Example response: ], "unprotect_access_levels": [ { - "id": 1, + "id": 1, "access_level": 40, "access_level_description": "Maintainers", "user_id": null, @@ -437,15 +437,15 @@ PATCH /projects/:id/protected_branches/:name curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_branches/feature-branch?allow_force_push=true&code_owner_approval_required=true" ``` -| Attribute | Type | Required | Description | +| Attribute | Type | Required | Description | | -------------------------------------------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the branch | -| `allow_force_push` | boolean | no | When enabled, members who can push to this branch can also force push. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. +| `name` | string | yes | The name of the branch. +| `allow_force_push` | boolean | no | When enabled, members who can push to this branch can also force push. +| `allowed_to_push` **(PREMIUM)** | array | no | Array of push access levels, with each described by a hash. +| `allowed_to_merge` **(PREMIUM)** | array | no | Array of merge access levels, with each described by a hash. +| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of unprotect access levels, with each described by a hash. The access level `No access` is not available for this field. | `code_owner_approval_required` **(PREMIUM)** | boolean | no | Prevent pushes to this branch if it matches an item in the [`CODEOWNERS` file](../user/project/code_owners.md). Defaults to `false`. | -| `allowed_to_push` **(PREMIUM)** | array | no | Array of push access levels, with each described by a hash. | -| `allowed_to_merge` **(PREMIUM)** | array | no | Array of merge access levels, with each described by a hash. | -| `allowed_to_unprotect` **(PREMIUM)** | array | no | Array of unprotect access levels, with each described by a hash. | Elements in the `allowed_to_push`, `allowed_to_merge` and `allowed_to_unprotect` arrays should be one of `user_id`, `group_id` or `access_level`, and take the form `{user_id: integer}`, `{group_id: integer}` or diff --git a/doc/api/protected_environments.md b/doc/api/protected_environments.md index 1ea3fc5adda..877add32fed 100644 --- a/doc/api/protected_environments.md +++ b/doc/api/protected_environments.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/api/protected_tags.md b/doc/api/protected_tags.md index 633ca441fad..d6928d7293c 100644 --- a/doc/api/protected_tags.md +++ b/doc/api/protected_tags.md @@ -8,17 +8,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w **Valid access levels** -Currently, these levels are recognized: +These access levels are recognized: -```plaintext -0 => No access -30 => Developer access -40 => Maintainer access -``` +- `0`: No access +- `30`: Developer role +- `40`: Maintainer role ## List protected tags -Gets a list of protected tags from a project. +Gets a list of [protected tags](../user/project/protected_tags.md) from a project. This function takes pagination parameters `page` and `per_page` to restrict the list of protected tags. ```plaintext @@ -27,10 +25,11 @@ GET /projects/:id/protected_tags | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_tags" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/protected_tags" ``` Example response: @@ -62,11 +61,12 @@ GET /projects/:id/protected_tags/:name | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the tag or wildcard | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the tag or wildcard. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_tags/release-1-0" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/protected_tags/release-1-0" ``` Example response: @@ -86,23 +86,35 @@ Example response: ## Protect repository tags -Protects a single repository tag or several project repository -tags using a wildcard protected tag. +Protects a single repository tag, or several project repository +tags, using a wildcard protected tag. ```plaintext POST /projects/:id/protected_tags ``` ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_tags?name=*-stable&create_access_level=30" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/protected_tags" -d '{ + "allowed_to_create" : [ + { + "user_id" : 1 + }, + { + "access_level" : 30 + } + ], + "create_access_level" : 30, + "name" : "*-stable" +}' ``` | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the tag or wildcard | -| `create_access_level` | string | no | Access levels allowed to create (defaults: `40`, Maintainer role) | -| `allowed_to_create` | array | no | Array of access levels allowed to create tags, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}` | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the tag or wildcard. | +| `allowed_to_create` | array | no | Array of access levels allowed to create tags, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}`. | +| `create_access_level` | string | no | Access levels allowed to create. Default: `40`, for Maintainer role. | Example response: @@ -128,10 +140,17 @@ DELETE /projects/:id/protected_tags/:name ``` ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/protected_tags/*-stable" +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/protected_tags/*-stable" ``` | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the tag | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](rest/index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | yes | The name of the tag. | + +## Related topics + +- [Tags API](tags.md) for all tags +- [Tags](../user/project/repository/tags/index.md) user documentation +- [Protected tags](../user/project/protected_tags.md) user documentation diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index f23f2b36ed0..0d8e335b620 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/releases/links.md b/doc/api/releases/links.md index 74f6b1c9efa..2294f161759 100644 --- a/doc/api/releases/links.md +++ b/doc/api/releases/links.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/resource_groups.md b/doc/api/resource_groups.md index cdba0c01f4f..552c2a3ecfb 100644 --- a/doc/api/resource_groups.md +++ b/doc/api/resource_groups.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/rest/deprecations.md b/doc/api/rest/deprecations.md new file mode 100644 index 00000000000..7bfc9e1f59e --- /dev/null +++ b/doc/api/rest/deprecations.md @@ -0,0 +1,86 @@ +--- +stage: Manage +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# REST API deprecations and removals + +The following API changes will occur when the v4 API is removed. + +The date of this change is unknown. +For details, see [issue 216456](https://gitlab.com/gitlab-org/gitlab/-/issues/216456) +and [issue 387485](https://gitlab.com/gitlab-org/gitlab/-/issues/387485). + +## `merged_by` API field + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/350534). + +The `merged_by` field in the [merge request API](../merge_requests.md#list-merge-requests) +has been deprecated in favor of the `merge_user` field which more correctly identifies who merged a merge request when +performing actions (merge when pipeline succeeds, add to merge train) other than a simple merge. + +API users are encouraged to use the new `merge_user` field instead. The `merged_by` field will be removed in v5 of the GitLab REST API. + +## `merge_status` API field + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/382032). + +The `merge_status` field in the [merge request API](../merge_requests.md#merge-status) +has been deprecated in favor of the `detailed_merge_status` field which more correctly identifies +all of the potential statuses that a merge request can be in. API users are encouraged to use the +new `detailed_merge_status` field instead. The `merge_status` field will be removed in v5 of the GitLab REST API. + +## Single merge request changes API endpoint + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/322117). + +The endpoint to get +[changes from a single merge request](../merge_requests.md#get-single-merge-request-changes) +has been deprecated in favor the +[list merge request diffs](../merge_requests.md#list-merge-request-diffs) endpoint. +API users are encouraged to switch to the new diffs endpoint instead. + +The `changes from a single merge request` endpoint will be removed in v5 of the GitLab REST API. + +## Approvers and Approver Group fields in Merge Request Approval API + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/353097). + +The endpoint to get the configuration of approvals for a project returns +empty arrays for `approvers` and `approval_groups`. +These fields were deprecated in favor of the endpoint to +[get project-level rules](../merge_request_approvals.md#get-project-level-rules) +for a merge request. API users are encouraged to switch to this endpoint instead. + +These fields will be removed from the `get configuration` endpoint in v5 of the GitLab REST API. + +## Runner usage of `active` replaced by `paused` + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351109). + +Occurrences of the `active` identifier in the GitLab Runner GraphQL API endpoints will be +renamed to `paused` in GitLab 16.0. + +- In v4 of the REST API, starting in GitLab 14.8, you can use the `paused` property in place of `active` +- In v5 of the REST API, this change will affect endpoints taking or returning `active` property, such as: + - `GET /runners` + - `GET /runners/all` + - `GET /runners/:id` / `PUT /runners/:id` + - `PUT --form "active=false" /runners/:runner_id` + - `GET /projects/:id/runners` / `POST /projects/:id/runners` + - `GET /groups/:id/runners` + +The 16.0 release of GitLab Runner will start using the `paused` property when registering runners. + +## Runner status will not return `paused` + +Breaking change. [Related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/344648). + +In a future v5 of the REST API, the endpoints for GitLab Runner will not return `paused` or `active`. + +A runner's status will only relate to runner contact status, such as: +`online`, `offline`, or `not_connected`. Status `paused` or `active` will no longer appear. + +When checking if a runner is `paused`, API users are advised to check the boolean attribute +`paused` to be `true` instead. When checking if a runner is `active`, check if `paused` is `false`. diff --git a/doc/api/rest/index.md b/doc/api/rest/index.md index 57d13f2a54f..91a70c3f2a9 100644 --- a/doc/api/rest/index.md +++ b/doc/api/rest/index.md @@ -311,17 +311,17 @@ The following table shows the possible return codes for API requests. | Return values | Description | |--------------------------|-------------| | `200 OK` | The `GET`, `PUT` or `DELETE` request was successful, and the resource itself is returned as JSON. | +| `201 Created` | The `POST` request was successful, and the resource is returned as JSON. | | `202 Accepted` | The `GET`, `PUT` or `DELETE` request was successful, and the resource is scheduled for processing. | | `204 No Content` | The server has successfully fulfilled the request, and there is no additional content to send in the response payload body. | -| `201 Created` | The `POST` request was successful, and the resource is returned as JSON. | | `304 Not Modified` | The resource hasn't been modified since the last request. | | `400 Bad Request` | A required attribute of the API request is missing. For example, the title of an issue is not given. | | `401 Unauthorized` | The user isn't authenticated. A valid [user token](#authentication) is necessary. | | `403 Forbidden` | The request isn't allowed. For example, the user isn't allowed to delete a project. | -| `404 Not Found` | A resource couldn't be accessed. For example, an ID for a resource couldn't be found. | +| `404 Not Found` | A resource couldn't be accessed. For example, an ID for a resource couldn't be found, or the user isn't authorized to access the resource. | | `405 Method Not Allowed` | The request isn't supported. | | `409 Conflict` | A conflicting resource already exists. For example, creating a project with a name that already exists. | -| `412` | The request was denied. This can happen if the `If-Unmodified-Since` header is provided when trying to delete a resource, which was modified in between. | +| `412 Precondition Failed`| The request was denied. This can happen if the `If-Unmodified-Since` header is provided when trying to delete a resource, which was modified in between. | | `422 Unprocessable` | The entity couldn't be processed. | | `429 Too Many Requests` | The user exceeded the [application rate limits](../../administration/instance_limits.md#rate-limits). | | `500 Server Error` | While handling the request, something went wrong on the server. | @@ -483,12 +483,13 @@ pagination headers. Keyset-based pagination is supported only for selected resources and ordering options: -| Resource | Options | Availability | -|:---------------------------------------------------------|:---------------------------------|:-------------------------------------------------------------------------------------------------------------| -| [Projects](../projects.md) | `order_by=id` only | Authenticated and unauthenticated users | -| [Groups](../groups.md) | `order_by=name`, `sort=asc` only | Unauthenticated users only | -| [Group audit events](../audit_events.md#group-audit-events) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333968) in GitLab 15.2) | -| [Jobs](../jobs.md) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362172) in GitLab 15.9) | +| Resource | Options | Availability | +|:----------------------------------------------------------------|:---------------------------------|:--------------------------------------------------------------------------------------------------------------| +| [Projects](../projects.md) | `order_by=id` only | Authenticated and unauthenticated users | +| [Groups](../groups.md) | `order_by=name`, `sort=asc` only | Unauthenticated users only | +| [Group audit events](../audit_events.md#group-audit-events) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333968) in GitLab 15.2) | +| [Project audit events](../audit_events.md#project-audit-events) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367528) in GitLab 15.10) | +| [Jobs](../jobs.md) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362172) in GitLab 15.9) | ### Pagination response headers diff --git a/doc/api/search.md b/doc/api/search.md index 3c3ad3f6ab9..f7fa7babe71 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -31,8 +31,8 @@ GET /search | `scope` | string | Yes | The scope to search in. Values include `projects`, `issues`, `merge_requests`, `milestones`, `snippet_titles`, `users`. [Additional scopes](#additional-scopes): `blobs`, `commits`, `notes`, `wiki_blobs`. | | `search` | string | Yes | The search query. | | `confidential` | boolean | No | Filter by confidentiality. Supports `issues` scope; other scopes are ignored. | -| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| -| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| +| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| +| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| | `state` | string | No | Filter by state. Supports `issues` and `merge_requests` scopes; other scopes are ignored. | ### Scope: `projects` @@ -407,6 +407,7 @@ Example response: "system": false, "noteable_id": 22, "noteable_type": "Issue", + "project_id": 6, "noteable_iid": 2 } ] @@ -449,8 +450,8 @@ GET /groups/:id/search | `scope` | string | Yes | The scope to search in. Values include `issues`, `merge_requests`, `milestones`, `projects`, `users`. [Additional scopes](#additional-scopes): `blobs`, `commits`, `notes`, `wiki_blobs`. | | `search` | string | Yes | The search query. | | `confidential` | boolean | No | Filter by confidentiality. Supports only `issues` scope; other scopes are ignored. | -| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| -| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| +| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| +| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| | `state` | string | No | Filter by state. Supports `issues` and `merge_requests` only; other scopes are ignored. | The response depends on the requested scope. @@ -796,6 +797,7 @@ Example response: "system": false, "noteable_id": 22, "noteable_type": "Issue", + "project_id": 6, "noteable_iid": 2 } ] @@ -839,8 +841,8 @@ GET /projects/:id/search | `search` | string | Yes | The search query. | | `confidential` | boolean | No | Filter by confidentiality. Supports `issues` scope; other scopes are ignored. | | `ref` | string | No | The name of a repository branch or tag to search on. The project's default branch is used by default. Applicable only for scopes `blobs`, `commits`, and `wiki_blobs`. | -| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| -| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for Advanced Search.| +| `order_by` | string | No | Allowed values are `created_at` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| +| `sort` | string | No | Allowed values are `asc` or `desc` only. If not set, results are sorted by `created_at` in descending order for basic search, or by the most relevant documents for advanced search.| | `state` | string | No | Filter by state. Supports the `issues` and `merge_requests` scopes; other scopes are ignored. | The response depends on the requested scope. @@ -1046,6 +1048,7 @@ Example response: "system": false, "noteable_id": 22, "noteable_type": "Issue", + "project_id": 6, "noteable_iid": 2 } ] diff --git a/doc/api/secure_files.md b/doc/api/secure_files.md index 11c718dde01..0fb12cf3ad2 100644 --- a/doc/api/secure_files.md +++ b/doc/api/secure_files.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference, api --- diff --git a/doc/api/settings.md b/doc/api/settings.md index 94ed2b2e3db..c3c48fba339 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -227,7 +227,8 @@ Example response: "can_create_group": false, "jira_connect_application_key": "123", "jira_connect_proxy_url": "http://gitlab.example.com", - "user_defaults_to_private_profile": true + "user_defaults_to_private_profile": true, + "projects_api_rate_limit_unauthenticated": 400 } ``` @@ -268,9 +269,9 @@ listed in the descriptions of the relevant settings. | `akismet_api_key` | string | required by: `akismet_enabled` | API key for Akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable Akismet spam protection. | | `allow_group_owners_to_manage_ldap` **(PREMIUM)** | boolean | no | Set to `true` to allow group owners to manage LDAP. | -| `allow_local_requests_from_hooks_and_services` | boolean | no | (Deprecated: Use `allow_local_requests_from_web_hooks_and_services` instead) Allow requests to the local network from hooks and services. | +| `allow_local_requests_from_hooks_and_services` | boolean | no | (Deprecated: Use `allow_local_requests_from_web_hooks_and_services` instead) Allow requests to the local network from webhooks and integrations. | | `allow_local_requests_from_system_hooks` | boolean | no | Allow requests to the local network from system hooks. | -| `allow_local_requests_from_web_hooks_and_services` | boolean | no | Allow requests to the local network from web hooks and services. | +| `allow_local_requests_from_web_hooks_and_services` | boolean | no | Allow requests to the local network from webhooks and integrations. | | `allow_runner_registration_token` | boolean | no | Allow using a registration token to create a runner. Defaults to `true`. | | `archive_builds_in_human_readable` | string | no | Set the duration for which the jobs are considered as old and expired. After that time passes, the jobs are archived and no longer able to be retried. Make it empty to never expire jobs. It has to be no less than 1 day, for example: <code>15 days</code>, <code>1 month</code>, <code>2 years</code>. | | `asset_proxy_enabled` | boolean | no | (**If enabled, requires:** `asset_proxy_url`) Enable proxying of assets. GitLab restart is required to apply changes. | @@ -282,7 +283,7 @@ listed in the descriptions of the relevant settings. | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It automatically builds, tests, and deploys applications based on a predefined CI/CD configuration. | | `automatic_purchased_storage_allocation` | boolean | no | Enabling this permits automatic allocation of purchased storage in a namespace. | -| `bulk_import_enabled` | boolean | no | Enable migrating GitLab groups by direct transfer. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. Requires [`bulk_import_projects` feature flag](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) to also migrate projects. Setting also [available](../user/admin_area/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer) in the Admin Area. | +| `bulk_import_enabled` | boolean | no | Enable migrating GitLab groups by direct transfer. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. Setting also [available](../user/admin_area/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer) in the Admin Area. | | `can_create_group` | boolean | no | Indicates whether users can create top-level groups. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367754) in GitLab 15.5. Defaults to `true`. | | `check_namespace_plan` **(PREMIUM)** | boolean | no | Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `commit_email_hostname` | string | no | Custom hostname (for private commit emails). | @@ -305,6 +306,7 @@ listed in the descriptions of the relevant settings. | `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_projects_limit` | integer | no | Project limit per user. Default is `100000`. | | `default_snippet_visibility` | string | no | What visibility level new snippets receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | +| `default_syntax_highlighting_theme` | integer | no | Default syntax highlighting theme for new users and users who are not signed in. See [IDs of available themes](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/themes.rb#L16). | `delayed_project_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed project deletion by default in new groups. Default is `false`. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), can only be enabled when `delayed_group_deletion` is true. | | `delayed_group_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed group deletion. Default is `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352959) in GitLab 15.0. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), disables and locks the group-level setting for delayed protect deletion when set to `false`. | | `deletion_adjourned_period` **(PREMIUM SELF)** | integer | no | The number of days to wait before deleting a project or group that is marked for deletion. Value must be between `1` and `90`. Defaults to `7`. [From GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/352960), a hook on `deletion_adjourned_period` sets the period to `1` on every update, and sets both `delayed_project_deletion` and `delayed_group_deletion` to `false` if the period is `0`. | @@ -315,7 +317,7 @@ listed in the descriptions of the relevant settings. | `disable_feed_token` | boolean | no | Disable display of RSS/Atom and calendar feed tokens. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231493) in GitLab 13.7. | | `disable_personal_access_token` **(PREMIUM SELF)** | boolean | no | Disable personal access tokens. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384201) in GitLab 15.7. | | `disabled_oauth_sign_in_sources` | array of strings | no | Disabled OAuth sign-in sources. | -| `dns_rebinding_protection_enabled` | boolean | no | Enforce DNS rebinding attack protection. | +| `dns_rebinding_protection_enabled` | boolean | no | Enforce DNS-rebinding attack protection. | | `domain_denylist_enabled` | boolean | no | (**If enabled, requires:** `domain_denylist`) Allows blocking sign-ups from emails from specific domains. | | `domain_denylist` | array of strings | no | Users with email addresses that match these domains **cannot** sign up. Wildcards allowed. Use separate lines for multiple entries. For example: `domain.com`, `*.domain.com`. | | `domain_allowlist` | array of strings | no | Force people to use only corporate emails for sign-up. Default is `null`, meaning there is no restriction. | @@ -368,6 +370,7 @@ listed in the descriptions of the relevant settings. | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | +| `gitlab_dedicated_instance` | boolean | no | Indicates whether the instance was provisioned for GitLab Dedicated. | | `grafana_enabled` | boolean | no | Enable Grafana. | | `grafana_url` | string | no | Grafana URL. | | `gravatar_enabled` | boolean | no | Enable Gravatar. | @@ -417,7 +420,7 @@ listed in the descriptions of the relevant settings. | `maven_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use repo.maven.apache.org as a default remote repository when the package is not found in the GitLab Package Registry for Maven. | | `npm_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use npmjs.org as a default remote repository when the package is not found in the GitLab Package Registry for npm. | | `pypi_package_requests_forwarding` **(PREMIUM)** | boolean | no | Use pypi.org as a default remote repository when the package is not found in the GitLab Package Registry for PyPI. | -| `outbound_local_requests_whitelist` | array of strings | no | Define a list of trusted domains or IP addresses to which local requests are allowed when local requests for hooks and services are disabled. +| `outbound_local_requests_whitelist` | array of strings | no | Define a list of trusted domains or IP addresses to which local requests are allowed when local requests for webhooks and integrations are disabled. | `pages_domain_verification_enabled` | boolean | no | Require users to prove ownership of custom domains. Domain verification is an essential security measure for public GitLab sites. Users are required to demonstrate they control a domain before it is enabled. | | `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | | `password_authentication_enabled_for_web` | boolean | no | Enable authentication for the web interface via a GitLab account password. Default is `true`. | @@ -434,6 +437,7 @@ listed in the descriptions of the relevant settings. | `plantuml_url` | string | required by: `plantuml_enabled` | The PlantUML instance URL for integration. | | `polling_interval_multiplier` | decimal | no | Interval multiplier used by endpoints that perform polling. Set to `0` to disable polling. | | `project_export_enabled` | boolean | no | Enable project export. | +| `projects_api_rate_limit_unauthenticated` | integer | no | [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112283) in GitLab 15.10. Max number of requests per 10 minutes per IP address for unauthenticated requests to the [list all projects API](projects.md#list-all-projects). Default: 400. To disable throttling set to 0.| | `prometheus_metrics_enabled` | boolean | no | Enable Prometheus metrics. | | `protected_ci_variables` | boolean | no | CI/CD variables are protected by default. | | `push_event_activities_limit` | integer | no | Number of changes (branches or tags) in a single push to determine whether individual push events or bulk push events are created. [Bulk push events are created](../user/admin_area/settings/push_event_activities_limit.md) if it surpasses that value. | diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index 1e1f226481c..bb1f3968cf3 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -11,7 +11,19 @@ This page describes the API for [suggesting changes](../user/project/merge_reque Every API call to suggestions must be authenticated. -## Applying a suggestion +## Create a suggestion + +To create a suggestion through the API, use the Discussions API to +[create a new thread in the merge request diff](discussions.md#create-new-merge-request-thread). +The format for suggestions is: + +````markdown +```suggestion:-3+0 +example text +``` +```` + +## Apply a suggestion Applies a suggested patch in a merge request. Users must have at least the Developer role to perform such action. @@ -43,7 +55,7 @@ Example response: } ``` -## Applying multiple suggestions +## Apply multiple suggestions ```plaintext PUT /suggestions/batch_apply diff --git a/doc/api/topics.md b/doc/api/topics.md index 8acf6bd645d..8f2aae9e070 100644 --- a/doc/api/topics.md +++ b/doc/api/topics.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/api/users.md b/doc/api/users.md index 4ecb3d48c0f..f877aa29b9c 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -41,7 +41,7 @@ GET /users You can also use `?search=` to search for users by name, username, or public email. For example, `/users?search=John`. When you search for a: -- Public email, you must use the full email address to get an exact match. +- Public email, you must use the full email address to get an exact match. A search might return a partial match. For example, if you search for the email `on@example.com`, the search can return both `on@example.com` and `jon@example.com`. - Name or username, you do not have to get an exact match because this is a fuzzy search. In addition, you can lookup users by username: diff --git a/doc/api/visual_review_discussions.md b/doc/api/visual_review_discussions.md index f966af82b10..561dc1a26a5 100644 --- a/doc/api/visual_review_discussions.md +++ b/doc/api/visual_review_discussions.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/architecture/blueprints/_template.md b/doc/architecture/blueprints/_template.md index f7dea60e9b7..64b79f996e0 100644 --- a/doc/architecture/blueprints/_template.md +++ b/doc/architecture/blueprints/_template.md @@ -50,6 +50,7 @@ Blueprint statuses you can use: - "accepted" - "ongoing" - "implemented" +- "postponed" - "rejected" --> @@ -125,6 +126,9 @@ but keep it simple! This should have enough detail that reviewers can understand exactly what you're proposing, but should not include things like API designs or implementation. The "Design Details" section below is for the real nitty-gritty. + +You might want to consider including the pros and cons of the proposed solution so that they can be +compared with the pros and cons of alternatives. --> ## Design and implementation details @@ -153,3 +157,12 @@ Diagrams authored in GitLab flavored markdown are preferred. In cases where that is not feasible, images should be placed under `images/` in the same directory as the `index.md` for the proposal. --> + +## Alternative Solutions + +<!-- +It might be a good idea to include a list of alternative solutions or paths considered, although it is not required. Include pros and cons for +each alternative solution/path. + +"Do nothing" and its pros and cons could be included in the list too. +--> diff --git a/doc/architecture/blueprints/cells/cells-feature-admin-area.md b/doc/architecture/blueprints/cells/cells-feature-admin-area.md new file mode 100644 index 00000000000..abcf4a5e263 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-admin-area.md @@ -0,0 +1,58 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Admin Area' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Admin Area + +In our Cells architecture proposal we plan to share all admin related tables in +GitLab. This allows simpler management of all Cells in one interface and reduces +the risk of settings diverging in different Cells. This introduces challenges +with admin pages that allow you to manage data that will be spread across all +Cells. + +## 1. Definition + +There are consequences for admin pages that contain data that spans "the whole +instance" as the Admin pages may be served by any Cell or possibly just 1 cell. +There are already many parts of the Admin interface that will have data that +spans many cells. For example lists of all Groups, Projects, Topics, Jobs, +Analytics, Applications and more. There are also administrative monitoring +capabilities in the Admin page that will span many cells such as the "Background +Jobs" and "Background Migrations" pages. + +## 2. Data flow + +## 3. Proposal + +We will need to decide how to handle these exceptions with a few possible +options: + +1. Move all these pages out into a dedicated per-cell Admin section. Probably + the URL will need to be routable to a single Cell like `/cells/<cell_id>/admin`, + then we can display this data per Cell. These pages will be distinct from + other Admin pages which control settings that are shared across all Cells. We + will also need to consider how this impacts self-managed customers and + whether, or not, this should be visible for single-cell instances of GitLab. +1. Build some aggregation interfaces for this data so that it can be fetched + from all Cells and presented in a single UI. This may be beneficial to an + administrator that needs to see and filter all data at a glance, especially + when they don't know which Cell the data is on. The downside, however, is + that building this kind of aggregation is very tricky when all the Cells are + designed to be totally independent, and it does also enforce more strict + requirements on compatibility between Cells. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md new file mode 100644 index 00000000000..5f0c8c2823d --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Agent for Kubernetes' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Agent for Kubernetes + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-backups.md b/doc/architecture/blueprints/cells/cells-feature-backups.md new file mode 100644 index 00000000000..80cac8764e6 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-backups.md @@ -0,0 +1,61 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Backups' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Backups + +Each cells will take its own backups, and consequently have its own isolated +backup / restore procedure. + +## 1. Definition + +GitLab Backup takes a backup of the PostgreSQL database used by the application, +and also Git repository data. + +## 2. Data flow + +Each cell has a number of application databases to back up (e.g. `main`, and `ci`). + +Additionally, there may be cluster-wide metadata tables (e.g. `users` table) +which is directly accesible via PostgreSQL. + +## 3. Proposal + +### 3.1. Cluster-wide metadata + +It is currently unknown how cluster-wide metadata tables will be accessible. We +may choose to have cluster-wide metadata tables backed up separately, or have +each cell back up its copy of cluster-wide metdata tables. + +### 3.2 Consistency + +#### 3.2.1 Take backups independently + +As each cell will communicate with each other via API, and there will be no joins +to the users table, it should be acceptable for each cell to take a backup +independently of each other. + +#### 3.2.2 Enforce snapshots + +We can require that each cell take a snapshot for the PostgreSQL databases at +around the same time to allow for a consistent-enough backup. + +## 4. Evaluation + +As the number of cells increases, it will likely not be feasible to take a +snapshot at the same time for all cells. Hence taking backups independently is +the better option. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-ci-runners.md b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md new file mode 100644 index 00000000000..619c78d19ec --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md @@ -0,0 +1,169 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: CI Runners' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: CI Runners + +GitLab in order to execute CI jobs [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), +very often managed by customer in their infrastructure. + +All CI jobs created as part of CI pipeline are run in a context of project +it poses a challenge how to manage GitLab Runners. + +## 1. Definition + +There are 3 different types of runners: + +- instance-wide: runners that are registered globally with specific tags (selection criteria) +- group runners: runners that execute jobs from a given top-level group or subprojects of that group +- project runners: runners that execute jobs from projects or many projects: some runners might + have projects assigned from projects in different top-level groups. + +This alongside with existing data structure where `ci_runners` is a table describing +all types of runners poses a challenge how the `ci_runners` should be managed in a Cells environment. + +## 2. Data flow + +GitLab Runners use a set of globally scoped endpoints to: + +- registration of a new runner via registration token `https://gitlab.com/api/v4/runners` + ([subject for removal](../runner_tokens/index.md)) (`registration token`) +- requests jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) +- upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) +- upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) +- download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) + +Currently three types of authentication tokens are used: + +- runner registration token ([subject for removal](../runner_tokens/index.md)) +- runner token representing an registered runner in a system with specific configuration (`tags`, `locked`, etc.) +- build token representing an ephemeral token giving a limited access to updating a specific + job, uploading artifacts, downloading dependent artifacts, downloading and uploading + container registry images + +Each of those endpoints do receive an authentication token via header (`JOB-TOKEN` for `/trace`) +or body parameter (`token` all other endpoints). + +Since the CI pipeline would be created in a context of a specific Cell it would be required +that pick of a build would have to be processed by that particular Cell. This requires +that build picking depending on a solution would have to be either: + +- routed to correct Cell for a first time +- be made to be two phase: request build from global pool, claim build on a specific Cell using a Cell specific URL + +## 3. Proposal + +This section describes various proposals. Reader should consider that those +proposals do describe solutions for different problems. Many or some aspects +of those proposals might be the solution to the stated problem. + +### 3.1. Authentication tokens + +Even though the paths for CI Runners are not routable they can be made routable with +those two possible solutions: + +- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with + a ticketing mechanism (based on `X-GitLab-Last-Update` header). Runner when first + starts sends a request to GitLab to which GitLab responds with either a build to pick + by runner. This value is completely controlled by GitLab. This allows GitLab + to use JWT or any other means to encode `cell` identifier that could be easily + decodable by Router. +- The majority of communication (in terms of volume) is using `build token` making it + the easiest target to change since GitLab is sole owner of the token that Runner later + uses for specific job. There were prior discussions about not storing `build token` + but rather using `JWT` token with defined scopes. Such token could encode the `cell` + to which router could easily route all requests. + +### 3.2. Request body + +- The most of used endpoints pass authentication token in request body. It might be desired + to use HTTP Headers as an easier way to access this information by Router without + a need to proxy requests. + +### 3.3. Instance-wide are Cell local + +We can pick a design where all runners are always registered and local to a given Cell: + +- Each Cell has it's own set of instance-wide runners that are updated at it's own pace +- The project runners can only be linked to projects from the same organization + creating strong isolation. +- In this model the `ci_runners` table is local to the Cell. +- In this model we would require the above endpoints to be scoped to a Cell in some way + or made routable. It might be via prefixing them, adding additional Cell parameter, + or providing much more robust way to decode runner token and match it to Cell. +- If routable token is used, we could move away from cryptographic random stored in + database to rather prefer to use JWT tokens that would encode +- The Admin Area showing registered Runners would have to be scoped to a Cell + +This model might be desired since it provides strong isolation guarantees. +This model does significantly increase maintenance overhead since each Cell is managed +separately. + +This model may require adjustments to runner tags feature so that projects have consistent runner experience across cells. + +### 3.4. Instance-wide are cluster-wide + +Contrary to proposal where all runners are Cell local, we can consider that runners +are global, or just instance-wide runners are global. + +However, this requires significant overhaul of system and to change the following aspects: + +- `ci_runners` table would likely have to be split decomposed into `ci_instance_runners`, ... +- all interfaces would have to be adopted to use correct table +- build queuing would have to be reworked to be two phase where each Cell would know of all pending + and running builds, but the actual claim of a build would happen against a Cell containing data +- likely `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables + increasing likelihood of creating hotspots in a system related to CI queueing + +This model makes it complex to implement from engineering side. Does make some data being shared +between Cells. Creates hotspots / scalability issues in a system (ex. during abuse) that +might impact experience of organizations on other Cells. + +### 3.5. GitLab CI Daemon + +Another potential solution to explore is to have a dedicated service responsible for builds queueing +owning it's database and working in a model of either sharded or celled service. There were prior +discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). + +If the service would be sharded: + +- depending on a model if runners are cluster-wide or cell-local this service would have to fetch + data from all Cells +- if the sharded service would be used we could adapt a model of either sharing database containing + `ci_pending_builds/ci_running_builds` with the service +- if the sharded service would be used we could consider a push model where each Cell pushes to CI/CD Daemon + builds that should be picked by Runner +- the sharded service would be aware which Cell is responsible for processing the given build and could + route processing requests to designated Cell + +If the service would be celled: + +- all expectations of routable endpoints are still valid + +In general usage of CI Daemon does not help significantly with the stated problem. However, this offers +a few upsides related to more efficient processing and decoupling model: push model and it opens a way +to offer stateful communication with GitLab Runners (ex. gRPC or Websockets). + +## 4. Evaluation + +Considering all solutions it appears that solution giving the most promise is: + +- use "instance-wide are Cell local" +- refine endpoints to have routable identities (either via specific paths, or better tokens) + +Other potential upsides is to get rid of `ci_builds.token` and rather use a `JWT token` +that can much better and easier encode wider set of scopes allowed by CI runner. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-container-registry.md b/doc/architecture/blueprints/cells/cells-feature-container-registry.md new file mode 100644 index 00000000000..b266de56205 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-container-registry.md @@ -0,0 +1,131 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Container Registry' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Container Registry + +GitLab Container Registry is a feature allowing to store Docker Container Images +in GitLab. You can read about GitLab integration [here](../../../user/packages/container_registry/index.md). + +## 1. Definition + +GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis +and Object Storage dependencies. Right now there's undergoing work to introduce +[Container Registry Metadata](../container_registry_metadata_database/index.md) +to optimize data storage and image retention policies of Container Registry. + +GitLab Container Registry is serving as a container for stored data, +but on it's own does not authenticate `docker login`. The `docker login` +is executed with user credentials (can be `personal access token`) +or CI build credentials (ephemeral `ci_builds.token`). + +Container Registry uses data deduplication. It means that the same blob +(image layer) that is shared between many projects is stored only once. +Each layer is hashed by `sha256`. + +The `docker login` does request JWT time-limited authentication token that +is signed by GitLab, but validated by Container Registry service. The JWT +token does store all authorized scopes (`container repository images`) +and operation types (`push` or `pull`). A single JWT authentication token +can be have many authorized scopes. This allows container registry and client +to mount existing blobs from another scopes. GitLab responds only with +authorized scopes. Then it is up to GitLab Container Registry to validate +if the given operation can be performed. + +The GitLab.com pages are always scoped to project. Each project can have many +container registry images attached. + +Currently in case of GitLab.com the actual registry service is served +via `https://registry.gitlab.com`. + +The main identifiable problems are: + +- the authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com +- the `https://registry.gitlab.com` that is run by external service and uses it's own data store +- the data deduplication, the Cells architecture with registry run in a Cell would reduce + efficiency of data storage + +## 2. Data flow + +### 2.1. Authorization request that is send by `docker login` + +```shell +curl \ + --user "username:password" \ + "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" +``` + +Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. + +```json +{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} +``` + +### 2.2. Docker client fetching tags + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list + +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 +``` + +### 2.3. Docker client fetching blobs and manifests + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 +``` + +## 3. Proposal + +### 3.1. Shard Container Registry separately to Cells architecture + +Due to it's architecture it extensive architecture and in general highly scalable +horizontal architecture it should be evaluated if the GitLab Container Registry +should be run not in Cell, but in a Cluster and be scaled independently. + +This might be easier, but would definitely not offer the same amount of data isolation. + +### 3.2. Run Container Registry within a Cell + +It appears that except `/jwt/auth` which would likely have to be processed by Router +(to decode `scope`) the container registry could be run as a local service of a Cell. + +The actual data at least in case of GitLab.com is not forwarded via registry, +but rather served directly from Object Storage / CDN. + +Its design encodes container repository image in a URL that is easily routable. +It appears that we could re-use the same stateless Router service in front of Container Registry +to serve manifests and blobs redirect. + +The only downside is increased complexity of managing standalone registry for each Cell, +but this might be desired approach. + +## 4. Evaluation + +There do not seem any theoretical problems with running GitLab Container Registry in a Cell. +Service seems that can be easily made routable to work well. + +The practical complexities are around managing complex service from infrastructure side. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md new file mode 100644 index 00000000000..cc6feb1c08a --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md @@ -0,0 +1,120 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Contributions: Forks' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Contributions: Forks + +[Forking workflow](../../../user/project/repository/forking_workflow.md) allows users +to copy existing project sources into their own namespace of choice (personal or group). + +## 1. Definition + +[Forking workflow](../../../user/project/repository/forking_workflow.md) is common workflow +with various usage patterns: + +- allows users to contribute back to upstream project +- persist repositories into their personal namespace +- copy to make changes and release as modified project + +Forks allow users not having write access to parent project to make changes. The forking workflow +is especially important for the Open Source community which is able to contribute back +to public projects. However, it is equally important in some companies which prefer the strong split +of responsibilites and tighter access control. The access to project is restricted +to designated list of developers. + +Forks enable: + +- tigther control of who can modify the upstream project +- split of the responsibilites: parent project might use CI configuration connecting to production systems +- run CI pipelines in context of fork in much more restrictive environment +- consider all forks to be unveted which reduces risks of leaking secrets, or any other information + tied with the project + +The forking model is problematic in Cells architecture for following reasons: + +- Forks are clones of existing repositories, forks could be created across different organizations, Cells and Gitaly shards. +- User can create merge request and contribute back to upstream project, this upstream project might in a different organization and Cell. +- The merge request CI pipeline is to executed in a context of source project, but presented in a context of target project. + +## 2. Data flow + +## 3. Proposals + +### 3.1. Intra-Cluster forks + +This proposal makes us to implement forks as a intra-ClusterCell forks where communication is done via API +between all trusted Cells of a cluster: + +- Forks when created, they are created always in context of user choice of group. +- Forks are isolated from Organization. +- Organization or group owner could disable forking across organizations or forking in general. +- When a Merge Request is created it is created in context of target project, referencing + external project on another Cell. +- To target project the merge reference is transfered that is used for presenting information + in context of target project. +- CI pipeline is fetched in context of source project as it-is today, the result is fetched into + Merge Request of target project. +- The Cell holding target project internally uses GraphQL to fetch status of source project + and include in context of the information for merge request. + +Upsides: + +- All existing forks continue to work as-is, as they are treated as intra-Cluster forks. + +Downsides: + +- The purpose of Organizations is to provide strong isolation between organizations + allowing to fork across does break security boundaries. +- However, this is no different to ability of users today to clone repository to local computer + and push it to any repository of choice. +- Access control of source project can be lower than those of target project. System today + requires that in order to contribute back the access level needs to be the same for fork and upstream. + +### 3.2. Forks are created in a personal namespace of the current organization + +Instead of creating projects across organizations, the forks are created in a user personal namespace +tied with the organization. Example: + +- Each user that is part of organization receives their personal namespace. For example for `GitLab Inc.` + it could be `gitlab.com/organization/gitlab-inc/@ayufan`. +- The user has to fork into it's own personal namespace of the organization. +- The user has that many personal namespaces as many organizations it belongs to. +- The personal namespace behaves similar to currently offered personal namespace. +- The user can manage and create projects within a personal namespace. +- The organization can prevent or disable usage of personal namespaces disallowing forks. +- All current forks are migrated into personal namespace of user in Organization. +- All forks are part of to the organization. +- The forks are not federated features. +- The personal namespace and forked project do not share configuration with parent project. + +### 3.3. Forks are created as internal projects under current project + +Instead of creating projects across organizations, the forks are attachments to existing projects. +Each user forking a project receives their unique project. Example: + +- For project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. +- Forks are created in a context of current organization, they do not cross organization boundaries + and are managed by the organization. +- Tied to the user (or any other user-provided name of the fork). +- The forks are not federated features. + +Downsides: + +- Does not answer how to handle and migrate all exisiting forks. +- Might share current group / project settings - breaking some security boundaries. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-dashboard.md b/doc/architecture/blueprints/cells/cells-feature-dashboard.md new file mode 100644 index 00000000000..37ed13ad9f9 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-dashboard.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Dashboard' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Dashboard + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-data-migration.md b/doc/architecture/blueprints/cells/cells-feature-data-migration.md new file mode 100644 index 00000000000..e24cd0323f4 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-data-migration.md @@ -0,0 +1,130 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Data migration' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Data migration + +It is essential for Cells architecture to provide a way to migrate data out of big Cells +into smaller ones. This describes various approaches to provide this type of split. + +We also need to handle for cases where data is already violating the expected +isolation constraints of Cells (ie. references cannot span multiple +organizations). We know that existing features like linked issues allowed users +to link issues across any projects regardless of their hierarchy. There are many +similar features. All of this data will need to be migrated in some way before +it can be split across different cells. This may mean some data needs to be +deleted, or the feature changed and modelled slightly differently before we can +properly split or migrate the organizations between cells. + +Having schema deviations across different Cells, which is a necessary +consequence of different databases, will also impact our ability to migrate +data between cells. Different schemas impact our ability to reliably replicate +data across cells and especially impact our ability to validate that the data is +correctly replicated. It might force us to only be able to move data between +cells when the schemas are all in sync (slowing down deployments and the +rebalancing process) or possibly only migrate from newer to older schemas which +would be complex. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +### 3.1. Split large Cells + +A single Cell can only be divided into many Cells. This is based on principle +that it is easier to create exact clone of an existing Cell in many replicas +out of which some will be made authoritative once migrated. Keeping those +replicas up-to date with Cell 0 is also much easier due to pre-existing +replication solutions that can replicate the whole systems: Geo, PostgreSQL +physical replication, etc. + +1. All data of an organization needs to not be divided across many Cells. +1. Split should be doable online. +1. New Cells cannot contain pre-existing data. +1. N Cells contain exact replica of Cell 0. +1. The data of Cell 0 is live replicated to as many Cells it needs to be split. +1. Once consensus is achieved between Cell 0 and N-Cells the organizations to be migrated away + are marked as read-only cluster-wide. +1. The `routes` is updated on for all organizations to be split to indicate an authoritative + Cell holding the most recent data, like `gitlab-org` on `cell-100`. +1. The data for `gitlab-org` on Cell 0, and on other non-authoritative N-Cells are dormant + and will be removed in the future. +1. All accesses to `gitlab-org` on a given Cell are validated about `cell_id` of `routes` + to ensure that given Cell is authoritative to handle the data. + +#### More challenges of this proposal + +1. There is no streaming replication capability for Elasticsearch, but you could + snapshot the whole Elasticsearch index and recreate, but this takes hours. + It could be handled by pausing Elasticsearch indexing on the initial cell during + the migration as indexing downtime is not a big issue, but this still needs + to be coordinated with the migration process +1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other + new data stores snapshots in an online system would likely lead to gaps + without a long downtime. You need to choose a sync point and at the sync + point you need to stop writes to perform the migration. The more data stores + there are to migrate at the same time the longer the write downtime for the + failover. We would also need to find a reliable place in the application to + actually block updates to all these systems with a high degree of + confidence. In the past we've only been confident by shutting down all rails + services because any rails process could write directly to any of these at + any time due to async workloads or other surprising code paths. +1. How to efficiently delete all the orphaned data. Locating all `ci_builds` + associated with half the organizations would be very expensive if we have to + do joins. We haven't yet determined if we'd want to store an `organization_id` + column on every table, but this is the kind of thing it would be helpful for. + +### 3.2. Migrate organization from an existing Cell + +This is different to split, as we intend to perform logical and selective replication +of data belonging to a single organization. + +Today this type of selective replication is only implemented by Gitaly where we can migrate +Git repository from a single Gitaly node to another with minimal downtime. + +In this model we would require identifying all resources belonging to a given organization: +database rows, object storage files, Git repositories, etc. and selectively copy them over +to another (likely) existing Cell importing data into it. Ideally ensuring that we can +perform logical replication live of all changed data, but change similarly to split +which Cell is authoritative for this organization. + +1. It is hard to identify all resources belonging to organization. +1. It requires either downtime for organization or a robust system to identify + live changes made. +1. It likely will require a full database structure analysis (more robust than project import/export) + to perform selective PostgreSQL logical replication. + +#### More challenges of this proposal + +1. Logical replication is still not performant enough to keep up with our + scale. Even if we could use logical replication we still don't have an + efficient way to filter data related to a single organization without + joining all the way to the `organizations` table which will slow down + logical replication dramatically. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-database-sequences.md b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md new file mode 100644 index 00000000000..7966ac4bfdf --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md @@ -0,0 +1,94 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Database Sequences' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Database Sequences + +GitLab today ensures that every database row create has unique ID, allowing +to access Merge Request, CI Job or Project by a known global ID. + +Cells will use many distinct and not connected databases, each of them having +a separate IDs for most of entities. + +It might be desirable to retain globally unique IDs for all database rows +to allow migrating resources between Cells in the future. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +This are some preliminary ideas how we can retain unique IDs across the system. + +### 3.1. UUID + +Instead of using incremental sequences use UUID (128 bit) that is stored in database. + +- This might break existing IDs and requires adding UUID column for all existing tables. +- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. + +### 3.2. Use Cell index encoded in ID + +Since significant number of tables already use 64 bit ID numbers we could use MSB to encode +Cell ID effectively enabling + +- This might limit amount of Cells that can be enabled in system, as we might decide to only + allocate 1024 possible Cell numbers. +- This might make IDs to be migratable between Cells, since even if entity from Cell 1 is migrated to Cell 100 + this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode Cell number and we would need + lookup table. +- This requires updating all IDs to 32 bits. + +### 3.3. Allocate sequence ranges from central place + +Each Cell might receive its own range of the sequences as they are consumed from a centrally managed place. +Once Cell consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. +Ranges would be tracked to provide a faster lookup table if a random access pattern is required. + +- This might make IDs to be migratable between Cells, since even if entity from Cell 1 is migrated to Cell 100 + this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode Cell number and we would need + much more robust lookup table as we could be breaking previously assigned sequence ranges. +- This does not require updating all IDs to 64 bits. +- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server +- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. + +### 3.4. Define only some tables to require unique IDs + +Maybe this is acceptable only for some tables to have a globally unique IDs. It could be projects, groups +and other top-level entities. All other tables like `merge_requests` would only offer Cell-local ID, +but when referenced outside it would rather use IID (an ID that is monotonic in context of a given resource, like project). + +- This makes the ID 10000 for `merge_requests` be present on all Cells, which might be sometimes confusing + as for uniqueness of the resource. +- This might make random access by ID (if ever needed) be impossible without using composite key, like: `project_id+merge_request_id`. +- This would require us to implement a transformation/generation of new ID if we need to migrate records to another cell. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. +- If IDs need to change when moving between cells this means that any links to records by ID would no longer work even if those links included the `project_id`. +- If we plan to allow these ids to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-git-access.md b/doc/architecture/blueprints/cells/cells-feature-git-access.md new file mode 100644 index 00000000000..3582c99c7dd --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-git-access.md @@ -0,0 +1,163 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Git Access' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Git Access + +This document describes impact of Cells architecture on all Git access (over HTTPS and SSH) +patterns providing explanation of how potentially those features should be changed +to work well with Cells. + +## 1. Definition + +Git access is done through out the application. It can be an operation performed by the system +(read Git repository) or by user (create a new file via Web IDE, `git clone` or `git push` via command line). + +The Cells architecture defines that all Git repositories will be local to the Cell, +so no repository could be shared with another Cell. + +The Cells architecture will require that any Git operation done can only be handled by a Cell holding +the data. It means that any operation either via Web interface, API, or GraphQL needs to be routed +to the correct Cell. It means that any `git clone` or `git push` operation can only be performed +in a context of a Cell. + +## 2. Data flow + +The are various operations performed today by the GitLab on a Git repository. This describes +the data flow how they behave today to better represent the impact. + +It appears that Git access does require changes only to a few endpoints that are scoped to project. +There appear to be different types of repositories: + +- Project: assigned to Group +- Wiki: additional repository assigned to Project +- Design: similar to Wiki, additional repository assigned to Project +- Snippet: creates a virtual project to hold repository, likely tied to the User + +### 2.1. Git clone over HTTPS + +Execution of: `git clone` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC InfoRefsUploadPack + Gitaly ->> User: Response + User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack + Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.2. Git clone over SSH + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.3. Git push over HTTPS + +Execution of: `git push` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC PostReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.4. Git push over SSHD + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC ReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.5. Create commit via Web + +Execution of `Add CHANGELOG` to repository: + +```mermaid +sequenceDiagram + Web ->> Puma: POST /gitlab-org/gitlab/-/create/main + Puma ->> Gitaly: RPC TreeEntry + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> Puma: Response + Puma ->> Web: See CHANGELOG +``` + +## 3. Proposal + +The Cells stateless router proposal requires that any ambiguous path (that is not routable) +will be made to be routable. It means that at least the following paths will have to be updated +do introduce a routable entity (project, group, or organization). + +Change: + +- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` +- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` +- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` +- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` + +Where: + +- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) +- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) + +## 4. Evaluation + +Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. + +The one major complication is supporting snippets, but this likely falls in the same category as for the approach +to support user's personal namespaces. + +## 4.1. Pros + +1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. + +## 4.2. Cons + +1. The sharing of repositories objects is limited to the given Cell and Gitaly node. +1. The across-Cells forks are likely impossible to be supported (discover: how this work today across different Gitaly node). diff --git a/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md new file mode 100644 index 00000000000..9d501918a21 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: GitLab Pages' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GitLab Pages + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-global-search.md b/doc/architecture/blueprints/cells/cells-feature-global-search.md new file mode 100644 index 00000000000..fdd56d7258d --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-global-search.md @@ -0,0 +1,47 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Global search' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Global search + +When we introduce multiple Cells we intend to isolate all services related to +those Cells. This will include Elasticsearch which means our current global +search functionality will not work. It may be possible to implement aggregated +search across all cells, but it is unlikely to be performant to do fan-out +searches across all cells especially once you start to do pagination which +requires setting the correct offset and page number for each search. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +Likely first versions of Cells will simply not support global searches and then +we may later consider if building global searches to support popular use cases +is worthwhile. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-graphql.md b/doc/architecture/blueprints/cells/cells-feature-graphql.md new file mode 100644 index 00000000000..8430d23c7f4 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-graphql.md @@ -0,0 +1,94 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: GraphQL' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GraphQL + +GitLab extensively uses GraphQL to perform efficient data query operations. +GraphQL due to it's nature is not directly routable. The way how GitLab uses +it calls the `/api/graphql` endpoint, and only query or mutation of body request +might define where the data can be accessed. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +There are at least two main ways to implement GraphQL in Cells architecture. + +### 3.1. GraphQL routable by endpoint + +Change `/api/graphql` to `/api/organization/<organization>/graphql`. + +- This breaks all existing usages of `/api/graphql` endpoint + since the API URI is changed. + +### 3.2. GraphQL routable by body + +As part of router parse GraphQL body to find a routable entity, like `project`. + +- This still makes the GraphQL query be executed only in context of a given Cell + and not allowing the data to be merged. + +```json +# Good example +{ + project(fullPath:"gitlab-org/gitlab") { + id + description + } +} + +# Bad example, since Merge Request is not routable +{ + mergeRequest(id: 1111) { + iid + description + } +} +``` + +### 3.3. Merging GraphQL Proxy + +Implement as part of router GraphQL Proxy which can parse body +and merge results from many Cells. + +- This might make pagination hard to achieve, or we might assume that + we execute many queries of which results are merged across all Cells. + +```json +{ + project(fullPath:"gitlab-org/gitlab"){ + id, description + } + group(fullPath:"gitlab-com") { + id, description + } +} +``` + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-organizations.md b/doc/architecture/blueprints/cells/cells-feature-organizations.md new file mode 100644 index 00000000000..9c2e33719ab --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-organizations.md @@ -0,0 +1,58 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Organizations' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Organizations + +One of the major designs of Cells architecture is strong isolation between Groups. +Organizations as described by this blueprint provides a way to have plausible UX +for joining together many Groups that are isolated from the rest of systems. + +## 1. Definition + +Cells do require that all groups and projects of a single organization can +only be stored on a single Cell since a Cell can only access data that it holds locally +and has very limited capabilities to read information from other Cells. + +Cells with Organizations do require strong isolation between organizations. + +It will have significant implications on various user-facing features, +like Todos, dropdowns allowing to select projects, references to other issues +or projects, or any other social functions present at GitLab. Today those functions +were able to reference anything in the whole system. With the introduction of +organizations such will be forbidden. + +This problem definition aims to answer effort and implications required to add +strong isolation between organizations to the system. Including features affected +and their data processing flow. The purpose is to ensure that our solution when +implemented consistently avoids data leakage between organizations residing on +a single Cell. + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md new file mode 100644 index 00000000000..467683ffadd --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Personal Namespaces' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Personal Namespaces + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md new file mode 100644 index 00000000000..c961857e4fa --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md @@ -0,0 +1,46 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Router Endpoints Classification' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Router Endpoints Classification + +Classification of all endpoints is essential to properly route request +hitting load balancer of a GitLab installation to a Cell that can serve it. + +Each Cell should be able to decode each request and classify for which Cell +it belongs to. + +GitLab currently implements hundreds of endpoints. This document tries +to describe various techniques that can be implemented to allow the Rails +to provide this information efficiently. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-schema-changes.md b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md new file mode 100644 index 00000000000..c43fc6f302e --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md @@ -0,0 +1,55 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Schema changes' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Schema changes + +When we introduce multiple Cells that own their own databases this will +complicate the process of making schema changes to Postgres and Elasticsearch. +Today we already need to be careful to make changes comply with our zero +downtime deployments. For example, +[when removing a column we need to make changes over 3 separate deployments](../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). +We have tooling like `post_migrate` that helps with these kinds of changes to +reduce the number of merge requests needed, but these will be complicated when +we are dealing with deploying multiple rails applications that will be at +different versions at any one time. This problem will be particularly tricky to +solve for shared databases like our plan to share the `users` related tables +among all Cells. + +A key benefit of Cells may be that it allows us to run different +customers on different versions of GitLab. We may choose to update our own cell +before all our customers giving us even more flexibility than our current +canary architecture. But doing this means that schema changes need to have even +more versions of backward compatibility support which could slow down +development as we need extra steps to make schema changes. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-secrets.md b/doc/architecture/blueprints/cells/cells-feature-secrets.md new file mode 100644 index 00000000000..44b65c27683 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-secrets.md @@ -0,0 +1,48 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Secrets' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Secrets + +Where possible, each cell should have its own distinct set of secrets. +However, there will be some secrets that will be required to be the same for all +cells in the cluster + +## 1. Definition + +GitLab has a lot of +[secrets](https://docs.gitlab.com/charts/installation/secrets.html) that needs +to be configured. + +Some secrets are for inter-component communication, e.g. `GitLab Shell secret`, +and used only within a cell. + +Some secrets are used for features, e.g. `ci_jwt_signing_key`. + +## 2. Data flow + +## 3. Proposal + +1. Secrets used for features will need to be consistent across all cells, so that the UX is consistent. + 1. This is especially true for the `db_key_base` secret which is used for + encrypting data at rest in the database - so that projects that are + transferred to another cell will continue to work. We do not want to have + to re-encrypt such rows when we move projects/groups between cells. +1. Secrets which are used for intra-cell communication only should be uniquely generated + per-cell. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-snippets.md b/doc/architecture/blueprints/cells/cells-feature-snippets.md new file mode 100644 index 00000000000..a40b44c7b74 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-snippets.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Snippets' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Snippets + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-template.md b/doc/architecture/blueprints/cells/cells-feature-template.md new file mode 100644 index 00000000000..6951421023d --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-template.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Problem A' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: A + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/cells-feature-uploads.md b/doc/architecture/blueprints/cells/cells-feature-uploads.md new file mode 100644 index 00000000000..9d3500b6c36 --- /dev/null +++ b/doc/architecture/blueprints/cells/cells-feature-uploads.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells: Uploads' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Uploads + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/images/iteration0-organizations-introduction.png b/doc/architecture/blueprints/cells/images/iteration0-organizations-introduction.png Binary files differindex 5725b0fa71f..5725b0fa71f 100644 --- a/doc/architecture/blueprints/pods/images/iteration0-organizations-introduction.png +++ b/doc/architecture/blueprints/cells/images/iteration0-organizations-introduction.png diff --git a/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png b/doc/architecture/blueprints/cells/images/pods-and-fulfillment.png Binary files differindex fea32d1800e..fea32d1800e 100644 --- a/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png +++ b/doc/architecture/blueprints/cells/images/pods-and-fulfillment.png diff --git a/doc/architecture/blueprints/cells/images/term-cell.png b/doc/architecture/blueprints/cells/images/term-cell.png Binary files differnew file mode 100644 index 00000000000..40e4a3e29c3 --- /dev/null +++ b/doc/architecture/blueprints/cells/images/term-cell.png diff --git a/doc/architecture/blueprints/cells/images/term-cluster.png b/doc/architecture/blueprints/cells/images/term-cluster.png Binary files differnew file mode 100644 index 00000000000..d54a3569a08 --- /dev/null +++ b/doc/architecture/blueprints/cells/images/term-cluster.png diff --git a/doc/architecture/blueprints/cells/images/term-organization.png b/doc/architecture/blueprints/cells/images/term-organization.png Binary files differnew file mode 100644 index 00000000000..172ff7e415a --- /dev/null +++ b/doc/architecture/blueprints/cells/images/term-organization.png diff --git a/doc/architecture/blueprints/pods/images/term-top-level-namespace.png b/doc/architecture/blueprints/cells/images/term-top-level-namespace.png Binary files differindex c1cd317d878..c1cd317d878 100644 --- a/doc/architecture/blueprints/pods/images/term-top-level-namespace.png +++ b/doc/architecture/blueprints/cells/images/term-top-level-namespace.png diff --git a/doc/architecture/blueprints/cells/index.md b/doc/architecture/blueprints/cells/index.md new file mode 100644 index 00000000000..54244265b30 --- /dev/null +++ b/doc/architecture/blueprints/cells/index.md @@ -0,0 +1,358 @@ +--- +status: accepted +creation-date: "2022-09-07" +authors: [ "@ayufan", "@fzimmer", "@DylanGriffith" ] +coach: "@ayufan" +approvers: [ "@fzimmer" ] +owning-stage: "~devops::enablement" +participating-stages: [] +--- + +# Cells + +This document is a work-in-progress and represents a very early state of the Cells design. Significant aspects are not documented, though we expect to add them in the future. + +## Summary + +Cells is a new architecture for our Software as a Service platform that is horizontally-scalable, resilient, and provides a more consistent user experience. It may also provide additional features in the future, such as data residency control (regions) and federated features. + +## Terminology + +We use the following terms to describe components and properties of the Cells architecture. + +### Cell + +> Pod was renamed to Cell in <https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/121163> + +A Cell is a set of infrastructure components that contains multiple top-level namespaces that belong to different organizations. The components include both datastores (PostgreSQL, Redis etc.) and stateless services (web etc.). The infrastructure components provided within a Cell are shared among organizations and their top-level namespaces but not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other. + + + +#### Cell properties + +- Each cell is independent from the others +- Infrastructure components are shared by organizations and their top-level namespaces within a Cell +- More Cells can be provisioned to provide horizontal scalability +- A failing Cell does not lead to failure of other Cells +- Noisy neighbor effects are limited to within a Cell +- Cells are not visible to organizations; it is an implementation detail +- Cells may be located in different geographical regions (for example, EU, US, JP, UK) + +Discouraged synonyms: GitLab instance, cluster, shard + +### Cluster + +A cluster is a collection of Cells. + + + +#### Cluster properties + +- A cluster holds cluster-wide metadata, for example Users, Routes, Settings. + +Discouraged synonyms: whale + +### Organizations + +GitLab references [Organizations in the initial set up](../../../topics/set_up_organization.md) and users can add a (free text) organization to their profile. There is no Organization entity established in the GitLab codebase. + +As part of delivering Cells, we propose the introduction of an `organization` entity. Organizations would represent billable entities or customers. + +Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). + +Organizations work under the following assumptions: + +1. Users care about what happens within their organizations. +1. Features need to work within an organization. +1. Only few features need to work across organizations. +1. Users understand that the majority of pages they view are only scoped to a single organization at a time. +1. Organizations are located on a single cell. + + + +#### Organization properties + +- Top-level namespaces belong to organizations +- Users can be members of different organizations +- Organizations are isolated from each other by default meaning that cross-namespace features will only work for namespaces that exist within a single organization +- User namespaces must not belong to an organization + +Discouraged synonyms: Billable entities, customers + +### Top-Level namespace + +A top-level namespace is the logical object container in the code that represents all groups, subgroups and projects that belong to an organization. + +A top-level namespace is the root of nested collection namespaces and projects. The namespace and its related entities form a tree-like hierarchy: Namespaces are the nodes of the tree, projects are the leaves. + +Example: + +`https://gitlab.com/gitlab-org/gitlab/`: + +- `gitlab-org` is a `top-level namespace`; the root for all groups and projects of an organization +- `gitlab` is a `project`; a project of the organization. + +Top-level namespaces may [be replaced by organizations](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals). This proposal only uses the term top-level namespaces as the organization definition is ongoing. + +Discouraged synonyms: Root-level namespace + + + +#### Top-level namespace properties + +- Top-level namespaces belonging to an organization are located on the same Cell +- Top-level namespaces can interact with other top-level namespaces that belong to the same organization + +### Users + +Users are available globally and not restricted to a single Cell. Users can be members of many different organizations with varying permissions. Inside organizations, users can create multiple top-level namespaces. User activity is not limited to a single organization but their contributions (for example TODOs) are only aggregated within an organization. This avoids the need for aggregating across cells. + +#### User properties + +- Users are shared globally across all Cells +- Users can create multiple top-level namespaces +- Users can be a member of multiple top-level namespaces +- Users can be a member of multiple organizations +- Users can administer organizations +- User activity is aggregated in an organization +- Every user has one personal namespace + +## Goals + +### Scalability + +The main goal of this new shared-infrastructure architecture is to provide additional scalability for our SaaS Platform. GitLab.com is largely monolithic and we have estimated (internal) that the current architecture has scalability limitations, even when database partitioning and decomposition are taken into account. + +Cells provide a horizontally scalable solution because additional Cells can be created based on demand. Cells can be provisioned and tuned as needed for optimal scalability. + +### Increased availability + +A major challenge for shared-infrastructure architectures is a lack of isolation between top-level namespaces. This can lead to noisy neighbor effects. A organization's behavior inside a top-level namespace can impact all other organizations. This is highly undesirable. Cells provide isolation at the cell level. A group of organizations is fully isolated from other organizations located on a different Cell. This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of shared infrastructure. + +Additionally, Cells provide a way to implement disaster recovery capabilities. Entire Cells may be replicated to read-only standbys with automatic failover capabilities. + +### A consistent experience + +Organizations should have the same user experience on our SaaS platform as they do on a self-managed GitLab instance. + +### Regions + +GitLab.com is only hosted within the United States of America. Organizations located in other regions have voiced demand for local SaaS offerings. Cells provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Cells may be deployed within different geographies. Depending on which of the organization's data is located outside a Cell, this may solve data residency and compliance problems. + +## Market segment + +Cells would provide a solution for organizations in the small to medium business (up to 100 users) and the mid-market segment (up to 2000 users). +(See [segmentation definitions](https://about.gitlab.com/handbook/sales/field-operations/gtm-resources/#segmentation).) +Larger organizations may benefit substantially from [GitLab Dedicated](../../../subscriptions/gitlab_dedicated/index.md). + +At this moment, GitLab.com has "social-network"-like capabilities that may not fit well into a more isolated organization model. Removing those features, however, possesses some challenges: + +1. How will existing `gitlab-org` contributors contribute to the namespace?? +1. How do we move existing top-level namespaces into the new model (effectively breaking their social features)? + +We should evaluate if the SMB and mid market segment is interested in these features, or if not having them is acceptable in most cases. + +### Self-managed + +For reasons of consistency, it is expected that self-managed instances will +adopt the cells architecture as well. To expand, self-managed instances can +continue with just a single Cell while supporting the option of adding additional +Cells. Organizations, and possible User decomposition will also be adopted for +self-managed instances. + +## High-level architecture problems to solve + +A number of technical issues need to be resolved to implement Cells (in no particular order). This section will be expanded. + +1. How are users of an organization routed to the correct Cell? +1. How do users authenticate? +1. How are Cells rebalanced? +1. How are Cells provisioned? +1. How can Cells implement disaster recovery capabilities? + +## Cross-section impact + +Cells is a fundamental architecture change that impacts other sections and stages. This section summarizes and links to other groups that may be impacted and highlights potential conflicts that need to be resolved. The Tenant Scale group is not responsible for achieving the goals of other groups but we want to ensure that dependencies are resolved. + +### Summary + +Based on discussions with other groups the net impact of introducing Cells and a new entity called organizations is mostly neutral. It may slow down development in some areas. We did not discover major blockers for other teams. + +1. We need to resolve naming conflicts (proposal is TBD) +1. Cells requires introducing Organizations. Organizations are a new entity **above** top-level groups. Because this is a new entity, it may impact the ability to consolidate settings for Group::Organization and influence their decision on [how to approach introducing a an organization](https://gitlab.com/gitlab-org/gitlab/-/issues/376285#approach-2-organization-is-built-on-top-of-top-level-groups) +1. Organizations may make it slightly easier for Fulfillment to realize their billing plans. + +### Impact on Group::Organization + +We synced with the Organization PM and Designer ([recording](https://youtu.be/b5Opn9cFWFk)) and discussed the similarities and differences between the Cells and Organization proposal ([presentation](https://docs.google.com/presentation/d/1FsUi22Up15b_tu6p2m-yLML3hCZ3rgrZrmzJAxUsNmU/edit?usp=sharing)). + +#### Goals of Group::Organization + +As defined in the [organization documentation](../../../user/organization/index.md): + +1. Create an entity to manage everything you do as a GitLab administrator, including: + 1. Defining and applying settings to all of your groups, subgroups, and projects. + 1. Aggregating data from all your groups, subgroups, and projects. +1. Reach feature parity between SaaS and self-managed installations, with all Admin Area settings moving to groups (?). Hardware controls remain on the instance level. + +The [organization roadmap outlines](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals) the current goals in detail. + +#### Potential conflicts with Cells + +- Organization defines a new entity as the primary organizational object for groups and projects. +- We will only introduce one entity +- Group::Organization highlighted the need to further validate the key assumption that users only care about what happens within their organization. + +### Impact on Fulfillment + +We synced with Fulfillment ([recording](https://youtu.be/FkQF3uF7vTY)) to discuss how Cells would impact them. Fulfillment is supportive of an entity above top-level namespaces. Their perspective is outline in [!5639](https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/5639/diffs). + +#### Goals of Fulfillment + +- Fulfillment has a longstanding plan to move billing from the top-level namespace to a level above. This would mean that a license applies for an organization and all its top-level namespaces. +- Fulfillment uses Zuora for billing and would like to have a 1-to-1 relationship between an organization and their Zuora entity called BillingAccount. They want to move away from tying a license to a single user. +- If a customer needs multiple organizations, the corresponding BillingAccounts can be rolled up into a consolidated billing account (similar to [AWS consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html)) +- Ideally, a self-managed instance has a single Organization by default, which should be enough for most customers. +- Fulfillment prefers only one additional entity. + +A rough representation of this is: + + + +#### Potential conflicts with Cells + +- There are no known conflicts between Fulfillment's plans and Cells + +## Iteration plan + +We can't ship the entire Cells architecture in one go - it is too large. Instead, we are adopting an iteration plan that provides value along the way. + +1. Introduce organizations +1. Migrate existing top-level namespaces to organizations +1. Create new organizations on `cell` +1. Migrate existing organizations from `cell` to `cell` +1. Add additional Cell capabilities (DR, Regions) + +### Iteration 0: Introduce organizations + +In the first iteration, we introduce the concept of an organization +as a way to group top-level namespaces together. Support for organizations **does not require any Cells work** but having them will make all subsequent iterations of Cells simpler. This is mainly because we can group top-level namespaces for a single organization onto a Cell. Within an organization all interactions work as normal but we eliminate any cross-organizational interactions except in well defined cases (e.g. forking). + +This means that we don't have a large number of cross-cell interactions. + +Introducing organizations allows GitLab to move towards a multi-tenant system that is similar to Discord's with a single user account but many different "servers" - our organizations - that allow users to switch context. This model harmonizes the UX across self-managed and our SaaS Platforms and is a good fit for Cells. + +Organizations solve the following problems: + +1. We can group top-level namespaces by organization. It is very similar to the initial concept of "instance groups". For example these two top-level namespaces would belong to the organization `GitLab`: + 1. `https://gitlab.com/gitlab-org/` + 1. `https://gitlab.com/gitlab-com/` +1. We can isolate organizations from each other. Top-level namespaces of the same organization can interact within organizations but are not allowed to interact with other namespaces in other organizations. This is useful for customers because it means an organization provides clear boundaries - similar to a self-managed instance. This means we don't have to aggregate user dashboards across everything and can locally scope them to organizations. +1. We don't need to define hierarchies inside an organization. It is a container that could be filled with whatever hierarchy / entity set makes sense (organization, top-level namespaces etc.) +1. Self-managed instances would set a default organization. +1. Organizations can control user-profiles in a central way. This could be achieved by having an organization specific user-profile. Such a profile makes it possible for the organization administrators to control the user role in a company, enforce user emails, or show a graphical indicator of a user being part of the organization. An example would be a "GitLab Employee stamp" on comments. + + + +#### Why would customers opt-in to Organizations? + +By introducing organizations and Cells we can improve the reliability, performance and availability of our SaaS Platforms. + +The first iteration of organizations would also have some benefits by providing more isolation. A simple example would be that `@` mentions could be scoped to an organization. + +Future iterations would create additional value but are beyond the scope of this blueprint. + +Organizations will likely be required in the future as well. + +#### Initial user experience + +1. We create a default `GitLab.com public` organization and assign all public top-level namespaces to it. This allows existing users to access all the data on GitLab.com, exactly as it does now. +1. Any user wanting to opt-in to the benefits of organizations will need to set a single default organization. Any attempts for these users to load a global page like `/dashboard` will end up redirecting to `/-/organizations/<DEFAULT_ORGANIZATION>/dashboard`. +1. New users that opted in to organizations will only ever see data that is related to a single organization. Upon login, data is shown for the default organization. It will be clear to the user how they can switch to a different organization. Users can still navigate to the `GitLab.com` organization but they won't see TODOs from their new organizations in any such views. Instead they'd need to navigate directly to `/organizations/my-company/-/dashboard`. + +### Migrating to Organizations + +Existing customers could also opt-in to migrate their existing top-level paid namespaces to become part of an organization. In most cases this will be a 1-to-1 mapping. But in some cases it may allow a customer to move multiple top-level namespaces into one organization (for example GitLab). + +Migrating to Organizations would be optional. We could even recruit a few beta testers early on to see if this works for them. GitLab itself could dogfood organizations and we'd surface a lot of issues restricting interactions with other namespaces. + +## Iteration 1 - Introduce Cell US 0 + +### GitLab.com as Cell US0 + +GitLab.com will be treated as the first cell `Cell US 0`. It will be unique and much larger compared to newly created cells. All existing top-level namespaces and organizations will remain on `Cell US 0` in the first iteration. + +### Users are globally available + +Users are globally available and the same for all cells. This means that user data needs to be handled separately, for example via decomposition, see [!95941](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95941). + +### Cell groundwork + +In this iteration, we'll lay all the groundwork to support a second Cell for new organizations. This will be transparent to customers. + +## Iteration 2 - Introduce Cell US 1 + +### Add new organizations to Cell US 1 + +After we are ready to support a second Cell, newly created organizations are located by default on `Cell US 1`. The user experience for organizations is already well established. + +### Migrate existing organizations from Cell US 0 to Cell US 1 + +We know that we'll have to move organizations from `Cell US 0` to other cells to reduce its size and ultimately retire the existing GitLab.com architecture. + +By introducing organizations early, we should be able to draw strong "boundaries" across organizations and support migrating existing organizations to a new Cell. + +This is likely going to be GitLab itself - if we can dogfood this, we are likely going to be successful with other organizations as well. + +## Iteration 3 - Introduce Regions + +We can now leverage the Cells architecture to introduce Regions. + +## Iteration 4 - Introduce cross-organizational interactions as needed + +Based on user research, we may want to change certain features to work across organizations. Examples include: + +- Specific features allow for cross-organization interactions, for example forking, search. + +## Technical Proposals + +The Cells architecture do have long lasting implications to data processing, location, scalability and the GitLab architecture. +This section links all different technical proposals that are being evaluated. + +- [Stateless Router That Uses a Cache to Pick Cell and Is Redirected When Wrong Cell Is Reached](proposal-stateless-router-with-buffering-requests.md) + +- [Stateless Router That Uses a Cache to Pick Cell and pre-flight `/api/v4/cells/learn`](proposal-stateless-router-with-routes-learning.md) + +## Impacted features + +The Cells architecture will impact many features requiring some of them to be rewritten, or changed significantly. +This is the list of known affected features with the proposed solutions. + +- [Cells: Git Access](cells-feature-git-access.md) +- [Cells: Data Migration](cells-feature-data-migration.md) +- [Cells: Database Sequences](cells-feature-database-sequences.md) +- [Cells: GraphQL](cells-feature-graphql.md) +- [Cells: Organizations](cells-feature-organizations.md) +- [Cells: Router Endpoints Classification](cells-feature-router-endpoints-classification.md) +- [Cells: Schema changes (Postgres and Elasticsearch migrations)](cells-feature-schema-changes.md) +- [Cells: Backups](cells-feature-backups.md) +- [Cells: Global Search](cells-feature-global-search.md) +- [Cells: CI Runners](cells-feature-ci-runners.md) +- [Cells: Admin Area](cells-feature-admin-area.md) +- [Cells: Secrets](cells-feature-secrets.md) +- [Cells: Container Registry](cells-feature-container-registry.md) +- [Cells: Contributions: Forks](cells-feature-contributions-forks.md) +- [Cells: Personal Namespaces](cells-feature-personal-namespaces.md) +- [Cells: Dashboard: Projects, Todos, Issues, Merge Requests, Activity, ...](cells-feature-dashboard.md) +- [Cells: Snippets](cells-feature-snippets.md) +- [Cells: Uploads](cells-feature-uploads.md) +- [Cells: GitLab Pages](cells-feature-gitlab-pages.md) +- [Cells: Agent for Kubernetes](cells-feature-agent-for-kubernetes.md) + +## Links + +- [Internal Pods presentation](https://docs.google.com/presentation/d/1x1uIiN8FR9fhL7pzFh9juHOVcSxEY7d2_q4uiKKGD44/edit#slide=id.ge7acbdc97a_0_155) +- [Cells Epic](https://gitlab.com/groups/gitlab-org/-/epics/7582) +- [Database Group investigation](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/doc/root-namespace-sharding.html) +- [Shopify Pods architecture](https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale) +- [Opstrace architecture](https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/architecture/overview.md) diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/architecture/blueprints/cells/pods-feature-admin-area.md index b4a6f7f66fb..354a55cacac 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/architecture/blueprints/cells/pods-feature-admin-area.md @@ -1,11 +1,11 @@ --- -redirect_to: '../../../update/background_migrations.md' -remove_date: '2023-03-11' +redirect_to: 'cells-feature-admin-area.md' +remove_date: '2023-06-14' --- -This document was moved to [another location](../../../update/background_migrations.md). +This document was moved to [another location](cells-feature-admin-area.md). -<!-- This redirect file can be deleted after <2023-03-11>. --> +<!-- This redirect file can be deleted after <2023-06-14>. --> <!-- Redirects that point to other docs in the same project expire in three months. --> <!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/architecture/blueprints/cells/pods-feature-agent-for-kubernetes.md index d1d65498fcc..c6a7e138508 100644 --- a/doc/administration/sidekiq/extra_sidekiq_routing.md +++ b/doc/architecture/blueprints/cells/pods-feature-agent-for-kubernetes.md @@ -1,11 +1,11 @@ --- -redirect_to: 'processing_specific_job_classes.md#routing-rules' -remove_date: '2023-02-01' +redirect_to: 'cells-feature-agent-for-kubernetes.md' +remove_date: '2023-06-14' --- -This document was moved to [another location](processing_specific_job_classes.md#routing-rules). +This document was moved to [another location](cells-feature-agent-for-kubernetes.md). -<!-- This redirect file can be deleted after <2023-02-01>. --> +<!-- This redirect file can be deleted after <2023-06-14>. --> <!-- Redirects that point to other docs in the same project expire in three months. --> <!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/directory_structure.md b/doc/architecture/blueprints/cells/pods-feature-backups.md index 34ee86d9ee5..6927142f993 100644 --- a/doc/development/directory_structure.md +++ b/doc/architecture/blueprints/cells/pods-feature-backups.md @@ -1,11 +1,11 @@ --- -redirect_to: 'software_design.md' -remove_date: '2023-01-24' +redirect_to: 'cells-feature-backups.md' +remove_date: '2023-06-14' --- -This document was moved to [another location](software_design.md) +This document was moved to [another location](cells-feature-backups.md). -<!-- This redirect file can be deleted after <2023-01-24>. --> +<!-- This redirect file can be deleted after <2023-06-14>. --> <!-- Redirects that point to other docs in the same project expire in three months. --> <!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-ci-runners.md b/doc/architecture/blueprints/cells/pods-feature-ci-runners.md new file mode 100644 index 00000000000..fd9c0094b2f --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-ci-runners.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-ci-runners.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-ci-runners.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-container-registry.md b/doc/architecture/blueprints/cells/pods-feature-container-registry.md new file mode 100644 index 00000000000..f6a1aa997c3 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-container-registry.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-container-registry.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-container-registry.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-contributions-forks.md b/doc/architecture/blueprints/cells/pods-feature-contributions-forks.md new file mode 100644 index 00000000000..441a0da11b7 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-contributions-forks.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-contributions-forks.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-contributions-forks.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-dashboard.md b/doc/architecture/blueprints/cells/pods-feature-dashboard.md new file mode 100644 index 00000000000..9404f3b647b --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-dashboard.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-dashboard.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-dashboard.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-data-migration.md b/doc/architecture/blueprints/cells/pods-feature-data-migration.md new file mode 100644 index 00000000000..023b54ae5e4 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-data-migration.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-data-migration.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-data-migration.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-database-sequences.md b/doc/architecture/blueprints/cells/pods-feature-database-sequences.md new file mode 100644 index 00000000000..785289081bb --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-database-sequences.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-database-sequences.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-database-sequences.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-git-access.md b/doc/architecture/blueprints/cells/pods-feature-git-access.md new file mode 100644 index 00000000000..bc44137f8d8 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-git-access.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-git-access.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-git-access.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-gitlab-pages.md b/doc/architecture/blueprints/cells/pods-feature-gitlab-pages.md new file mode 100644 index 00000000000..8043155b1a6 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-gitlab-pages.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-gitlab-pages.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-gitlab-pages.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-global-search.md b/doc/architecture/blueprints/cells/pods-feature-global-search.md new file mode 100644 index 00000000000..d472b6a9ff6 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-global-search.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-global-search.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-global-search.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-graphql.md b/doc/architecture/blueprints/cells/pods-feature-graphql.md new file mode 100644 index 00000000000..23d7b0bb856 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-graphql.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-graphql.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-graphql.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-organizations.md b/doc/architecture/blueprints/cells/pods-feature-organizations.md new file mode 100644 index 00000000000..92ab37a21bf --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-organizations.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-organizations.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-organizations.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-personal-namespaces.md b/doc/architecture/blueprints/cells/pods-feature-personal-namespaces.md new file mode 100644 index 00000000000..0f05c7d83cb --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-personal-namespaces.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-personal-namespaces.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-personal-namespaces.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-router-endpoints-classification.md b/doc/architecture/blueprints/cells/pods-feature-router-endpoints-classification.md new file mode 100644 index 00000000000..91de121f694 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-router-endpoints-classification.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-router-endpoints-classification.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-router-endpoints-classification.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-schema-changes.md b/doc/architecture/blueprints/cells/pods-feature-schema-changes.md new file mode 100644 index 00000000000..5f7de0eb486 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-schema-changes.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-schema-changes.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-schema-changes.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-secrets.md b/doc/architecture/blueprints/cells/pods-feature-secrets.md new file mode 100644 index 00000000000..5c482e1c986 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-secrets.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-secrets.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-secrets.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-snippets.md b/doc/architecture/blueprints/cells/pods-feature-snippets.md new file mode 100644 index 00000000000..867c9b49955 --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-snippets.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-snippets.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-snippets.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-template.md b/doc/architecture/blueprints/cells/pods-feature-template.md new file mode 100644 index 00000000000..e1150c3426f --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-template.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-template.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-template.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/pods-feature-uploads.md b/doc/architecture/blueprints/cells/pods-feature-uploads.md new file mode 100644 index 00000000000..7280f70ebdb --- /dev/null +++ b/doc/architecture/blueprints/cells/pods-feature-uploads.md @@ -0,0 +1,11 @@ +--- +redirect_to: 'cells-feature-uploads.md' +remove_date: '2023-06-14' +--- + +This document was moved to [another location](cells-feature-uploads.md). + +<!-- This redirect file can be deleted after <2023-06-14>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md new file mode 100644 index 00000000000..12102429b03 --- /dev/null +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md @@ -0,0 +1,648 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells Stateless Router Proposal' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Proposal: Stateless Router + +We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related +tables so that they can be shared between all cells and allow any cell to +authenticate a user and route requests to the correct cell. Cells may receive +requests for the resources they don't own, but they know how to redirect back +to the correct cell. + +The router is stateless and does not read from the `routes` database which +means that all interactions with the database still happen from the Rails +monolith. This architecture also supports regions by allowing for low traffic +databases to be replicated across regions. + +Users are not directly exposed to the concept of Cells but instead they see +different data dependent on their chosen "organization". +[Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the +application and allow us to decide which request route to which cell, since an +organization can only be on a single cell. + +## Differences + +The main difference between this proposal and the one [with learning routes](proposal-stateless-router-with-routes-learning.md) +is that this proposal always sends requests to any of the Cells. If the requests cannot be processed, +the requests will be bounced back with relevant headers. This requires that request to be buffered. +It allows that request decoding can be either via URI or Body of request by Rails. +This means that each request might be sent more than once and be processed more than once as result. + +The [with learning routes proposal](proposal-stateless-router-with-routes-learning.md) requires that +routable information is always encoded in URI, and the router sends a pre-flight request. + +## Summary in diagrams + +This shows how a user request routes via DNS to the nearest router and the router chooses a cell to send the request to. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + end +``` + +<details><summary>More detail</summary> + +This shows that the router can actually send requests to any cell. The user will +get the closest router to them geographically. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + end + router_eu-.->cell_us0; + router_eu-.->cell_us1; + router_us-.->cell_eu0; + router_us-.->cell_eu1; +``` + +</details> + +<details><summary>Even more detail</summary> + +This shows the databases. `gitlab_users` and `gitlab_routes` exist only in the +US region but are replicated to other regions. Replication does not have an +arrow because it's too hard to read the diagram. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + db_gitlab_users[(gitlab_users Primary)]; + db_gitlab_routes[(gitlab_routes Primary)]; + db_gitlab_users_replica[(gitlab_users Replica)]; + db_gitlab_routes_replica[(gitlab_routes Replica)]; + db_cell_us0[(gitlab_main/gitlab_ci Cell US0)]; + db_cell_us1[(gitlab_main/gitlab_ci Cell US1)]; + db_cell_eu0[(gitlab_main/gitlab_ci Cell EU0)]; + db_cell_eu1[(gitlab_main/gitlab_ci Cell EU1)]; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + cell_eu0-->db_cell_eu0; + cell_eu0-->db_gitlab_users_replica; + cell_eu0-->db_gitlab_routes_replica; + cell_eu1-->db_gitlab_users_replica; + cell_eu1-->db_gitlab_routes_replica; + cell_eu1-->db_cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + cell_us0-->db_cell_us0; + cell_us0-->db_gitlab_users; + cell_us0-->db_gitlab_routes; + cell_us1-->db_gitlab_users; + cell_us1-->db_gitlab_routes; + cell_us1-->db_cell_us1; + end + router_eu-.->cell_us0; + router_eu-.->cell_us1; + router_us-.->cell_eu0; + router_us-.->cell_eu1; +``` + +</details> + +## Summary of changes + +1. Tables related to User data (including profile settings, authentication credentials, personal access tokens) are decomposed into a `gitlab_users` schema +1. The `routes` table is decomposed into `gitlab_routes` schema +1. The `application_settings` (and probably a few other instance level tables) are decomposed into `gitlab_admin` schema +1. A new column `routes.cell_id` is added to `routes` table +1. A new Router service exists to choose which cell to route a request to. +1. A new concept will be introduced in GitLab called an organization and a user can select a "default organization" and this will be a user level setting. The default organization is used to redirect users away from ambiguous routes like `/dashboard` to organization scoped routes like `/organizations/my-organization/-/dashboard`. Legacy users will have a special default organization that allows them to keep using global resources on `Cell US0`. All existing namespaces will initially move to this public organization. +1. If a cell receives a request for a `routes.cell_id` that it does not own it returns a `302` with `X-Gitlab-Cell-Redirect` header so that the router can send the request to the correct cell. The correct cell can also set a header `X-Gitlab-Cell-Cache` which contains information about how this request should be cached to remember the cell. For example if the request was `/gitlab-org/gitlab` then the header would encode `/gitlab-org/* => Cell US0` (for example, any requests starting with `/gitlab-org/` can always be routed to `Cell US0` +1. When the cell does not know (from the cache) which cell to send a request to it just picks a random cell within it's region +1. Writes to `gitlab_users` and `gitlab_routes` are sent to a primary PostgreSQL server in our `US` region but reads can come from replicas in the same region. This will add latency for these writes but we expect they are infrequent relative to the rest of GitLab. + +## Detailed explanation of default organization in the first iteration + +All users will get a new column `users.default_organization` which they can +control in user settings. We will introduce a concept of the +`GitLab.com Public` organization. This will be set as the default organization for all existing +users. This organization will allow the user to see data from all namespaces in +`Cell US0` (for example, our original GitLab.com instance). This behavior can be invisible to +existing users such that they don't even get told when they are viewing a +global page like `/dashboard` that it's even scoped to an organization. + +Any new users with a default organization other than `GitLab.com Public` will have +a distinct user experience and will be fully aware that every page they load is +only ever scoped to a single organization. These users can never +load any global pages like `/dashboard` and will end up being redirected to +`/organizations/<DEFAULT_ORGANIZATION>/-/dashboard`. This may also be the case +for legacy APIs and such users may only ever be able to use APIs scoped to a +organization. + +## Detailed explanation of Admin Area settings + +We believe that maintaining and synchronizing Admin Area settings will be +frustrating and painful so to avoid this we will decompose and share all Admin Area +settings in the `gitlab_admin` schema. This should be safe (similar to other +shared schemas) because these receive very little write traffic. + +In cases where different cells need different settings (for example, the +Elasticsearch URL), we will either decide to use a templated +format in the relevant `application_settings` row which allows it to be dynamic +per cell. Alternatively if that proves difficult we'll introduce a new table +called `per_cell_application_settings` and this will have 1 row per cell to allow +setting different settings per cell. It will still be part of the `gitlab_admin` +schema and shared which will allow us to centrally manage it and simplify +keeping settings in sync for all cells. + +## Pros + +1. Router is stateless and can live in many regions. We use Anycast DNS to resolve to nearest region for the user. +1. Cells can receive requests for namespaces in the wrong cell and the user + still gets the right response as well as caching at the router that + ensures the next request is sent to the correct cell so the next request + will go to the correct cell +1. The majority of the code still lives in `gitlab` rails codebase. The Router doesn't actually need to understand how GitLab URLs are composed. +1. Since the responsibility to read and write `gitlab_users`, + `gitlab_routes` and `gitlab_admin` still lives in Rails it means minimal + changes will be needed to the Rails application compared to extracting + services that need to isolate the domain models and build new interfaces. +1. Compared to a separate routing service this allows the Rails application + to encode more complex rules around how to map URLs to the correct cell + and may work for some existing API endpoints. +1. All the new infrastructure (just a router) is optional and a single-cell + self-managed installation does not even need to run the Router and there are + no other new services. + +## Cons + +1. `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases may need to be + replicated across regions and writes need to go across regions. We need to + do an analysis on write TPS for the relevant tables to determine if this is + feasible. +1. Sharing access to the database from many different Cells means that they are + all coupled at the Postgres schema level and this means changes to the + database schema need to be done carefully in sync with the deployment of all + Cells. This limits us to ensure that Cells are kept in closely similar + versions compared to an architecture with shared services that have an API + we control. +1. Although most data is stored in the right region there can be requests + proxied from another region which may be an issue for certain types + of compliance. +1. Data in `gitlab_users` and `gitlab_routes` databases must be replicated in + all regions which may be an issue for certain types of compliance. +1. The router cache may need to be very large if we get a wide variety of URLs + (for example, long tail). In such a case we may need to implement a 2nd level of + caching in user cookies so their frequently accessed pages always go to the + right cell the first time. +1. Having shared database access for `gitlab_users` and `gitlab_routes` + from multiple cells is an unusual architecture decision compared to + extracting services that are called from multiple cells. +1. It is very likely we won't be able to find cacheable elements of a + GraphQL URL and often existing GraphQL endpoints are heavily dependent on + ids that won't be in the `routes` table so cells won't necessarily know + what cell has the data. As such we'll probably have to update our GraphQL + calls to include an organization context in the path like + `/api/organizations/<organization>/graphql`. +1. This architecture implies that implemented endpoints can only access data + that are readily accessible on a given Cell, but are unlikely + to aggregate information from many Cells. +1. All unknown routes are sent to the latest deployment which we assume to be `Cell US0`. + This is required as newly added endpoints will be only decodable by latest cell. + This Cell could later redirect to correct one that can serve the given request. + Since request processing might be heavy some Cells might receive significant amount + of traffic due to that. + +## Example database configuration + +Handling shared `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases, while having dedicated `gitlab_main` and `gitlab_ci` databases should already be handled by the way we use `config/database.yml`. We should also, already be able to handle the dedicated EU replicas while having a single US primary for `gitlab_users` and `gitlab_routes`. Below is a snippet of part of the database configuration for the Cell architecture described above. + +<details><summary>Cell US0</summary> + +```yaml +# config/database.yml +production: + main: + host: postgres-main.cell-us0.primary.consul + load_balancing: + discovery: postgres-main.cell-us0.replicas.consul + ci: + host: postgres-ci.cell-us0.primary.consul + load_balancing: + discovery: postgres-ci.cell-us0.replicas.consul + users: + host: postgres-users-primary.consul + load_balancing: + discovery: postgres-users-replicas.us.consul + routes: + host: postgres-routes-primary.consul + load_balancing: + discovery: postgres-routes-replicas.us.consul + admin: + host: postgres-admin-primary.consul + load_balancing: + discovery: postgres-admin-replicas.us.consul +``` + +</details> + +<details><summary>Cell EU0</summary> + +```yaml +# config/database.yml +production: + main: + host: postgres-main.cell-eu0.primary.consul + load_balancing: + discovery: postgres-main.cell-eu0.replicas.consul + ci: + host: postgres-ci.cell-eu0.primary.consul + load_balancing: + discovery: postgres-ci.cell-eu0.replicas.consul + users: + host: postgres-users-primary.consul + load_balancing: + discovery: postgres-users-replicas.eu.consul + routes: + host: postgres-routes-primary.consul + load_balancing: + discovery: postgres-routes-replicas.eu.consul + admin: + host: postgres-admin-primary.consul + load_balancing: + discovery: postgres-admin-replicas.eu.consul +``` + +</details> + +## Request flows + +1. `gitlab-org` is a top level namespace and lives in `Cell US0` in the `GitLab.com Public` organization +1. `my-company` is a top level namespace and lives in `Cell EU0` in the `my-organization` organization + +### Experience for paying user that is part of `my-organization` + +Such a user will have a default organization set to `/my-organization` and will be +unable to load any global routes outside of this organization. They may load other +projects/namespaces but their MR/Todo/Issue counts at the top of the page will +not be correctly populated in the first iteration. The user will be aware of +this limitation. + +#### Navigates to `/my-company/my-project` while logged in + +1. User is in Europe so DNS resolves to the router in Europe +1. They request `/my-company/my-project` without the router cache, so the router chooses randomly `Cell EU1` +1. `Cell EU1` does not have `/my-company`, but it knows that it lives in `Cell EU0` so it redirects the router to `Cell EU0` +1. `Cell EU0` returns the correct response as well as setting the cache headers for the router `/my-company/* => Cell EU0` +1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Cell EU0` + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu1: GET /my-company/my-project + cell_eu1->>router_eu: 302 /my-company/my-project X-Gitlab-Cell-Redirect={cell:Cell EU0} + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... X-Gitlab-Cell-Cache={path_prefix:/my-company/} +``` + +#### Navigates to `/my-company/my-project` while not logged in + +1. User is in Europe so DNS resolves to the router in Europe +1. The router does not have `/my-company/*` cached yet so it chooses randomly `Cell EU1` +1. `Cell EU1` redirects them through a login flow +1. Still they request `/my-company/my-project` without the router cache, so the router chooses a random cell `Cell EU1` +1. `Cell EU1` does not have `/my-company`, but it knows that it lives in `Cell EU0` so it redirects the router to `Cell EU0` +1. `Cell EU0` returns the correct response as well as setting the cache headers for the router `/my-company/* => Cell EU0` +1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Cell EU0` + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu1: GET /my-company/my-project + cell_eu1->>user: 302 /users/sign_in?redirect=/my-company/my-project + user->>router_eu: GET /users/sign_in?redirect=/my-company/my-project + router_eu->>cell_eu1: GET /users/sign_in?redirect=/my-company/my-project + cell_eu1->>user: <h1>Sign in... + user->>router_eu: POST /users/sign_in?redirect=/my-company/my-project + router_eu->>cell_eu1: POST /users/sign_in?redirect=/my-company/my-project + cell_eu1->>user: 302 /my-company/my-project + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu1: GET /my-company/my-project + cell_eu1->>router_eu: 302 /my-company/my-project X-Gitlab-Cell-Redirect={cell:Cell EU0} + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... X-Gitlab-Cell-Cache={path_prefix:/my-company/} +``` + +#### Navigates to `/my-company/my-other-project` after last step + +1. User is in Europe so DNS resolves to the router in Europe +1. The router cache now has `/my-company/* => Cell EU0`, so the router chooses `Cell EU0` +1. `Cell EU0` returns the correct response as well as the cache header again + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... X-Gitlab-Cell-Cache={path_prefix:/my-company/} +``` + +#### Navigates to `/gitlab-org/gitlab` after last step + +1. User is in Europe so DNS resolves to the router in Europe +1. The router has no cached value for this URL so randomly chooses `Cell EU0` +1. `Cell EU0` redirects the router to `Cell US0` +1. `Cell US0` returns the correct response as well as the cache header again + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_us0 as Cell US0 + user->>router_eu: GET /gitlab-org/gitlab + router_eu->>cell_eu0: GET /gitlab-org/gitlab + cell_eu0->>router_eu: 302 /gitlab-org/gitlab X-Gitlab-Cell-Redirect={cell:Cell US0} + router_eu->>cell_us0: GET /gitlab-org/gitlab + cell_us0->>user: <h1>GitLab.org... X-Gitlab-Cell-Cache={path_prefix:/gitlab-org/} +``` + +In this case the user is not on their "default organization" so their TODO +counter will not include their normal todos. We may choose to highlight this in +the UI somewhere. A future iteration may be able to fetch that for them from +their default organization. + +#### Navigates to `/` + +1. User is in Europe so DNS resolves to the router in Europe +1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) +1. The Router choose `Cell EU0` randomly +1. The Rails application knows the users default organization is `/my-organization`, so + it redirects the user to `/organizations/my-organization/-/dashboard` +1. The Router has a cached value for `/organizations/my-organization/*` so it then sends the + request to `POD EU0` +1. `Cell EU0` serves up a new page `/organizations/my-organization/-/dashboard` which is the same + dashboard view we have today but scoped to an organization clearly in the UI +1. The user is (optionally) presented with a message saying that data on this page is only + from their default organization and that they can change their default + organization if it's not right. + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + user->>router_eu: GET / + router_eu->>cell_eu0: GET / + cell_eu0->>user: 302 /organizations/my-organization/-/dashboard + user->>router: GET /organizations/my-organization/-/dashboard + router->>cell_eu0: GET /organizations/my-organization/-/dashboard + cell_eu0->>user: <h1>My Company Dashboard... X-Gitlab-Cell-Cache={path_prefix:/organizations/my-organization/} +``` + +#### Navigates to `/dashboard` + +As above, they will end up on `/organizations/my-organization/-/dashboard` as +the rails application will already redirect `/` to the dashboard page. + +### Navigates to `/not-my-company/not-my-project` while logged in (but they don't have access since this project/group is private) + +1. User is in Europe so DNS resolves to the router in Europe +1. The router knows that `/not-my-company` lives in `Cell US1` so sends the request to this +1. The user does not have access so `Cell US1` returns 404 + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_us1 as Cell US1 + user->>router_eu: GET /not-my-company/not-my-project + router_eu->>cell_us1: GET /not-my-company/not-my-project + cell_us1->>user: 404 +``` + +#### Creates a new top level namespace + +The user will be asked which organization they want the namespace to belong to. +If they select `my-organization` then it will end up on the same cell as all +other namespaces in `my-organization`. If they select nothing we default to +`GitLab.com Public` and it is clear to the user that this is isolated from +their existing organization such that they won't be able to see data from both +on a single page. + +### Experience for GitLab team member that is part of `/gitlab-org` + +Such a user is considered a legacy user and has their default organization set to +`GitLab.com Public`. This is a "meta" organization that does not really exist but +the Rails application knows to interpret this organization to mean that they are +allowed to use legacy global functionality like `/dashboard` to see data across +namespaces located on `Cell US0`. The rails backend also knows that the default cell to render any ambiguous +routes like `/dashboard` is `Cell US0`. Lastly the user will be allowed to +navigate to organizations on another cell like `/my-organization` but when they do the +user will see a message indicating that some data may be missing (for example, the +MRs/Issues/Todos) counts. + +#### Navigates to `/gitlab-org/gitlab` while not logged in + +1. User is in the US so DNS resolves to the US router +1. The router knows that `/gitlab-org` lives in `Cell US0` so sends the request + to this cell +1. `Cell US0` serves up the response + +```mermaid +sequenceDiagram + participant user as User + participant router_us as Router US + participant cell_us0 as Cell US0 + user->>router_us: GET /gitlab-org/gitlab + router_us->>cell_us0: GET /gitlab-org/gitlab + cell_us0->>user: <h1>GitLab.org... X-Gitlab-Cell-Cache={path_prefix:/gitlab-org/} +``` + +#### Navigates to `/` + +1. User is in US so DNS resolves to the router in US +1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) +1. The Router chooses `Cell US1` randomly +1. The Rails application knows the users default organization is `GitLab.com Public`, so + it redirects the user to `/dashboards` (only legacy users can see + `/dashboard` global view) +1. Router does not have a cache for `/dashboard` route (specifically rails never tells it to cache this route) +1. The Router chooses `Cell US1` randomly +1. The Rails application knows the users default organization is `GitLab.com Public`, so + it allows the user to load `/dashboards` (only legacy users can see + `/dashboard` global view) and redirects to router the legacy cell which is `Cell US0` +1. `Cell US0` serves up the global view dashboard page `/dashboard` which is the same + dashboard view we have today + +```mermaid +sequenceDiagram + participant user as User + participant router_us as Router US + participant cell_us0 as Cell US0 + participant cell_us1 as Cell US1 + user->>router_us: GET / + router_us->>cell_us1: GET / + cell_us1->>user: 302 /dashboard + user->>router_us: GET /dashboard + router_us->>cell_us1: GET /dashboard + cell_us1->>router_us: 302 /dashboard X-Gitlab-Cell-Redirect={cell:Cell US0} + router_us->>cell_us0: GET /dashboard + cell_us0->>user: <h1>Dashboard... +``` + +#### Navigates to `/my-company/my-other-project` while logged in (but they don't have access since this project is private) + +They get a 404. + +### Experience for non-authenticated users + +Flow is similar to authenticated users except global routes like `/dashboard` will +redirect to the login page as there is no default organization to choose from. + +### A new customers signs up + +They will be asked if they are already part of an organization or if they'd +like to create one. If they choose neither they end up no the default +`GitLab.com Public` organization. + +### An organization is moved from 1 cell to another + +TODO + +### GraphQL/API requests which don't include the namespace in the URL + +TODO + +### The autocomplete suggestion functionality in the search bar which remembers recent issues/MRs + +TODO + +### Global search + +TODO + +## Administrator + +### Loads `/admin` page + +1. Router picks a random cell `Cell US0` +1. Cell US0 redirects user to `/admin/cells/cellus0` +1. Cell US0 renders an Admin Area page and also returns a cache header to cache `/admin/cellss/cellus0/* => Cell US0`. The Admin Area page contains a dropdown list showing other cells they could select and it changes the query parameter. + +Admin Area settings in Postgres are all shared across all cells to avoid +divergence but we still make it clear in the URL and UI which cell is serving +the Admin Area page as there is dynamic data being generated from these pages and +the operator may want to view a specific cell. + +## More Technical Problems To Solve + +### Replicating User Sessions Between All Cells + +Today user sessions live in Redis but each cell will have their own Redis instance. We already use a dedicated Redis instance for sessions so we could consider sharing this with all cells like we do with `gitlab_users` PostgreSQL database. But an important consideration will be latency as we would still want to mostly fetch sessions from the same region. + +An alternative might be that user sessions get moved to a JWT payload that encodes all the session data but this has downsides. For example, it is difficult to expire a user session, when their password changes or for other reasons, if the session lives in a JWT controlled by the user. + +### How do we migrate between Cells + +Migrating data between cells will need to factor all data stores: + +1. PostgreSQL +1. Redis Shared State +1. Gitaly +1. Elasticsearch + +### Is it still possible to leak the existence of private groups via a timing attack? + +If you have router in EU, and you know that EU router by default redirects +to EU located Cells, you know their latency (lets assume 10 ms). Now, if your +request is bounced back and redirected to US which has different latency +(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was +returned by US Cell and know that your 404 is in fact 403. + +We may defer this until we actually implement a cell in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. + +One technique to mitigate this risk might be to have the router add a random +delay to any request that returns 404 from a cell. + +## Should runners be shared across all cells? + +We have 2 options and we should decide which is easier: + +1. Decompose runner registration and queuing tables and share them across all + cells. This may have implications for scalability, and we'd need to consider + if this would include group/project runners as this may have scalability + concerns as these are high traffic tables that would need to be shared. +1. Runners are registered per-cell and, we probably have a separate fleet of + runners for every cell or just register the same runners to many cells which + may have implications for queueing + +## How do we guarantee unique ids across all cells for things that cannot conflict? + +This project assumes at least namespaces and projects have unique ids across +all cells as many requests need to be routed based on their ID. Since those +tables are across different databases then guaranteeing a unique ID will +require a new solution. There are likely other tables where unique IDs are +necessary and depending on how we resolve routing for GraphQL and other APIs +and other design goals it may be determined that we want the primary key to be +unique for all tables. diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md new file mode 100644 index 00000000000..7f25441c75f --- /dev/null +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md @@ -0,0 +1,672 @@ +--- +stage: enablement +group: Tenant Scale +comments: false +description: 'Cells Stateless Router Proposal' +--- + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Proposal: Stateless Router + +We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related +tables so that they can be shared between all cells and allow any cell to +authenticate a user and route requests to the correct cell. Cells may receive +requests for the resources they don't own, but they know how to redirect back +to the correct cell. + +The router is stateless and does not read from the `routes` database which +means that all interactions with the database still happen from the Rails +monolith. This architecture also supports regions by allowing for low traffic +databases to be replicated across regions. + +Users are not directly exposed to the concept of Cells but instead they see +different data dependent on their chosen "organization". +[Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the +application and allow us to decide which request route to which cell, since an +organization can only be on a single cell. + +## Differences + +The main difference between this proposal and one [with buffering requests](proposal-stateless-router-with-buffering-requests.md) +is that this proposal uses a pre-flight API request (`/api/v4/cells/learn`) to redirect the request body to the correct Cell. +This means that each request is sent exactly once to be processed, but the URI is used to decode which Cell it should be directed. + +## Summary in diagrams + +This shows how a user request routes via DNS to the nearest router and the router chooses a cell to send the request to. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + end +``` + +### More detail + +This shows that the router can actually send requests to any cell. The user will +get the closest router to them geographically. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + end + router_eu-.->cell_us0; + router_eu-.->cell_us1; + router_us-.->cell_eu0; + router_us-.->cell_eu1; +``` + +### Even more detail + +This shows the databases. `gitlab_users` and `gitlab_routes` exist only in the +US region but are replicated to other regions. Replication does not have an +arrow because it's too hard to read the diagram. + +```mermaid +graph TD; + user((User)); + dns[DNS]; + router_us(Router); + router_eu(Router); + cell_us0{Cell US0}; + cell_us1{Cell US1}; + cell_eu0{Cell EU0}; + cell_eu1{Cell EU1}; + db_gitlab_users[(gitlab_users Primary)]; + db_gitlab_routes[(gitlab_routes Primary)]; + db_gitlab_users_replica[(gitlab_users Replica)]; + db_gitlab_routes_replica[(gitlab_routes Replica)]; + db_cell_us0[(gitlab_main/gitlab_ci Cell US0)]; + db_cell_us1[(gitlab_main/gitlab_ci Cell US1)]; + db_cell_eu0[(gitlab_main/gitlab_ci Cell EU0)]; + db_cell_eu1[(gitlab_main/gitlab_ci Cell EU1)]; + user-->dns; + dns-->router_us; + dns-->router_eu; + subgraph Europe + router_eu-->cell_eu0; + router_eu-->cell_eu1; + cell_eu0-->db_cell_eu0; + cell_eu0-->db_gitlab_users_replica; + cell_eu0-->db_gitlab_routes_replica; + cell_eu1-->db_gitlab_users_replica; + cell_eu1-->db_gitlab_routes_replica; + cell_eu1-->db_cell_eu1; + end + subgraph United States + router_us-->cell_us0; + router_us-->cell_us1; + cell_us0-->db_cell_us0; + cell_us0-->db_gitlab_users; + cell_us0-->db_gitlab_routes; + cell_us1-->db_gitlab_users; + cell_us1-->db_gitlab_routes; + cell_us1-->db_cell_us1; + end + router_eu-.->cell_us0; + router_eu-.->cell_us1; + router_us-.->cell_eu0; + router_us-.->cell_eu1; +``` + +## Summary of changes + +1. Tables related to User data (including profile settings, authentication credentials, personal access tokens) are decomposed into a `gitlab_users` schema +1. The `routes` table is decomposed into `gitlab_routes` schema +1. The `application_settings` (and probably a few other instance level tables) are decomposed into `gitlab_admin` schema +1. A new column `routes.cell_id` is added to `routes` table +1. A new Router service exists to choose which cell to route a request to. +1. If a router receives a new request it will send `/api/v4/cells/learn?method=GET&path_info=/group-org/project` to learn which Cell can process it +1. A new concept will be introduced in GitLab called an organization +1. We require all existing endpoints to be routable by URI, or be fixed to a specific Cell for processing. This requires changing ambiguous endpoints like `/dashboard` to be scoped like `/organizations/my-organization/-/dashboard` +1. Endpoints like `/admin` would be routed always to the specific Cell, like `cell_0` +1. Each Cell can respond to `/api/v4/cells/learn` and classify each endpoint +1. Writes to `gitlab_users` and `gitlab_routes` are sent to a primary PostgreSQL server in our `US` region but reads can come from replicas in the same region. This will add latency for these writes but we expect they are infrequent relative to the rest of GitLab. + +## Pre-flight request learning + +While processing a request the URI will be decoded and a pre-flight request +will be sent for each non-cached endpoint. + +When asking for the endpoint GitLab Rails will return information about +the routable path. GitLab Rails will decode `path_info` and match it to +an existing endpoint and find a routable entity (like project). The router will +treat this as short-lived cache information. + +1. Prefix match: `/api/v4/cells/learn?method=GET&path_info=/gitlab-org/gitlab-test/-/issues` + + ```json + { + "path": "/gitlab-org/gitlab-test", + "cell": "cell_0", + "source": "routable" + } + ``` + +1. Some endpoints might require an exact match: `/api/v4/cells/learn?method=GET&path_info=/-/profile` + + ```json + { + "path": "/-/profile", + "cell": "cell_0", + "source": "fixed", + "exact": true + } + ``` + +## Detailed explanation of default organization in the first iteration + +All users will get a new column `users.default_organization` which they can +control in user settings. We will introduce a concept of the +`GitLab.com Public` organization. This will be set as the default organization for all existing +users. This organization will allow the user to see data from all namespaces in +`Cell US0` (ie. our original GitLab.com instance). This behavior can be invisible to +existing users such that they don't even get told when they are viewing a +global page like `/dashboard` that it's even scoped to an organization. + +Any new users with a default organization other than `GitLab.com Public` will have +a distinct user experience and will be fully aware that every page they load is +only ever scoped to a single organization. These users can never +load any global pages like `/dashboard` and will end up being redirected to +`/organizations/<DEFAULT_ORGANIZATION>/-/dashboard`. This may also be the case +for legacy APIs and such users may only ever be able to use APIs scoped to a +organization. + +## Detailed explanation of Admin Area settings + +We believe that maintaining and synchronizing Admin Area settings will be +frustrating and painful so to avoid this we will decompose and share all Admin Area +settings in the `gitlab_admin` schema. This should be safe (similar to other +shared schemas) because these receive very little write traffic. + +In cases where different cells need different settings (eg. the +Elasticsearch URL), we will either decide to use a templated +format in the relevant `application_settings` row which allows it to be dynamic +per cell. Alternatively if that proves difficult we'll introduce a new table +called `per_cell_application_settings` and this will have 1 row per cell to allow +setting different settings per cell. It will still be part of the `gitlab_admin` +schema and shared which will allow us to centrally manage it and simplify +keeping settings in sync for all cells. + +## Pros + +1. Router is stateless and can live in many regions. We use Anycast DNS to resolve to nearest region for the user. +1. Cells can receive requests for namespaces in the wrong cell and the user + still gets the right response as well as caching at the router that + ensures the next request is sent to the correct cell so the next request + will go to the correct cell +1. The majority of the code still lives in `gitlab` rails codebase. The Router doesn't actually need to understand how GitLab URLs are composed. +1. Since the responsibility to read and write `gitlab_users`, + `gitlab_routes` and `gitlab_admin` still lives in Rails it means minimal + changes will be needed to the Rails application compared to extracting + services that need to isolate the domain models and build new interfaces. +1. Compared to a separate routing service this allows the Rails application + to encode more complex rules around how to map URLs to the correct cell + and may work for some existing API endpoints. +1. All the new infrastructure (just a router) is optional and a single-cell + self-managed installation does not even need to run the Router and there are + no other new services. + +## Cons + +1. `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases may need to be + replicated across regions and writes need to go across regions. We need to + do an analysis on write TPS for the relevant tables to determine if this is + feasible. +1. Sharing access to the database from many different Cells means that they are + all coupled at the Postgres schema level and this means changes to the + database schema need to be done carefully in sync with the deployment of all + Cells. This limits us to ensure that Cells are kept in closely similar + versions compared to an architecture with shared services that have an API + we control. +1. Although most data is stored in the right region there can be requests + proxied from another region which may be an issue for certain types + of compliance. +1. Data in `gitlab_users` and `gitlab_routes` databases must be replicated in + all regions which may be an issue for certain types of compliance. +1. The router cache may need to be very large if we get a wide variety of URLs + (ie. long tail). In such a case we may need to implement a 2nd level of + caching in user cookies so their frequently accessed pages always go to the + right cell the first time. +1. Having shared database access for `gitlab_users` and `gitlab_routes` + from multiple cells is an unusual architecture decision compared to + extracting services that are called from multiple cells. +1. It is very likely we won't be able to find cacheable elements of a + GraphQL URL and often existing GraphQL endpoints are heavily dependent on + ids that won't be in the `routes` table so cells won't necessarily know + what cell has the data. As such we'll probably have to update our GraphQL + calls to include an organization context in the path like + `/api/organizations/<organization>/graphql`. +1. This architecture implies that implemented endpoints can only access data + that are readily accessible on a given Cell, but are unlikely + to aggregate information from many Cells. +1. All unknown routes are sent to the latest deployment which we assume to be `Cell US0`. + This is required as newly added endpoints will be only decodable by latest cell. + Likely this is not a problem for the `/cells/learn` is it is lightweight + to process and this should not cause a performance impact. + +## Example database configuration + +Handling shared `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases, while having dedicated `gitlab_main` and `gitlab_ci` databases should already be handled by the way we use `config/database.yml`. We should also, already be able to handle the dedicated EU replicas while having a single US primary for `gitlab_users` and `gitlab_routes`. Below is a snippet of part of the database configuration for the Cell architecture described above. + +**Cell US0**: + +```yaml +# config/database.yml +production: + main: + host: postgres-main.cell-us0.primary.consul + load_balancing: + discovery: postgres-main.cell-us0.replicas.consul + ci: + host: postgres-ci.cell-us0.primary.consul + load_balancing: + discovery: postgres-ci.cell-us0.replicas.consul + users: + host: postgres-users-primary.consul + load_balancing: + discovery: postgres-users-replicas.us.consul + routes: + host: postgres-routes-primary.consul + load_balancing: + discovery: postgres-routes-replicas.us.consul + admin: + host: postgres-admin-primary.consul + load_balancing: + discovery: postgres-admin-replicas.us.consul +``` + +**Cell EU0**: + +```yaml +# config/database.yml +production: + main: + host: postgres-main.cell-eu0.primary.consul + load_balancing: + discovery: postgres-main.cell-eu0.replicas.consul + ci: + host: postgres-ci.cell-eu0.primary.consul + load_balancing: + discovery: postgres-ci.cell-eu0.replicas.consul + users: + host: postgres-users-primary.consul + load_balancing: + discovery: postgres-users-replicas.eu.consul + routes: + host: postgres-routes-primary.consul + load_balancing: + discovery: postgres-routes-replicas.eu.consul + admin: + host: postgres-admin-primary.consul + load_balancing: + discovery: postgres-admin-replicas.eu.consul +``` + +## Request flows + +1. `gitlab-org` is a top level namespace and lives in `Cell US0` in the `GitLab.com Public` organization +1. `my-company` is a top level namespace and lives in `Cell EU0` in the `my-organization` organization + +### Experience for paying user that is part of `my-organization` + +Such a user will have a default organization set to `/my-organization` and will be +unable to load any global routes outside of this organization. They may load other +projects/namespaces but their MR/Todo/Issue counts at the top of the page will +not be correctly populated in the first iteration. The user will be aware of +this limitation. + +#### Navigates to `/my-company/my-project` while logged in + +1. User is in Europe so DNS resolves to the router in Europe +1. They request `/my-company/my-project` without the router cache, so the router chooses randomly `Cell EU1` +1. The `/cells/learn` is sent to `Cell EU1`, which responds that resource lives on `Cell EU0` +1. `Cell EU0` returns the correct response +1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Cell EU0` + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu1: /api/v4/cells/learn?method=GET&path_info=/my-company/my-project + cell_eu1->>router_eu: {path: "/my-company", cell: "cell_eu0", source: "routable"} + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/my-company/my-project` while not logged in + +1. User is in Europe so DNS resolves to the router in Europe +1. The router does not have `/my-company/*` cached yet so it chooses randomly `Cell EU1` +1. The `/cells/learn` is sent to `Cell EU1`, which responds that resource lives on `Cell EU0` +1. `Cell EU0` redirects them through a login flow +1. User requests `/users/sign_in`, uses random Cell to run `/cells/learn` +1. The `Cell EU1` responds with `cell_0` as a fixed route +1. User after login requests `/my-company/my-project` which is cached and stored in `Cell EU0` +1. `Cell EU0` returns the correct response + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu1: /api/v4/cells/learn?method=GET&path_info=/my-company/my-project + cell_eu1->>router_eu: {path: "/my-company", cell: "cell_eu0", source: "routable"} + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: 302 /users/sign_in?redirect=/my-company/my-project + user->>router_eu: GET /users/sign_in?redirect=/my-company/my-project + router_eu->>cell_eu1: /api/v4/cells/learn?method=GET&path_info=/users/sign_in + cell_eu1->>router_eu: {path: "/users", cell: "cell_eu0", source: "fixed"} + router_eu->>cell_eu0: GET /users/sign_in?redirect=/my-company/my-project + cell_eu0-->>user: <h1>Sign in... + user->>router_eu: POST /users/sign_in?redirect=/my-company/my-project + router_eu->>cell_eu0: POST /users/sign_in?redirect=/my-company/my-project + cell_eu0->>user: 302 /my-company/my-project + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu0: GET /my-company/my-project + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/my-company/my-other-project` after last step + +1. User is in Europe so DNS resolves to the router in Europe +1. The router cache now has `/my-company/* => Cell EU0`, so the router chooses `Cell EU0` +1. `Cell EU0` returns the correct response as well as the cache header again + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_eu1 as Cell EU1 + user->>router_eu: GET /my-company/my-project + router_eu->>cell_eu0: GET /my-company/my-project + cell_eu0->>user: <h1>My Project... +``` + +#### Navigates to `/gitlab-org/gitlab` after last step + +1. User is in Europe so DNS resolves to the router in Europe +1. The router has no cached value for this URL so randomly chooses `Cell EU0` +1. `Cell EU0` redirects the router to `Cell US0` +1. `Cell US0` returns the correct response as well as the cache header again + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + participant cell_us0 as Cell US0 + user->>router_eu: GET /gitlab-org/gitlab + router_eu->>cell_eu0: /api/v4/cells/learn?method=GET&path_info=/gitlab-org/gitlab + cell_eu0->>router_eu: {path: "/gitlab-org", cell: "cell_us0", source: "routable"} + router_eu->>cell_us0: GET /gitlab-org/gitlab + cell_us0->>user: <h1>GitLab.org... +``` + +In this case the user is not on their "default organization" so their TODO +counter will not include their normal todos. We may choose to highlight this in +the UI somewhere. A future iteration may be able to fetch that for them from +their default organization. + +#### Navigates to `/` + +1. User is in Europe so DNS resolves to the router in Europe +1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) +1. The Router choose `Cell EU0` randomly +1. The Rails application knows the users default organization is `/my-organization`, so + it redirects the user to `/organizations/my-organization/-/dashboard` +1. The Router has a cached value for `/organizations/my-organization/*` so it then sends the + request to `POD EU0` +1. `Cell EU0` serves up a new page `/organizations/my-organization/-/dashboard` which is the same + dashboard view we have today but scoped to an organization clearly in the UI +1. The user is (optionally) presented with a message saying that data on this page is only + from their default organization and that they can change their default + organization if it's not right. + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_eu0 as Cell EU0 + user->>router_eu: GET / + router_eu->>cell_eu0: GET / + cell_eu0->>user: 302 /organizations/my-organization/-/dashboard + user->>router: GET /organizations/my-organization/-/dashboard + router->>cell_eu0: GET /organizations/my-organization/-/dashboard + cell_eu0->>user: <h1>My Company Dashboard... X-Gitlab-Cell-Cache={path_prefix:/organizations/my-organization/} +``` + +#### Navigates to `/dashboard` + +As above, they will end up on `/organizations/my-organization/-/dashboard` as +the rails application will already redirect `/` to the dashboard page. + +### Navigates to `/not-my-company/not-my-project` while logged in (but they don't have access since this project/group is private) + +1. User is in Europe so DNS resolves to the router in Europe +1. The router knows that `/not-my-company` lives in `Cell US1` so sends the request to this +1. The user does not have access so `Cell US1` returns 404 + +```mermaid +sequenceDiagram + participant user as User + participant router_eu as Router EU + participant cell_us1 as Cell US1 + user->>router_eu: GET /not-my-company/not-my-project + router_eu->>cell_us1: GET /not-my-company/not-my-project + cell_us1->>user: 404 +``` + +#### Creates a new top level namespace + +The user will be asked which organization they want the namespace to belong to. +If they select `my-organization` then it will end up on the same cell as all +other namespaces in `my-organization`. If they select nothing we default to +`GitLab.com Public` and it is clear to the user that this is isolated from +their existing organization such that they won't be able to see data from both +on a single page. + +### Experience for GitLab team member that is part of `/gitlab-org` + +Such a user is considered a legacy user and has their default organization set to +`GitLab.com Public`. This is a "meta" organization that does not really exist but +the Rails application knows to interpret this organization to mean that they are +allowed to use legacy global functionality like `/dashboard` to see data across +namespaces located on `Cell US0`. The rails backend also knows that the default cell to render any ambiguous +routes like `/dashboard` is `Cell US0`. Lastly the user will be allowed to +navigate to organizations on another cell like `/my-organization` but when they do the +user will see a message indicating that some data may be missing (eg. the +MRs/Issues/Todos) counts. + +#### Navigates to `/gitlab-org/gitlab` while not logged in + +1. User is in the US so DNS resolves to the US router +1. The router knows that `/gitlab-org` lives in `Cell US0` so sends the request + to this cell +1. `Cell US0` serves up the response + +```mermaid +sequenceDiagram + participant user as User + participant router_us as Router US + participant cell_us0 as Cell US0 + user->>router_us: GET /gitlab-org/gitlab + router_us->>cell_us0: GET /gitlab-org/gitlab + cell_us0->>user: <h1>GitLab.org... +``` + +#### Navigates to `/` + +1. User is in US so DNS resolves to the router in US +1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) +1. The Router chooses `Cell US1` randomly +1. The Rails application knows the users default organization is `GitLab.com Public`, so + it redirects the user to `/dashboards` (only legacy users can see + `/dashboard` global view) +1. Router does not have a cache for `/dashboard` route (specifically rails never tells it to cache this route) +1. The Router chooses `Cell US1` randomly +1. The Rails application knows the users default organization is `GitLab.com Public`, so + it allows the user to load `/dashboards` (only legacy users can see + `/dashboard` global view) and redirects to router the legacy cell which is `Cell US0` +1. `Cell US0` serves up the global view dashboard page `/dashboard` which is the same + dashboard view we have today + +```mermaid +sequenceDiagram + participant user as User + participant router_us as Router US + participant cell_us0 as Cell US0 + participant cell_us1 as Cell US1 + user->>router_us: GET / + router_us->>cell_us1: GET / + cell_us1->>user: 302 /dashboard + user->>router_us: GET /dashboard + router_us->>cell_us1: /api/v4/cells/learn?method=GET&path_info=/dashboard + cell_us1->>router_us: {path: "/dashboard", cell: "cell_us0", source: "routable"} + router_us->>cell_us0: GET /dashboard + cell_us0->>user: <h1>Dashboard... +``` + +#### Navigates to `/my-company/my-other-project` while logged in (but they don't have access since this project is private) + +They get a 404. + +### Experience for non-authenticated users + +Flow is similar to logged in users except global routes like `/dashboard` will +redirect to the login page as there is no default organization to choose from. + +### A new customers signs up + +They will be asked if they are already part of an organization or if they'd +like to create one. If they choose neither they end up no the default +`GitLab.com Public` organization. + +### An organization is moved from 1 cell to another + +TODO + +### GraphQL/API requests which don't include the namespace in the URL + +TODO + +### The autocomplete suggestion functionality in the search bar which remembers recent issues/MRs + +TODO + +### Global search + +TODO + +## Administrator + +### Loads `/admin` page + +1. The `/admin` is locked to `Cell US0` +1. Some endpoints of `/admin`, like Projects in Admin are scoped to a Cell + and users needs to choose the correct one in a dropdown, which results in endpoint + like `/admin/cells/cell_0/projects`. + +Admin Area settings in Postgres are all shared across all cells to avoid +divergence but we still make it clear in the URL and UI which cell is serving +the Admin Area page as there is dynamic data being generated from these pages and +the operator may want to view a specific cell. + +## More Technical Problems To Solve + +### Replicating User Sessions Between All Cells + +Today user sessions live in Redis but each cell will have their own Redis instance. We already use a dedicated Redis instance for sessions so we could consider sharing this with all cells like we do with `gitlab_users` PostgreSQL database. But an important consideration will be latency as we would still want to mostly fetch sessions from the same region. + +An alternative might be that user sessions get moved to a JWT payload that encodes all the session data but this has downsides. For example, it is difficult to expire a user session, when their password changes or for other reasons, if the session lives in a JWT controlled by the user. + +### How do we migrate between Cells + +Migrating data between cells will need to factor all data stores: + +1. PostgreSQL +1. Redis Shared State +1. Gitaly +1. Elasticsearch + +### Is it still possible to leak the existence of private groups via a timing attack? + +If you have router in EU, and you know that EU router by default redirects +to EU located Cells, you know their latency (lets assume 10 ms). Now, if your +request is bounced back and redirected to US which has different latency +(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was +returned by US Cell and know that your 404 is in fact 403. + +We may defer this until we actually implement a cell in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. + +One technique to mitigate this risk might be to have the router add a random +delay to any request that returns 404 from a cell. + +## Should runners be shared across all cells? + +We have 2 options and we should decide which is easier: + +1. Decompose runner registration and queuing tables and share them across all + cells. This may have implications for scalability, and we'd need to consider + if this would include group/project runners as this may have scalability + concerns as these are high traffic tables that would need to be shared. +1. Runners are registered per-cell and, we probably have a separate fleet of + runners for every cell or just register the same runners to many cells which + may have implications for queueing + +## How do we guarantee unique ids across all cells for things that cannot conflict? + +This project assumes at least namespaces and projects have unique ids across +all cells as many requests need to be routed based on their ID. Since those +tables are across different databases then guaranteeing a unique ID will +require a new solution. There are likely other tables where unique IDs are +necessary and depending on how we resolve routing for GraphQL and other APIs +and other design goals it may be determined that we want the primary key to be +unique for all tables. diff --git a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md index ebe3c72adfc..3e86d30df1d 100644 --- a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md +++ b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md @@ -1,7 +1,11 @@ --- -stage: none -group: unassigned comments: false +status: ongoing +creation-date: "2022-05-31" +authors: [ "@grzesiek" ] +coach: [ "@ayufan", "@grzesiek" ] +approvers: [ "@jreporter", "@cheryl.li" ] +owning-stage: "~devops::verify" description: 'Pipeline data partitioning design' --- @@ -803,9 +807,11 @@ DRIs: | Role | Who | |---------------------|------------------------------------------------| | Author | Grzegorz Bizon, Principal Engineer | -| Recommender | Kamil Trzciński, Senior Distingiushed Engineer | -| Product Manager | James Heimbuck, Senior Product Manager | -| Engineering Manager | Scott Hampton, Engineering Manager | +| Recommender | Kamil Trzciński, Senior Distinguished Engineer | +| Product Leadership | Jackie Porter, Director of Product Management | +| Engineering Leadership | Caroline Simpson, Engineering Manager / Cheryl Li, Senior Engineering Manager | | Lead Engineer | Marius Bobin, Senior Backend Engineer | +| Senior Engineer | Maxime Orefice, Senior Backend Engineer | +| Senior Engineer | Tianwen Chen, Senior Backend Engineer | <!-- vale gitlab.Spelling = YES --> diff --git a/doc/architecture/blueprints/ci_pipeline_components/dev_workflow.md b/doc/architecture/blueprints/ci_pipeline_components/dev_workflow.md new file mode 100644 index 00000000000..266ea13c275 --- /dev/null +++ b/doc/architecture/blueprints/ci_pipeline_components/dev_workflow.md @@ -0,0 +1,155 @@ +--- +stage: verify +group: pipeline authoring +comments: false +description: 'Development workflow for a components repository' +--- + +# Development workflow for a components repository + +## Summary + +This page describes the process of creating a components repository. +It describes all the necessary steps, from the creation of the project to having new releases displayed in the +catalog page. + +## 1. Create a new project + +First, create a new project and add a `README.md` file, which is a planned future +requirement for a repository to become a catalog resource. + +## 2. Create a component inside the repository + +If you intend to have only one component in the repository, you can define it in the root directory. +Otherwise, create a directory for the component. +For more information, see the [directory structure of a components repository](index.md#structure-of-a-components-repository). + +This example defines a single component in the root directory. + +Create a `template.yml` file that contains the configuration we want to provide as a component: + +```yaml +spec: + inputs: + stage: + default: test +--- +.component-default-job: + image: busybox + stage: $[[ inputs.stage ]] + +component-job-1: + extends: .component-default-job + script: echo job 1 + +component-job-2: + extends: .component-default-job + script: echo job 2 +``` + +The example component configuration above adds two jobs, `component-job-1` and `component-job-2`, to a pipeline. + +## 3. Test changes in CI + +To test any changes pushed to our component, we create a `.gitlab-ci.yml` in the root directory: + +```yaml +## +# This configuration expects an access token with read-only access to the API +# to be saved as in a masked CI/CD variable named 'API_TOKEN' + +include: + # Leverage predefined variables to refer to the current project and SHA + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + +stages: [test, release] + +# Expect all `component-job-*` jobs are added +ensure-jobs-added: + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job-"))) | length'` + if [ "$count" != "2" ]; then + exit 1 + fi + +# Ensure that a project description exists, because it will be important to display +# the resource in the catalog. +check-description: + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID" + desc=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq '.description'` + if [ "$desc" = "null" ]; then + echo "Description not set. Please set a projet description" + exit 1 + else + echo "Description set" + fi + +# Ensure that a `README.md` exists in the root directory as it represents the +# documentation for the whole components repository. +check-readme: + image: busybox + script: ls README.md || (echo "Please add a README.md file" && exit 1) + +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` + +This pipeline contains examples of several tasks: + +- Use the component to ensure that the final configuration uses valid syntax. + This also ensures that the minimal requirements for the component to work are in place, + like inputs and secrets. +- Test that the created pipeline has the expected characteristics. + For example, ensure the `component-job-*` jobs are added to the pipeline. + - We call the [pipeline API endpoint](../../../api/pipelines.md#get-a-single-pipeline) with `curl` + and parse the data via `jq`. + - With this technique users could check things like ensuring certain jobs were included, + the job has the right properties set, or the log contains the expected output. +- Ensure that the project description is set. +- Ensure that the repository contains a `README.md` file. +- Create a [release automatically](../../../ci/yaml/index.md#release). When a tag is created and follows specific regex, create a release + after all previous checks pass. + +## 4. Run a pipeline + +Now run a new pipeline for the `main` branch, by pushing a change or manually running a pipeline: + + + +## 5. Create a tag + +As the pipeline for `main` is green, we can now [create our first tag](../../../user/project/repository/tags/index.md#create-a-tag): `v1.0`. + +As soon as the `v1.0` tag is created, we see a tag pipeline start. +This time the pipeline also has a `create-release` job in the `release` stage: + + + +When the `create-release` job finishes we should see the new release available in the **Releases** menu: + + + +## 6. Publish the repository to the catalog + +To ensure that both the components repository and the new release is visible in the CI Catalog, +we need to publish it. + +Publishing a components repository makes it a catalog resource. + +The API endpoint for this action is under development. +For more details read the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387065). diff --git a/doc/architecture/blueprints/ci_pipeline_components/img/new_release.png b/doc/architecture/blueprints/ci_pipeline_components/img/new_release.png Binary files differnew file mode 100644 index 00000000000..eed5c55d5e3 --- /dev/null +++ b/doc/architecture/blueprints/ci_pipeline_components/img/new_release.png diff --git a/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_main.png b/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_main.png Binary files differnew file mode 100644 index 00000000000..8b03b96ba7e --- /dev/null +++ b/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_main.png diff --git a/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_tag.png b/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_tag.png Binary files differnew file mode 100644 index 00000000000..d0814a479ae --- /dev/null +++ b/doc/architecture/blueprints/ci_pipeline_components/img/pipeline_tag.png diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index b1aee7c4217..c1094bc4290 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -92,9 +92,12 @@ This section defines some terms that are used throughout this document. With the identifying abstract concepts and are subject to changes as we refine the design by discovering new insights. - **Component** Is the reusable unit of pipeline configuration. -- **Project** Is the GitLab project attached to a repository. A project can contain multiple components. -- **Catalog** is the collection of projects that are set to contain components. -- **Version** is the release name of a tag in the project, which allows components to be pinned to a specific revision. +- **Components repository** represents a collection of CI components stored in the same project. +- **Project** is the GitLab project attached to a single components repository. +- **Catalog** is a collection of resources like components repositories. +- **Catalog resource** is the single item displayed in the catalog. A components repository is a catalog resource. +- **Version** is a specific revision of catalog resource. It maps to the released tag in the project, + which allows components to be pinned to a specific revision. ## Definition of pipeline component @@ -202,7 +205,6 @@ A component YAML file: - Should be **validated statically** (for example: using JSON schema validators). ```yaml ---- spec: inputs: website: @@ -217,18 +219,13 @@ spec: # content of the component ``` -Components that are released in the catalog must have a `README.md` file at the root directory of the repository. -The `README.md` represents the documentation for the specific component, hence it's recommended -even when not releasing versions in the catalog. - ### The component version The version of the component can be (in order of highest priority first): 1. A commit SHA - For example: `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` -1. A released tag - For example: `gitlab.com/gitlab-org/dast@1.0` +1. A tag - For example: `gitlab.com/gitlab-org/dast@1.0` 1. A special moving target version that points to the most recent released tag - For example: `gitlab.com/gitlab-org/dast@~latest` -1. An unreleased tag - For example: `gitlab.com/gitlab-org/dast@rc-1.0` 1. A branch name - For example: `gitlab.com/gitlab-org/dast@master` If a tag and branch exist with the same name, the tag takes precedence over the branch. @@ -237,26 +234,31 @@ takes precedence over the tag. As we want to be able to reference any revisions (even those not released), a component must be defined in a Git repository. -NOTE: When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches the commit SHA of the current pipeline context. -## Components project +## Components repository + +A components repository is a GitLab project/repository that exclusively hosts one or more pipeline components. -A components project is a GitLab project/repository that exclusively hosts one or more pipeline components. +A components repository can be a catalog resource. For a components repository it's highly recommended to set +an appropriate avatar and project description to improve discoverability in the catalog. -For components projects it's highly recommended to set an appropriate avatar and project description -to improve discoverability in the catalog. +Components repositories that are released in the catalog must have a `README.md` file at the root directory of the repository. +The `README.md` represents the documentation of the components repository, hence it's recommended +even when not listing the repository in the catalog. -### Structure of a components project +### Structure of a components repository -A project can host one or more components depending on whether the author wants to define a single component -per project or include multiple cohesive components under the same project. +A components repository can host one or more components. The author can decide whether to define a single component +per repository or include multiple cohesive components in the same repository. -Let's imagine we are developing a component that runs RSpec tests for a Rails app. We create a component project +A components repository is identified by the project full path. + +Let's imagine we are developing a component that runs RSpec tests for a Rails app. We create a project called `myorg/rails-rspec`. -The following directory structure would support 1 component per project: +The following directory structure would support 1 component per repository: ```plaintext . @@ -267,10 +269,10 @@ The following directory structure would support 1 component per project: The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. -The component is now identified by the path `gitlab.com/myorg/rails-rspec` and we expect a `template.yml` file -and `README.md` located in the root directory of the repository. +The component is now identified by the path `gitlab.com/myorg/rails-rspec` which also maps to the +project path. We expect a `template.yml` file and `README.md` to be located in the root directory of the repository. -The following directory structure would support multiple components per project: +The following directory structure would support multiple components per repository: ```plaintext . @@ -319,7 +321,6 @@ This limitation encourages cohesion at project level and keeps complexity low. If the component takes any input parameters they must be specified according to the following schema: ```yaml ---- spec: inputs: website: # by default all declared inputs are mandatory. @@ -363,7 +364,6 @@ Input parameters are validated as soon as possible: 1. Interpolate input parameters inside the component's content. ```yaml ---- spec: inputs: environment: @@ -452,7 +452,6 @@ include: Then the configuration being included must specify the inputs by defining a specification section in the YAML: ```yaml ---- spec: inputs: foo: @@ -496,7 +495,6 @@ deploy-app: To solve the problem of `Run Pipeline` UI form we could fully leverage the `inputs` specifications: ```yaml ---- spec: inputs: concurrency: @@ -516,35 +514,114 @@ spec: # rest of the pipeline config ``` -### Limits +## CI Catalog -Any MVC that exposes a feature should be added with limitations from the beginning. -It's safer to add new features with restrictions than trying to limit a feature after it's being used. -We can always soften the restrictions later depending on user demand. +The CI Catalog is an index of resources that users can leverage in CI/CD. It initially +contains a list of components repositories that users can discover and use in their pipelines. -Some limits we could consider adding: +In the future, the Catalog could contain also other types of resources (for example: +integrations, project templates, etc.). -- number of components that a single project can contain/export -- number of imports that a `.gitlab-ci.yml` file can use -- number of imports that a component can declare/use -- max level of nested imports -- max length of the exported component name +To list a components repository in the Catalog we need to mark the project as being a +catalog resource. We do that initially with an API endpoint, similar to changing a project setting. + +Once a project is marked as a "catalog resource" it can be displayed in the Catalog. + +We could create a database record when the API endpoint is used and remove the record when +the same is disabled/removed. + +## Catalog resource + +Upon publishing, a catalog resource should have at least following attributes: + +- `path`: to be uniquely identified. +- `name`: for components repository this could be the project name. +- `documentation`: we would use the `README.md` file which would be mandatory. +- `versions`: one or more releases of the resource. + +Other properties of a catalog resource: + +- `description`: for components repository this could be the project description. +- `avatar image`: we could use the project avatar. +- indicators of popularity (stars, forks). +- categorization: user should select a category and or define search tags + +As soon as a components repository is marked as being a "catalog resource" +we should be seeing the resource listed in the Catalog. + +Initially for the resource, the project may not have any released tags. +Users would be able to use the components repository by specifying a branch name or +commit SHA for the version. However, these types of version qualifiers should not +be listed in the catalog resource's page for various reasons: + +- The list of branches and tags can get very large. +- Branches and tags may not be meaningful for the end-user. +- Branches and tags don't communicate versioning thoroughly. + +## Releasing new resource versions to the Catalog -## Publishing components +The versions that should be displayed for the resource should be the project [releases](../../../user/project/releases/index.md). +Creating project releases is an official act of versioning a resource. -Users will be able to publish CI Components into a CI Catalog. This can happen -in a CI pipeline job, similarly to how software is being deployed following -Continuous Delivery principles. This will allow us to guardrail the quality of -components being deployed. To ensure that the CI Components meet quality -standards users will be able to test them before publishing new versions in the +A resource page would have: + +- The latest release in evidence (for example: the default version). +- The ability to inspect and use past releases of the resource. +- The documentation represented by the `README.md`. + +Users should be able to release new versions of the resource in a CI pipeline job, +similar to how software is being deployed following Continuous Delivery principles. + +To ensure that the components repository and the including components +meet quality standards, users can test them before releasing new versions in the CI Catalog. -Once a project containing components gets published we will index components' +Some examples of checks we can run during the release of a new resource version: + +- Ensure the project contains a `README.md` in the root directory. +- Ensure the project description exists. +- If an index of available components is present for a components repository, ensure each + component has valid YAML. + +Once a new release for the project gets created we index the resource's metadata. We want to initially index as much metadata as possible, to gain more flexibility in how we design CI Catalog's main page. We don't want to be -constrained by the lack of data available to properly visualize CI Components -in CI Catalog. In order to do that, we may need to find all components that are -being published, read their `spec` metadata and index what we find there. +constrained by the lack of data available to properly visualize resources in +the CI Catalog. To do that, we may need to find all resources that are +being released and index their data and metadata. +For example: index the content of `spec:` section for CI components. + +See an [example of development workflow](dev_workflow.md) for a components repository. + +## Note about future resource types + +In the future, to support multiple types of resources in the Catalog we could +require a file `catalog-resource.yml` to be defined in the root directory of the project: + +```yaml +name: DAST +description: Scan a web endpoint to find vulnerabilities +category: security +tags: [dynamic analysis, security scanner] +type: components_repository +``` + +This file could also be used for indexing metadata about the content of the resource. +For example, users could list the components in the repository and we can index +further data for search purpose: + +```yaml +name: DAST +description: Scan a web endpoint to find vulnerabilities +category: security +tags: [dynamic analysis, security scanner] +type: components_repository +metadata: + components: + - all-scans + - scan-x + - scan-y +``` ## Implementation guidelines @@ -585,6 +662,20 @@ being published, read their `spec` metadata and index what we find there. components from GitLab.com or from repository exports. - Iterate on feedback. +## Limits + +Any MVC that exposes a feature should be added with limitations from the beginning. +It's safer to add new features with restrictions than trying to limit a feature after it's being used. +We can always soften the restrictions later depending on user demand. + +Some limits we could consider adding: + +- number of components that a single project can contain/export +- number of imports that a `.gitlab-ci.yml` file can use +- number of imports that a component can declare/use +- max level of nested imports +- max length of the exported component name + ## Who Proposal: diff --git a/doc/architecture/blueprints/clickhouse_usage/index.md b/doc/architecture/blueprints/clickhouse_usage/index.md new file mode 100644 index 00000000000..2781ea15a55 --- /dev/null +++ b/doc/architecture/blueprints/clickhouse_usage/index.md @@ -0,0 +1,52 @@ +--- +status: proposed +creation-date: "2023-02-02" +authors: [ "@nhxnguyen" ] +coach: "@grzesiek" +approvers: [ "@dorrino", "@nhxnguyen" ] +owning-stage: "~devops::data_stores" +participating-stages: ["~section::ops", "~section::dev"] +--- + +# ClickHouse Usage at GitLab + +## Summary + +[ClickHouse](https://clickhouse.com/) is an open-source column-oriented database management system. It can efficiently filter, aggregate, and sum across large numbers of rows. In FY23, GitLab selected ClickHouse as its standard data store for features with big data and insert-heavy requirements such as Observability and Analytics. This blueprint is a product of the [ClickHouse working group](https://about.gitlab.com/company/team/structure/working-groups/clickhouse-datastore/). It serves as a high-level blueprint to ClickHouse adoption at GitLab and references other blueprints addressing specific ClickHouse-related technical challenges. + +## Motivation + +In FY23-Q2, the Monitor:Observability team developed and shipped a [ClickHouse data platform](https://gitlab.com/groups/gitlab-org/-/epics/7772) to store and query data for Error Tracking and other observability features. Other teams have also begun to incorporate ClickHouse into their current or planned architectures. Given the growing interest in ClickHouse across product development teams, it is important to have a cohesive strategy for developing features using ClickHouse. This will allow teams to more efficiently leverage ClickHouse and ensure that we can maintain and support this functionality effectively for SaaS and self-managed customers. + +### Goals + +As ClickHouse has already been selected for use at GitLab, our main goal now is to ensure successful adoption of ClickHouse across GitLab. It is helpful to break down this goal according to the different phases of the product development workflow. + +1. Plan: Make it easy for development teams to understand if ClickHouse is the right fit for their feature. +1. Develop and Test: Give teams the best practices and frameworks to develop ClickHouse-backed features. +1. Launch: Support ClickHouse-backed features for SaaS and self-managed. +1. Improve: Successfully scale our usage of ClickHouse. + +### Non-Goals + +## Proposals + +The following are links to proposals in the form of blueprints that address technical challenges to using ClickHouse across a wide variety of features. + +1. Scalable data ingestion pipeline. + - How do we ingest large volumes of data from GitLab into ClickHouse either directly or by replicating existing data? +1. Supporting ClickHouse for self-managed installations. + - For which use-cases and scales does it make sense to run ClickHouse for self-managed and what are the associated costs? + - How can we best support self-managed installation of ClickHouse for different types/sizes of environments? + - Consider using the [Opstrace ClickHouse operator](https://gitlab.com/gitlab-org/opstrace/opstrace/-/tree/main/clickhouse-operator) as the basis for a canonical distribution. + - Consider exposing Clickhouse backend as [GitLab Plus](https://gitlab.com/groups/gitlab-org/-/epics/308) to combine benefits of using self-managed instance and GitLab-managed database. + - Should we develop abstractions for querying and data ingestion to avoid requiring ClickHouse for small-scale installations? +1. Abstraction layer for features to leverage both ClickHouse or PostreSQL. + - What are the benefits and tradeoffs? For example, how would this impact our automated migration and query testing? +1. Security recommendations and secure defaults for ClickHouse usage. + +Note that we are still formulating proposals and will update the blueprint accordingly. + +## Best Practices + +Best practices and guidelines for developing performant and scalable features using ClickHouse are located in the [ClickHouse developer documentation](../../../development/database/clickhouse/index.md). diff --git a/doc/architecture/blueprints/search/code_search_with_zoekt.md b/doc/architecture/blueprints/code_search_with_zoekt/index.md index d0d347f1ff4..ca74460b98a 100644 --- a/doc/architecture/blueprints/search/code_search_with_zoekt.md +++ b/doc/architecture/blueprints/code_search_with_zoekt/index.md @@ -57,7 +57,7 @@ customers that wish to participate in the trial. The main goals of this integration will be to implement the following highly requested improvements to code search: -1. [Exact match (substring match) code searches in Advanced Search](https://gitlab.com/gitlab-org/gitlab/-/issues/325234) +1. [Exact match (substring match) code searches in advanced search](https://gitlab.com/gitlab-org/gitlab/-/issues/325234) 1. [Support regular expressions with Advanced Global Search](https://gitlab.com/gitlab-org/gitlab/-/issues/4175) 1. [Support multiple line matches in the same file](https://gitlab.com/gitlab-org/gitlab/-/issues/668) diff --git a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md index 7fecbd1de71..12254fa7920 100644 --- a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md +++ b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md @@ -340,7 +340,7 @@ What was done? spec.add_dependency 'graphql-docs' spec.add_dependency 'grape' end - ``` + ``` 1. Move routes to the `engines/web_engine/config/routes.rb` file @@ -380,59 +380,59 @@ What was done? 1. Configure GitLab when to load the engine. - In GitLab `config/engines.rb`, we can configure when do we want to load our engines by relying on our `Gitlab::Runtime` + In GitLab `config/engines.rb`, we can configure when do we want to load our engines by relying on our `Gitlab::Runtime` - ```ruby - # config/engines.rb - # Load only in case we are running web_server or rails console - if Gitlab::Runtime.puma? || Gitlab::Runtime.console? - require 'web_engine' - end - ``` + ```ruby + # config/engines.rb + # Load only in case we are running web_server or rails console + if Gitlab::Runtime.puma? || Gitlab::Runtime.console? + require 'web_engine' + end + ``` 1. Configure Engine - Our Engine inherits from the `Rails::Engine` class. This way this gem notifies Rails that - there's an engine at the specified path so it will correctly mount the engine inside - the application, performing tasks such as adding the app directory of the engine to - the load path for models, mailers, controllers, and views. - A file at `lib/web_engine/engine.rb`, is identical in function to a standard Rails - application's `config/application.rb` file. This way engines can access a configuration - object which contains configuration shared by all railties and the application. - Additionally, each engine can access `autoload_paths`, `eager_load_paths`, and `autoload_once_paths` - settings which are scoped to that engine. - - ```ruby - module WebEngine - class Engine < ::Rails::Engine - config.eager_load_paths.push(*%W[#{config.root}/lib - #{config.root}/app/graphql/resolvers/concerns - #{config.root}/app/graphql/mutations/concerns - #{config.root}/app/graphql/types/concerns]) - - if Gitlab.ee? - ee_paths = config.eager_load_paths.each_with_object([]) do |path, memo| - ee_path = config.root - .join('ee', Pathname.new(path).relative_path_from(config.root)) - memo << ee_path.to_s - end - # Eager load should load CE first - config.eager_load_paths.push(*ee_paths) - end - end - end - ``` + Our Engine inherits from the `Rails::Engine` class. This way this gem notifies Rails that + there's an engine at the specified path so it will correctly mount the engine inside + the application, performing tasks such as adding the app directory of the engine to + the load path for models, mailers, controllers, and views. + A file at `lib/web_engine/engine.rb`, is identical in function to a standard Rails + application's `config/application.rb` file. This way engines can access a configuration + object which contains configuration shared by all railties and the application. + Additionally, each engine can access `autoload_paths`, `eager_load_paths`, and `autoload_once_paths` + settings which are scoped to that engine. + + ```ruby + module WebEngine + class Engine < ::Rails::Engine + config.eager_load_paths.push(*%W[#{config.root}/lib + #{config.root}/app/graphql/resolvers/concerns + #{config.root}/app/graphql/mutations/concerns + #{config.root}/app/graphql/types/concerns]) + + if Gitlab.ee? + ee_paths = config.eager_load_paths.each_with_object([]) do |path, memo| + ee_path = config.root + .join('ee', Pathname.new(path).relative_path_from(config.root)) + memo << ee_path.to_s + end + # Eager load should load CE first + config.eager_load_paths.push(*ee_paths) + end + end + end + ``` 1. Testing - We adapted CI to test `engines/web_engine/` as a self-sufficient component of stack. + We adapted CI to test `engines/web_engine/` as a self-sufficient component of stack. - - We moved `spec` as-is files to the `engines/web_engine/spec` folder - - We moved `ee/spec` as-is files to the `engines/web_engine/ee/spec` folder - - We control specs from main application using environment variable `TEST_WEB_ENGINE` - - We added new CI job that will run `engines/web_engine/spec` tests separately using `TEST_WEB_ENGINE` environment variable. - - We added new CI job that will run `engines/web_engine/ee/spec` tests separately using `TEST_WEB_ENGINE` environment variable. - - We are running all white box frontend tests with `TEST_WEB_ENGINE=true` + - We moved `spec` as-is files to the `engines/web_engine/spec` folder + - We moved `ee/spec` as-is files to the `engines/web_engine/ee/spec` folder + - We control specs from main application using environment variable `TEST_WEB_ENGINE` + - We added new CI job that will run `engines/web_engine/spec` tests separately using `TEST_WEB_ENGINE` environment variable. + - We added new CI job that will run `engines/web_engine/ee/spec` tests separately using `TEST_WEB_ENGINE` environment variable. + - We are running all white box frontend tests with `TEST_WEB_ENGINE=true` #### Results diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index 97853075607..88034f60caf 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -4,7 +4,8 @@ creation-date: "2021-02-07" authors: [ "@alexpooley", "@ifarkas" ] coach: "@grzesiek" approvers: [ "@m_gill", "@mushakov" ] -owning-stage: "~devops::plan" +author-stage: "~devops::plan" +owning-stage: "~devops::data_stores" participating-stages: [] --- @@ -147,4 +148,4 @@ The initial iteration will provide a framework to house features under `Namespac ## Related topics -- [Workspace developer documentation](../../../development/workspace/index.md) +- [Organization developer documentation](../../../development/organization/index.md) diff --git a/doc/architecture/blueprints/gitlab_agent_deployments/index.md b/doc/architecture/blueprints/gitlab_agent_deployments/index.md index 96e361d7531..0e5b5f0b248 100644 --- a/doc/architecture/blueprints/gitlab_agent_deployments/index.md +++ b/doc/architecture/blueprints/gitlab_agent_deployments/index.md @@ -374,6 +374,8 @@ Here is an example of GraphQL query: lastDeployment(status: SUCCESS) { agent { id + name + project kubernetesNamespace } } diff --git a/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md index c5bd2440b0c..82d1596bc90 100644 --- a/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md +++ b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md @@ -93,9 +93,9 @@ Additionally, since we intend to ingest data via Prometheus `remote_write` API, We also need to make sure to avoid writing a lot of small writes into Clickhouse, therefore it’d be prudent to batch data before writing it into Clickhouse. -We must also make sure ingestion remains decoupled with `Storage` so as to reduce undue dependence on a given storage implementation. While we do intend to use Clickhouse as our backing storage for any foreseeable future, this ensures we do not tie ourselves in into Clickhouse too much should future business requirements warrant the usage of a different backend/technology. A good way to implement this in Golang would be our implementations adhering to a standard interface, the following for example: +We must also make sure ingestion remains decoupled with `Storage` so as to reduce undue dependence on a given storage implementation. While we do intend to use Clickhouse as our backing storage for any foreseeable future, this ensures we do not tie ourselves in into Clickhouse too much should future business requirements warrant the usage of a different backend/technology. A good way to implement this in Go would be our implementations adhering to a standard interface, the following for example: -```golang +```go type Storage interface { Read( ctx context.Context, diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md index 4a8eeaf86a9..6718705a7c8 100644 --- a/doc/architecture/blueprints/object_storage/index.md +++ b/doc/architecture/blueprints/object_storage/index.md @@ -53,7 +53,7 @@ This has led to increased complexity across the board, from development [we no longer recommend](../../../administration/nfs.md) to our users and is no longer in use on GitLab.com. - Understanding all the moving parts and the flow is extremely - complicated: we have CarrierWave, Fog, Golang S3/Azure SDKs, all + complicated: we have CarrierWave, Fog, Go S3/Azure SDKs, all being used, and that complicates testing as well. - Fog and CarrierWave are not maintained to the level of the native SDKs (for example, AWS S3 SDK), so we have to maintain or monkey diff --git a/doc/architecture/blueprints/pods/images/term-cluster.png b/doc/architecture/blueprints/pods/images/term-cluster.png Binary files differdeleted file mode 100644 index 87e4d631551..00000000000 --- a/doc/architecture/blueprints/pods/images/term-cluster.png +++ /dev/null diff --git a/doc/architecture/blueprints/pods/images/term-organization.png b/doc/architecture/blueprints/pods/images/term-organization.png Binary files differdeleted file mode 100644 index 4c82c62b8f4..00000000000 --- a/doc/architecture/blueprints/pods/images/term-organization.png +++ /dev/null diff --git a/doc/architecture/blueprints/pods/images/term-pod.png b/doc/architecture/blueprints/pods/images/term-pod.png Binary files differdeleted file mode 100644 index d8f79df2f29..00000000000 --- a/doc/architecture/blueprints/pods/images/term-pod.png +++ /dev/null diff --git a/doc/architecture/blueprints/pods/index.md b/doc/architecture/blueprints/pods/index.md index 077303be30f..5c15f880a54 100644 --- a/doc/architecture/blueprints/pods/index.md +++ b/doc/architecture/blueprints/pods/index.md @@ -1,356 +1,11 @@ --- -status: accepted -creation-date: "2022-09-07" -authors: [ "@ayufan", "@fzimmer", "@DylanGriffith" ] -coach: "@ayufan" -approvers: [ "@fzimmer" ] -owning-stage: "~devops::enablement" -participating-stages: [] +redirect_to: '../cells/index.md' +remove_date: '2023-06-13' --- -# Pods +This document was moved to [another location](../cells/index.md). -This document is a work-in-progress and represents a very early state of the Pods design. Significant aspects are not documented, though we expect to add them in the future. - -## Summary - -Pods is a new architecture for our Software as a Service platform that is horizontally-scalable, resilient, and provides a more consistent user experience. It may also provide additional features in the future, such as data residency control (regions) and federated features. - -## Terminology - -We use the following terms to describe components and properties of the Pods architecture. - -### Pod - -A Pod is a set of infrastructure components that contains multiple top-level namespaces that belong to different organizations. The components include both datastores (PostgreSQL, Redis etc.) and stateless services (web etc.). The infrastructure components provided within a Pod are shared among organizations and their top-level namespaces but not shared with other Pods. This isolation of infrastructure components means that Pods are independent from each other. - - - -#### Pod properties - -- Each pod is independent from the others -- Infrastructure components are shared by organizations and their top-level namespaces within a Pod -- More Pods can be provisioned to provide horizontal scalability -- A failing Pod does not lead to failure of other Pods -- Noisy neighbor effects are limited to within a Pod -- Pods are not visible to organizations; it is an implementation detail -- Pods may be located in different geographical regions (for example, EU, US, JP, UK) - -Discouraged synonyms: GitLab instance, cluster, shard - -### Cluster - -A cluster is a collection of Pods. - - - -#### Cluster properties - -- A cluster holds cluster-wide metadata, for example Users, Routes, Settings. - -Discouraged synonyms: whale - -### Organizations - -GitLab references [Organizations in the initial set up](../../../topics/set_up_organization.md) and users can add a (free text) organization to their profile. There is no Organization entity established in the GitLab codebase. - -As part of delivering Pods, we propose the introduction of an `organization` entity. Organizations would represent billable entities or customers. - -Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). - -Organizations work under the following assumptions: - -1. Users care about what happens within their organizations. -1. Features need to work within an organization. -1. Only few features need to work across organizations. -1. Users understand that the majority of pages they view are only scoped to a single organization at a time. -1. Organizations are located on a single pod. - - - -#### Organization properties - -- Top-level namespaces belong to organizations -- Users can be members of different organizations -- Organizations are isolated from each other by default meaning that cross-namespace features will only work for namespaces that exist within a single organization -- User namespaces must not belong to an organization - -Discouraged synonyms: Billable entities, customers - -### Top-Level namespace - -A top-level namespace is the logical object container in the code that represents all groups, subgroups and projects that belong to an organization. - -A top-level namespace is the root of nested collection namespaces and projects. The namespace and its related entities form a tree-like hierarchy: Namespaces are the nodes of the tree, projects are the leaves. - -Example: - -`https://gitlab.com/gitlab-org/gitlab/`: - -- `gitlab-org` is a `top-level namespace`; the root for all groups and projects of an organization -- `gitlab` is a `project`; a project of the organization. - -Top-level namespaces may [be replaced by workspaces](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals). This proposal only uses the term top-level namespaces as the workspace definition is ongoing. - -Discouraged synonyms: Root-level namespace - - - -#### Top-level namespace properties - -- Top-level namespaces belonging to an organization are located on the same Pod -- Top-level namespaces can interact with other top-level namespaces that belong to the same organization - -### Users - -Users are available globally and not restricted to a single Pod. Users can be members of many different organizations with varying permissions. Inside organizations, users can create multiple top-level namespaces. User activity is not limited to a single organization but their contributions (for example TODOs) are only aggregated within an organization. This avoids the need for aggregating across pods. - -#### User properties - -- Users are shared globally across all Pods -- Users can create multiple top-level namespaces -- Users can be a member of multiple top-level namespaces -- Users can be a member of multiple organizations -- Users can administer organizations -- User activity is aggregated in an organization -- Every user has one personal namespace - -## Goals - -### Scalability - -The main goal of this new shared-infrastructure architecture is to provide additional scalability for our SaaS Platform. GitLab.com is largely monolithic and we have estimated (internal) that the current architecture has scalability limitations, even when database partitioning and decomposition are taken into account. - -Pods provide a horizontally scalable solution because additional Pods can be created based on demand. Pods can be provisioned and tuned as needed for optimal scalability. - -### Increased availability - -A major challenge for shared-infrastructure architectures is a lack of isolation between top-level namespaces. This can lead to noisy neighbor effects. A organization's behavior inside a top-level namespace can impact all other organizations. This is highly undesirable. Pods provide isolation at the pod level. A group of organizations is fully isolated from other organizations located on a different Pod. This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of shared infrastructure. - -Additionally, Pods provide a way to implement disaster recovery capabilities. Entire Pods may be replicated to read-only standbys with automatic failover capabilities. - -### A consistent experience - -Organizations should have the same user experience on our SaaS platform as they do on a self-managed GitLab instance. - -### Regions - -GitLab.com is only hosted within the United States of America. Organizations located in other regions have voiced demand for local SaaS offerings. Pods provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Pods may be deployed within different geographies. Depending on which of the organization's data is located outside a Pod, this may solve data residency and compliance problems. - -## Market segment - -Pods would provide a solution for organizations in the small to medium business (up to 100 users) and the mid-market segment (up to 2000 users). -(See [segmentation definitions](https://about.gitlab.com/handbook/sales/field-operations/gtm-resources/#segmentation).) -Larger organizations may benefit substantially from [GitLab Dedicated](../../../subscriptions/gitlab_dedicated/index.md). - -At this moment, GitLab.com has "social-network"-like capabilities that may not fit well into a more isolated organization model. Removing those features, however, possesses some challenges: - -1. How will existing `gitlab-org` contributors contribute to the namespace?? -1. How do we move existing top-level namespaces into the new model (effectively breaking their social features)? - -We should evaluate if the SMB and mid market segment is interested in these features, or if not having them is acceptable in most cases. - -### Self-managed - -For reasons of consistency, it is expected that self-managed instances will -adopt the pods architecture as well. To expand, self-managed instances can -continue with just a single Pod while supporting the option of adding additional -Pods. Organizations, and possible User decomposition will also be adopted for -self-managed instances. - -## High-level architecture problems to solve - -A number of technical issues need to be resolved to implement Pods (in no particular order). This section will be expanded. - -1. How are users of an organization routed to the correct Pod? -1. How do users authenticate? -1. How are Pods rebalanced? -1. How are Pods provisioned? -1. How can Pods implement disaster recovery capabilities? - -## Cross-section impact - -Pods is a fundamental architecture change that impacts other sections and stages. This section summarizes and links to other groups that may be impacted and highlights potential conflicts that need to be resolved. The Pods group is not responsible for achieving the goals of other groups but we want to ensure that dependencies are resolved. - -### Summary - -Based on discussions with other groups the net impact of introducing Pods and a new entity called organizations is mostly neutral. It may slow down development in some areas. We did not discover major blockers for other teams. - -1. We need to resolve naming conflicts (proposal is TBD) -1. Pods requires introducing Organizations. Organizations are a new entity **above** top-level groups. Because this is a new entity, it may impact the ability to consolidate settings for Group Workspace and influence their decision on [how to approach introducing a workspace](https://gitlab.com/gitlab-org/gitlab/-/issues/376285#approach-2-workspace-is-built-on-top-of-top-level-groups) -1. Organizations may make it slightly easier for Fulfillment to realize their billing plans. - -### Impact on Group Manage Workspace - -We synced with the Workspace PM and Designer ([recording](https://youtu.be/b5Opn9cFWFk)) and discussed the similarities and differences between the Pods and Workspace proposal ([presentation](https://docs.google.com/presentation/d/1FsUi22Up15b_tu6p2m-yLML3hCZ3rgrZrmzJAxUsNmU/edit?usp=sharing)). - -#### Goals of Group Manage Workspace - -As defined in the [workspace documentation](../../../user/workspace/index.md): - -1. Create an entity to manage everything you do as a GitLab administrator, including: - 1. Defining and applying settings to all of your groups, subgroups, and projects. - 1. Aggregating data from all your groups, subgroups, and projects. -1. Reach feature parity between SaaS and self-managed installations, with all Admin Area settings moving to groups (?). Hardware controls remain on the instance level. - -The [workspace roadmap outlines](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals) the current goals in detail. - -#### Potential conflicts with Pods - -- Workspace and Organization are different terms for the same entity. Both define a new entity as the primary organizational object for groups and projects. This is mainly a semantic difference and **we need to decide on a name** following [user research to decide if workspace](https://gitlab.com/gitlab-org/ux-research/-/issues/2147). This is also driven by the fact that the Remote Development team is looking at better names and [are considering the term Workspace as well](https://gitlab.com/gitlab-com/Product/-/issues/4812). -- We will only introduce one entity -- Group workspace highlighted the need to further validate the key assumption that users only care about what happens within their organization. - -### Impact on Fulfillment - -We synced with Fulfillment ([recording](https://youtu.be/FkQF3uF7vTY)) to discuss how Pods would impact them. Fulfillment is supportive of an entity above top-level namespaces. Their perspective is outline in [!5639](https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/5639/diffs). - -#### Goals of Fulfillment - -- Fulfillment has a longstanding plan to move billing from the top-level namespace to a level above. This would mean that a license applies for an organization and all its top-level namespaces. -- Fulfillment uses Zuora for billing and would like to have a 1-to-1 relationship between an organization and their Zuora entity called BillingAccount. They want to move away from tying a license to a single user. -- If a customer needs multiple organizations, the corresponding BillingAccounts can be rolled up into a consolidated billing account (similar to [AWS consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html)) -- Ideally, a self-managed instance has a single Organization by default, which should be enough for most customers. -- Fulfillment prefers only one additional entity. - -A rough representation of this is: - - - -#### Potential conflicts with Pods - -- There are no known conflicts between Fulfillment's plans and Pods - -## Iteration plan - -We can't ship the entire Pods architecture in one go - it is too large. Instead, we are adopting an iteration plan that provides value along the way. - -1. Introduce organizations -1. Migrate existing top-level namespaces to organizations -1. Create new organizations on `pod_0` -1. Migrate existing organizations from `pod_0` to `pod_n` -1. Add additional Pod capabilities (DR, Regions) - -### Iteration 0: Introduce organizations - -In the first iteration, we introduce the concept of an organization -as a way to group top-level namespaces together. Support for organizations **does not require any Pods work** but having them will make all subsequent iterations of Pods simpler. This is mainly because we can group top-level namespaces for a single organization onto a Pod. Within an organization all interactions work as normal but we eliminate any cross-organizational interactions except in well defined cases (e.g. forking). - -This means that we don't have a large number of cross-pod interactions. - -Introducing organizations allows GitLab to move towards a multi-tenant system that is similar to Discord's with a single user account but many different "servers" - our organizations - that allow users to switch context. This model harmonizes the UX across self-managed and our SaaS Platforms and is a good fit for Pods. - -Organizations solve the following problems: - -1. We can group top-level namespaces by organization. It is very similar to the initial concept of "instance groups". For example these two top-level namespaces would belong to the organization `GitLab`: - 1. `https://gitlab.com/gitlab-org/` - 1. `https://gitlab.com/gitlab-com/` -1. We can isolate organizations from each other. Top-level namespaces of the same organization can interact within organizations but are not allowed to interact with other namespaces in other organizations. This is useful for customers because it means an organization provides clear boundaries - similar to a self-managed instance. This means we don't have to aggregate user dashboards across everything and can locally scope them to organizations. -1. We don't need to define hierarchies inside an organization. It is a container that could be filled with whatever hierarchy / entity set makes sense (workspaces, top-level namespaces etc.) -1. Self-managed instances would set a default organization. -1. Organizations can control user-profiles in a central way. This could be achieved by having an organization specific user-profile. Such a profile makes it possible for the organization administrators to control the user role in a company, enforce user emails, or show a graphical indicator of a user being part of the organization. An example would be a "GitLab Employee stamp" on comments. - - - -#### Why would customers opt-in to Organizations? - -By introducing organizations and Pods we can improve the reliability, performance and availability of our SaaS Platforms. - -The first iteration of organizations would also have some benefits by providing more isolation. A simple example would be that `@` mentions could be scoped to an organization. - -Future iterations would create additional value but are beyond the scope of this blueprint. - -Organizations will likely be required in the future as well. - -#### Initial user experience - -1. We create a default `GitLab.com public` organization and assign all public top-level namespaces to it. This allows existing users to access all the data on GitLab.com, exactly as it does now. -1. Any user wanting to opt-in to the benefits of organizations will need to set a single default organization. Any attempts for these users to load a global page like `/dashboard` will end up redirecting to `/-/organizations/<DEFAULT_ORGANIZATION>/dashboard`. -1. New users that opted in to organizations will only ever see data that is related to a single organization. Upon login, data is shown for the default organization. It will be clear to the user how they can switch to a different organization. Users can still navigate to the `GitLab.com` organization but they won't see TODOs from their new organizations in any such views. Instead they'd need to navigate directly to `/organizations/my-company/-/dashboard`. - -### Migrating to Organizations - -Existing customers could also opt-in to migrate their existing top-level paid namespaces to become part of an organization. In most cases this will be a 1-to-1 mapping. But in some cases it may allow a customer to move multiple top-level namespaces into one organization (for example GitLab). - -Migrating to Organizations would be optional. We could even recruit a few beta testers early on to see if this works for them. GitLab itself could dogfood organizations and we'd surface a lot of issues restricting interactions with other namespaces. - -## Iteration 1 - Introduce Pod US 0 - -### GitLab.com as Pod US0 - -GitLab.com will be treated as the first pod `Pod US 0`. It will be unique and much larger compared to newly created pods. All existing top-level namespaces and organizations will remain on `Pod US 0` in the first iteration. - -### Users are globally available - -Users are globally available and the same for all pods. This means that user data needs to be handled separately, for example via decomposition, see [!95941](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95941). - -### Pod groundwork - -In this iteration, we'll lay all the groundwork to support a second Pod for new organizations. This will be transparent to customers. - -## Iteration 2 - Introduce Pod US 1 - -### Add new organizations to Pod US 1 - -After we are ready to support a second Pod, newly created organizations are located by default on `Pod US 1`. The user experience for organizations is already well established. - -### Migrate existing organizations from Pod US 0 to Pod US 1 - -We know that we'll have to move organizations from `Pod US 0` to other pods to reduce its size and ultimately retire the existing GitLab.com architecture. - -By introducing organizations early, we should be able to draw strong "boundaries" across organizations and support migrating existing organizations to a new Pod. - -This is likely going to be GitLab itself - if we can dogfood this, we are likely going to be successful with other organizations as well. - -## Iteration 3 - Introduce Regions - -We can now leverage the Pods architecture to introduce Regions. - -## Iteration 4 - Introduce cross-organizational interactions as needed - -Based on user research, we may want to change certain features to work across organizations. Examples include: - -- Specific features allow for cross-organization interactions, for example forking, search. - -## Technical Proposals - -The Pods architecture do have long lasting implications to data processing, location, scalability and the GitLab architecture. -This section links all different technical proposals that are being evaluated. - -- [Stateless Router That Uses a Cache to Pick Pod and Is Redirected When Wrong Pod Is Reached](proposal-stateless-router-with-buffering-requests.md) - -- [Stateless Router That Uses a Cache to Pick Pod and pre-flight `/api/v4/pods/learn`](proposal-stateless-router-with-routes-learning.md) - -## Impacted features - -The Pods architecture will impact many features requiring some of them to be rewritten, or changed significantly. -This is the list of known affected features with the proposed solutions. - -- [Pods: Git Access](pods-feature-git-access.md) -- [Pods: Data Migration](pods-feature-data-migration.md) -- [Pods: Database Sequences](pods-feature-database-sequences.md) -- [Pods: GraphQL](pods-feature-graphql.md) -- [Pods: Organizations](pods-feature-organizations.md) -- [Pods: Router Endpoints Classification](pods-feature-router-endpoints-classification.md) -- [Pods: Schema changes (Postgres and Elasticsearch migrations)](pods-feature-schema-changes.md) -- [Pods: Backups](pods-feature-backups.md) -- [Pods: Global Search](pods-feature-global-search.md) -- [Pods: CI Runners](pods-feature-ci-runners.md) -- [Pods: Admin Area](pods-feature-admin-area.md) -- [Pods: Secrets](pods-feature-secrets.md) -- [Pods: Container Registry](pods-feature-container-registry.md) -- [Pods: Contributions: Forks](pods-feature-contributions-forks.md) -- [Pods: Personal Namespaces](pods-feature-personal-namespaces.md) -- [Pods: Dashboard: Projects, Todos, Issues, Merge Requests, Activity, ...](pods-feature-dashboard.md) -- [Pods: Snippets](pods-feature-snippets.md) -- [Pods: Uploads](pods-feature-uploads.md) -- [Pods: GitLab Pages](pods-feature-gitlab-pages.md) -- [Pods: Agent for Kubernetes](pods-feature-agent-for-kubernetes.md) - -## Links - -- [Internal Pods presentation](https://docs.google.com/presentation/d/1x1uIiN8FR9fhL7pzFh9juHOVcSxEY7d2_q4uiKKGD44/edit#slide=id.ge7acbdc97a_0_155) -- [Pods Epic](https://gitlab.com/groups/gitlab-org/-/epics/7582) -- [Database Group investigation](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/doc/root-namespace-sharding.html) -- [Shopify Pods architecture](https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale) -- [Opstrace architecture](https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/architecture/overview.md) +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-admin-area.md b/doc/architecture/blueprints/pods/pods-feature-admin-area.md index 7efaa383510..0f02a4a88ba 100644 --- a/doc/architecture/blueprints/pods/pods-feature-admin-area.md +++ b/doc/architecture/blueprints/pods/pods-feature-admin-area.md @@ -1,58 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Admin Area' +redirect_to: '../cells/cells-feature-admin-area.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-admin-area.md). -# Pods: Admin Area - -In our Pods architecture proposal we plan to share all admin related tables in -GitLab. This allows simpler management of all Pods in one interface and reduces -the risk of settings diverging in different Pods. This introduces challenges -with admin pages that allow you to manage data that will be spread across all -Pods. - -## 1. Definition - -There are consequences for admin pages that contain data that spans "the whole -instance" as the Admin pages may be served by any Pod or possibly just 1 pod. -There are already many parts of the Admin interface that will have data that -spans many pods. For example lists of all Groups, Projects, Topics, Jobs, -Analytics, Applications and more. There are also administrative monitoring -capabilities in the Admin page that will span many pods such as the "Background -Jobs" and "Background Migrations" pages. - -## 2. Data flow - -## 3. Proposal - -We will need to decide how to handle these exceptions with a few possible -options: - -1. Move all these pages out into a dedicated per-pod Admin section. Probably - the URL will need to be routable to a single Pod like `/pods/<pod_id>/admin`, - then we can display this data per Pod. These pages will be distinct from - other Admin pages which control settings that are shared across all Pods. We - will also need to consider how this impacts self-managed customers and - whether, or not, this should be visible for single-pod instances of GitLab. -1. Build some aggregation interfaces for this data so that it can be fetched - from all Pods and presented in a single UI. This may be beneficial to an - administrator that needs to see and filter all data at a glance, especially - when they don't know which Pod the data is on. The downside, however, is - that building this kind of aggregation is very tricky when all the Pods are - designed to be totally independent, and it does also enforce more strict - requirements on compatibility between Pods. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md b/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md index f390c751b8b..f28cc447e0a 100644 --- a/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md +++ b/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Agent for Kubernetes' +redirect_to: '../cells/cells-feature-agent-for-kubernetes.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-agent-for-kubernetes.md). -# Pods: Agent for Kubernetes - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-backups.md b/doc/architecture/blueprints/pods/pods-feature-backups.md index 5e4de42f473..db22317cf75 100644 --- a/doc/architecture/blueprints/pods/pods-feature-backups.md +++ b/doc/architecture/blueprints/pods/pods-feature-backups.md @@ -1,61 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Backups' +redirect_to: '../cells/cells-feature-backups.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-backups.md). -# Pods: Backups - -Each pods will take its own backups, and consequently have its own isolated -backup / restore procedure. - -## 1. Definition - -GitLab Backup takes a backup of the PostgreSQL database used by the application, -and also Git repository data. - -## 2. Data flow - -Each pod has a number of application databases to back up (e.g. `main`, and `ci`). - -Additionally, there may be cluster-wide metadata tables (e.g. `users` table) -which is directly accesible via PostgreSQL. - -## 3. Proposal - -### 3.1. Cluster-wide metadata - -It is currently unknown how cluster-wide metadata tables will be accessible. We -may choose to have cluster-wide metadata tables backed up separately, or have -each pod back up its copy of cluster-wide metdata tables. - -### 3.2 Consistency - -#### 3.2.1 Take backups independently - -As each pod will communicate with each other via API, and there will be no joins -to the users table, it should be acceptable for each pod to take a backup -independently of each other. - -#### 3.2.2 Enforce snapshots - -We can require that each pod take a snapshot for the PostgreSQL databases at -around the same time to allow for a consistent-enough backup. - -## 4. Evaluation - -As the number of pods increases, it will likely not be feasible to take a -snapshot at the same time for all pods. Hence taking backups independently is -the better option. - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-ci-runners.md b/doc/architecture/blueprints/pods/pods-feature-ci-runners.md index b75515a916f..1985bb21884 100644 --- a/doc/architecture/blueprints/pods/pods-feature-ci-runners.md +++ b/doc/architecture/blueprints/pods/pods-feature-ci-runners.md @@ -1,169 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: CI Runners' +redirect_to: '../cells/cells-feature-ci-runners.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-ci-runners.md). -# Pods: CI Runners - -GitLab in order to execute CI jobs [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), -very often managed by customer in their infrastructure. - -All CI jobs created as part of CI pipeline are run in a context of project -it poses a challenge how to manage GitLab Runners. - -## 1. Definition - -There are 3 different types of runners: - -- instance-wide: runners that are registered globally with specific tags (selection criteria) -- group runners: runners that execute jobs from a given top-level group or subprojects of that group -- project runners: runners that execute jobs from projects or many projects: some runners might - have projects assigned from projects in different top-level groups. - -This alongside with existing data structure where `ci_runners` is a table describing -all types of runners poses a challenge how the `ci_runners` should be managed in a Pods environment. - -## 2. Data flow - -GitLab Runners use a set of globally scoped endpoints to: - -- registration of a new runner via registration token `https://gitlab.com/api/v4/runners` - ([subject for removal](../runner_tokens/index.md)) (`registration token`) -- requests jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) -- upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) -- upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) -- download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) - -Currently three types of authentication tokens are used: - -- runner registration token ([subject for removal](../runner_tokens/index.md)) -- runner token representing an registered runner in a system with specific configuration (`tags`, `locked`, etc.) -- build token representing an ephemeral token giving a limited access to updating a specific - job, uploading artifacts, downloading dependent artifacts, downloading and uploading - container registry images - -Each of those endpoints do receive an authentication token via header (`JOB-TOKEN` for `/trace`) -or body parameter (`token` all other endpoints). - -Since the CI pipeline would be created in a context of a specific Pod it would be required -that pick of a build would have to be processed by that particular Pod. This requires -that build picking depending on a solution would have to be either: - -- routed to correct Pod for a first time -- be made to be two phase: request build from global pool, claim build on a specific Pod using a Pod specific URL - -## 3. Proposal - -This section describes various proposals. Reader should consider that those -proposals do describe solutions for different problems. Many or some aspects -of those proposals might be the solution to the stated problem. - -### 3.1. Authentication tokens - -Even though the paths for CI Runners are not routable they can be made routable with -those two possible solutions: - -- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with - a ticketing mechanism (based on `X-GitLab-Last-Update` header). Runner when first - starts sends a request to GitLab to which GitLab responds with either a build to pick - by runner. This value is completely controlled by GitLab. This allows GitLab - to use JWT or any other means to encode `pod` identifier that could be easily - decodable by Router. -- The majority of communication (in terms of volume) is using `build token` making it - the easiest target to change since GitLab is sole owner of the token that Runner later - uses for specific job. There were prior discussions about not storing `build token` - but rather using `JWT` token with defined scopes. Such token could encode the `pod` - to which router could easily route all requests. - -### 3.2. Request body - -- The most of used endpoints pass authentication token in request body. It might be desired - to use HTTP Headers as an easier way to access this information by Router without - a need to proxy requests. - -### 3.3. Instance-wide are Pod local - -We can pick a design where all runners are always registered and local to a given Pod: - -- Each Pod has it's own set of instance-wide runners that are updated at it's own pace -- The project runners can only be linked to projects from the same organization - creating strong isolation. -- In this model the `ci_runners` table is local to the Pod. -- In this model we would require the above endpoints to be scoped to a Pod in some way - or made routable. It might be via prefixing them, adding additional Pod parameter, - or providing much more robust way to decode runner token and match it to Pod. -- If routable token is used, we could move away from cryptographic random stored in - database to rather prefer to use JWT tokens that would encode -- The Admin Area showing registered Runners would have to be scoped to a Pod - -This model might be desired since it provides strong isolation guarantees. -This model does significantly increase maintenance overhead since each Pod is managed -separately. - -This model may require adjustments to runner tags feature so that projects have consistent runner experience across pods. - -### 3.4. Instance-wide are cluster-wide - -Contrary to proposal where all runners are Pod local, we can consider that runners -are global, or just instance-wide runners are global. - -However, this requires significant overhaul of system and to change the following aspects: - -- `ci_runners` table would likely have to be split decomposed into `ci_instance_runners`, ... -- all interfaces would have to be adopted to use correct table -- build queuing would have to be reworked to be two phase where each Pod would know of all pending - and running builds, but the actual claim of a build would happen against a Pod containing data -- likely `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables - increasing likelihood of creating hotspots in a system related to CI queueing - -This model makes it complex to implement from engineering side. Does make some data being shared -between Pods. Creates hotspots / scalability issues in a system (ex. during abuse) that -might impact experience of organizations on other Pods. - -### 3.5. GitLab CI Daemon - -Another potential solution to explore is to have a dedicated service responsible for builds queueing -owning it's database and working in a model of either sharded or podded service. There were prior -discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). - -If the service would be sharded: - -- depending on a model if runners are cluster-wide or pod-local this service would have to fetch - data from all Pods -- if the sharded service would be used we could adapt a model of either sharing database containing - `ci_pending_builds/ci_running_builds` with the service -- if the sharded service would be used we could consider a push model where each Pod pushes to CI/CD Daemon - builds that should be picked by Runner -- the sharded service would be aware which Pod is responsible for processing the given build and could - route processing requests to designated Pod - -If the service would be podded: - -- all expectations of routable endpoints are still valid - -In general usage of CI Daemon does not help significantly with the stated problem. However, this offers -a few upsides related to more efficient processing and decoupling model: push model and it opens a way -to offer stateful communication with GitLab Runners (ex. gRPC or Websockets). - -## 4. Evaluation - -Considering all solutions it appears that solution giving the most promise is: - -- use "instance-wide are Pod local" -- refine endpoints to have routable identities (either via specific paths, or better tokens) - -Other potential upsides is to get rid of `ci_builds.token` and rather use a `JWT token` -that can much better and easier encode wider set of scopes allowed by CI runner. - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-container-registry.md b/doc/architecture/blueprints/pods/pods-feature-container-registry.md index d47913fbc2a..9d2bbb3febe 100644 --- a/doc/architecture/blueprints/pods/pods-feature-container-registry.md +++ b/doc/architecture/blueprints/pods/pods-feature-container-registry.md @@ -1,131 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Container Registry' +redirect_to: '../cells/cells-feature-container-registry.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-container-registry.md). -# Pods: Container Registry - -GitLab Container Registry is a feature allowing to store Docker Container Images -in GitLab. You can read about GitLab integration [here](../../../user/packages/container_registry/index.md). - -## 1. Definition - -GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis -and Object Storage dependencies. Right now there's undergoing work to introduce -[Container Registry Metadata](../container_registry_metadata_database/index.md) -to optimize data storage and image retention policies of Container Registry. - -GitLab Container Registry is serving as a container for stored data, -but on it's own does not authenticate `docker login`. The `docker login` -is executed with user credentials (can be `personal access token`) -or CI build credentials (ephemeral `ci_builds.token`). - -Container Registry uses data deduplication. It means that the same blob -(image layer) that is shared between many projects is stored only once. -Each layer is hashed by `sha256`. - -The `docker login` does request JWT time-limited authentication token that -is signed by GitLab, but validated by Container Registry service. The JWT -token does store all authorized scopes (`container repository images`) -and operation types (`push` or `pull`). A single JWT authentication token -can be have many authorized scopes. This allows container registry and client -to mount existing blobs from another scopes. GitLab responds only with -authorized scopes. Then it is up to GitLab Container Registry to validate -if the given operation can be performed. - -The GitLab.com pages are always scoped to project. Each project can have many -container registry images attached. - -Currently in case of GitLab.com the actual registry service is served -via `https://registry.gitlab.com`. - -The main identifiable problems are: - -- the authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com -- the `https://registry.gitlab.com` that is run by external service and uses it's own data store -- the data deduplication, the Pods architecture with registry run in a Pod would reduce - efficiency of data storage - -## 2. Data flow - -### 2.1. Authorization request that is send by `docker login` - -```shell -curl \ - --user "username:password" \ - "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" -``` - -Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. - -```json -{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} -``` - -### 2.2. Docker client fetching tags - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list - -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 -``` - -### 2.3. Docker client fetching blobs and manifests - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 -``` - -## 3. Proposal - -### 3.1. Shard Container Registry separately to Pods architecture - -Due to it's architecture it extensive architecture and in general highly scalable -horizontal architecture it should be evaluated if the GitLab Container Registry -should be run not in Pod, but in a Cluster and be scaled independently. - -This might be easier, but would definitely not offer the same amount of data isolation. - -### 3.2. Run Container Registry within a Pod - -It appears that except `/jwt/auth` which would likely have to be processed by Router -(to decode `scope`) the container registry could be run as a local service of a Pod. - -The actual data at least in case of GitLab.com is not forwarded via registry, -but rather served directly from Object Storage / CDN. - -Its design encodes container repository image in a URL that is easily routable. -It appears that we could re-use the same stateless Router service in front of Container Registry -to serve manifests and blobs redirect. - -The only downside is increased complexity of managing standalone registry for each Pod, -but this might be desired approach. - -## 4. Evaluation - -There do not seem any theoretical problems with running GitLab Container Registry in a Pod. -Service seems that can be easily made routable to work well. - -The practical complexities are around managing complex service from infrastructure side. - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md b/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md index 566ae50ec49..38bdef35329 100644 --- a/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md +++ b/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md @@ -1,120 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Contributions: Forks' +redirect_to: '../cells/cells-feature-contributions-forks.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-contributions-forks.md). -# Pods: Contributions: Forks - -[Forking workflow](../../../user/project/repository/forking_workflow.md) allows users -to copy existing project sources into their own namespace of choice (personal or group). - -## 1. Definition - -[Forking workflow](../../../user/project/repository/forking_workflow.md) is common workflow -with various usage patterns: - -- allows users to contribute back to upstream project -- persist repositories into their personal namespace -- copy to make changes and release as modified project - -Forks allow users not having write access to parent project to make changes. The forking workflow -is especially important for the Open Source community which is able to contribute back -to public projects. However, it is equally important in some companies which prefer the strong split -of responsibilites and tighter access control. The access to project is restricted -to designated list of developers. - -Forks enable: - -- tigther control of who can modify the upstream project -- split of the responsibilites: parent project might use CI configuration connecting to production systems -- run CI pipelines in context of fork in much more restrictive environment -- consider all forks to be unveted which reduces risks of leaking secrets, or any other information - tied with the project - -The forking model is problematic in Pods architecture for following reasons: - -- Forks are clones of existing repositories, forks could be created across different organizations, Pods and Gitaly shards. -- User can create merge request and contribute back to upstream project, this upstream project might in a different organization and Pod. -- The merge request CI pipeline is to executed in a context of source project, but presented in a context of target project. - -## 2. Data flow - -## 3. Proposals - -### 3.1. Intra-Cluster forks - -This proposal makes us to implement forks as a intra-ClusterPod forks where communication is done via API -between all trusted Pods of a cluster: - -- Forks when created, they are created always in context of user choice of group. -- Forks are isolated from Organization. -- Organization or group owner could disable forking across organizations or forking in general. -- When a Merge Request is created it is created in context of target project, referencing - external project on another Pod. -- To target project the merge reference is transfered that is used for presenting information - in context of target project. -- CI pipeline is fetched in context of source project as it-is today, the result is fetched into - Merge Request of target project. -- The Pod holding target project internally uses GraphQL to fetch status of source project - and include in context of the information for merge request. - -Upsides: - -- All existing forks continue to work as-is, as they are treated as intra-Cluster forks. - -Downsides: - -- The purpose of Organizations is to provide strong isolation between organizations - allowing to fork across does break security boundaries. -- However, this is no different to ability of users today to clone repository to local computer - and push it to any repository of choice. -- Access control of source project can be lower than those of target project. System today - requires that in order to contribute back the access level needs to be the same for fork and upstream. - -### 3.2. Forks are created in a personal namespace of the current organization - -Instead of creating projects across organizations, the forks are created in a user personal namespace -tied with the organization. Example: - -- Each user that is part of organization receives their personal namespace. For example for `GitLab Inc.` - it could be `gitlab.com/organization/gitlab-inc/@ayufan`. -- The user has to fork into it's own personal namespace of the organization. -- The user has that many personal namespaces as many organizations it belongs to. -- The personal namespace behaves similar to currently offered personal namespace. -- The user can manage and create projects within a personal namespace. -- The organization can prevent or disable usage of personal namespaces disallowing forks. -- All current forks are migrated into personal namespace of user in Organization. -- All forks are part of to the organization. -- The forks are not federated features. -- The personal namespace and forked project do not share configuration with parent project. - -### 3.3. Forks are created as internal projects under current project - -Instead of creating projects across organizations, the forks are attachments to existing projects. -Each user forking a project receives their unique project. Example: - -- For project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. -- Forks are created in a context of current organization, they do not cross organization boundaries - and are managed by the organization. -- Tied to the user (or any other user-provided name of the fork). -- The forks are not federated features. - -Downsides: - -- Does not answer how to handle and migrate all exisiting forks. -- Might share current group / project settings - breaking some security boundaries. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-dashboard.md b/doc/architecture/blueprints/pods/pods-feature-dashboard.md index e63d912b4c9..1d92b891aff 100644 --- a/doc/architecture/blueprints/pods/pods-feature-dashboard.md +++ b/doc/architecture/blueprints/pods/pods-feature-dashboard.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Dashboard' +redirect_to: '../cells/cells-feature-dashboard.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-dashboard.md). -# Pods: Dashboard - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-data-migration.md b/doc/architecture/blueprints/pods/pods-feature-data-migration.md index fbe97316dcc..c06006a86dc 100644 --- a/doc/architecture/blueprints/pods/pods-feature-data-migration.md +++ b/doc/architecture/blueprints/pods/pods-feature-data-migration.md @@ -1,130 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Data migration' +redirect_to: '../cells/cells-feature-data-migration.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-data-migration.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Data migration - -It is essential for Pods architecture to provide a way to migrate data out of big Pods -into smaller ones. This describes various approaches to provide this type of split. - -We also need to handle for cases where data is already violating the expected -isolation constraints of Pods (ie. references cannot span multiple -organizations). We know that existing features like linked issues allowed users -to link issues across any projects regardless of their hierarchy. There are many -similar features. All of this data will need to be migrated in some way before -it can be split across different pods. This may mean some data needs to be -deleted, or the feature changed and modelled slightly differently before we can -properly split or migrate the organizations between pods. - -Having schema deviations across different Pods, which is a necessary -consequence of different databases, will also impact our ability to migrate -data between pods. Different schemas impact our ability to reliably replicate -data across pods and especially impact our ability to validate that the data is -correctly replicated. It might force us to only be able to move data between -pods when the schemas are all in sync (slowing down deployments and the -rebalancing process) or possibly only migrate from newer to older schemas which -would be complex. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -### 3.1. Split large Pods - -A single Pod can only be divided into many Pods. This is based on principle -that it is easier to create exact clone of an existing Pod in many replicas -out of which some will be made authoritative once migrated. Keeping those -replicas up-to date with Pod 0 is also much easier due to pre-existing -replication solutions that can replicate the whole systems: Geo, PostgreSQL -physical replication, etc. - -1. All data of an organization needs to not be divided across many Pods. -1. Split should be doable online. -1. New Pods cannot contain pre-existing data. -1. N Pods contain exact replica of Pod 0. -1. The data of Pod 0 is live replicated to as many Pods it needs to be split. -1. Once consensus is achieved between Pod 0 and N-Pods the organizations to be migrated away - are marked as read-only cluster-wide. -1. The `routes` is updated on for all organizations to be split to indicate an authoritative - Pod holding the most recent data, like `gitlab-org` on `pod-100`. -1. The data for `gitlab-org` on Pod 0, and on other non-authoritative N-Pods are dormant - and will be removed in the future. -1. All accesses to `gitlab-org` on a given Pod are validated about `pod_id` of `routes` - to ensure that given Pod is authoritative to handle the data. - -#### More challenges of this proposal - -1. There is no streaming replication capability for Elasticsearch, but you could - snapshot the whole Elasticsearch index and recreate, but this takes hours. - It could be handled by pausing Elasticsearch indexing on the initial pod during - the migration as indexing downtime is not a big issue, but this still needs - to be coordinated with the migration process -1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other - new data stores snapshots in an online system would likely lead to gaps - without a long downtime. You need to choose a sync point and at the sync - point you need to stop writes to perform the migration. The more data stores - there are to migrate at the same time the longer the write downtime for the - failover. We would also need to find a reliable place in the application to - actually block updates to all these systems with a high degree of - confidence. In the past we've only been confident by shutting down all rails - services because any rails process could write directly to any of these at - any time due to async workloads or other surprising code paths. -1. How to efficiently delete all the orphaned data. Locating all `ci_builds` - associated with half the organizations would be very expensive if we have to - do joins. We haven't yet determined if we'd want to store an `organization_id` - column on every table, but this is the kind of thing it would be helpful for. - -### 3.2. Migrate organization from an existing Pod - -This is different to split, as we intend to perform logical and selective replication -of data belonging to a single organization. - -Today this type of selective replication is only implemented by Gitaly where we can migrate -Git repository from a single Gitaly node to another with minimal downtime. - -In this model we would require identifying all resources belonging to a given organization: -database rows, object storage files, Git repositories, etc. and selectively copy them over -to another (likely) existing Pod importing data into it. Ideally ensuring that we can -perform logical replication live of all changed data, but change similarly to split -which Pod is authoritative for this organization. - -1. It is hard to identify all resources belonging to organization. -1. It requires either downtime for organization or a robust system to identify - live changes made. -1. It likely will require a full database structure analysis (more robust than project import/export) - to perform selective PostgreSQL logical replication. - -#### More challenges of this proposal - -1. Logical replication is still not performant enough to keep up with our - scale. Even if we could use logical replication we still don't have an - efficient way to filter data related to a single organization without - joining all the way to the `organizations` table which will slow down - logical replication dramatically. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-database-sequences.md b/doc/architecture/blueprints/pods/pods-feature-database-sequences.md index 0a8bb4d250e..9c4d6c5e290 100644 --- a/doc/architecture/blueprints/pods/pods-feature-database-sequences.md +++ b/doc/architecture/blueprints/pods/pods-feature-database-sequences.md @@ -1,94 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Database Sequences' +redirect_to: '../cells/cells-feature-database-sequences.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-database-sequences.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Database Sequences - -GitLab today ensures that every database row create has unique ID, allowing -to access Merge Request, CI Job or Project by a known global ID. - -Pods will use many distinct and not connected databases, each of them having -a separate IDs for most of entities. - -It might be desirable to retain globally unique IDs for all database rows -to allow migrating resources between Pods in the future. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -This are some preliminary ideas how we can retain unique IDs across the system. - -### 3.1. UUID - -Instead of using incremental sequences use UUID (128 bit) that is stored in database. - -- This might break existing IDs and requires adding UUID column for all existing tables. -- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. - -### 3.2. Use Pod index encoded in ID - -Since significant number of tables already use 64 bit ID numbers we could use MSB to encode -Pod ID effectively enabling - -- This might limit amount of Pods that can be enabled in system, as we might decide to only - allocate 1024 possible Pod numbers. -- This might make IDs to be migratable between Pods, since even if entity from Pod 1 is migrated to Pod 100 - this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode Pod number and we would need - lookup table. -- This requires updating all IDs to 32 bits. - -### 3.3. Allocate sequence ranges from central place - -Each Pod might receive its own range of the sequences as they are consumed from a centrally managed place. -Once Pod consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. -Ranges would be tracked to provide a faster lookup table if a random access pattern is required. - -- This might make IDs to be migratable between Pods, since even if entity from Pod 1 is migrated to Pod 100 - this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode Pod number and we would need - much more robust lookup table as we could be breaking previously assigned sequence ranges. -- This does not require updating all IDs to 64 bits. -- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server -- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. - -### 3.4. Define only some tables to require unique IDs - -Maybe this is acceptable only for some tables to have a globally unique IDs. It could be projects, groups -and other top-level entities. All other tables like `merge_requests` would only offer Pod-local ID, -but when referenced outside it would rather use IID (an ID that is monotonic in context of a given resource, like project). - -- This makes the ID 10000 for `merge_requests` be present on all Pods, which might be sometimes confusing - as for uniqueness of the resource. -- This might make random access by ID (if ever needed) be impossible without using composite key, like: `project_id+merge_request_id`. -- This would require us to implement a transformation/generation of new ID if we need to migrate records to another pod. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. -- If IDs need to change when moving between pods this means that any links to records by ID would no longer work even if those links included the `project_id`. -- If we plan to allow these ids to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-git-access.md b/doc/architecture/blueprints/pods/pods-feature-git-access.md index 9bda2d1de9c..1a0df0f9569 100644 --- a/doc/architecture/blueprints/pods/pods-feature-git-access.md +++ b/doc/architecture/blueprints/pods/pods-feature-git-access.md @@ -1,163 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Git Access' +redirect_to: '../cells/cells-feature-git-access.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-git-access.md). -# Pods: Git Access - -This document describes impact of Pods architecture on all Git access (over HTTPS and SSH) -patterns providing explanation of how potentially those features should be changed -to work well with Pods. - -## 1. Definition - -Git access is done through out the application. It can be an operation performed by the system -(read Git repository) or by user (create a new file via Web IDE, `git clone` or `git push` via command line). - -The Pods architecture defines that all Git repositories will be local to the Pod, -so no repository could be shared with another Pod. - -The Pods architecture will require that any Git operation done can only be handled by a Pod holding -the data. It means that any operation either via Web interface, API, or GraphQL needs to be routed -to the correct Pod. It means that any `git clone` or `git push` operation can only be performed -in a context of a Pod. - -## 2. Data flow - -The are various operations performed today by the GitLab on a Git repository. This describes -the data flow how they behave today to better represent the impact. - -It appears that Git access does require changes only to a few endpoints that are scoped to project. -There appear to be different types of repositories: - -- Project: assigned to Group -- Wiki: additional repository assigned to Project -- Design: similar to Wiki, additional repository assigned to Project -- Snippet: creates a virtual project to hold repository, likely tied to the User - -### 2.1. Git clone over HTTPS - -Execution of: `git clone` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC InfoRefsUploadPack - Gitaly ->> User: Response - User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack - Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.2. Git clone over SSH - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.3. Git push over HTTPS - -Execution of: `git push` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC PostReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.4. Git push over SSHD - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC ReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.5. Create commit via Web - -Execution of `Add CHANGELOG` to repository: - -```mermaid -sequenceDiagram - Web ->> Puma: POST /gitlab-org/gitlab/-/create/main - Puma ->> Gitaly: RPC TreeEntry - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> Puma: Response - Puma ->> Web: See CHANGELOG -``` - -## 3. Proposal - -The Pods stateless router proposal requires that any ambiguous path (that is not routable) -will be made to be routable. It means that at least the following paths will have to be updated -do introduce a routable entity (project, group, or organization). - -Change: - -- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` -- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` -- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` -- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` - -Where: - -- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) -- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) - -## 4. Evaluation - -Supporting Git repositories if a Pod can access only its own repositories does not appear to be complex. - -The one major complication is supporting snippets, but this likely falls in the same category as for the approach -to support user's personal namespaces. - -## 4.1. Pros - -1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. - -## 4.2. Cons - -1. The sharing of repositories objects is limited to the given Pod and Gitaly node. -1. The across-Pods forks are likely impossible to be supported (discover: how this work today across different Gitaly node). +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md b/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md index 932f996d8ba..4c7f162434e 100644 --- a/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md +++ b/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: GitLab Pages' +redirect_to: '../cells/cells-feature-gitlab-pages.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-gitlab-pages.md). -# Pods: GitLab Pages - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-global-search.md b/doc/architecture/blueprints/pods/pods-feature-global-search.md index 5ea863ac646..035e95219e4 100644 --- a/doc/architecture/blueprints/pods/pods-feature-global-search.md +++ b/doc/architecture/blueprints/pods/pods-feature-global-search.md @@ -1,47 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Global search' +redirect_to: '../cells/cells-feature-global-search.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-global-search.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Global search - -When we introduce multiple Pods we intend to isolate all services related to -those Pods. This will include Elasticsearch which means our current global -search functionality will not work. It may be possible to implement aggregated -search across all pods, but it is unlikely to be performant to do fan-out -searches across all pods especially once you start to do pagination which -requires setting the correct offset and page number for each search. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -Likely first versions of Pods will simply not support global searches and then -we may later consider if building global searches to support popular use cases -is worthwhile. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-graphql.md b/doc/architecture/blueprints/pods/pods-feature-graphql.md index 87c8391fbb3..f0f01a2b120 100644 --- a/doc/architecture/blueprints/pods/pods-feature-graphql.md +++ b/doc/architecture/blueprints/pods/pods-feature-graphql.md @@ -1,94 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: GraphQL' +redirect_to: '../cells/cells-feature-graphql.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-graphql.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: GraphQL - -GitLab extensively uses GraphQL to perform efficient data query operations. -GraphQL due to it's nature is not directly routable. The way how GitLab uses -it calls the `/api/graphql` endpoint, and only query or mutation of body request -might define where the data can be accessed. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -There are at least two main ways to implement GraphQL in Pods architecture. - -### 3.1. GraphQL routable by endpoint - -Change `/api/graphql` to `/api/organization/<organization>/graphql`. - -- This breaks all existing usages of `/api/graphql` endpoint - since the API URI is changed. - -### 3.2. GraphQL routable by body - -As part of router parse GraphQL body to find a routable entity, like `project`. - -- This still makes the GraphQL query be executed only in context of a given Pod - and not allowing the data to be merged. - -```json -# Good example -{ - project(fullPath:"gitlab-org/gitlab") { - id - description - } -} - -# Bad example, since Merge Request is not routable -{ - mergeRequest(id: 1111) { - iid - description - } -} -``` - -### 3.3. Merging GraphQL Proxy - -Implement as part of router GraphQL Proxy which can parse body -and merge results from many Pods. - -- This might make pagination hard to achieve, or we might assume that - we execute many queries of which results are merged across all Pods. - -```json -{ - project(fullPath:"gitlab-org/gitlab"){ - id, description - } - group(fullPath:"gitlab-com") { - id, description - } -} -``` - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-organizations.md b/doc/architecture/blueprints/pods/pods-feature-organizations.md index a0a87458767..f801f739374 100644 --- a/doc/architecture/blueprints/pods/pods-feature-organizations.md +++ b/doc/architecture/blueprints/pods/pods-feature-organizations.md @@ -1,58 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Organizations' +redirect_to: '../cells/cells-feature-organizations.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-organizations.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Organizations - -One of the major designs of Pods architecture is strong isolation between Groups. -Organizations as described by this blueprint provides a way to have plausible UX -for joining together many Groups that are isolated from the rest of systems. - -## 1. Definition - -Pods do require that all groups and projects of a single organization can -only be stored on a single Pod since a Pod can only access data that it holds locally -and has very limited capabilities to read information from other Pods. - -Pods with Organizations do require strong isolation between organizations. - -It will have significant implications on various user-facing features, -like Todos, dropdowns allowing to select projects, references to other issues -or projects, or any other social functions present at GitLab. Today those functions -were able to reference anything in the whole system. With the introduction of -organizations such will be forbidden. - -This problem definition aims to answer effort and implications required to add -strong isolation between organizations to the system. Including features affected -and their data processing flow. The purpose is to ensure that our solution when -implemented consistently avoids data leakage between organizations residing on -a single Pod. - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md b/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md index f78044bb551..237eb5f9d64 100644 --- a/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md +++ b/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Personal Namespaces' +redirect_to: '../cells/cells-feature-personal-namespaces.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-personal-namespaces.md). -# Pods: Personal Namespaces - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md b/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md index bf0969fcb38..b9e85c29481 100644 --- a/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md +++ b/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md @@ -1,46 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Router Endpoints Classification' +redirect_to: '../cells/cells-feature-router-endpoints-classification.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-router-endpoints-classification.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Router Endpoints Classification - -Classification of all endpoints is essential to properly route request -hitting load balancer of a GitLab installation to a Pod that can serve it. - -Each Pod should be able to decode each request and classify for which Pod -it belongs to. - -GitLab currently implements hundreds of endpoints. This document tries -to describe various techniques that can be implemented to allow the Rails -to provide this information efficiently. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-schema-changes.md b/doc/architecture/blueprints/pods/pods-feature-schema-changes.md index ae7c288028d..a57f76ad9d4 100644 --- a/doc/architecture/blueprints/pods/pods-feature-schema-changes.md +++ b/doc/architecture/blueprints/pods/pods-feature-schema-changes.md @@ -1,55 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Schema changes' +redirect_to: '../cells/cells-feature-schema-changes.md' +remove_date: '2023-06-13' --- -DISCLAIMER: -This page may contain information related to upcoming products, features and -functionality. It is important to note that the information presented is for -informational purposes only, so please do not rely on the information for -purchasing or planning purposes. Just like with all projects, the items -mentioned on the page are subject to change or delay, and the development, -release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +This document was moved to [another location](../cells/cells-feature-schema-changes.md). -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Pods: Schema changes - -When we introduce multiple Pods that own their own databases this will -complicate the process of making schema changes to Postgres and Elasticsearch. -Today we already need to be careful to make changes comply with our zero -downtime deployments. For example, -[when removing a column we need to make changes over 3 separate deployments](../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). -We have tooling like `post_migrate` that helps with these kinds of changes to -reduce the number of merge requests needed, but these will be complicated when -we are dealing with deploying multiple rails applications that will be at -different versions at any one time. This problem will be particularly tricky to -solve for shared databases like our plan to share the `users` related tables -among all Pods. - -A key benefit of Pods may be that it allows us to run different -customers on different versions of GitLab. We may choose to update our own pod -before all our customers giving us even more flexibility than our current -canary architecture. But doing this means that schema changes need to have even -more versions of backward compatibility support which could slow down -development as we need extra steps to make schema changes. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-secrets.md b/doc/architecture/blueprints/pods/pods-feature-secrets.md index f18a41dc0fb..f33b98add21 100644 --- a/doc/architecture/blueprints/pods/pods-feature-secrets.md +++ b/doc/architecture/blueprints/pods/pods-feature-secrets.md @@ -1,48 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Secrets' +redirect_to: '../cells/cells-feature-secrets.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-secrets.md). -# Pods: Secrets - -Where possible, each pod should have its own distinct set of secrets. -However, there will be some secrets that will be required to be the same for all -pods in the cluster - -## 1. Definition - -GitLab has a lot of -[secrets](https://docs.gitlab.com/charts/installation/secrets.html) that needs -to be configured. - -Some secrets are for inter-component communication, e.g. `GitLab Shell secret`, -and used only within a pod. - -Some secrets are used for features, e.g. `ci_jwt_signing_key`. - -## 2. Data flow - -## 3. Proposal - -1. Secrets used for features will need to be consistent across all pods, so that the UX is consistent. - 1. This is especially true for the `db_key_base` secret which is used for - encrypting data at rest in the database - so that projects that are - transferred to another pod will continue to work. We do not want to have - to re-encrypt such rows when we move projects/groups between pods. -1. Secrets which are used for intra-pod communication only should be uniquely generated - per-pod. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-snippets.md b/doc/architecture/blueprints/pods/pods-feature-snippets.md index 1bb866ca958..42d3c401dba 100644 --- a/doc/architecture/blueprints/pods/pods-feature-snippets.md +++ b/doc/architecture/blueprints/pods/pods-feature-snippets.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Snippets' +redirect_to: '../cells/cells-feature-snippets.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-snippets.md). -# Pods: Snippets - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-template.md b/doc/architecture/blueprints/pods/pods-feature-template.md index dfae21b5406..acc8e329725 100644 --- a/doc/architecture/blueprints/pods/pods-feature-template.md +++ b/doc/architecture/blueprints/pods/pods-feature-template.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Problem A' +redirect_to: '../cells/cells-feature-template.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-template.md). -# Pods: A - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/pods-feature-uploads.md b/doc/architecture/blueprints/pods/pods-feature-uploads.md index 634f3ef9560..1de4c138843 100644 --- a/doc/architecture/blueprints/pods/pods-feature-uploads.md +++ b/doc/architecture/blueprints/pods/pods-feature-uploads.md @@ -1,29 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods: Uploads' +redirect_to: '../cells/cells-feature-uploads.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/cells-feature-uploads.md). -# Pods: Uploads - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md index adc523e90c2..4c135c5dbc3 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md @@ -1,648 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods Stateless Router Proposal' +redirect_to: '../cells/proposal-stateless-router-with-buffering-requests.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/proposal-stateless-router-with-buffering-requests.md). -# Proposal: Stateless Router - -We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related -tables so that they can be shared between all pods and allow any pod to -authenticate a user and route requests to the correct pod. Pods may receive -requests for the resources they don't own, but they know how to redirect back -to the correct pod. - -The router is stateless and does not read from the `routes` database which -means that all interactions with the database still happen from the Rails -monolith. This architecture also supports regions by allowing for low traffic -databases to be replicated across regions. - -Users are not directly exposed to the concept of Pods but instead they see -different data dependent on their chosen "organization". -[Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which pod, since an -organization can only be on a single pod. - -## Differences - -The main difference between this proposal and the one [with learning routes](proposal-stateless-router-with-routes-learning.md) -is that this proposal always sends requests to any of the Pods. If the requests cannot be processed, -the requests will be bounced back with relevant headers. This requires that request to be buffered. -It allows that request decoding can be either via URI or Body of request by Rails. -This means that each request might be sent more than once and be processed more than once as result. - -The [with learning routes proposal](proposal-stateless-router-with-routes-learning.md) requires that -routable information is always encoded in URI, and the router sends a pre-flight request. - -## Summary in diagrams - -This shows how a user request routes via DNS to the nearest router and the router chooses a pod to send the request to. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - end -``` - -<details><summary>More detail</summary> - -This shows that the router can actually send requests to any pod. The user will -get the closest router to them geographically. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - end - router_eu-.->pod_us0; - router_eu-.->pod_us1; - router_us-.->pod_eu0; - router_us-.->pod_eu1; -``` - -</details> - -<details><summary>Even more detail</summary> - -This shows the databases. `gitlab_users` and `gitlab_routes` exist only in the -US region but are replicated to other regions. Replication does not have an -arrow because it's too hard to read the diagram. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - db_gitlab_users[(gitlab_users Primary)]; - db_gitlab_routes[(gitlab_routes Primary)]; - db_gitlab_users_replica[(gitlab_users Replica)]; - db_gitlab_routes_replica[(gitlab_routes Replica)]; - db_pod_us0[(gitlab_main/gitlab_ci Pod US0)]; - db_pod_us1[(gitlab_main/gitlab_ci Pod US1)]; - db_pod_eu0[(gitlab_main/gitlab_ci Pod EU0)]; - db_pod_eu1[(gitlab_main/gitlab_ci Pod EU1)]; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - pod_eu0-->db_pod_eu0; - pod_eu0-->db_gitlab_users_replica; - pod_eu0-->db_gitlab_routes_replica; - pod_eu1-->db_gitlab_users_replica; - pod_eu1-->db_gitlab_routes_replica; - pod_eu1-->db_pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - pod_us0-->db_pod_us0; - pod_us0-->db_gitlab_users; - pod_us0-->db_gitlab_routes; - pod_us1-->db_gitlab_users; - pod_us1-->db_gitlab_routes; - pod_us1-->db_pod_us1; - end - router_eu-.->pod_us0; - router_eu-.->pod_us1; - router_us-.->pod_eu0; - router_us-.->pod_eu1; -``` - -</details> - -## Summary of changes - -1. Tables related to User data (including profile settings, authentication credentials, personal access tokens) are decomposed into a `gitlab_users` schema -1. The `routes` table is decomposed into `gitlab_routes` schema -1. The `application_settings` (and probably a few other instance level tables) are decomposed into `gitlab_admin` schema -1. A new column `routes.pod_id` is added to `routes` table -1. A new Router service exists to choose which pod to route a request to. -1. A new concept will be introduced in GitLab called an organization and a user can select a "default organization" and this will be a user level setting. The default organization is used to redirect users away from ambiguous routes like `/dashboard` to organization scoped routes like `/organizations/my-organization/-/dashboard`. Legacy users will have a special default organization that allows them to keep using global resources on `Pod US0`. All existing namespaces will initially move to this public organization. -1. If a pod receives a request for a `routes.pod_id` that it does not own it returns a `302` with `X-Gitlab-Pod-Redirect` header so that the router can send the request to the correct pod. The correct pod can also set a header `X-Gitlab-Pod-Cache` which contains information about how this request should be cached to remember the pod. For example if the request was `/gitlab-org/gitlab` then the header would encode `/gitlab-org/* => Pod US0` (for example, any requests starting with `/gitlab-org/` can always be routed to `Pod US0` -1. When the pod does not know (from the cache) which pod to send a request to it just picks a random pod within it's region -1. Writes to `gitlab_users` and `gitlab_routes` are sent to a primary PostgreSQL server in our `US` region but reads can come from replicas in the same region. This will add latency for these writes but we expect they are infrequent relative to the rest of GitLab. - -## Detailed explanation of default organization in the first iteration - -All users will get a new column `users.default_organization` which they can -control in user settings. We will introduce a concept of the -`GitLab.com Public` organization. This will be set as the default organization for all existing -users. This organization will allow the user to see data from all namespaces in -`Pod US0` (for example, our original GitLab.com instance). This behavior can be invisible to -existing users such that they don't even get told when they are viewing a -global page like `/dashboard` that it's even scoped to an organization. - -Any new users with a default organization other than `GitLab.com Public` will have -a distinct user experience and will be fully aware that every page they load is -only ever scoped to a single organization. These users can never -load any global pages like `/dashboard` and will end up being redirected to -`/organizations/<DEFAULT_ORGANIZATION>/-/dashboard`. This may also be the case -for legacy APIs and such users may only ever be able to use APIs scoped to a -organization. - -## Detailed explanation of Admin Area settings - -We believe that maintaining and synchronizing Admin Area settings will be -frustrating and painful so to avoid this we will decompose and share all Admin Area -settings in the `gitlab_admin` schema. This should be safe (similar to other -shared schemas) because these receive very little write traffic. - -In cases where different pods need different settings (for example, the -Elasticsearch URL), we will either decide to use a templated -format in the relevant `application_settings` row which allows it to be dynamic -per pod. Alternatively if that proves difficult we'll introduce a new table -called `per_pod_application_settings` and this will have 1 row per pod to allow -setting different settings per pod. It will still be part of the `gitlab_admin` -schema and shared which will allow us to centrally manage it and simplify -keeping settings in sync for all pods. - -## Pros - -1. Router is stateless and can live in many regions. We use Anycast DNS to resolve to nearest region for the user. -1. Pods can receive requests for namespaces in the wrong pod and the user - still gets the right response as well as caching at the router that - ensures the next request is sent to the correct pod so the next request - will go to the correct pod -1. The majority of the code still lives in `gitlab` rails codebase. The Router doesn't actually need to understand how GitLab URLs are composed. -1. Since the responsibility to read and write `gitlab_users`, - `gitlab_routes` and `gitlab_admin` still lives in Rails it means minimal - changes will be needed to the Rails application compared to extracting - services that need to isolate the domain models and build new interfaces. -1. Compared to a separate routing service this allows the Rails application - to encode more complex rules around how to map URLs to the correct pod - and may work for some existing API endpoints. -1. All the new infrastructure (just a router) is optional and a single-pod - self-managed installation does not even need to run the Router and there are - no other new services. - -## Cons - -1. `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases may need to be - replicated across regions and writes need to go across regions. We need to - do an analysis on write TPS for the relevant tables to determine if this is - feasible. -1. Sharing access to the database from many different Pods means that they are - all coupled at the Postgres schema level and this means changes to the - database schema need to be done carefully in sync with the deployment of all - Pods. This limits us to ensure that Pods are kept in closely similar - versions compared to an architecture with shared services that have an API - we control. -1. Although most data is stored in the right region there can be requests - proxied from another region which may be an issue for certain types - of compliance. -1. Data in `gitlab_users` and `gitlab_routes` databases must be replicated in - all regions which may be an issue for certain types of compliance. -1. The router cache may need to be very large if we get a wide variety of URLs - (for example, long tail). In such a case we may need to implement a 2nd level of - caching in user cookies so their frequently accessed pages always go to the - right pod the first time. -1. Having shared database access for `gitlab_users` and `gitlab_routes` - from multiple pods is an unusual architecture decision compared to - extracting services that are called from multiple pods. -1. It is very likely we won't be able to find cacheable elements of a - GraphQL URL and often existing GraphQL endpoints are heavily dependent on - ids that won't be in the `routes` table so pods won't necessarily know - what pod has the data. As such we'll probably have to update our GraphQL - calls to include an organization context in the path like - `/api/organizations/<organization>/graphql`. -1. This architecture implies that implemented endpoints can only access data - that are readily accessible on a given Pod, but are unlikely - to aggregate information from many Pods. -1. All unknown routes are sent to the latest deployment which we assume to be `Pod US0`. - This is required as newly added endpoints will be only decodable by latest pod. - This Pod could later redirect to correct one that can serve the given request. - Since request processing might be heavy some Pods might receive significant amount - of traffic due to that. - -## Example database configuration - -Handling shared `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases, while having dedicated `gitlab_main` and `gitlab_ci` databases should already be handled by the way we use `config/database.yml`. We should also, already be able to handle the dedicated EU replicas while having a single US primary for `gitlab_users` and `gitlab_routes`. Below is a snippet of part of the database configuration for the Pod architecture described above. - -<details><summary>Pod US0</summary> - -```yaml -# config/database.yml -production: - main: - host: postgres-main.pod-us0.primary.consul - load_balancing: - discovery: postgres-main.pod-us0.replicas.consul - ci: - host: postgres-ci.pod-us0.primary.consul - load_balancing: - discovery: postgres-ci.pod-us0.replicas.consul - users: - host: postgres-users-primary.consul - load_balancing: - discovery: postgres-users-replicas.us.consul - routes: - host: postgres-routes-primary.consul - load_balancing: - discovery: postgres-routes-replicas.us.consul - admin: - host: postgres-admin-primary.consul - load_balancing: - discovery: postgres-admin-replicas.us.consul -``` - -</details> - -<details><summary>Pod EU0</summary> - -```yaml -# config/database.yml -production: - main: - host: postgres-main.pod-eu0.primary.consul - load_balancing: - discovery: postgres-main.pod-eu0.replicas.consul - ci: - host: postgres-ci.pod-eu0.primary.consul - load_balancing: - discovery: postgres-ci.pod-eu0.replicas.consul - users: - host: postgres-users-primary.consul - load_balancing: - discovery: postgres-users-replicas.eu.consul - routes: - host: postgres-routes-primary.consul - load_balancing: - discovery: postgres-routes-replicas.eu.consul - admin: - host: postgres-admin-primary.consul - load_balancing: - discovery: postgres-admin-replicas.eu.consul -``` - -</details> - -## Request flows - -1. `gitlab-org` is a top level namespace and lives in `Pod US0` in the `GitLab.com Public` organization -1. `my-company` is a top level namespace and lives in `Pod EU0` in the `my-organization` organization - -### Experience for paying user that is part of `my-organization` - -Such a user will have a default organization set to `/my-organization` and will be -unable to load any global routes outside of this organization. They may load other -projects/namespaces but their MR/Todo/Issue counts at the top of the page will -not be correctly populated in the first iteration. The user will be aware of -this limitation. - -#### Navigates to `/my-company/my-project` while logged in - -1. User is in Europe so DNS resolves to the router in Europe -1. They request `/my-company/my-project` without the router cache, so the router chooses randomly `Pod EU1` -1. `Pod EU1` does not have `/my-company`, but it knows that it lives in `Pod EU0` so it redirects the router to `Pod EU0` -1. `Pod EU0` returns the correct response as well as setting the cache headers for the router `/my-company/* => Pod EU0` -1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Pod EU0` - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu1: GET /my-company/my-project - pod_eu1->>router_eu: 302 /my-company/my-project X-Gitlab-Pod-Redirect={pod:Pod EU0} - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... X-Gitlab-Pod-Cache={path_prefix:/my-company/} -``` - -#### Navigates to `/my-company/my-project` while not logged in - -1. User is in Europe so DNS resolves to the router in Europe -1. The router does not have `/my-company/*` cached yet so it chooses randomly `Pod EU1` -1. `Pod EU1` redirects them through a login flow -1. Still they request `/my-company/my-project` without the router cache, so the router chooses a random pod `Pod EU1` -1. `Pod EU1` does not have `/my-company`, but it knows that it lives in `Pod EU0` so it redirects the router to `Pod EU0` -1. `Pod EU0` returns the correct response as well as setting the cache headers for the router `/my-company/* => Pod EU0` -1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Pod EU0` - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu1: GET /my-company/my-project - pod_eu1->>user: 302 /users/sign_in?redirect=/my-company/my-project - user->>router_eu: GET /users/sign_in?redirect=/my-company/my-project - router_eu->>pod_eu1: GET /users/sign_in?redirect=/my-company/my-project - pod_eu1->>user: <h1>Sign in... - user->>router_eu: POST /users/sign_in?redirect=/my-company/my-project - router_eu->>pod_eu1: POST /users/sign_in?redirect=/my-company/my-project - pod_eu1->>user: 302 /my-company/my-project - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu1: GET /my-company/my-project - pod_eu1->>router_eu: 302 /my-company/my-project X-Gitlab-Pod-Redirect={pod:Pod EU0} - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... X-Gitlab-Pod-Cache={path_prefix:/my-company/} -``` - -#### Navigates to `/my-company/my-other-project` after last step - -1. User is in Europe so DNS resolves to the router in Europe -1. The router cache now has `/my-company/* => Pod EU0`, so the router chooses `Pod EU0` -1. `Pod EU0` returns the correct response as well as the cache header again - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... X-Gitlab-Pod-Cache={path_prefix:/my-company/} -``` - -#### Navigates to `/gitlab-org/gitlab` after last step - -1. User is in Europe so DNS resolves to the router in Europe -1. The router has no cached value for this URL so randomly chooses `Pod EU0` -1. `Pod EU0` redirects the router to `Pod US0` -1. `Pod US0` returns the correct response as well as the cache header again - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_us0 as Pod US0 - user->>router_eu: GET /gitlab-org/gitlab - router_eu->>pod_eu0: GET /gitlab-org/gitlab - pod_eu0->>router_eu: 302 /gitlab-org/gitlab X-Gitlab-Pod-Redirect={pod:Pod US0} - router_eu->>pod_us0: GET /gitlab-org/gitlab - pod_us0->>user: <h1>GitLab.org... X-Gitlab-Pod-Cache={path_prefix:/gitlab-org/} -``` - -In this case the user is not on their "default organization" so their TODO -counter will not include their normal todos. We may choose to highlight this in -the UI somewhere. A future iteration may be able to fetch that for them from -their default organization. - -#### Navigates to `/` - -1. User is in Europe so DNS resolves to the router in Europe -1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) -1. The Router choose `Pod EU0` randomly -1. The Rails application knows the users default organization is `/my-organization`, so - it redirects the user to `/organizations/my-organization/-/dashboard` -1. The Router has a cached value for `/organizations/my-organization/*` so it then sends the - request to `POD EU0` -1. `Pod EU0` serves up a new page `/organizations/my-organization/-/dashboard` which is the same - dashboard view we have today but scoped to an organization clearly in the UI -1. The user is (optionally) presented with a message saying that data on this page is only - from their default organization and that they can change their default - organization if it's not right. - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - user->>router_eu: GET / - router_eu->>pod_eu0: GET / - pod_eu0->>user: 302 /organizations/my-organization/-/dashboard - user->>router: GET /organizations/my-organization/-/dashboard - router->>pod_eu0: GET /organizations/my-organization/-/dashboard - pod_eu0->>user: <h1>My Company Dashboard... X-Gitlab-Pod-Cache={path_prefix:/organizations/my-organization/} -``` - -#### Navigates to `/dashboard` - -As above, they will end up on `/organizations/my-organization/-/dashboard` as -the rails application will already redirect `/` to the dashboard page. - -### Navigates to `/not-my-company/not-my-project` while logged in (but they don't have access since this project/group is private) - -1. User is in Europe so DNS resolves to the router in Europe -1. The router knows that `/not-my-company` lives in `Pod US1` so sends the request to this -1. The user does not have access so `Pod US1` returns 404 - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_us1 as Pod US1 - user->>router_eu: GET /not-my-company/not-my-project - router_eu->>pod_us1: GET /not-my-company/not-my-project - pod_us1->>user: 404 -``` - -#### Creates a new top level namespace - -The user will be asked which organization they want the namespace to belong to. -If they select `my-organization` then it will end up on the same pod as all -other namespaces in `my-organization`. If they select nothing we default to -`GitLab.com Public` and it is clear to the user that this is isolated from -their existing organization such that they won't be able to see data from both -on a single page. - -### Experience for GitLab team member that is part of `/gitlab-org` - -Such a user is considered a legacy user and has their default organization set to -`GitLab.com Public`. This is a "meta" organization that does not really exist but -the Rails application knows to interpret this organization to mean that they are -allowed to use legacy global functionality like `/dashboard` to see data across -namespaces located on `Pod US0`. The rails backend also knows that the default pod to render any ambiguous -routes like `/dashboard` is `Pod US0`. Lastly the user will be allowed to -navigate to organizations on another pod like `/my-organization` but when they do the -user will see a message indicating that some data may be missing (for example, the -MRs/Issues/Todos) counts. - -#### Navigates to `/gitlab-org/gitlab` while not logged in - -1. User is in the US so DNS resolves to the US router -1. The router knows that `/gitlab-org` lives in `Pod US0` so sends the request - to this pod -1. `Pod US0` serves up the response - -```mermaid -sequenceDiagram - participant user as User - participant router_us as Router US - participant pod_us0 as Pod US0 - user->>router_us: GET /gitlab-org/gitlab - router_us->>pod_us0: GET /gitlab-org/gitlab - pod_us0->>user: <h1>GitLab.org... X-Gitlab-Pod-Cache={path_prefix:/gitlab-org/} -``` - -#### Navigates to `/` - -1. User is in US so DNS resolves to the router in US -1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) -1. The Router chooses `Pod US1` randomly -1. The Rails application knows the users default organization is `GitLab.com Public`, so - it redirects the user to `/dashboards` (only legacy users can see - `/dashboard` global view) -1. Router does not have a cache for `/dashboard` route (specifically rails never tells it to cache this route) -1. The Router chooses `Pod US1` randomly -1. The Rails application knows the users default organization is `GitLab.com Public`, so - it allows the user to load `/dashboards` (only legacy users can see - `/dashboard` global view) and redirects to router the legacy pod which is `Pod US0` -1. `Pod US0` serves up the global view dashboard page `/dashboard` which is the same - dashboard view we have today - -```mermaid -sequenceDiagram - participant user as User - participant router_us as Router US - participant pod_us0 as Pod US0 - participant pod_us1 as Pod US1 - user->>router_us: GET / - router_us->>pod_us1: GET / - pod_us1->>user: 302 /dashboard - user->>router_us: GET /dashboard - router_us->>pod_us1: GET /dashboard - pod_us1->>router_us: 302 /dashboard X-Gitlab-Pod-Redirect={pod:Pod US0} - router_us->>pod_us0: GET /dashboard - pod_us0->>user: <h1>Dashboard... -``` - -#### Navigates to `/my-company/my-other-project` while logged in (but they don't have access since this project is private) - -They get a 404. - -### Experience for non-authenticated users - -Flow is similar to authenticated users except global routes like `/dashboard` will -redirect to the login page as there is no default organization to choose from. - -### A new customers signs up - -They will be asked if they are already part of an organization or if they'd -like to create one. If they choose neither they end up no the default -`GitLab.com Public` organization. - -### An organization is moved from 1 pod to another - -TODO - -### GraphQL/API requests which don't include the namespace in the URL - -TODO - -### The autocomplete suggestion functionality in the search bar which remembers recent issues/MRs - -TODO - -### Global search - -TODO - -## Administrator - -### Loads `/admin` page - -1. Router picks a random pod `Pod US0` -1. Pod US0 redirects user to `/admin/pods/podus0` -1. Pod US0 renders an Admin Area page and also returns a cache header to cache `/admin/podss/podus0/* => Pod US0`. The Admin Area page contains a dropdown list showing other pods they could select and it changes the query parameter. - -Admin Area settings in Postgres are all shared across all pods to avoid -divergence but we still make it clear in the URL and UI which pod is serving -the Admin Area page as there is dynamic data being generated from these pages and -the operator may want to view a specific pod. - -## More Technical Problems To Solve - -### Replicating User Sessions Between All Pods - -Today user sessions live in Redis but each pod will have their own Redis instance. We already use a dedicated Redis instance for sessions so we could consider sharing this with all pods like we do with `gitlab_users` PostgreSQL database. But an important consideration will be latency as we would still want to mostly fetch sessions from the same region. - -An alternative might be that user sessions get moved to a JWT payload that encodes all the session data but this has downsides. For example, it is difficult to expire a user session, when their password changes or for other reasons, if the session lives in a JWT controlled by the user. - -### How do we migrate between Pods - -Migrating data between pods will need to factor all data stores: - -1. PostgreSQL -1. Redis Shared State -1. Gitaly -1. Elasticsearch - -### Is it still possible to leak the existence of private groups via a timing attack? - -If you have router in EU, and you know that EU router by default redirects -to EU located Pods, you know their latency (lets assume 10 ms). Now, if your -request is bounced back and redirected to US which has different latency -(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was -returned by US Pod and know that your 404 is in fact 403. - -We may defer this until we actually implement a pod in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. - -One technique to mitigate this risk might be to have the router add a random -delay to any request that returns 404 from a pod. - -## Should runners be shared across all pods? - -We have 2 options and we should decide which is easier: - -1. Decompose runner registration and queuing tables and share them across all - pods. This may have implications for scalability, and we'd need to consider - if this would include group/project runners as this may have scalability - concerns as these are high traffic tables that would need to be shared. -1. Runners are registered per-pod and, we probably have a separate fleet of - runners for every pod or just register the same runners to many pods which - may have implications for queueing - -## How do we guarantee unique ids across all pods for things that cannot conflict? - -This project assumes at least namespaces and projects have unique ids across -all pods as many requests need to be routed based on their ID. Since those -tables are across different databases then guaranteeing a unique ID will -require a new solution. There are likely other tables where unique IDs are -necessary and depending on how we resolve routing for GraphQL and other APIs -and other design goals it may be determined that we want the primary key to be -unique for all tables. +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md index 1156e65f6aa..093d5d7acc6 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md @@ -1,672 +1,11 @@ --- -stage: enablement -group: pods -comments: false -description: 'Pods Stateless Router Proposal' +redirect_to: '../cells/proposal-stateless-router-with-routes-learning.md' +remove_date: '2023-06-13' --- -This document is a work-in-progress and represents a very early state of the -Pods design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Pods, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. +This document was moved to [another location](../cells/proposal-stateless-router-with-routes-learning.md). -# Proposal: Stateless Router - -We will decompose `gitlab_users`, `gitlab_routes` and `gitlab_admin` related -tables so that they can be shared between all pods and allow any pod to -authenticate a user and route requests to the correct pod. Pods may receive -requests for the resources they don't own, but they know how to redirect back -to the correct pod. - -The router is stateless and does not read from the `routes` database which -means that all interactions with the database still happen from the Rails -monolith. This architecture also supports regions by allowing for low traffic -databases to be replicated across regions. - -Users are not directly exposed to the concept of Pods but instead they see -different data dependent on their chosen "organization". -[Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which pod, since an -organization can only be on a single pod. - -## Differences - -The main difference between this proposal and one [with buffering requests](proposal-stateless-router-with-buffering-requests.md) -is that this proposal uses a pre-flight API request (`/api/v4/pods/learn`) to redirect the request body to the correct Pod. -This means that each request is sent exactly once to be processed, but the URI is used to decode which Pod it should be directed. - -## Summary in diagrams - -This shows how a user request routes via DNS to the nearest router and the router chooses a pod to send the request to. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - end -``` - -### More detail - -This shows that the router can actually send requests to any pod. The user will -get the closest router to them geographically. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - end - router_eu-.->pod_us0; - router_eu-.->pod_us1; - router_us-.->pod_eu0; - router_us-.->pod_eu1; -``` - -### Even more detail - -This shows the databases. `gitlab_users` and `gitlab_routes` exist only in the -US region but are replicated to other regions. Replication does not have an -arrow because it's too hard to read the diagram. - -```mermaid -graph TD; - user((User)); - dns[DNS]; - router_us(Router); - router_eu(Router); - pod_us0{Pod US0}; - pod_us1{Pod US1}; - pod_eu0{Pod EU0}; - pod_eu1{Pod EU1}; - db_gitlab_users[(gitlab_users Primary)]; - db_gitlab_routes[(gitlab_routes Primary)]; - db_gitlab_users_replica[(gitlab_users Replica)]; - db_gitlab_routes_replica[(gitlab_routes Replica)]; - db_pod_us0[(gitlab_main/gitlab_ci Pod US0)]; - db_pod_us1[(gitlab_main/gitlab_ci Pod US1)]; - db_pod_eu0[(gitlab_main/gitlab_ci Pod EU0)]; - db_pod_eu1[(gitlab_main/gitlab_ci Pod EU1)]; - user-->dns; - dns-->router_us; - dns-->router_eu; - subgraph Europe - router_eu-->pod_eu0; - router_eu-->pod_eu1; - pod_eu0-->db_pod_eu0; - pod_eu0-->db_gitlab_users_replica; - pod_eu0-->db_gitlab_routes_replica; - pod_eu1-->db_gitlab_users_replica; - pod_eu1-->db_gitlab_routes_replica; - pod_eu1-->db_pod_eu1; - end - subgraph United States - router_us-->pod_us0; - router_us-->pod_us1; - pod_us0-->db_pod_us0; - pod_us0-->db_gitlab_users; - pod_us0-->db_gitlab_routes; - pod_us1-->db_gitlab_users; - pod_us1-->db_gitlab_routes; - pod_us1-->db_pod_us1; - end - router_eu-.->pod_us0; - router_eu-.->pod_us1; - router_us-.->pod_eu0; - router_us-.->pod_eu1; -``` - -## Summary of changes - -1. Tables related to User data (including profile settings, authentication credentials, personal access tokens) are decomposed into a `gitlab_users` schema -1. The `routes` table is decomposed into `gitlab_routes` schema -1. The `application_settings` (and probably a few other instance level tables) are decomposed into `gitlab_admin` schema -1. A new column `routes.pod_id` is added to `routes` table -1. A new Router service exists to choose which pod to route a request to. -1. If a router receives a new request it will send `/api/v4/pods/learn?method=GET&path_info=/group-org/project` to learn which Pod can process it -1. A new concept will be introduced in GitLab called an organization -1. We require all existing endpoints to be routable by URI, or be fixed to a specific Pod for processing. This requires changing ambiguous endpoints like `/dashboard` to be scoped like `/organizations/my-organization/-/dashboard` -1. Endpoints like `/admin` would be routed always to the specific Pod, like `pod_0` -1. Each Pod can respond to `/api/v4/pods/learn` and classify each endpoint -1. Writes to `gitlab_users` and `gitlab_routes` are sent to a primary PostgreSQL server in our `US` region but reads can come from replicas in the same region. This will add latency for these writes but we expect they are infrequent relative to the rest of GitLab. - -## Pre-flight request learning - -While processing a request the URI will be decoded and a pre-flight request -will be sent for each non-cached endpoint. - -When asking for the endpoint GitLab Rails will return information about -the routable path. GitLab Rails will decode `path_info` and match it to -an existing endpoint and find a routable entity (like project). The router will -treat this as short-lived cache information. - -1. Prefix match: `/api/v4/pods/learn?method=GET&path_info=/gitlab-org/gitlab-test/-/issues` - - ```json - { - "path": "/gitlab-org/gitlab-test", - "pod": "pod_0", - "source": "routable" - } - ``` - -1. Some endpoints might require an exact match: `/api/v4/pods/learn?method=GET&path_info=/-/profile` - - ```json - { - "path": "/-/profile", - "pod": "pod_0", - "source": "fixed", - "exact": true - } - ``` - -## Detailed explanation of default organization in the first iteration - -All users will get a new column `users.default_organization` which they can -control in user settings. We will introduce a concept of the -`GitLab.com Public` organization. This will be set as the default organization for all existing -users. This organization will allow the user to see data from all namespaces in -`Pod US0` (ie. our original GitLab.com instance). This behavior can be invisible to -existing users such that they don't even get told when they are viewing a -global page like `/dashboard` that it's even scoped to an organization. - -Any new users with a default organization other than `GitLab.com Public` will have -a distinct user experience and will be fully aware that every page they load is -only ever scoped to a single organization. These users can never -load any global pages like `/dashboard` and will end up being redirected to -`/organizations/<DEFAULT_ORGANIZATION>/-/dashboard`. This may also be the case -for legacy APIs and such users may only ever be able to use APIs scoped to a -organization. - -## Detailed explanation of Admin Area settings - -We believe that maintaining and synchronizing Admin Area settings will be -frustrating and painful so to avoid this we will decompose and share all Admin Area -settings in the `gitlab_admin` schema. This should be safe (similar to other -shared schemas) because these receive very little write traffic. - -In cases where different pods need different settings (eg. the -Elasticsearch URL), we will either decide to use a templated -format in the relevant `application_settings` row which allows it to be dynamic -per pod. Alternatively if that proves difficult we'll introduce a new table -called `per_pod_application_settings` and this will have 1 row per pod to allow -setting different settings per pod. It will still be part of the `gitlab_admin` -schema and shared which will allow us to centrally manage it and simplify -keeping settings in sync for all pods. - -## Pros - -1. Router is stateless and can live in many regions. We use Anycast DNS to resolve to nearest region for the user. -1. Pods can receive requests for namespaces in the wrong pod and the user - still gets the right response as well as caching at the router that - ensures the next request is sent to the correct pod so the next request - will go to the correct pod -1. The majority of the code still lives in `gitlab` rails codebase. The Router doesn't actually need to understand how GitLab URLs are composed. -1. Since the responsibility to read and write `gitlab_users`, - `gitlab_routes` and `gitlab_admin` still lives in Rails it means minimal - changes will be needed to the Rails application compared to extracting - services that need to isolate the domain models and build new interfaces. -1. Compared to a separate routing service this allows the Rails application - to encode more complex rules around how to map URLs to the correct pod - and may work for some existing API endpoints. -1. All the new infrastructure (just a router) is optional and a single-pod - self-managed installation does not even need to run the Router and there are - no other new services. - -## Cons - -1. `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases may need to be - replicated across regions and writes need to go across regions. We need to - do an analysis on write TPS for the relevant tables to determine if this is - feasible. -1. Sharing access to the database from many different Pods means that they are - all coupled at the Postgres schema level and this means changes to the - database schema need to be done carefully in sync with the deployment of all - Pods. This limits us to ensure that Pods are kept in closely similar - versions compared to an architecture with shared services that have an API - we control. -1. Although most data is stored in the right region there can be requests - proxied from another region which may be an issue for certain types - of compliance. -1. Data in `gitlab_users` and `gitlab_routes` databases must be replicated in - all regions which may be an issue for certain types of compliance. -1. The router cache may need to be very large if we get a wide variety of URLs - (ie. long tail). In such a case we may need to implement a 2nd level of - caching in user cookies so their frequently accessed pages always go to the - right pod the first time. -1. Having shared database access for `gitlab_users` and `gitlab_routes` - from multiple pods is an unusual architecture decision compared to - extracting services that are called from multiple pods. -1. It is very likely we won't be able to find cacheable elements of a - GraphQL URL and often existing GraphQL endpoints are heavily dependent on - ids that won't be in the `routes` table so pods won't necessarily know - what pod has the data. As such we'll probably have to update our GraphQL - calls to include an organization context in the path like - `/api/organizations/<organization>/graphql`. -1. This architecture implies that implemented endpoints can only access data - that are readily accessible on a given Pod, but are unlikely - to aggregate information from many Pods. -1. All unknown routes are sent to the latest deployment which we assume to be `Pod US0`. - This is required as newly added endpoints will be only decodable by latest pod. - Likely this is not a problem for the `/pods/learn` is it is lightweight - to process and this should not cause a performance impact. - -## Example database configuration - -Handling shared `gitlab_users`, `gitlab_routes` and `gitlab_admin` databases, while having dedicated `gitlab_main` and `gitlab_ci` databases should already be handled by the way we use `config/database.yml`. We should also, already be able to handle the dedicated EU replicas while having a single US primary for `gitlab_users` and `gitlab_routes`. Below is a snippet of part of the database configuration for the Pod architecture described above. - -**Pod US0**: - -```yaml -# config/database.yml -production: - main: - host: postgres-main.pod-us0.primary.consul - load_balancing: - discovery: postgres-main.pod-us0.replicas.consul - ci: - host: postgres-ci.pod-us0.primary.consul - load_balancing: - discovery: postgres-ci.pod-us0.replicas.consul - users: - host: postgres-users-primary.consul - load_balancing: - discovery: postgres-users-replicas.us.consul - routes: - host: postgres-routes-primary.consul - load_balancing: - discovery: postgres-routes-replicas.us.consul - admin: - host: postgres-admin-primary.consul - load_balancing: - discovery: postgres-admin-replicas.us.consul -``` - -**Pod EU0**: - -```yaml -# config/database.yml -production: - main: - host: postgres-main.pod-eu0.primary.consul - load_balancing: - discovery: postgres-main.pod-eu0.replicas.consul - ci: - host: postgres-ci.pod-eu0.primary.consul - load_balancing: - discovery: postgres-ci.pod-eu0.replicas.consul - users: - host: postgres-users-primary.consul - load_balancing: - discovery: postgres-users-replicas.eu.consul - routes: - host: postgres-routes-primary.consul - load_balancing: - discovery: postgres-routes-replicas.eu.consul - admin: - host: postgres-admin-primary.consul - load_balancing: - discovery: postgres-admin-replicas.eu.consul -``` - -## Request flows - -1. `gitlab-org` is a top level namespace and lives in `Pod US0` in the `GitLab.com Public` organization -1. `my-company` is a top level namespace and lives in `Pod EU0` in the `my-organization` organization - -### Experience for paying user that is part of `my-organization` - -Such a user will have a default organization set to `/my-organization` and will be -unable to load any global routes outside of this organization. They may load other -projects/namespaces but their MR/Todo/Issue counts at the top of the page will -not be correctly populated in the first iteration. The user will be aware of -this limitation. - -#### Navigates to `/my-company/my-project` while logged in - -1. User is in Europe so DNS resolves to the router in Europe -1. They request `/my-company/my-project` without the router cache, so the router chooses randomly `Pod EU1` -1. The `/pods/learn` is sent to `Pod EU1`, which responds that resource lives on `Pod EU0` -1. `Pod EU0` returns the correct response -1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Pod EU0` - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu1: /api/v4/pods/learn?method=GET&path_info=/my-company/my-project - pod_eu1->>router_eu: {path: "/my-company", pod: "pod_eu0", source: "routable"} - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... -``` - -#### Navigates to `/my-company/my-project` while not logged in - -1. User is in Europe so DNS resolves to the router in Europe -1. The router does not have `/my-company/*` cached yet so it chooses randomly `Pod EU1` -1. The `/pods/learn` is sent to `Pod EU1`, which responds that resource lives on `Pod EU0` -1. `Pod EU0` redirects them through a login flow -1. User requests `/users/sign_in`, uses random Pod to run `/pods/learn` -1. The `Pod EU1` responds with `pod_0` as a fixed route -1. User after login requests `/my-company/my-project` which is cached and stored in `Pod EU0` -1. `Pod EU0` returns the correct response - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu1: /api/v4/pods/learn?method=GET&path_info=/my-company/my-project - pod_eu1->>router_eu: {path: "/my-company", pod: "pod_eu0", source: "routable"} - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: 302 /users/sign_in?redirect=/my-company/my-project - user->>router_eu: GET /users/sign_in?redirect=/my-company/my-project - router_eu->>pod_eu1: /api/v4/pods/learn?method=GET&path_info=/users/sign_in - pod_eu1->>router_eu: {path: "/users", pod: "pod_eu0", source: "fixed"} - router_eu->>pod_eu0: GET /users/sign_in?redirect=/my-company/my-project - pod_eu0-->>user: <h1>Sign in... - user->>router_eu: POST /users/sign_in?redirect=/my-company/my-project - router_eu->>pod_eu0: POST /users/sign_in?redirect=/my-company/my-project - pod_eu0->>user: 302 /my-company/my-project - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu0: GET /my-company/my-project - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... -``` - -#### Navigates to `/my-company/my-other-project` after last step - -1. User is in Europe so DNS resolves to the router in Europe -1. The router cache now has `/my-company/* => Pod EU0`, so the router chooses `Pod EU0` -1. `Pod EU0` returns the correct response as well as the cache header again - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_eu1 as Pod EU1 - user->>router_eu: GET /my-company/my-project - router_eu->>pod_eu0: GET /my-company/my-project - pod_eu0->>user: <h1>My Project... -``` - -#### Navigates to `/gitlab-org/gitlab` after last step - -1. User is in Europe so DNS resolves to the router in Europe -1. The router has no cached value for this URL so randomly chooses `Pod EU0` -1. `Pod EU0` redirects the router to `Pod US0` -1. `Pod US0` returns the correct response as well as the cache header again - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - participant pod_us0 as Pod US0 - user->>router_eu: GET /gitlab-org/gitlab - router_eu->>pod_eu0: /api/v4/pods/learn?method=GET&path_info=/gitlab-org/gitlab - pod_eu0->>router_eu: {path: "/gitlab-org", pod: "pod_us0", source: "routable"} - router_eu->>pod_us0: GET /gitlab-org/gitlab - pod_us0->>user: <h1>GitLab.org... -``` - -In this case the user is not on their "default organization" so their TODO -counter will not include their normal todos. We may choose to highlight this in -the UI somewhere. A future iteration may be able to fetch that for them from -their default organization. - -#### Navigates to `/` - -1. User is in Europe so DNS resolves to the router in Europe -1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) -1. The Router choose `Pod EU0` randomly -1. The Rails application knows the users default organization is `/my-organization`, so - it redirects the user to `/organizations/my-organization/-/dashboard` -1. The Router has a cached value for `/organizations/my-organization/*` so it then sends the - request to `POD EU0` -1. `Pod EU0` serves up a new page `/organizations/my-organization/-/dashboard` which is the same - dashboard view we have today but scoped to an organization clearly in the UI -1. The user is (optionally) presented with a message saying that data on this page is only - from their default organization and that they can change their default - organization if it's not right. - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_eu0 as Pod EU0 - user->>router_eu: GET / - router_eu->>pod_eu0: GET / - pod_eu0->>user: 302 /organizations/my-organization/-/dashboard - user->>router: GET /organizations/my-organization/-/dashboard - router->>pod_eu0: GET /organizations/my-organization/-/dashboard - pod_eu0->>user: <h1>My Company Dashboard... X-Gitlab-Pod-Cache={path_prefix:/organizations/my-organization/} -``` - -#### Navigates to `/dashboard` - -As above, they will end up on `/organizations/my-organization/-/dashboard` as -the rails application will already redirect `/` to the dashboard page. - -### Navigates to `/not-my-company/not-my-project` while logged in (but they don't have access since this project/group is private) - -1. User is in Europe so DNS resolves to the router in Europe -1. The router knows that `/not-my-company` lives in `Pod US1` so sends the request to this -1. The user does not have access so `Pod US1` returns 404 - -```mermaid -sequenceDiagram - participant user as User - participant router_eu as Router EU - participant pod_us1 as Pod US1 - user->>router_eu: GET /not-my-company/not-my-project - router_eu->>pod_us1: GET /not-my-company/not-my-project - pod_us1->>user: 404 -``` - -#### Creates a new top level namespace - -The user will be asked which organization they want the namespace to belong to. -If they select `my-organization` then it will end up on the same pod as all -other namespaces in `my-organization`. If they select nothing we default to -`GitLab.com Public` and it is clear to the user that this is isolated from -their existing organization such that they won't be able to see data from both -on a single page. - -### Experience for GitLab team member that is part of `/gitlab-org` - -Such a user is considered a legacy user and has their default organization set to -`GitLab.com Public`. This is a "meta" organization that does not really exist but -the Rails application knows to interpret this organization to mean that they are -allowed to use legacy global functionality like `/dashboard` to see data across -namespaces located on `Pod US0`. The rails backend also knows that the default pod to render any ambiguous -routes like `/dashboard` is `Pod US0`. Lastly the user will be allowed to -navigate to organizations on another pod like `/my-organization` but when they do the -user will see a message indicating that some data may be missing (eg. the -MRs/Issues/Todos) counts. - -#### Navigates to `/gitlab-org/gitlab` while not logged in - -1. User is in the US so DNS resolves to the US router -1. The router knows that `/gitlab-org` lives in `Pod US0` so sends the request - to this pod -1. `Pod US0` serves up the response - -```mermaid -sequenceDiagram - participant user as User - participant router_us as Router US - participant pod_us0 as Pod US0 - user->>router_us: GET /gitlab-org/gitlab - router_us->>pod_us0: GET /gitlab-org/gitlab - pod_us0->>user: <h1>GitLab.org... -``` - -#### Navigates to `/` - -1. User is in US so DNS resolves to the router in US -1. Router does not have a cache for `/` route (specifically rails never tells it to cache this route) -1. The Router chooses `Pod US1` randomly -1. The Rails application knows the users default organization is `GitLab.com Public`, so - it redirects the user to `/dashboards` (only legacy users can see - `/dashboard` global view) -1. Router does not have a cache for `/dashboard` route (specifically rails never tells it to cache this route) -1. The Router chooses `Pod US1` randomly -1. The Rails application knows the users default organization is `GitLab.com Public`, so - it allows the user to load `/dashboards` (only legacy users can see - `/dashboard` global view) and redirects to router the legacy pod which is `Pod US0` -1. `Pod US0` serves up the global view dashboard page `/dashboard` which is the same - dashboard view we have today - -```mermaid -sequenceDiagram - participant user as User - participant router_us as Router US - participant pod_us0 as Pod US0 - participant pod_us1 as Pod US1 - user->>router_us: GET / - router_us->>pod_us1: GET / - pod_us1->>user: 302 /dashboard - user->>router_us: GET /dashboard - router_us->>pod_us1: /api/v4/pods/learn?method=GET&path_info=/dashboard - pod_us1->>router_us: {path: "/dashboard", pod: "pod_us0", source: "routable"} - router_us->>pod_us0: GET /dashboard - pod_us0->>user: <h1>Dashboard... -``` - -#### Navigates to `/my-company/my-other-project` while logged in (but they don't have access since this project is private) - -They get a 404. - -### Experience for non-authenticated users - -Flow is similar to logged in users except global routes like `/dashboard` will -redirect to the login page as there is no default organization to choose from. - -### A new customers signs up - -They will be asked if they are already part of an organization or if they'd -like to create one. If they choose neither they end up no the default -`GitLab.com Public` organization. - -### An organization is moved from 1 pod to another - -TODO - -### GraphQL/API requests which don't include the namespace in the URL - -TODO - -### The autocomplete suggestion functionality in the search bar which remembers recent issues/MRs - -TODO - -### Global search - -TODO - -## Administrator - -### Loads `/admin` page - -1. The `/admin` is locked to `Pod US0` -1. Some endpoints of `/admin`, like Projects in Admin are scoped to a Pod - and users needs to choose the correct one in a dropdown, which results in endpoint - like `/admin/pods/pod_0/projects`. - -Admin Area settings in Postgres are all shared across all pods to avoid -divergence but we still make it clear in the URL and UI which pod is serving -the Admin Area page as there is dynamic data being generated from these pages and -the operator may want to view a specific pod. - -## More Technical Problems To Solve - -### Replicating User Sessions Between All Pods - -Today user sessions live in Redis but each pod will have their own Redis instance. We already use a dedicated Redis instance for sessions so we could consider sharing this with all pods like we do with `gitlab_users` PostgreSQL database. But an important consideration will be latency as we would still want to mostly fetch sessions from the same region. - -An alternative might be that user sessions get moved to a JWT payload that encodes all the session data but this has downsides. For example, it is difficult to expire a user session, when their password changes or for other reasons, if the session lives in a JWT controlled by the user. - -### How do we migrate between Pods - -Migrating data between pods will need to factor all data stores: - -1. PostgreSQL -1. Redis Shared State -1. Gitaly -1. Elasticsearch - -### Is it still possible to leak the existence of private groups via a timing attack? - -If you have router in EU, and you know that EU router by default redirects -to EU located Pods, you know their latency (lets assume 10 ms). Now, if your -request is bounced back and redirected to US which has different latency -(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was -returned by US Pod and know that your 404 is in fact 403. - -We may defer this until we actually implement a pod in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. - -One technique to mitigate this risk might be to have the router add a random -delay to any request that returns 404 from a pod. - -## Should runners be shared across all pods? - -We have 2 options and we should decide which is easier: - -1. Decompose runner registration and queuing tables and share them across all - pods. This may have implications for scalability, and we'd need to consider - if this would include group/project runners as this may have scalability - concerns as these are high traffic tables that would need to be shared. -1. Runners are registered per-pod and, we probably have a separate fleet of - runners for every pod or just register the same runners to many pods which - may have implications for queueing - -## How do we guarantee unique ids across all pods for things that cannot conflict? - -This project assumes at least namespaces and projects have unique ids across -all pods as many requests need to be routed based on their ID. Since those -tables are across different databases then guaranteeing a unique ID will -require a new solution. There are likely other tables where unique IDs are -necessary and depending on how we resolve routing for GraphQL and other APIs -and other design goals it may be determined that we want the primary key to be -unique for all tables. +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/architecture/blueprints/rate_limiting/index.md b/doc/architecture/blueprints/rate_limiting/index.md index b466a54e922..92364040962 100644 --- a/doc/architecture/blueprints/rate_limiting/index.md +++ b/doc/architecture/blueprints/rate_limiting/index.md @@ -375,7 +375,7 @@ hierarchy. Choosing a proper solution will require a thoughtful research. - Implementing a separate Go library which uses the same backend (for example, Redis) for rate limiting. 1. **SDK for Satellite Services (Owning Team)** - - Build Golang SDK. + - Build Go SDK. - Create examples showcasing usage of the new rate limits SDK. 1. **Team fan out for Satellite Services (Stage Groups)** diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index 69a10674d7d..359eadb18c5 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -183,14 +183,17 @@ CREATE TABLE ci_runners ( ) ``` -The `ci_builds_metadata` table shall reference `ci_runner_machines`. +A new `p_ci_runner_machine_builds` table joins the `ci_runner_machines` and `ci_builds` tables, to avoid +adding more pressure to those tables. We might consider a more efficient way to store `contacted_at` than updating the existing record. ```sql -CREATE TABLE ci_builds_metadata ( - ... +CREATE TABLE p_ci_runner_machine_builds ( + partition_id bigint DEFAULT 100 NOT NULL, + build_id bigint NOT NULL, runner_machine_id bigint NOT NULL -); +) +PARTITION BY LIST (partition_id); CREATE TABLE ci_runner_machines ( id bigint NOT NULL, @@ -370,44 +373,53 @@ scope. | GitLab Rails app | `%15.8` | Create database migration to add `config` column to `ci_runner_machines` table. | | GitLab Runner | `%15.9` | Start sending `system_id` value in `POST /jobs/request` request and other follow-up requests that require identifying the unique system. | | GitLab Rails app | `%15.9` | Create service similar to `StaleGroupRunnersPruneCronWorker` service to clean up `ci_runner_machines` records instead of `ci_runners` records.<br/>Existing service continues to exist but focuses only on legacy runners. | -| GitLab Rails app | `%15.9` | [Feature flag] Rollout of `create_runner_machine`. | +| GitLab Rails app | `%15.9` | Implement the `create_runner_machine` [feature flag](../../../administration/feature_flags.md). | | GitLab Rails app | `%15.9` | Create `ci_runner_machines` record in `POST /runners/verify` request if the runner token is prefixed with `glrt-`. | | GitLab Rails app | `%15.9` | Use runner token + `system_id` JSON parameters in `POST /jobs/request` request in the [heartbeat request](https://gitlab.com/gitlab-org/gitlab/blob/c73c96a8ffd515295842d72a3635a8ae873d688c/lib/api/ci/helpers/runner.rb#L14-20) to update the `ci_runner_machines` cache/table. | -| GitLab Rails app | `%15.9` | [Feature flag] Enable runner creation workflow (`create_runner_workflow`). | +| GitLab Rails app | `%15.9` | Implement the `create_runner_workflow_for_admin` [feature flag](../../../administration/feature_flags.md). | | GitLab Rails app | `%15.9` | Implement `create_{instance|group|project}_runner` permissions. | | GitLab Rails app | `%15.9` | Rename `ci_runner_machines.machine_xid` column to `system_xid` to be consistent with `system_id` passed in APIs. | -| GitLab Rails app | `%15.10` | Drop `ci_runner_machines.machine_xid` column. | -| GitLab Rails app | `%15.11` | Remove the ignore rule for `ci_runner_machines.machine_xid` column. | +| GitLab Rails app | `%15.10` | Remove the ignore rule for `ci_runner_machines.machine_xid` column. | +| GitLab Rails app | `%15.10` | Replace `ci_builds_metadata.runner_machine_id` with a new join table. | +| GitLab Rails app | `%15.11` | Drop `ci_builds_metadata.runner_machine_id` column. | +| GitLab Rails app | `%16.0` | Remove the ignore rule for `ci_builds_metadata.runner_machine_id` column. | ### Stage 4 - Create runners from the UI | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Rails app | `%15.9` | Implement new GraphQL user-authenticated API to create a new runner. | | GitLab Rails app | `%15.9` | [Add prefix to newly generated runner authentication tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/383198). | +| GitLab Rails app | `%15.9` | Add new runner field for with token that is used in registration | +| GitLab Rails app | `%15.9` | Implement new GraphQL user-authenticated API to create a new runner. | | GitLab Rails app | `%15.10` | Return token and runner ID information from `/runners/verify` REST endpoint. | | GitLab Runner | `%15.10` | [Modify register command to allow new flow with glrt- prefixed authentication tokens](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29613). | -| GitLab Rails app | `%15.10` | Implement UI to create new runner. | -| GitLab Rails app | `%15.10` | GraphQL changes to `CiRunner` type. | -| GitLab Rails app | `%15.10` | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) | +| GitLab Runner | `%15.10` | Make the `gitlab-runner register` command happen in a single operation. | +| GitLab Rails app | `%15.10` | Define feature flag and policies for "New Runner creation workflow" for groups and projects. | +| GitLab Rails app | `%15.10` | Only update runner `contacted_at` and `status` when polled for jobs. | +| GitLab Rails app | `%15.10` | Add GraphQL type to represent runner machines under `CiRunner`. | +| GitLab Rails app | `%15.10` | Implement UI to create new instance runner. | +| GitLab Rails app | `%15.11` | Update service and mutation to accept groups and projects. | +| GitLab Rails app | `%15.11` | Implement UI to create new group/project runners. | +| GitLab Rails app | `%15.11` | GraphQL changes to `CiRunner` type. (?) | +| GitLab Rails app | `%15.11` | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) | | GitLab Rails app | `%15.11` | Adapt `POST /api/v4/runners` REST endpoint to accept a request from an authorized user with a scope instead of a registration token. | +| GitLab Runner | `%15.11` | Handle glrt- runner tokens in `unregister` command. | ### Stage 5 - Optional disabling of registration token | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Rails app | `%15.11` | Adapt `register_{group|project}_runner` permissions to take [application setting](https://gitlab.com/gitlab-org/gitlab/-/issues/386712) in consideration. | -| GitLab Rails app | `%15.11` | Add UI to allow disabling use of registration tokens at project or group level. | -| GitLab Rails app | `%15.11` | Introduce `:enforce_create_runner_workflow` feature flag (disabled by default) to control whether use of registration tokens is allowed. | -| GitLab Rails app | `%15.11` | Make [`POST /api/v4/runners` endpoint](../../../api/runners.md#register-a-new-runner) permanently return `HTTP 410 Gone` if either `allow_runner_registration_token` setting or `:enforce_create_runner_workflow` feature flag disables registration tokens.<br/>A future v5 version of the API should return `HTTP 404 Not Found`. | -| GitLab Rails app | `%15.11` | Start refusing job requests that don't include a unique ID, if either `allow_runner_registration_token` setting or `:enforce_create_runner_workflow` feature flag disables registration tokens. | -| GitLab Rails app | `%15.11` | Hide legacy UI showing registration with a registration token, if `:enforce_create_runner_workflow` feature flag disables registration tokens. | +| GitLab Rails app | | Adapt `register_{group|project}_runner` permissions to take [application setting](https://gitlab.com/gitlab-org/gitlab/-/issues/386712) in consideration. | +| GitLab Rails app | | Add UI to allow disabling use of registration tokens at project or group level. | +| GitLab Rails app | | Introduce `:enforce_create_runner_workflow` feature flag (disabled by default) to control whether use of registration tokens is allowed. | +| GitLab Rails app | | Make [`POST /api/v4/runners` endpoint](../../../api/runners.md#register-a-new-runner) permanently return `HTTP 410 Gone` if either `allow_runner_registration_token` setting or `:enforce_create_runner_workflow` feature flag disables registration tokens.<br/>A future v5 version of the API should return `HTTP 404 Not Found`. | +| GitLab Rails app | | Start refusing job requests that don't include a unique ID, if either `allow_runner_registration_token` setting or `:enforce_create_runner_workflow` feature flag disables registration tokens. | +| GitLab Rails app | | Hide legacy UI showing registration with a registration token, if `:enforce_create_runner_workflow` feature flag disables registration tokens. | ### Stage 6 - Enforcement | Component | Milestone | Changes | |------------------|----------:|---------| -| GitLab Runner | `%16.0` | Do not allow runner to start if `.runner_system_id` file cannot be written. | | GitLab Rails app | `%16.6` | Enable `:enforce_create_runner_workflow` feature flag by default. | | GitLab Rails app | `%16.6` | Start reject job requests that don't include `system_id` value. | @@ -495,7 +507,7 @@ gitlab-runner register --executor "shell" \ --url "https://gitlab.com/" \ --non-interactive \ - --registration-token="grlt-2CR8_eVxiioB1QmzPZwa" + --registration-token="glrt-2CR8_eVxiioB1QmzPZwa" ``` ### How does this change impact auto-scaling scenarios? diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 1aa579b842a..d7c5b089116 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -139,13 +139,14 @@ You can override cache settings without overwriting the global cache by using `policy` for one job: ```yaml -cache: &global_cache - key: $CI_COMMIT_REF_SLUG - paths: - - node_modules/ - - public/ - - vendor/ - policy: pull-push +default: + cache: &global_cache + key: $CI_COMMIT_REF_SLUG + paths: + - node_modules/ + - public/ + - vendor/ + policy: pull-push job: cache: @@ -570,7 +571,7 @@ You can clear the cache in the GitLab UI: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the upper right, select **Clear runner caches**. +1. In the upper-right corner, select **Clear runner caches**. On the next commit, your CI/CD jobs use a new cache. diff --git a/doc/ci/chatops/img/gitlab-chatops-icon-small.png b/doc/ci/chatops/img/gitlab-chatops-icon-small.png Binary files differdeleted file mode 100644 index 71cc5dba5cf..00000000000 --- a/doc/ci/chatops/img/gitlab-chatops-icon-small.png +++ /dev/null diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index f0efb5fc884..bb3c8d537f3 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -9,62 +9,70 @@ type: index, concepts, howto > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4466) in GitLab Ultimate 10.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) to GitLab Free in 11.9. -> - `CHAT_USER_ID` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341798) in GitLab 14.4. +> - `CHAT_USER_ID` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341798) in GitLab 14.4. -GitLab ChatOps provides a method to interact with CI/CD jobs through chat services -like Slack. Many organizations' discussion, collaboration, and troubleshooting takes -place in chat services. Having a method to run CI/CD jobs with output -posted back to the channel can significantly augment your team's workflow. +Use GitLab ChatOps to interact with CI/CD jobs through chat services +like Slack. -## How GitLab ChatOps works +Many organizations use chat services to collaborate, troubleshoot, and plan work. With ChatOps, +you can discuss work with your team, run CI/CD jobs, and view job output, all from the same +application. -GitLab ChatOps is built upon [GitLab CI/CD](../index.md) and -[Slack Slash Commands](../../user/project/integrations/slack_slash_commands.md). -ChatOps provides a `run` action for [slash commands](../../integration/slash_commands.md) -with the following arguments: +## ChatOps workflow and CI/CD configuration -- A `<job name>` to execute. -- The `<job arguments>`. +ChatOps looks for the specified job in the +[`.gitlab-ci.yml`](../yaml/index.md) on the project's default +branch. If the job is found, ChatOps creates a pipeline that contains +only the specified job. If you set `when: manual`, ChatOps creates the +pipeline, but the job doesn't start automatically. + +A job run with ChatOps has the same functionality as a job run from +GitLab. The job can use existing [CI/CD variables](../variables/index.md#predefined-cicd-variables) like +`GITLAB_USER_ID` to perform additional rights validation, but these +variables can be [overridden](../variables/index.md#cicd-variable-precedence). + +You should set [`rules`](../yaml/index.md#rules) so the job does not +run as part of the standard CI/CD pipeline. ChatOps passes the following [CI/CD variables](../variables/index.md#predefined-cicd-variables) to the job: -- `CHAT_INPUT` contains any additional arguments. -- `CHAT_CHANNEL` is set to the name of channel the action was triggered in. -- `CHAT_USER_ID` is set to the chat service's user ID of the user who triggered the slash command. +- `CHAT_INPUT` - The arguments passed to `/project-name run`. +- `CHAT_CHANNEL` - The name of the chat channel the job is run from. +- `CHAT_USER_ID` - The chat service ID of the user who runs the job. -When executed, ChatOps looks up the specified job name and attempts to match it -to a corresponding job in [`.gitlab-ci.yml`](../yaml/index.md). If a matching job -is found on the default branch, a pipeline containing only that job is scheduled. After the -job completes: +When the job runs: -- If the job completes in *less than 30 minutes*, the ChatOps sends the job's output to Slack. -- If the job completes in *more than 30 minutes*, the job must use the +- If the job completes in less than 30 minutes, ChatOps sends the job output to the chat channel. +- If the job completes in more than 30 minutes, you must use a method like the [Slack API](https://api.slack.com/) to send data to the channel. -To use the `run` command, you must have at least the -Developer role. -If a job shouldn't be able to be triggered from chat, you can set the job to `except: [chat]`. +## Run a CI/CD job + +You can run a CI/CD job from chat with the `/project-name run` +[slash command](../../integration/slash_commands.md). + +Prerequisites: + +- You must have at least the Developer role for the project. + +To run a CI/CD job: + +- In the chat client, enter `/project-name run <job name> <arguments>`. -## Best practices for ChatOps CI jobs +ChatOps schedules a pipeline that contains only the specified job. -Since ChatOps is built upon GitLab CI/CD, the job has all the same features and -functions available. Consider these best practices when creating ChatOps jobs: +### Exclude a job from ChatOps -- GitLab strongly recommends you set [`rules`](../yaml/index.md#rules) so the job does not run as part - of the standard CI pipeline. -- If the job is set to `when: manual`, ChatOps creates the pipeline, but the job waits to be started. -- ChatOps provides limited support for access control. If the user triggering the - slash command has at least the Developer role - in the project, the job runs. The job itself can use existing - [CI/CD variables](../variables/index.md#predefined-cicd-variables) like - `GITLAB_USER_ID` to perform additional rights validation, but - these variables can be [overridden](../variables/index.md#cicd-variable-precedence). +To prevent a job from being run from chat: -### Controlling the ChatOps reply +- In `.gitlab-ci.yml`, set the job to `except: [chat]`. -The output for jobs with a single command is sent to the channel as a reply. For -example, the chat reply of the following job is `Hello World` in the channel: +## Customize the ChatOps reply + +ChatOps sends the output for a job with a single command to the +channel as a reply. For example, when the following job runs, +the chat reply is `Hello world`: ```yaml stages: @@ -78,13 +86,12 @@ hello-world: - echo "Hello World" ``` -Jobs that contain multiple commands (or `before_script`) return additional -content in the chat reply. In these cases, both the commands and their output are -included, with the commands wrapped in ANSI color codes. +If the job contains multiple commands, or if `before_script` is set, ChatOps sends the commands +and their output to the channel. The commands are wrapped in ANSI color codes. -To selectively reply with the output of one command, its output must be bounded by -the `chat_reply` section. For example, the following job lists the files in the -current directory: +To selectively reply with the output of one command, place the output +in a `chat_reply` section. For example, the following job lists the +files in the current directory: ```yaml stages: @@ -99,28 +106,8 @@ ls: - echo -e "section_start:$( date +%s ):chat_reply\r\033[0K\n$( ls -la )\nsection_end:$( date +%s ):chat_reply\r\033[0K" ``` -## GitLab ChatOps examples - -The GitLab.com team created a repository of [common ChatOps scripts](https://gitlab.com/gitlab-com/chatops) -they use to interact with our Production instance of GitLab. Administrators of -other GitLab instances may find them useful. They can serve as inspiration for ChatOps -scripts you can write to interact with your own applications. - -## GitLab ChatOps icon - -The [official GitLab ChatOps icon](img/gitlab-chatops-icon.png) is available for download. -You can find and download the official GitLab ChatOps icon here. - - - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +## Related topics -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +- Download the [official GitLab ChatOps icon](img/gitlab-chatops-icon.png). +- The GitLab team maintains a [repository of common ChatOps scripts](https://gitlab.com/gitlab-com/chatops) + they use to interact with GitLab.com. diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index 920e7cca2cb..71210467018 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/cloud_deployment/heroku.md b/doc/ci/cloud_deployment/heroku.md index 541ba89dca8..7c15bf5b8cd 100644 --- a/doc/ci/cloud_deployment/heroku.md +++ b/doc/ci/cloud_deployment/heroku.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index a8826a6fdb5..79684d9d627 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/cloud_services/aws/index.md b/doc/ci/cloud_services/aws/index.md index 484a159cd2b..733105e9ddf 100644 --- a/doc/ci/cloud_services/aws/index.md +++ b/doc/ci/cloud_services/aws/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/cloud_services/azure/index.md b/doc/ci/cloud_services/azure/index.md index 321f9849632..c99e30e8bac 100644 --- a/doc/ci/cloud_services/azure/index.md +++ b/doc/ci/cloud_services/azure/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/cloud_services/google_cloud/index.md b/doc/ci/cloud_services/google_cloud/index.md index 516a2d05cd1..bcb2681403a 100644 --- a/doc/ci/cloud_services/google_cloud/index.md +++ b/doc/ci/cloud_services/google_cloud/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/cloud_services/index.md b/doc/ci/cloud_services/index.md index 9304e3562d9..88e7d88e22a 100644 --- a/doc/ci/cloud_services/index.md +++ b/doc/ci/cloud_services/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index a8c192dc944..674f9b6035a 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -9,12 +9,12 @@ type: concepts, howto You can use GitLab CI/CD with Docker to create Docker images. For example, you can create a Docker image of your application, -test it, and publish it to a container registry. +test it, and push it to a container registry. To run Docker commands in your CI/CD jobs, you must configure -GitLab Runner to support `docker` commands. +GitLab Runner to support `docker` commands. This method requires `privileged` mode. -If you want to build Docker images without enabling privileged mode on the runner, +If you want to build Docker images without enabling `privileged` mode on the runner, you can use a [Docker alternative](#docker-alternatives). ## Enable Docker commands in your CI/CD jobs @@ -58,7 +58,7 @@ the Docker commands, but needs permission to do so. sudo -u gitlab-runner -H docker info ``` -1. In GitLab, to verify that everything works, add `docker info` to `.gitlab-ci.yml`: +1. In GitLab, add `docker info` to `.gitlab-ci.yml` to verify that Docker is working: ```yaml before_script: @@ -70,10 +70,10 @@ the Docker commands, but needs permission to do so. - docker run my-docker-image /script/to/run/tests ``` -You can now use `docker` commands (and install `docker-compose` if needed). +You can now use `docker` commands (and install Docker Compose if needed). -When you add `gitlab-runner` to the `docker` group, you are effectively granting `gitlab-runner` full root permissions. -For more information, see the [security of the `docker` group](https://blog.zopyx.com/on-docker-security-docker-group-considered-harmful/). +When you add `gitlab-runner` to the `docker` group, you effectively grant `gitlab-runner` full root permissions. +For more information, see [security of the `docker` group](https://blog.zopyx.com/on-docker-security-docker-group-considered-harmful/). ### Use Docker-in-Docker @@ -83,15 +83,15 @@ For more information, see the [security of the `docker` group](https://blog.zopy - The executor uses a [container image of Docker](https://hub.docker.com/_/docker/), provided by Docker, to run your CI/CD jobs. -The Docker image has all of the `docker` tools installed and can run +The Docker image includes all of the `docker` tools and can run the job script in context of the image in privileged mode. -We recommend you use Docker-in-Docker with TLS enabled, +You should use Docker-in-Docker with TLS enabled, which is supported by [GitLab.com shared runners](../runners/index.md). -You should always specify a specific version of the image, like `docker:20.10.16`. +You should always pin a specific version of the image, like `docker:20.10.16`. If you use a tag like `docker:stable`, you have no control over which version is used. -Unpredictable behavior can result, especially when new versions are released. +This can cause incompatibility problems when new versions are released. #### Use the Docker executor with Docker-in-Docker @@ -101,14 +101,11 @@ You can use the Docker executor to run jobs in a Docker container. > Introduced in GitLab Runner 11.11. -The Docker daemon supports connections over TLS. In Docker 19.03.12 and later, -TLS is the default. +The Docker daemon supports connections over TLS. TLS is the default in Docker 19.03.12 and later. WARNING: -This task enables `--docker-privileged`. When you do this, you are effectively disabling all of -the security mechanisms of containers and exposing your host to privilege -escalation. Doing this can lead to container breakout. For more information, -see the official Docker documentation about +This task enables `--docker-privileged`, which effectively disables the container's security mechanisms and exposes your host to privilege +escalation. This action can cause container breakout. For more information, see [runtime privilege and Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). To use Docker-in-Docker with TLS enabled: @@ -134,10 +131,9 @@ To use Docker-in-Docker with TLS enabled: you must always use `privileged = true` in your Docker containers. - This command mounts `/certs/client` for the service and build container, which is needed for the Docker client to use the - certificates in that directory. For more information on how - Docker with TLS works, see <https://hub.docker.com/_/docker/#tls>. + certificates in that directory. For more information, see [the Docker image documentation](https://hub.docker.com/_/docker/). - The previous command creates a `config.toml` entry similar to this: + The previous command creates a `config.toml` entry similar to the following example: ```toml [[runners]] @@ -155,7 +151,7 @@ To use Docker-in-Docker with TLS enabled: [runners.cache.gcs] ``` -1. You can now use `docker` in the job script. Note the inclusion of the +1. You can now use `docker` in the job script. You should include the `docker:20.10.16-dind` service: ```yaml @@ -193,7 +189,7 @@ To use Docker-in-Docker with TLS enabled: ##### Docker-in-Docker with TLS disabled in the Docker executor -Sometimes you might have legitimate reasons to disable TLS. +Sometimes there are legitimate reasons to disable TLS. For example, you have no control over the GitLab Runner configuration that you are using. @@ -215,7 +211,7 @@ Assuming that the runner's `config.toml` is similar to: [runners.cache.gcs] ``` -You can now use `docker` in the job script. Note the inclusion of the +You can now use `docker` in the job script. You should include the `docker:20.10.16-dind` service: ```yaml @@ -254,7 +250,7 @@ build: #### Use the Kubernetes executor with Docker-in-Docker -You can use the Kubernetes executor to run jobs in a Docker container. +You can use the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html) to run jobs in a Docker container. ##### Docker-in-Docker with TLS enabled in Kubernetes @@ -280,7 +276,7 @@ To use Docker-in-Docker with TLS enabled in Kubernetes: medium = "Memory" ``` -1. You can now use `docker` in the job script. Note the inclusion of the +1. You can now use `docker` in the job script. You should include the `docker:20.10.16-dind` service: ```yaml @@ -324,25 +320,23 @@ To use Docker-in-Docker with TLS enabled in Kubernetes: - docker run my-docker-image /script/to/run/tests ``` -#### Limitations of Docker-in-Docker +#### Known issues with Docker-in-Docker -Docker-in-Docker is the recommended configuration, but is -not without its own challenges: +Docker-in-Docker is the recommended configuration, but you should be aware of the following issues: - **The `docker-compose` command**: This command is not available in this configuration by default. - To use `docker-compose` in your job scripts, follow the `docker-compose` + To use `docker-compose` in your job scripts, follow the Docker Compose [installation instructions](https://docs.docker.com/compose/install/). -- **Cache**: Each job runs in a new environment. Concurrent jobs work fine, - because every build gets its own instance of Docker engine and they don't conflict with each other. - However, jobs can be slower because there's no caching of layers. +- **Cache**: Each job runs in a new environment. Because every build gets its own instance of the Docker engine, concurrent jobs do not cause conflicts. + However, jobs can be slower because there's no caching of layers. See [Docker layer caching](#make-docker-in-docker-builds-faster-with-docker-layer-caching). - **Storage drivers**: By default, earlier versions of Docker use the `vfs` storage driver, which copies the file system for each job. Docker 17.09 and later use `--storage-driver overlay2`, which is the recommended storage driver. See [Using the OverlayFS driver](#use-the-overlayfs-driver) for details. -- **Root file system**: Because the `docker:20.10.16-dind` container and the runner container don't share their +- **Root file system**: Because the `docker:20.10.16-dind` container and the runner container do not share their root file system, you can use the job's working directory as a mount point for child containers. For example, if you have files you want to share with a - child container, you might create a subdirectory under `/builds/$CI_PROJECT_PATH` - and use it as your mount point. For a more detailed explanation, view + child container, you could create a subdirectory under `/builds/$CI_PROJECT_PATH` + and use it as your mount point. For a more detailed explanation, see [issue #41227](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227). ```yaml @@ -362,16 +356,16 @@ NOTE: If you bind the Docker socket and you are [using GitLab Runner 11.11 or later](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1261), you can no longer use `docker:20.10.16-dind` as a service. Volume bindings -are done to the services as well, making these incompatible. +also affect services, making them incompatible. #### Use the Docker executor with Docker socket binding To make Docker available in the context of the image, you need to mount `/var/run/docker.sock` into the launched containers. To do this with the Docker -executor, you need to add `"/var/run/docker.sock:/var/run/docker.sock"` to the +executor, add `"/var/run/docker.sock:/var/run/docker.sock"` to the [Volumes in the `[runners.docker]` section](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). -Your configuration should look something like this: +Your configuration should look similar to this example: ```toml [[runners]] @@ -388,7 +382,7 @@ Your configuration should look something like this: Insecure = false ``` -You can also do this while registering your runner by providing the following options: +To mount `/var/run/docker.sock` while registering your runner, include the following options: ```shell sudo gitlab-runner register -n \ @@ -402,10 +396,10 @@ sudo gitlab-runner register -n \ ##### Enable registry mirror for `docker:dind` service -When the Docker daemon starts inside of the service container, it uses -the default configuration. You may want to configure a +When the Docker daemon starts inside the service container, it uses +the default configuration. You might want to configure a [registry mirror](https://docs.docker.com/registry/recipes/mirror/) for -performance improvements and to ensure you don't reach Docker Hub rate limits. +performance improvements and to ensure you do not exceed Docker Hub rate limits. ###### The service in the `.gitlab-ci.yml` file @@ -474,9 +468,9 @@ content: ``` Update the `config.toml` file to mount the file to -`/etc/docker/daemon.json`. This would mount the file for **every** -container that is created by GitLab Runner. The configuration is -picked up by the `dind` service. +`/etc/docker/daemon.json`. This mounts the file for **every** +container created by GitLab Runner. The configuration is +detected by the `dind` service. ```toml [[runners]] @@ -516,13 +510,12 @@ kubectl create configmap docker-daemon --namespace gitlab-runner --from-file /tm ``` NOTE: -Make sure to use the namespace that the Kubernetes executor for GitLab Runner uses -to create job pods in. +You must use the namespace that the Kubernetes executor for GitLab Runner uses to create job pods. After the ConfigMap is created, you can update the `config.toml` file to mount the file to `/etc/docker/daemon.json`. This update -mounts the file for **every** container that is created by GitLab Runner. -The configuration is picked up by the `dind` service. +mounts the file for **every** container created by GitLab Runner. +The `dind` service detects this configuration. ```toml [[runners]] @@ -537,22 +530,21 @@ The configuration is picked up by the `dind` service. sub_path = "daemon.json" ``` -##### Limitations of Docker socket binding +##### Known issues with Docker socket binding When you use Docker socket binding, you avoid running Docker in privileged mode. However, the implications of this method are: -- By sharing the Docker daemon, you are effectively disabling all - the security mechanisms of containers and exposing your host to privilege - escalation, which can lead to container breakout. For example, if a project - ran `docker rm -f $(docker ps -a -q)` it would remove the GitLab Runner +- By sharing the Docker daemon, you effectively disable all + the container's security mechanisms and expose your host to privilege + escalation. This can cause container breakout. For example, if a project + ran `docker rm -f $(docker ps -a -q)`, it would remove the GitLab Runner containers. -- Concurrent jobs may not work; if your tests - create containers with specific names, they may conflict with each other. -- Any containers spawned by Docker commands are siblings of the runner rather - than children of the runner. This may have complications and limitations that - are unsuitable for your workflow. -- Sharing files and directories from the source repository into containers may not +- Concurrent jobs might not work. If your tests + create containers with specific names, they might conflict with each other. +- Any containers created by Docker commands are siblings of the runner, rather + than children of the runner. This might cause complications for your workflow. +- Sharing files and directories from the source repository into containers might not work as expected. Volume mounting is done in the context of the host machine, not the build container. For example: @@ -560,8 +552,8 @@ the implications of this method are: docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` -You don't need to include the `docker:20.10.16-dind` service, like you do when -you're using the Docker-in-Docker executor: +You do not need to include the `docker:20.10.16-dind` service, like you do when +you use the Docker-in-Docker executor: ```yaml image: docker:20.10.16 @@ -580,7 +572,7 @@ build: When you use Docker-in-Docker, the [standard authentication methods](using_docker_images.md#access-an-image-from-a-private-container-registry) -don't work because a fresh Docker daemon is started with the service. +do not work, because a fresh Docker daemon is started with the service. ### Option 1: Run `docker login` @@ -612,14 +604,14 @@ empty or remove it. If you are an administrator for GitLab Runner, you can mount a file with the authentication configuration to `~/.docker/config.json`. -Then every job that the runner picks up is authenticated already. If you +Then every job that the runner picks up is already authenticated. If you are using the official `docker:20.10.16` image, the home directory is under `/root`. If you mount the configuration file, any `docker` command that modifies the `~/.docker/config.json` fails. For example, `docker login` fails, because the file is mounted as read-only. Do not change it from -read-only, because problems occur. +read-only, because this causes problems. Here is an example of `/opt/.docker/config.json` that follows the [`DOCKER_AUTH_CONFIG`](using_docker_images.md#determine-your-docker_auth_config-data) @@ -685,7 +677,7 @@ If you already have defined, you can use the variable and save it in `~/.docker/config.json`. -There are multiple ways to define this authentication: +You can define this authentication in several ways: - In [`pre_build_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) in the runner configuration file. @@ -718,24 +710,23 @@ build: When using Docker-in-Docker, Docker downloads all layers of your image every time you create a build. Recent versions of Docker (Docker 1.13 and later) can -use a pre-existing image as a cache during the `docker build` step. This considerably -speeds up the build process. +use a pre-existing image as a cache during the `docker build` step. This significantly +accelerates the build process. ### How Docker caching works -When running `docker build`, each command in `Dockerfile` results in a layer. -These layers are kept around as a cache and can be reused if there haven't been -any changes. Change in one layer causes all subsequent layers to be recreated. +When running `docker build`, each command in `Dockerfile` creates a layer. +These layers are retained as a cache and can be reused if there have been no changes. Change in one layer causes the recreation of all subsequent layers. -You can specify a tagged image to be used as a cache source for the `docker build` -command by using the `--cache-from` argument. Multiple images can be specified +To specify a tagged image to be used as a cache source for the `docker build` +command, use the `--cache-from` argument. Multiple images can be specified as a cache source by using multiple `--cache-from` arguments. Any image that's used -with the `--cache-from` argument must first be pulled +with the `--cache-from` argument must be pulled (using `docker pull`) before it can be used as a cache source. ### Docker caching example -Here's a `.gitlab-ci.yml` file that shows how to use Docker caching: +This example `.gitlab-ci.yml` file shows how to use Docker caching: ```yaml image: docker:20.10.16 @@ -768,35 +759,33 @@ In the `script` section for the `build` stage: cache (see the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if available, and tags it. 1. The last two commands push the tagged Docker images to the container registry - so that they may also be used as cache for subsequent builds. + so that they can also be used as cache for subsequent builds. ## Use the OverlayFS driver NOTE: The shared runners on GitLab.com use the `overlay2` driver by default. -By default, when using `docker:dind`, Docker uses the `vfs` storage driver which -copies the file system on every run. This is a disk-intensive operation -which can be avoided if a different driver is used, for example `overlay2`. +By default, when using `docker:dind`, Docker uses the `vfs` storage driver, which +copies the file system on every run. You can avoid this disk-intensive operation by using a different driver, for example `overlay2`. ### Requirements -1. Make sure a recent kernel is used, preferably `>= 4.2`. +1. Ensure a recent kernel is used, preferably `>= 4.2`. 1. Check whether the `overlay` module is loaded: ```shell sudo lsmod | grep overlay ``` - If you see no result, then it isn't loaded. To load it use: + If you see no result, then the module is not loaded. To load the module, use: ```shell sudo modprobe overlay ``` - If everything went fine, you need to make sure module is loaded on reboot. - On Ubuntu systems, this is done by editing `/etc/modules`. Just add the - following line into it: + If the module loaded, you must make sure the module loads on reboot. + On Ubuntu systems, do this by adding the following line to `/etc/modules`: ```plaintext overlay @@ -823,7 +812,7 @@ environment variable in the environment = ["DOCKER_DRIVER=overlay2"] ``` -If you're running multiple runners, you have to modify all configuration files. +If you're running multiple runners, you must modify all configuration files. Read more about the [runner configuration](https://docs.gitlab.com/runner/configuration/) and [using the OverlayFS storage driver](https://docs.docker.com/storage/storagedriver/overlayfs-driver/). @@ -833,8 +822,8 @@ and [using the OverlayFS storage driver](https://docs.docker.com/storage/storage To build Docker images without enabling privileged mode on the runner, you can use one of these alternatives: -- [`kaniko`](using_kaniko.md). -- [`buildah`](https://github.com/containers/buildah). +- [`kaniko`](using_kaniko.md) +- [`buildah`](https://github.com/containers/buildah) For example, with `buildah`: @@ -868,7 +857,7 @@ build: ## Use the GitLab Container Registry -After you've built a Docker image, you can push it up to the built-in +After you've built a Docker image, you can push it to the [GitLab Container Registry](../../user/packages/container_registry/build_and_push_images.md#use-gitlab-cicd). ## Troubleshooting @@ -879,18 +868,18 @@ This is a common error when you are using [Docker-in-Docker](#use-docker-in-docker) v19.03 or later. -This issue occurs because Docker starts on TLS automatically. +This error occurs because Docker starts on TLS automatically. -- If this is your first time setting it up, read +- If this is your first time setting it up, see [use the Docker executor with the Docker image](#use-docker-in-docker). -- If you are upgrading from v18.09 or earlier, read our +- If you are upgrading from v18.09 or earlier, see the [upgrade guide](https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/). -This error can also occur with the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind) when attempts are made to access the Docker-in-Docker service before it has had time to fully start up. For a more detailed explanation, see [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27215). +This error can also occur with the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind) when attempts are made to access the Docker-in-Docker service before it has fully started up. For a more detailed explanation, see [issue 27215](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27215). ### Docker `no such host` error -You may get an error that says +You might get an error that says `docker: error during connect: Post https://docker:2376/v1.40/containers/create: dial tcp: lookup docker on x.x.x.x:53: no such host`. This issue can occur when the service's image name diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index 089ecefb373..c4f685b18ec 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Require approvals prior to deploying to a Protected Environment --- diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index cf82238564e..327118c8a86 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index 479b783202d..2ee0dc2ee11 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/environments/external_deployment_tools.md b/doc/ci/environments/external_deployment_tools.md index ff3172f0e02..89a1f8565a9 100644 --- a/doc/ci/environments/external_deployment_tools.md +++ b/doc/ci/environments/external_deployment_tools.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -42,45 +42,45 @@ Here is an example setup that creates a `success` deployment record in GitLab wh 1. Create a new webhook. You can save the following manifest file and apply it by `kubectl apply -n argocd -f <manifiest-file-path>`: - ```yaml - apiVersion: v1 - kind: ConfigMap - metadata: - name: argocd-notifications-cm - data: - trigger.on-deployed: | - - description: Application is synced and healthy. Triggered once per commit. - oncePer: app.status.sync.revision - send: - - gitlab-deployment-status - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' - template.gitlab-deployment-status: | - webhook: - gitlab: - method: POST - path: /projects/<your-project-id>/deployments - body: | - { - "status": "success", - "environment": "production", - "sha": "{{.app.status.operationState.operation.sync.revision}}", - "ref": "main", - "tag": "false" - } - service.webhook.gitlab: | - url: https://gitlab.com/api/v4 - headers: - - name: PRIVATE-TOKEN - value: <your-access-token> - - name: Content-type - value: application/json - ``` + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: argocd-notifications-cm + data: + trigger.on-deployed: | + - description: Application is synced and healthy. Triggered once per commit. + oncePer: app.status.sync.revision + send: + - gitlab-deployment-status + when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' + template.gitlab-deployment-status: | + webhook: + gitlab: + method: POST + path: /projects/<your-project-id>/deployments + body: | + { + "status": "success", + "environment": "production", + "sha": "{{.app.status.operationState.operation.sync.revision}}", + "ref": "main", + "tag": "false" + } + service.webhook.gitlab: | + url: https://gitlab.com/api/v4 + headers: + - name: PRIVATE-TOKEN + value: <your-access-token> + - name: Content-type + value: application/json + ``` 1. Create a new subscription in your application: - ```shell - kubectl patch app <your-app-name> -n argocd -p '{"metadata": {"annotations": {"notifications.argoproj.io/subscribe.on-deployed.gitlab":""}}}' --type merge - ``` + ```shell + kubectl patch app <your-app-name> -n argocd -p '{"metadata": {"annotations": {"notifications.argoproj.io/subscribe.on-deployed.gitlab":""}}}' --type merge + ``` NOTE: If a deployment wasn't created as expected, you can troubleshoot with [`argocd-notifications` tool](https://argocd-notifications.readthedocs.io/en/stable/troubleshooting/). diff --git a/doc/ci/environments/img/environments_project_home.png b/doc/ci/environments/img/environments_project_home.png Binary files differindex 36b33e260f0..10d9de4f415 100644 --- a/doc/ci/environments/img/environments_project_home.png +++ b/doc/ci/environments/img/environments_project_home.png diff --git a/doc/ci/environments/incremental_rollouts.md b/doc/ci/environments/incremental_rollouts.md index 10cda68c4b5..8df2d8ead6c 100644 --- a/doc/ci/environments/incremental_rollouts.md +++ b/doc/ci/environments/incremental_rollouts.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 60450692794..c9b212c2915 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference disqus_identifier: 'https://docs.gitlab.com/ee/ci/environments.html' @@ -159,6 +159,19 @@ deploy_review_app: - main ``` +### Rename an environment + +> - Renaming an environment by using the UI was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68550) in GitLab 14.3. +> - Renaming an environment by using the API was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/338897) in GitLab 15.9 and is planned for removal in GitLab 16.0. + +You cannot rename an environment by using the UI, and the API method was deprecated in GitLab 15.9. + +To achieve the same result as renaming an environment: + +1. [Stop the existing environment](#stop-an-environment-through-the-ui). +1. [Delete the existing environment](#delete-an-environment). +1. [Create a new environment](#create-a-static-environment) with the desired name. + ## Deployment tier of environments > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. @@ -207,7 +220,7 @@ The `when: manual` action: You can find the play button in the pipelines, environments, deployments, and jobs views. -## Configure Kubernetes deployments (DEPRECATED) +## Configure Kubernetes deployments (deprecated) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -708,7 +721,7 @@ You can view a deployment's expiration date in the GitLab UI. 1. On the left sidebar, select **Deployments > Environments**. 1. Select the name of the deployment. -In the upper left, next to the environment name, the expiration date is displayed. +In the upper-left corner, next to the environment name, the expiration date is displayed. #### Override a deployment's scheduled stop time @@ -717,7 +730,7 @@ You can manually override a deployment's expiration date. 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Deployments > Environments**. 1. Select the deployment name. -1. in the upper right, select the thumbtack (**{thumbtack}**). +1. in the upper-right corner, select the thumbtack (**{thumbtack}**).  @@ -866,7 +879,7 @@ It may take a minute or two for data to appear after initial deployment. Metric charts can be embedded in GitLab Flavored Markdown. See [Embedding Metrics in GitLab Flavored Markdown](../../operations/metrics/embed.md) for more details. -### Web terminals (DEPRECATED) +### Web terminals (deprecated) > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -991,19 +1004,6 @@ like [Review Apps](../review_apps/index.md) (`review/*`). The most specific spec takes precedence over the other wildcard matching. In this case, the `review/feature-1` spec takes precedence over `review/*` and `*` specs. -### Rename an environment - -> Renaming environments through the UI was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68550) in GitLab 14.3. Renaming environments through the API was deprecated and [is planned to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/338897) in GitLab 15.0. - -Renaming an environment through the UI is not possible. -Instead, you need to delete the old environment and create a new one: - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Deployments > Environments**. -1. Find the environment and stop it. -1. Delete the environment. -1. Create a new environment with your preferred name. - ## Related topics - [Use GitLab CI to deploy to multiple environments (blog post)](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index fc49be798ec..c81a5ab1378 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 7bb14c4c900..6295c131fb9 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index 533b6519d9a..81e70e5cf82 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md index 67b4ec6b279..74b4581af99 100644 --- a/doc/ci/examples/deployment/index.md +++ b/doc/ci/examples/deployment/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 7b5690e2d3d..3385aca1ef2 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments author: Vincent Tunru author_gitlab: Vinnl diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md index 6738c55b6fa..b05c872d624 100644 --- a/doc/ci/examples/index.md +++ b/doc/ci/examples/index.md @@ -76,7 +76,7 @@ choose one of these templates: - [dotNET Core (`dotNET-Core.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/dotNET-Core.gitlab-ci.yml) - [Elixir (`Elixir.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Elixir.gitlab-ci.yml) - [Flutter (`Flutter.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Flutter.gitlab-ci.yml) -- [Golang (`Go.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml) +- [Go (`Go.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml) - [Gradle (`Gradle.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml) - [Grails (`Grails.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Grails.gitlab-ci.yml) - [iOS with fastlane (`iOS-Fastlane.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/iOS-Fastlane.gitlab-ci.yml) diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 062bd602c29..d56f14617db 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -126,7 +126,7 @@ We'll use this variable in the `.gitlab-ci.yml` later, to easily connect to our  -We also need to add the public key to **Project** > **Settings** > **Repository** as a [Deploy Key](../../../user/project/deploy_keys/index.md), which gives us the ability to access our repository from the server through [SSH protocol](../../../gitlab-basics/command-line-commands.md#start-working-on-your-project). +We also need to add the public key to **Project** > **Settings** > **Repository** as a [Deploy Key](../../../user/project/deploy_keys/index.md), which gives us the ability to access our repository from the server through the SSH protocol. ```shell # As the deployer user on the server diff --git a/doc/ci/examples/semantic-release.md b/doc/ci/examples/semantic-release.md index 9f299448d52..0b4b2054610 100644 --- a/doc/ci/examples/semantic-release.md +++ b/doc/ci/examples/semantic-release.md @@ -90,7 +90,7 @@ As part of publishing a package, semantic-release increases the version number i <!-- markdownlint-disable MD044 --> -1. On the top bar, in the upper right, select your avatar. +1. On the top bar, in the upper-right corner, select your avatar. 1. On the left sidebar, select **Access Tokens**. 1. In the **Token name** box, enter a token name. 1. Under **select scopes**, select the **api** checkbox. diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index e62c38fc1ec..bf07af5e761 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." type: concepts diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index d9cfbdf124e..db916d8d125 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -65,7 +65,9 @@ tries to steal tokens from other jobs. You can control what projects a CI/CD job token can access to increase the job token's security. A job token might give extra permissions that aren't necessary -to access specific private resources. +to access specific private resources. The job token scope only controls access +to private projects. If an accessed project is public or internal, token scoping does +not apply. If a job token is leaked, it could potentially be used to access private data to the job token's user. By limiting the job token access scope, private data cannot @@ -78,14 +80,15 @@ see [epic 3559](https://gitlab.com/groups/gitlab-org/-/epics/3559). > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346298/) in GitLab 15.9. [Deployed behind the `:inbound_ci_scoped_job_token` feature flag](../../user/feature_flags.md), enabled by default. -Control your project's job token scope by creating an **inbound** allowlist of projects which can -access your project through its `CI_JOB_TOKEN`. +Create an **inbound** allowlist of projects which can access your project through +their `CI_JOB_TOKEN`. -For example, you can add project `B` to the inbound allowlist for project `A`. Jobs -in the pipeline for "allowed project" `B` can now use the CI/CD job token to authenticate -API calls to access project `A`. +For example, project `A` can add project `B` to the inbound allowlist. CI/CD jobs +in project `B` (the "allowed project") can now use their CI/CD job token to +authenticate API calls to access project `A`. If project `A` is public or internal, +the project can be accessed by project `B` without adding it to the inbound allowlist. -By default the allowlist includes your current project. +By default the inbound allowlist of any project only includes itself. It is a security risk to disable this feature, so project maintainers or owners should keep this setting enabled at all times. Add projects to the allowlist only when cross-project @@ -113,6 +116,8 @@ To disable the inbound job token scope allowlist: 1. Toggle **Allow access to this project with a CI_JOB_TOKEN** to disabled. Enabled by default in new projects. +You can also disable the allowlist [with the API](../../api/graphql/reference/index.md#mutationprojectcicdsettingsupdate). + ### Add a project to the inbound job token scope allowlist You can add projects to the inbound allowlist for a project. Projects added to the allowlist @@ -133,6 +138,8 @@ To add a project: 1. Under **Allow CI job tokens from the following projects to access this project**, add projects to the allowlist. +You can also add a target project to the allowlist [with the API](../../api/graphql/reference/index.md#mutationcijobtokenscopeaddproject). + ### Limit your project's job token access > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. @@ -155,8 +162,8 @@ limited only by the user's access permissions. For example, when the setting is enabled, jobs in a pipeline in project `A` have a `CI_JOB_TOKEN` scope limited to project `A`. If the job needs to use the token to make an API request to a private project `B`, then `B` must be added to the allowlist for `A`. -If project `B` is public or internal, it's not required to be added to the allowlist. -The job token scope is only for controlling access to private projects. +If project `B` is public or internal, you do not need to add +`B` to the allowlist to grant access. ### Configure the outbound job token scope diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 5e69ecff7b8..b9c2ee409b8 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -339,8 +339,8 @@ In the example above: - `date +%s`: The Unix timestamp (for example `1560896352`). - `my_first_section`: The name given to the section. - `\r\e[0K`: Prevents the section markers from displaying in the rendered (colored) - job log, but they are displayed in the raw job log. To see them, in the upper right - of the job log, select **{doc-text}** (**Show complete raw**). + job log, but they are displayed in the raw job log. To see them, in the upper-right corner + of the job log, select **Show complete raw** (**{doc-text}**). - `\r`: carriage return. - `\e[0K`: clear line ANSI escape code. diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 5a553228ba1..f728e9b9e17 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -255,7 +255,7 @@ For very active repositories with a large number of references and files, you ca enabled on all Gitaly servers, we found that we no longer need a pre-clone step for `gitlab-org/gitlab` development. - Optimize your CI/CD jobs by seeding repository data in a pre-clone step with the [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See - [SaaS runners on Linux](../runners/saas/linux_saas_runner.md#pre-clone-script) for details. + [SaaS runners on Linux](../runners/saas/linux_saas_runner.md#pre-clone-script-deprecated) for details. Besides speeding up pipelines in large and active projects, seeding the repository data also helps avoid `429 Too many requests` errors from Cloudflare. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 74e0f0cd5ef..119a0e5853e 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -26,7 +26,7 @@ To check CI/CD configuration with the CI lint tool: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the upper right, select **CI lint**. +1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Validate**. @@ -47,7 +47,7 @@ To simulate a pipeline: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **CI/CD > Pipelines**. -1. In the upper right, select **CI lint**. +1. In the upper-right corner, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Simulate pipeline creation for the default branch**. 1. Select **Validate**. diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 727977b3bc8..1a7155c8195 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -70,7 +70,7 @@ simulations in the [CI Lint tool](../lint.md#simulate-a-pipeline). > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/357219) in GitLab 15.1. You can review configuration added with the [`include`](../yaml/index.md#include) -keyword in the pipeline editor. In the upper right, select the file tree (**{file-tree}**) +keyword in the pipeline editor. In the upper-right corner, select the file tree (**{file-tree}**) to see a list of all included configuration files. Selected files open in a new tab for review. diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index 4cbaa77b029..e1eb2f53910 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -19,7 +19,7 @@ end-to-end duration of a pipeline. On GitLab.com: -- CI/CD minutes quotas are enabled for both public and private projects, but public +- CI/CD minutes quotas are enabled for all projects, but certain projects [consume CI/CD minutes at a slower rate](#cost-factor). - The base monthly CI/CD minutes quota for a GitLab.com [namespace](../../user/namespace/index.md) is determined by its [license tier](https://about.gitlab.com/pricing/). @@ -201,13 +201,12 @@ can be higher than the end-to-end duration of a pipeline. The cost factors for jobs running on shared runners on GitLab.com are: -- `1` for internal and private projects. -- `0.5` for public projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). -- `0.008` for public forks of public projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). For every 125 minutes of job execution time, +- `1` for internal, public, and private projects. +- Exceptions for public projects: + - `0.5` for projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). + - `0.008` for forks of projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). For every 125 minutes of job execution time, you use 1 CI/CD minute. -- `1` for other public projects, after October 1, 2022 (previously `0.04`). - For every 1 minute of job execution time, you use 1 CI/CD minute. -- Calculated differently for [community contributions to GitLab projects](#cost-factor-for-community-contributions-to-gitlab-projects). +- Discounted dynamically for [community contributions to GitLab projects](#cost-factor-for-community-contributions-to-gitlab-projects). The cost factors on self-managed instances are: diff --git a/doc/ci/pipelines/downstream_pipelines.md b/doc/ci/pipelines/downstream_pipelines.md index e4560cd882d..1e4654e69fe 100644 --- a/doc/ci/pipelines/downstream_pipelines.md +++ b/doc/ci/pipelines/downstream_pipelines.md @@ -231,37 +231,43 @@ configuration for jobs that use the Windows runner, like scripts, use <code>\ ### Run child pipelines with merge request pipelines -To trigger a child pipeline as a [merge request pipeline](merge_request_pipelines.md): +Pipelines, including child pipelines, run as branch pipelines by default when not using +[`rules`](../yaml/index.md#rules) or [`workflow:rules`](../yaml/index.md#workflowrules). +To configure child pipelines to run when triggered from a [merge request (parent) pipeline](merge_request_pipelines.md), use `rules` or `workflow:rules`. +For example, using `rules`: -1. Set the trigger job to run on merge requests in the parent pipeline's configuration file: +1. Set the parent pipeline's trigger job to run on merge requests: ```yaml - microservice_a: + trigger-child-pipeline-job: trigger: - include: path/to/microservice_a.yml + include: path/to/child-pipeline-configuration.yml rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" ``` -1. Configure the child pipeline jobs to run in merge request pipelines with [`rules`](../yaml/index.md#rules) - or [`workflow:rules`](../yaml/index.md#workflowrules). - For example, with `rules` in a child pipeline's configuration file: +1. Use `rules` to configure the child pipeline jobs to run when triggered by the parent pipeline: ```yaml job1: - script: echo "Child pipeline job 1" + script: echo "This child pipeline job runs any time the parent pipeline triggers it." rules: - - if: $CI_MERGE_REQUEST_ID + - if: $CI_PIPELINE_SOURCE == "parent_pipeline" job2: - script: echo "Child pipeline job 2" + script: echo "This child pipeline job runs only when the parent pipeline is a merge request pipeline" rules: - if: $CI_MERGE_REQUEST_ID ``` - In child pipelines, `$CI_PIPELINE_SOURCE` always has a value of `parent_pipeline` - and cannot be used to identify merge request pipelines. Use `$CI_MERGE_REQUEST_ID` - instead, which is always present in merge request pipelines. +In child pipelines, `$CI_PIPELINE_SOURCE` always has a value of `parent_pipeline`, so: + +- You can use `if: $CI_PIPELINE_SOURCE == "parent_pipeline"` to ensure child pipeline jobs always run. +- You _can't_ use `if: $CI_PIPELINE_SOURCE == "merge_request_event"` to configure child pipeline + jobs to run for merge request pipelines. Instead, use `if: $CI_MERGE_REQUEST_ID` + to set child pipeline jobs to run only when the parent pipeline is a merge request pipeline. The parent pipeline's + [`CI_MERGE_REQUEST_*` predefined variables](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) + are passed to the child pipeline jobs. ### Specify a branch for multi-project pipelines @@ -281,7 +287,7 @@ Use: - The `project` keyword to specify the full path to the downstream project. In [GitLab 15.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367660), you can use [variable expansion](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). -- The `branch` keyword to specify the name of a branch or [tag](../../topics/git/tags.md) +- The `branch` keyword to specify the name of a branch or [tag](../../user/project/repository/tags/index.md) in the project specified by `project`. You can use variable expansion. ## Trigger a multi-project pipeline by using the API @@ -309,18 +315,32 @@ trigger_pipeline: > Hover behavior for pipeline cards [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/) in GitLab 13.2. In the [pipeline graph view](index.md#view-full-pipeline-graph), downstream pipelines display -as a list of cards on the right of the graph. Hover over the pipeline's card to view -which job triggered the downstream pipeline. +as a list of cards on the right of the graph. From this view, you can: -### Retry a downstream pipeline +- Select a trigger job to see the triggered downstream pipeline's jobs. +- Select **Expand jobs** **{chevron-lg-right}** on a pipeline card to expand the view + with the downstream pipeline's jobs. You can view one downstream pipeline at a time. +- Hover over a pipeline card to have the job that triggered the downstream pipeline highlighted. + +### Retry failed and canceled jobs in a downstream pipeline > - Retry from graph view [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354974) in GitLab 15.0 [with a flag](../../administration/feature_flags.md) named `downstream_retry_action`. Disabled by default. > - Retry from graph view [generally available and feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/357406) in GitLab 15.1. -To retry a completed downstream pipeline, select **Retry** (**{retry}**): +To retry failed and canceled jobs, select **Retry** (**{retry}**): - From the downstream pipeline's details page. -- On the pipeline's card in the [pipeline graph view](index.md#view-full-pipeline-graph). +- On the pipeline's card in the pipeline graph view. + +### Recreate a downstream pipeline + +> Retry trigger job from graph view [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367547) in GitLab 15.10 [with a flag](../../administration/feature_flags.md) named `ci_recreate_downstream_pipeline`. Disabled by default. + +You can recreate a downstream pipeline by retrying its corresponding trigger job. The newly created downstream pipeline replaces the current downstream pipeline in the pipeline graph. + +To recreate a downstream pipeline: + +- Select **Run again** (**{retry}**) on the trigger job's card in the pipeline graph view. ### Cancel a downstream pipeline @@ -330,7 +350,7 @@ To retry a completed downstream pipeline, select **Retry** (**{retry}**): To cancel a downstream pipeline that is still running, select **Cancel** (**{cancel}**): - From the downstream pipeline's details page. -- On the pipeline's card in the [pipeline graph view](index.md#view-full-pipeline-graph). +- On the pipeline's card in the pipeline graph view. ### Mirror the status of a downstream pipeline in the trigger job @@ -365,13 +385,9 @@ trigger_job: After you trigger a multi-project pipeline, the downstream pipeline displays to the right of the [pipeline graph](index.md#visualize-pipelines). - - In [pipeline mini graphs](index.md#pipeline-mini-graphs), the downstream pipeline displays to the right of the mini graph. - - ## Fetch artifacts from an upstream pipeline Use [`needs:project`](../yaml/index.md#needsproject) to fetch artifacts from an @@ -604,8 +620,9 @@ the ones defined in the upstream project take precedence. ### Pass dotenv variables created in a job **(PREMIUM)** -You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) -and [`needs:project`](../yaml/index.md#needsproject). +You can pass variables to a downstream job with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) +and [`needs:project`](../yaml/index.md#needsproject). These variables are only available in +the script of the job and can't be used to configure it, for example with `rules` or `artifact:paths`. For example, in a [multi-project pipeline](#multi-project-pipelines): @@ -656,6 +673,16 @@ With multi-project pipelines, the trigger job fails and does not create the down to run pipelines against the protected branch. See [pipeline security for protected branches](index.md#pipeline-security-on-protected-branches) for more information. +### Job in child pipeline is not created when the pipeline runs + +If the parent pipeline is a [merge request pipeline](merge_request_pipelines.md), +the child pipeline must [use `workflow:rules` or `rules` to ensure the jobs run](#run-child-pipelines-with-merge-request-pipelines). + +If no jobs in the child pipeline can run due to missing or incorrect `rules` configuration: + +- The child pipeline fails to start. +- The parent pipeline's trigger job fails with: `downstream pipeline can not be created, Pipeline will not run for the selected trigger. The rules configuration prevented any jobs from being added to the pipeline.` + ### `Ref is ambiguous` You cannot trigger a multi-project pipeline with a tag when a branch exists with the same diff --git a/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png b/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png Binary files differdeleted file mode 100644 index aadf8bb0979..00000000000 --- a/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png +++ /dev/null diff --git a/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png b/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png Binary files differdeleted file mode 100644 index 48a0ca9d84f..00000000000 --- a/doc/ci/pipelines/img/pipeline_mini_graph_v15_0.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index fa04cb6cb92..d2038dbcb62 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -423,8 +423,7 @@ You can group the jobs by: - [Job dependencies](#view-job-dependencies-in-the-pipeline-graph), which arranges jobs based on their [`needs`](../yaml/index.md#needs) dependencies. -[Multi-project pipeline graphs](downstream_pipelines.md#view-multi-project-pipelines-in-pipeline-graphs) help -you visualize the entire pipeline, including all cross-project inter-dependencies. +Multi-project pipeline graphs help you visualize the entire pipeline, including all cross-project inter-dependencies. If a stage contains more than 100 jobs, only the first 100 jobs are listed in the pipeline graph. The remaining jobs still run as usual. To see the jobs: diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 75c9852d3d0..7db4596bb1d 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' --- @@ -264,7 +264,7 @@ artifacts and log. You must be: To delete a job: 1. Go to a job's detail page. -1. In the upper right of the job's log, select **Erase job log** (**{remove}**). +1. In the upper-right corner of the job's log, select **Erase job log** (**{remove}**). 1. On the confirmation dialog, select **OK**. ## Expose job artifacts in the merge request UI diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 7e18c11f234..881e94a6559 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -140,14 +140,7 @@ the parent project. Additionally, if you do not trust the fork project's runner, running the pipeline in the parent project uses the parent project's trusted runners. WARNING: -Fork merge requests can contain malicious code that tries to steal secrets in the -parent project when the pipeline runs, even before merge. As a reviewer, carefully -check the changes in the merge request before triggering the pipeline. If you trigger -the pipeline by selecting **Run pipeline** or applying a suggestion, GitLab shows -a warning that you must accept before the pipeline runs. If you trigger the pipeline -by using any other method, including the API, [`/rebase` quick action](../../user/project/quick_actions.md#issues-merge-requests-and-epics), -or [**Rebase** option](../../user/project/merge_requests/methods/index.md#rebasing-in-semi-linear-merge-methods), -**no warning displays**. +Fork merge requests can contain malicious code that tries to steal secrets in the parent project when the pipeline runs, even before merge. As a reviewer, carefully check the changes in the merge request before triggering the pipeline. Unless you trigger the pipeline through the API or the [`/rebase` quick action](../../user/project/quick_actions.md#issues-merge-requests-and-epics), GitLab shows a warning that you must accept before the pipeline runs. Otherwise, **no warning displays**. Prerequisites: diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index e4a3416f60b..dadf631e2bb 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index aacaa110041..31c848d5274 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 3633863915c..98dc02c1041 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -71,7 +71,7 @@ is selected. ## Auto-cancel redundant pipelines -You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: +You can set pending or running pipelines to cancel automatically when a pipeline for new changes runs on the same branch. You can enable this in the project settings: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index 7cf71f2a3ea..3835b5f2df2 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Control the job concurrency in GitLab CI/CD --- diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 7a0f43f9ec5..9e69ab1ccb8 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 0fd4bff1bff..41dda5da0b2 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -155,8 +155,11 @@ different places. ### Determine the IP address of a shared runner -To view the IP address of a shared runner you must have administrator access to -the GitLab instance. To determine this: +Prerequisite: + +- You must have administrator access to the instance. + +To determine the IP address of a shared runner: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **CI/CD > Runners**. diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 41cde709143..d3bbd7d8816 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -20,7 +20,7 @@ in your [subscription plan](https://about.gitlab.com/pricing/). ## Security for GitLab SaaS runners -GitLab SaaS runners on Linux and Windows run on Google Compute Platform. The [Google Infrastructure Security Design Overview whitepaper](https://cloud.google.com/docs/security/infrastructure/design/resources/google_infrastructure_whitepaper_fa.pdf) provides an overview of how Google designs security into its technical infrastructure. The GitLab [Trust Center](https://about.gitlab.com/security/) and [GitLab Security Compliance Controls](https://about.staging.gitlab.com/handbook/engineering/security/security-assurance/security-compliance/sec-controls.html) pages provide an overview of the Security and Compliance controls that govern the GitLab SaaS runners. +GitLab SaaS runners on Linux and Windows run on Google Compute Platform. The [Google Infrastructure Security Design Overview whitepaper](https://cloud.google.com/docs/security/infrastructure/design/resources/google_infrastructure_whitepaper_fa.pdf) provides an overview of how Google designs security into its technical infrastructure. The GitLab [Trust Center](https://about.gitlab.com/security/) and [GitLab Security Compliance Controls](https://about.staging.gitlab.com/handbook/engineering/security/security-assurance/security-compliance/sec-controls.html) pages provide an overview of the security and compliance controls that govern the GitLab SaaS runners. The runner that serves as a Runner Manager automatically initiates the creation and deletion of the virtual machines (VMs) used for CI jobs. When the Runner Manager picks up a GitLab SaaS CI job, it automatically executes that job on a new VM. There is no human or manual intervention in this process. The following section provides an overview of the additional built-in layers that harden the security of the GitLab Runner SaaS CI build environment. diff --git a/doc/ci/runners/register_runner.md b/doc/ci/runners/register_runner.md new file mode 100644 index 00000000000..ec10a1299dc --- /dev/null +++ b/doc/ci/runners/register_runner.md @@ -0,0 +1,33 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Register a shared runner **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383139) in GitLab 15.10. [Deployed behind the `create_runner_workflow_for_admin` flag](../../administration/feature_flags.md), disabled by default. + +FLAG: +On self-managed GitLab, this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `create_runner_workflow_for_admin`. +On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. + +Prerequisites: + +- You must be an administrator to register a shared runner. + +To register a [shared runner](../runners/runners_scope.md#shared-runners): + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **CI/CD > Runners**. +1. Select **New instance runner**. +1. Select a platform. +1. Optional. Enter a description. +1. Optional. Enter a maintenance note. +1. Optional. Enter configurations for the runner. +1. Select **Submit**. +1. Follow the instructions to register the runner from the command line. + +NOTE: +The token only displays in the UI for a short period of time during registration, +and is then saved in `config.toml`. diff --git a/doc/ci/runners/saas/linux_saas_runner.md b/doc/ci/runners/saas/linux_saas_runner.md index 58c8a6d0815..e9ac91409af 100644 --- a/doc/ci/runners/saas/linux_saas_runner.md +++ b/doc/ci/runners/saas/linux_saas_runner.md @@ -88,12 +88,15 @@ Below are the settings for SaaS runners on Linux. and [#4070](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4070). NOTE: -The final disk space your jobs can use is less than 25 GB. Some disk space -allocated to the instance is occupied by the operating system, the Docker image, -and a copy of your cloned repository. +SaaS runner instances are provisioned with a 25 GB storage volume. The underlying disk space of the storage volume +is shared by the operating system, the Docker image, and a copy of your cloned repository. +This means that the available free disk space that your jobs can use is **less than 25 GB**. -## Pre-clone script +## Pre-clone script (deprecated) +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/391896) in GitLab 15.9 +and is planned for removal in 16.0. Use [`pre_get_sources_script`](../../../ci/yaml/index.md#hookspre_get_sources_script) instead. This change is a breaking change. With SaaS runners on Linux, you can run commands in a CI/CD job before the runner attempts to run `git init` and `git fetch` to download a GitLab repository. The diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md index 1fc7afea7e6..7aa0f33fc59 100644 --- a/doc/ci/runners/saas/macos/environment.md +++ b/doc/ci/runners/saas/macos/environment.md @@ -20,11 +20,11 @@ Each time you run a job that requires tooling or dependencies not available in t GitLab SaaS provides macOS build machines on Apple servers with Intel x86-64 processors. The expectation is that virtual machines running on the Apple M1 chip will be available in the second half of 2022. -At this time there is only one available machine type offered, `gbc-macos-large`. +At this time there is only one available machine type offered, `shared-macos-amd64`. | Instance type | vCPUS | Memory (GB) | | --------- | --- | ------- | -| `gbc-macos-large` | 4 | 10 | +| `shared-macos-amd64` | 4 | 10 | ## VM images diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 96e294ff828..21120732fbe 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -84,4 +84,4 @@ In SaaS runners on macOS, the objective is to make 90% of CI jobs start executin - If the VM image does not include the specific software version you need for your job, then the job execution time will increase as the required software needs to be fetched and installed. - At this time, it is not possible to bring your own OS image. -- The keychain for user `gitlab` is not publicly available. You must create a keychain instead. +- The keychain for user `gitlab` is not publicly available. You must create a keychain instead. diff --git a/doc/ci/secrets/id_token_authentication.md b/doc/ci/secrets/id_token_authentication.md index f622bc2a7b1..b10763b40d6 100644 --- a/doc/ci/secrets/id_token_authentication.md +++ b/doc/ci/secrets/id_token_authentication.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- @@ -45,7 +45,7 @@ The following fields are included in each ID token: | [`iss`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.1) | Always | Issuer of the token, which is the domain of the GitLab instance ("issuer" claim). | | [`jti`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.7) | Always | Unique identifier for the token ("JWT ID" claim). | | [`nbf`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.5) | Always | The time after which the token becomes valid ("not before" claim). | -| [`sub`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.2) | Always | The job ID ("subject" claim). | +| [`sub`](https://www.rfc-editor.org/rfc/rfc7519.html#section-4.1.2) | Always | `project_path:{group}/{project}:ref_type:{type}:ref:{branch_name}` ("subject" claim). | | `deployment_tier` | Job specifies an environment | [Deployment tier](../environments/index.md#deployment-tier-of-environments) of the environment the job specifies. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363590) in GitLab 15.2. | | `environment_protected` | Job specifies an environment | `true` if specified environment is protected, `false` otherwise. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9. | | `environment` | Job specifies an environment | Environment the job specifies. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9. | diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 14f771431e5..fa8fe2a36a8 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md index 10baa57cabb..d432db351bb 100644 --- a/doc/ci/secure_files/index.md +++ b/doc/ci/secure_files/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index f26751c56e7..eca4c62deb2 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -269,7 +269,7 @@ test: | `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to the Docker `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | | `alias` (1) | no | 9.4 | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | -| `variables` (2) | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). | +| `variables` (2) | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). Service variables cannot reference themselves. | (1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index c1154485018..ab767697cbc 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 4088e5e82c6..8ad9e27de8b 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -1,12 +1,12 @@ --- stage: Plan -group: Certify +group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments description: Test cases in GitLab can help your teams create testing scenarios in their existing development platform. type: reference --- -# Test Cases **(ULTIMATE)** +# Test cases **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233479) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/241983) in GitLab 13.7. @@ -24,14 +24,14 @@ Prerequisite: To create a test case in a GitLab project: -1. Go to **CI/CD > Test Cases**. +1. Go to **CI/CD > Test cases**. 1. Select **New test case**. You are taken to the new test case form. Here you can enter the new case's title, [description](../../user/markdown.md), attach a file, and assign [labels](../../user/project/labels.md). 1. Select **Submit test case**. You are taken to view the new test case. ## View a test case -You can view all test cases in the project in the Test Cases list. Filter the +You can view all test cases in the project in the test cases list. Filter the issue list with a search query, including labels or the test case's title. Prerequisite: @@ -43,7 +43,7 @@ Whether you can view an test case depends on the [project visibility level](../. To view a test case: -1. In a project, go to **CI/CD > Test Cases**. +1. In a project, go to **CI/CD > Test cases**. 1. Select the title of the test case you want to view. You are taken to the test case page.  @@ -77,7 +77,7 @@ To archive a test case, on the test case's page, select **Archive test case**. To view archived test cases: -1. Go to **CI/CD > Test Cases**. +1. Go to **CI/CD > Test cases**. 1. Select **Archived**. ## Reopen an archived test case diff --git a/doc/ci/testing/accessibility_testing.md b/doc/ci/testing/accessibility_testing.md index fa57371a7d5..58bb4308095 100644 --- a/doc/ci/testing/accessibility_testing.md +++ b/doc/ci/testing/accessibility_testing.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/browser_performance_testing.md b/doc/ci/testing/browser_performance_testing.md index ff013f0037e..175ecef7f09 100644 --- a/doc/ci/testing/browser_performance_testing.md +++ b/doc/ci/testing/browser_performance_testing.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md index 4b826991bb5..ff7e11e9c71 100644 --- a/doc/ci/testing/code_quality.md +++ b/doc/ci/testing/code_quality.md @@ -44,6 +44,7 @@ Code Quality results are shown in the: - Merge request widget - Merge request changes view - Pipeline details view +- Project quality view ### Merge request widget @@ -73,6 +74,12 @@ tab of the pipeline's details page.  +### Project quality view + +The project quality view displays an overview of the code quality findings. + + + ## Enable Code Quality Prerequisites: diff --git a/doc/ci/testing/fail_fast_testing.md b/doc/ci/testing/fail_fast_testing.md index 4ad1656c8d6..57e21beef06 100644 --- a/doc/ci/testing/fail_fast_testing.md +++ b/doc/ci/testing/fail_fast_testing.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/img/code_quality_summary_15_9.png b/doc/ci/testing/img/code_quality_summary_15_9.png Binary files differnew file mode 100644 index 00000000000..60077123488 --- /dev/null +++ b/doc/ci/testing/img/code_quality_summary_15_9.png diff --git a/doc/ci/testing/index.md b/doc/ci/testing/index.md index 41d474f0e60..5fd86561b9e 100644 --- a/doc/ci/testing/index.md +++ b/doc/ci/testing/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md index 7e527fae562..cf870bcec55 100644 --- a/doc/ci/testing/load_performance_testing.md +++ b/doc/ci/testing/load_performance_testing.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/metrics_reports.md b/doc/ci/testing/metrics_reports.md index e084e4d3bc7..9257ba8990e 100644 --- a/doc/ci/testing/metrics_reports.md +++ b/doc/ci/testing/metrics_reports.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md index e2a1a4a2c5e..1b29304d1af 100644 --- a/doc/ci/testing/test_coverage_visualization.md +++ b/doc/ci/testing/test_coverage_visualization.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -33,7 +33,7 @@ This format was originally developed for Java, but most coverage analysis framew for other languages have plugins to add support for it, like: - [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby) -- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Golang) +- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Go) Other coverage analysis frameworks support the format out of the box, for example: diff --git a/doc/ci/testing/unit_test_report_examples.md b/doc/ci/testing/unit_test_report_examples.md index 87426fc8496..c63e225a2a7 100644 --- a/doc/ci/testing/unit_test_report_examples.md +++ b/doc/ci/testing/unit_test_report_examples.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -274,3 +274,24 @@ phpunit: reports: junit: report.xml ``` + +## Rust + +This example uses [cargo2junit](https://crates.io/crates/cargo2junit), +which is installed in the current directory. +To retrieve JSON output from `cargo test`, you must enable the nightly compiler. + +```yaml +run unittests: + image: rust:latest + stage: test + before_script: + - cargo install --root . cargo2junit + script: + - cargo test -- -Z unstable-options --format json --report-time | bin/cargo2junit > report.xml + artifacts: + when: always + reports: + junit: + - report.xml +``` diff --git a/doc/ci/testing/unit_test_reports.md b/doc/ci/testing/unit_test_reports.md index 0fe9b2b6d64..88a5d90d30e 100644 --- a/doc/ci/testing/unit_test_reports.md +++ b/doc/ci/testing/unit_test_reports.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index 16530ea20b6..393b128c658 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: tutorial --- @@ -201,3 +201,13 @@ doesn't exist, GitLab returns `The requested URL returned error: 400`. For example, you might accidentally use `main` for the branch name in a project that uses a different branch name for its default branch. + +Another possible cause for this error is a rule that prevents creation of the pipelines when `CI_PIPELINE_SOURCE` value is `trigger`, such as: + +```yaml +rules: + - if: $CI_PIPELINE_SOURCE == "trigger" + when: never +``` + +Review your [`workflow:rules`](../yaml/index.md#workflowrules) to ensure a pipeline can be created when `CI_PIPELINE_SOURCE` value is `trigger`. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 17ce184ee28..30c067db36a 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -37,7 +37,7 @@ If you need to link to the schema directly, it is at: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json. +https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json ``` To see the full list of custom tags covered by the CI/CD schema, check the diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index ec5c6c240a6..24eee4253a8 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -19,6 +19,12 @@ or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-varia Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) to execute scripts. Each shell has its own set of reserved variable names. +To ensure consistent behavior, you should always put variable values in single or double quotes. +Variables are internally parsed by the [Psych YAML parser](https://docs.ruby-lang.org/en/master/Psych.html), +so quoted and unquoted variables might be parsed differently. For example, `VAR1: 012345` +is interpreted as an octal value, so the value becomes `5349`, but `VAR1: "012345"` is parsed +as a string with a value of `012345`. + > For more information about advanced use of GitLab CI/CD: > > - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Get to productivity faster with these [7 advanced GitLab CI workflow hacks](https://about.gitlab.com/webcast/7cicd-hacks/) @@ -189,7 +195,7 @@ You can make a CI/CD variable available to all projects and groups in a GitLab i Prerequisite: -- You must have administrator access. +- You must have administrator access to the instance. To add an instance variable: diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index b9557b98066..45dc1a7fe37 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 6ea1f454467..c20d9be51e7 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Authoring +group: Pipeline Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index 151a043da5e..8740cf100ac 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -22,7 +22,7 @@ date for the artifacts. Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or pipeline features from each job. -To be able to browse the report output files, include the [`artifacts:paths`](index.md#artifactspaths) keyword. +To browse the report output files, ensure you include the [`artifacts:paths`](index.md#artifactspaths) keyword in your job definition. NOTE: Combined reports in parent pipelines using [artifacts from child pipelines](index.md#needspipelinejob) is diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index bf0b7444e78..6d34a3034f7 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -20,13 +20,6 @@ To include a single configuration file, use either of these syntax options: include: '/templates/.after-script-template.yml' ``` -- `include` with a single file, and you specify the `include` type: - - ```yaml - include: - remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - ``` - ## Include an array of configuration files You can include an array of configuration files: @@ -159,12 +152,18 @@ do not change. This method is called *merging*. ### Merge method for `include` -For a file containing `include` directives, the included files are read in order (possibly -recursively), and the configuration in these files is likewise merged in order. If the parameters overlap, the last included file takes precedence. Finally, the directives in the -file itself are merged with the configuration from the included files. +The `include` configuration merges with the main configuration file with this process: + +- Included files are read in the order defined in the configuration file, and + the included configuration is merged together in the same order. +- If an included file also uses `include`, that nested `include` configuration is merged first (recursively). +- If parameters overlap, the last included file takes precedence when merging the configuration + from the included files. +- After all configuration added with `include` is merged together, the main configuration + is merged with the included configuration. This merge method is a _deep merge_, where hash maps are merged at any depth in the -configuration. To merge hash map A (containing the configuration merged so far) and B (the next piece +configuration. To merge hash map "A" (that contains the configuration merged so far) and "B" (the next piece of configuration), the keys and values are processed as follows: - When the key only exists in A, use the key and value from A. @@ -172,9 +171,7 @@ of configuration), the keys and values are processed as follows: - When the key exists in both A and B, and one of the values is not a hash map, use the value from B. - Otherwise, use the key and value from B. -For example: - -We have a configuration consisting of two files. +For example, with a configuration that consists of two files: - The `.gitlab-ci.yml` file: @@ -211,7 +208,7 @@ We have a configuration consisting of two files. dotenv: deploy.env ``` -The merged result: +The merged result is: ```yaml variables: @@ -374,7 +371,7 @@ In `include` sections in your `.gitlab-ci.yml` file, you can use: - [Project variables](../variables/index.md#for-a-project). - [Group variables](../variables/index.md#for-a-group). - [Instance variables](../variables/index.md#for-an-instance). -- Project [predefined variables](../variables/predefined_variables.md). +- Project [predefined variables](../variables/predefined_variables.md) (`CI_PROJECT_*`). - In GitLab 14.2 and later, the `$CI_COMMIT_REF_NAME` [predefined variable](../variables/predefined_variables.md). When used in `include`, the `CI_COMMIT_REF_NAME` variable returns the full @@ -386,15 +383,7 @@ In GitLab 14.5 and later, you can also use: - [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule). - [Manual pipeline run variables](../pipelines/index.md#run-a-pipeline-manually). -- Pipeline [predefined variables](../variables/predefined_variables.md). - - YAML files are parsed before the pipeline is created, so the following pipeline predefined variables - are **not** available: - - - `CI_PIPELINE_ID` - - `CI_PIPELINE_URL` - - `CI_PIPELINE_IID` - - `CI_PIPELINE_CREATED_AT` +- The `CI_PIPELINE_SOURCE` and `CI_PIPELINE_TRIGGERED` [predefined variables](../variables/predefined_variables.md). For example: @@ -517,3 +506,20 @@ When the pipeline runs, GitLab: # This matches all `.yml` files only in subfolders of `configs`. include: 'configs/**/*.yml' ``` + +## Troubleshooting + +### `Maximum of 150 nested includes are allowed!` error + +The maximum number of [nested included files](#use-nested-includes) for a pipeline is 150. +If you receive the `Maximum 150 includes are allowed` error message in your pipeline, +it's likely that either: + +- Some of the nested configuration includes an overly large number of additional nested `include` configuration. +- There is an accidental loop in the nested includes. For example, `include1.yml` includes + `include2.yml` which includes `include1.yml`, creating a recursive loop. + +To help reduce the risk of this happening, edit the pipeline configuration file +with the [pipeline editor](../pipeline_editor/index.md), which validates if the +limit is reached. You can remove one included file at a time to try to narrow down +which configuration file is the source of the loop or excessive included files. diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index d5c0fe1d41d..6ba9d455278 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -136,11 +136,17 @@ The `include` files are: - Always evaluated first and then merged with the content of the `.gitlab-ci.yml` file, regardless of the position of the `include` keyword. -You can [nest](includes.md#use-nested-includes) up to 100 includes. In [GitLab 14.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/28987), -the same file can be included multiple times in nested includes, but duplicates are ignored. +You can have up to 150 includes per pipeline, including [nested](includes.md#use-nested-includes) includes: -In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/28212), -the time limit to resolve all files is 30 seconds. +- In [GitLab 15.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/367150) you can have up to 150 includes. + In nested includes, the same file can be included multiple times, but duplicated includes + count towards the limit. +- From [GitLab 14.9 to GitLab 15.9](https://gitlab.com/gitlab-org/gitlab/-/issues/28987), you can have up to 100 includes. + The same file can be included multiple times in nested includes, but duplicates are ignored. +- In GitLab 14.9 and earlier you can have up to 100 includes, but the same file can not + be included multiple times. + +The time limit to resolve all files is 30 seconds. **Keyword type**: Global keyword. @@ -153,6 +159,8 @@ the time limit to resolve all files is 30 seconds. **Additional details**: +- Only [certain CI/CD variables](includes.md#use-variables-with-include) can be used + with `include` keywords. - Use merging to customize and override included CI/CD configurations with local - You can override included configuration by having the same job name or global keyword in the `.gitlab-ci.yml` file. The two configurations are merged together, and the @@ -171,7 +179,7 @@ the time limit to resolve all files is 30 seconds. #### `include:local` -Use `include:local` to include a file that is in the same repository as the project running the pipeline. +Use `include:local` to include a file that is in the same repository as the configuration file containing the `include` keyword. Use `include:local` instead of symbolic links. **Keyword type**: Global keyword. @@ -201,8 +209,8 @@ include: '.gitlab-ci-production.yml' - The `.gitlab-ci.yml` file and the local file must be on the same branch. - You can't include local files through Git submodules paths. -- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the same project, - so you can use local, project, remote, or template includes. +- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the project containing the configuration file with the `include` keyword, not the project running the pipeline. + You can use local, project, remote, or template includes. #### `include:project` @@ -220,8 +228,7 @@ use `include:project` and `include:file`. The YAML files must have the `.yml` or `.yaml` extension. - `include:ref`: Optional. The ref to retrieve the file from. Defaults to the `HEAD` of the project when not specified. - -You can use [certain CI/CD variables](includes.md#use-variables-with-include). +- You can use [certain CI/CD variables](includes.md#use-variables-with-include). **Example of `include:project`**: @@ -252,8 +259,8 @@ include: **Additional details**: -- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the target project. - You can use `local` (relative to the target project), `project`, `remote`, or `template` includes. +- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the project containing the configuration file with the nested `include` keyword. + You can use `local` (relative to the project containing the configuration file with the `include` keyword), `project`, `remote`, or `template` includes. - When the pipeline starts, the `.gitlab-ci.yml` file configuration included by all methods is evaluated. The configuration is a snapshot in time and persists in the database. GitLab does not reflect any changes to the referenced `.gitlab-ci.yml` file configuration until the next pipeline starts. @@ -422,23 +429,30 @@ A configuration with different pipeline names depending on the pipeline conditio ```yaml variables: - PIPELINE_NAME: 'Default pipeline name' # A default is not required. + PROJECT1_PIPELINE_NAME: 'Default pipeline name' # A default is not required. workflow: - name: '$PIPELINE_NAME' + name: '$PROJECT1_PIPELINE_NAME' rules: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' variables: - PIPELINE_NAME: 'MR pipeline: $CI_COMMIT_BRANCH' + PROJECT1_PIPELINE_NAME: 'MR pipeline: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' - if: '$CI_MERGE_REQUEST_LABELS =~ /pipeline:run-in-ruby3/' variables: - PIPELINE_NAME: 'Ruby 3 pipeline' + PROJECT1_PIPELINE_NAME: 'Ruby 3 pipeline' ``` **Additional details**: - If the name is an empty string, the pipeline is not assigned a name. A name consisting of only CI/CD variables could evaluate to an empty string if all the variables are also empty. +- `workflow:rules:variables` become [global variables](#variables) available in all jobs, + including [`trigger`](#trigger) jobs which forward variables to downstream pipelines by default. + If the downstream pipeline uses the same variable, the [variable is overwritten](../variables/index.md#cicd-variable-precedence) + by the upstream variable value. Be sure to either: + - Use a unique variable name in every project's pipeline configuration, like `PROJECT1_PIPELINE_NAME`. + - Use [`inherit:variables`](#inheritvariables) in the trigger job and list the + exact variables you want to forward to the downstream pipeline. #### `workflow:rules` @@ -552,6 +566,16 @@ When the branch is something else: - job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. - job2's `DEPLOY_VARIABLE` is `default-deploy`. +**Additional details**: + +- `workflow:rules:variables` become [global variables](#variables) available in all jobs, + including [`trigger`](#trigger) jobs which forward variables to downstream pipelines by default. + If the downstream pipeline uses the same variable, the [variable is overwritten](../variables/index.md#cicd-variable-precedence) + by the upstream variable value. Be sure to either: + - Use unique variable names in every project's pipeline configuration, like `PROJECT1_VARIABLE_NAME`. + - Use [`inherit:variables`](#inheritvariables) in the trigger job and list the + exact variables you want to forward to the downstream pipeline. + ## Job keywords The following topics explain how to use keywords to configure CI/CD pipelines. @@ -938,10 +962,8 @@ job: #### `artifacts:public` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's recommended for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223273) in GitLab 13.8 [with a flag](../../user/feature_flags.md) named `non_public_artifacts`, disabled by default. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/322454) in GitLab 15.10. Artifacts created with `artifacts:public` before 15.10 are not guaranteed to remain private after this update. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, @@ -1924,12 +1946,8 @@ rspec: ### `hooks` -> Introduced in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. -The feature is not ready for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356850) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/381840) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. Use `hooks` to specify lists of commands to execute on the runner at certain stages of job execution, like before retrieving the Git repository. @@ -1943,7 +1961,8 @@ at certain stages of job execution, like before retrieving the Git repository. #### `hooks:pre_get_sources_script` -> Introduced in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356850) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/381840) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. Use `hooks:pre_get_sources_script` to specify a list of commands to execute on the runner before retrieving the Git repository and any submodules. You can use it @@ -4348,18 +4367,19 @@ child3: ### `variables` -Use `variables` to define [CI/CD variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file), -which are configurable values that are passed to jobs. - -Variables are always available in `script`, `before_script`, and `after_script` commands. -You can also use variables as inputs in some job keywords. +Use `variables` to define [CI/CD variables](../variables/index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file) for jobs. **Keyword type**: Global and job keyword. You can use it at the global level, and also at the job level. -If you define `variables` at the global level, each variable is copied to -every job configuration when the pipeline is created. If the job already has that -variable defined, the [job-level variable takes precedence](../variables/index.md#cicd-variable-precedence). +If you define `variables` as a [global keyword](#keywords), it behaves like default variables +for all jobs. Each variable is copied to every job configuration when the pipeline is created. +If the job already has that variable defined, the [job-level variable takes precedence](../variables/index.md#cicd-variable-precedence). + +Variables defined at the global-level cannot be used as inputs for other global keywords +like [`include`](includes.md#use-variables-with-include). These variables can only +be used at the job-level, in `script`, `before_script`, and `after_script` sections, +as well as inputs in some job keywords like [`rules`](../jobs/job_control.md#cicd-variable-expressions). **Possible inputs**: Variable name and value pairs: diff --git a/doc/cloud_seed/index.md b/doc/cloud_seed/index.md index 1021c7f7700..9ce18950506 100644 --- a/doc/cloud_seed/index.md +++ b/doc/cloud_seed/index.md @@ -1,5 +1,5 @@ --- -stage: Release +stage: Configure group: Incubation info: Cloud Seed (formerly 5mp) is a GitLab Incubation Engineering program. No technical writer assigned to this group. --- diff --git a/doc/development/elasticsearch.md b/doc/development/advanced_search.md index 935964a9a90..73a8191b789 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/advanced_search.md @@ -4,16 +4,16 @@ group: Global Search info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Elasticsearch knowledge +# Advanced search development guidelines -This area is to maintain a compendium of useful information when working with Elasticsearch. +This page includes information about developing and working with Elasticsearch. Information on how to enable Elasticsearch and perform the initial indexing is in the [Elasticsearch integration documentation](../integration/advanced_search/elasticsearch.md#enable-advanced-search). ## Deep Dive -In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on the GitLab [Elasticsearch integration](../integration/advanced_search/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. +In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`) on the GitLab [Elasticsearch integration](../integration/advanced_search/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details might have changed, it should still serve as a good introduction. In August 2020, a second Deep Dive was hosted, focusing on [GitLab-specific architecture for multi-indices support](#zero-downtime-reindexing-with-multiple-indices). The <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=0WdPR9oB2fg) and the [slides](https://lulalala.gitlab.io/gitlab-elasticsearch-deepdive/) are available. Everything covered in this deep dive was accurate as of GitLab 13.3. @@ -42,6 +42,16 @@ After initial indexing is complete, create, update, and delete operations for al Search queries are generated by the concerns found in [`ee/app/models/concerns/elastic`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! +### Custom routing + +[Custom routing](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html#_searching_with_custom_routing) +is used in Elasticsearch for document types that are associated with a project. The routing format is `project_<project_id>`. Routing is set +during indexing and searching operations. Some of the benefits and tradeoffs to using custom routing are: + +- Project scoped searches are much faster. +- Routing is not used if too many shards would be hit for global and group scoped searches. +- Shard size imbalance might occur. + ## Existing Analyzers/Tokenizers/Filters These are all defined in [`ee/lib/elastic/latest/config.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb) @@ -184,305 +194,6 @@ If the current version is `v12p1`, and we need to create a new version for `v12p 1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1` 1. Make changes to files under the `latest` folder as needed -## Creating a new Advanced Search migration - -> This functionality was introduced by [#234046](https://gitlab.com/gitlab-org/gitlab/-/issues/234046). - -NOTE: -This only supported for indices created with GitLab 13.0 or greater. - -In the [`ee/elastic/migrate/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/elastic/migrate) folder, create a new file with the filename format `YYYYMMDDHHMMSS_migration_name.rb`. This format is the same for Rails database migrations. - -```ruby -# frozen_string_literal: true - -class MigrationName < Elastic::Migration - # Important: Any updates to the Elastic index mappings must be replicated in the respective - # configuration files: - # - `Elastic::Latest::Config`, for the main index. - # - `Elastic::Latest::<Type>Config`, for standalone indices. - - def migrate - end - - # Check if the migration has completed - # Return true if completed, otherwise return false - def completed? - end -end -``` - -Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All migrations not executed -are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) -cron worker sequentially. - -To update Elastic index mappings, apply the configuration to the respective files: - -- For the main index: [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). -- For standalone indices: `Elastic::Latest::<Type>Config`. - -Migrations can be built with a retry limit and have the ability to be [failed and marked as halted](https://gitlab.com/gitlab-org/gitlab/-/blob/66e899b6637372a4faf61cfd2f254cbdd2fb9f6d/ee/lib/elastic/migration.rb#L40). -Any data or index cleanup needed to support migration retries should be handled within the migration. - -### Migration helpers - -The following migration helpers are available in `ee/app/workers/concerns/elastic/`: - -#### `Elastic::MigrationBackfillHelper` - -Backfills a specific field in an index. In most cases, the mapping for the field should already be added. - -Requires the `index_name` and `field_name` methods. - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationBackfillHelper - - private - - def index_name - Issue.__elasticsearch__.index_name - end - - def field_name - :schema_version - end -end -``` - -#### `Elastic::MigrationUpdateMappingsHelper` - -Updates a mapping in an index by calling `put_mapping` with the mapping specified. - -Requires the `index_name` and `new_mappings` methods. - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationUpdateMappingsHelper - - private - - def index_name - Issue.__elasticsearch__.index_name - end - - def new_mappings - { - schema_version: { - type: 'short' - } - } - end -end -``` - -#### `Elastic::MigrationRemoveFieldsHelper` - -Removes specified fields from an index. - -Requires the `index_name`, `document_type` methods. If there is one field to remove, add the `field_to_remove` method, otherwise add `fields_to_remove` with an array of fields. - -Checks in batches if any documents that match `document_type` have the fields specified in Elasticsearch. If documents exist, uses a Painless script to perform `update_by_query`. - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationRemoveFieldsHelper - - batched! - throttle_delay 1.minute - - private - - def index_name - User.__elasticsearch__.index_name - end - - def document_type - 'user' - end - - def fields_to_remove - %w[two_factor_enabled has_projects] - end -end -``` - -The default batch size is `10_000`. You can override this value by specifying `BATCH_SIZE`: - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationRemoveFieldsHelper - - batched! - BATCH_SIZE = 100 - - ... -end -``` - -#### `Elastic::MigrationObsolete` - -Marks a migration as obsolete when it's no longer required. - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationObsolete -end -``` - -#### `Elastic::MigrationHelper` - -Contains methods you can use when a migration doesn't fit the previous examples. - -```ruby -class MigrationName < Elastic::Migration - include Elastic::MigrationHelper - - def migrate - ... - end - - def completed? - ... - end -end -``` - -### Migration options supported by the `Elastic::MigrationWorker` - -[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) supports the following migration options: - -- `batched!` - Allow the migration to run in batches. If set, the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) -will re-enqueue itself with a delay which is set using the `throttle_delay` option described below. The batching -must be handled within the `migrate` method, this setting controls the re-enqueuing only. - -- `batch_size` - Sets the number of documents modified during a `batched!` migration run. This size should be set to a value which allows the updates -enough time to finish. This can be tuned in combination with the `throttle_delay` option described below. The batching -must be handled within a custom `migrate` method or by using the [`Elastic::MigrationBackfillHelper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/concerns/elastic/migration_backfill_helper.rb) -`migrate` method which uses this setting. Default value is 1000 documents. - -- `throttle_delay` - Sets the wait time in between batch runs. This time should be set high enough to allow each migration batch -enough time to finish. Additionally, the time should be less than 30 minutes since that is how often the -[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) -cron worker runs. Default value is 5 minutes. - -- `pause_indexing!` - Pause indexing while the migration runs. This setting will record the indexing setting before -the migration runs and set it back to that value when the migration is completed. - -- `space_requirements!` - Verify that enough free space is available in the cluster when the migration runs. This setting - will halt the migration if the storage required is not available when the migration runs. The migration must provide - the space required in bytes by defining a `space_required_bytes` method. - -- `retry_on_failure` - Enable the retry on failure feature. By default, it retries - the migration 30 times. After it runs out of retries, the migration is marked as halted. - To customize the number of retries, pass the `max_attempts` argument: - `retry_on_failure max_attempts: 10` - -```ruby -# frozen_string_literal: true - -class BatchedMigrationName < Elastic::Migration - # Declares a migration should be run in batches - batched! - throttle_delay 10.minutes - pause_indexing! - space_requirements! - retry_on_failure - - # ... -end -``` - -### Multi-version compatibility - -These Advanced Search migrations, like any other GitLab changes, need to support the case where -[multiple versions of the application are running at the same time](multi_version_compatibility.md). - -Depending on the order of deployment, it's possible that the migration -has started or finished and there's still a server running the application code from before the -migration. We need to take this into consideration until we can -[ensure all Advanced Search migrations start after the deployment has finished](https://gitlab.com/gitlab-org/gitlab/-/issues/321619). - -### Reverting a migration - -Because Elasticsearch does not support transactions, we always need to design our -migrations to accommodate a situation where the application -code is reverted after the migration has started or after it is finished. - -For this reason we generally defer destructive actions (for example, deletions after -some data is moved) to a later merge request after the migrations have -completed successfully. To be safe, for self-managed customers we should also -defer it to another release if there is risk of important data loss. - -### Best practices for Advanced Search migrations - -Follow these best practices for best results: - -- When working in batches, keep the batch size under 9,000 documents - and `throttle_delay` for at least 3 minutes. The bulk indexer is set to run - every 1 minute and process a batch of 10,000 documents. These limits - allow the bulk indexer time to process records before another migration - batch is attempted. -- To ensure that document counts are up to date, it is recommended to refresh - the index before checking if a migration is completed. -- Add logging statements to each migration when the migration starts, when a - completion check occurs, and when the migration is completed. These logs - are helpful when debugging issues with migrations. -- Pause indexing if you're using any Elasticsearch Reindex API operations. -- Consider adding a retry limit if there is potential for the migration to fail. - This ensures that migrations can be halted if an issue occurs. - -## Deleting Advanced Search migrations in a major version upgrade - -Since our Advanced Search migrations usually require us to support multiple -code paths for a long period of time, it's important to clean those up when we -safely can. - -We choose to use GitLab major version upgrades as a safe time to remove -backwards compatibility for indices that have not been fully migrated. We -[document this in our upgrade documentation](../update/index.md#upgrading-to-a-new-major-version). -We also choose to replace the migration code with the halted migration -and remove tests so that: - -- We don't need to maintain any code that is called from our Advanced Search - migrations. -- We don't waste CI time running tests for migrations that we don't support - anymore. -- Operators who have not run this migration and who upgrade directly to the - target version will see a message prompting them to reindex from scratch. - -To be extra safe, we will not delete migrations that were created in the last -minor version before the major upgrade. So, if we are upgrading to `%14.0`, -we should not delete migrations that were only added in `%13.12`. This is an -extra safety net as we expect there are migrations that get merged that may -take multiple weeks to finish on GitLab.com. It would be bad if we upgraded -GitLab.com to `%14.0` before the migrations in `%13.12` were finished. Since -our deployments to GitLab.com are automated and we currently don't have -automated checks to prevent this, the extra precaution is warranted. -Additionally, even if we did have automated checks to prevent it, we wouldn't -actually want to hold up GitLab.com deployments on Advanced Search migrations, -as they may still have another week to go, and that's too long to block -deployments. - -### Process for removing migrations - -For every migration that was created 2 minor versions before the major version -being upgraded to, we do the following: - -1. Confirm the migration has actually completed successfully for GitLab.com. -1. Replace the content of the migration with: - - ```ruby - include Elastic::MigrationObsolete - ``` - -1. Delete any spec files to support this migration. -1. Remove any logic handling backwards compatibility for this migration. You - can find this by looking for - `Elastic::DataMigrationService.migration_has_finished?(:migration_name_in_lowercase)`. -1. Create a merge request with these changes. Noting that we should not - accidentally merge this before the major release is started. - ## Performance Monitoring ### Prometheus diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 94abbda9671..d29d9b9651f 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -875,7 +875,7 @@ module Types graphql_name 'IssuableSeverity' description 'Incident severity' - ::IssuableSeverity.severities.keys.each do |severity| + ::IssuableSeverity.severities.each_key do |severity| value severity.upcase, value: severity, description: "#{severity.titleize} severity." end end diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 006d0a01abb..828ef21ba9a 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -333,6 +333,17 @@ expect(response).to match_response_schema('merge_requests') Also see [verifying N+1 performance](#verifying-with-tests) in tests. +## Error handling + +When throwing an error with a message that is meant to be user-facing, you should +use the error message utility function contained in `lib/gitlab/utils/error_message.rb`. +It adds a prefix to the error message, making it distinguishable from non-user-facing error messages. +Please make sure that the Frontend is aware of the prefix usage and is using the according utils. + +```ruby +Gitlab::Utils::ErrorMessage.to_user_facing('Example user-facing error-message') +``` + ## Include a changelog entry All client-facing changes **must** include a [changelog entry](changelog.md). diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index bd4587333e0..f48088a6e08 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -24,7 +24,7 @@ to be emitted from the rails application: ## Existing SLIs -1. [`rails_request_apdex`](rails_request_apdex.md) +1. [`rails_request`](rails_request.md) 1. `global_search_apdex` 1. `global_search_error_rate` 1. `global_search_indexing_apdex` diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request.md index dc9d67b0a2b..b3ee326aa87 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request.md @@ -4,7 +4,7 @@ group: Scalability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Rails request Apdex SLI +# Rails request SLIs (service level indicators) > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 @@ -12,21 +12,33 @@ NOTE: This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_observability/index.md#error-budget) by default. You can [opt in](#error-budget-attribution-and-ownership). -The request Apdex SLI (Service Level Indicator) is [an SLI defined in the application](index.md). -It measures the duration of successful requests as an indicator for +The request Apdex SLI and the error rate SLI are [SLIs defined in the application](index.md). + +The request Apdex measures the duration of successful requests as an indicator for application performance. This includes the REST and GraphQL API, and the -regular controller endpoints. It consists of these counters: +regular controller endpoints. + +The error rate measures unsuccessful requests as an indicator for +server misbehavior. This includes the REST API, and the +regular controller endpoints. -1. `gitlab_sli:rails_request_apdex:total`: This counter gets +1. `gitlab_sli_rails_request_apdex_total`: This counter gets incremented for every request that did not result in a response with a `5xx` status code. It ensures slow failures are not counted twice, because the request is already counted in the error SLI. -1. `gitlab_sli:rails_request_apdex:success_total`: This counter gets +1. `gitlab_sli_rails_request_apdex_success_total`: This counter gets incremented for every successful request that performed faster than the [defined target duration depending on the endpoint's urgency](#adjusting-request-urgency). -Both these counters are labeled with: +1. `gitlab_sli_rails_request_error_total`: This counter gets + incremented for every request that resulted in a response + with a `5xx` status code. + +1. `gitlab_sli_rails_request_total`: This counter gets + incremented for every request. + +These counters are labeled with: 1. `endpoint_id`: The identification of the Rails Controller or the Grape-API endpoint. @@ -195,6 +207,14 @@ class Boards::ListsController < ApplicationController end ``` +A custom RSpec matcher is available to check endpoint's request urgency in the controller specs: + +```ruby +specify do + expect(get(:index, params: request_params)).to have_request_urgency(:medium) +end +``` + ### Grape endpoints To specify the urgency for an entire API class: @@ -228,6 +248,15 @@ get 'client/features', urgency: :low do end ``` +A custom RSpec matcher is also compatible with grape endpoints' specs: + +```ruby + +specify do + expect(get(api('/avatar'), params: { email: 'public@example.com' })).to have_request_urgency(:medium) +end +``` + WARNING: We can't specify the urgency at the namespace level. The directive is ignored when doing so. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 96b70e2fbd8..7a9c3572406 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -448,7 +448,7 @@ Consul is a tool for service discovery and configuration. Consul is distributed, - [Source](../integration/advanced_search/elasticsearch.md) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/elasticsearch.md) - Layer: Core Service (Data) -- GitLab.com: [Get Advanced Search working on GitLab.com (Closed)](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. +- GitLab.com: [Get advanced search working on GitLab.com (Closed)](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. Elasticsearch is a distributed RESTful search engine built for the cloud. @@ -695,7 +695,7 @@ Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics. - [Source](../install/installation.md#6-database) - Layer: Core Service (Data) - Process: `postgresql` -- GitLab.com: [PostgreSQL](../user/gitlab_com/index.md#postgresql) +- GitLab.com: [PostgreSQL](https://about.gitlab.com/handbook/engineering/infrastructure/database/) GitLab packages the popular Database to provide storage for Application meta data and user information. @@ -1101,12 +1101,29 @@ PostgreSQL: GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced configuration files include: -- `gitlab.yml`: GitLab configuration +- `gitlab.yml`: GitLab Rails configuration - `puma.rb`: Puma web server settings - `database.yml`: Database connection settings GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. +#### Adding a new setting in GitLab Rails + +Settings which belong in `gitlab.yml` include those related to: + +- How the application is wired together across multiple services. For example, Gitaly addresses, Redis addresses, Postgres addresses, and Consul addresses. +- Distributed Tracing configuration, and some observability configurations. For example, histogram bucket boundaries. +- Anything that needs to be configured during Rails initialization, possibly before the Postgres connection has been established. + +Many other settings are better placed in the app itself, in `ApplicationSetting`. Managing settings in UI is usually a better user experience compared to managing configuration files. With respect to development cost, modifying `gitlab.yml` often seems like a faster iteration, but when you consider all the deployment methods below, it may be a poor tradeoff. + +When adding a setting to `gitlab.yml`: + +1. Ensure that it is also + [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml#adding-a-new-setting-to-gitlabyml). +1. Ensure that it is also [added to Charts](https://docs.gitlab.com/charts/development/style_guide.html), if needed. +1. Ensure that it is also [added to GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/support/templates/gitlab/config/gitlab.yml.erb). + ### Maintenance tasks [GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index 8df5121a2f7..c7c723d1686 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -4,7 +4,7 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Audit Event Guide +# Audit event development guidelines This guide provides an overview of how Audit Events work, and how to instrument new audit events. diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index 53033cd19ff..b2281f9c032 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Auto DevOps development guide +# Auto DevOps development guidelines This document provides a development guide for contributors to [Auto DevOps](../topics/autodevops/index.md). diff --git a/doc/development/backend/create_source_code_be/gitaly_touch_points.md b/doc/development/backend/create_source_code_be/gitaly_touch_points.md index 6ee2c7f0e51..c689af2f150 100644 --- a/doc/development/backend/create_source_code_be/gitaly_touch_points.md +++ b/doc/development/backend/create_source_code_be/gitaly_touch_points.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## RPCs -Gitaly is a wrapper around the `git` binary, running in a [Gitaly Cluster](../../../administration/gitaly/index.md). It provides managed access to the file system housing the `git` repositories, via Golang Remote Procedure Calls (RPCs). Other functions are access optimization, caching, and a form of pagination against the file system. +Gitaly is a wrapper around the `git` binary, running in a [Gitaly Cluster](../../../administration/gitaly/index.md). It provides managed access to the file system housing the `git` repositories, using Go Remote Procedure Calls (RPCs). Other functions are access optimization, caching, and a form of pagination against the file system. The comprehensive [Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/beginners_guide.md) is focused on making updates to Gitaly, and offers many insights into how to understand the Gitaly code. diff --git a/doc/development/backend/create_source_code_be/index.md b/doc/development/backend/create_source_code_be/index.md index 77c98982210..d894a14b006 100644 --- a/doc/development/backend/create_source_code_be/index.md +++ b/doc/development/backend/create_source_code_be/index.md @@ -4,16 +4,16 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create: Source Code Backend +# Create: Source Code backend -The Create:Source Code BE team focuses on the GitLab suite of Source Code Management +The Create: Source Code backend (BE) team focuses on the GitLab suite of Source Code Management (SCM) tools. It is responsible for all backend aspects of the product categories that fall under the [Source Code group](https://about.gitlab.com/handbook/product/categories/#source-code-group) of the [Create stage](https://about.gitlab.com/handbook/product/categories/#create-stage) of the [DevOps lifecycle](https://about.gitlab.com/handbook/product/categories/#devops-stages). We interface with the Gitaly and Code Review teams, and work closely with the -[Create:Source Code Frontend team](https://about.gitlab.com/handbook/engineering/development/dev/create/create-source-code-fe/). The features +[Create: Source Code frontend team](https://about.gitlab.com/handbook/engineering/development/dev/create/create-source-code-fe/). The features we work with are listed on the [Features by Group Page](https://about.gitlab.com/handbook/product/categories/features/#createsource-code-group). @@ -39,7 +39,7 @@ To learn about the reasoning behind our creation of `gitlab-sshd`, read the blog ### Gitaly touch points -Gitaly is a Golang RPC service which handles all the `git` calls made by GitLab. +Gitaly is a Go RPC service which handles all the `git` calls made by GitLab. GitLab is not exposed directly, and all traffic comes through Create: Source Code. For more information, read [Gitaly touch points](gitaly_touch_points.md). diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 0d9c7348915..23976d09389 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -37,13 +37,14 @@ idea is to have one ETL pipeline for each relation to be imported. ### API -The current [Project](../user/project/settings/import_export.md) and [Group](../user/group/settings/import_export.md) Import are file based, so they require an export -step to generate the file to be imported. +The current [project](../user/project/settings/import_export.md) and +[group](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so +they require an export step to generate the file to be imported. -GitLab Group migration leverages on [GitLab API](../api/rest/index.md) to speed the migration. +Group migration by direct transfer leverages the [GitLab API](../api/rest/index.md) to speed the migration. And, because we're on the road to [GraphQL](../api/graphql/index.md), -GitLab Group Migration will be contributing towards to expand the GraphQL API coverage, which benefits both GitLab +Group migration by direct transfer can contribute to expanding GraphQL API coverage, which benefits both GitLab and its users. ### Namespace diff --git a/doc/development/caching.md b/doc/development/caching.md index 9b3f9a4215e..dff94b68ca0 100644 --- a/doc/development/caching.md +++ b/doc/development/caching.md @@ -332,11 +332,11 @@ views/projects/_home_panel:462ad2485d7d6957e03ceba2c6717c29/projects/16-20210316 ``` 1. The view name and template tree digest - `views/projects/_home_panel:462ad2485d7d6957e03ceba2c6717c29` + `views/projects/_home_panel:462ad2485d7d6957e03ceba2c6717c29` 1. The model name, ID, and `updated_at` values - `projects/16-20210316142425469452` + `projects/16-20210316142425469452` 1. The symbol we passed in, converted to a string - `tag_list` + `tag_list` ### Look for diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 389623e68d8..42f4b5dd6f3 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -23,13 +23,13 @@ Settings are not cascading by default. To define a cascading setting, take the f 1. In the `NamespaceSetting` model, define the new attribute using the `cascading_attr` helper method. You can use an array to define multiple attributes on a single line. - ```ruby - class NamespaceSetting - include CascadingNamespaceSettingAttribute + ```ruby + class NamespaceSetting + include CascadingNamespaceSettingAttribute - cascading_attr :delayed_project_removal - end - ``` + cascading_attr :delayed_project_removal + end + ``` 1. Create the database columns. @@ -37,21 +37,21 @@ Settings are not cascading by default. To define a cascading setting, take the f The helper creates four columns, two each in `namespace_settings` and `application_settings`. - ```ruby - class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[2.1] - include Gitlab::Database::MigrationHelpers::CascadingNamespaceSettings + ```ruby + class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[2.1] + include Gitlab::Database::MigrationHelpers::CascadingNamespaceSettings - enable_lock_retries! + enable_lock_retries! - def up - add_cascading_namespace_setting :delayed_project_removal, :boolean, default: false, null: false - end + def up + add_cascading_namespace_setting :delayed_project_removal, :boolean, default: false, null: false + end - def down - remove_cascading_namespace_setting :delayed_project_removal - end - end - ``` + def down + remove_cascading_namespace_setting :delayed_project_removal + end + end + ``` Existing settings being converted to a cascading setting will require individual migrations to add columns and change existing columns. Use the specifications diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 27bcffe7560..56b5fa8c976 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -94,7 +94,7 @@ EE: true uses system fonts for all text." - Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). -- Any change that introduces an [Advanced Search migration](elasticsearch.md#creating-a-new-advanced-search-migration) +- Any change that introduces an [advanced search migration](search/advanced_search_migration_styleguide.md#creating-a-new-advanced-search-migration) **must** have a changelog entry. - A fix for a regression introduced and then fixed in the same release (such as fixing a bug introduced during a monthly release candidate) **should not** diff --git a/doc/development/cicd/cicd_tables.md b/doc/development/cicd/cicd_tables.md new file mode 100644 index 00000000000..c86540c10f0 --- /dev/null +++ b/doc/development/cicd/cicd_tables.md @@ -0,0 +1,119 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Add new tables to the CI database + +The [pipeline data partitioning](../../architecture/blueprints/ci_data_decay/pipeline_partitioning.md) +design blueprint describes how to partition existing tables in the CI domain. However, +you still need to add tables for new features. Sometimes these tables hold +references to larger tables that need to be partitioned. To reduce future +work, all tables that use a `belongs_to` association to partitionable tables +should be partitioned from the start. + +## Create a new routing table + +The database helpers for creating tables do not accept partitioning options, +so the best solution is to create the tables using raw SQL: + +```ruby + enable_lock_retries! + + def up + execute(<<~SQL) + CREATE TABLE p_ci_examples ( + id bigint NOT NULL, + partition_id bigint NOT NULL, + build_id bigint NOT NULL, + PRIMARY KEY (id, partition_id), + CONSTRAINT fk_bb490f12fe_p FOREIGN KEY (partition_id, build_id) + REFERENCES ci_builds(partition_id, id) + ON UPDATE CASCADE ON DELETE CASCADE + ) + PARTITION BY LIST (partition_id); + SQL + end + + def down + drop_table :p_ci_examples + end +``` + +This table is called a routing table and it does not hold any data. The +data is stored in partitions. + +When creating the routing table: + +- The table name must start with the `p_` prefix. There are analyzers in place to ensure that all queries go + through the routing tables and do not access the partitions directly. +- Each new table needs a `partition_id` column and its value must equal + the value from the related association. In this example, that is `ci_builds`. All resources + belonging to a pipeline share the same `partition_id` value. +- The primary key must have the columns ordered this way to allow efficient + search only by `id`. +- The foreign key constraint must include the `ON UPDATE CASCADE` option because + the `partition_id` value should be able to update it for re-balancing the + partitions. + +## Create the first partition + +Usually, you rely on the application to create the initial partition at boot time. +However, due to the high traffic on the CI tables and the large number of nodes, +it can be difficult to acquire a lock on the referenced table. +Consequently, during deployment, a node may fail to start. +To prevent this failure, you must ensure that the partition is already in place before +the application runs: + +```ruby + disable_ddl_transaction! + + def up + with_lock_retries do + connection.execute(<<~SQL) + LOCK TABLE ci_builds IN SHARE UPDATE EXCLUSIVE MODE; + LOCK TABLE ONLY p_ci_examples IN ACCESS EXCLUSIVE MODE; + SQL + + connection.execute(<<~SQL) + CREATE TABLE IF NOT EXISTS gitlab_partitions_dynamic.ci_examples_100 + PARTITION OF p_ci_examples + FOR VALUES IN (100); + SQL + end + end +``` + +Partitions are created in `gitlab_partitions_dynamic` schema. + +When creating a partition, remember: + +- Partition names do not use the `p_` prefix. +- The default value for `partition_id` is `100`. + +## Cascade the partition value + +To cascade the partition value, the module should use the `Ci::Partitionable` module: + +```ruby +class Ci::Example < Ci::ApplicationRecord + include Ci::Partitionable + + self.table_name = :p_ci_examples + self.primary_key = :id + + belongs_to :build, class_name: 'Ci::Build' + partitionable scope: :build, partitioned: true +end +``` + +## Manage partitions + +The model must be included in the [`PARTITIONABLE_MODELS`](https://gitlab.com/gitlab-org/gitlab/-/blob/920147293ae304639915f66b260dc14e4f629850/app/models/concerns/ci/partitionable.rb#L25-44) +list because it is used to test that the `partition_id` is +propagated correctly. + +If it's missing, specifying `partitioned: true` creates the first partition. The model also needs to be registered in the +[`postgres_partitioning.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/920147293ae304639915f66b260dc14e4f629850/config/initializers/postgres_partitioning.rb) +initializer. diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 641d0ea4f6f..2cc8fca02dd 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, concepts, howto --- -# CI/CD development documentation +# CI/CD development guidelines Development guides that are specific to CI/CD are listed here: @@ -147,6 +147,7 @@ This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/ There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the runner is registered to: - Select jobs for shared runner (instance level) + - Utilizes a fair scheduling algorithm which prioritizes projects with fewer running builds - Select jobs for group runner - Select jobs for project runner diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 4f6799d12d8..1bf4a780e26 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -436,7 +436,6 @@ To add a metric definition for a new template: ```yaml - name: p_ci_templates_my_template_name category: ci_templates - redis_slot: ci_templates aggregation: weekly ``` diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md index 107a1588e18..2034f57d995 100644 --- a/doc/development/code_intelligence/index.md +++ b/doc/development/code_intelligence/index.md @@ -4,7 +4,7 @@ group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Code Intelligence +# Code intelligence development guidelines > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/1576) in GitLab 13.1. diff --git a/doc/development/code_owners/index.md b/doc/development/code_owners/index.md new file mode 100644 index 00000000000..fa0385777e8 --- /dev/null +++ b/doc/development/code_owners/index.md @@ -0,0 +1,132 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Code Owners development guidelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219916) in GitLab 15.10. + +This document was created to help contributors understand the code design of +[Code Owners](../../user/project/code_owners.md). You should read this +document before making changes to the code for this feature. + +This document is intentionally limited to an overview of how the code is +designed, as code can change often. To understand how a specific part of the +feature works, view the code and the specs. The details here explain how the +major components of the Code Owners feature work. + +NOTE: +This document should be updated when parts of the codebase referenced in this +document are updated, removed, or new parts are added. + +## Business logic + +All of the business logic for code owners is located in the `Gitlab::CodeOwners` +namespace. Code Owners is an EE-only feature, so the files only exist in the `./ee` directory. + +- `Gitlab::CodeOwners`: the main module used to interact with the code owner rules. + - Defined in `./ee/lib/gitlab/code_owners.rb`. +- `Gitlab::CodeOwners::Loader`: finds the correct `CODEOWNER` file and loads the + content into a `Gitlab::CodeOwners::File` instance. + - Defined in `./ee/lib/gitlab/code_owners/loader.rb`. +- `Gitlab::CodeOwners::ReferenceExtractor`: extracts `CODEOWNER` user, group, + and email references from texts. + - Defined in `./ee/lib/gitlab/code_owners/reference_extractor.rb`. +- `Gitlab::CodeOwners::UsersLoader`: the correct `CODEOWNER` file and loads the + content into a `Gitlab::CodeOwners::File` instance. + - Defined in `./ee/lib/gitlab/code_owners/users_loader.rb`. +- `Gitlab::CodeOwners::GroupsLoader`: finds the correct `CODEOWNER` file and loads + the content into a `Gitlab::CodeOwners::File` instance. + - Defined in `./ee/lib/gitlab/code_owners/groups_loader.rb`. +- `Gitlab::CodeOwners::File`: wraps a `CODEOWNERS` file and exposes the data through + the class' public methods. + - Defined in `./ee/lib/gitlab/code_owners/file.rb`. +- `Gitlab::CodeOwners::Entry`: wraps an entry (a pattern and owners line) in a + `CODEOWNERS` file and exposes the data through the class' public methods. + - Defined in `./ee/lib/gitlab/code_owners/entry.rb`. +- `Gitlab::CodeOwners::Validator`: validates no files in the `CODEOWNERS` entries + have been changed when a user pushes to a protected branch with `require_code_owner_approval` enabled. + - Defined in `./ee/lib/gitlab/code_owners/validator.rb`. + +## Related models + +### `ProtectedBranch` + +The `ProtectedBranch` model is defined in `app/models/protected_branch.rb` and +extended in `ee/app/ee/models/protected_branch.rb`. The EE version includes a column +named `require_code_owner_approval` which prevents changes from being pushed directly +to the branch being protected if the file is listed in `CODEOWNERS`. + +### `ApprovalMergeRequestRule` + +The `ApprovalMergeRequestRule` model is defined in `ee/app/models/approval_merge_request_rule.rb`. +The model stores approval rules for a merge request. We use multiple rule types, +including a `code_owner` type rule. + +## Controllers and Services + +The following controllers and services below are being used for the approval +rules feature to work: + +### `Api::Internal::Base` + +This `/internal/allowed` endpoint is called when pushing to GitLab to ensure the +user is allowed to push. The `/internal/allowed` endpoint performs a `Gitlab::Checks::DiffCheck`. +In EE, this includes code owner checks. + +Defined in `lib/api/internal/base.rb`. + +### `Repositories::GitHttpController` + +When changes are pushed to GitLab over HTTP, the controller performs an access check +to ensure the user is allowed to push. The checks perform a `Gitlab::Checks::DiffCheck`. +In EE, this includes Code Owner checks. + +Defined in `app/controllers/repositories/git_http_controller.rb`. + +### `EE::Gitlab::Checks::DiffCheck` + +This module extends the CE `Gitlab::Checks::DiffChecks` class and adds code owner +validation. It uses the `Gitlab::CodeOwner::Validator` class to verify users are +not pushing files listed in `CODEOWNER` directly to a protected branch while the +branch requires code owner approval. + +### `MergeRequests::SyncCodeOwnerApprovalRules` + +This service is defined in `services/merge_requests/sync_code_owner_approval_rules.rb` and used for: + +- Deleting outdated code owner approval rules when new changes are pushed to a merge request. +- Creating code owner approval rules for each changed file in a merge request that is also listed in the `CODEOWNER` file. + +## Flow + +These flowcharts should help explain the flow from the controllers down to the +models for different features. + +### Push changes to a protected branch with `require_code_owner_approval` enabled + +```mermaid +graph TD + Api::Internal::Base --> Gitlab::GitAccess + Gitlab::GitAccess --> Gitlab::Checks::DiffCheck + Gitlab::Checks::DiffCheck --> Gitlab::CodeOwners::Validator + Gitlab::CodeOwners::Validator --> ProtectedBranch + Gitlab::CodeOwners::Validator --> Gitlab::CodeOwners::Loader + Gitlab::CodeOwners::Loader --> Gitlab::CodeOwners::Entry +``` + +### Sync code owner rules to merge request approval rules + +```mermaid +graph TD + EE::ProtectedBranches::CreateService --> MergeRequest::SyncCodeOwnerApprovalRules + EE::MergeRequestRefreshService --> MergeRequest::SyncCodeOwnerApprovalRules + EE::MergeRequests::ReloadMergeHeadDiffService --> MergeRequest::SyncCodeOwnerApprovalRules + EE::MergeRequests::CreateService --> MergeRequests::SyncCodeOwnerApprovalRulesWorker + EE::MergeRequests::UpdateService --> MergeRequests::SyncCodeOwnerApprovalRulesWorker + MergeRequests::SyncCodeOwnerApprovalRulesWorker --> MergeRequest::SyncCodeOwnerApprovalRules + MergeRequest::SyncCodeOwnerApprovalRules --> id1{delete outdated code owner rules} + MergeRequest::SyncCodeOwnerApprovalRules --> id2{create rule for each code owner entry} +``` diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 245fb2152cd..ed9404187ae 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -68,6 +68,9 @@ When a suitable [domain expert](#domain-experts) isn't available, you can choose To find a domain expert: +- In the Merge Request approvals widget, select [View eligible approvers](../user/project/merge_requests/approvals/rules.md#eligible-approvers). + This widget shows recommended and required approvals per area of the codebase. + These rules are defined in [Code Owners](../user/project/merge_requests/approvals/rules.md#code-owners-as-eligible-approvers). - View the list of team members who work in the [stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages) related to the merge request. - View team members' domain expertise on the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page or on the [GitLab team page](https://about.gitlab.com/company/team/). Domains are self-identified, so use your judgment to map the changes on your merge request to a domain. - Look for team members who have contributed to the files in the merge request. View the logs by running `git log <file>`. @@ -237,7 +240,7 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering ##### Compliance -1. You have confirmed that the correct [MR type label](contributing/issue_workflow.md#type-labels) has been applied. +1. You have confirmed that the correct [MR type label](labels/index.md) has been applied. ### The responsibility of the merge request author @@ -532,7 +535,7 @@ WARNING: Before taking the decision to merge: - Set the milestone. -- Confirm that the correct [MR type label](contributing/issue_workflow.md#type-labels) is applied. +- Confirm that the correct [MR type label](labels/index.md#type-labels) is applied. - Consider warnings and errors from danger bot, code quality, and other reports. Unless a strong case can be made for the violation, these should be resolved before merging. A comment must be posted if the MR is merged with any failed job. @@ -718,12 +721,7 @@ Enterprise Edition instance. This has some implications: cached value returns (say, from a string or nil to an array), change the cache key at the same time. 1. **Settings** should be added as a - [last resort](https://about.gitlab.com/handbook/product/product-principles/#convention-over-configuration). - If you're adding a new setting in `gitlab.yml`: - 1. Try to avoid that, and add to `ApplicationSetting` instead. - 1. Ensure that it is also - [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml#adding-a-new-setting-to-gitlabyml). - 1. Ensure that it is also [added to Charts](https://docs.gitlab.com/charts/development/style_guide.html), if needed. + [last resort](https://about.gitlab.com/handbook/product/product-principles/#convention-over-configuration). See [Adding a new setting to GitLab Rails](architecture.md#adding-a-new-setting-in-gitlab-rails). 1. **File system access** is not possible in a [cloud-native architecture](architecture.md#adapting-existing-and-introducing-new-components). Ensure that we support object storage for any file storage we need to perform. For more information, see the [uploads documentation](uploads/index.md). diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index aec487ed184..210370f9816 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -12,7 +12,7 @@ Follow these guidelines when contributing or reviewing design and user interface advice and best practices for code review in general. The basis for most of these guidelines is [Pajamas](https://design.gitlab.com/), -GitLab design system. We encourage you to [contribute to Pajamas](https://design.gitlab.com/get-started/contribute/) +GitLab design system. We encourage you to [contribute to Pajamas](https://design.gitlab.com/get-started/contributing/) with additions and improvements. ## Merge request reviews @@ -127,7 +127,7 @@ At any moment, but usually _during_ or _after_ the design's implementation: - Contribute [issues to Pajamas](https://design.gitlab.com/get-started/contributing#contribute-an-issue) for additions or enhancements to the design system. -- Create issues with the [`~UX debt`](issue_workflow.md#technical-and-ux-debt) +- Create issues with the [`~UX debt`](../labels/index.md#technical-and-ux-debt) label for intentional deviations from the agreed-upon UX requirements due to time or feasibility challenges, linking back to the corresponding issues or merge requests. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 55827e00e43..6a9dd34e703 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -36,7 +36,7 @@ for audiences of all ages. If you would like to contribute to GitLab: - Issues with the - [`~Seeking community contributions` label](issue_workflow.md#label-for-community-contributors) + [`~Seeking community contributions` label](../labels/index.md#label-for-community-contributors) are a great place to start. - Optimizing our tests is another great opportunity to contribute. You can use [RSpec profiling statistics](https://gitlab-org.gitlab.io/rspec_profiling_stats/) to identify @@ -49,11 +49,11 @@ If you would like to contribute to GitLab: The general flow of contributing to GitLab is: -1. [Create a fork](../../user/project/repository/forking_workflow.md#creating-a-fork) - of GitLab. In some cases, you will want to set up the +1. [Read about](https://gitlab.com/gitlab-community/meta) and [request access](https://gitlab.com/gitlab-community/meta#request-access-to-community-forks) + to the GitLab Community forks. In some cases, you will want to set up the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) to - [develop against your fork](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#develop-in-your-own-gitlab-fork). -1. Make your changes in your fork. + [develop against the GitLab community fork](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#develop-in-your-own-gitlab-fork). +1. Create a feature branch and make changes in the [GitLab community fork](https://gitlab.com/gitlab-community/gitlab). 1. When you're ready, [create a new merge request](../../user/project/merge_requests/creating_merge_requests.md). 1. In the merge request's description: - Ensure you provide complete and accurate information. @@ -107,30 +107,16 @@ your code has multiple disciplines, you may mention multiple merge request coach GitLab receives a lot of community contributions. If your code has not been reviewed within two working days of its initial submission, you can ask for help with `@gitlab-bot help`. -#### Addition of external libraries - -When submitting code to GitLab, you may feel that your contribution requires the aid of an external -library. If your code includes an external library, please provide a link to the library, as well as -reasons for including it. - -Mention a maintainer in merge requests that contain: - -- More than 500 changes. -- Any major [breaking changes](../deprecation_guidelines/index.md). -- External libraries. - -If you are not sure who to mention, the reviewer will do this for you early in the merge request process. - #### Issues workflow This [documentation](issue_workflow.md) outlines the current issue workflow: - [Issue triaging](issue_workflow.md#issue-triaging) -- [Labels](issue_workflow.md#labels) +- [Labels](../labels/index.md) - [Feature proposals](issue_workflow.md#feature-proposals) - [Issue weight](issue_workflow.md#issue-weight) - [Regression issues](issue_workflow.md#regression-issues) -- [Technical and UX debt](issue_workflow.md#technical-and-ux-debt) +- [Technical and UX debt](../labels/index.md#technical-and-ux-debt) - [Technical debt in follow-up issues](issue_workflow.md#technical-debt-in-follow-up-issues) #### Merge requests workflow diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index b55fef25302..583b39ebe56 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -37,300 +37,7 @@ the affected files to find someone. We also have triage automation in place, described [in our handbook](https://about.gitlab.com/handbook/engineering/quality/triage-operations/). -## Labels - -To allow for asynchronous issue handling, we use [milestones](https://gitlab.com/groups/gitlab-org/-/milestones) -and [labels](https://gitlab.com/gitlab-org/gitlab/-/labels). Leads and product managers handle most of the -scheduling into milestones. Labeling is a task for everyone. (For some projects, labels can be set only by GitLab team members and not by community contributors). - -Most issues will have labels for at least one of the following: - -- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::maintenance"`. -- Stage. For example: `~"devops::plan"` or `~"devops::create"`. -- Group. For example: `~"group::source code"`, `~"group::knowledge"`, or `~"group::editor"`. -- Category. For example: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, or `~"Category:Templates"`. -- Feature. For example: `~wiki`, `~ldap`, `~api`, `~issues`, or `~"merge requests"`. -- Department: `~UX`, `~Quality` -- Team: `~"Technical Writing"`, `~Delivery` -- Specialization: `~frontend`, `~backend`, `~documentation` -- Release Scoping: `~Deliverable`, `~Stretch`, `~"Next Patch Release"` -- Priority: `~"priority::1"`, `~"priority::2"`, `~"priority::3"`, `~"priority::4"` -- Severity: `~"severity::1"`, `~"severity::2"`, `~"severity::3"`, `~"severity::4"` - -Please add `~"breaking change"` label if the issue can be considered as a [breaking change](../deprecation_guidelines/index.md). - -Please add `~security` label if the issue is related to application security. - -All labels, their meaning and priority are defined on the -[labels page](https://gitlab.com/gitlab-org/gitlab/-/labels). - -If you come across an issue that has none of these, and you're allowed to set -labels, you can _always_ add the type, stage, group, and often the category/feature labels. - -### Type labels - -Type labels are very important. They define what kind of issue this is. Every -issue should have one and only one. - -The SSOT for type and subtype labels is [available in the handbook](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification). - -A number of type labels have a priority assigned to them, which automatically -makes them float to the top, depending on their importance. - -Type labels are always lowercase, and can have any color, besides blue (which is -already reserved for category labels). - -The descriptions on the [labels page](https://gitlab.com/groups/gitlab-org/-/labels) -explain what falls under each type label. - -The GitLab handbook documents [when something is a bug](https://about.gitlab.com/handbook/product/product-processes/#bug-issues) and [when it is a feature request](https://about.gitlab.com/handbook/product/product-processes/#feature-issues). - -### Stage labels - -Stage labels specify which [stage](https://about.gitlab.com/handbook/product/categories/#hierarchy) the issue belongs to. - -#### Naming and color convention - -Stage labels respects the `devops::<stage_key>` naming convention. -`<stage_key>` is the stage key as it is in the single source of truth for stages at -<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml> -with `_` replaced with a space. - -For instance, the "Manage" stage is represented by the `~"devops::manage"` label in -the `gitlab-org` group since its key under `stages` is `manage`. - -The current stage labels can be found by [searching the labels list for `devops::`](https://gitlab.com/groups/gitlab-org/-/labels?search=devops::). - -These labels are [scoped labels](../../user/project/labels.md#scoped-labels) -and thus are mutually exclusive. - -The Stage labels are used to generate the [direction pages](https://about.gitlab.com/direction/) automatically. - -### Group labels - -Group labels specify which [groups](https://about.gitlab.com/company/team/structure/#product-groups) the issue belongs to. - -It's highly recommended to add a group label, as it's used by our triage -automation to -[infer the correct stage label](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues-and-merge-requests). - -#### Naming and color convention - -Group labels respects the `group::<group_key>` naming convention and -their color is `#A8D695`. -`<group_key>` is the group key as it is in the single source of truth for groups at -<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml>, -with `_` replaced with a space. - -For instance, the "Pipeline Execution" group is represented by the -~"group::pipeline execution" label in the `gitlab-org` group since its key -under `stages.manage.groups` is `pipeline_execution`. - -The current group labels can be found by [searching the labels list for `group::`](https://gitlab.com/groups/gitlab-org/-/labels?search=group::). - -These labels are [scoped labels](../../user/project/labels.md#scoped-labels) -and thus are mutually exclusive. - -You can find the groups listed in the [Product Stages, Groups, and Categories](https://about.gitlab.com/handbook/product/categories/) page. - -We use the term group to map down product requirements from our product stages. -As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. - -### Category labels - -From the handbook's -[Product stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/#hierarchy) -page: - -> Categories are high-level capabilities that may be a standalone product at -another company, such as Portfolio Management, for example. - -It's highly recommended to add a category label, as it's used by our triage -automation to -[infer the correct group and stage labels](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues). - -If you are an expert in a particular area, it makes it easier to find issues to -work on. You can also subscribe to those labels to receive an email each time an -issue is labeled with a category label corresponding to your expertise. - -#### Naming and color convention - -Category labels respects the `Category:<Category Name>` naming convention and -their color is `#428BCA`. -`<Category Name>` is the category name as it is in the single source of truth for categories at -<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. - -For instance, the "DevOps Reports" category is represented by the -~"Category:DevOps Reports" label in the `gitlab-org` group since its -`devops_reports.name` value is "DevOps Reports". - -If a category's label doesn't respect this naming convention, it should be specified -with [the `label` attribute](https://about.gitlab.com/handbook/marketing/digital-experience/website/#category-attributes) -in <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. - -### Feature labels - -From the handbook's -[Product stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/#hierarchy) -page: - -> Features: Small, discrete functionalities, for example Issue weights. Some common -features are listed within parentheses to facilitate finding responsible PMs by keyword. - -It's highly recommended to add a feature label if no category label applies, as -it's used by our triage automation to -[infer the correct group and stage labels](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues). - -If you are an expert in a particular area, it makes it easier to find issues to -work on. You can also subscribe to those labels to receive an email each time an -issue is labeled with a feature label corresponding to your expertise. - -Examples of feature labels are `~wiki`, `~ldap`, `~api`, `~issues`, and `~"merge requests"`. - -#### Naming and color convention - -Feature labels are all-lowercase. - -### Facet labels - -To track additional information or context about created issues, developers may -add _facet labels_. Facet labels are also sometimes used for issue prioritization -or for measurements (such as time to close). An example of a facet label is the -~customer label, which indicates customer interest. - -### Department labels - -The current department labels are: - -- ~UX -- ~Quality - -### Team labels - -**Important**: Most of the historical team labels (like Manage or Plan) are -now deprecated in favor of [Group labels](#group-labels) and [Stage labels](#stage-labels). - -Team labels specify what team is responsible for this issue. -Assigning a team label makes sure issues get the attention of the appropriate -people. - -The current team labels are: - -- ~Delivery -- ~"Technical Writing" - -#### Naming and color convention - -Team labels are always capitalized so that they show up as the first label for -any issue. - -### Specialization labels - -These labels narrow the [specialization](https://about.gitlab.com/company/team/structure/#specialist) on a unit of work. - -- ~frontend -- ~backend -- ~documentation - -### Release scoping labels - -Release Scoping labels help us clearly communicate expectations of the work for the -release. There are three levels of Release Scoping labels: - -- ~Deliverable: Issues that are expected to be delivered in the current - milestone. -- ~Stretch: Issues that are a stretch goal for delivering in the current - milestone. If these issues are not done in the current release, they will - strongly be considered for the next release. -- ~"Next Patch Release": Issues to put in the next patch release. Work on these - first, and add the `~"Pick into X.Y"` label to the merge request, along with the - appropriate milestone. - -Each issue scheduled for the current milestone should be labeled ~Deliverable -or ~"Stretch". Any open issue for a previous milestone should be labeled -~"Next Patch Release", or otherwise rescheduled to a different milestone. - -### Priority labels - -We have the following priority labels: - -- ~"priority::1" -- ~"priority::2" -- ~"priority::3" -- ~"priority::4" - -Please refer to the issue triage [priority label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#priority) section in our handbook to see how it's used. - -### Severity labels - -We have the following severity labels: - -- ~"severity::1" -- ~"severity::2" -- ~"severity::3" -- ~"severity::4" - -Please refer to the issue triage [severity label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity) section in our handbook to see how it's used. - -### Label for community contributors - -There are many issues that have a clear solution with uncontroversial benefit to GitLab users. -However, GitLab might not have the capacity for all these proposals in the current roadmap. -These issues are labeled ~"Seeking community contributions" because we welcome merge requests to resolve them. - -Community contributors can submit merge requests for any issue they want, but -the ~"Seeking community contributions" label has a special meaning. It points to -changes that: - -1. We already agreed on, -1. Are well-defined, -1. Are likely to get accepted by a maintainer. - -We want to avoid a situation when a contributor picks an -~"Seeking community contributions" issue and then their merge request gets closed, -because we realize that it does not fit our vision, or we want to solve it in a -different way. - -We manually add the ~"Seeking community contributions" label to issues -that fit the criteria described above. -We do not automatically add this label, because it requires human evaluation. - -We recommend people that have never contributed to any open source project to -look for issues labeled `~"Seeking community contributions"` with a -[weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None&weight=1) or the `~"good for new contributors"` -[label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) -attached to it. -More experienced contributors are very welcome to tackle -[any of them](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None). - -For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues -with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&label_name[]=Community+challenge). -If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom -GitLab merchandise. - -If you've decided that you would like to work on an issue, please @-mention -the [appropriate product manager](https://about.gitlab.com/handbook/product/#who-to-talk-to-for-what) -as soon as possible. The product manager will then pull in appropriate GitLab team -members to further discuss scope, design, and technical considerations. This will -ensure that your contribution is aligned with the GitLab product and minimize -any rework and delay in getting it merged into main. - -GitLab team members who apply the ~"Seeking community contributions" label to an issue -should update the issue description with a responsible product manager, inviting -any potential community contributor to @-mention per above. - -### Stewardship label - -For issues related to the open source stewardship of GitLab, -there is the ~"stewardship" label. - -This label is to be used for issues in which the stewardship of GitLab -is a topic of discussion. For instance if GitLab Inc. is planning to add -features from GitLab EE to GitLab CE, related issues would be labeled with -~"stewardship". - -A recent example of this was the issue for -[bringing the time tracking API to GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/25517#note_20019084). +For information about which labels to apply to issues, see [Labels](../labels/index.md). ## Feature proposals @@ -399,31 +106,6 @@ The release manager will [update the notes](https://gitlab.com/gitlab-org/release-tools/blob/master/doc/pro-tips.md#update-the-regression-issue) in the regression issue as fixes are addressed. -## Technical and UX debt - -In order to track things that can be improved in the GitLab codebase, -we use the ~"technical debt" label in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). -We use the ~"UX debt" label when we choose to deviate from the MVC, in a way that harms the user experience. - -These labels should be added to issues that describe things that can be improved, -shortcuts that have been taken, features that need additional attention, and all -other things that have been left behind due to high velocity of development. -For example, code that needs refactoring should use the ~"technical debt" label, -something that didn't ship according to our Design System guidelines should -use the ~"UX debt" label. - -Everyone can create an issue, though you may need to ask for adding a specific -label, if you do not have permissions to do it by yourself. Additional labels -can be combined with these labels, to make it easier to schedule -the improvements for a release. - -Issues tagged with these labels have the same priority like issues -that describe a new feature to be introduced in GitLab, and should be scheduled -for a release by the appropriate person. - -Make sure to mention the merge request that the ~"technical debt" issue or -~"UX debt" issue is associated with in the description of the issue. - ## Technical debt in follow-up issues It's common to discover technical debt during development of a new feature. In @@ -459,6 +141,6 @@ and assignee. The maintainer must always agree before an outstanding discussion is resolved in this manner, and will be the one to create the issue. The title and description should be of the same quality as those created -[in the usual manner](#technical-and-ux-debt) - in particular, the issue title +[in the usual manner](../labels/index.md#technical-and-ux-debt) - in particular, the issue title **must not** begin with `Follow-up`! The creating maintainer should also expect to be involved in some capacity when work begins on the follow-up issue. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 01bfdae5999..f39d93a39bc 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -9,11 +9,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w We welcome merge requests from everyone, with fixes and improvements to GitLab code, tests, and documentation. The issues that are specifically suitable -for community contributions have the [`Seeking community contributions`](issue_workflow.md#label-for-community-contributors) +for community contributions have the +[`Seeking community contributions`](../labels/index.md#label-for-community-contributors) label, but you are free to contribute to any issue you want. +## Working from issues + +If you find an issue, please submit a merge request with a fix or improvement, +if you can, and include tests. + +If you want to add a new feature that is not labeled, it is best to first create +an issue (if there isn't one already) and leave a comment asking for it +to be labeled as `Seeking community contributions`. See the [feature proposals](issue_workflow.md#feature-proposals) +section. + +If you don't know how to fix the issue but can write a test that exposes the +issue, we will accept that as well. In general, bug fixes that include a +regression test are merged quickly. New features without proper tests +might be slower to receive feedback. + +If you are new to GitLab development (or web development in general), see the +[how to contribute](index.md#how-to-contribute) section to get started with +some potentially easy issues. + +## Merge request ownership + If an issue is marked for the current milestone at any time, even -when you are working on it, a GitLab team member may take over the merge request to ensure the work is finished before the release date. +when you are working on it, a GitLab team member may take over the merge request to ensure the work is finished before the release date. If a contributor is no longer actively working on a submitted merge request, we can: @@ -31,28 +53,18 @@ we credit the original author by adding a changelog entry crediting the author and optionally include the original author on at least one of the commits within the MR. -If you want to add a new feature that is not labeled, it is best to first create -an issue (if there isn't one already) and leave a comment asking for it -to be labeled as `Seeking community contributions`. See the [feature proposals](issue_workflow.md#feature-proposals) -section. +## Merge request guidelines for contributors + +### Getting started Merge requests should be submitted to the appropriate project at GitLab.com, for example [GitLab](https://gitlab.com/gitlab-org/gitlab/-/merge_requests), [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests), or [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests). -If you are new to GitLab development (or web development in general), see the -[how to contribute](index.md#how-to-contribute) section to get started with -some potentially easy issues. - -To start developing GitLab, download the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) +To start developing GitLab locally, download the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) and see the [Development section](../../index.md) for the required guidelines. -## Merge request guidelines for contributors - -If you find an issue, please submit a merge request with a fix or improvement, -if you can, and include tests. - NOTE: Consider placing your code behind a feature flag if you think it might affect production availability. Not sure? Read [When to use feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags). @@ -64,20 +76,19 @@ this by tagging them in an MR before submitting the code for review. Talking to team members can be helpful when making design decisions. Communicating the intent behind your changes can also help expedite merge request reviews. -If -you don't know how to fix the issue but can write a test that exposes the -issue, we will accept that as well. In general, bug fixes that include a -regression test are merged quickly. New features without proper tests -might be slower to receive feedback. +### Creating a merge request To create a merge request: -1. [Fork](../../user/project/repository/forking_workflow.md) the project into - your personal namespace (or group) on GitLab.com. -1. Create a feature branch in your fork (don't work off your [default branch](../../user/project/repository/branches/default.md)). +1. [Read about](https://gitlab.com/gitlab-community/meta) and [request access](https://gitlab.com/gitlab-community/meta#request-access-to-community-forks) + to the GitLab Community forks. In some cases, you will want to set up the + [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) to + [develop against the GitLab community fork](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#develop-in-your-own-gitlab-fork). +1. Create a feature branch in the [GitLab community fork](https://gitlab.com/gitlab-community/gitlab) + (don't work off the [default branch](../../user/project/repository/branches/default.md)). 1. Follow the [commit messages guidelines](#commit-messages-guidelines). 1. If you have multiple commits, combine them into a few logically organized commits. -1. Push the commits to your working branch in your fork. +1. Push the commits to your working branch in the fork. 1. Submit a merge request (MR) against the default branch of the upstream project. 1. The MR title should describe the change you want to make. 1. The MR description should give a reason for your change. @@ -191,6 +202,10 @@ To make sure that your merge request can be approved, please ensure that it meet the contribution acceptance criteria below: 1. The change is as small as possible. +1. If the merge request contains more than 500 changes: + - Explain the reason + - Mention a maintainer +1. Mention any major [breaking changes](../deprecation_guidelines/index.md). 1. Include proper tests and make all tests pass (unless it contains a test exposing a bug in existing code). Every new class should have corresponding unit tests, even if the class is exercised at a higher level, such as a feature test. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 28ce8e6ff4b..d24875e559a 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -5,7 +5,7 @@ group: Development info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Style guides +# Development style guides ## Editor/IDE styling standardization @@ -30,7 +30,9 @@ We were using Overcommit prior to Lefthook, so you may want to uninstall it firs ### Install Lefthook -1. Install the `lefthook` Ruby gem: +1. You can install lefthook in [different ways](https://github.com/evilmartians/lefthook/blob/master/docs/install.md#install-lefthook). + If you do not choose to install it globally (e.g. via Homebrew or package managers), and only want to use it for the GitLab project, + you can install the Ruby gem via: ```shell bundle install @@ -39,12 +41,18 @@ We were using Overcommit prior to Lefthook, so you may want to uninstall it firs 1. Install Lefthook managed Git hooks: ```shell + # If installed globally + lefthook install + # Or if installed via ruby gem bundle exec lefthook install ``` 1. Test Lefthook is working by running the Lefthook `pre-push` Git hook: ```shell + # If installed globally + lefthook run pre-push + # Or if installed via ruby gem bundle exec lefthook run pre-push ``` @@ -57,6 +65,18 @@ Lefthook is configured with a combination of: - Project configuration in [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml). - Any [local configuration](https://github.com/evilmartians/lefthook/blob/master/README.md#local-config). +### Lefthook auto-fixing files + +We have a custom lefthook target to run all the linters with auto-fix capabilities, +but just on the files which changed in your branch. + +```shell +# If installed globally +lefthook run auto-fix +# Or if installed via ruby gem +bundle exec lefthook run auto-fix +``` + ### Disable Lefthook temporarily To disable Lefthook temporarily, you can set the `LEFTHOOK` environment variable to `0`. For instance: diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index b08eaed2afa..ef1e563b668 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -141,27 +141,27 @@ To enable the Dangerfile on another existing GitLab project, complete the follow 1. Add [`gitlab-dangerfiles`](https://rubygems.org/gems/gitlab-dangerfiles) to your `Gemfile`. 1. Create a `Dangerfile` with the following content: - ```ruby - require "gitlab-dangerfiles" + ```ruby + require "gitlab-dangerfiles" - Gitlab::Dangerfiles.for_project(self, &:import_defaults) - ``` + Gitlab::Dangerfiles.for_project(self, &:import_defaults) + ``` 1. Add the following to your CI/CD configuration: - ```yaml - include: - - project: 'gitlab-org/quality/pipeline-common' - file: - - '/ci/danger-review.yml' - rules: - - if: $CI_SERVER_HOST == "gitlab.com" - ``` + ```yaml + include: + - project: 'gitlab-org/quality/pipeline-common' + file: + - '/ci/danger-review.yml' + rules: + - if: $CI_SERVER_HOST == "gitlab.com" + ``` 1. If your project is in the `gitlab-org` group, you don't need to set up any token as the `DANGER_GITLAB_API_TOKEN` variable is available at the group level. If not, follow these last steps: - 1. Create a [Project access tokens](../user/project/settings/project_access_tokens.md). - 1. Add the token as a CI/CD project variable named `DANGER_GITLAB_API_TOKEN`. + 1. Create a [Project access tokens](../user/project/settings/project_access_tokens.md). + 1. Add the token as a CI/CD project variable named `DANGER_GITLAB_API_TOKEN`. You should add the ~"Danger bot" label to the merge request before sending it for review. diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index 2c2999e69d6..823fb49a9ab 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -155,13 +155,13 @@ To limit impact on GitLab.com, a process exists to validate them asynchronously during weekend hours. Due to generally lower traffic and fewer deployments, FK validation can proceed at a lower level of risk. -### Schedule foreign key validation for a low-impact time +#### Schedule foreign key validation for a low-impact time 1. [Schedule the FK to be validated](#schedule-the-fk-to-be-validated). 1. [Verify the MR was deployed and the FK is valid in production](#verify-the-mr-was-deployed-and-the-fk-is-valid-in-production). 1. [Add a migration to validate the FK synchronously](#add-a-migration-to-validate-the-fk-synchronously). -### Schedule the FK to be validated +#### Schedule the FK to be validated 1. Create a merge request containing a post-deployment migration, which prepares the foreign key for asynchronous validation. @@ -198,7 +198,7 @@ def down end ``` -### Verify the MR was deployed and the FK is valid in production +#### Verify the MR was deployed and the FK is valid in production 1. Verify that the post-deploy migration was executed on GitLab.com using ChatOps with `/chatops run auto_deploy status <merge_sha>`. If the output returns `db/gprd`, @@ -208,7 +208,7 @@ end 1. Use [Database Lab](database_lab.md) to check if validation was successful. Ensure the output does not indicate the foreign key is `NOT VALID`. -### Add a migration to validate the FK synchronously +#### Add a migration to validate the FK synchronously After the foreign key is valid on the production database, create a second merge request that validates the foreign key synchronously. The schema changes @@ -240,19 +240,20 @@ end ``` -## Test database FK changes locally +### Test database FK changes locally You must test the database foreign key changes locally before creating a merge request. -### Verify the foreign keys validated asynchronously +#### Verify the foreign keys validated asynchronously Use the asynchronous helpers on your local environment to test changes for validating a foreign key: -1. Enable the feature flags by running `Feature.enable(:database_async_foreign_key_validation)` - and `Feature.enable(:database_reindexing)` in the Rails console. +1. Enable the feature flag by running `Feature.enable(:database_async_foreign_key_validation)` + in the Rails console. 1. Run `bundle exec rails db:migrate` so that it creates an entry in the async validation table. -1. Run `bundle exec rails gitlab:db:reindex` so that the FK is validated asynchronously. +1. Run `bundle exec rails gitlab:db:validate_async_constraints:all` so that the FK is validated + asynchronously on all databases. 1. To verify the foreign key, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d+ table_name` to check that your diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index c6fe6d16faf..67dccd99a6c 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -34,10 +34,13 @@ Background migrations can help when: - Populating one column based on JSON stored in another column. - Migrating data that depends on the output of external services. (For example, an API.) -NOTE: -If the batched background migration is part of an important upgrade, it must be announced -in the release post. Discuss with your Project Manager if you're unsure if the migration falls -into this category. +### Notes + +- If the batched background migration is part of an important upgrade, it must be announced + in the release post. Discuss with your Project Manager if you're unsure if the migration falls + into this category. +- You should use the [generator](#generator) to create batched background migrations, + so that required files are created by default. ## Isolation @@ -311,6 +314,22 @@ NOTE: When applying additional filters, it is important to ensure they are properly covered by an index to optimize `EachBatch` performance. In the example above we need an index on `(type, id)` to support the filters. See [the `EachBatch` documentation for more information](iterating_tables_in_batches.md). +## Generator + +The custom generator `batched_background_migration` scaffolds necessary files and +accepts `table_name`, `column_name`, and `feature_category` as arguments. Usage: + +```shell +bundle exec rails g batched_background_migration my_batched_migration --table_name=<table-name> --column_name=<column-name> --feature_category=<feature-category> +``` + +This command creates these files: + +- `db/post_migrate/20230214231008_queue_my_batched_migration.rb` +- `spec/migrations/20230214231008_queue_my_batched_migration_spec.rb` +- `lib/gitlab/background_migration/my_batched_migration.rb` +- `spec/lib/gitlab/background_migration/my_batched_migration_spec.rb` + ## Example The `routes` table has a `source_type` field that's used for a polymorphic relationship. @@ -319,8 +338,13 @@ the work is migrating data from the `source_id` column into a new singular forei Because we intend to delete old rows later, there's no need to update them as part of the background migration. -1. Start by defining our migration class, which should inherit - from `Gitlab::BackgroundMigration::BatchedMigrationJob`: +1. Start by using the generator to create batched background migration files: + + ```shell + bundle exec rails g batched_background_migration BackfillRouteNamespaceId --table_name=routes --column_name=id --feature_category=source_code_management + ``` + +1. Update the migration job (subclass of `BatchedMigrationJob`) to copy `source_id` values to `namespace_id`: ```ruby class Gitlab::BackgroundMigration::BackfillRouteNamespaceId < BatchedMigrationJob @@ -344,10 +368,10 @@ background migration. ``` NOTE: - Job classes must be subclasses of `BatchedMigrationJob` to be + Job classes inherit from `BatchedMigrationJob` to ensure they are correctly handled by the batched migration framework. Any subclass of - `BatchedMigrationJob` is initialized with necessary arguments to - execute the batch, as well as a connection to the tracking database. + `BatchedMigrationJob` is initialized with the necessary arguments to + execute the batch, and a connection to the tracking database. 1. Create a database migration that adds a new trigger to the database. Example: @@ -380,12 +404,14 @@ background migration. end ``` -1. Create a post-deployment migration that queues the migration for existing data: +1. Update the created post-deployment migration with required delay and batch sizes: ```ruby class QueueBackfillRoutesNamespaceId < Gitlab::Database::Migration[2.1] MIGRATION = 'BackfillRouteNamespaceId' DELAY_INTERVAL = 2.minutes + BATCH_SIZE = 1000 + SUB_BATCH_SIZE = 100 restrict_gitlab_migration gitlab_schema: :gitlab_main @@ -394,7 +420,9 @@ background migration. MIGRATION, :routes, :id, - job_interval: DELAY_INTERVAL + job_interval: DELAY_INTERVAL, + batch_size: BATCH_SIZE, + sub_batch_size: SUB_BATCH_SIZE ) end @@ -719,6 +747,99 @@ You can view failures in two ways: WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; ``` +### Executing a particular batch on the database testing pipeline + +NOTE: +Only [database maintainers](https://gitlab.com/groups/gitlab-org/maintainers/database/-/group_members?with_inherited_permissions=exclude) can view the database testing pipeline artifacts. Ask one for help if you need to use this method. + +Let's assume that a batched background migration failed on a particular batch on GitLab.com and you want to figure out which query failed and why. At the moment, we don't have a good way to retrieve query information (especially the query parameters) and rerunning the entire migration with more logging would be a long process. + +Fortunately you can leverage our [database migration pipeline](database_migration_pipeline.md) to rerun a particular batch with additional logging and/or fix to see if it solves the problem. + +<!-- vale gitlab.Substitutions = NO --> +For an example see [Draft: Test PG::CardinalityViolation fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110910) but make sure to read the entire section. + +To do that, you need to: + +1. Find the batch `start_id` and `end_id` +1. Create a regular migration +1. Apply a workaround for our migration helpers (optional) +1. Start the database migration pipeline + +#### 1. Find the batch `start_id` and `end_id` + +You should be able to find those in [Kibana][#viewing-failure-error-logs]. + +#### 2. Create a regular migration + +Schedule the batch in the `up` block of a regular migration: + +```ruby +def up + instance = Gitlab::BackgroundMigration::YourBackgroundMigrationClass.new( + start_id: <batch start_id>, + end_id: <batch end_id>, + batch_table: <table name>, + batch_column: <batching column>, + sub_batch_size: <sub batch size>, + pause_ms: <miliseconds between batches>, + job_arguments: <job arguments if any>, + connection: connection + ) + + instance.perform +end + + +def down + # no-op +end +``` + +#### 3. Apply a workaround for our migration helpers (optional) + +If your batched background migration touches tables from a schema other than the one you specified by using `restrict_gitlab_migration` helper (example: the scheduling migration has `restrict_gitlab_migration gitlab_schema: :gitlab_main` but the background job uses tables from the `:gitlab_ci` schema) then the migration will fail. To prevent that from happening you'll have to monkey patch database helpers so they don't fail the testing pipeline job: + +1. Add the schema names to [`RestrictGitlabSchema`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb#L57) + +```diff +diff --git a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb +index b8d1d21a0d2d2a23d9e8c8a0a17db98ed1ed40b7..912e20659a6919f771045178c66828563cb5a4a1 100644 +--- a/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb ++++ b/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb +@@ -55,7 +55,7 @@ def unmatched_schemas + end + + def allowed_schemas_for_connection +- Gitlab::Database.gitlab_schemas_for_connection(connection) ++ Gitlab::Database.gitlab_schemas_for_connection(connection) << :gitlab_ci + end + end + end +``` + +1. Add the schema names to [`RestrictAllowedSchemas`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb#L82) + +```diff +diff --git a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb +index 4ae3622479f0800c0553959e132143ec9051898e..d556ec7f55adae9d46a56665ce02de782cb09f2d 100644 +--- a/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb ++++ b/lib/gitlab/database/query_analyzers/restrict_allowed_schemas.rb +@@ -79,7 +79,7 @@ def restrict_to_dml_only(parsed) + tables = self.dml_tables(parsed) + schemas = self.dml_schemas(tables) + +- if (schemas - self.allowed_gitlab_schemas).any? ++ if (schemas - (self.allowed_gitlab_schemas << :gitlab_ci)).any? + raise DMLAccessDeniedError, \ + "Select/DML queries (SELECT/UPDATE/DELETE) do access '#{tables}' (#{schemas.to_a}) " \ + "which is outside of list of allowed schemas: '#{self.allowed_gitlab_schemas}'. " \ +``` + +#### 4. Start the database migration pipeline + +Create a Draft merge request with your changes and trigger the manual `db:gitlabcom-database-testing` job. + ### Adding indexes to support batched background migrations Sometimes it is necessary to add a new or temporary index to support a batched background migration. diff --git a/doc/development/database/clickhouse/merge_request_analytics.md b/doc/development/database/clickhouse/merge_request_analytics.md new file mode 100644 index 00000000000..34da71d6c4c --- /dev/null +++ b/doc/development/database/clickhouse/merge_request_analytics.md @@ -0,0 +1,355 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Merge request analytics with ClickHouse + +The [merge request analytics feature](../../../user/analytics/merge_request_analytics.md) +shows statistics about the merged merge requests in the project and also exposes record-level metadata. +Aggregations include: + +- **Average time to merge**: The duration between the creation time and the merge time. +- **Monthly aggregations**: A chart of 12 months of the merged merge requests. + +Under the chart, the user can see the paginated list of merge requests, 12 months per page. + +You can filter by: + +- Author +- Assignee +- Labels +- Milestone +- Source branch +- Target branch + +## Current performance problems + +- The aggregation queries require specialized indexes, which cost additional + disk space (index-only scans). +- Querying the whole 12 months is slow (statement timeout). Instead, the frontend + requests data per month (12 database queries). +- Even with specialized indexes, making the feature available on the group level + would not be feasible due to the large volume of merge requests. + +## Example queries + +Get the number of merge requests merged in a given month: + +```sql +SELECT COUNT(*) +FROM "merge_requests" +INNER JOIN "merge_request_metrics" ON "merge_request_metrics"."merge_request_id" = "merge_requests"."id" +WHERE (NOT EXISTS + (SELECT 1 + FROM "banned_users" + WHERE (merge_requests.author_id = banned_users.user_id))) + AND "merge_request_metrics"."target_project_id" = 278964 + AND "merge_request_metrics"."merged_at" >= '2022-12-01 00:00:00' + AND "merge_request_metrics"."merged_at" <= '2023-01-01 00:00:00' +``` + +The `merge_request_metrics` table was de-normalized (by adding `target_project_id`) +to improve the first-page load time. The query itself works well for smaller date ranges, +however, it can time out as the date range increases. + +After an extra filter is added, the query becomes more complex because it must also +filter the `merge_requests` table: + +```sql +SELECT COUNT(*) +FROM "merge_requests" +INNER JOIN "merge_request_metrics" ON "merge_request_metrics"."merge_request_id" = "merge_requests"."id" +WHERE (NOT EXISTS + (SELECT 1 + FROM "banned_users" + WHERE (merge_requests.author_id = banned_users.user_id))) + AND "merge_requests"."author_id" IN + (SELECT "users"."id" + FROM "users" + WHERE (LOWER("users"."username") IN (LOWER('ahegyi')))) + AND "merge_request_metrics"."target_project_id" = 278964 + AND "merge_request_metrics"."merged_at" >= '2022-12-01 00:00:00' + AND "merge_request_metrics"."merged_at" <= '2023-01-01 00:00:00' +``` + +To calculate mean time to merge, we also query the total time between the +merge request creation time and merge time. + +```sql +SELECT EXTRACT(epoch + FROM SUM(AGE(merge_request_metrics.merged_at, merge_request_metrics.created_at))) +FROM "merge_requests" +INNER JOIN "merge_request_metrics" ON "merge_request_metrics"."merge_request_id" = "merge_requests"."id" +WHERE (NOT EXISTS + (SELECT 1 + FROM "banned_users" + WHERE (merge_requests.author_id = banned_users.user_id))) + AND "merge_requests"."author_id" IN + (SELECT "users"."id" + FROM "users" + WHERE (LOWER("users"."username") IN (LOWER('ahegyi')))) + AND "merge_request_metrics"."target_project_id" = 278964 + AND "merge_request_metrics"."merged_at" >= '2022-08-01 00:00:00' + AND "merge_request_metrics"."merged_at" <= '2022-09-01 00:00:00' + AND "merge_request_metrics"."merged_at" > "merge_request_metrics"."created_at" +LIMIT 1 +``` + +## Store merge request data in ClickHouse + +Several other use cases exist for storing and querying merge request data in +ClickHouse. In this document, we focus on this particular feature. + +The core data exists in the `merge_request_metrics` and in the `merge_requests` +database tables. Some filters require extra tables to be joined: + +- `banned_users`: Filter out merge requests created by banned users. +- `labels`: A merge request can have one or more assigned labels. +- `assignees`: A merge request can have one or more assignees. +- `merged_at`: The `merged_at` column is located in the `merge_request_metrics` table. + +The `merge_requests` table contains data that can be filtered directly: + +- **Author**: via the `author_id` column. +- **Milestone**: via the `milestone_id` column. +- **Source branch**. +- **Target branch**. +- **Project**: via the `project_id` column. + +### Keep ClickHouse data up to date + +Replicating or syncing the `merge_requests` table is unfortunately not enough. +Separate queries to associated tables are required to insert one de-normalized +`merge_requests` row into the ClickHouse database. + +Change detection is non-trivial to implement. A few corners we could cut: + +- The feature is available for GitLab Premium and GitLab Ultimate customers. + We don't have to sync all the data, but instead sync only the `merge_requests` records + which are part of licensed groups. +- Data changes (often) happen via the `MergeRequest` services, where bumping the + `updated_at` timestamp column is mostly consistent. Some sort of incremental + synchronization process could be implemented. +- We only need to query the merged merge requests. After the merge, the record rarely changes. + +### Database table structure + +The database table structure uses de-normalization to make all required columns +available in one database table. This eliminates the need for `JOINs`. + +```sql +CREATE TABLE merge_requests +( + `id` UInt64, + `project_id` UInt64 DEFAULT 0 NOT NULL, + `author_id` UInt64 DEFAULT 0 NOT NULL, + `milestone_id` UInt64 DEFAULT 0 NOT NULL, + `label_ids` Array(UInt64) DEFAULT [] NOT NULL, + `assignee_ids` Array(UInt64) DEFAULT [] NOT NULL, + `source_branch` String DEFAULT '' NOT NULL, + `target_branch` String DEFAULT '' NOT NULL, + `merged_at` DateTime64(6, 'UTC') NOT NULL, + `created_at` DateTime64(6, 'UTC') DEFAULT now() NOT NULL, + `updated_at` DateTime64(6, 'UTC') DEFAULT now() NOT NULL +) +ENGINE = ReplacingMergeTree(updated_at) +ORDER BY (project_id, merged_at, id); +``` + +Similarly to the [activity data example](gitlab_activity_data.md), we use the +`ReplacingMergeTree` engine. Several columns of the merge request record may change, +so keeping the table up-to-date is important. + +The database table is ordered by the `project_id, merged_at, id` columns. This ordering +optimizes the table data for our use case: querying the `merged_at` column in a project. + +## Rewrite the count query + +First, let's generate some data for the table. + +```sql +INSERT INTO merge_requests (id, project_id, author_id, milestone_id, label_ids, merged_at, created_at) +SELECT id, project_id, author_id, milestone_id, label_ids, merged_at, created_at +FROM generateRandom('id UInt64, project_id UInt8, author_id UInt8, milestone_id UInt8, label_ids Array(UInt8), merged_at DateTime64(6, \'UTC\'), created_at DateTime64(6, \'UTC\')') +LIMIT 1000000; +``` + +NOTE: +Some integer data types were cast as `UInt8` so it is highly probable that they +have same values across different rows. + +The original count query only aggregated data for one month. With ClickHouse, we can +attempt aggregating the data for the whole year. + +PostgreSQL-based count query: + +```sql +SELECT COUNT(*) +FROM "merge_requests" +INNER JOIN "merge_request_metrics" ON "merge_request_metrics"."merge_request_id" = "merge_requests"."id" +WHERE (NOT EXISTS + (SELECT 1 + FROM "banned_users" + WHERE (merge_requests.author_id = banned_users.user_id))) + AND "merge_request_metrics"."target_project_id" = 278964 + AND "merge_request_metrics"."merged_at" >= '2022-12-01 00:00:00' + AND "merge_request_metrics"."merged_at" <= '2023-01-01 00:00:00' +``` + +ClickHouse query: + +```sql +SELECT + toYear(merged_at) AS year, + toMonth(merged_at) AS month, + COUNT(*) +FROM merge_requests +WHERE + project_id = 200 + AND merged_at BETWEEN '2022-01-01 00:00:00' + AND '2023-01-01 00:00:00' +GROUP BY year, month +``` + +The query processed a significantly lower number of rows compared to the generated data. +The `ORDER BY` clause (primary key) is helping the query execution: + +```plaintext +11 rows in set. Elapsed: 0.010 sec. +Processed 8.19 thousand rows, 131.07 KB (783.45 thousand rows/s., 12.54 MB/s.) +``` + +## Rewrite the Mean time to merge query + +The query calculates the mean time to merge as: +`duration(created_at, merged_at) / merge_request_count`. The calculation is done in +two separate steps: + +1. Request the monthly counts and the monthly duration values. +1. Sum the counts to get the yearly count. +1. Sum the durations to get the yearly duration. +1. Divide the durations by the count. + +In ClickHouse, we can calculate the mean time to merge with one query: + +```sql +SELECT + SUM( + dateDiff('second', merged_at, created_at) / 3600 / 24 + ) / COUNT(*) AS mean_time_to_merge -- mean_time_to_merge is in days +FROM merge_requests +WHERE + project_id = 200 + AND merged_at BETWEEN '2022-01-01 00:00:00' + AND '2023-01-01 00:00:00' +``` + +## Filtering + +The database queries above can be used as base queries. You can add more filters. +For example, filtering for a label and a milestone: + +```sql +SELECT + toYear(merged_at) AS year, + toMonth(merged_at) AS month, + COUNT(*) +FROM merge_requests +WHERE + project_id = 200 + AND milestone_id = 15 + AND has(label_ids, 118) + AND -- array includes 118 + merged_at BETWEEN '2022-01-01 00:00:00' + AND '2023-01-01 00:00:00' +GROUP BY year, month +``` + +Optimizing a particular filter is usually done with a database index. This particular +query reads 8000 rows: + +```plaintext +1 row in set. Elapsed: 0.016 sec. +Processed 8.19 thousand rows, 589.99 KB (505.38 thousand rows/s., 36.40 MB/s.) +``` + +Adding an index on `milestone_id`: + +```sql +ALTER TABLE merge_requests +ADD + INDEX milestone_id_index milestone_id TYPE minmax GRANULARITY 10; +ALTER TABLE + merge_requests MATERIALIZE INDEX milestone_id_index; +``` + +On the generated data, adding the index didn't improve the performance. + +### Banned users filter + +A recently added feature in GitLab filters out merge requests where the author is +banned by the admins. The banned users are tracked on the instance level in the +`banned_users` database table. + +#### Idea 1: Enumerate the banned user IDs + +This would require no structural changes to the ClickHouse database schema. +We could query the banned users in the project and filter the values out in query time. + +Get the banned users (in PostgreSQL): + +```sql +SELECT user_id FROM banned_users +``` + +In ClickHouse + +```sql +SELECT + toYear(merged_at) AS year, + toMonth(merged_at) AS month, + COUNT(*) +FROM merge_requests +WHERE + author_id NOT IN (1, 2, 3, 4) AND -- banned users + project_id = 200 + AND milestone_id = 15 + AND has(label_ids, 118) AND -- array includes 118 + merged_at BETWEEN '2022-01-01 00:00:00' + AND '2023-01-01 00:00:00' +GROUP BY year, month +``` + +The problem with this approach is that the number of banned users could increase significantly which would make the query bigger and slower. + +#### Idea 2: replicate the `banned_users` table + +Assuming that the `banned_users table` doesn't grow to millions of rows, we could +attempt to periodically sync the whole table to ClickHouse. With this approach, +a mostly consistent `banned_users` table could be used in the ClickHouse database query: + +```sql +SELECT + toYear(merged_at) AS year, + toMonth(merged_at) AS month, + COUNT(*) +FROM merge_requests +WHERE + author_id NOT IN (SELECT user_id FROM banned_users) AND + project_id = 200 AND + milestone_id = 15 AND + has(label_ids, 118) AND -- array includes 118 + merged_at BETWEEN '2022-01-01 00:00:00' AND '2023-01-01 00:00:00' +GROUP BY year, month +``` + +Alternatively, the `banned_users` table could be stored as a +[dictionary](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts) +to further improve the query performance. + +#### Idea 3: Alter the feature + +For analytical calculations, it might be acceptable to drop this particular filter. +This approach assumes that including the merge requests of banned users doesn't skew the statistics significantly. diff --git a/doc/development/database/clickhouse/tiered_storage.md b/doc/development/database/clickhouse/tiered_storage.md new file mode 100644 index 00000000000..d9026f47e28 --- /dev/null +++ b/doc/development/database/clickhouse/tiered_storage.md @@ -0,0 +1,138 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tiered Storages in ClickHouse + +NOTE: +The MergeTree table engine in ClickHouse supports tiered storage. +See the documentation for [Using Multiple Block Devices for Data Storage](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes) +for details on setup and further explanation. + +Quoting from the [MergeTree documentation](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes): + +<!-- vale gitlab.Simplicity = NO --> + +> MergeTree family table engines can store data on multiple block devices. For example, +> it can be useful when the data of a certain table are implicitly split into "hot" and "cold". +> The most recent data is regularly requested but requires only a small amount of space. +> On the contrary, the fat-tailed historical data is requested rarely. + +<!-- vale gitlab.Simplicity = YES --> + +When used with remote storage backends such as +[Amazon S3](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-s3), +this makes a very efficient storage scheme. It allows for storage policies, which +allows data to be on local disks for a period of time and then move it to object storage. + +An [example configuration](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes_configure) can look like this: + +```xml +<storage_configuration> + <disks> + <fast_ssd> + <path>/mnt/fast_ssd/clickhouse/</path> + </fast_ssd> + <gcs> + <support_batch_delete>false</support_batch_delete> + <type>s3</type> + <endpoint>https://storage.googleapis.com/${BUCKET_NAME}/${ROOT_FOLDER}/</endpoint> + <access_key_id>${SERVICE_ACCOUNT_HMAC_KEY}</access_key_id> + <secret_access_key>${SERVICE_ACCOUNT_HMAC_SECRET}</secret_access_key> + <metadata_path>/var/lib/clickhouse/disks/gcs/</metadata_path> + </gcs> + ... + </disks> + ... + <policies> + + <move_from_local_disks_to_gcs> <!-- policy name --> + <volumes> + <hot> <!-- volume name --> + <disk>fast_ssd</disk> <!-- disk name --> + </hot> + <cold> + <disk>gcs</disk> + </cold> + </volumes> + <move_factor>0.2</move_factor> + <!-- The move factor determines when to move data from hot volume to cold. + See ClickHouse docs for more details. --> + </moving_from_ssd_to_hdd> + .... +</storage_configuration> +``` + +In this storage policy, two volumes are defined `hot` and `cold`. After the `hot` volume is filled with occupancy of `disk_size * move_factor`, the data is being moved to Google Cloud Storage (GCS). + +If this storage policy is not the default, create tables by attaching the storage policies. For example: + +```sql +CREATE TABLE key_value_table ( + event_date Date, + key String, + value String, +) ENGINE = MergeTree +ORDER BY (key) +PARTITION BY toYYYYMM(event_date) +SETTINGS storage_policy = 'move_from_local_disks_to_gcs' +``` + +NOTE: +In this storage policy, the move happens implicitly. It is also possible to keep +_hot_ data on local disks for a fixed period of time and then move them as _cold_. + +This approach is possible with +[Table TTLs](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#mergetree-table-ttl), +which are also available with MergeTree table engine. + +The ClickHouse documentation shows this feature in detail, in the example of +[implementing a hot - warm - cold architecture](https://clickhouse.com/docs/en/guides/developer/ttl/#implementing-a-hotwarmcold-architecture). + +You can take a similar approach for the example shown above. First, adjust the storage policy: + +```xml +<storage_configuration> + ... + <policies> + <local_disk_and_gcs> <!-- policy name --> + <volumes> + <hot> <!-- volume name --> + <disk>fast_ssd</disk> <!-- disk name --> + </hot> + <cold> + <disk>gcs</disk> + </cold> + </volumes> + </local_disk_and_gcs> + .... +</storage_configuration> +``` + +Then create the table as: + +```sql +CREATE TABLE another_key_value_table ( + event_date Date, + key String, + value String, +) ENGINE = MergeTree +ORDER BY (key) +PARTITION BY toYYYYMM(event_date) +TTL + event_date TO VOLUME 'hot', + event_date + INTERVAL 1 YEAR TO VOLUME 'cold' +SETTINGS storage_policy = 'local_disk_and_gcs'; +``` + +This creates the table so that data older than 1 year (evaluated against the +`event_date` column) is moved to GCS. Such a storage policy can be helpful for append-only +tables (like audit events) where only the most recent data is accessed frequently. +You can drop the data altogether, which can be a regulatory requirement. + +We don't mention modifying TTLs in this guide, but that is possible as well. +See ClickHouse documentation for +[modifying TTL](https://clickhouse.com/docs/en/sql-reference/statements/alter/ttl/#modify-ttl) +for details. diff --git a/doc/development/database/database_dictionary.md b/doc/development/database/database_dictionary.md index b7e6fa4b5b3..515bf00c6f0 100644 --- a/doc/development/database/database_dictionary.md +++ b/doc/development/database/database_dictionary.md @@ -26,6 +26,7 @@ feature_categories: description: Represents a Terraform state backend introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26619 milestone: '13.0' +gitlab_schema: gitlab_main ``` ## Adding tables diff --git a/doc/development/database/database_lab.md b/doc/development/database/database_lab.md index 162fc597cc4..e428b9d4b0a 100644 --- a/doc/development/database/database_lab.md +++ b/doc/development/database/database_lab.md @@ -12,6 +12,17 @@ on replicated production data. Unlike a typical read-only production replica, in also create, update, and delete rows. You can also test the performance of schema changes, like additional indexes or columns, in an isolated copy of production data. +## Database Lab quick start + +1. [Visit the console](https://console.postgres.ai/). +1. Select **Sign in with Google**. (Not GitLab, as you need Google SSO to connect with our project.) +1. After you sign in, select the GitLab organization and then visit "Ask Joe" in the sidebar. +1. Select the database you're testing against: + - Most queries for the GitLab project run against `gitlab-production-tunnel-pg12`. + - If the query is for a CI table, select `gitlab-production-ci`. + - If the query is for the container registry, select `gitlab-production-registry`. +1. Type `explain <Query Text>` in the chat box to get a plan. + ## Access Database Lab Engine Access to the DLE is helpful for: @@ -21,7 +32,7 @@ Access to the DLE is helpful for: To access the DLE's services, you can: -- Perform query testing in the `#database_lab` Slack channel, or in the Postgres.ai web console. +- Perform query testing in the Postgres.ai web console. Employees access both services with their GitLab Google account. Query testing provides `EXPLAIN` (analyze, buffers) plans for queries executed there. - Migration testing by triggering a job as a part of a merge request. @@ -40,8 +51,6 @@ This procedure is similar to [Rails console access with Teleport](https://gitlab You can access Database Lab's query analysis features either: -- In the `#database_lab` Slack channel. Shows everyone's commands and results, but - your own commands are still isolated in their own clone. - In [the Postgres.ai web console](https://console.postgres.ai/GitLab/joe-instances). Shows only the commands you run. diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index 06e16b4c7f1..2344ee3f942 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/171) in GitLab 14.2. With the [automated migration testing pipeline](https://gitlab.com/gitlab-org/database-team/gitlab-com-database-testing) -we can automatically test migrations in a production-like environment (similar to `#database-lab`). +we can automatically test migrations in a production-like environment (using [Database Lab](database_lab.md)). It is based on an [architecture blueprint](../../architecture/blueprints/database_testing/index.md). Migration testing is enabled in the [GitLab project](https://gitlab.com/gitlab-org/gitlab) diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index aa92134018d..933bbe9c060 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -53,9 +53,8 @@ that require a more in-depth discussion between the database reviewers and maint - [Database Office Hours Agenda](https://docs.google.com/document/d/1wgfmVL30F8SdMg-9yY6Y8djPSxWNvKmhR5XmsvYX1EI/edit). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [YouTube playlist with past recordings](https://www.youtube.com/playlist?list=PL05JrBw4t0Kp-kqXeiF7fF7cFYaKtdqXM). -You should also join the [#database-lab](understanding_explain_plans.md#database-lab-engine) -Slack channel and get familiar with how to use Joe, the Slackbot that provides developers -with their own clone of the production database. +Get familiar with using [Database Lab from postgres.ai](database_lab.md), a bot that +provides developers with their own clone of the production database. Understanding and efficiently using `EXPLAIN` plans is at the core of the database review process. The following guides provide a quick introduction and links to follow on more advanced topics: diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 8f22eaac496..2cb8509e203 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -4,7 +4,7 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Database guides +# Database development guidelines ## Database Reviews @@ -109,6 +109,8 @@ including the major methods: - [Introduction](clickhouse/index.md) - [Optimizing query execution](clickhouse/optimization.md) - [Rebuild GitLab features using ClickHouse 1: Activity data](clickhouse/gitlab_activity_data.md) +- [Rebuild GitLab features using ClickHouse 2: Merge Request analytics](clickhouse/merge_request_analytics.md) +- [Tiered Storage in ClickHouse](clickhouse/tiered_storage.md) ## Miscellaneous diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index d22e3209096..c1b30a4cbbe 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Pods +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/database/query_performance.md b/doc/development/database/query_performance.md index 73a6a40f801..10ab726940a 100644 --- a/doc/development/database/query_performance.md +++ b/doc/development/database/query_performance.md @@ -44,7 +44,7 @@ automatically includes these options. If you are making a warm cache query, you see only the `shared hits`. -For example in #database-lab: +For example, using [Database Lab](database_lab.md): ```plaintext Shared buffers: @@ -60,7 +60,7 @@ Buffers: shared hit=7323 If the cache is cold, you also see `reads`. -In #database-lab: +Using [Database Lab](database_lab.md): ```plaintext Shared buffers: diff --git a/doc/development/database/required_stops.md b/doc/development/database/required_stops.md index 46fabb5c1b4..b706babbc5e 100644 --- a/doc/development/database/required_stops.md +++ b/doc/development/database/required_stops.md @@ -11,6 +11,36 @@ disruptive effect on customers. Before adding a required stop, consider if any alternative approaches exist to avoid a required stop. Sometimes a required stop is unavoidable. In those cases, follow the instructions below. +## Common scenarios that require stops + +### Long running migrations being finalized + +If a migration takes a long time, it could cause a large number of customers to encounter timeouts +during upgrades. The increased support volume may cause us to introduce a required stop. While any +background migration may cause these issues with particularly large customers, we typically only +introduce stops when the impact is widespread. + +- **Cause:** When an upgrade takes more than an hour, omnibus times out. +- **Mitigation:** Schedule finalization for the first minor version after the next required stop. + +### Improperly finalized background migrations + +You may need to introduce a required stop for mitigation when: + +- A background migration is not finalized, and +- A migration is written that depends on that background migration. + +- **Cause:** The dependent migration may fail if the background migration is incomplete. +- **Mitigation:** Ensure that all background migrations are finalized before authoring dependent migrations. + +### Bugs in migration related tooling + +In a few circumstances, bugs in migration related tooling has required us to introduce stops. While we aim +to prevent these in testing, sometimes they happen. + +- **Cause:** There have been a few different causes where we recognized these too late. +- **Mitigation:** Typically we try to backport fixes for migrations, but in some cases this is not possible. + ## Before the required stop is released Before releasing a known required stop, complete these steps. If the required stop diff --git a/doc/development/database/understanding_explain_plans.md b/doc/development/database/understanding_explain_plans.md index 094bd6b346f..560744430f9 100644 --- a/doc/development/database/understanding_explain_plans.md +++ b/doc/development/database/understanding_explain_plans.md @@ -714,8 +714,7 @@ SQL optimization tool - [Joe Bot](https://gitlab.com/postgres-ai/joe). Database Lab Engine provides developers with their own clone of the production database, while Joe Bot helps with exploring execution plans. -Joe Bot is available in the [`#database-lab`](https://gitlab.slack.com/archives/CLJMDRD8C) channel on Slack, -and through its [web interface](https://console.postgres.ai/gitlab/joe-instances). +Joe Bot is available through its [web interface](https://console.postgres.ai/gitlab/joe-instances). With Joe Bot you can execute DDL statements (like creating indexes, tables, and columns) and get query plans for `SELECT`, `UPDATE`, and `DELETE` statements. @@ -792,34 +791,6 @@ Planning time: 0.411 ms Execution time: 0.113 ms ``` -### ChatOps - -GitLab team members can also use our ChatOps solution, available in Slack -using the [`/chatops` slash command](../chatops_on_gitlabcom.md). - -NOTE: -While ChatOps is still available, the recommended way to generate execution plans is to use [Database Lab Engine](#database-lab-engine). - -You can use ChatOps to get a query plan by running the following: - -```sql -/chatops run explain SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) -``` - -Visualising the plan using <https://explain.depesz.com/> is also supported: - -```sql -/chatops run explain --visual SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) -``` - -Quoting the query is not necessary. - -For more information about the available options, run: - -```sql -/chatops run explain --help -``` - ## Further reading A more extensive guide on understanding query plans can be found in diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 048482960f4..1f653f8af94 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -68,7 +68,7 @@ Refer to [Preparation when adding or modifying queries](#preparation-when-adding A merge request **author**'s role is to: - Decide whether a database review is needed. -- If database review is needed, add the ~database label. +- If database review is needed, add the `~database` label. - [Prepare the merge request for a database review](#how-to-prepare-the-merge-request-for-a-database-review). - Provide the [required](#required) artifacts prior to submitting the MR. @@ -99,7 +99,7 @@ The MR author should request a review from the suggested database the suggested database **maintainer**. If reviewer roulette didn't suggest a database reviewer & maintainer, -make sure you have applied the ~database label and rerun the +make sure you have applied the `~database` label and rerun the `danger-review` CI job, or pick someone from the [`@gl-database` team](https://gitlab.com/groups/gl-database/-/group_members). @@ -122,14 +122,14 @@ the following preparations into account. - When [high-traffic](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) tables are involved in the migration, use the [`enable_lock_retries`](migration_style_guide.md#retry-mechanism-when-acquiring-database-locks) method to enable lock-retries. Review the relevant [examples in our documentation](migration_style_guide.md#usage-with-transactional-migrations) for use cases and solutions. - Ensure RuboCop checks are not disabled unless there's a valid reason to. - When adding an index to a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), -test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slack channel and add the execution time to the MR description: - - Execution time largely varies between `#database-lab` and GitLab.com, but an elevated execution time from `#database-lab` + test its execution using `CREATE INDEX CONCURRENTLY` in [Database Lab](database/database_lab.md) and add the execution time to the MR description: + - Execution time largely varies between Database Lab and GitLab.com, but an elevated execution time from Database Lab can give a hint that the execution on GitLab.com is also considerably high. - - If the execution from `#database-lab` is longer than `1h`, the index should be moved to a [post-migration](database/post_deployment_migrations.md). + - If the execution from Database Lab is longer than `1h`, the index should be moved to a [post-migration](database/post_deployment_migrations.md). Keep in mind that in this case you may need to split the migration and the application changes in separate releases to ensure the index is in place when the code that needs it is deployed. - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage. - - This job runs migrations in a production-like environment (similar to `#database_lab`) and posts to the MR its findings (queries, runtime, size change). + - This job runs migrations in a [Database Lab](database/database_lab.md) clone and posts to the MR its findings (queries, runtime, size change). - Review migration runtimes and any warnings. #### Preparation when adding data migrations @@ -173,13 +173,11 @@ Include in the MR description: ##### Query Plans - The query plan for each raw SQL query included in the merge request along with the link to the query plan following each raw SQL snippet. -- Provide a public link to the plan from either: - - [postgres.ai](https://postgres.ai/): Follow the link in `#database-lab` and generate a shareable, public link - by selecting **Share** in the upper right corner. - - [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com): Paste both the plan and the query used in the form. +- Provide a link to the plan from [postgres.ai](database/database_lab.md), provided by the chatbot. + - If it's not possible to get an accurate picture in Database Lab, you may need to seed a development environment, and instead provide links + from [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com). Be sure to paste both the plan + and the query used in the form. - When providing query plans, make sure it hits enough data: - - You can use a GitLab production replica to test your queries on a large scale, - through the `#database-lab` Slack channel or through [ChatOps](database/understanding_explain_plans.md#chatops). - To produce a query plan with enough data, you can use the IDs of: - The `gitlab-org` namespace (`namespace_id = 9970`), for queries involving a group. - The `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`) projects, for queries involving a project. @@ -188,7 +186,7 @@ Include in the MR description: - That means that no query plan should return 0 records or less records than the provided limit (if a limit is included). If a query is used in batching, a proper example batch with adequate included results should be identified and provided. - If your queries belong to a new feature in GitLab.com and thus they don't return data in production: - You may analyze the query and to provide the plan from a local environment. - - `#database-lab` and [postgres.ai](https://postgres.ai/) both allow updates to data (`exec UPDATE issues SET ...`) and creation of new tables and columns (`exec ALTER TABLE issues ADD COLUMN ...`). + - [postgres.ai](https://postgres.ai/) allows updates to data (`exec UPDATE issues SET ...`) and creation of new tables and columns (`exec ALTER TABLE issues ADD COLUMN ...`). - More information on how to find the number of actual returned records in [Understanding EXPLAIN plans](database/understanding_explain_plans.md) - For query changes, it is best to provide both the SQL queries along with the plan _before_ and _after_ the change. This helps spot differences quickly. @@ -231,7 +229,7 @@ Include in the MR description: - [Check indexes are present for foreign keys](migration_style_guide.md#adding-foreign-key-constraints) - Ensure that migrations execute in a transaction or only contain concurrent index/foreign key helpers (with transactions disabled) - - If an index to a large table is added and its execution time was elevated (more than 1h) on `#database-lab`: + - If an index to a large table is added and its execution time was elevated (more than 1h) on [Database Lab](database/database_lab.md): - Ensure it was added in a post-migration. - Maintainer: After the merge request is merged, notify Release Managers about it on `#f_upcoming_release` Slack channel. - Check consistency with `db/structure.sql` and that migrations are [reversible](migration_style_guide.md#reversibility) @@ -269,8 +267,7 @@ Include in the MR description: - Check for any overly complex queries and queries the author specifically points out for review (if any) - If not present, ask the author to provide SQL queries and query plans - (for example, by using [ChatOps](database/understanding_explain_plans.md#chatops) or direct - database access) + using [Database Lab](database/database_lab.md) - For given queries, review parameters regarding data distribution - [Check query plans](database/understanding_explain_plans.md) and suggest improvements to queries (changing the query, schema or adding indexes and similar) diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index e532fa82011..bc14b0f127c 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -62,6 +62,8 @@ A breaking change can be considered major if it affects many users, or represent Deprecations should be announced on the [Deprecated feature removal schedule](../../update/deprecations.md). +Deprecations should be announced [no later than the third milestone preceding intended removal](https://about.gitlab.com/handbook/product/gitlab-the-product/#process-for-deprecating-and-removing-a-feature). + Do not include the deprecation announcement in the merge request that introduces a code change for the deprecation. Use a separate MR to create a deprecation entry. For steps to create a deprecation entry, see [Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). @@ -77,7 +79,7 @@ However, at GitLab, we [give agency](https://about.gitlab.com/handbook/values/#g Generally, feature or configuration can be removed/changed only on major release. It also should be [deprecated in advance](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). -For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/rest/index.md#compatibility-guidelines) guidelines. +For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../documentation/restful_api_styleguide.md#deprecations) guidelines. For configuration removals, see the [Omnibus deprecation policy](../../administration/package_information/deprecation_policy.md). diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 79d0ff84713..bd9e7035290 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Distributed Tracing - development guidelines +# Distributed tracing development guidelines GitLab is instrumented for distributed tracing. Distributed Tracing in GitLab is currently considered **experimental**, as it has not yet been tested at scale on GitLab.com. diff --git a/doc/development/distribution/index.md b/doc/development/distribution/index.md new file mode 100644 index 00000000000..168e3854eca --- /dev/null +++ b/doc/development/distribution/index.md @@ -0,0 +1,35 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +description: "GitLab's development guidelines for Distribution" +--- + +# Contribute to GitLab Distribution + +Learn how to add new components and services to the GitLab application. + +## Support all package methods + +Additions must support both Omnibus GitLab and Cloud Native GitLab. Changes +to one must be made to the other to retain feature parity. + +## Contributing + +The primary projects handled by Distribution are listed below. For more +information, visit the [Distribution team engineering handbook page](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/) +or select one of the subsections in the navigation bar. + +### GitLab application + +- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) +- [Cloud Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG) +- [GitLab Operator](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator) +- [GitLab Chart](https://gitlab.com/gitlab-org/charts/gitlab) + +### Components and tools + +- [Omnibus GitLab Builder](https://gitlab.com/gitlab-org/gitlab-omnibus-builder) +- [Omnibus Fork](https://gitlab.com/gitlab-org/omnibus) +- [GitLab Logger](https://gitlab.com/gitlab-org/cloud-native/gitlab-logger) +- [Issue Bot](https://gitlab.com/gitlab-org/distribution/issue-bot) diff --git a/doc/development/documentation/alpha_beta.md b/doc/development/documentation/alpha_beta.md new file mode 100644 index 00000000000..4b5c24d512f --- /dev/null +++ b/doc/development/documentation/alpha_beta.md @@ -0,0 +1,43 @@ +--- +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects +stage: none +group: unassigned +--- + +# Document Alpha, Beta, LA features + +Some features are not generally available and are instead considered +[Alpha, Beta, or Limited Availability](../../policy/alpha-beta-support.md). + +When you document a feature in one of these three statuses: + +- Add `(Alpha)`, `(Beta)`, or `(Limited Availability)` in parentheses after the page or topic title. +- Do not include `(Alpha)`, `(Beta)`, or `(Limited Availability)` in the left nav. +- Ensure the version history lists the feature's status. + +These features are usually behind a feature flag, which follow [these documentation guidelines](feature_flags.md). + +If you add details of how users should enroll, or how to contact the team with issues, +the `FLAG:` note should be above these details. For example: + +```markdown +## Great new feature (Alpha) + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, ask an administrator to enable the feature flag named example_flag. +On GitLab.com, this feature is not available. This feature is not ready for production use. + +Use this great new feature when you need to do this new thing. + +This feature is in Alpha. To join the list of users testing this feature, +do this thing. If you find a bug, [open an issue](link). +``` + +When the feature is released, remove: + +- The text in parentheses. +- Any language about the feature not being ready for production. +- The feature flag information. + +Ensure the version history is up-to-date. diff --git a/doc/development/documentation/contribute.md b/doc/development/documentation/contribute.md new file mode 100644 index 00000000000..8b08743c6e9 --- /dev/null +++ b/doc/development/documentation/contribute.md @@ -0,0 +1,83 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Contribute to the GitLab documentation + +Everyone is welcome to update the GitLab documentation! + +## Work without an issue + +You don't need an issue to update the documentation. + +On [https://docs.gitlab.com](https://docs.gitlab.com), at the bottom of any page, +you can select **View page source** or **Edit in Web IDE** and [get started with a merge request](#open-your-merge-request). + +You can alternately: + +- Choose a page [in the `/doc` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) + and edit it from there. +- Try installing and running the [Vale linting tool](testing.md#vale) + and fixing the resulting issues. + +When you're developing code, the workflow for updating docs is slightly different. +For details, see the [merge request workflow](../contributing/merge_request_workflow.md). + +## Search available issues + +If you're looking for an open issue, you can +[review the list of documentation issues curated specifically for new contributors](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=documentation&label_name%5B%5D=docs-only&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=20). + +When you find an issue you'd like to work on: + +- If the issue is already assigned to someone, pick a different one. +- If the issue is unassigned, add a comment and ask to work on the issue. For a Hackathon, use `@docs-hackathon`. Otherwise, use `@gl-docsteam`. For example: + + ```plaintext + @docs-hackathon I would like to work on this issue + ``` + +- Do not ask for more than three issues at a time. + +## Open your merge request + +When you are ready to update the documentation: + +1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). +1. In the upper-right corner, select **Fork**. Forking makes a copy of the repository on GitLab.com. +1. In your fork, find the documentation page in the `\doc` directory. +1. If you know Git, make your changes and open a merge request. + If not, follow these steps: + 1. In the upper-right corner, select **Edit** if it is visible. + If it is not, select the down arrow (**{chevron-lg-down}**) next to + **Open in Web IDE** or **Gitpod**, and select **Edit**. + 1. In the **Commit message** text box, enter a commit message. + Use 3-5 words, start with a capital letter, and do not end with a period. + 1. Select **Commit changes**. + 1. On the left sidebar, select **Merge requests**. + 1. Select **New merge request**. + 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. + For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. + 1. Select **Compare branches and continue**. A new merge request opens. + 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. + 1. Select **Create merge request**. + +## Ask for help + +Ask for help from the Technical Writing team if you: + +- Need help to choose the correct place for documentation. +- Want to discuss a documentation idea or outline. +- Want to request any other help. + +To identify someone who can help you: + +1. Locate the Technical Writer for the relevant + [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). +1. Either: + - If urgent help is required, directly assign the Technical Writer in the issue or in the merge request. + - If non-urgent help is required, ping the Technical Writer in the issue or merge request. + +If you are a member of the GitLab Slack workspace, you can request help in the `#docs` channel. diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 37be2178592..986252eedac 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -7,33 +7,32 @@ description: "GitLab development - how to document features deployed behind feat # Document features deployed behind feature flags -GitLab uses [feature flags](../feature_flags/index.md) to strategically roll -out the deployment of its own features. The way we document a feature behind a -feature flag depends on its state (enabled or disabled). When the state -changes, the developer who made the change **must update the documentation** -accordingly. +GitLab uses [feature flags](../feature_flags/index.md) to roll +out the deployment of its own features. + +When the state of a feature flag changes, the developer who made the change +**must update the documentation**. ## When to document features behind a feature flag Every feature introduced to the codebase, even if it's behind a disabled feature flag, must be documented. For more information, see -[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). +[the discussion that led to this decision](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47917#note_459984428). [Alpha, Beta, or Limited Availability](../../policy/alpha-beta-support.md) features are usually behind a feature flag, and must also be documented. For more information, see [Document Alpha, Beta, LA features](alpha_beta.md). + +When the feature is [implemented in multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development), +discuss the plan with your technical writer. -When the feature is [implemented over multiple merge requests](../feature_flags/index.md#feature-flags-in-gitlab-development), -discuss the exact documentation plan with your technical writer. Consider -creating a dedicated documentation issue if the feature: +You can create a documentation issue and delay the documentation if the feature: - Is far-reaching (makes changes across many areas of GitLab), like navigation changes. - Includes many MRs. - Affects more than a few documentation pages. - Is not fully functional if the feature flag is enabled for testing. -If you and the technical writer agree to delay the product documentation (for example, until after testing), -collaborate with the TW to create a documentation issue detailing the plan for adding the content. -The PM and EM should be included in the discussions to make sure the task of adding the documentation is assigned and scheduled. -Despite any planned delays, every feature flag in the codebase is automatically listed at -<https://docs.gitlab.com/ee/user/feature_flags.html#available-feature-flags>, -even when the feature is not fully functional. +The PM, EM, and writer should make sure the documentation work is assigned and scheduled. + +Every feature flag in the codebase is [in the documentation](../../user/feature_flags.md), +even when the feature is not fully functional or otherwise documented. ## How to add feature flag documentation @@ -57,13 +56,6 @@ Possible version history entries are: > - [Generally available](issue-link) in GitLab X.Y. Feature flag `flag_name` removed. ``` -You can combine entries if they happened in the same release: - -```markdown -> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. -``` - ## Use a note to describe the state of the feature flag Information about feature flags should be in a `FLAG` note at the start of the topic (just below the version history). @@ -144,3 +136,41 @@ And, when the feature is done and fully available to all users: > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. > - [Generally available](issue-link) in GitLab 14.0. Feature flag `forti_token_cloud` removed. ``` + +## Simplify long version history + +The version history can get long, but you can sometimes simplify or remove entries. + +Combine entries if they happened in the same release: + +- Before: + + ```markdown + > - [Introduced](issue-link) in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. + > - [Enabled on GitLab.com](issue-link) in GitLab 14.3. + > - [Enabled on self-managed](issue-link) in GitLab 14.3. + ``` + +- After: + + ```markdown + > - [Introduced](issue-link) in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. + > - [Enabled on GitLab.com and self-managed](issue-link) in GitLab 14.3. + ``` + +Remove `Enabled on GitLab.com` entries when the feature is enabled by default for both GitLab.com and self-managed: + +- Before: + + ```markdown + > - [Introduced](issue-link) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. + > - [Enabled on GitLab.com](issue-link) in GitLab 15.9. + > - [Generally available](issue-link) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. + ``` + +- After: + + ```markdown + > - [Introduced](issue-link) in GitLab 15.6 [with a flag](../../administration/feature_flags.md) named `ci_hooks_pre_get_sources_script`. Disabled by default. + > - [Generally available](issue-link) in GitLab 15.10. Feature flag `ci_hooks_pre_get_sources_script` removed. + ``` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index fd9e885e097..d1f5ee8f9f3 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -107,8 +107,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w --- ``` -If you need help determining the correct stage, read [Ask for help](workflow.md#ask-for-help). - ### Redirection metadata The following metadata should be added when a page is moved to another location: @@ -157,19 +155,15 @@ You can use a Rake task to update the `CODEOWNERS` file. #### Update the `CODEOWNERS` file -To update the `CODEOWNERS` file: +When groups or [TW assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) +change, you must update the `CODEOWNERS` file: -1. Review the [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) - in the [`codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) - file. If any assignments have changed: - 1. Update the `codeowners.rake` file with the changes. - 1. Assign the merge request to a technical writing manager for review and merge. -1. After the changes to `codeowners.rake` are merged, go to the root of the `gitlab` repository. +1. Update the [stage and group metadata](#stage-and-group-metadata) for any affected doc pages, if necessary. If there are many changes, you can do this step in a separate MR. +1. Update the [`codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) file with the changes. +1. Go to the root of the `gitlab` repository. 1. Run the Rake task with this command: `bundle exec rake tw:codeowners` -1. Review the command output for any pages that need attention to - their metadata. Handle any needed changes in a separate merge request. -1. Add the changes to the CODEOWNERS file to Git: `git add .gitlab/CODEOWNERS` -1. Commit your changes to your branch, and push your branch up to `origin`. +1. Review the changes in the `CODEOWNERS` file. +1. Add and commit all your changes and push your branch up to `origin`. 1. Create a merge request and assign it to a technical writing manager for review. ## Move, rename, or delete a page diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index a92d58ead96..93afbf7e6dd 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -132,7 +132,7 @@ To deprecate an attribute: ``` To widely announce a deprecation, or if it's a breaking change, -[update the deprecations and removals documentation pages](../deprecation_guidelines/index.md#update-the-deprecations-and-removals-documentation-pages). +[update the REST API deprecations and removals page](../../api/rest/deprecations.md). ## Method description diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 74437ea46c9..7f69a1f0d54 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -779,14 +779,18 @@ As much as possible, use text that follows one of these patterns: For example: -- `For more information, see [merge requests](../../../user/project/merge_requests/index.md).` -- `To create a review app, see [review apps](../../../ci/review_apps/index.md).` +- `For more information, see [merge requests](LINK).` +- `To create a review app, see [review apps](LINK).` You can expand on this text by using phrases like `For more information about this feature, see...` -Do not to use alternate phrases, like `Learn more about...` or -`To read more...`. +Do not to use the following constructions: + +- `Learn more about...` +- `To read more...`. +- `For more information, see the [Merge requests](LINK) page.` +- `For more information, see the [Merge requests](LINK) documentation.` #### Descriptive text rather than `here` diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index e64fd4df7ff..1ea95367d01 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -82,6 +82,11 @@ If you can add the phrase "by zombies" to the phrase, the construction is passive. For example, `The button is selected by zombies` is passive. `Zombies select the button` is active. +## Admin Area + +Use title case for **Admin Area** to refer to the area of the UI that you access when you select **Main menu > Admin**. +This area of the UI says **Admin Area** at the top of the page and on the menu. + ## administrator Use **administrator access** instead of **admin** when talking about a user's access level. @@ -99,10 +104,9 @@ Instead of: - To do this thing, you must have the Admin role. -## Admin Area +## advanced search -Use title case **Admin Area** to refer to the area of the UI that you access when you select **Main menu > Admin**. -This area of the UI says **Admin Area** at the top of the page and on the menu. +Use lowercase for **advanced search** to refer to the faster, more efficient search across the entire GitLab instance. ## agent @@ -113,7 +117,7 @@ For example: - Install the agent in your cluster. - Select an agent from the list. -Do not use title case **GitLab Agent** or **GitLab Agent for Kubernetes**. +Do not use title case for **GitLab Agent** or **GitLab Agent for Kubernetes**. ## agent access token @@ -530,9 +534,12 @@ Use title case for **Geo**. Do not make **GitLab** possessive (GitLab's). This guidance follows [GitLab Trademark Guidelines](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/brand/brand-activation/trademark-guidelines/). -## GitLab.com +## GitLab Flavored Markdown -**GitLab.com** refers to the GitLab instance managed by GitLab itself. +When possible, spell out [**GitLab Flavored Markdown**](../../../user/markdown.md). +([Vale](../testing.md#vale) rule: [`GLFM.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) + +If you must abbreviate, do not use **GFM**. Use **GLFM** instead. ## GitLab Helm chart, GitLab chart @@ -545,26 +552,30 @@ Do not use **the `gitlab` chart**, **the GitLab Chart**, or **the cloud-native c You use the **GitLab Helm chart** to deploy **cloud-native GitLab** in a Kubernetes cluster. -## GitLab Flavored Markdown +## GitLab Pages -When possible, spell out [**GitLab Flavored Markdown**](../../../user/markdown.md). -([Vale](../testing.md#vale) rule: [`GLFM.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) +For consistency and branding, use **GitLab Pages** rather than **Pages**. -If you must abbreviate, do not use **GFM**. Use **GLFM** instead. +However, if you use **GitLab Pages** for the first mention on a page or in the UI, +you can use **Pages** thereafter. + +## GitLab Runner + +Use title case for **GitLab Runner**. This is the product you install. See also [runners](#runner-runners) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). ## GitLab SaaS **GitLab SaaS** refers to the product license that provides access to GitLab.com. It does not refer to the GitLab instance managed by GitLab itself. -## GitLab Runner - -Use title case for **GitLab Runner**. This is the product you install. See also [runners](#runner-runners) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). - ## GitLab self-managed Use **GitLab self-managed** to refer to the product license for GitLab instances managed by customers themselves. +## GitLab.com + +**GitLab.com** refers to the GitLab instance managed by GitLab itself. + ## guide We want to speak directly to users. On `docs.gitlab.com`, do not use **guide** as part of a page title. @@ -730,7 +741,7 @@ Do not use **limitations**. Use **known issues** instead. ## log in, log on -Do not use **log in** or **log on**. Use [sign in](#sign-in) instead. If the user interface has **Log in**, you can use it. +Do not use **log in** or **log on**. Use [sign in](#sign-in-sign-in) instead. If the user interface has **Log in**, you can use it. ## logged-in user, logged in user @@ -933,6 +944,17 @@ An Owner is the highest role a user can have. Use title case for the GitLab Package Registry. +## page + +If you write a phrase like, "On the **Issues** page," ensure steps for how to get to the page are nearby. Otherwise, people might not know what the **Issues** page is. + +The page name should be visible in the UI at the top of the page. +If it is not, you should be able to get the name from the breadcrumb. + +The docs should match the case in the UI, and the page name should be bold. For example: + +- On the **Test cases** page, ... + ## permissions Do not use [**roles**](#roles) and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. @@ -947,6 +969,12 @@ Use lowercase for **personal access token**. Do not use **please**. For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +## prerequisites + +Use **prerequisites** when documenting the steps before a task. Do not use **requirements**. + +For more information, see [the task topic type](../topic_types/task.md). + ## press Use **press** when talking about keyboard keys. For example: @@ -1009,6 +1037,12 @@ Do not use **Reporter permissions**. A user who is assigned the Reporter role ha Use title case for **Repository Mirroring**. +## requirements + +Use **prerequisites** when documenting the steps before a task. Do not use **requirements**. + +For more information, see [the task topic type](../topic_types/task.md). + ## respectively Avoid **respectively** and be more precise instead. @@ -1120,14 +1154,17 @@ Use **setup** as a noun, and **set up** as a verb. For example: - Your remote office setup is amazing. - To set up your remote office correctly, consider the ergonomics of your work area. -## sign in +## sign in, sign-in -Use **sign in** or **sign in to**. +Use **sign in** or **sign in to** as a verb to describe the action of signing in. Do not use **sign on** or **sign into**, or **log on**, **log in**, or **log into**. If the user interface has different words, use those. +You can use **sign-in** as a noun or adjective. For example, **sign-in page** or +**sign-in restrictions**. + You can use **single sign-on**. ## sign up @@ -1314,7 +1351,10 @@ See also [downgrade](#downgrade) and [roll back](#roll-back). ## upper left, upper right -Use **upper left** and **upper right** instead of **top left** and **top right**. Hyphenate as adjectives (for example, **upper-left corner**). +Use **upper-left corner** and **upper-right corner** to provide direction in the UI. +If the UI element is not in a corner, use **upper left** and **upper right**. + +Do not use **top left** and **top right**. For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/u/upper-left-upper-right). diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index 8d23a5f322e..f596ffc9b8b 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -60,6 +60,22 @@ For example, `Create an issue`. If several tasks on a page share prerequisites, you can create a separate topic with the title `Prerequisites`. +## When a task has only one step + +If you need to write a task that has only one step, make that step an unordered list item. +This format helps the step stand out, while keeping it consistent with the rules +for lists. + +For example: + +```markdown +# Create a merge request + +To create a merge request: + +- In the upper-right corner, select **New merge request**. +``` + ### When more than one way exists to perform a task If more than one way exists to perform a task in the UI, you should diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md index 30a0d4d4cf4..859e7265a2a 100644 --- a/doc/development/documentation/versions.md +++ b/doc/development/documentation/versions.md @@ -13,7 +13,7 @@ including when features were introduced and when they were updated or removed. ## View older documentation versions Previous versions of the documentation are available on `docs.gitlab.com`. -To view a previous version, select the **Versions** button in the upper right. +To view a previous version, in the upper-right corner, select **Versions**. To view versions that are not available on `docs.gitlab.com`: @@ -89,8 +89,9 @@ voters to agree. When features are deprecated and removed, update the related documentation. -API documentation follows these guidelines, but the GraphQL docs use -a [separate process](../api_graphql_styleguide.md#deprecating-schema-items). +NOTE: +A separate process exists for [GraphQL docs](../api_graphql_styleguide.md#deprecating-schema-items) +and [REST API docs](restful_api_styleguide.md#deprecations). ### Deprecate a page or topic diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index fe5453a4a58..1a4194aebd9 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -4,148 +4,39 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# How to update GitLab documentation +# Documentation workflow -Anyone can contribute to the GitLab documentation! You can create a merge request for documentation when: +Documentation at GitLab follows a workflow. -- You find errors or other room for improvement in existing documentation. -- You have an idea for all-new documentation that would help a GitLab user or administrator to - accomplish their work with GitLab. +## Before merging -If you are working on a feature or enhancement, use the -[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#documentation-for-a-product-change). +Ensure your documentation includes: -## Do not use ChatGPT or AI-generated content for the docs - -GitLab documentation is distributed under the [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/), which presupposes that GitLab owns the documentation. - -Under current law in the US and the EU, it’s possible that AI-generated works might either: - -- not be owned by anyone because they weren't created by a human, or -- belong to the AI training data’s creator, if the AI verbatim reproduces content that it trained on - -If the documentation contains AI-generated content, GitLab probably wouldn't own this content, which would risk invalidating the CC BY-SA 4.0 license. +- [Product badges](styleguide/index.md#product-tier-badges). +- The GitLab [version](versions.md) that introduced the feature. +- Accurate [links](styleguide/index.md#links). +- Accurate [user permissions](../../user/permissions.md). -Contributions to GitLab documentation are made under either our [DCO or our CLA terms](https://about.gitlab.com/community/contribute/dco-cla/). In both, contributors have to make certain certifications about the authorship of their work that they can't validly make for AI-generated text. - -For these reasons, do not add AI-generated content to the documentation. - -## How to update the docs - -If you are not a GitLab team member, or do not have the Developer role for the GitLab repository, to update GitLab documentation: - -1. Select an [issue](https://about.gitlab.com/handbook/product/ux/technical-writing/#community-contribution-opportunities) you'd like to work on. - - For a Hackathon, mention `@docs-hackathon` in a comment and ask for the issue to be assigned to you. - To be fair to other contributors, if you see someone has already asked to work on the issue, choose another issue. - - If you're not taking part in a Hackathon, you don't need an issue to open a merge request. - If you are looking for issues to work on and don't see any that suit you, you can always fix [Vale](testing.md#vale) issues. -1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). -1. In the upper right, select **Fork**. Forking makes a copy of the repository on GitLab.com. -1. In your fork, find the documentation page in the `\doc` directory. -1. If you know Git, make your changes and open a merge request. - If not, follow these steps: - 1. In the upper right, select **Edit** if it is visible. If it is not, select the down arrow (**{chevron-lg-down}**) next to **Open in Web IDE** or **Gitpod**, and select **Edit**. - 1. In the **Commit message** text box, enter a commit message. Use 3-5 words, start with a capital letter, and do not end with a period. - 1. Select **Commit changes**. - 1. On the left sidebar, select **Merge requests**. - 1. Select **New merge request**. - 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. - For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. - 1. Select **Compare branches and continue**. A new merge request opens. - 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. - 1. Select **Create merge request**. - -If you need help while working on the page, view: - -- The [Style Guide](styleguide/index.md). -- The [Word list](styleguide/word_list.md) -- The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). - -### Ask for help - -Ask for help from the Technical Writing team if you: - -- Need help to choose the correct place for documentation. -- Want to discuss a documentation idea or outline. -- Want to request any other help. - -To identify someone who can help you: - -1. Locate the Technical Writer for the relevant - [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). -1. Either: - - If urgent help is required, directly assign the Technical Writer in the issue or in the merge request. - - If non-urgent help is required, ping the Technical Writer in the issue or merge request. - -If you are a member of the GitLab Slack workspace, you can request help in `#docs`. +Ensure you've followed the [style guide](styleguide/index.md) and [word list](styleguide/word_list.md). ## Documentation labels -When you author an issue or merge request, you must add these labels: +When you author an issue or merge request, choose the +[Documentation template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md). +It includes these labels, which are added to the merge request: -- A [type label](../contributing/issue_workflow.md#type-labels), either `~"type::feature"` or `~"type::maintenance"`. -- A [stage label](../contributing/issue_workflow.md#stage-labels) and [group label](../contributing/issue_workflow.md#group-labels). +- A [type label](../labels/index.md#type-labels), either `~"type::feature"` or `~"type::maintenance"`. +- A [stage label](../labels/index.md#stage-labels) and [group label](../labels/index.md#group-labels). For example, `~devops::create` and `~group::source code`. -- A `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). +- A `~documentation` [specialization label](../labels/index.md#specialization-labels). A member of the Technical Writing team adds these labels: - A [documentation scoped label](../../user/project/labels.md#scoped-labels) with the `docs::` prefix. For example, `~docs::improvement`. -- The [`~Technical Writing` team label](../contributing/issue_workflow.md#team-labels). - -## Reviewing and merging - -Anyone with the Maintainer role to the relevant GitLab project can -merge documentation changes. Maintainers must make a good-faith effort to ensure that the content: - -- Is clear and sufficiently easy for the intended audience to navigate and understand. -- Meets the [Documentation Guidelines](index.md) and [Style Guide](styleguide/index.md). +- The [`~Technical Writing` team label](../labels/index.md#team-labels). -If the author or reviewer has any questions, they can mention the writer who is assigned to the relevant -[DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). - -The process involves the following: - -- Primary Reviewer. Review by a [code reviewer](https://about.gitlab.com/handbook/engineering/projects/) - or other appropriate colleague to confirm accuracy, clarity, and completeness. This can be skipped - for minor fixes without substantive content changes. -- Technical Writer (Optional). If not completed for a merge request before merging, must be scheduled - post-merge. Schedule post-merge reviews only if an urgent merge is required. To request a: - - Pre-merge review, assign the Technical Writer listed for the applicable - [DevOps stage group](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). - - Post-merge review, see [Post-merge reviews](#post-merge-reviews). -- Maintainer. For merge requests, Maintainers: - - Can always request any of the above reviews. - - Review before or after a Technical Writer review. - - Ensure the given release milestone is set. - - Ensure the appropriate labels are applied, including any required to pick a merge request into - a release. - - Ensure that, if there has not been a Technical Writer review completed or scheduled, they - [create the required issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review), assign it to the Technical Writer of the given stage group, - and link it from the merge request. - -The process is reflected in the **Documentation** -[merge request template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md). - -### Before merging - -Ensure the following if skipping an initial Technical Writer review: - -- [Product badges](styleguide/index.md#product-tier-badges) are applied. -- The GitLab [version](versions.md) that - introduced the feature is included. -- Changes to topic titles don't affect in-app hyperlinks. -- Specific [user permissions](../../user/permissions.md) are documented. -- New documents are linked from higher-level indexes, for discoverability. -- The style guide is followed: - - For [directories and files](site_architecture/folder_structure.md). - - For [images](styleguide/index.md#images). - -Merge requests that change the location of documentation must always be reviewed by a Technical -Writer before merging. - -### Post-merge reviews +## Post-merge reviews If not assigned to a Technical Writer for review prior to merging, a review must be scheduled immediately after merge by the developer or maintainer. For this, @@ -174,8 +65,25 @@ Remember: - The Technical Writer can also help decide that documentation can be merged without Technical writer review, with the review to occur soon after merge. -## Other ways to help +## Do not use ChatGPT or AI-generated content for the docs + +GitLab documentation is distributed under the [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/), which presupposes that GitLab owns the documentation. -If you have ideas for further documentation resources please -[create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Documentation) -using the Documentation template. +Under current law in the US and the EU, it’s possible that AI-generated works might either: + +- not be owned by anyone because they weren't created by a human, or +- belong to the AI training data's creator, if the AI verbatim reproduces content that it trained on + +If the documentation contains AI-generated content, GitLab probably wouldn't own this content, which would risk invalidating the CC BY-SA 4.0 license. + +Contributions to GitLab documentation are made under either our [DCO or our CLA terms](https://about.gitlab.com/community/contribute/dco-cla/). In both, contributors have to make certain certifications about the authorship of their work that they can't validly make for AI-generated text. + +For these reasons, do not add AI-generated content to the documentation. + +## Related topics + +- [Reviews and levels of edit](https://about.gitlab.com/handbook/product/ux/technical-writing/#reviews) +- [Technical writing assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) +- The [Style Guide](styleguide/index.md) +- The [Word list](styleguide/word_list.md) +- The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/) diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 4eb5bedef1c..bc997c37e66 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -18,6 +18,49 @@ info: To determine the technical writer assigned to the Stage/Group associated w [EE features list](https://about.gitlab.com/features/). <!-- markdownlint-enable MD044 --> +## SaaS only feature + +Use the following guidelines when you develop a feature that is only applicable for SaaS (for example, a CustomersDot integration). + +1. It is recommended you use an application setting. This enables + granular settings so that each SaaS instance can switch things according to + their need. +1. If application setting is not possible, helpers such as `Gitlab.com?` can be + used. However, this comes with drawbacks as listed in the epic to + [remove these helpers](https://gitlab.com/groups/gitlab-org/-/epics/7374). + 1. Consider performance and availability impact on other SaaS instances. For example, + [GitLab JH overrides](https://jihulab.com/gitlab-cn/gitlab/-/blob/main-jh/jh/lib/jh/gitlab/saas.rb) + SaaS helpers, so that it returns true for `Gitlab.com?`. + +### Simulate a SaaS instance + +If you're developing locally and need your instance to simulate the SaaS (GitLab.com) +version of the product: + +1. Export this environment variable: + + ```shell + export GITLAB_SIMULATE_SAAS=1 + ``` + + There are many ways to pass an environment variable to your local GitLab instance. + For example, you can create an `env.runit` file in the root of your GDK with the above snippet. + +1. Enable **Allow use of licensed EE features** to make licensed EE features available to projects + only if the project namespace's plan includes the feature. + + 1. Visit **Admin > Settings > General**. + 1. Expand **Account and limit**. + 1. Select the **Allow use of licensed EE features** checkbox. + 1. Select **Save changes**. + +1. Ensure the group you want to test the EE feature for is actually using an EE plan: + 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, select **Overview > Groups**. + 1. Identify the group you want to modify, and select **Edit**. + 1. Scroll to **Permissions and group features**. For **Plan**, select `Ultimate`. + 1. Select **Save changes**. + ## Implement a new EE feature If you're developing a GitLab Premium or GitLab Ultimate licensed feature, use these steps to @@ -74,9 +117,9 @@ To guard your licensed feature: 1. Optional. If your global feature is also available to namespaces with a paid plan, combine two feature identifiers to allow both administrators and group users. For example: - ```ruby - License.feature_available?(:my_feature_name) || group.licensed_feature_available?(:my_feature_name_for_namespace) # Both admins and group members can see this EE feature - ``` + ```ruby + License.feature_available?(:my_feature_name) || group.licensed_feature_available?(:my_feature_name_for_namespace) # Both admins and group members can see this EE feature + ``` ### Simulate a CE instance when unlicensed @@ -100,15 +143,15 @@ To simulate a CE instance without deleting the license in a GDK: 1. Create an `env.runit` file in the root of your GDK with the line: - ```shell - export FOSS_ONLY=1 - ``` + ```shell + export FOSS_ONLY=1 + ``` 1. Then restart the GDK: - ```shell - gdk restart rails && gdk restart webpack - ``` + ```shell + gdk restart rails && gdk restart webpack + ``` Remove the line in `env.runit` if you want to revert back to an EE installation, and repeat step 2. @@ -137,35 +180,6 @@ To do so: bin/rspec spec/features/<path_to_your_spec> ``` -### Simulate a SaaS instance - -If you're developing locally and need your instance to simulate the SaaS (GitLab.com) -version of the product: - -1. Export this environment variable: - - ```shell - export GITLAB_SIMULATE_SAAS=1 - ``` - - There are many ways to pass an environment variable to your local GitLab instance. - For example, you can create an `env.runit` file in the root of your GDK with the above snippet. - -1. Enable **Allow use of licensed EE features** to make licensed EE features available to projects - only if the project namespace's plan includes the feature. - - 1. Visit **Admin > Settings > General**. - 1. Expand **Account and limit**. - 1. Select the **Allow use of licensed EE features** checkbox. - 1. Click **Save changes**. - -1. Ensure that the group for which you want to test the EE feature, is actually using an EE plan: - 1. On the top bar, select **Main menu > Admin**. - 1. On the left sidebar, select **Overview > Groups**. - 1. Identify the group you want to modify, and select **Edit**. - 1. Scroll to **Permissions and group features**. For **Plan**, select `Ultimate`. - 1. Select **Save changes**. - ### Run CI pipelines in a FOSS context By default, merge request pipelines for development run in an EE-context only. If you are @@ -580,11 +594,11 @@ Resolving an EE template path that is relative to the CE view path doesn't work. For `render` and `render_if_exists`, they search for the EE partial first, and then CE partial. They would only render a particular partial, not all partials with the same name. We could take the advantage of this, so that -the same partial path (for example, `shared/issuable/form/default_templates`) could +the same partial path (for example, `projects/settings/archive`) could be referring to the CE partial in CE (that is, -`app/views/shared/issuable/form/_default_templates.html.haml`), while EE +`app/views/projects/settings/_archive.html.haml`), while EE partial in EE (that is, -`ee/app/views/shared/issuable/form/_default_templates.html.haml`). This way, +`ee/app/views/projects/settings/_archive.html.haml`). This way, we could show different things between CE and EE. However sometimes we would also want to reuse the CE partial in EE partial @@ -594,21 +608,19 @@ would be tedious to do so. In this case, we could as well just use `render_ce` which would ignore any EE partials. One example would be -`ee/app/views/shared/issuable/form/_default_templates.html.haml`: +`ee/app/views/projects/settings/_archive.html.haml`: ```haml -- if @project.feature_available?(:issuable_default_templates) - = render_ce 'shared/issuable/form/default_templates' -- elsif show_promotions? - = render 'shared/promotions/promote_issue_templates' +- return if @project.marked_for_deletion? += render_ce 'projects/settings/archive' ``` In the above example, we can't use -`render 'shared/issuable/form/default_templates'` because it would find the +`render 'projects/settings/archive'` because it would find the same EE partial, causing infinite recursion. Instead, we could use `render_ce` so it ignores any partials in `ee/` and then it would render the CE partial -(that is, `app/views/shared/issuable/form/_default_templates.html.haml`) -for the same path (that is, `shared/issuable/form/default_templates`). This way +(that is, `app/views/projects/settings/_archive.html.haml`) +for the same path (that is, `projects/settings/archive`). This way we could easily wrap around the CE partial. ### Code in `lib/gitlab/background_migration/` diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index af45603782f..b00131b12f3 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -345,7 +345,7 @@ Keep in mind that: - When you add `:hover` styles, in most cases you should add `:focus` styles too so that the styling is applied for both mouse **and** keyboard users. - If you remove an interactive element's `outline`, make sure you maintain visual focus state in another way such as with `box-shadow`. -See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/keyboard-only/) for more detail. +See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility/keyboard-only) for more detail. ## `tabindex` @@ -516,6 +516,55 @@ GitLab-specific examples are assignee and label dropdowns. Building such widgets require ARIA to make them understandable to screen readers. Proper research and testing should be done to ensure compliance with [WCAG](https://www.w3.org/WAI/standards-guidelines/wcag/). +## Automated accessibility testing with axe + +You can use [axe-core](https://github.com/dequelabs/axe-core) [gems](https://github.com/dequelabs/axe-core-gems) +to run automated accessibility tests in feature tests. + +Axe provides the custom matcher `be_axe_clean`, which can be used like the following: + +```ruby +# spec/features/settings_spec.rb +it 'passes axe automated accessibility testing', :js do + visit_settings_page + + wait_for_requests # ensures page is fully loaded + + expect(page).to be_axe_clean +end +``` + +If needed, you can scope testing to a specific area of the page by using `within`. + +Axe also provides specific [clauses](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#clauses), +for example: + +```ruby +expect(page).to be_axe_clean.within '[data-testid="element"]' + +# run only WCAG 2.0 Level AA rules +expect(page).to be_axe_clean.according_to :wcag2aa + +# specifies which rule to skip +expect(page).to be_axe_clean.skipping :'link-in-text-block' + +# clauses can be chained +expect(page).to be_axe_clean.within('[data-testid="element"]') + .according_to(:wcag2aa) +``` + +Axe does not test hidden regions, such as inactive menus or modal windows. To test +hidden regions for accessibility, write tests that activate or render the regions visible +and run the matcher again. + +### Known accessibility violations + +This section documents violations where a recommendation differs with the [design system](https://design.gitlab.com/): + +- `link-in-text-block`: For now, use the `skipping` clause to skip `:'link-in-text-block'` + rule to fix the violation. After this is fixed as part of [issue 1444](https://gitlab.com/gitlab-org/gitlab-services/design.gitlab.com/-/issues/1444) + and underline is added to the `GlLink` component, this clause can be removed. + ## Resources ### Viewing the browser accessibility tree diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 6d13f419430..5c7fe68fec5 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -64,7 +64,7 @@ Instead, you should obtain an instance of the `ContentEditor` class by listening ```html <script> -import { createAlert } from '~/flash'; +import { createAlert } from '~/alert'; import { __ } from '~/locale'; export default { diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 3b349d880c0..995730796b4 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -69,7 +69,7 @@ banner on top of the component examples indicates that: > component. For example, at the time of writing, this type of warning can be observed for -[all form components](https://design.gitlab.com/components/form/). It, however, +all form components, such as the [checkbox](https://design.gitlab.com/components/checkbox). It, however, doesn't imply that the component should not be used. GitLab always asks to use `<gl-*>` components whenever a suitable component exists. @@ -192,7 +192,7 @@ To see what polyfills are being used: 1. Select the [`compile-production-assets`](https://gitlab.com/gitlab-org/gitlab/-/jobs/641770154) job. 1. In the right-hand sidebar, scroll to **Job Artifacts**, and select **Browse**. 1. Select the **webpack-report** folder to open it, and select **index.html**. -1. In the upper left corner of the page, select the right arrow **{chevron-lg-right}** +1. In the upper-left corner of the page, select the right arrow (**{chevron-lg-right}**) to display the explorer. 1. In the **Search modules** field, enter `gitlab/node_modules/core-js` to see which polyfills are being loaded and where: diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 8ae0458e47c..5f0855d8f91 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -63,17 +63,17 @@ the GraphQL extension, follow these steps: 1. Add an `apollo.config.js` file to the root of your `gitlab` local directory. 1. Populate the file with the following content: - ```javascript - module.exports = { - client: { - includes: ['./app/assets/javascripts/**/*.graphql', './ee/app/assets/javascripts/**/*.graphql'], - service: { - name: 'GitLab', - localSchemaFile: './tmp/tests/graphql/gitlab_schema.graphql', - }, - }, - }; - ``` + ```javascript + module.exports = { + client: { + includes: ['./app/assets/javascripts/**/*.graphql', './ee/app/assets/javascripts/**/*.graphql'], + service: { + name: 'GitLab', + localSchemaFile: './tmp/tests/graphql/gitlab_schema.graphql', + }, + }, + }; + ``` 1. Restart VS Code. @@ -84,10 +84,8 @@ Our GraphQL API can be explored via GraphiQL at your instance's [GitLab GraphQL API Reference documentation](../../api/graphql/reference) where needed. -You can check all existing queries and mutations on the right side -of GraphiQL in its **Documentation explorer**. You can also -write queries and mutations directly on the left tab and check -their execution by selecting **Execute query** in the upper left: +To check all existing queries and mutations, on the right side of GraphiQL, select **Documentation explorer**. +To check the execution of the queries and mutations you've written, in the upper-left corner, select **Execute query**.  @@ -107,7 +105,9 @@ Default client accepts two parameters: `resolvers` and `config`. ### Multiple client queries for the same object -If you are making multiple queries to the same Apollo client object you might encounter the following error: `Cache data may be lost when replacing the someProperty field of a Query object. To address this problem, either ensure all objects of SomeEntityhave an id or a custom merge function`. We are already checking `ID` presence for every GraphQL type that has an `ID`, so this shouldn't be the case. Most likely, the `SomeEntity` type doesn't have an `ID` property, and to fix this warning we need to define a custom merge function. +If you are making multiple queries to the same Apollo client object you might encounter the following error: `Cache data may be lost when replacing the someProperty field of a Query object. To address this problem, either ensure all objects of SomeEntityhave an id or a custom merge function`. We are already checking `id` presence for every GraphQL type that has an `id`, so this shouldn't be the case (unless you see this warning when running unit tests; in this case please ensure your mocked responses contain an `id` whenever it's requested). + +When `SomeEntity` type doesn't have an `id` property in the GraphQL schema, to fix this warning we need to define a custom merge function. We have some client-wide types with `merge: true` defined in the default client as [`typePolicies`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/lib/graphql.js) (this means that Apollo will merge existing and incoming responses in the case of subsequent queries). Consider adding `SomeEntity` there or defining a custom merge function for it. @@ -264,9 +264,7 @@ Read more about [Vue Apollo](https://github.com/vuejs/vue-apollo) in the [Vue Ap ### Local state with Apollo -It is possible to manage an application state with Apollo by using [client-site resolvers](#using-client-side-resolvers) -or [type policies with reactive variables](#using-type-policies-with-reactive-variables) when creating your default -client. +It is possible to manage an application state with Apollo when creating your default client. #### Using client-side resolvers @@ -326,6 +324,32 @@ export default { } ``` +Instead of using `writeQuery`, we can create a type policy that will return `user` on every attempt of reading the `userQuery` from the cache: + +```javascript +const defaultClient = createDefaultClient({}, { + cacheConfig: { + typePolicies: { + Query: { + fields: { + user: { + read(data) { + return data || { + user: { + name: 'John', + surname: 'Doe', + age: 30 + }, + } + } + } + } + } + } + } +}); +``` + Along with creating local data, we can also extend existing GraphQL types with `@client` fields. This is extremely helpful when we need to mock an API response for fields not yet added to our GraphQL API. ##### Mocking API response with local Apollo cache @@ -390,122 +414,9 @@ For each attempt to fetch a version, our client fetches `id` and `sha` from the Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.app/guide/local-state.html#local-state). -#### Using type policies with reactive variables - -Apollo Client 3 offers an alternative to [client-side resolvers](#using-client-side-resolvers) by using -[reactive variables to store client state](https://www.apollographql.com/docs/react/local-state/reactive-variables/). - -**NOTE:** -We are still learning the best practices for both **type policies** and **reactive vars**. -Take a moment to improve this guide or [leave a comment](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/100) -if you use it! - -In the example below we define a `@client` query and its `typedefs`: - -```javascript -// ./graphql/typedefs.graphql -extend type Query { - localData: String! -} -``` - -```javascript -// ./graphql/get_local_data.query.graphql -query getLocalData { - localData @client -} -``` - -Similar to resolvers, your `typePolicies` execute when the `@client` query is used. However, -using `makeVar` triggers every relevant active Apollo query to reactively update when the state -mutates. - -```javascript -// ./graphql/local_state.js - -import { makeVar } from '@apollo/client/core'; -import typeDefs from './typedefs.graphql'; - -export const createLocalState = () => { - // set an initial value - const localDataVar = makeVar(''); - - const cacheConfig = { - typePolicies: { - Query: { - fields: { - localData() { - // obtain current value - // triggers when `localDataVar` is updated - return localDataVar(); - }, - }, - }, - }, - }; - - // methods that update local state - const localMutations = { - setLocalData(newData) { - localDataVar(newData); - }, - clearData() { - localDataVar(''); - }, - }; - - return { - cacheConfig, - typeDefs, - localMutations, - }; -}; -``` - -Pass the cache configuration to your Apollo Client: - -```javascript -// index.js - -// ... -import createDefaultClient from '~/lib/graphql'; -import { createLocalState } from './graphql/local_state'; - -const { cacheConfig, typeDefs, localMutations } = createLocalState(); - -const apolloProvider = new VueApollo({ - defaultClient: createDefaultClient({}, { cacheConfig, typeDefs }), -}); - -return new Vue({ - el, - apolloProvider, - provide: { - // inject local state mutations to your app - localMutations, - }, - render(h) { - return h(MyApp); - }, -}); -``` - -Wherever used, the local query updates as the state updates thanks to the **reactive variable**. - ### Using with Vuex -When the Apollo Client is used in Vuex and fetched data is stored in the Vuex store, the Apollo Client cache does not need to be enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. With Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache by passing a valid `fetchPolicy` option to its constructor: - -```javascript -import fetchPolicies from '~/graphql_shared/fetch_policy_constants'; - -export const gqClient = createGqClient( - {}, - { - fetchPolicy: fetchPolicies.NO_CACHE, - }, -); -``` +We do not recommend creating new applications with Vuex and Apollo Client combined ### Working on GraphQL-based features when frontend and backend are not in sync @@ -1314,73 +1225,6 @@ If you use the RubyMine IDE, and have marked the `tmp` directory as `gitlab/tmp/tests/graphql`. This will allow the **JS GraphQL** plugin to automatically find and index the schema. -#### Testing Apollo components - -If we use `ApolloQuery` or `ApolloMutation` in our components, to test their functionality we need to add a stub first: - -```javascript -import { ApolloMutation } from 'vue-apollo'; - -function createComponent(props = {}) { - wrapper = shallowMount(MyComponent, { - sync: false, - propsData: { - ...props, - }, - stubs: { - ApolloMutation, - }, - }); -} -``` - -`ApolloMutation` component exposes `mutate` method via scoped slot. If we want to test this method, we need to add it to mocks: - -```javascript -const mutate = jest.fn().mockResolvedValue(); -const $apollo = { - mutate, -}; - -function createComponent(props = {}) { - wrapper = shallowMount(MyComponent, { - sync: false, - propsData: { - ...props, - }, - stubs: { - ApolloMutation, - }, - mocks: { - $apollo, - } - }); -} -``` - -Then we can check if `mutate` is called with correct variables: - -```javascript -const mutationVariables = { - mutation: createNoteMutation, - update: expect.anything(), - variables: { - input: { - noteableId: 'noteable-id', - body: 'test', - discussionId: '0', - }, - }, -}; - -it('calls mutation on submitting form ', () => { - createComponent() - findReplyForm().vm.$emit('submitForm'); - - expect(mutate).toHaveBeenCalledWith(mutationVariables); -}); -``` - #### Mocking Apollo Client To test the components with Apollo operations, we need to mock an Apollo Client in our unit tests. We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client and [`createMockApollo` helper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/mock_apollo_helper.js) we created on top of it. @@ -1393,38 +1237,40 @@ import Vue from 'vue'; Vue.use(VueApollo); -function createMockApolloProvider() { - return createMockApollo(requestHandlers); -} +describe('Some component with Apollo mock', () => { + let wrapper; -function createComponent(options = {}) { - const { mockApollo } = options; - ... - return shallowMount(..., { - apolloProvider: mockApollo, - ... - }); -} + function createComponent(options = {}) { + wrapper = shallowMount(...); + } +}) ``` -After this, you can control whether you need a variable for `mockApollo` and assign it in the appropriate `describe`-scope: +After this, we need to create a mocked Apollo provider: ```javascript -describe('Some component', () => { +import createMockApollo from 'helpers/mock_apollo_helper'; + +describe('Some component with Apollo mock', () => { let wrapper; + let mockApollo; - describe('with Apollo mock', () => { - let mockApollo; + function createComponent(options = {}) { + mockApollo = createMockApollo(...) - beforeEach(() => { - mockApollo = createMockApolloProvider(); - wrapper = createComponent({ mockApollo }); + wrapper = shallowMount(SomeComponent, { + apolloProvider: mockApollo }); - }); -}); + } + + afterEach(() => { + // we need to ensure we don't have provider persisted between tests + mockApollo = null + }) +}) ``` -In the `createMockApolloProvider`-factory, we need to define an array of _handlers_ for every query or mutation: +Now, we need to define an array of _handlers_ for every query or mutation. Handlers should be mock functions that return either a correct query response, or an error: ```javascript import getDesignListQuery from '~/design_management/graphql/queries/get_design_list.query.graphql'; @@ -1435,72 +1281,70 @@ describe('Some component with Apollo mock', () => { let wrapper; let mockApollo; - function createMockApolloProvider() { - Vue.use(VueApollo); - - const requestHandlers = [ - [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], - [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], - ]; - ... + function createComponent(options = { + designListHandler: jest.fn().mockResolvedValue(designListQueryResponse) + }) { + mockApollo = createMockApollo([ + [getDesignListQuery, options.designListHandler], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + [moveDesignMutation, jest.fn().mockResolvedValue(moveDesignMutationResponse)], + ]) + + wrapper = shallowMount(SomeComponent, { + apolloProvider: mockApollo + }); } }) ``` -After this, we need to create a mock Apollo Client instance using a helper: +When mocking resolved values, ensure the structure of the response is the same +as the actual API response. For example, root property should be `data`: ```javascript -import createMockApollo from 'helpers/mock_apollo_helper'; - -describe('Some component', () => { - let wrapper; - - function createMockApolloProvider() { - Vue.use(VueApollo); - - const requestHandlers = [ - [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], - [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], - ]; - - return createMockApollo(requestHandlers); - } - - function createComponent(options = {}) { - const { mockApollo } = options; - - return shallowMount(Index, { - apolloProvider: mockApollo, - }); - } - - describe('with Apollo mock', () => { - let mockApollo; - - beforeEach(() => { - mockApollo = createMockApolloProvider(); - wrapper = createComponent({ mockApollo }); - }); - }); -}); +const designListQueryResponse = { + data: { + project: { + id: '1', + issue: { + id: 'issue-1', + designCollection: { + copyState: 'READY', + designs: { + nodes: [ + { + id: '3', + event: 'NONE', + filename: 'fox_3.jpg', + notesCount: 1, + image: 'image-3', + imageV432x230: 'image-3', + currentUserTodos: { + nodes: [], + }, + }, + ], + }, + versions: { + nodes: [], + }, + }, + }, + }, + }, +}; ``` -When mocking resolved values, ensure the structure of the response is the same -as the actual API response. For example, root property should be `data`. - When testing queries, keep in mind they are promises, so they need to be _resolved_ to render a result. Without resolving, we can check the `loading` state of the query: ```javascript it('renders a loading state', () => { - const mockApollo = createMockApolloProvider(); - const wrapper = createComponent({ mockApollo }); + const wrapper = createComponent(); expect(wrapper.findComponent(LoadingSpinner).exists()).toBe(true) }); it('renders designs list', async () => { - const mockApollo = createMockApolloProvider(); - const wrapper = createComponent({ mockApollo }); + const wrapper = createComponent(); await waitForPromises() @@ -1511,17 +1355,10 @@ it('renders designs list', async () => { If we need to test a query error, we need to mock a rejected value as request handler: ```javascript -function createMockApolloProvider() { - ... - const requestHandlers = [ - [getDesignListQuery, jest.fn().mockRejectedValue(new Error('GraphQL error')], - ]; - ... -} -... - it('renders error if query fails', async () => { - const wrapper = createComponent(); + const wrapper = createComponent({ + designListHandler: jest.fn.mockRejectedValue('Houston, we have a problem!') + }); await waitForPromises() @@ -1529,38 +1366,28 @@ it('renders error if query fails', async () => { }) ``` -Request handlers can also be passed to component factory as a parameter. - Mutations could be tested the same way: ```javascript -function createMockApolloProvider({ - moveHandler = jest.fn().mockResolvedValue(moveDesignMutationResponse), -}) { - Vue.use(VueApollo); - - moveDesignHandler = moveHandler; - - const requestHandlers = [ - [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], - [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], - [moveDesignMutation, moveDesignHandler], - ]; - - return createMockApollo(requestHandlers); -} - -function createComponent(options = {}) { - const { mockApollo } = options; + const moveDesignHandlerSuccess = jest.fn().mockResolvedValue(moveDesignMutationResponse) + + function createComponent(options = { + designListHandler: jest.fn().mockResolvedValue(designListQueryResponse), + moveDesignHandler: moveDesignHandlerSuccess + }) { + mockApollo = createMockApollo([ + [getDesignListQuery, options.designListHandler], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + [moveDesignMutation, moveDesignHandler], + ]) + + wrapper = shallowMount(SomeComponent, { + apolloProvider: mockApollo + }); + } - return shallowMount(Index, { - apolloProvider: mockApollo, - }); -} -... it('calls a mutation with correct parameters and reorders designs', async () => { - const mockApollo = createMockApolloProvider({}); - const wrapper = createComponent({ mockApollo }); + const wrapper = createComponent(); wrapper.find(VueDraggable).vm.$emit('change', { moved: { @@ -1569,7 +1396,7 @@ it('calls a mutation with correct parameters and reorders designs', async () => }, }); - expect(moveDesignHandler).toHaveBeenCalled(); + expect(moveDesignHandlerSuccess).toHaveBeenCalled(); await waitForPromises(); @@ -1726,121 +1553,29 @@ query fetchLocalUser { ```javascript import fetchLocalUserQuery from '~/design_management/graphql/queries/fetch_local_user.query.graphql'; -function createMockApolloProvider() { - Vue.use(VueApollo); - - const requestHandlers = [ - [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], - [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], - ]; - - const mockApollo = createMockApollo(requestHandlers, {}); - mockApollo.clients.defaultClient.cache.writeQuery({ - query: fetchLocalUserQuery, - data: { - fetchLocalUser: { - __typename: 'User', - name: 'Test', - }, - }, - }); - - return mockApollo; -} - -function createComponent(options = {}) { - const { mockApollo } = options; - - return shallowMount(Index, { - apolloProvider: mockApollo, - }); -} -``` - -Sometimes it is necessary to control what the local resolver returns and inspect how it is called by the component. This can be done by mocking your local resolver: - -```javascript -import fetchLocalUserQuery from '~/design_management/graphql/queries/fetch_local_user.query.graphql'; - -function createMockApolloProvider(options = {}) { - Vue.use(VueApollo); - const { fetchLocalUserSpy } = options; - - const mockApollo = createMockApollo([], { - Query: { - fetchLocalUser: fetchLocalUserSpy, - }, - }); - - // Necessary for local resolvers to be activated - mockApollo.clients.defaultClient.cache.writeQuery({ - query: fetchLocalUserQuery, - data: {}, - }); - - return mockApollo; -} -``` - -In the test you can then control what the spy is supposed to do and inspect the component after the request have returned: - -```javascript -describe('My Index test with `createMockApollo`', () => { +describe('Some component with Apollo mock', () => { let wrapper; - let fetchLocalUserSpy; - - afterEach(() => { - wrapper.destroy(); - fetchLocalUserSpy = null; - }); - - describe('when loading', () => { - beforeEach(() => { - const mockApollo = createMockApolloProvider(); - wrapper = createComponent({ mockApollo }); - }); - - it('displays the loader', () => { - // Assess that the loader is present - }); - }); - - describe('with data', () => { - beforeEach(async () => { - fetchLocalUserSpy = jest.fn().mockResolvedValue(localUserQueryResponse); - const mockApollo = createMockApolloProvider(fetchLocalUserSpy); - wrapper = createComponent({ mockApollo }); - await waitForPromises(); - }); - - it('should fetch data once', () => { - expect(fetchLocalUserSpy).toHaveBeenCalledTimes(1); - }); - - it('displays data', () => { - // Assess that data is present - }); - }); - - describe('with error', () => { - const error = 'Error!'; - - beforeEach(async () => { - fetchLocalUserSpy = jest.fn().mockRejectedValueOnce(error); - const mockApollo = createMockApolloProvider(fetchLocalUserSpy); - wrapper = createComponent({ mockApollo }); - await waitForPromises(); - }); + let mockApollo; - it('should fetch data once', () => { - expect(fetchLocalUserSpy).toHaveBeenCalledTimes(1); + function createComponent(options = { + designListHandler: jest.fn().mockResolvedValue(designListQueryResponse) + }) { + mockApollo = createMockApollo([...]) + mockApollo.clients.defaultClient.cache.writeQuery({ + query: fetchLocalUserQuery, + data: { + fetchLocalUser: { + __typename: 'User', + name: 'Test', + }, + }, }); - it('displays the error', () => { - // Assess that the error is displayed + wrapper = shallowMount(SomeComponent, { + apolloProvider: mockApollo }); - }); -}); + } +}) ``` When you need to configure the mocked apollo client's caching behavior, @@ -1854,21 +1589,15 @@ const defaultCacheOptions = { ``` ```javascript -function createMockApolloProvider({ props = {}, requestHandlers } = {}) { - Vue.use(VueApollo); - - const mockApollo = createMockApollo( - requestHandlers, - {}, - { - dataIdFromObject: (object) => - // eslint-disable-next-line no-underscore-dangle - object.__typename === 'Requirement' ? object.iid : defaultDataIdFromObject(object), - }, - ); - - return mockApollo; -} +mockApollo = createMockApollo( + requestHandlers, + {}, + { + dataIdFromObject: (object) => + // eslint-disable-next-line no-underscore-dangle + object.__typename === 'Requirement' ? object.iid : defaultDataIdFromObject(object), + }, +); ``` ## Handling errors diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index 5ad918d466b..4242dd837f8 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -352,7 +352,6 @@ To generate these known events for a single widget: 1. Repeat step 6, but change the `data_source` to `redis_hll`. 1. Add each of the HLL metrics to `lib/gitlab/usage_data_counters/known_events/code_review_events.yml`: 1. `name` = (the event) - 1. `redis_slot` = `code_review` 1. `category` = `code_review` 1. `aggregation` = `weekly` 1. Add each event (those listed in the command in step 7, replacing `test_reports` diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 3aa901093f0..20609718217 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -142,21 +142,21 @@ To use the Vue performance plugin: 1. Import the plugin: - ```javascript - import PerformancePlugin from '~/performance/vue_performance_plugin'; - ``` + ```javascript + import PerformancePlugin from '~/performance/vue_performance_plugin'; + ``` 1. Use it before initializing your Vue application: - ```javascript - Vue.use(PerformancePlugin, { - components: [ - 'IdeTreeList', - 'FileTree', - 'RepoEditor', - ] - }); - ``` + ```javascript + Vue.use(PerformancePlugin, { + components: [ + 'IdeTreeList', + 'FileTree', + 'RepoEditor', + ] + }); + ``` The plugin accepts the list of components, performance of which should be measured. The components should be specified by their `name` option. diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index 5f2e8c1e029..9f52850041d 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -1,6 +1,6 @@ --- stage: Create -group: Editor +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -35,7 +35,7 @@ Vue component, but the integration of Source Editor is generally straightforward const editor = new SourceEditor({ // Editor Options. // The list of all accepted options can be found at - // https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.EditorOption.html + // https://microsoft.github.io/monaco-editor/docs.html }); ``` @@ -61,7 +61,7 @@ An instance of Source Editor accepts the following configuration options: ## API The editor uses the same public API as -[provided by Monaco editor](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) +[provided by Monaco editor](https://microsoft.github.io/monaco-editor/docs.html) with additional functions on the instance level: | Function | Arguments | Description diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index e57c117bc39..8e0814ad96b 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -16,15 +16,15 @@ To build and launch Storybook locally, in the root directory of the `gitlab` pro 1. Install Storybook dependencies: - ```shell - yarn storybook:install - ``` + ```shell + yarn storybook:install + ``` 1. Build the Storybook site: - ```shell - yarn storybook:start - ``` + ```shell + yarn storybook:start + ``` ## Adding components to Storybook @@ -35,14 +35,14 @@ To add a story: 1. Create a new `.stories.js` file in the same directory as the Vue component. The filename should have the same prefix as the Vue component. - ```txt - vue_shared/ - ├─ components/ - │ ├─ sidebar - │ | ├─ todo_toggle - │ | | ├─ todo_button.vue - │ │ | ├─ todo_button.stories.js - ``` + ```txt + vue_shared/ + ├─ components/ + │ ├─ sidebar + │ | ├─ todo_toggle + │ | | ├─ todo_button.vue + │ │ | ├─ todo_button.stories.js + ``` 1. Write the story as per the [official Storybook instructions](https://storybook.js.org/docs/vue/writing-stories/introduction/) diff --git a/doc/development/fe_guide/style/html.md b/doc/development/fe_guide/style/html.md index b1cce29bc61..c92f77e9033 100644 --- a/doc/development/fe_guide/style/html.md +++ b/doc/development/fe_guide/style/html.md @@ -58,7 +58,7 @@ Button tags requires a `type` attribute according to the [W3C HTML specification ### Blank target -Arbitrarily opening links in a new tab is not recommended, so refer to the [Pajamas guidelines on links](https://design.gitlab.com/product-foundations/interaction/#links) when considering adding `target="_blank"` to links. +Arbitrarily opening links in a new tab is not recommended, so refer to the [Pajamas guidelines on links](https://design.gitlab.com/components/link) when considering adding `target="_blank"` to links. When using `target="_blank"` with `a` tags, you must also add the `rel="noopener noreferrer"` attribute. This prevents a security vulnerability [documented by JitBit](https://www.jitbit.com/alexblog/256-targetblank---the-most-underestimated-vulnerability-ever/). diff --git a/doc/development/fe_guide/style/index.md b/doc/development/fe_guide/style/index.md index 94ed9544cf5..552b769d6f6 100644 --- a/doc/development/fe_guide/style/index.md +++ b/doc/development/fe_guide/style/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab development style guides +# Frontend style guides See below for the relevant style guides, guidelines, linting, and other information for developing GitLab. diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 3e3a79dd7bb..b35ffdd8669 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -329,3 +329,22 @@ Only export the constants as a collection (array, or object) when there is a nee // good, if the constants need to be iterated over export const VARIANTS = [VARIANT_WARNING, VARIANT_ERROR]; ``` + +## Error handling + +When catching a server-side error you should use the error message +utility function contained in `app/assets/javascripts/lib/utils/error_message.js`. +This utility parses the received error message and checks for a prefix that indicates +whether the message is meant to be user-facing or not. The utility returns +an object with the message, and a boolean indicating whether the message is meant to be user-facing or not. Please make sure that the Backend is aware of the utils usage and is adding the prefix +to the error message accordingly. + +```javascript +import { parseErrorMessage } from '~/lib/utils/error_message'; + +onError(error) { + const { message, userFacing } = parseErrorMessage(error); + + const errorMessage = userFacing ? message : genericErrorText; +} +``` diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index a21d7c4577b..e9d2a724d9d 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -59,63 +59,66 @@ Check the [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) for mor 1. Explicitly define data being passed into the Vue app - ```javascript - // bad - return new Vue({ - el: '#element', - components: { - componentName - }, - provide: { - ...someDataset - }, - props: { - ...anotherDataset - }, - render: createElement => createElement('component-name'), - })); - - // good - const { foobar, barfoo } = someDataset; - const { foo, bar } = anotherDataset; - - return new Vue({ - el: '#element', - components: { - componentName - }, - provide: { - foobar, - barfoo - }, - props: { - foo, - bar - }, - render: createElement => createElement('component-name'), - })); - ``` - - We discourage the use of the spread operator in this specific case in - order to keep our codebase explicit, discoverable, and searchable. - This applies in any place where we would benefit from the above, such as - when [initializing Vuex state](../vuex.md#why-not-just-spread-the-initial-state). - The pattern above also enables us to easily parse non scalar values during - instantiation. - - ```javascript - return new Vue({ - el: '#element', - components: { - componentName - }, - props: { - foo, - bar: parseBoolean(bar) - }, - render: createElement => createElement('component-name'), - })); - ``` + ```javascript + // bad + return new Vue({ + el: '#element', + name: 'ComponentNameRoot', + components: { + componentName + }, + provide: { + ...someDataset + }, + props: { + ...anotherDataset + }, + render: createElement => createElement('component-name'), + })); + + // good + const { foobar, barfoo } = someDataset; + const { foo, bar } = anotherDataset; + + return new Vue({ + el: '#element', + name: 'ComponentNameRoot', + components: { + componentName + }, + provide: { + foobar, + barfoo + }, + props: { + foo, + bar + }, + render: createElement => createElement('component-name'), + })); + ``` + + We discourage the use of the spread operator in this specific case in + order to keep our codebase explicit, discoverable, and searchable. + This applies in any place where we would benefit from the above, such as + when [initializing Vuex state](../vuex.md#why-not-just-spread-the-initial-state). + The pattern above also enables us to easily parse non scalar values during + instantiation. + + ```javascript + return new Vue({ + el: '#element', + name: 'ComponentNameRoot', + components: { + componentName + }, + props: { + foo, + bar: parseBoolean(bar) + }, + render: createElement => createElement('component-name'), + })); + ``` ## Naming @@ -404,7 +407,7 @@ When using `v-for` you need to provide a *unique* `:key` attribute for each item ``` 1. When using `v-for` with `template` and there is more than one child element, the `:key` values -must be unique. It's advised to use `kebab-case` namespaces. + must be unique. It's advised to use `kebab-case` namespaces. ```html <template v-for="(item, index) in items"> @@ -467,8 +470,9 @@ Creating a global, mutable wrapper provides a number of advantages, including th ``` - Use a `beforeEach` block to mount the component (see -[the `createComponent` factory](#the-createcomponent-factory) for more information). -- Use an `afterEach` block to destroy the component, for example, `wrapper.destroy()`. + [the `createComponent` factory](#the-createcomponent-factory) for more information). +- Automatically destroy the component after the test is run with [`enableAutoDestroy`](https://v1.test-utils.vuejs.org/api/#enableautodestroy-hook) + set in [`shared_test_setup.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/d0bdc8370ef17891fd718a4578e41fef97cf065d/spec/frontend/__helpers__/shared_test_setup.js#L20). #### The `createComponent` factory @@ -538,42 +542,42 @@ describe('MyComponent', () => { #### `createComponent` best practices 1. Consider using a single (or a limited number of) object arguments over many arguments. - Defining single parameters for common data like `props` is okay, - but keep in mind our [JavaScript style guide](javascript.md#limit-number-of-parameters) and - stay within the parameter number limit: + Defining single parameters for common data like `props` is okay, + but keep in mind our [JavaScript style guide](javascript.md#limit-number-of-parameters) and + stay within the parameter number limit: - ```javascript - // bad - function createComponent(data, props, methods, isLoading, mountFn) { } + ```javascript + // bad + function createComponent(data, props, methods, isLoading, mountFn) { } - // good - function createComponent({ data, props, methods, stubs, isLoading } = {}) { } + // good + function createComponent({ data, props, methods, stubs, isLoading } = {}) { } - // good - function createComponent(props = {}, { data, methods, stubs, isLoading } = {}) { } - ``` + // good + function createComponent(props = {}, { data, methods, stubs, isLoading } = {}) { } + ``` 1. If you require both `mount` _and_ `shallowMount` within the same set of tests, it -can be useful define a `mountFn` parameter for the `createComponent` factory that accepts -the mounting function (`mount` or `shallowMount`) to be used to mount the component: + can be useful define a `mountFn` parameter for the `createComponent` factory that accepts + the mounting function (`mount` or `shallowMount`) to be used to mount the component: - ```javascript - import { shallowMount } from '@vue/test-utils'; + ```javascript + import { shallowMount } from '@vue/test-utils'; - function createComponent({ mountFn = shallowMount } = {}) { } - ``` + function createComponent({ mountFn = shallowMount } = {}) { } + ``` 1. Use the `mountExtended` and `shallowMountExtended` helpers to expose `wrapper.findByTestId()`: - ```javascript - import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; - import { SomeComponent } from 'components/some_component.vue'; + ```javascript + import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; + import { SomeComponent } from 'components/some_component.vue'; - let wrapper; + let wrapper; - const createWrapper = () => { wrapper = shallowMountExtended(SomeComponent); }; - const someButton = () => wrapper.findByTestId('someButtonTestId'); - ``` + const createWrapper = () => { wrapper = shallowMountExtended(SomeComponent); }; + const someButton = () => wrapper.findByTestId('someButtonTestId'); + ``` ### Setting component state @@ -581,70 +585,70 @@ the mounting function (`mount` or `shallowMount`) to be used to mount the compon component state wherever possible. Instead, set the component's [`propsData`](https://v1.test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: - ```javascript - // bad - wrapper = shallowMount(MyComponent); - wrapper.setProps({ - myProp: 'my cool prop' - }); + ```javascript + // bad + wrapper = shallowMount(MyComponent); + wrapper.setProps({ + myProp: 'my cool prop' + }); - // good - wrapper = shallowMount({ propsData: { myProp: 'my cool prop' } }); - ``` + // good + wrapper = shallowMount({ propsData: { myProp: 'my cool prop' } }); + ``` - The exception here is when you wish to test component reactivity in some way. - For example, you may want to test the output of a component when after a particular watcher has - executed. Using `setProps` to test such behavior is okay. + The exception here is when you wish to test component reactivity in some way. + For example, you may want to test the output of a component when after a particular watcher has + executed. Using `setProps` to test such behavior is okay. ### Accessing component state 1. When accessing props or attributes, prefer the `wrapper.props('myProp')` syntax over `wrapper.props().myProp` or `wrapper.vm.myProp`: - ```javascript - // good - expect(wrapper.props().myProp).toBe(true); - expect(wrapper.attributes().myAttr).toBe(true); + ```javascript + // good + expect(wrapper.props().myProp).toBe(true); + expect(wrapper.attributes().myAttr).toBe(true); - // better - expect(wrapper.props('myProp').toBe(true); - expect(wrapper.attributes('myAttr')).toBe(true); - ``` + // better + expect(wrapper.props('myProp').toBe(true); + expect(wrapper.attributes('myAttr')).toBe(true); + ``` 1. When asserting multiple props, check the deep equality of the `props()` object with [`toEqual`](https://jestjs.io/docs/expect#toequalvalue): - ```javascript - // good - expect(wrapper.props('propA')).toBe('valueA'); - expect(wrapper.props('propB')).toBe('valueB'); - expect(wrapper.props('propC')).toBe('valueC'); - - // better - expect(wrapper.props()).toEqual({ - propA: 'valueA', - propB: 'valueB', - propC: 'valueC', - }); - ``` + ```javascript + // good + expect(wrapper.props('propA')).toBe('valueA'); + expect(wrapper.props('propB')).toBe('valueB'); + expect(wrapper.props('propC')).toBe('valueC'); + + // better + expect(wrapper.props()).toEqual({ + propA: 'valueA', + propB: 'valueB', + propC: 'valueC', + }); + ``` 1. If you are only interested in some of the props, you can use [`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): - ```javascript - // good - expect(wrapper.props()).toEqual(expect.objectContaining({ - propA: 'valueA', - propB: 'valueB', - })); + ```javascript + // good + expect(wrapper.props()).toEqual(expect.objectContaining({ + propA: 'valueA', + propB: 'valueB', + })); - // better - expect(wrapper.props()).toMatchObject({ - propA: 'valueA', - propB: 'valueB', - }); - ``` + // better + expect(wrapper.props()).toMatchObject({ + propA: 'valueA', + propB: 'valueB', + }); + ``` ## The JavaScript/Vue Accord @@ -659,16 +663,16 @@ The goal of this accord is to make sure we are all on the same page. 1. We avoid adding new jQuery events when they are not required. Instead of adding new jQuery events take a look at [different methods to do the same task](https://v2.vuejs.org/v2/api/#vm-emit). 1. You may query the `window` object one time, while bootstrapping your application for application -specific data (for example, `scrollTo` is ok to access anytime). Do this access during the -bootstrapping of your application. + specific data (for example, `scrollTo` is ok to access anytime). Do this access during the + bootstrapping of your application. 1. You may have a temporary but immediate need to create technical debt by writing code that does -not follow our standards, to be refactored later. Maintainers need to be ok with the tech debt in -the first place. An issue should be created for that tech debt to evaluate it further and discuss. -In the coming months you should fix that tech debt, with its priority to be determined by maintainers. + not follow our standards, to be refactored later. Maintainers need to be ok with the tech debt in + the first place. An issue should be created for that tech debt to evaluate it further and discuss. + In the coming months you should fix that tech debt, with its priority to be determined by maintainers. 1. When creating tech debt you must write the tests for that code before hand and those tests may -not be rewritten. For example, jQuery tests rewritten to Vue tests. + not be rewritten. For example, jQuery tests rewritten to Vue tests. 1. You may choose to use VueX as a centralized state management. If you choose not to use VueX, you -must use the *store pattern* which can be found in the -[Vue.js documentation](https://v2.vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch). + must use the *store pattern* which can be found in the + [Vue.js documentation](https://v2.vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch). 1. Once you have chosen a centralized state-management solution you must use it for your entire -application. Don't mix and match your state-management solutions. + application. Don't mix and match your state-management solutions. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index cea47bc0e4c..119a91e33b5 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -76,7 +76,13 @@ component, is that you avoid creating a fixture or an HTML element in the unit t `initSimpleApp` is a helper function that streamlines the process of mounting a component in Vue.js. It accepts two arguments: a selector string representing the mount point in the HTML, and a Vue component. -To use `initSimpleApp`, include the HTML element in the page with the appropriate selector and add a data-view-model attribute containing a JSON object. Then, import the desired Vue component and pass it along with the selector to `initSimpleApp`. This mounts the component at the specified location. +To use `initSimpleApp`: + +1. Include an HTML element in the page with an ID or unique class. +1. Add a data-view-model attribute containing a JSON object. +1. Import the desired Vue component, and pass it along with a valid CSS selector string + that selects the HTML element to `initSimpleApp`. This string mounts the component + at the specified location. `initSimpleApp` automatically retrieves the content of the data-view-model attribute as a JSON object and passes it as props to the mounted Vue component. This can be used to pre-populate the component with data. @@ -138,6 +144,7 @@ const { endpoint } = el.dataset; return new Vue({ el, + name: 'MyComponentRoot', render(createElement) { return createElement('my-component', { provide: { @@ -198,6 +205,7 @@ const { endpoint } = el.dataset; return new Vue({ el, + name: 'MyComponentRoot', render(createElement) { return createElement('my-component', { props: { @@ -252,6 +260,7 @@ export const initUserForm = () => { return new Vue({ el, + name: 'UserFormRoot', render(h) { return h(UserForm, { props: { @@ -305,6 +314,7 @@ initializing our Vue instance, and the data should be provided as `props` to the ```javascript return new Vue({ el: '.js-vue-app', + name: 'MyComponentRoot', render(createElement) { return createElement('my-component', { props: { @@ -443,6 +453,22 @@ Composition API allows you to place the logic in the `<script>` section of the c </script> ``` +### `v-bind` limitations + +Avoid using `v-bind="$attrs"` unless absolutely necessary. You might need this when +developing a native control wrapper. (This is a good candidate for a `gitlab-ui` component.) +In any other cases, always prefer using `props` and explicit data flow. + +Using `v-bind="$attrs"` leads to: + +1. A loss in component's contract. The `props` were designed specifically + to address this problem. +1. High maintenance cost for each component in the tree. `v-bind="$attrs"` is specifically + hard to debug because you must scan the whole hierarchy of components to understand + the data flow. +1. Problems during migration to Vue 3. `$attrs` in Vue 3 include event listeners which + could cause unexpected side-effects after Vue 3 migration is completed. + ### Aim to have one API style per component When adding `setup()` property to Vue component, consider refactoring it to Composition API entirely. It's not always feasible, especially for large components, but we should aim to have one API style per component for readability and maintainability. @@ -608,8 +634,7 @@ describe('~/todos/app.vue', () => { }); afterEach(() => { - // IMPORTANT: Clean up the component instance and axios mock adapter - wrapper.destroy(); + // IMPORTANT: Clean up the axios mock adapter mock.restore(); }); diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 01ee50fb6ca..52278c94e9f 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -6,14 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vuex -When there's a clear benefit to separating state management from components (for example, due to state complexity) we recommend using [Vuex](https://vuex.vuejs.org) over any other Flux pattern. Otherwise, feel free to manage state in the components. - -Vuex should be strongly considered when: - -- You expect multiple parts of the application to react to state changes. -- There's a need to share data between multiple components. -- There are complex interactions with Backend, for example, multiple API calls. -- The app involves interacting with backend via both traditional REST API and GraphQL (especially when moving the REST API over to GraphQL is a pending backend task). +[Vuex](https://vuex.vuejs.org) should no longer be considered a preferred path to store management and is currently in its legacy phase. This means it is acceptable to add upon existing `Vuex` stores, but we strongly recommend reducing store sizes over time and eventually migrating fully to [Apollo](https://www.apollographql.com/) whenever it's possible. Before adding any new `Vuex` store to an application, first ensure that the `Vue` application you plan to add it into **does not use** `Apollo`. `Vuex` and `Apollo` should not be combined unless absolutely necessary. Please consider reading through [our GraphQL documentation](../fe_guide/graphql.md) for more guidelines on how you can build `Apollo` based applications. The information included in this page is explained in more detail in the official [Vuex documentation](https://vuex.vuejs.org). @@ -97,7 +90,7 @@ In this file, we write the actions that call mutations for handling a list of us ```javascript import * as types from './mutation_types'; import axios from '~/lib/utils/axios_utils'; - import { createAlert } from '~/flash'; + import { createAlert } from '~/alert'; export const fetchUsers = ({ state, dispatch }) => { commit(types.REQUEST_USERS); @@ -305,6 +298,7 @@ export default () => { return new Vue({ el, + name: 'AwesomeVueRoot', store: createStore(el.dataset), render: h => h(AwesomeVueApp) }); @@ -462,10 +456,6 @@ describe('component', () => { createComponent(); }); - afterEach(() => { - wrapper.destroy(); - }); - it('should show a user', async () => { const user = { name: 'Foo', diff --git a/doc/development/fe_guide/widgets.md b/doc/development/fe_guide/widgets.md index edb8559da48..6cd8e6c091c 100644 --- a/doc/development/fe_guide/widgets.md +++ b/doc/development/fe_guide/widgets.md @@ -62,11 +62,11 @@ Because we need different GraphQL queries and mutations for different sidebars, ```javascript export const assigneesQueries = { - [IssuableType.Issue]: { + [TYPE_ISSUE]: { query: getIssueParticipants, mutation: updateAssigneesMutation, }, - [IssuableType.MergeRequest]: { + [TYPE_MERGE_REQUEST]: { query: getMergeRequestParticipants, mutation: updateMergeRequestParticipantsMutation, }, diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index 141b24161b4..c3cee596d8a 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -79,7 +79,7 @@ Consult these topics for information on contributing to specific GitLab features - [Adding a new Redis instance](redis/new_redis_instance.md) - [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) -- [Elasticsearch integration docs](elasticsearch.md) +- [Advanced search integration docs](advanced_search.md) - [Working with merge request diffs](diffs.md) - [Approval Rules](merge_request_concepts/approval_rules.md) - [Repository mirroring](repository_mirroring.md) @@ -127,7 +127,6 @@ See [database guidelines](database/index.md). - [Security Scanners](integrations/secure.md) - [Secure Partner Integration](integrations/secure_partner_integration.md) - [How to run Jenkins in development environment](integrations/jenkins.md) -- [How to run local CodeSandbox integration for Web IDE Live Preview](integrations/codesandbox.md) The following integration guides are internal. Some integrations require access to administrative accounts of third-party services and are available only for GitLab team members to contribute to: diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 3adf5248b8d..f8a592f98f5 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -84,6 +84,8 @@ When a feature has successfully been environment and verified as safe and working, you can roll out the change to GitLab.com (production). +If a feature is [deprecated](../../update/deprecations.md), do not enable the flag. + #### Communicate the change Some feature flag changes on GitLab.com should be communicated with diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 7370697b082..9b6876ac0bc 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -35,7 +35,7 @@ should be leveraged: the status of a feature behind the feature flag in the documentation and with other stakeholders. The issue description should be updated with the feature flag name and whether it is defaulted on or off as soon it is evident that a feature flag is needed. -- Merge requests that introduce a feature flag, update its state, or remove them +- Merge requests that introduce a feature flag, update its state, or remove the existing feature flag because a feature is deemed stable must have the ~"feature flag" label assigned. diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 147ff5fa6e9..f0634107ba5 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -67,9 +67,8 @@ listed here that also do not work properly in FIPS mode: - [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) supports a reduced set of [analyzers](../user/application_security/sast/index.md#fips-enabled-images) when operating in FIPS-compliant mode. -- Advanced Search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. +- Advanced search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. - [Gravatar or Libravatar-based profile images](../administration/libravatar.md) are not FIPS-compliant. -- [Personal Access Tokens](../user/profile/personal_access_tokens.md) are not available for use or creation. Additionally, these package repositories are disabled in FIPS mode: @@ -275,104 +274,55 @@ all: gitlab_charts_custom_config_file: '/path/to/gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/charts.yml' ``` -Now create `charts.yml` in the location specified above and specify tags with a `-fips` suffix. For example: +Now create `charts.yml` in the location specified above and specify tags with a `-fips` suffix. -```yaml -global: - image: - pullPolicy: Always - certificates: - image: - tag: master-fips - kubectl: - image: - tag: master-fips - -gitlab: - gitaly: - image: - tag: master-fips - gitlab-exporter: - image: - tag: master-fips - gitlab-shell: - image: - tag: main-fips # The default branch is main, not master - gitlab-mailroom: - image: - tag: master-fips - gitlab-pages: - image: - tag: master-fips - migrations: - image: - tag: master-fips - sidekiq: - image: - tag: master-fips - toolbox: - image: - tag: master-fips - webservice: - image: - tag: master-fips - workhorse: - tag: master-fips - -nginx-ingress: - controller: - image: - repository: registry.gitlab.com/gitlab-org/cloud-native/charts/gitlab-ingress-nginx/controller - tag: v1.2.1-fips - pullPolicy: Always - digest: sha256:c4222b7ab3836b9be2a7649cff4b2e6ead34286dfdf3a7b04eb34fdd3abb0334 -``` - -The above example shows a FIPS-enabled [`nginx-ingress`](https://github.com/kubernetes/ingress-nginx) image. -See our [Charts documentation on FIPS](https://docs.gitlab.com/charts/advanced/fips/index.html) for more details. +See our [Charts documentation on FIPS](https://docs.gitlab.com/charts/advanced/fips/index.html) for more details, including +an [example values file](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/examples/fips/values.yaml) as a reference. You can also use release tags, but the versioning is tricky because each component may use its own versioning scheme. For example, for GitLab v15.2: ```yaml global: + image: + tagSuffix: -fips certificates: image: - tag: 20211220-r0-fips + tag: 20211220-r0 kubectl: image: - tag: 1.18.20-fips + tag: 1.18.20 gitlab: gitaly: image: - tag: v15.2.0-fips + tag: v15.2.0 gitlab-exporter: image: - tag: 11.17.1-fips + tag: 11.17.1 gitlab-shell: image: - tag: v14.9.0-fips + tag: v14.9.0 gitlab-mailroom: image: - tag: v15.2.0-fips + tag: v15.2.0 gitlab-pages: image: - tag: v1.61.0-fips + tag: v1.61.0 migrations: image: - tag: v15.2.0-fips + tag: v15.2.0 sidekiq: image: - tag: v15.2.0-fips + tag: v15.2.0 toolbox: image: - tag: v15.2.0-fips + tag: v15.2.0 webservice: image: - tag: v15.2.0-fips + tag: v15.2.0 workhorse: - tag: v15.2.0-fips + tag: v15.2.0 ``` ## FIPS Performance Benchmarking @@ -496,7 +446,7 @@ irb(main):001:0> require 'openssl'; OpenSSL.fips_mode ### Go -Google maintains a [`dev.boringcrypto` branch](https://github.com/golang/go/tree/dev.boringcrypto) in the Golang compiler +Google maintains a [`dev.boringcrypto` branch](https://github.com/golang/go/tree/dev.boringcrypto) in the Go compiler that makes it possible to statically link BoringSSL, a FIPS-validated module forked from OpenSSL. However, BoringSSL is not intended for public use. diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 36ef1bcd834..38c0d77bcae 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Gemfile guidelines +# Gemfile development guidelines When adding a new entry to `Gemfile`, or upgrading an existing dependency pay attention to the following rules. @@ -60,6 +60,8 @@ This means that new dependencies should, at a minimum, meet the following criter - There are no issues open that we know may impact the availability or performance of GitLab. - The project is tested using some form of test automation. The test suite must be passing using the Ruby version currently used by GitLab. +- CI builds for all supported platforms must succeed using the new dependency. For more information, see + how to [build a package for testing](build_test_package.md#building-a-package-for-testing). - If the project uses a C extension, consider requesting an additional review from a C or MRI domain expert. C extensions can greatly impact GitLab stability and performance. @@ -95,6 +97,8 @@ does not contain any hidden dependencies on our application code. In general, we want to think carefully before doing this as there are also disadvantages: +### Potential disadvantages + 1. Gems - even those maintained by GitLab - do not necessarily go through the same [code review process](code_review.md) as the main Rails application. @@ -106,9 +110,23 @@ also disadvantages: community's needs. In general, if we are not using the latest version of our own gem, that might be a warning sign. +### Create and publish a Ruby gem + In the case where we do want to extract some library code we've written to a gem, go through these steps: +1. Determine a suitable name for the gem. If it's a GitLab-owned gem, prefix + the gem name with `gitlab-`. For example, `gitlab-sidekiq-fetcher`. +1. Create the gem or fork as necessary. +1. Ensure the `gitlab_rubygems` group is an owner of the new gem by running: + + ```shell + gem owner <gem-name> --add gitlab_rubygems + ``` + +1. [Publish the gem to rubygems.org](https://guides.rubygems.org/publishing/#publishing-to-rubygemsorg) +1. Visit `https://rubygems.org/gems/<gem-name>` and verify that the gem published + successfully and `gitlab_rubygems` is also an owner. 1. Start with the code in the Rails application. Here it's fine to have the code in `lib/` and loaded automatically. We can skip this step if the step below makes more sense initially. diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index b4f5501ccac..addab512f89 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -4,7 +4,7 @@ group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Gitaly developers guide +# Gitaly development guidelines [Gitaly](https://gitlab.com/gitlab-org/gitaly) is a high-level Git RPC service used by GitLab Rails, Workhorse and GitLab Shell. @@ -14,7 +14,7 @@ Workhorse and GitLab Shell. <!-- vale gitlab.Spelling = NO --> In May 2019, Bob Van Landuyt -hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) +hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`) on the [Gitaly project](https://gitlab.com/gitlab-org/gitaly). It included how to contribute to it as a Ruby developer, and shared domain-specific knowledge with anyone who may work in this part of the codebase in the future. @@ -257,13 +257,13 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. 1. Create a package scoped flag name: - ```golang + ```go var findAllTagsFeatureFlag = "go-find-all-tags" ``` 1. Create a switch in the code using the `featureflag` package: - ```golang + ```go if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { // go implementation } else { @@ -273,7 +273,7 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. 1. Create Prometheus metrics: - ```golang + ```go var findAllTagsRequests = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "gitaly_find_all_tags_requests_total", @@ -297,7 +297,7 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. 1. Set headers in tests: - ```golang + ```go import ( "google.golang.org/grpc/metadata" diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 17b4a28f57d..2d9ae768d33 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -12,10 +12,11 @@ necessary to import GitHub projects into a GitLab instance. The GitHub importer offers two different types of importers: a sequential importer and a parallel importer. The Rake task `import:github` uses the -sequential importer, while everything else uses the parallel importer. The -difference between these two importers is quite simple: the sequential importer -does all work in a single thread, making it more useful for debugging purposes -or Rake tasks. The parallel importer, on the other hand, uses Sidekiq. +sequential importer, and everything else uses the parallel importer. The +difference between these two importers is: + +- The sequential importer does all the work in a single thread, so it's more suited for debugging purposes or Rake tasks. +- The parallel importer uses Sidekiq. ## Requirements @@ -179,10 +180,10 @@ Advancing stages is done in one of two ways: The first approach should only be used by workers that perform all their work in a single thread, while `AdvanceStageWorker` should be used for everything else. -The way `AdvanceStageWorker` works is fairly simple. When scheduling a job it +When you schedule a job, `AdvanceStageWorker` is given a project ID, a list of Redis keys, and the name of the next stage. The Redis keys (produced by `Gitlab::JobWaiter`) are used to check if the -currently running stage has been completed or not. If the stage has not yet been +running stage has been completed or not. If the stage has not yet been completed `AdvanceStageWorker` reschedules itself. After a stage finishes `AdvanceStageworker` refreshes the import JID (more on this below) and schedule the worker of the next stage. @@ -324,14 +325,6 @@ The last log entry reports the number of objects fetched and imported: } ``` -## Errors when importing large projects - -The GitHub importer may encounter errors when importing large projects. For help with this, see the -documentation for the following use cases: - -- [Alternative way to import notes and diff notes](../user/project/import/github.md#alternative-way-to-import-notes-and-diff-notes) -- [Reduce GitHub API request objects per page](../user/project/import/github.md#reduce-github-api-request-objects-per-page) - ## Metrics dashboards To assess the GitHub importer health, the [GitHub importer dashboard](https://dashboards.gitlab.net/d/importers-github-importer/importers-github-importer) diff --git a/doc/development/gitlab_flavored_markdown/index.md b/doc/development/gitlab_flavored_markdown/index.md index f115ae9a11c..064a7ecbfa9 100644 --- a/doc/development/gitlab_flavored_markdown/index.md +++ b/doc/development/gitlab_flavored_markdown/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w <!-- vale gitlab.GitLabFlavoredMarkdown = NO --> -# GitLab Flavored Markdown (GLFM) developer documentation +# GitLab Flavored Markdown (GLFM) development guidelines This page contains the MVC for the developer documentation for GitLab Flavored Markdown (GLFM). For the user documentation about Markdown in GitLab, refer to diff --git a/doc/development/gitlab_shell/index.md b/doc/development/gitlab_shell/index.md index 7097fd48cea..3a1af0fc9e9 100644 --- a/doc/development/gitlab_shell/index.md +++ b/doc/development/gitlab_shell/index.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Shell +# GitLab Shell development guidelines [](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) [](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) [](https://codeclimate.com/github/gitlabhq/gitlab-shell) @@ -21,8 +21,8 @@ Ruby to build and test, but not to run. GitLab Shell runs on `port 22` on an Omnibus installation. To use a regular SSH service, configure it on an alternative port. -Download and install the current version of Go from [golang.org](https://go.dev/dl/). -We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy) +Download and install the [current version of Go](https://go.dev/dl/). +We follow the [Go Release Policy](https://go.dev/doc/devel/release#policy) and support: - The current stable version. diff --git a/doc/development/gitpod_internals.md b/doc/development/gitpod_internals.md index a4674df758d..a4b340916dd 100644 --- a/doc/development/gitpod_internals.md +++ b/doc/development/gitpod_internals.md @@ -25,6 +25,6 @@ You can find this webhook in [Webhook Settings in `gitlab-org/gitlab`](https://g If a webhook failed to connect for a long time, then it may have been disabled in the project. -To re-enable a failing or failed webhook, send a test request in [Webhook Settings](https://gitlab.com/gitlab-org/gitlab/-/hooks). See [Re-enable disabled webhooks page](https://docs.gitlab.com/15.4/ee/user/project/integrations/webhooks.html#re-enable-disabled-webhooks) for more details. +To re-enable a failing or failed webhook, send a test request in [Webhook Settings](https://gitlab.com/gitlab-org/gitlab/-/hooks). See [Re-enable disabled webhooks page](../user/project/integrations/webhooks.md#re-enable-disabled-webhooks) for more details. After re-enabling, check the prebuilds' health in a [project's prebuilds](https://gitpod.io/t/gitlab-org/gitlab/prebuilds) and confirm that prebuilds start without any errors. diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index b3ec0a15c37..f71fe7b8dac 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -26,7 +26,7 @@ by Distribution: ## Supporting multiple Go versions -Individual Golang projects need to support multiple Go versions because: +Individual Go projects need to support multiple Go versions because: - When a new version of Go is released, we should start integrating it into the CI pipelines to verify compatibility with the new compiler. - We must support the [official Omnibus GitLab Go version](#updating-go-version), which may be behind the latest minor release. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 508219cee43..6092a05afff 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -124,7 +124,7 @@ lint: # Write the code coverage report to gl-code-quality-report.json # and print linting issues to stdout in the format: path/to/file:line description # remove `--issues-exit-code 0` or set to non-zero to fail the job if linting issues are detected - - golangci-lint run --issues-exit-code 0 --out-format code-climate | tee gl-code-quality-report.json | jq -r '.[] | "\(.location.path):\(.location.lines.begin) \(.description)"' + - golangci-lint run --issues-exit-code 0 --print-issued-lines=false --out-format code-climate:gl-code-quality-report.json,line-number artifacts: reports: codequality: gl-code-quality-report.json @@ -216,7 +216,7 @@ When comparing expected and actual values in tests, use and others to improve readability when comparing structs, errors, large portions of text, or JSON documents: -```golang +```go type TestData struct { // ... } @@ -291,7 +291,7 @@ easier to debug. For example: -```golang +```go // Wrap the error return nil, fmt.Errorf("get cache %s: %w", f.Name, err) @@ -462,7 +462,7 @@ allocations. **Don't:** -```golang +```go var s2 []string for _, val := range s1 { s2 = append(s2, val) @@ -471,8 +471,8 @@ for _, val := range s1 { **Do:** -```golang -s2 := make([]string, 0, size) +```go +s2 := make([]string, 0, len(s1)) for _, val := range s1 { s2 = append(s2, val) } @@ -494,7 +494,7 @@ If the scanner report is small, less than 35 lines, then feel free to [inline th The [go-cmp](https://github.com/google/go-cmp) package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: -```golang +```go package main import ( diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index cc7232cd793..1957662317a 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -103,7 +103,7 @@ are very appreciative of the work done by translators and proofreaders! - Portuguese - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [Crowdin](https://crowdin.com/profile/ldiogotrindade) - Portuguese, Brazilian - - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra), [Crowdin](https://crowdin.com/profile/paulogomes.rep) + - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra) - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) - Horberlan Brito - [GitLab](https://gitlab.com/horberlan), [Crowdin](https://crowdin.com/profile/horberlan) diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index d182bd8333e..48b780b50bf 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/img/feature-flag-metrics.png b/doc/development/img/feature-flag-metrics.png Binary files differindex ce8702d47eb..f94c0fc3e46 100644 --- a/doc/development/img/feature-flag-metrics.png +++ b/doc/development/img/feature-flag-metrics.png diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 7f5a0faf8fb..ed5854f8833 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -4,28 +4,29 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Test Import Project +# Test import project For testing, we can import our own [GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/) project (named `gitlabhq` in this case) under a group named `qa-perf-testing`. Project tarballs that can be used for testing can be found over on the [performance-data](https://gitlab.com/gitlab-org/quality/performance-data) project. A different project could be used if required. -There are several options for importing the project into your GitLab environment. They are detailed as follows with the assumption that the recommended group `qa-perf-testing` and project `gitlabhq` are being set up. +You can import the project into your GitLab environment in a number of ways. They are detailed as follows with the +assumption that the recommended group `qa-perf-testing` and project `gitlabhq` are being set up. ## Importing the project -There are several ways to import a project. +Use one of these methods to import the test project. -### Importing via UI +### Import by using the UI -The first option is to [import the Project tarball file via the GitLab UI](../user/project/settings/import_export.md#import-a-project-and-its-data): +The first option is to [import the project tarball file by using the GitLab UI](../user/project/settings/import_export.md#import-a-project-and-its-data): -1. Create the group `qa-perf-testing` -1. Import the [GitLab FOSS project tarball](https://gitlab.com/gitlab-org/quality/performance-data/-/blob/master/projects_export/gitlabhq_export.tar.gz) into the Group. +1. Create the group `qa-perf-testing`. +1. Import the [GitLab FOSS project tarball](https://gitlab.com/gitlab-org/quality/performance-data/-/blob/master/projects_export/gitlabhq_export.tar.gz) into the group. It should take up to 15 minutes for the project to fully import. You can head to the project's main page for the current status. This method ignores all the errors silently (including the ones related to `GITALY_DISABLE_REQUEST_LIMITS`) and is used by GitLab users. For development and testing, check the other methods below. -### Importing via the `import-project` script +### Import by using the `import-project` script A convenient script, [`bin/import-project`](https://gitlab.com/gitlab-org/quality/performance/blob/master/bin/import-project), is provided with [performance](https://gitlab.com/gitlab-org/quality/performance) project to import the Project tarball into a GitLab environment via API from the terminal. @@ -42,7 +43,7 @@ bin/import-project --help The process should take up to 15 minutes for the project to import fully. The script checks the status periodically and exits after the import has completed. -### Importing via GitHub +### Import by using GitHub There is also an option to [import the project via GitHub](../user/project/import/github.md): @@ -51,126 +52,12 @@ There is also an option to [import the project via GitHub](../user/project/impor This method takes longer to import than the other methods and depends on several factors. It's recommended to use the other methods. -### Importing via a Rake task - -> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. - -This script was introduced in GitLab 12.6 for importing large GitLab project exports. - -As part of this script we also disable direct upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts). - -We can run this script from the terminal: - -Parameters: - -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `username` | string | yes | User name | -| `namespace_path` | string | yes | Namespace path | -| `project_path` | string | yes | Project name | -| `archive_path` | string | yes | Path to the exported project tarball you want to import | - -```shell -bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" -``` - -If you're running Omnibus, run the following Rake task: - -```shell -gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" -``` - -#### Enable verbose output - -To make the import Rake task more verbose, use the `IMPORT_DEBUG` environment variable: - -```shell -IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" -``` - -#### Troubleshooting - -Check the common errors listed below, what they mean, and how to fix them. - -##### `Exception: undefined method 'name' for nil:NilClass` - -The `username` is not valid. - -##### `Exception: undefined method 'full_path' for nil:NilClass` - -The `namespace_path` does not exist. -For example, one of the groups or subgroups is mistyped or missing -or you've specified the project name in the path. - -The task only creates the project. -If you want to import it to a new group or subgroup then create it first. - -##### `Exception: No such file or directory @ rb_sysopen - (filename)` - -The specified project export file in `archive_path` is missing. - -##### `Exception: Permission denied @ rb_sysopen - (filename)` - -The specified project export file cannot be accessed by the `git` user. - -Setting the file owner to `git:git`, changing the file permissions to `0400`, and moving it to a -public folder (for example `/tmp/`) fixes the issue. - -##### `Name can contain only letters, digits, emojis ...` - -```plaintext -Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, -digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start -with '-', end in '.git', or end in '.atom'. -``` - -The project name specified in `project_path` is not valid for one of the specified reasons. - -Only put the project name in `project_path`. For example, if you provide a path of subgroups -it fails with this error as `/` is not a valid character in a project name. - -##### `Name has already been taken and Path has already been taken` - -A project with that name already exists. - -##### `Exception: Error importing repository into (namespace) - No space left on device` - -The disk has insufficient space to complete the import. - -During import, the tarball is cached in your configured `shared_path` directory. Verify the -disk has enough free space to accommodate both the cached tarball and the unpacked -project files on disk. - -##### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import - -If you receive a `Total number of not imported relations: XX` message, and issues -aren't created during the import, check [exceptions_json.log](../administration/logs/index.md#exceptions_jsonlog). -You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`, -where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you -are likely hitting the issue with rebalancing of `relative_position` field of the issues. - -```ruby -# Check the current maximum value of relative_position -Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) - -# Run the rebalancing process and check if the maximum value of relative_position has changed -Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute -Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) -``` - -Repeat the import attempt after that and check if the issues are imported successfully. - -##### Gitaly calls error when importing - -If you're attempting to import a large project into a development environment, you may see Gitaly throw an error about too many calls or invocations, for example: - -```plaintext -Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? -``` +### Import by using a Rake task -This is due to a [n+1 calls limit being set for development setups](gitaly.md#toomanyinvocationserror-errors). You can work around this by setting `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable, restarting your development environment and importing again. +To import the test project by using a Rake task, see +[Import large projects](../administration/raketasks/project_import_export.md#import-large-projects). -### Importing via the Rails console +### Import by using the Rails console The last option is to import a project using a Rails console: @@ -245,8 +132,9 @@ bundle exec rails r /path_to_script/script.rb project_name /path_to_extracted_p ## Access token setup -Many of the tests also require a GitLab Personal Access Token. This is due to numerous endpoints themselves requiring authentication. +Many of the tests also require a GitLab personal access token because numerous endpoints require authentication themselves. -[The official GitLab docs detail how to create this token](../user/profile/personal_access_tokens.md#create-a-personal-access-token). The tests require that the token is generated by an administrator and that it has the `API` and `read_repository` permissions. +[The GitLab documentation details how to create this token](../user/profile/personal_access_tokens.md#create-a-personal-access-token). +The tests require that the token is generated by an administrator and that it has the `API` and `read_repository` permissions. Details on how to use the Access Token with each type of test are found in their respective documentation. diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md deleted file mode 100644 index 4553ed2966f..00000000000 --- a/doc/development/integrations/codesandbox.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -stage: none -group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-01' ---- - -# Set up local CodeSandbox development environment (removed) - -WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108627) in GitLab 15.8 -and is planned for removal in 15.9. This change is a breaking change. - -This guide walks through setting up a local [CodeSandbox repository](https://github.com/codesandbox/codesandbox-client) and integrating it with a local GitLab instance. CodeSandbox -is used to power the Web IDE [Live Preview feature](../../user/project/web_ide/index.md#live-preview-removed). Having a local CodeSandbox setup is useful for debugging upstream issues or -creating upstream contributions like [this one](https://github.com/codesandbox/codesandbox-client/pull/5137). - -## Initial setup - -Before using CodeSandbox with your local GitLab instance, you must: - -1. Enable HTTPS on your GDK. CodeSandbox uses Service Workers that require `https`. - Follow the GDK [NGINX configuration instructions](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/nginx.md) to enable HTTPS for GDK. -1. Clone the [`codesandbox-client` project](https://github.com/codesandbox/codesandbox-client) - locally. If you plan on contributing upstream, you might want to fork and clone first. -1. Optional. Use correct `python` and `nodejs` versions. Otherwise, `yarn` may fail to - install or build some packages. If you're using `asdf` you can run the following commands: - - ```shell - asdf local nodejs 10.14.2 - asdf local python 2.7.18 - ``` - -1. Run the following commands in the `codesandbox-client` project checkout: - - ```shell - # This might be necessary for the `prepublishOnly` job that is run later - yarn global add lerna - - # Install packages - yarn - ``` - - You can run `yarn build:clean` to clean up the build assets. - -## Use local GitLab instance with local CodeSandbox - -GitLab integrates with two parts of CodeSandbox: - -- An npm package called `smooshpack` (called `sandpack` in the `codesandbox-client` project). - This exposes an entrypoint for us to kick off CodeSandbox's bundler. -- A server that houses CodeSandbox assets for bundling and previewing. This is hosted - on a separate server for security. - -Each time you want to run GitLab and CodeSandbox together, you need to perform the -steps in the following sections. - -### Use local `smooshpack` for GitLab - -GitLab usually satisfies its `smooshpack` dependency with a remote module, but we want -to use a locally-built module. To build and use a local `smooshpack` module: - -1. In the `codesandbox-client` project directory, run: - - ```shell - cd standalone-packages/sandpack - yarn link - - # (Optional) you might want to start a development build - yarn run start - ``` - - Now, in the GitLab project, you can run `yarn link "smooshpack"`. `yarn` looks - for `smooshpack` **on disk** as opposed to the one hosted remotely. - -1. In the `gitlab` project directory, run: - - ```shell - # Remove and reinstall node_modules just to be safe - rm -rf node_modules - yarn install - - # Use the "smooshpack" package on disk - yarn link "smooshpack" - ``` - -### Fix possible GDK webpack problem - -`webpack` in GDK can fail to find packages inside a linked package. This step can help -you avoid `webpack` breaking with messages saying that it can't resolve packages from -`smooshpack/dist/sandpack.es5.js`. - -In the `codesandbox-client` project directory, run: - -```shell -cd standalone-packages - -mkdir node_modules -ln -s $PATH_TO_LOCAL_GITLAB/node_modules/core-js ./node_modules/core-js -``` - -### Start building CodeSandbox app assets - -In the `codesandbox-client` project directory: - -```shell -cd packages/app - -yarn start:sandpack-sandbox -``` - -### Create HTTPS proxy for CodeSandbox `sandpack` assets - -Because we need `https`, we need to create a proxy to the webpack server. We can use -[`http-server`](https://www.npmjs.com/package/http-server), which can do this proxying -out of the box: - -```shell -npx http-server --proxy http://localhost:3000 -S -C $PATH_TO_CERT_PEM -K $PATH_TO_KEY_PEM -p 8044 -d false -``` - -### Update `bundler_url` setting in GitLab (removed) - -WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108627) in GitLab 15.8 -and is planned for removal in 15.9. This change is a breaking change. - -We need to update our `application_setting_implementation.rb` to point to the server that hosts the -CodeSandbox `sandpack` assets. For instance, if these assets are hosted by a server at `https://sandpack.local:8044`: - -```patch -diff --git a/app/models/application_setting_implementation.rb b/app/models/application_setting_implementation.rb -index 6eed627b502..1824669e881 100644 ---- a/app/models/application_setting_implementation.rb -+++ b/app/models/application_setting_implementation.rb -@@ -391,7 +391,7 @@ def static_objects_external_storage_enabled? - # This will eventually be configurable - # https://gitlab.com/gitlab-org/gitlab/-/issues/208161 - def web_ide_clientside_preview_bundler_url -- 'https://sandbox-prod.gitlab-static.net' -+ 'https://sandpack.local:8044' - end - - private - -``` - -NOTE: -You can apply this patch by copying it to your clipboard and running `pbpaste | git apply`. - -You may want to restart the GitLab Rails server after making this change: - -```shell -gdk restart rails-web -``` diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index ceb64ba2bb7..6f42cb4d4fe 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: "GitLab's development guidelines for Integrations" --- -# Integrations development guide +# Integrations development guidelines This page provides development guidelines for implementing [GitLab integrations](../../user/project/integrations/index.md), which are part of our [main Rails project](https://gitlab.com/gitlab-org/gitlab). @@ -315,7 +315,7 @@ When developing a new integration, we also recommend you gate the availability b You can provide help text in the integration form, including links to off-site documentation, as described above in [Customize the frontend form](#customize-the-frontend-form). Refer to -our [usability guidelines](https://design.gitlab.com/usability/helping-users/) for help text. +our [usability guidelines](https://design.gitlab.com/usability/contextual-help) for help text. For more detailed documentation, provide a page in `doc/user/project/integrations`, and link it from the [Integrations overview](../../user/project/integrations/index.md). diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index 6baccdca327..8b05cc3168e 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -26,9 +26,9 @@ GitLab does not allow requests to localhost or the local network by default. Whe 1. Log into your GitLab instance as an administrator. 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Network**. -1. Expand **Outbound requests** and check the following checkboxes: +1. Expand **Outbound requests**, and select the following checkboxes: - - **Allow requests to the local network from web hooks and services** + - **Allow requests to the local network from webhooks and integrations** - **Allow requests to the local network from system hooks** For more details about GitLab webhooks, see [Webhooks and insecure internal web services](../../security/webhooks.md). diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index eca4d9775c5..4789280c09d 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Set up a development environment +# Set up a Jira development environment The following are required to install and test the app: @@ -101,23 +101,23 @@ The following steps describe setting up an environment to test the GitLab OAuth 1. Start a Gitpod session and open the rails console. - ```shell - bundle exec rails console - ``` + ```shell + bundle exec rails console + ``` 1. Enable the feature flag. - ```shell - Feature.enable(:jira_connect_oauth) - ``` + ```shell + Feature.enable(:jira_connect_oauth) + ``` 1. On your GitLab instance, go to **Admin > Applications**. 1. Create a new application with the following settings: - - Name: `Jira Connect` - - Redirect URI: `YOUR_GITPOD_INSTANCE/-/jira_connect/oauth_callbacks` - - Scopes: `api` - - Trusted: **No** - - Confidential: **No** + - Name: `Jira Connect` + - Redirect URI: `YOUR_GITPOD_INSTANCE/-/jira_connect/oauth_callbacks` + - Scopes: `api` + - Trusted: **No** + - Confidential: **No** 1. Copy the Application ID. 1. Go to **Admin > Settings > General**. 1. Scroll down and expand the GitLab for Jira App section. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 002579d9b83..bc0a490f555 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -234,22 +234,13 @@ then `artifacts:reports:dependency_scanning` must be set to `depscan.json`. ### Exit code -Following the POSIX exit code standard, the scanner exits with 0 for success and any number from 1 to 255 for anything else. +Following the POSIX exit code standard, the scanner exits with either `0` for success or `1` for failure. Success also includes the case when vulnerabilities are found. When a CI job fails, security report results are not ingested by GitLab, even if the job -[allows failure](../../ci/yaml/index.md#allow_failure). The report artifacts are still uploaded to GitLab and available +[allows failure](../../ci/yaml/index.md#allow_failure). However, the report artifacts are still uploaded to GitLab and available for [download in the pipeline security tab](../../user/application_security/vulnerability_report/pipeline.md#download-security-scan-outputs). -When executing a scanning job using the [Docker-in-Docker privileged mode](../../user/application_security/sast/index.md#requirements), -we reserve the following standard exit codes. - -| Orchestrator Exit Code | Description | -|------------------------|----------------------------------| -| 3 | No match, no compatible analyzer | -| 4 | Project directory empty | -| 5 | No compatible Docker image | - ### Logging The scanner should log error messages and warnings so that users can easily investigate @@ -412,8 +403,6 @@ The `id` should not collide with any other analyzers or scanners another integra ##### Scan Primary Identifiers -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.4 [with a flag](../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. Disabled by default. - The `scan.primary_identifiers` field is an optional field containing an array of [primary identifiers](../../user/application_security/terminology/index.md#primary-identifier)). This is an exhaustive list of all rulesets for which the analyzer performed the scan. @@ -422,7 +411,7 @@ Even when the [`Vulnerabilities`](#vulnerabilities) array for a given scan may b should contain the complete list of potential identifiers to inform the Rails application of which rules were executed. -When populated, the Rails application automatically resolves previously detected vulnerabilities as no +When populated, the Rails application [may automatically resolve previously detected vulnerabilities](../../user/application_security/iac_scanning/index.md#automatic-vulnerability-resolution) as no longer relevant when their primary identifier is not included. ##### Name, message, and description diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index b19e431ebc6..aa10bbeda9c 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -37,13 +37,11 @@ is stored in a file at the path configured in `config/gitlab.yml` by default this is in the root of the rails app named `.gitlab_shell_secret` -To authenticate using that token, clients read the contents of that -file, and include the token Base64 encoded in a `secret_token` parameter -or in the `Gitlab-Shared-Secret` header. +To authenticate using that token, clients: -NOTE: -The internal API used by GitLab Pages, and GitLab agent server (`kas`) uses JSON Web Token (JWT) -authentication, which is different from GitLab Shell. +1. Read the contents of that file. +1. Use the file contents to generate a JSON Web Token (`JWT`). +1. Pass the JWT in the `Gitlab-Shell-Api-Request` header. ## Git Authentication @@ -78,7 +76,7 @@ POST /internal/allowed Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ "http://localhost:3001/api/v4/internal/allowed" ``` @@ -128,7 +126,7 @@ information for LFS clients when the repository is accessed over SSH. Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" ``` @@ -148,12 +146,12 @@ curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ ## Authorized Keys Check This endpoint is called by the GitLab Shell authorized keys -check. Which is called by OpenSSH for +check. Which is called by OpenSSH or GitLab SSHD for [fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md). | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `key` | string | yes | SSH key as passed by OpenSSH to GitLab Shell | +| `key` | string | yes | An authorized key used for public key authentication. | ```plaintext GET /internal/authorized_keys @@ -162,7 +160,7 @@ GET /internal/authorized_keys Example request: ```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>" +curl --request GET --header "Gitlab-Shell-Api-Request: <JWT token>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key>" ``` Example response: @@ -197,7 +195,7 @@ GET /internal/discover Example request: ```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7" +curl --request GET --header "Gitlab-Shell-Api-Request: <JWT token>" "http://localhost:3001/api/v4/internal/discover?key_id=7" ``` Example response: @@ -226,7 +224,7 @@ GET /internal/check Example request: ```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check" +curl --request GET --header "Gitlab-Shell-Api-Request: <JWT token>" "http://localhost:3001/api/v4/internal/check" ``` Example response: @@ -263,7 +261,7 @@ GET /internal/two_factor_recovery_codes Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" ``` @@ -311,7 +309,7 @@ POST /internal/personal_access_token Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ "http://localhost:3001/api/v4/internal/personal_access_token" ``` @@ -348,7 +346,7 @@ POST /internal/error_tracking/allowed Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "project_id=111&public_key=generated-error-tracking-key" \ "http://localhost:3001/api/v4/internal/error_tracking/allowed" ``` @@ -379,7 +377,7 @@ POST /internal/pre_receive Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" ``` @@ -412,7 +410,7 @@ POST /internal/post_receive Example Request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ +curl --request POST --header "Gitlab-Shell-Api-Request: <JWT token>" \ --data "gl_repository=project-7" --data "identifier=user-1" \ --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ "http://localhost:3001/api/v4/internal/post_receive" diff --git a/doc/development/issue_types.md b/doc/development/issue_types.md index 465f539e026..d0d513af781 100644 --- a/doc/development/issue_types.md +++ b/doc/development/issue_types.md @@ -4,7 +4,7 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Issue Types (DEPRECATED) +# Issue Types (deprecated) WARNING: We are deprecating Issue Types as of GitLab 14.2 in favor of [Work Items and Work Item Types](work_items.md). diff --git a/doc/development/json.md b/doc/development/json.md index 8a2575401fb..bdb7f73ab62 100644 --- a/doc/development/json.md +++ b/doc/development/json.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# JSON Guidelines +# JSON development guidelines At GitLab we handle a lot of JSON data. To best ensure we remain performant when handling large JSON encodes or decodes, we use our own JSON class diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index e44e2af4371..99737e71b22 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Kubernetes integration - development guidelines +# Kubernetes integration development guidelines This document provides various guidelines when developing for the GitLab [Kubernetes integration](../user/infrastructure/clusters/index.md). diff --git a/doc/development/labels/index.md b/doc/development/labels/index.md new file mode 100644 index 00000000000..af4df4adee2 --- /dev/null +++ b/doc/development/labels/index.md @@ -0,0 +1,348 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Labels + +To allow for asynchronous issue handling, we use [milestones](https://gitlab.com/groups/gitlab-org/-/milestones) +and [labels](https://gitlab.com/gitlab-org/gitlab/-/labels). Leads and product managers handle most of the +scheduling into milestones. Labeling is a task for everyone. (For some projects, labels can be set only by GitLab team members and not by community contributors). + +Most issues will have labels for at least one of the following: + +- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::maintenance"`. +- Stage. For example: `~"devops::plan"` or `~"devops::create"`. +- Group. For example: `~"group::source code"`, `~"group::knowledge"`, or `~"group::editor"`. +- Category. For example: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, or `~"Category:Templates"`. +- Feature. For example: `~wiki`, `~ldap`, `~api`, `~issues`, or `~"merge requests"`. +- Department: `~UX`, `~Quality` +- Team: `~"Technical Writing"`, `~Delivery` +- Specialization: `~frontend`, `~backend`, `~documentation` +- Release Scoping: `~Deliverable`, `~Stretch`, `~"Next Patch Release"` +- Priority: `~"priority::1"`, `~"priority::2"`, `~"priority::3"`, `~"priority::4"` +- Severity: `~"severity::1"`, `~"severity::2"`, `~"severity::3"`, `~"severity::4"` + +Please add `~"breaking change"` label if the issue can be considered as a [breaking change](../deprecation_guidelines/index.md). + +Please add `~security` label if the issue is related to application security. + +All labels, their meaning and priority are defined on the +[labels page](https://gitlab.com/gitlab-org/gitlab/-/labels). + +If you come across an issue that has none of these, and you're allowed to set +labels, you can _always_ add the type, stage, group, and often the category/feature labels. + +## Type labels + +Type labels are very important. They define what kind of issue this is. Every +issue should have one and only one. + +The SSOT for type and subtype labels is [available in the handbook](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification). + +A number of type labels have a priority assigned to them, which automatically +makes them float to the top, depending on their importance. + +Type labels are always lowercase, and can have any color, besides blue (which is +already reserved for category labels). + +The descriptions on the [labels page](https://gitlab.com/groups/gitlab-org/-/labels) +explain what falls under each type label. + +The GitLab handbook documents [when something is a bug](https://about.gitlab.com/handbook/product/product-processes/#bug-issues) and [when it is a feature request](https://about.gitlab.com/handbook/product/product-processes/#feature-issues). + +## Stage labels + +Stage labels specify which [stage](https://about.gitlab.com/handbook/product/categories/#hierarchy) the issue belongs to. + +### Naming and color convention + +Stage labels respects the `devops::<stage_key>` naming convention. +`<stage_key>` is the stage key as it is in the single source of truth for stages at +<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml> +with `_` replaced with a space. + +For instance, the "Manage" stage is represented by the `~"devops::manage"` label in +the `gitlab-org` group since its key under `stages` is `manage`. + +The current stage labels can be found by [searching the labels list for `devops::`](https://gitlab.com/groups/gitlab-org/-/labels?search=devops::). + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels) +and thus are mutually exclusive. + +The Stage labels are used to generate the [direction pages](https://about.gitlab.com/direction/) automatically. + +## Group labels + +Group labels specify which [groups](https://about.gitlab.com/company/team/structure/#product-groups) the issue belongs to. + +It's highly recommended to add a group label, as it's used by our triage +automation to +[infer the correct stage label](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues-and-merge-requests). + +### Naming and color convention + +Group labels respects the `group::<group_key>` naming convention and +their color is `#A8D695`. +`<group_key>` is the group key as it is in the single source of truth for groups at +<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml>, +with `_` replaced with a space. + +For instance, the "Pipeline Execution" group is represented by the +`~"group::pipeline execution"` label in the `gitlab-org` group since its key +under `stages.manage.groups` is `pipeline_execution`. + +The current group labels can be found by [searching the labels list for `group::`](https://gitlab.com/groups/gitlab-org/-/labels?search=group::). + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels) +and thus are mutually exclusive. + +You can find the groups listed in the [Product Stages, Groups, and Categories](https://about.gitlab.com/handbook/product/categories/) page. + +We use the term group to map down product requirements from our product stages. +As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. + +## Category labels + +From the handbook's +[Product stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/#hierarchy) +page: + +> Categories are high-level capabilities that may be a standalone product at +another company, such as Portfolio Management, for example. + +It's highly recommended to add a category label, as it's used by our triage +automation to +[infer the correct group and stage labels](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues). + +If you are an expert in a particular area, it makes it easier to find issues to +work on. You can also subscribe to those labels to receive an email each time an +issue is labeled with a category label corresponding to your expertise. + +### Naming and color convention + +Category labels respects the `Category:<Category Name>` naming convention and +their color is `#428BCA`. +`<Category Name>` is the category name as it is in the single source of truth for categories at +<https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. + +For instance, the "DevOps Reports" category is represented by the +`~"Category:DevOps Reports"` label in the `gitlab-org` group since its +`devops_reports.name` value is "DevOps Reports". + +If a category's label doesn't respect this naming convention, it should be specified +with [the `label` attribute](https://about.gitlab.com/handbook/marketing/digital-experience/website/#category-attributes) +in <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. + +## Feature labels + +From the handbook's +[Product stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/#hierarchy) +page: + +> Features: Small, discrete functionalities, for example Issue weights. Some common +features are listed within parentheses to facilitate finding responsible PMs by keyword. + +It's highly recommended to add a feature label if no category label applies, as +it's used by our triage automation to +[infer the correct group and stage labels](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-labelling-of-issues). + +If you are an expert in a particular area, it makes it easier to find issues to +work on. You can also subscribe to those labels to receive an email each time an +issue is labeled with a feature label corresponding to your expertise. + +Examples of feature labels are `~wiki`, `~ldap`, `~api`, `~issues`, and `~"merge requests"`. + +### Naming and color convention + +Feature labels are all-lowercase. + +## Workflow labels + +Issues use the following workflow labels to specify the current issue status: + +- `~"workflow::awaiting security release"` +- `~"workflow::blocked"` +- `~"workflow::complete"` +- `~"workflow::design"` +- `~"workflow::feature-flagged"` +- `~"workflow::in dev"` +- `~"workflow::in review"` +- `~"workflow::planning breakdown"` +- `~"workflow::problem validation"` +- `~"workflow::production"` +- `~"workflow::ready for design"` +- `~"workflow::ready for development"` +- `~"workflow::refinement"` +- `~"workflow::scheduling"` +- `~"workflow::solution validation"` +- `~"workflow::start"` +- `~"workflow::validation backlog"` +- `~"workflow::verification"` + +## Facet labels + +To track additional information or context about created issues, developers may +add _facet labels_. Facet labels are also sometimes used for issue prioritization +or for measurements (such as time to close). An example of a facet label is the +`~"customer"` label, which indicates customer interest. + +## Department labels + +The current department labels are: + +- `~"UX"` +- `~"Quality"` + +## Team labels + +**Important**: Most of the historical team labels (like Manage or Plan) are +now deprecated in favor of [Group labels](#group-labels) and [Stage labels](#stage-labels). + +Team labels specify what team is responsible for this issue. +Assigning a team label makes sure issues get the attention of the appropriate +people. + +The current team labels are: + +- `~"Delivery"~` +- `~"Technical Writing"` + +### Naming and color convention + +Team labels are always capitalized so that they show up as the first label for +any issue. + +## Specialization labels + +These labels narrow the [specialization](https://about.gitlab.com/company/team/structure/#specialist) on a unit of work. + +- `~"frontend"` +- `~"backend"` +- `~"documentation"` + +## Release scoping labels + +Release Scoping labels help us clearly communicate expectations of the work for the +release. There are three levels of Release Scoping labels: + +- `~"Deliverable"`: Issues that are expected to be delivered in the current + milestone. +- `~"Stretch"`: Issues that are a stretch goal for delivering in the current + milestone. If these issues are not done in the current release, they will + strongly be considered for the next release. +- `~"Next Patch Release"`: Issues to put in the next patch release. Work on these + first, and add the `~"Pick into X.Y"` label to the merge request, along with the + appropriate milestone. + +Each issue scheduled for the current milestone should be labeled `~"Deliverable"~` +or `~"Stretch"`. Any open issue for a previous milestone should be labeled +`~"Next Patch Release"`, or otherwise rescheduled to a different milestone. + +## Priority labels + +We have the following priority labels: + +- `~"priority::1"` +- `~"priority::2"` +- `~"priority::3"` +- `~"priority::4"` + +Please refer to the issue triage [priority label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#priority) section in our handbook to see how it's used. + +## Severity labels + +We have the following severity labels: + +- `~"severity::1"` +- `~"severity::2"` +- `~"severity::3"` +- `~"severity::4"` + +Please refer to the issue triage [severity label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity) section in our handbook to see how it's used. + +## Label for community contributors + +There are many issues that have a clear solution with uncontroversial benefit to GitLab users. +However, GitLab might not have the capacity for all these proposals in the current roadmap. +These issues are labeled `~"Seeking community contributions"` because we welcome merge requests to resolve them. + +Community contributors can submit merge requests for any issue they want, but +the `~"Seeking community contributions"` label has a special meaning. It points to +changes that: + +1. We already agreed on, +1. Are well-defined, +1. Are likely to get accepted by a maintainer. + +We want to avoid a situation when a contributor picks an +~"Seeking community contributions" issue and then their merge request gets closed, +because we realize that it does not fit our vision, or we want to solve it in a +different way. + +We manually add the `~"Seeking community contributions"` label to issues +that fit the criteria described above. +We do not automatically add this label, because it requires human evaluation. + +We recommend people that have never contributed to any open source project to +look for issues labeled `~"Seeking community contributions"` with a +[weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None&weight=1) or the `~"quick win"` +[label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=quick%20win&assignee_id=None) +attached to it. +More experienced contributors are very welcome to tackle +[any of them](https://gitlab.com/groups/gitlab-org/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&assignee_id=None). + +For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues +with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=Seeking+community+contributions&label_name[]=Community+challenge). +If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom +GitLab merchandise. + +If you've decided that you would like to work on an issue, please @-mention +the [appropriate product manager](https://about.gitlab.com/handbook/product/#who-to-talk-to-for-what) +as soon as possible. The product manager will then pull in appropriate GitLab team +members to further discuss scope, design, and technical considerations. This will +ensure that your contribution is aligned with the GitLab product and minimize +any rework and delay in getting it merged into main. + +GitLab team members who apply the `~"Seeking community contributions"` label to an issue +should update the issue description with a responsible product manager, inviting +any potential community contributor to @-mention per above. + +## Stewardship label + +For issues related to the open source stewardship of GitLab, +there is the `~"stewardship"` label. + +This label is to be used for issues in which the stewardship of GitLab +is a topic of discussion. For instance if GitLab Inc. is planning to add +features from GitLab EE to GitLab CE, related issues would be labeled with +`~"stewardship"`. + +A recent example of this was the issue for +[bringing the time tracking API to GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/25517#note_20019084). + +## Technical and UX debt + +In order to track things that can be improved in the GitLab codebase, +we use the `~"technical debt"` label in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). +We use the `~"UX debt"` label when we choose to deviate from the MVC, in a way that harms the user experience. + +These labels should be added to issues that describe things that can be improved, +shortcuts that have been taken, features that need additional attention, and all +other things that have been left behind due to high velocity of development. +For example, code that needs refactoring should use the `~"technical debt"` label, +something that didn't ship according to our Design System guidelines should +use the `~"UX debt"` label. + +Everyone can create an issue, though you may need to ask for adding a specific +label, if you do not have permissions to do it by yourself. Additional labels +can be combined with these labels, to make it easier to schedule +the improvements for a release. + +Issues tagged with these labels have the same priority like issues +that describe a new feature to be introduced in GitLab, and should be scheduled +for a release by the appropriate person. + +Make sure to mention the merge request that the `~"technical debt"` issue or +`~"UX debt"` issue is associated with in the description of the issue. diff --git a/doc/development/lfs.md b/doc/development/lfs.md index ec91f9f3c8b..8f6d54e08e3 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -4,14 +4,14 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Git LFS developer information +# Git LFS development guidelines This page contains developer-centric information for GitLab team members. For the user documentation, see [Git Large File Storage](../topics/git/lfs/index.md). ## Deep Dive -In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) +In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`) on the GitLab [Git LFS](../topics/git/lfs/index.md) implementation to share domain-specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=Yyxwcksr0Qc), diff --git a/doc/development/logging.md b/doc/development/logging.md index 538fc7ccfe1..6840445a0a5 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Developers Guide to Logging +# Logging development guidelines [GitLab Logs](../administration/logs/index.md) play a critical role for both administrators and GitLab team members to diagnose problems in the field. @@ -169,6 +169,28 @@ Resources: - [Elasticsearch mapping - avoiding type gotchas](https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html#_avoiding_type_gotchas) - [Elasticsearch mapping types]( https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) +#### Include a class attribute + +Structured logs should always include a `class` attribute to make all entries logged from a particular place in the code findable. +To automatically add the `class` attribute, you can include the +[`Gitlab::Loggable` module](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/loggable.rb) and use the `build_structured_payload` method. + +```ruby +class MyClass + include ::Gitlab::Loggable + + def my_method + logger.info(build_structured_payload(message: 'log message', project_id: project_id)) + end + + private + + def logger + @logger ||= Gitlab::AppJsonLogger.build + end +end +``` + #### Logging durations Similar to timezones, choosing the right time unit to log can impose avoidable overhead. So, whenever diff --git a/doc/development/merge_request_concepts/approval_rules.md b/doc/development/merge_request_concepts/approval_rules.md index d119644cd7c..d6000d48706 100644 --- a/doc/development/merge_request_concepts/approval_rules.md +++ b/doc/development/merge_request_concepts/approval_rules.md @@ -4,7 +4,7 @@ group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Approval Rules development guide +# Approval rules development guidelines This document explains the backend design and flow of all related functionality about [merge request approval rules](../../user/project/merge_requests/approvals/index.md). diff --git a/doc/development/merge_request_concepts/diffs/index.md b/doc/development/merge_request_concepts/diffs/index.md index 8ef3f01aba9..c2dec5c4753 100644 --- a/doc/development/merge_request_concepts/diffs/index.md +++ b/doc/development/merge_request_concepts/diffs/index.md @@ -17,7 +17,7 @@ We rely on different sources to present diffs. These include: <!-- vale gitlab.Spelling = NO --> In January 2019, Oswaldo Ferreira hosted a Deep Dive (GitLab team members only: -`https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab Diffs and Commenting on Diffs +`https://gitlab.com/gitlab-org/create-stage/-/issues/1`) on GitLab Diffs and Commenting on Diffs functionality to share domain-specific knowledge with anyone who may work in this part of the codebase in the future: diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md index 14d9582ad84..8e9b586d5b0 100644 --- a/doc/development/merge_request_concepts/index.md +++ b/doc/development/merge_request_concepts/index.md @@ -34,7 +34,7 @@ This area of the merge request is where all of the options and commit messages a Reports are widgets within the merge request that report information about changes within the merge request. These widgets provide information to better help the author understand the changes and further improvements to the proposed changes. -[Design Documentation](https://design.gitlab.com/regions/merge-request-reports/) +[Design Documentation](https://design.gitlab.com/patterns/merge-request-reports)  diff --git a/doc/development/merge_request_concepts/performance.md b/doc/development/merge_request_concepts/performance.md index 740b8f1607b..3b2a097ea2d 100644 --- a/doc/development/merge_request_concepts/performance.md +++ b/doc/development/merge_request_concepts/performance.md @@ -260,7 +260,7 @@ It re-instantiates project object for each build, instead of using the same in-m In this particular case the workaround is fairly easy: ```ruby -ActiveRecord::Associations::Preloader.new.preload(pipeline, [builds: :project]) +ActiveRecord::Associations::Preloader.new(records: pipeline, associations: [builds: :project]).call pipeline.builds.each do |build| build.to_json(only: [:name], include: [project: { only: [:name]}]) diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 4625489146e..9f2ad1f7769 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Data Stores +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -307,7 +307,7 @@ which is a "versioned" class. For new migrations, the latest version should be u can be looked up in `Gitlab::Database::Migration::MIGRATION_CLASSES`) to use the latest version of migration helpers. -In this example, we use version 2.0 of the migration class: +In this example, we use version 2.1 of the migration class: ```ruby class TestMigration < Gitlab::Database::Migration[2.1] @@ -323,7 +323,7 @@ version of migration helpers automatically. Migration helpers and versioning were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68986) in GitLab 14.3. For merge requests targeting previous stable branches, use the old format and still inherit from -`ActiveRecord::Migration[6.1]` instead of `Gitlab::Database::Migration[2.0]`. +`ActiveRecord::Migration[6.1]` instead of `Gitlab::Database::Migration[2.1]`. ## Retry mechanism when acquiring database locks @@ -345,7 +345,7 @@ on the `users` table once it has been enqueued. More information about PostgreSQL locks: [Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html) -For stability reasons, GitLab.com has a specific [`statement_timeout`](../user/gitlab_com/index.md#postgresql) +For stability reasons, GitLab.com has a short `statement_timeout` set. When the migration is invoked, any database query has a fixed time to execute. In a worst-case scenario, the request sits in the lock queue, blocking other queries for the duration of the configured statement timeout, diff --git a/doc/development/navigation_sidebar.md b/doc/development/navigation_sidebar.md new file mode 100644 index 00000000000..495f30a796c --- /dev/null +++ b/doc/development/navigation_sidebar.md @@ -0,0 +1,38 @@ +--- +stage: Manage +group: Foundations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Navigation sidebar + +Follow these guidelines when contributing additions or changes to the +[redesigned](https://gitlab.com/groups/gitlab-org/-/epics/9044) navigation +sidebar. + +These guidelines reflect the current state of the navigation sidebar. However, +the sidebar is a work in progress, and so is this documentation. + +## Enable the new navigation sidebar + +To enable the new navigation sidebar: + +- Enable the `super_sidebar_nav` feature flag. +- Select your avatar, then turn on the **New navigation** toggle. + +## Adding page-specific Vue content + +Pages can render arbitrary content into the sidebar using the `SidebarPortal` +component. Content passed to its default slot is rendered below that +page's navigation items in the sidebar. + +NOTE: +Only one instance of this component on a given page is supported. This is to +avoid ordering issues and cluttering the sidebar. + +NOTE: +Arbitrary content is allowed, but nav items should be implemented by +subclassing `::Sidebars::Panel`. + +NOTE: +Do not use the `SidebarPortalTarget` component. It is internal to the sidebar. diff --git a/doc/development/organization/index.md b/doc/development/organization/index.md new file mode 100644 index 00000000000..6a643603170 --- /dev/null +++ b/doc/development/organization/index.md @@ -0,0 +1,159 @@ +--- +comments: false +type: index, dev +stage: Data Stores +group: Tenant Scale +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" +description: "Development Guidelines: learn about organization when developing GitLab." +--- + +# Organization + +The [Organization initiative](../../user/organization/index.md) focuses on reaching feature parity between +SaaS and self-managed installations. + +## Consolidate groups and projects + +- [Architecture blueprint](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + +One facet of the Organization initiative is to consolidate groups and projects, +addressing the feature disparity between them. Some features, such as epics, are +only available at the group level. Some features, such as issues, are only available +at the project level. Other features, such as milestones, are available to both groups +and projects. + +We receive many requests to add features either to the group or project level. +Moving features around to different levels is problematic on multiple levels: + +- It requires engineering time to move the features. +- It requires UX overhead to maintain mental models of feature availability. +- It creates redundant code. + +When features are copied from one level (project, group, or instance) to another, +the copies often have small, nuanced differences between them. These nuances cause +extra engineering time when fixes are needed, because the fix must be copied to +several locations. These nuances also create different user experiences when the +feature is used in different places. + +A solution for this problem is to consolidate groups and projects into a single +entity, `namespace`. The work on this solution is split into several phases and +is tracked in [epic 6473](https://gitlab.com/groups/gitlab-org/-/epics/6473). + +### Phase 1 + +- [Phase 1 epic](https://gitlab.com/groups/gitlab-org/-/epics/6697). +- **Goals**: + 1. Ensure every project receives a corresponding record in the `namespaces` + table with `type='Project'`. + 1. For user namespaces, the type changes from `NULL` to `User`. + +We should make sure that projects, and the project namespace, are equivalent: + +- **Create project:** use Rails callbacks to ensure a new project namespace is + created for each project. Project namespace records should contain `created_at` and + `updated_at` attributes equal to the project's `created_at`/`updated_at` attributes. +- **Update project:** use the `after_save` callback in Rails to ensure some + attributes are kept in sync between project and project namespaces. + Read [`project#after_save`](https://gitlab.com/gitlab-org/gitlab/blob/6d26634e864d7b748dda0e283eb2477362263bc3/app/models/project.rb#L101-L101) + for more information. +- **Delete project:** use FKs cascade delete or Rails callbacks to ensure when a `Project` + or its `ProjectNamespace` is removed, its corresponding `ProjectNamespace` or `Project` + is also removed. +- **Transfer project to a different group:** make sure that when a project is transferred, + its corresponding project namespace is transferred to the same group. +- **Transfer group:** make sure when transferring a group that all of its sub-projects, + either direct or through descendant groups, have their corresponding project + namespaces transferred correctly as well. +- **Export or import project** + - **Export project** continues to export only the project, and not its project namespace, + in this phase. The project namespace does not contain any specific information + to export at this point. Eventually we want the project namespace to be exported as well. + - **Import project** creates a new project, so the project namespace is created through + Rails `after_save` callback on the project model. +- **Export or import group:** when importing or exporting a `Group`, projects are not + included in the operation. If that feature is changed to include `Project` when its group is + imported or exported, the logic must include their corresponding project namespaces + in the import or export. + +After ensuring these points, run a database migration to create a `ProjectNamespace` +record for every `Project`. Project namespace records created during the migration +should have `created_at` and `updated_at` attributes set to the migration runtime. +The project namespaces' `created_at` and `updated_at` attributes would not match +their corresponding project's `created_at` and `updated_at` attributes. We want +the different dates to help audit any of the created project namespaces, in case we need it. +After this work completes, we must migrate data as described in +[Backfill `ProjectNamespace` for every Project](https://gitlab.com/gitlab-org/gitlab/-/issues/337100). + +### Phase 2 + +- [Phase 2 epic](https://gitlab.com/groups/gitlab-org/-/epics/6768). +- **Goal**: Link `ProjectNamespace` to other entities on the database level. + +In this phase: + +- Communicate the changes company-wide at the engineering level. We want to make + engineers aware of the upcoming changes, even though teams are not expected to + collaborate actively until phase 3. +- Raise awareness to avoid regressions, and conflicting or duplicate work that + can be dealt with before phase 3. + +### Phase 3 + +- [Phase 3 epic](https://gitlab.com/groups/gitlab-org/-/epics/6585). +- **Goal**: Achieve feature parity between the namespace types. +Problems to solve as part of this phase: + +- Routes handling through `ProjectNamespace` rather than `Project`. +- Memberships handling. +- Policies handling. +- Import and export. +- Other interactions between project namespace and project models. + +Phase 3 is when the active migration of features from `Project` to `ProjectNamespace`, +or directly to `Namespace`, happens. + +### How to plan features that interact with Group and ProjectNamespace + +As of now, every Project in the system has a record in the `namespaces` table. This makes it possible to +use common interface to create features that are shared between Groups and Projects. Shared behavior can be added using +a concerns mechanism. Because the `Namespace` model is responsible for `UserNamespace` methods as well, it is discouraged +to use the `Namespace` model for shared behavior for Projects and Groups. + +#### Resource-based features + +To migrate resource-based features, existing functionality will need to be supported. This can be achieved in two Phases. + +**Phase 1 - Setup** + +- Link into the namespaces table + - Add a column to the table + - For example, in issues a `project id` points to the projects table. We need to establish a link to the `namespaces` table. + - Modify code so that any new record already has the correct data in it + - Backfill + +**Phase 2 - Prerequisite work** + +- Investigate the permission model as well as any performance concerns related to that. + - Permissions need to be checked and kept in place. +- Investigate what other models need to support namespaces for functionality dependent on features you migrate in Phase 1. +- Adjust CRUD services and APIs (REST and GraphQL) to point to the new column you added in Phase 1. +- Consider performance when fetching resources. + +Introducing new functionality is very much dependent on every single team and feature. + +#### Settings-related features + +Right now, cascading settings are available for `NamespaceSettings`. By creating `ProjectNamespace`, +we can use this framework to make sure that some settings are applicable on the project level as well. + +When working on settings, we need to make sure that: + +- They are not used in `join` queries or modify those queries. +- Updating settings is taken into consideration. +- If we want to move from project to project namespace, we follow a similar database process to the one described in [Phase 1](#phase-1). + +## Related topics + +- [Consolidating groups and projects](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + architecture documentation +- [Organization user documentation](../../user/organization/index.md) diff --git a/doc/development/packages/debian_repository.md b/doc/development/packages/debian_repository.md index 26a33c548d8..1523d777e7b 100644 --- a/doc/development/packages/debian_repository.md +++ b/doc/development/packages/debian_repository.md @@ -116,7 +116,7 @@ sequenceDiagram ProcessChangesService->>+ExtractChangesMetadataService: Extract changesmetadata ExtractChangesMetadataService->>+ExtractMetadataService: Extract file metadata ExtractMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file - ExtractMetadataService->>+ExtractDebMetadataService: If .deb or .udeb + ExtractMetadataService->>+ExtractDebMetadataService: If .deb, .udeb or ddeb ExtractDebMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file ParseDebian822Service-->>-ExtractDebMetadataService: Parse String as Debian RFC822 control data format ExtractDebMetadataService-->>-ExtractMetadataService: Return the parsed control file @@ -127,7 +127,7 @@ sequenceDiagram loop process files listed in .changes ProcessChangesService->>+ExtractMetadataService: Process file ExtractMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file - ExtractMetadataService->>+ExtractDebMetadataService: If .deb or .udeb + ExtractMetadataService->>+ExtractDebMetadataService: If .deb, .udeb or ddeb ExtractDebMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file ParseDebian822Service-->>-ExtractDebMetadataService: Parse String as Debian RFC822 control data format ExtractDebMetadataService-->>-ExtractMetadataService: Return the parsed control file diff --git a/doc/development/packages/index.md b/doc/development/packages/index.md index fa0e9f5d926..4f027825422 100644 --- a/doc/development/packages/index.md +++ b/doc/development/packages/index.md @@ -4,7 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Package and container registry documentation +# Package and container registry development guidelines The documentation for package and container registry development is split into two groups. diff --git a/doc/development/permissions.md b/doc/development/permissions.md index bf7f99de1ab..3abadc98501 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -4,7 +4,7 @@ group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Implementing permissions +# Permission development guidelines There are multiple types of permissions across GitLab, and when implementing anything that deals with permissions, all of them should be considered. @@ -60,7 +60,7 @@ Additionally, the following project features can have different visibility level - Pipelines - Analytics - Requirements -- Security & Compliance +- Security and Compliance - Wiki - Snippets - Pages @@ -198,3 +198,124 @@ Say, for example, we extract the whole endpoint into a service. The `can?` check - If the finder doesn't accept `current_user`, and therefore doesn't check permissions, then probably no. - If the finder accepts `current_user`, and doesn't check permissions, then it would be a good idea to double check other usages of the finder, and we might consider adding authorization. - If the finder accepts `current_user`, and already checks permissions, then either we need to add our case, or the existing checks are appropriate. + +### Refactoring permissions + +#### Finding existing permissions checks + +As mentioned [above](#where-should-permissions-be-checked), permissions are +often checked in multiple locations for a single endpoint or web request. As a +result, finding the list of authorization checks that are run for a given endpoint +can be challenging. + +To assist with this, you can locally set `GITLAB_DEBUG_POLICIES=true`. + +This outputs information about which abilities are checked in the requests +made in any specs that you run. The output also includes the line of code where the +authorization check was made. Caller information is especially helpful in cases +where there is metaprogramming used because those cases are difficult to find by +grepping for ability name strings. + +Example: + +```shell +# example spec run + +GITLAB_DEBUG_POLICIES=true bundle exec rspec spec/controllers/groups_controller_spec.rb:162 + +# permissions debug output when spec is run; if multiple policy checks are run they will all be in the debug output. + +POLICY CHECK DEBUG -> policy: GlobalPolicy, ability: create_group, called_from: ["/gitlab/app/controllers/application_controller.rb:245:in `can?'", "/gitlab/app/controllers/groups_controller.rb:255:in `authorize_create_group!'"] +``` + +This flag is meant to help learn more about authorization checks while +refactoring and should not remain enabled for any specs on the default branch. + +#### Understanding logic for individual abilities + +References to an ability may appear in a `DeclarativePolicy` class many times +and depend on conditions and rules which reference other abilities. As a result, +it can be challenging to know exactly which conditions apply to a particular +ability. + +`DeclarativePolicy` provides a `ability_map` for each Policy class, which +pulls all Rules for an ability into an array. + +Example: + +```ruby +> GroupPolicy.ability_map.map.select { |k,v| k == :read_group_member } +=> {:read_group_member=>[[:enable, #<Rule can?(:read_group)>], [:prevent, #<Rule ~can_read_group_member>]]} + +> GroupPolicy.ability_map.map.select { |k,v| k == :read_group } +=> {:read_group=> + [[:enable, #<Rule public_group>], + [:enable, #<Rule logged_in_viewable>], + [:enable, #<Rule guest>], + [:enable, #<Rule admin>], + [:enable, #<Rule has_projects>], + [:enable, #<Rule read_package_registry_deploy_token>], + [:enable, #<Rule write_package_registry_deploy_token>], + [:prevent, #<Rule all?(~public_group, ~admin, user_banned_from_group)>], + [:enable, #<Rule auditor>], + [:prevent, #<Rule needs_new_sso_session>], + [:prevent, #<Rule all?(ip_enforcement_prevents_access, ~owner, ~auditor)>]]} +``` + +`DeclarativePolicy` also provides a `debug` method that can be used to +understand the logic tree for a specific object and actor. The output is similar +to the list of rules from `ability_map`. But, `DeclarativePolicy` stops +evaluating rules once one `prevent`s an ability, so it is possible that +not all conditions are called. + +Example: + +```ruby +policy = GroupPolicy.new(User.last, Group.last) +policy.debug(:read_group) + +- [0] enable when public_group ((@custom_guest_user1 : Group/139)) +- [0] enable when logged_in_viewable ((@custom_guest_user1 : Group/139)) +- [0] enable when admin ((@custom_guest_user1 : Group/139)) +- [0] enable when auditor ((@custom_guest_user1 : Group/139)) +- [14] prevent when all?(~public_group, ~admin, user_banned_from_group) ((@custom_guest_user1 : Group/139)) +- [14] prevent when needs_new_sso_session ((@custom_guest_user1 : Group/139)) +- [16] enable when guest ((@custom_guest_user1 : Group/139)) +- [16] enable when has_projects ((@custom_guest_user1 : Group/139)) +- [16] enable when read_package_registry_deploy_token ((@custom_guest_user1 : Group/139)) +- [16] enable when write_package_registry_deploy_token ((@custom_guest_user1 : Group/139)) + [21] prevent when all?(ip_enforcement_prevents_access, ~owner, ~auditor) ((@custom_guest_user1 : Group/139)) + +=> #<DeclarativePolicy::Runner::State:0x000000015c665050 + @called_conditions= + #<Set: { + "/dp/condition/GroupPolicy/public_group/Group:139", + "/dp/condition/GroupPolicy/logged_in_viewable/User:83,Group:139", + "/dp/condition/BasePolicy/admin/User:83", + "/dp/condition/BasePolicy/auditor/User:83", + "/dp/condition/GroupPolicy/user_banned_from_group/User:83,Group:139", + "/dp/condition/GroupPolicy/needs_new_sso_session/User:83,Group:139", + "/dp/condition/GroupPolicy/guest/User:83,Group:139", + "/dp/condition/GroupPolicy/has_projects/User:83,Group:139", + "/dp/condition/GroupPolicy/read_package_registry_deploy_token/User:83,Group:139", + "/dp/condition/GroupPolicy/write_package_registry_deploy_token/User:83,Group:139"}>, + @enabled=false, + @prevented=true> +``` + +#### Testing that individual policies are equivalent + +You can use the `'equivalent project policy abilities'` shared example to ensure +that 2 project policy abilities are equivalent for all project visibility levels +and access levels. + +Example: + +```ruby + context 'when refactoring read_pipeline_schedule and read_pipeline' do + let(:old_policy) { :read_pipeline_schedule } + let(:new_policy) { :read_pipeline } + + it_behaves_like 'equivalent policies' + end +``` diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index 240d98a855f..51cdd12a8c1 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -100,7 +100,7 @@ The `rules` definitions for full Jest tests are defined at `.frontend:rules:jest We run only the predictive RSpec & Jest jobs for fork pipelines, unless the `pipeline:run-all-rspec` label is set on the MR. The goal is to reduce the CI/CD minutes consumed by fork pipelines. -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170). +See the [experiment issue](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1170). ## Fail-fast job in merge request pipelines @@ -392,6 +392,8 @@ to detect, and fail if any changes introduced in the merge request has zero cove The `rspec:undercoverage` job obtains coverage data from the `rspec:coverage` job. +If the `rspec:undercoverage` job detects missing coverage due to a CE method being overridden in EE, add the `pipeline:run-as-if-foss` label to the merge request and start a new pipeline. + In the event of an emergency, or false positive from this job, add the `pipeline:skip-undercoverage` label to the merge request to allow this job to fail. @@ -442,7 +444,7 @@ so that they're ultimately fixed by their respective group. The automatic skipping of flaky tests can still be enabled by setting the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable to `true`. -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069). +See the [experiment issue](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1069). ### Automatic retry of failing tests in a separate process @@ -451,7 +453,7 @@ RSpec process. The goal is to get rid of most side-effects from previous tests t We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job. -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148). +See the [experiment issue](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1148). ## Compatibility testing @@ -484,19 +486,17 @@ This should let us: Our test suite runs against PG12 as GitLab.com runs on PG12 and [Omnibus defaults to PG12 for new installs and upgrades](../../administration/package_information/postgresql_versions.md). -We do run our test suite against PG11 and PG13 on nightly scheduled pipelines. - -We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). +We do run our test suite against PG13 on nightly scheduled pipelines. #### Current versions testing -| Where? | PostgreSQL version | Ruby version | -|------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------| -| Merge requests | 12 (default version), 11 for DB library changes | 3.0 (default version) | -| `master` branch commits | 12 (default version), 11 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version), 11 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `ruby2` branch (every odd-numbered hour), see below. | 12 (default version), 11 for DB library changes | 2.7 | -| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 11, 13 | 3.0 (default version) | +| Where? | PostgreSQL version | Ruby version | +|------------------------------------------------------------------------------------------------|--------------------------|-----------------------| +| Merge requests | 12 (default version) | 3.0 (default version) | +| `master` branch commits | 12 (default version) | 3.0 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version) | 3.0 (default version) | +| `maintenance` scheduled pipelines for the `ruby2` branch (every odd-numbered hour), see below. | 12 (default version) | 2.7 | +| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 13 | 3.0 (default version) | There are 2 pipeline schedules used for testing Ruby 2.7. One is triggering a pipeline in `ruby2-sync` branch, which updates the `ruby2` branch with latest @@ -518,7 +518,6 @@ We follow the [PostgreSQL versions shipped with Omnibus GitLab](../../administra | PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | | -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | | PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | -| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | | PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | ### Redis versions testing diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 9ff4e5a35ec..bd96f2f2872 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -136,8 +136,6 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k | `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | | `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | | `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | -| `.use-pg11` | Allows a job to run the `postgres` 11 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | | `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-pg13` | Allows a job to use the `postgres` 13 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 5f2df91edf3..d0eef0c86bd 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -147,4 +147,4 @@ We no longer use this optimization for `gitlab-org/gitlab` because the [pack-obj allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../../ci/runners/saas/linux_saas_runner.md#pre-clone-script). +[defined by GitLab.com shared runners](../../ci/runners/saas/linux_saas_runner.md#pre-clone-script-deprecated). diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index c72110bc253..41237dcdff4 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -4,7 +4,7 @@ group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Product Qualified Lead (PQL) development guide +# Product Qualified Lead (PQL) development guidelines The Product Qualified Lead (PQL) funnel connects our users with our team members. Read more about [PQL product principles](https://about.gitlab.com/handbook/product/product-principles/#product-qualified-leads-pqls). diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md index 3320f3134fb..31537b21527 100644 --- a/doc/development/project_templates.md +++ b/doc/development/project_templates.md @@ -35,7 +35,7 @@ installed: 1. In your selected namespace, create a public project. 1. Add the project content you want to use in the template. Do not include unnecessary assets or dependencies. For an example, [see this project](https://gitlab.com/gitlab-org/project-templates/dotnetcore). -1. When the project is ready for review, [create an issue](https://gitlab.com/gitlab-org/gitlab/issues) with a link to your project. +1. When the project is ready for review, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues) with a link to your project. In your issue, mention the Create:Source Code [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) for the Templates feature. diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 834a20239fc..590ac547c2d 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Working with Prometheus Metrics +# Prometheus metrics development guidelines ## Adding to the library diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md index 08d2f46c1cd..e9b52b81c0e 100644 --- a/doc/development/python_guide/index.md +++ b/doc/development/python_guide/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Python Development Guidelines +# Python development guidelines GitLab requires Python as a dependency for [reStructuredText](https://docutils.sourceforge.io/rst.html) markup rendering. diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md index a541de020fb..32295cc0e43 100644 --- a/doc/development/rails_update.md +++ b/doc/development/rails_update.md @@ -61,49 +61,49 @@ To efficiently and quickly find which Rails change caused the spec failure you c 1. Clone the `rails` project in a folder of your choice. For example, it might be the GDK root dir: - ```shell - cd <GDK_FOLDER> - git clone https://github.com/rails/rails.git - ``` + ```shell + cd <GDK_FOLDER> + git clone https://github.com/rails/rails.git + ``` 1. Replace the `gem 'rails'` line in GitLab `Gemfile` with: - ```ruby - gem 'rails', ENV['RAILS_VERSION'], path: ENV['RAILS_FOLDER'] - ``` + ```ruby + gem 'rails', ENV['RAILS_VERSION'], path: ENV['RAILS_FOLDER'] + ``` 1. Set the `RAILS_FOLDER` environment variable with the folder you cloned Rails into: - ```shell - export RAILS_FOLDER="<GDK_FOLDER>/rails" - ``` + ```shell + export RAILS_FOLDER="<GDK_FOLDER>/rails" + ``` 1. Change the directory to `RAILS_FOLDER` and set the range for the `git bisect` command: - ```shell - cd $RAILS_FOLDER - git bisect start <NEW_VERSION_TAG> <OLD_VERSION_TAG> - ``` + ```shell + cd $RAILS_FOLDER + git bisect start <NEW_VERSION_TAG> <OLD_VERSION_TAG> + ``` - Where `<NEW_VERSION_TAG>` is the tag where the spec is red and `<OLD_VERSION_TAG>` is the one with the green spec. - For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. - Replace `<NEW_VERSION_TAG>` with the tag where the spec is red and `<OLD_VERSION_TAG>` with the one with the green spec. For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. - In the output, you can see how many steps approximately it takes to find the commit. + Where `<NEW_VERSION_TAG>` is the tag where the spec is red and `<OLD_VERSION_TAG>` is the one with the green spec. + For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. + Replace `<NEW_VERSION_TAG>` with the tag where the spec is red and `<OLD_VERSION_TAG>` with the one with the green spec. For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. + In the output, you can see how many steps approximately it takes to find the commit. 1. Start the `git bisect` process and pass spec's filenames to `scripts/rails-update-bisect` as arguments. It can be faster to pick only one example instead of an entire spec file. - ```shell - git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb - # OR - git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb:7 - ``` + ```shell + git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb + # OR + git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb:7 + ``` 1. When the process is completed, `git bisect` prints the commit hash, which you can use to find the corresponding MR in the [`rails/rails`](https://github.com/rails/rails) repository. 1. Execute `git bisect reset` to exit the `bisect` mode. 1. Revert the changes to `Gemfile`: - ```shell - git checkout -- Gemfile - ``` + ```shell + git checkout -- Gemfile + ``` ### Follow-up reading material diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index cd7f8cba39b..82e96befd11 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -258,7 +258,7 @@ One way to generate the initial list is to run the Rake task `rubocop:todo:gener bundle exec rake rubocop:todo:generate ``` -To generate TODO list for specific RuboCop rules, pass them comma-separated as +To generate TODO list for specific RuboCop rules, pass them comma-seperated as argument to the Rake task: ```shell diff --git a/doc/development/real_time.md b/doc/development/real_time.md index 2aa48fed8eb..99d09170646 100644 --- a/doc/development/real_time.md +++ b/doc/development/real_time.md @@ -4,7 +4,7 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Real-Time Features +# Real-time features This guide contains instructions on how to safely roll out new real-time features. diff --git a/doc/development/redis.md b/doc/development/redis.md index 68cab9ac38d..a2dbf52822e 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Redis guidelines +# Redis development guidelines ## Redis instances @@ -188,6 +188,26 @@ it 'avoids N+1 Redis calls to forks_count key' do end ``` +You also can use special matchers `exceed_redis_calls_limit` and +`exceed_redis_command_calls_limit` to define an upper limit for +a number of Redis calls: + +```ruby +it 'avoids N+1 Redis calls' do + control = RedisCommands::Recorder.new { visit_page } + + expect(control).not_to exceed_redis_calls_limit(1) +end +``` + +```ruby +it 'avoids N+1 sadd Redis calls' do + control = RedisCommands::Recorder.new { visit_page } + + expect(control).not_to exceed_redis_command_calls_limit(:sadd, 1) +end +``` + These tests can help to identify N+1 problems related to Redis calls, and make sure that the fix for them works as expected. diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index 0a030a0877f..c1f69f4d103 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -177,19 +177,15 @@ bin/feature-flag use_primary_store_as_default_for_foo ``` By enabling `use_primary_and_secondary_stores_for_foo` feature flag, our `Gitlab::Redis::Foo` will use `MultiStore` to write to both new Redis instance -and the [old (fallback-instance)](#fallback-instance). -If we fail to fetch data from the new instance, we will fallback and read from the old Redis instance. -We can monitor logs for `Gitlab::Redis::MultiStore::ReadFromPrimaryError`, and also the Prometheus counter `gitlab_redis_multi_store_read_fallback_total`. +and the [old (fallback-instance)](#fallback-instance). All read commands are performed only on the default store which is controlled using the +`use_primary_store_as_default_for_foo` feature flag. By enabling `use_primary_store_as_default_for_foo` feature flag, +the `MultiStore` uses `primary_store` (new instance) as default Redis store. For `pipelined` commands (`pipelined` and `multi`), we execute the entire operation in both stores and then compare the results. If they differ, we emit a `Gitlab::Redis::MultiStore:PipelinedDiffError` error, and track it in the `gitlab_redis_multi_store_pipelined_diff_error_total` Prometheus counter. -Once we stop seeing those errors, this means that we are no longer relying on the data stored on the old Redis store. -At this point, we are probably safe to move the traffic to the new Redis store. - -By enabling `use_primary_store_as_default_for_foo` feature flag, the `MultiStore` will use `primary_store` (new instance) as default Redis store. - -Once this feature flag is enabled, we can disable `use_primary_and_secondary_stores_for_foo` feature flag. +After a period of time for the new store to be populated, we can perform external validation to compare the state of both stores. +Upon satisfactory validation results, we are probably safe to move the traffic to the new Redis store. We can disable `use_primary_and_secondary_stores_for_foo` feature flag. This will allow the MultiStore to read and write only from the primary Redis store (new store), moving all the traffic to the new Redis store. Once we have moved all our traffic to the primary store, our data migration is complete. @@ -206,17 +202,44 @@ MultiStore implements read and write Redis commands separately. - `smembers` - `scard` +- 'exists' +- 'exists?' +- 'get' +- 'hexists' +- 'hget' +- 'hgetall' +- 'hlen' +- 'hmget' +- 'hscan_each' +- 'mapped_hmget' +- 'mget' +- 'scan_each' +- 'scard' +- 'sismember' +- 'smembers' +- 'sscan' +- 'sscan_each' +- 'ttl' +- 'zscan_each' + ##### Write commands -- `set` -- `setnx` -- `setex` -- `sadd` -- `srem` -- `del` -- `pipelined` -- `flushdb` -- `rpush` +- 'del' +- 'eval' +- 'expire' +- 'flushdb' +- 'hdel' +- 'hset' +- 'incr' +- 'incrby' +- 'mapped_hmset' +- 'rpush' +- 'sadd' +- 'set' +- 'setex' +- 'setnx' +- 'srem' +- 'unlink' ##### `pipelined` commands @@ -227,7 +250,8 @@ Thus, excluding the Redis operations performed, the block should be idempotent. - `multi` When a command outside of the supported list is used, `method_missing` will pass it to the old Redis instance and keep track of it. -This ensures that anything unexpected behaves like it would before. +This ensures that anything unexpected behaves like it would before. In development or test environment, an error would be raised for early +detection. NOTE: By tracking `gitlab_redis_multi_store_method_missing_total` counter and `Gitlab::Redis::MultiStore::MethodMissingError`, @@ -237,7 +261,6 @@ a developer will need to add an implementation for missing Redis commands before | error | message | |---------------------------------------------------|---------------------------------------------------------------------------------------------| -| `Gitlab::Redis::MultiStore::ReadFromPrimaryError` | Value not found on the Redis primary store. Read from the Redis secondary store successful. | | `Gitlab::Redis::MultiStore::PipelinedDiffError` | `pipelined` command executed on both stores successfully but results differ between them. | | `Gitlab::Redis::MultiStore::MethodMissingError` | Method missing. Falling back to execute method on the Redis secondary store. | @@ -245,7 +268,6 @@ a developer will need to add an implementation for missing Redis commands before | Metrics name | Type | Labels | Description | |-------------------------------------------------------|--------------------|----------------------------|----------------------------------------------------------| -| `gitlab_redis_multi_store_read_fallback_total` | Prometheus Counter | `command`, `instance_name` | Client side Redis MultiStore reading fallback total | | `gitlab_redis_multi_store_pipelined_diff_error_total` | Prometheus Counter | `command`, `instance_name` | Redis MultiStore `pipelined` command diff between stores | | `gitlab_redis_multi_store_method_missing_total` | Prometheus Counter | `command`, `instance_name` | Client side Redis MultiStore method missing total | diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index a9164398afb..6d95dec823b 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w <!-- vale gitlab.Spelling = NO --> -In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) +In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/-/issues/1`) on the GitLab [Pull Repository Mirroring functionality](../user/project/repository/mirror/pull.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), diff --git a/doc/development/rubocop_development_guide.md b/doc/development/rubocop_development_guide.md index 2ff94f65232..1fdc0fbe78c 100644 --- a/doc/development/rubocop_development_guide.md +++ b/doc/development/rubocop_development_guide.md @@ -5,7 +5,7 @@ group: Development info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# RuboCop rule development guide +# RuboCop rule development guidelines Our codebase style is defined and enforced by [RuboCop](https://github.com/rubocop-hq/rubocop). @@ -43,7 +43,7 @@ Before adding a new cop to enforce a given style, make sure to discuss it with y We maintain cops across several Ruby code bases, and not all of them are specific to the GitLab application. When creating a new cop that could be applied to multiple applications, we encourage you -to add it to our [`gitlab-styles`](https://gitlab.com/gitlab-org/gitlab-styles) gem. +to add it to our [`gitlab-styles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles) gem. If the cop targets rules that only apply to the main GitLab application, it should be added to [GitLab](https://gitlab.com/gitlab-org/gitlab) instead. @@ -60,7 +60,7 @@ A grace period can safely be lifted as soon as there are no warnings for 2 weeks 1. Enable the new cop in `.rubocop.yml` (if not already done via [`gitlab-styles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles)). 1. [Generate TODOs for the new cop](rake_tasks.md#generate-initial-rubocop-todo-list). 1. [Set the new cop to `grace period`](#cop-grace-period). -1. Create an issue to fix TODOs and encourage community contributions (via ~"good for new contributors" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=good%20for%20new%20contributors&label_name%5B%5D=static%20code%20analysis&first_page_size=20). +1. Create an issue to fix TODOs and encourage community contributions (via ~"quick win" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=quick%20wins&label_name%5B%5D=static%20code%20analysis&first_page_size=20). 1. Create an issue to remove `grace period` after 2 weeks of silence in the `#f_rubocop` Slack channel. [See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903). ## Silenced offenses diff --git a/doc/development/search/advanced_search_migration_styleguide.md b/doc/development/search/advanced_search_migration_styleguide.md new file mode 100644 index 00000000000..2f8cd036dcf --- /dev/null +++ b/doc/development/search/advanced_search_migration_styleguide.md @@ -0,0 +1,311 @@ +--- +stage: Data Stores +group: Global Search +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Advanced search migration style guide + +## Creating a new advanced search migration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6. + +NOTE: +This functionality is only supported for indices created in GitLab 13.0 and later. + +In the [`ee/elastic/migrate/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/elastic/migrate) folder, create a new file with the filename format `YYYYMMDDHHMMSS_migration_name.rb`. This format is the same for Rails database migrations. + +```ruby +# frozen_string_literal: true + +class MigrationName < Elastic::Migration + # Important: Any updates to the Elastic index mappings must be replicated in the respective + # configuration files: + # - `Elastic::Latest::Config`, for the main index. + # - `Elastic::Latest::<Type>Config`, for standalone indices. + + def migrate + end + + # Check if the migration has completed + # Return true if completed, otherwise return false + def completed? + end +end +``` + +Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All migrations not executed +are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) +cron worker sequentially. + +To update Elastic index mappings, apply the configuration to the respective files: + +- For the main index: [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). +- For standalone indices: `Elastic::Latest::<Type>Config`. + +Migrations can be built with a retry limit and have the ability to be [failed and marked as halted](https://gitlab.com/gitlab-org/gitlab/-/blob/66e899b6637372a4faf61cfd2f254cbdd2fb9f6d/ee/lib/elastic/migration.rb#L40). +Any data or index cleanup needed to support migration retries should be handled in the migration. + +### Migration helpers + +The following migration helpers are available in `ee/app/workers/concerns/elastic/`: + +#### `Elastic::MigrationBackfillHelper` + +Backfills a specific field in an index. In most cases, the mapping for the field should already be added. + +Requires the `index_name` and `field_name` methods. + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationBackfillHelper + + private + + def index_name + Issue.__elasticsearch__.index_name + end + + def field_name + :schema_version + end +end +``` + +#### `Elastic::MigrationUpdateMappingsHelper` + +Updates a mapping in an index by calling `put_mapping` with the mapping specified. + +Requires the `index_name` and `new_mappings` methods. + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationUpdateMappingsHelper + + private + + def index_name + Issue.__elasticsearch__.index_name + end + + def new_mappings + { + schema_version: { + type: 'short' + } + } + end +end +``` + +#### `Elastic::MigrationRemoveFieldsHelper` + +Removes specified fields from an index. + +Requires the `index_name`, `document_type` methods. If there is one field to remove, add the `field_to_remove` method, otherwise add `fields_to_remove` with an array of fields. + +Checks in batches if any documents that match `document_type` have the fields specified in Elasticsearch. If documents exist, uses a Painless script to perform `update_by_query`. + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationRemoveFieldsHelper + + batched! + throttle_delay 1.minute + + private + + def index_name + User.__elasticsearch__.index_name + end + + def document_type + 'user' + end + + def fields_to_remove + %w[two_factor_enabled has_projects] + end +end +``` + +The default batch size is `10_000`. You can override this value by specifying `BATCH_SIZE`: + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationRemoveFieldsHelper + + batched! + BATCH_SIZE = 100 + + ... +end +``` + +#### `Elastic::MigrationObsolete` + +Marks a migration as obsolete when it's no longer required. + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationObsolete +end +``` + +#### `Elastic::MigrationHelper` + +Contains methods you can use when a migration doesn't fit the previous examples. + +```ruby +class MigrationName < Elastic::Migration + include Elastic::MigrationHelper + + def migrate + ... + end + + def completed? + ... + end +end +``` + +### Migration options supported by the `Elastic::MigrationWorker` + +[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) supports the following migration options: + +- `batched!` - Allow the migration to run in batches. If set, [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) + re-enqueues itself with a delay which is set using the `throttle_delay` option described below. The batching + must be handled in the `migrate` method. This setting controls the re-enqueuing only. + +- `batch_size` - Sets the number of documents modified during a `batched!` migration run. This size should be set to a value which allows the updates + enough time to finish. This can be tuned in combination with the `throttle_delay` option described below. The batching + must be handled in a custom `migrate` method or by using the [`Elastic::MigrationBackfillHelper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/concerns/elastic/migration_backfill_helper.rb) + `migrate` method which uses this setting. Default value is 1000 documents. + +- `throttle_delay` - Sets the wait time in between batch runs. This time should be set high enough to allow each migration batch + enough time to finish. Additionally, the time should be less than 5 minutes because that is how often the + [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) + cron worker runs. The default value is 3 minutes. + +- `pause_indexing!` - Pause indexing while the migration runs. This setting records the indexing setting before + the migration runs and set it back to that value when the migration is completed. + +- `space_requirements!` - Verify that enough free space is available in the cluster when the migration runs. This setting + halts the migration if the storage required is not available when the migration runs. The migration must provide + the space required in bytes by defining a `space_required_bytes` method. + +- `retry_on_failure` - Enable the retry on failure feature. By default, it retries + the migration 30 times. After it runs out of retries, the migration is marked as halted. + To customize the number of retries, pass the `max_attempts` argument: + `retry_on_failure max_attempts: 10` + +```ruby +# frozen_string_literal: true + +class BatchedMigrationName < Elastic::Migration + # Declares a migration should be run in batches + batched! + throttle_delay 10.minutes + pause_indexing! + space_requirements! + retry_on_failure + + # ... +end +``` + +### Multi-version compatibility + +These advanced search migrations, like any other GitLab changes, need to support the case where +[multiple versions of the application are running at the same time](../multi_version_compatibility.md). + +Depending on the order of deployment, it's possible that the migration +has started or finished and there's still a server running the application code from before the +migration. We need to take this into consideration until we can +[ensure all advanced search migrations start after the deployment has finished](https://gitlab.com/gitlab-org/gitlab/-/issues/321619). + +### Reverting a migration + +Because Elasticsearch does not support transactions, we always need to design our +migrations to accommodate a situation where the application +code is reverted after the migration has started or after it is finished. + +For this reason we generally defer destructive actions (for example, deletions after +some data is moved) to a later merge request after the migrations have +completed successfully. To be safe, for self-managed customers we should also +defer it to another release if there is risk of important data loss. + +### Best practices for advanced search migrations + +Follow these best practices for best results: + +- Order all migrations for each document type so that any migrations that use + [`Elastic::MigrationUpdateMappingsHelper`](#elasticmigrationupdatemappingshelper) + are executed before migrations that use the + [`Elastic::MigrationBackfillHelper`](#elasticmigrationbackfillhelper). This avoids + reindexing the same documents multiple times if all of the migrations are unapplied + and reduces the backfill time. +- When working in batches, keep the batch size under 9,000 documents. + The bulk indexer is set to run every minute and process a batch + of 10,000 documents. This way, the bulk indexer has time to + process records before another migration batch is attempted. +- To ensure that document counts are up to date, you should refresh + the index before checking if a migration is completed. +- Add logging statements to each migration when the migration starts, when a + completion check occurs, and when the migration is completed. These logs + are helpful when debugging issues with migrations. +- Pause indexing if you're using any Elasticsearch Reindex API operations. +- Consider adding a retry limit if there is potential for the migration to fail. + This ensures that migrations can be halted if an issue occurs. + +## Deleting advanced search migrations in a major version upgrade + +Because our advanced search migrations usually require us to support multiple +code paths for a long period of time, it's important to clean those up when we +safely can. + +We choose to use GitLab major version upgrades as a safe time to remove +backwards compatibility for indices that have not been fully migrated. We +[document this in our upgrade documentation](../../update/index.md#upgrading-to-a-new-major-version). +We also choose to replace the migration code with the halted migration +and remove tests so that: + +- We don't need to maintain any code that is called from our advanced search + migrations. +- We don't waste CI time running tests for migrations that we don't support + anymore. +- Operators who have not run this migration and who upgrade directly to the + target version see a message prompting them to reindex from scratch. + +To be extra safe, we do not delete migrations that were created in the last +minor version before the major upgrade. So, if we are upgrading to `%14.0`, +we should not delete migrations that were only added in `%13.12`. This +extra safety net allows for migrations that might +take multiple weeks to finish on GitLab.com. It would be bad if we upgraded +GitLab.com to `%14.0` before the migrations in `%13.12` were finished. Because +our deployments to GitLab.com are automated and we don't have +automated checks to prevent this, the extra precaution is warranted. +Additionally, even if we did have automated checks to prevent it, we wouldn't +actually want to hold up GitLab.com deployments on advanced search migrations, +as they may still have another week to go, and that's too long to block +deployments. + +### Process for removing migrations + +For every migration that was created 2 minor versions before the major version +being upgraded to, we do the following: + +1. Confirm the migration has actually completed successfully for GitLab.com. +1. Replace the content of the migration with: + + ```ruby + include Elastic::MigrationObsolete + ``` + +1. Delete any spec files to support this migration. +1. Remove any logic handling backwards compatibility for this migration. You + can find this by looking for + `Elastic::DataMigrationService.migration_has_finished?(:migration_name_in_lowercase)`. +1. Create a merge request with these changes. Noting that we should not + accidentally merge this before the major release is started. diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md index 5ac5118aae8..f74d31a1fb3 100644 --- a/doc/development/sec/index.md +++ b/doc/development/sec/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, concepts, howto --- -# Sec section development +# Sec section development guidelines The Sec section is responsible for GitLab application security features, the "Sec" part of DevSecOps. Development guides that are specific to the Sec section are listed here. @@ -88,7 +88,7 @@ In most other cases, the `identifiers` collection is unordered, where the remain Any time the primary identifier changes and a project pipeline is re-run, ingestion of the new report will "orphan" the previous DB record. Because our processing logic relies on generating a delta of two different vulnerabilities, it can end up looking rather confusing. For example: -[!Screenshot of primary identifier mismatch in MR widget](img/primary_identifier_changed_v15_6.png) + After being [merged](../integrations/secure.md#tracking-and-merging-vulnerabilities), the previous vulnerability is listed as "remediated" and the introduced as ["detected"](../../user/application_security/vulnerabilities/index.md#vulnerability-status-values). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 6c64e3b2acc..4f644dd018e 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -5,7 +5,7 @@ group: Development info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" --- -# Secure Coding Guidelines +# Secure coding development guidelines This document contains descriptions and guidelines for addressing security vulnerabilities commonly identified in the GitLab codebase. They are intended @@ -432,7 +432,7 @@ References: ### Select examples of past XSS issues affecting GitLab -- [Stored XSS in user status](https://gitlab.com/gitlab-org/gitlab-foss/issues/55320) +- [Stored XSS in user status](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55320) - [XSS vulnerability on custom project templates form](https://gitlab.com/gitlab-org/gitlab/-/issues/197302) - [Stored XSS in branch names](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55320) - [Stored XSS in merge request pages](https://gitlab.com/gitlab-org/gitlab/-/issues/35096) @@ -542,11 +542,11 @@ print(p.join('log', '/etc/passwd', '')) # renders the path to "/etc/passwd", which is not what we expect! ``` -#### Golang +#### Go -Golang has similar behavior with [`path.Clean`](https://pkg.go.dev/path#example-Clean). Remember that with many file systems, using `../../../../` traverses up to the root directory. Any remaining `../` are ignored. This example may give an attacker access to `/etc/passwd`: +Go has similar behavior with [`path.Clean`](https://pkg.go.dev/path#example-Clean). Remember that with many file systems, using `../../../../` traverses up to the root directory. Any remaining `../` are ignored. This example may give an attacker access to `/etc/passwd`: -```golang +```go path.Clean("/../../etc/passwd") // renders the path to "etc/passwd"; the file path is relative to whatever the current directory is path.Clean("../../etc/passwd") @@ -601,7 +601,7 @@ Go has built-in protections that usually prevent an attacker from successfully i Consider the following example: -```golang +```go package main import ( @@ -620,7 +620,7 @@ This echoes `"1; cat /etc/passwd"`. **Do not** use `sh`, as it bypasses internal protections: -```golang +```go out, _ = exec.Command("sh", "-c", "echo 1 | cat /etc/passwd").Output() ``` @@ -646,15 +646,15 @@ And the following cipher suites (according to the [RFC 8446](https://datatracker - `TLS_AES_128_GCM_SHA256` - `TLS_AES_256_GCM_SHA384` -*Note*: **Golang** does [not support](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676) all cipher suites with TLS 1.3. +*Note*: **Go** does [not support](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676) all cipher suites with TLS 1.3. ##### Implementation examples ##### TLS 1.3 -For TLS 1.3, **Golang** only supports [3 cipher suites](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676), as such we only need to set the TLS version: +For TLS 1.3, **Go** only supports [3 cipher suites](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676), as such we only need to set the TLS version: -```golang +```go cfg := &tls.Config{ MinVersion: tls.VersionTLS13, } @@ -678,9 +678,9 @@ response = GitLab::HTTP.perform_request(Net::HTTP::Get, 'https://gitlab.com', ss ##### TLS 1.2 -**Golang** does support multiple cipher suites that we do not want to use with TLS 1.2. We need to explicitly list authorized ciphers: +**Go** does support multiple cipher suites that we do not want to use with TLS 1.2. We need to explicitly list authorized ciphers: -```golang +```go func secureCipherSuites() []uint16 { return []uint16{ tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, @@ -692,7 +692,7 @@ func secureCipherSuites() []uint16 { And then use `secureCipherSuites()` in `tls.Config`: -```golang +```go tls.Config{ (...), CipherSuites: secureCipherSuites(), @@ -852,6 +852,31 @@ In the example above, the `is_admin?` method is overwritten when passing it to t Working with archive files like `zip`, `tar`, `jar`, `war`, `cpio`, `apk`, `rar` and `7z` presents an area where potentially critical security vulnerabilities can sneak into an application. +### Utilities for safely working with archive files + +There are common utilities that can be used to securely work with archive files. + +#### Ruby + +| Archive type | Utility | +|--------------|-------------| +| `zip` | `SafeZip` | + +#### `SafeZip` + +SafeZip provides a safe interface to extract specific directories or files within a `zip` archive through the `SafeZip::Extract` class. + +Example: + +```ruby +Dir.mktmpdir do |tmp_dir| + SafeZip::Extract.new(zip_file_path).extract(files: ['index.html', 'app/index.js'], to: tmp_dir) + SafeZip::Extract.new(zip_file_path).extract(directories: ['src/', 'test/'], to: tmp_dir) +rescue SafeZip::Extract::EntrySizeError + raise Error, "Path `#{file_path}` has invalid size in the zip!" +end +``` + ### Zip Slip In 2018, the security company Snyk [released a blog post](https://security.snyk.io/research/zip-slip-vulnerability) describing research into a widespread and critical vulnerability present in many libraries and applications which allows an attacker to overwrite arbitrary files on the server file system which, in many cases, can be leveraged to achieve remote code execution. The vulnerability was dubbed Zip Slip. @@ -895,7 +920,7 @@ end #### Go -```golang +```go // unzip INSECURELY extracts source zip file to destination. func unzip(src, dest string) error { r, err := zip.OpenReader(src) @@ -991,7 +1016,7 @@ end You are encouraged to use the secure archive utilities provided by [LabSec](https://gitlab.com/gitlab-com/gl-security/appsec/labsec) which will handle Zip Slip and other types of vulnerabilities for you. The LabSec utilities are also context aware which makes it possible to cancel or timeout extractions: -```golang +```go package main import "gitlab-com/gl-security/appsec/labsec/archive/zip" @@ -1016,7 +1041,7 @@ func main() { In case the LabSec utilities do not fit your needs, here is an example for extracting a zip file with protection against Zip Slip attacks: -```golang +```go // unzip extracts source zip file to destination with protection against Zip Slip attacks. func unzip(src, dest string) error { r, err := zip.OpenReader(src) @@ -1093,7 +1118,7 @@ end #### Go -```golang +```go // printZipContents INSECURELY prints contents of files in a zip file. func printZipContents(src string) error { r, err := zip.OpenReader(src) @@ -1161,7 +1186,7 @@ You are encouraged to use the secure archive utilities provided by [LabSec](http In case the LabSec utilities do not fit your needs, here is an example for extracting a zip file with protection against symlink attacks: -```golang +```go // printZipContents prints contents of files in a zip file with protection against symlink attacks. func printZipContents(src string) error { r, err := zip.OpenReader(src) diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index ef2e7e6edf5..5bfb81c1d00 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -19,7 +19,7 @@ To implement a new metric in Service Ping, follow these steps: 1. [Name and place the metric](metrics_dictionary.md#metric-key_path) 1. [Test counters manually using your Rails console](#test-counters-manually-using-your-rails-console) 1. [Generate the SQL query](#generate-the-sql-query) -1. [Optimize queries with `#database-lab`](#optimize-queries-with-database-lab) +1. [Optimize queries with Database Lab](#optimize-queries-with-database-lab) 1. [Add the metric definition to the Metrics Dictionary](#add-the-metric-definition) 1. [Add the metric to the Versions Application](#add-the-metric-to-the-versions-application) 1. [Create a merge request](#create-a-merge-request) @@ -174,7 +174,7 @@ Errors return a value of `-1`. WARNING: This functionality estimates a distinct count of a specific ActiveRecord_Relation in a given column, -which uses the [HyperLogLog](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) algorithm. +which uses the [HyperLogLog](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf) algorithm. As the HyperLogLog algorithm is probabilistic, the **results always include error**. The highest encountered error rate is 4.9%. @@ -330,48 +330,36 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P ```yaml - name: users_creating_epics - category: epics_usage - redis_slot: users aggregation: weekly - feature_flag: track_epics_activity ``` Keys: - `name`: unique event name. - Name format for Redis HLL events `<name>_<redis_slot>`. + Name format for Redis HLL events `{hll_counters}_<name>` [See Metric name](metrics_dictionary.md#metric-name) for a complete guide on metric naming suggestion. - Consider including in the event's name the Redis slot to be able to count totals for a specific category. - Example names: `users_creating_epics`, `users_triggering_security_scans`. - - `category`: event category. Used for getting total counts for events in a category, for easier - access to a group of events. - - `redis_slot`: optional Redis slot. Default value: event name. Only event data that is stored in the same slot - can be aggregated. Ensure keys are in the same slot. For example: - `users_creating_epics` with `redis_slot: 'users'` builds Redis key - `{users}_creating_epics-2020-34`. If `redis_slot` is not defined the Redis key will - be `{users_creating_epics}-2020-34`. - Recommended slots to use are: `users`, `projects`. This is the value we count. - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. Aggregation on a `daily` basis does not pull more fine grained data. - - `feature_flag`: if no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. 1. Use one of the following methods to track the event: - - In the controller using the `RedisTracking` module and the following format: + - In the controller using the `ProductAnalyticsTracking` module and the following format: ```ruby - track_event(*controller_actions, name:, conditions: nil, destinations: [:redis_hll], &block) + track_event(*controller_actions, name:, action:, label:, conditions: nil, destinations: [:redis_hll], &block) ``` Arguments: - `controller_actions`: the controller actions to track. - `name`: the event name. + - `action`: required if destination is `:snowplow. Action name for the triggered event. See [event schema](../snowplow/index.md#event-schema) for more details. + - `label`: required if destination is `:snowplow. Label for the triggered event. See [event schema](../snowplow/index.md#event-schema) for more details. - `conditions`: optional custom conditions. Uses the same format as Rails callbacks. - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: `:redis_hll`. - `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`. @@ -381,10 +369,14 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P ```ruby # controller class ProjectsController < Projects::ApplicationController - include RedisTracking + include ProductAnalyticsTracking skip_before_action :authenticate_user!, only: :show - track_event :index, :show, name: 'users_visiting_projects' + track_event :index, :show, + name: 'users_visiting_projects', + action: 'user_perform_visit', + label: 'redis_hll_counters.users_visiting_project_monthly', + destinations: %i[redis_hll snowplow] def index render html: 'index' @@ -497,24 +489,12 @@ We have the following recommendations for [adding new events](#add-new-events): - Event aggregation: weekly. - When adding new metrics, use a [feature flag](../../operations/feature_flags.md) to control the impact. -- For feature flags triggered by another service, set `default_enabled: false`, - - Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change +It's recommended to disable the new feature flag by default (set `default_enabled: false`). +- Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change ##### Enable or disable Redis HLL tracking -Events are tracked behind optional [feature flags](../feature_flags/index.md) due to concerns for Redis performance and scalability. - -For a full list of events and corresponding feature flags, see the [`known_events/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. - -To enable or disable tracking for specific event in <https://gitlab.com> or <https://about.staging.gitlab.com>, run commands such as the following to -[enable or disable the corresponding feature](../feature_flags/index.md). - -```shell -/chatops run feature set <feature_name> true -/chatops run feature set <feature_name> false -``` - -We can also disable tracking completely by using the global flag: +We can disable tracking completely by using the global flag: ```shell /chatops run feature set redis_hll_tracking true @@ -529,16 +509,6 @@ For each event we add metrics for the weekly and monthly time frames, and totals - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. - `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. -Redis HLL implementation calculates total metrics when both of these conditions are met: - -- The category is manually included in [CATEGORIES_FOR_TOTALS](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/hll_redis_counter.rb#L21). -- There is more than one metric for the same category, aggregation, and Redis slot. - -We add total unique counts for the weekly and monthly time frames where applicable: - -- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. -- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. - Example of `redis_hll_counters` data: ```ruby @@ -563,6 +533,7 @@ Example of `redis_hll_counters` data: "ide_edit_total_unique_counts_weekly"=>0, "ide_edit_total_unique_counts_monthly"=>0} } +} ``` Example: @@ -674,10 +645,9 @@ pry(main)> Gitlab::UsageData.count(User.active) (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 ``` -## Optimize queries with `#database-lab` +## Optimize queries with Database Lab -`#database-lab` is a Slack channel that uses a production-sized environment to test your queries. -Paste the SQL query into `#database-lab` to see how the query performs at scale. +[Database Lab](../database/database_lab.md) is a service that uses a production clone to test queries. - GitLab.com's production database has a 15 second timeout. - Any single query must stay below the [1 second execution time](../database/query_performance.md#timing-guidelines-for-queries) with cold caches. @@ -693,7 +663,7 @@ to a merge request description: - Query generated for the index and time. - Migration output for up and down execution. -We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). +For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). ### Optimization recommendations and examples @@ -751,7 +721,7 @@ To set up Service Ping locally, you must: Make sure you run `docker-compose up` to start a PostgreSQL and Redis instance. 1. Point GitLab to the Versions Application endpoint instead of the default endpoint: 1. Open [service_ping/submit_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L5) locally and modify `STAGING_BASE_URL`. - 1. Set it to the local Versions Application URL: `http://localhost:3000/usage_data`. + 1. Set it to the local Versions Application URL: `http://localhost:3000`. ### Test local setup @@ -846,7 +816,6 @@ you must fulfill the following requirements: 1. All events listed at `events` attribute must come from [`known_events/*.yml`](#known-events-are-added-automatically-in-service-data-payload) files. -1. All events listed at `events` attribute must have the same `redis_slot` attribute. 1. All events listed at `events` attribute must have the same `aggregation` attribute. 1. `time_frame` does not include `all` value, which is unavailable for Redis sourced aggregated metrics. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 14c9cb33446..e938de9e253 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -4,7 +4,7 @@ group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Service Ping Guide +# Service Ping development guidelines > - Introduced in GitLab Ultimate 11.2, more statistics. > - In GitLab 14.1, [renamed from Usage Ping to Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/5990). In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version. @@ -91,23 +91,23 @@ sequenceDiagram the required URL is <https://version.gitlab.com/>. 1. In case of an error, it will be reported to the Version application along with following pieces of information: - - `uuid` - GitLab instance unique identifier - - `hostname` - GitLab instance hostname - - `version` - GitLab instance current versions - - `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence - - `message` - Error message - - <pre> - <code> - { - "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", - "hostname"=>"127.0.0.1", - "version"=>"14.7.0-pre", - "elapsed"=>0.006946, - "message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...' - } - </code> - </pre> + - `uuid` - GitLab instance unique identifier + - `hostname` - GitLab instance hostname + - `version` - GitLab instance current versions + - `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence + - `message` - Error message + + <pre> + <code> + { + "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", + "hostname"=>"127.0.0.1", + "version"=>"14.7.0-pre", + "elapsed"=>0.006946, + "message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...' + } + </code> + </pre> 1. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics: @@ -135,7 +135,7 @@ sequenceDiagram ] } } - ``` +``` ### On a Geo secondary site @@ -177,7 +177,7 @@ The following is example content of the Service Ping payload. "recorded_at": "2020-04-17T07:43:54.162+00:00", "edition": "EEU", "license_md5": "00000000000000000000000000000000", - "license_sha256: "0000000000000000000000000000000000000000000000000000000000000000", + "license_sha256": "0000000000000000000000000000000000000000000000000000000000000000", "license_id": null, "historical_max_users": 999, "licensee": { diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 28581f81f94..90ce776af19 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -44,9 +44,9 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`, `license`. | | `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `operational`, `optional`, `subscription`, `standard`. The default value is `optional`.| | `instrumentation_class` | yes | `string`; [the class that implements the metric](metrics_instrumentation.md). | -| `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | +| `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. | | `performance_indicator_type` | no | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, `umau` or `customer_health_score`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). | -| `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | +| `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | | `milestone` | yes | The milestone when the metric is introduced and when it's available to self-managed instances with the official GitLab release. | | `milestone_removed` | no | The milestone when the metric is removed. | | `introduced_by_url` | no | The URL to the merge request that introduced the metric to be available for self-managed instances. | diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 80500ccc723..14ec4b8c9a6 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -154,17 +154,17 @@ end You can use Redis metrics to track events not kept in the database, for example, a count of how many times the search bar has been used. -[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97009). +[Example of a merge request that adds `Redis` metrics](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103455). The `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class. -Count unique values for `source_code_pushes` event. - Required options: - `event`: the event name. - `prefix`: the value of the `PREFIX` constant used in the counter classes from the `Gitlab::UsageDataCounters` namespace. +Count unique values for `source_code_pushes` event. + ```yaml time_frame: all data_source: redis diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 8a8ceae1f4c..7fcbafe50f7 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -82,10 +82,6 @@ To remove a metric: 1. Create an issue for removing the metric if none exists yet. The issue needs to outline why the metric should be deleted. You can use this issue to document the removal process. -1. Check the following YAML files and verify the metric is not used in an aggregate: - - [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) - - [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) - 1. Verify the metric is not used to calculate the conversational index. The conversational index is a measure that reports back to self-managed instances to inform administrators of the progress of DevOps adoption for the instance. @@ -115,7 +111,7 @@ To remove a metric: can be safely removed from Service Ping. Use this [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/15266) for guidance. -1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment in the issue regarding the deletion of the metric. +1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment in the issue from step 1 regarding the deletion of the metric. Many Service Ping metrics are relied upon for health score and XMAU reporting and unexpected changes to those metrics could break reporting. 1. After you verify the metric can be safely removed, diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index d78a005d76b..25f62fbcc98 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Guidelines for shell commands in the GitLab codebase +# Shell command development guidelines This document contains guidelines for working with processes and files in the GitLab codebase. These guidelines are meant to make your code more reliable _and_ secure. diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index b417a099228..97663b0b41c 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -46,30 +46,30 @@ following example deprecates and then removes `arg2` from the `perform_async` me 1. Provide a default value (usually `nil`) and use a comment to mark the argument as deprecated in the coming minor release. (Release M) - ```ruby - class ExampleWorker - # Keep arg2 parameter for backwards compatibility. - def perform(object_id, arg1, arg2 = nil) - # ... - end - end - ``` + ```ruby + class ExampleWorker + # Keep arg2 parameter for backwards compatibility. + def perform(object_id, arg1, arg2 = nil) + # ... + end + end + ``` 1. One minor release later, stop using the argument in `perform_async`. (Release M+1) - ```ruby - ExampleWorker.perform_async(object_id, arg1) - ``` + ```ruby + ExampleWorker.perform_async(object_id, arg1) + ``` 1. At the next major release, remove the value from the worker class. (Next major release) - ```ruby - class ExampleWorker - def perform(object_id, arg1) - # ... - end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id, arg1) + # ... + end + end + ``` ### Add an argument @@ -84,29 +84,29 @@ This approach requires multiple releases. 1. Add the argument to the worker with a default value (Release M). - ```ruby - class ExampleWorker - def perform(object_id, new_arg = nil) - # ... - end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id, new_arg = nil) + # ... + end + end + ``` 1. Add the new argument to all the invocations of the worker (Release M+1). - ```ruby - ExampleWorker.perform_async(object_id, new_arg) - ``` + ```ruby + ExampleWorker.perform_async(object_id, new_arg) + ``` 1. Remove the default value (Release M+2). - ```ruby - class ExampleWorker - def perform(object_id, new_arg) - # ... - end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id, new_arg) + # ... + end + end + ``` #### Parameter hash @@ -115,13 +115,13 @@ uses a parameter hash. 1. Use a parameter hash in the worker to allow future flexibility. - ```ruby - class ExampleWorker - def perform(object_id, params = {}) - # ... - end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id, params = {}) + # ... + end + end + ``` ## Removing worker classes @@ -131,54 +131,54 @@ To remove a worker class, follow these steps over two minor releases: 1. Remove any code that enqueues the jobs. - For example, if there is a UI component or an API endpoint that a user can interact with that results in the worker instance getting enqueued, make sure those surface areas are either removed or updated in a way that the worker instance is no longer enqueued. + For example, if there is a UI component or an API endpoint that a user can interact with that results in the worker instance getting enqueued, make sure those surface areas are either removed or updated in a way that the worker instance is no longer enqueued. - This ensures that instances related to the worker class are no longer being enqueued. + This ensures that instances related to the worker class are no longer being enqueued. 1. Ensure both the frontend and backend code no longer relies on any of the work that used to be done by the worker. 1. In the relevant worker classes, replace the contents of the `perform` method with a no-op, while keeping any arguments in tact. - For example, if you're working with the following `ExampleWorker`: + For example, if you're working with the following `ExampleWorker`: - ```ruby - class ExampleWorker - def perform(object_id) - SomeService.run!(object_id) - end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id) + SomeService.run!(object_id) + end + end + ``` - Implementing the no-op might look like this: + Implementing the no-op might look like this: - ```ruby - class ExampleWorker - def perform(object_id); end - end - ``` + ```ruby + class ExampleWorker + def perform(object_id); end + end + ``` - By implementing this no-op, you can avoid unnecessary cycles once any deprecated jobs that are still enqueued eventually get processed. + By implementing this no-op, you can avoid unnecessary cycles once any deprecated jobs that are still enqueued eventually get processed. ### In a subsequent, separate minor release 1. Delete the worker class file and follow the guidance in our [Sidekiq queues documentation](../sidekiq/index.md#sidekiq-queues) around running Rake tasks to regenerate/update related files. 1. Add a migration (not a post-deployment migration) that uses `sidekiq_remove_jobs`: - ```ruby - class RemoveMyDeprecatedWorkersJobInstances < Gitlab::Database::Migration[2.0] - DEPRECATED_JOB_CLASSES = %w[ - MyDeprecatedWorkerOne - MyDeprecatedWorkerTwo - ] - - def up - sidekiq_remove_jobs(job_klasses: DEPRECATED_JOB_CLASSES) - end - - def down - # This migration removes any instances of deprecated workers and cannot be undone. - end - end - ``` + ```ruby + class RemoveMyDeprecatedWorkersJobInstances < Gitlab::Database::Migration[2.0] + DEPRECATED_JOB_CLASSES = %w[ + MyDeprecatedWorkerOne + MyDeprecatedWorkerTwo + ] + + def up + sidekiq_remove_jobs(job_klasses: DEPRECATED_JOB_CLASSES) + end + + def down + # This migration removes any instances of deprecated workers and cannot be undone. + end + end + ``` ## Renaming queues diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index 355f5a3b753..2010a21130d 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Sidekiq guides +# Sidekiq development guidelines We use [Sidekiq](https://github.com/mperham/sidekiq) as our background job processor. These guides are for writing jobs that works well on diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md index 794a9a0160c..dc2214a40ed 100644 --- a/doc/development/snowplow/event_dictionary_guide.md +++ b/doc/development/snowplow/event_dictionary_guide.md @@ -39,8 +39,8 @@ Each event is defined in a separate YAML file consisting of the following fields | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the event. | | `milestone` | no | The milestone when the event is introduced. | | `introduced_by_url` | no | The URL to the merge request that introduced the event. | -| `distributions` | yes | The [distributions](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. Can be set to one or more of `ce` or `ee`. | -| `tiers` | yes | The [tiers]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. Can be set to one or more of `free`, `premium`, or `ultimate`. | +| `distributions` | yes | The [distributions](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. Can be set to one or more of `ce` or `ee`. | +| `tiers` | yes | The [tiers](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/) where the tracked feature is available. Can be set to one or more of `free`, `premium`, or `ultimate`. | ### Example event definition @@ -79,14 +79,13 @@ tiers: Use the dedicated [event definition generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/snowplow_event_definition_generator.rb) to create new event definitions. -The `category` and `action` of each event are included in the filename to enforce uniqueness. +The `category` and `action` of each event are included in the filename to standardize file naming. The generator takes three options: - `--ee`: Indicates if the event is for EE. - `--category=CATEGORY`: Indicates the `category` of the event. - `--action=ACTION`: Indicates the `action` of the event. -- `--force`: Overwrites the existing event definition, if one already exists. ```shell bundle exec rails generate gitlab:snowplow_event_definition --category Groups::EmailCampaignsController --action click diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 40b8b7b3da8..ae90b45b1f0 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -150,89 +150,89 @@ To implement Vue component tracking: 1. Import the `Tracking` library and call the `mixin` method: - ```javascript - import Tracking from '~/tracking'; + ```javascript + import Tracking from '~/tracking'; - const trackingMixin = Tracking.mixin(); + const trackingMixin = Tracking.mixin(); - // Optionally provide default properties - // const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); - ``` + // Optionally provide default properties + // const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); + ``` 1. Use the mixin in the component: - ```javascript - export default { - mixins: [trackingMixin], - // Or - // mixins: [Tracking.mixin()], - // mixins: [Tracking.mixin({ label: 'right_sidebar' })], - - data() { - return { - expanded: false, - }; - }, - }; - ``` + ```javascript + export default { + mixins: [trackingMixin], + // Or + // mixins: [Tracking.mixin()], + // mixins: [Tracking.mixin({ label: 'right_sidebar' })], + + data() { + return { + expanded: false, + }; + }, + }; + ``` 1. You can specify tracking options in by creating a `tracking` data object or computed property: - ```javascript - export default { - name: 'RightSidebar', - - mixins: [Tracking.mixin()], - - data() { - return { - expanded: false, - variant: '', - tracking: { - label: 'right_sidebar', - // property: '', - // value: '', - // experiment: '', - // extra: {}, - }, - }; - }, - - // Or - // computed: { - // tracking() { - // return { - // property: this.variant, - // extra: { expanded: this.expanded }, - // }; - // }, - // }, - }; - ``` + ```javascript + export default { + name: 'RightSidebar', + + mixins: [Tracking.mixin()], + + data() { + return { + expanded: false, + variant: '', + tracking: { + label: 'right_sidebar', + // property: '', + // value: '', + // experiment: '', + // extra: {}, + }, + }; + }, + + // Or + // computed: { + // tracking() { + // return { + // property: this.variant, + // extra: { expanded: this.expanded }, + // }; + // }, + // }, + }; + ``` 1. Call the `track` method. Tracking options can be passed as the second parameter: - ```javascript - this.track('click_button', { - label: 'right_sidebar', - }); - ``` + ```javascript + this.track('click_button', { + label: 'right_sidebar', + }); + ``` - Or use the `track` method in the template: + Or use the `track` method in the template: - ```html - <template> - <div> - <button data-testid="toggle" @click="toggle">Toggle</button> + ```html + <template> + <div> + <button data-testid="toggle" @click="toggle">Toggle</button> - <div v-if="expanded"> - <p>Hello world!</p> - <button @click="track('click_button')">Track another event</button> - </div> - </div> - </template> - ``` + <div v-if="expanded"> + <p>Hello world!</p> + <button @click="track('click_button')">Track another event</button> + </div> + </div> + </template> + ``` #### Testing example @@ -473,9 +473,7 @@ For a video tutorial, see the [Snowplow plugin walk through](https://www.youtube 1. To open the extension, select the Snowplow Inspector icon beside the address bar. 1. Click around on a webpage with Snowplow to see JavaScript events firing in the inspector window. -### Test backend events - -#### Snowplow Micro +### Test backend events with Snowplow Micro [Snowplow Micro](https://snowplow.io/blog/introducing-snowplow-micro/) is a Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro @@ -484,50 +482,10 @@ records the same events as the full Snowplow pipeline. To query events, use the It can be set up automatically using [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit). See the [how-to docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/snowplow_micro.md) for more details. -Optionally, you can set it up manually, using the following instructions. - -#### Set up Snowplow Micro manually - -To install and run Snowplow Micro, complete these steps to modify the -[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): - -1. Ensure [Docker is installed](https://docs.docker.com/get-docker/) and running. - -1. To install Snowplow Micro, clone the settings in -[this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). - -1. Navigate to the directory with the cloned project, - and start the appropriate Docker container: - - ```shell - ./snowplow-micro.sh - ``` - 1. Set the environment variable to tell the GDK to use Snowplow Micro in development. This overrides two `application_settings` options: - `snowplow_enabled` setting will instead return `true` from `Gitlab::Tracking.enabled?` - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever port is set for `snowplow_micro.port` GDK setting) from `Gitlab::Tracking.collector_hostname`. - - ```shell - gdk config set snowplow_micro.enabled true - ``` - - Optionally, you can set the port for you Snowplow Micro instance as well (the default value is `9090`): - - ```shell - gdk config set snowplow_micro.port 8080 - ``` - -1. Regenerate the project YAML config: - - ```shell - gdk reconfigure - ``` - -1. Restart GDK: - - ```shell - gdk restart - ``` +With Snowplow Micro set up you can now manually test backend Snowplow events: 1. Send a test Snowplow event from the Rails console: diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 6acbd72175e..276b5913890 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -4,7 +4,7 @@ group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Snowplow +# Snowplow development guidelines Snowplow is an enterprise-grade marketing and Product Intelligence platform that tracks how users engage with our website and application. diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index ee2c1eb95df..da7f2bc2781 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -28,7 +28,7 @@ events or touches Snowplow related files. - For frontend events, when relevant, add a screenshot of the event in the [testing tool](implementation.md#develop-and-test-snowplow) used. - For backend events, when relevant, add the output of the - [Snowplow Micro](implementation.md#snowplow-micro) good events + [Snowplow Micro](implementation.md#test-backend-events-with-snowplow-micro) good events `GET http://localhost:9090/micro/good` (it might be a good idea to reset with `GET http://localhost:9090/micro/reset` first). - Update the [Event Dictionary](event_dictionary_guide.md). @@ -41,3 +41,4 @@ events or touches Snowplow related files. - If needed, check that the events are firing locally using one of the [testing tools](implementation.md#develop-and-test-snowplow) available. - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. +- If the snowplow event mirrors a RedisHLL event, then tag @mdrussell to review if the payload is usable for this purpose. diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md index b2d780b1563..f2812cd6de9 100644 --- a/doc/development/spam_protection_and_captcha/exploratory_testing.md +++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md @@ -32,6 +32,7 @@ Enable any relevant feature flag, if the spam/CAPTCHA support is behind a featur 1. For **Site key**, use: `6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI` 1. For **Secret key**, use: `6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe` 1. Go to **Admin -> Settings -> Reporting** settings: `http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings` + 1. Expand the **Spam and Anti-bot Protection** section. 1. Select **Enable reCAPTCHA**. Enabling for login is not required unless you are testing that feature. 1. Enter the **Site key** and **Secret key**. 1. To set up Akismet: diff --git a/doc/development/spam_protection_and_captcha/model_and_services.md b/doc/development/spam_protection_and_captcha/model_and_services.md index 9c5d389a2f5..2c7ec8c3f50 100644 --- a/doc/development/spam_protection_and_captcha/model_and_services.md +++ b/doc/development/spam_protection_and_captcha/model_and_services.md @@ -35,9 +35,9 @@ To do this: designate which fields to consider the "`title`" or "`description`". For example, this line designates the `content` field as the `description`: - ```ruby - attr_spammable :content, spam_description: true - ``` + ```ruby + attr_spammable :content, spam_description: true + ``` 1. Add a `#check_for_spam?` method implementation: diff --git a/doc/development/stage_group_observability/img/error_budgets_kibana_dashboard_v15_10.png b/doc/development/stage_group_observability/img/error_budgets_kibana_dashboard_v15_10.png Binary files differnew file mode 100644 index 00000000000..e4f54b579c1 --- /dev/null +++ b/doc/development/stage_group_observability/img/error_budgets_kibana_dashboard_v15_10.png diff --git a/doc/development/stage_group_observability/index.md b/doc/development/stage_group_observability/index.md index b275b0bfec2..ba17b4cc73a 100644 --- a/doc/development/stage_group_observability/index.md +++ b/doc/development/stage_group_observability/index.md @@ -68,11 +68,11 @@ component can have two indicators: and [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) services, that threshold is **5 seconds** when not opted in to the - [`rails_requests` SLI](../application_slis/rails_request_apdex.md). + [`rails_request` SLI](../application_slis/rails_request.md). We've made this target configurable in [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). - To learn how to customize the request Apdex, see - [Rails request Apdex SLI](../application_slis/rails_request_apdex.md). + To customize the request Apdex, see + [Rails request SLIs](../application_slis/rails_request.md). This new Apdex measurement is not part of the error budget until you [opt in](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1451). @@ -136,3 +136,39 @@ For example, see the `server` component of the `web-pages` service:  To add more SLIs tailored to specific features, you can use an [Application SLI](../application_slis/index.md). + +## Kibana dashboard for error budgets + +For a detailed analysis you can use [a specialized Kibana dashboard](https://log.gprd.gitlab.net/goto/771b5c10-c0ec-11ed-85ed-e7557b0a598c), like this: + + + +Description: + +- **Apdex requests over limit (graph)** - Displays only requests that exceeded their + target duration. +- **Apdex operations over-limit duration (graph)** - Displays the distribution of duration + components (database, Redis, Gitaly, and Rails app). +- **Apdex requests** (pie chart) - Displays the percentage of `2xx`, `3xx`, `4xx` and + `5xx` requests. +- **Slow request component distribution** - Highlights the component responsible + for Apdex violation. +- **Apdex operations over limit** (table) - Displays a number of operations over + limit for each endpoint. +- **Apdex requests over limit** - Displays a list of individual requests responsible + for Apdex violation. + +### Use the dashboard + +1. Select the feature category you want to investigate. + 1. Scroll to the **Feature Category** section. Enter the feature name. + 1. Select **Apply changes**. Selected results contain only requests related to this feature category. +1. Select the time frame for the investigation. +1. Review dashboard and pay attention to the type of failures. + +Questions to answer: + +1. Does the failure pattern look like a spike? Or does it persist? +1. Does the failure look related to a particular component? (database, Redis, ...) +1. Does the failure affect a specific endpoint? Or is it system-wide? +1. Does the failure appear caused by infrastructure incidents? diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 54d3f368bbd..a9d17472a9f 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -507,6 +507,8 @@ If needed, you can scope interactions within a specific area of the page by usin As you will likely be scoping to an element such as a `div`, which typically does not have a label, you may use a `data-testid` selector in this case. +You can use the `be_axe_clean` matcher to run [axe automated accessibility testing](../fe_guide/accessibility.md#automated-accessibility-testing-with-axe) in feature tests. + ##### Externalized contents Test expectations against externalized contents should call the same diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 31d68bb9f4f..cf23792e239 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -42,11 +42,11 @@ rake contracts:merge_requests:test:merge_requests[contract_merge_requests] #### Verify the contracts in Pact Broker -By default, the Rake tasks will verify the locally stored contracts. In order to verify the contracts published in the Pact Broker, we need to set the `PACT_BROKER` environment variable to `true`. It is important to point out here that the file path and file name of the provider test is what is used to find the contract in the Pact Broker which is why it is important to make sure the [provider test naming conventions](#provider-naming) are followed. +By default, the Rake tasks will verify the locally stored contracts. In order to verify the contracts published in the Pact Broker, we need to set the `PACT_BROKER` environment variable to `true` and the `QA_PACT_BROKER_HOST` to the URL of the Pact Broker. It is important to point out here that the file path and file name of the provider test is what is used to find the contract in the Pact Broker which is why it is important to make sure the [provider test naming conventions](#provider-naming) are followed. ## Publish contracts to Pact Broker -The contracts generated by the consumer test can be published to a hosted Pact Broker by going to `spec/contracts` and running the `publish-contracts.sh` script. +The contracts generated by the consumer test can be published to a hosted Pact Broker by setting the `QA_PACT_BROKER_HOST` environment variable and running the [`publish-contracts.sh`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/publish-contracts.sh) script. ## Test suite folder structure and naming conventions diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index b81379d89b2..07a9bcb2f5c 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -87,7 +87,7 @@ file `basic_login_spec.rb`. See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block) WARNING: -The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2` +The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/550) in `13.2` in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead. ### The outer `RSpec.describe` block diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 777a9f47339..fd184c13e5e 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -16,9 +16,9 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case -Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). +Every test should have a corresponding test case in the [GitLab project test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). If a test case issue does not yet exist, any GitLab team member can create a new test case in -the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project +the **[CI/CD > Test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases)** page of the GitLab project with a placeholder title. After the test case URL is linked to a test in the code, when the test is run in a pipeline that has reporting enabled, the `report-results` script automatically updates the test case and the results issue. diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index c1389b3ac0e..c9e60c06732 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -52,5 +52,5 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance. | | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | +| `:testcase` | The link to the test case issue in the [GitLab Project test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | | `:transient` | The test tests transient bugs. It is excluded by default. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 4a947e59d5f..0d6d7a161c7 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -219,7 +219,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r # Using a full image address GITLAB_QA_ACCESS_TOKEN=your-token-here gitlab-qa Test::Integration::Geo registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 --no-teardown - ``` + ``` You can use the `--no-tests` option to build the containers only, and then run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from your GDK to complete setup and run tests. However, there might be configuration issues if your GDK and the containers are based on different GitLab versions. With the `--no-teardown` option, GitLab-QA uses the same GitLab version for the GitLab containers and the GitLab QA container used to configure the Geo setup. @@ -369,15 +369,15 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: 1. Run the GitLab container: - ```shell - sudo docker run \ - --hostname localhost \ - --net test \ - --publish 443:443 --publish 80:80 --publish 22:22 \ - --name gitlab \ - --env GITLAB_OMNIBUS_CONFIG="gitlab_rails['ldap_enabled'] = true; gitlab_rails['ldap_servers'] = {\"main\"=>{\"label\"=>\"LDAP\", \"host\"=>\"ldap-server.test\", \"port\"=>389, \"uid\"=>\"uid\", \"bind_dn\"=>\"cn=admin,dc=example,dc=org\", \"password\"=>\"admin\", \"encryption\"=>\"plain\", \"verify_certificates\"=>false, \"base\"=>\"dc=example,dc=org\", \"user_filter\"=>\"\", \"group_base\"=>\"ou=Global Groups,dc=example,dc=org\", \"admin_group\"=>\"AdminGroup\", \"external_groups\"=>\"\", \"sync_ssh_keys\"=>false}}; gitlab_rails['ldap_sync_worker_cron'] = '* * * * *'; gitlab_rails['ldap_group_sync_worker_cron'] = '* * * * *'; " \ - gitlab/gitlab-ee:latest - ``` + ```shell + sudo docker run \ + --hostname localhost \ + --net test \ + --publish 443:443 --publish 80:80 --publish 22:22 \ + --name gitlab \ + --env GITLAB_OMNIBUS_CONFIG="gitlab_rails['ldap_enabled'] = true; gitlab_rails['ldap_servers'] = {\"main\"=>{\"label\"=>\"LDAP\", \"host\"=>\"ldap-server.test\", \"port\"=>389, \"uid\"=>\"uid\", \"bind_dn\"=>\"cn=admin,dc=example,dc=org\", \"password\"=>\"admin\", \"encryption\"=>\"plain\", \"verify_certificates\"=>false, \"base\"=>\"dc=example,dc=org\", \"user_filter\"=>\"\", \"group_base\"=>\"ou=Global Groups,dc=example,dc=org\", \"admin_group\"=>\"AdminGroup\", \"external_groups\"=>\"\", \"sync_ssh_keys\"=>false}}; gitlab_rails['ldap_sync_worker_cron'] = '* * * * *'; gitlab_rails['ldap_group_sync_worker_cron'] = '* * * * *'; " \ + gitlab/gitlab-ee:latest + ``` 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index f476815bf32..c626a4fa81c 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -251,9 +251,9 @@ which would give us the minimal test combination to reproduce the failure: for the list under `Knapsack node specs:` in the CI job output log. 1. Save the list of specs as a file, and run: - ```shell - cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky - ``` + ```shell + cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky + ``` If there is an order-dependency issue, the script above will print the minimal reproduction. diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 85d807eceb1..6134e0f9959 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -520,6 +520,25 @@ it('waits for an event', () => { }); ``` +### Manipulate `gon` object + +`gon` (or `window.gon`) is a global object used to pass data from the backend. If your test depends +on its value you can directly modify it: + +```javascript +describe('when logged in', () => { + beforeEach(() => { + gon.current_user_id = 1; + }); + + it('shows message', () => { + expect(wrapper.text()).toBe('Logged in!'); + }); +}) +``` + +`gon` is reset in every test to ensure tests are isolated. + ### Ensuring that tests are isolated Tests are normally architected in a pattern which requires a recurring setup of the component under test. This is often achieved by making use of the `beforeEach` hook. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index b074a9e34f3..7958d6db706 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -276,7 +276,7 @@ find a way to limit it to only us.** ## Other resources - [Review apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) -- [Stability issues](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/212) +- [Stability issues](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/212) ### Helpful command line tools diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index c93a5fd539b..187d30c12be 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -300,7 +300,7 @@ graph RL - **DOM**: Testing on the real DOM ensures your components work in the intended environment. - Part of DOM testing is delegated to [cross-browser testing](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/45). + Part of DOM testing is delegated to [cross-browser testing](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/45). - **Properties or state of components**: On this level, all tests can only perform actions a user would do. For example: to change the state of a component, a click event would be fired. diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md index b5509f5934e..a62e8ea2d58 100644 --- a/doc/development/uploads/index.md +++ b/doc/development/uploads/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Uploads development guide +# Uploads development guidelines Uploads are an integral part of many GitLab features. To understand how GitLab handles uploads, this page provides an overview of the key mechanisms for transferring files to a storage destination. diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md index 6955f9c31cd..5575297ad6b 100644 --- a/doc/development/uploads/working_with_uploads.md +++ b/doc/development/uploads/working_with_uploads.md @@ -95,7 +95,7 @@ They consist of: Example: -```golang +```go u.route("PUT", apiProjectPattern+`packages/nuget/`, mimeMultipartUploader), ``` diff --git a/doc/development/ux/index.md b/doc/development/ux/index.md new file mode 100644 index 00000000000..784a59a3a4a --- /dev/null +++ b/doc/development/ux/index.md @@ -0,0 +1,26 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Contribute to UX design + +## UX Design + +These instructions are specifically for those wanting to make UX design contributions to GitLab. + +The UX department at GitLab uses [Figma](https://www.figma.com/) for all of its designs, and you can see our [Design Repository documentation](https://gitlab.com/gitlab-org/gitlab-design/blob/master/README.md#getting-started) for details on working with our files. + +You may leverage the [Pajamas UI Kit](https://www.figma.com/community/file/781156790581391771) in Figma to create mockups for your proposals. However, we will also gladly accept handmade drawings and sketches, wireframes, manipulated DOM screenshots, or prototypes. You can find design resources documentation in our [Design System](https://design.gitlab.com/). Use it to understand where and when to use common design solutions. + +## Contributing to Pajamas + +To contribute to [Pajamas design system](https://design.gitlab.com/) and the [UI kit](https://www.figma.com/community/file/781156790581391771), follow the [contribution guidelines](https://design.gitlab.com/get-started/contribute) documented in the handbook. While the instructions are code-focused, they will help you understand the overall process of contributing. + +## Contributing to other issues + +1. Review the list of available issues that are currently [accepting UX contribution](https://gitlab.com/groups/gitlab-org/-/issues/?sort=weight&state=opened&label_name%5B%5D=UX&label_name%5B%5D=workflow%3A%3Aready%20for%20design&label_name%5B%5D=Accepting%20UX%20contributions&first_page_size=20). +1. Find an issue that does not have an Assignee to ensure someone else is not working on a solution. Add the `~"workflow::design"` and `~"Community contribution"` labels and mention `@gitlab-com/gitlab-ux/reviewers` to request they assign the issue to you. +1. Add your design proposal to the issue description/[design management](../../user/project/issues/design_management.md) section. Remember to keep the scope of the proposal/change small following our [MVCs guidelines](https://about.gitlab.com/handbook/values/#minimal-viable-change-mvc). +1. If you have any questions or are ready for a review of your proposal, mention `@gitlab-com/gitlab-ux/reviewers` in a comment to make your request. diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 77a32b62e32..634c337793f 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -4,7 +4,7 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Value stream analytics development guide +# Value stream analytics development guidelines For information on how to configure value stream analytics (VSA) in GitLab, see our [analytics documentation](../user/analytics/value_stream_analytics.md). @@ -65,7 +65,7 @@ of the stage. Stages are configurable by the user within the pairing rules defin IDs are identical. - The stage event hash ID is later used to store the aggregated data in partitioned database tables. -Historically, value stream analytics defined [7 stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/cycle_analytics/default_stages.rb) +Historically, value stream analytics defined [six stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/cycle_analytics/default_stages.rb) which are always available to the end-users regardless of the subscription. ### Value streams diff --git a/doc/development/wikis.md b/doc/development/wikis.md index acb7ed335d3..2f97931f924 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: "GitLab's development guidelines for Wikis" --- -# Wikis development guide +# Wikis development guidelines > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227027) in GitLab 13.5. diff --git a/doc/development/windows.md b/doc/development/windows.md index bf56e16344a..99085b4b5f8 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -13,7 +13,7 @@ This is a guide for how to get a Windows development virtual machine on Google C ## Why Windows in Google Cloud? -Use of Microsoft Windows operating systems on company laptops is banned under the GitLab [Approved Operating Systems policy](https://about.gitlab.com/handbook/security/approved_os.html#windows). +Use of Microsoft Windows operating systems on company laptops is banned under the GitLab [Approved Operating Systems policy](https://about.gitlab.com/handbook/it/operating-systems/#windows). This can make it difficult to develop features for the Windows platforms. Using GCP allows us to have a temporary Windows machine that can be removed once we're done with it. diff --git a/doc/development/workspace/index.md b/doc/development/workspace/index.md index 0e0b6943a0b..ca404702d72 100644 --- a/doc/development/workspace/index.md +++ b/doc/development/workspace/index.md @@ -1,159 +1,11 @@ --- -comments: false -type: index, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines" -description: "Development Guidelines: learn about organization when developing GitLab." +redirect_to: '../organization/index.md' +remove_date: '2023-06-13' --- -# Organization +This document was moved to [another location](../organization/index.md). -The [Organization initiative](../../user/workspace/index.md) focuses on reaching feature parity between -SaaS and self-managed installations. - -## Consolidate groups and projects - -- [Architecture blueprint](../../architecture/blueprints/consolidating_groups_and_projects/index.md) - -One facet of the Organization initiative is to consolidate groups and projects, -addressing the feature disparity between them. Some features, such as epics, are -only available at the group level. Some features, such as issues, are only available -at the project level. Other features, such as milestones, are available to both groups -and projects. - -We receive many requests to add features either to the group or project level. -Moving features around to different levels is problematic on multiple levels: - -- It requires engineering time to move the features. -- It requires UX overhead to maintain mental models of feature availability. -- It creates redundant code. - -When features are copied from one level (project, group, or instance) to another, -the copies often have small, nuanced differences between them. These nuances cause -extra engineering time when fixes are needed, because the fix must be copied to -several locations. These nuances also create different user experiences when the -feature is used in different places. - -A solution for this problem is to consolidate groups and projects into a single -entity, `namespace`. The work on this solution is split into several phases and -is tracked in [epic 6473](https://gitlab.com/groups/gitlab-org/-/epics/6473). - -### Phase 1 - -- [Phase 1 epic](https://gitlab.com/groups/gitlab-org/-/epics/6697). -- **Goals**: - 1. Ensure every project receives a corresponding record in the `namespaces` - table with `type='Project'`. - 1. For user namespaces, the type changes from `NULL` to `User`. - -We should make sure that projects, and the project namespace, are equivalent: - -- **Create project:** use Rails callbacks to ensure a new project namespace is - created for each project. Project namespace records should contain `created_at` and - `updated_at` attributes equal to the project's `created_at`/`updated_at` attributes. -- **Update project:** use the `after_save` callback in Rails to ensure some - attributes are kept in sync between project and project namespaces. - Read [`project#after_save`](https://gitlab.com/gitlab-org/gitlab/blob/6d26634e864d7b748dda0e283eb2477362263bc3/app/models/project.rb#L101-L101) - for more information. -- **Delete project:** use FKs cascade delete or Rails callbacks to ensure when a `Project` - or its `ProjectNamespace` is removed, its corresponding `ProjectNamespace` or `Project` - is also removed. -- **Transfer project to a different group:** make sure that when a project is transferred, - its corresponding project namespace is transferred to the same group. -- **Transfer group:** make sure when transferring a group that all of its sub-projects, - either direct or through descendant groups, have their corresponding project - namespaces transferred correctly as well. -- **Export or import project** - - **Export project** continues to export only the project, and not its project namespace, - in this phase. The project namespace does not contain any specific information - to export at this point. Eventually we want the project namespace to be exported as well. - - **Import project** creates a new project, so the project namespace is created through - Rails `after_save` callback on the project model. -- **Export or import group:** when importing or exporting a `Group`, projects are not - included in the operation. If that feature is changed to include `Project` when its group is - imported or exported, the logic must include their corresponding project namespaces - in the import or export. - -After ensuring these points, run a database migration to create a `ProjectNamespace` -record for every `Project`. Project namespace records created during the migration -should have `created_at` and `updated_at` attributes set to the migration runtime. -The project namespaces' `created_at` and `updated_at` attributes would not match -their corresponding project's `created_at` and `updated_at` attributes. We want -the different dates to help audit any of the created project namespaces, in case we need it. -After this work completes, we must migrate data as described in -[Backfill `ProjectNamespace` for every Project](https://gitlab.com/gitlab-org/gitlab/-/issues/337100). - -### Phase 2 - -- [Phase 2 epic](https://gitlab.com/groups/gitlab-org/-/epics/6768). -- **Goal**: Link `ProjectNamespace` to other entities on the database level. - -In this phase: - -- Communicate the changes company-wide at the engineering level. We want to make - engineers aware of the upcoming changes, even though teams are not expected to - collaborate actively until phase 3. -- Raise awareness to avoid regressions, and conflicting or duplicate work that - can be dealt with before phase 3. - -### Phase 3 - -- [Phase 3 epic](https://gitlab.com/groups/gitlab-org/-/epics/6585). -- **Goal**: Achieve feature parity between the namespace types. -Problems to solve as part of this phase: - -- Routes handling through `ProjectNamespace` rather than `Project`. -- Memberships handling. -- Policies handling. -- Import and export. -- Other interactions between project namespace and project models. - -Phase 3 is when the active migration of features from `Project` to `ProjectNamespace`, -or directly to `Namespace`, happens. - -### How to plan features that interact with Group and ProjectNamespace - -As of now, every Project in the system has a record in the `namespaces` table. This makes it possible to -use common interface to create features that are shared between Groups and Projects. Shared behavior can be added using -a concerns mechanism. Because the `Namespace` model is responsible for `UserNamespace` methods as well, it is discouraged -to use the `Namespace` model for shared behavior for Projects and Groups. - -#### Resource-based features - -To migrate resource-based features, existing functionality will need to be supported. This can be achieved in two Phases. - -**Phase 1 - Setup** - -- Link into the namespaces table - - Add a column to the table - - For example, in issues a `project id` points to the projects table. We need to establish a link to the `namespaces` table. - - Modify code so that any new record already has the correct data in it - - Backfill - -**Phase 2 - Prerequisite work** - -- Investigate the permission model as well as any performance concerns related to that. - - Permissions need to be checked and kept in place. -- Investigate what other models need to support namespaces for functionality dependent on features you migrate in Phase 1. -- Adjust CRUD services and APIs (REST and GraphQL) to point to the new column you added in Phase 1. -- Consider performance when fetching resources. - -Introducing new functionality is very much dependent on every single team and feature. - -#### Settings-related features - -Right now, cascading settings are available for `NamespaceSettings`. By creating `ProjectNamespace`, -we can use this framework to make sure that some settings are applicable on the project level as well. - -When working on settings, we need to make sure that: - -- They are not used in `join` queries or modify those queries. -- Updating settings is taken into consideration. -- If we want to move from project to project namespace, we follow a similar database process to the one described in [Phase 1](#phase-1). - -## Related topics - -- [Consolidating groups and projects](../../architecture/blueprints/consolidating_groups_and_projects/index.md) - architecture documentation -- [Organization user documentation](../../user/workspace/index.md) +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/drawers/advanced_search_syntax.md b/doc/drawers/advanced_search_syntax.md index 7556c8bdfaf..6e732bd3175 100644 --- a/doc/drawers/advanced_search_syntax.md +++ b/doc/drawers/advanced_search_syntax.md @@ -13,6 +13,7 @@ source: /doc/user/search/advanced_search.md | Syntax | Description | Example | |--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `"` | Exact search | [`"gem sidekiq"`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=%22gem+sidekiq%22) | +| `~` | Fuzzy search | [`J~ Doe`](https://gitlab.com/search?scope=users&search=j%7E+doe) | | <code>|</code> | Or | [<code>display | banner</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+%7C+banner) | | `+` | And | [`display +banner`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=display+%2Bbanner&snippets=) | | `-` | Exclude | [`display -banner`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+-banner) | diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index 05731f93605..0915b9b8e21 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -7,89 +7,95 @@ type: howto # Add a file to a repository **(FREE)** -Adding files to a repository is a small, but key task. Bringing files in to a repository, -such as code, images, or documents, allows them to be tracked by Git, even though they -may have been created elsewhere. - -You can add a file to a repository in your [terminal](#add-a-file-using-the-command-line), and -then push to GitLab. You can also use the [web interface](../user/project/repository/web_editor.md#upload-a-file), -which may be a simpler solution. - -If you need to create a file first, for example a `README.md` text file, that can -also be done from the [terminal](command-line-commands.md#create-a-text-file-in-the-current-directory) or -[web interface](../user/project/repository/web_editor.md#create-a-file). - -## Add a file using the command line - -Open a [terminal/shell](command-line-commands.md), and change into the folder of your -GitLab project. This usually means running the following command until you get -to the desired destination: - -```shell -cd <destination folder> -``` - -[Create a new branch](../tutorials/make_your_first_git_commit.md#create-a-branch-and-make-changes) to add your file into. Submitting changes directly -to the default branch should be avoided unless your project is very small and you're the -only person working on it. - -You can also [switch to an existing branch](start-using-git.md#switch-to-a-branch) -if you have one already. - -Using your standard tool for copying files (for example, Finder in macOS, or File Explorer -on Windows), put the file into a directory in the GitLab project. - -Check if your file is actually present in the directory (if you're on Windows, -use `dir` instead): - -```shell -ls -``` - -You should see the name of the file in the list shown. - -Check the status: - -```shell -git status -``` - -Your file's name should appear in red, so `git` took notice of it! Now add it -to the repository: - -```shell -git add <name of file> -``` - -Check the status again, your file's name should have turned green: - -```shell -git status -``` - -Commit (save) your file to the repository: - -```shell -git commit -m "DESCRIBE COMMIT IN A FEW WORDS" -``` - -Now you can push (send) your changes (in the branch `<branch-name>`) to GitLab -(the Git remote named 'origin'): - -```shell -git push origin <branch-name> -``` - -Your image is added to your branch in your repository in GitLab. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +Adding files to a repository is a small, but key, task. No matter where the code, +images, or documents were created, Git tracks them after you add them to your repository. + +## Add an existing file + +To add an existing file to your repository, either: + +- Upload the file from the GitLab UI. +- Add a file to your repository from the command line, then push the file up to GitLab. + +### From the UI + +If you are unfamiliar with the command line, use the +[Web Editor](../user/project/repository/web_editor.md) to upload a file from the GitLab UI: + +<!-- Original source for this list: doc/user/project/repository/web_editor.md#upload-a-file --> +<!-- For why we duplicated the info, see https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111072#note_1267429478 --> + +1. On the top bar, select **Main menu > Projects** and find your project. +1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). +1. From the dropdown list, select **Upload file**. +1. Complete the fields. To create a merge request with the uploaded file, ensure the **Start a new merge request with these changes** toggle is turned on. +1. Select **Upload file**. + +### From the command line + +To add a new file from the command line: + +1. Open a terminal (or shell) window. +1. Use the "change directory" (`cd`) command to go to your GitLab project's folder. + Run the `cd DESTINATION` command, changing `DESTINATION` to the location of your folder. +1. Choose a Git branch to work in. You can either: + - [Create a new branch](../tutorials/make_your_first_git_commit.md#create-a-branch-and-make-changes) + to add your file into. Don't submit changes directly to the default branch of your + repository unless your project is very small and you're the only person working on it. + - [Switch to an existing branch](start-using-git.md#switch-to-a-branch). +1. Copy the file into the appropriate directory in your project. Use your standard tool + for copying files, such as Finder in macOS, or File Explorer in Windows. +1. In your terminal window, confirm that your file is present in the directory: + - Windows: Use the `dir` command. + - All other operating systems: Use the `ls` command. + You should see the name of the file in the list shown. +1. Check the status of your file with the `git status` command. Your file's name + should be red. Files listed in red are in your file system, but Git isn't tracking them yet. +1. Tell Git to track this file with the `git add FILENAME` command, replacing `FILENAME` + with the name of your file. +1. Check the status of your file again with the `git status` command. Your file's name + should be green. Files listed in green are tracked locally by Git, but still + need to be committed and pushed. +1. Commit (save) your file to your local copy of your project's Git repository: + + ```shell + git commit -m "DESCRIBE COMMIT IN A FEW WORDS" + ``` + +1. Push (send) your changes from your copy of the repository, up to GitLab. + In this command, `origin` refers to the copy of the repository stored at GitLab. + Replace `BRANCHNAME` with the name of your branch: + + ```shell + git push origin BRANCHNAME + ``` + +1. Git prepares, compresses, and sends the data. Lines from the remote repository + (here, GitLab) are prefixed with `remote:` like this: + + ```plaintext + Enumerating objects: 9, done. + Counting objects: 100% (9/9), done. + Delta compression using up to 10 threads + Compressing objects: 100% (5/5), done. + Writing objects: 100% (5/5), 1.84 KiB | 1.84 MiB/s, done. + Total 5 (delta 3), reused 0 (delta 0), pack-reused 0 + remote: + remote: To create a merge request for BRANCHNAME, visit: + remote: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/new?merge_request%5Bsource_branch%5D=BRANCHNAME + remote: + To https://gitlab.com/gitlab-org/gitlab.git + * [new branch] BRANCHNAME -> BRANCHNAME + branch 'BRANCHNAME' set up to track 'origin/BRANCHNAME'. + ``` + +Your file is now copied from your local copy of the repository, up to the remote +repository at GitLab. To create a merge request, copy the link sent back from the remote +repository and paste it into a browser window. + +## Add a new file + +To create a new file (like a `README.md` text file) in your repository, either: + +- [Create the file](../user/project/repository/web_editor.md#create-a-file) from the GitLab UI. +- Create the file from the terminal. diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index 07ab9365693..2850669ce57 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -1,123 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../user/index.md' +remove_date: '2023-06-09' --- -# Edit files through the command line **(FREE)** +This document was moved to [another location](../user/index.md). -When [working with Git from the command line](start-using-git.md), you need to -use more than just the Git commands. There are several basic commands that you should -learn to make full use of the command line. - -## Start working on your project - -To work on a Git project locally (from your own computer), with the command line, -first you need to [clone (copy) it](start-using-git.md#clone-a-repository) to -your computer. - -## Working with files on the command line - -This section has examples of some basic shell commands that you might find useful. -For more information, search the web for _bash commands_. - -Alternatively, you can edit files using your choice of editor (IDE), or the GitLab user -interface (not locally). - -### Common commands - -The list below is not exhaustive, but contains many of the most commonly used commands. - -| Command | Description | -|--------------------------------|---------------------------------------------| -| `cd NAME-OF-DIRECTORY` | Go into a directory to work in it | -| `cd ..` | Go back one directory | -| `ls` | List what's in the current directory | -| `ls a*` | List what's in the current directory that starts with `a` | -| `ls *.md` | List what's in the current directory that ends with `.md` | -| `mkdir NAME-OF-YOUR-DIRECTORY` | Create a new directory | -| `cat README.md` | Display the contents of a [text file you created previously](#create-a-text-file-in-the-current-directory) | -| `pwd` | Show the current directory | -| `clear` | Clear the shell window | - -### Create a text file in the current directory - -To create a text file from the command line, for example `README.md`, follow these -steps: - -```shell -touch README.md -nano README.md -#### ADD YOUR INFORMATION -#### Press: control + X -#### Type: Y -#### Press: enter -``` - -### Remove a file or directory - -It's easy to delete (remove) a file or directory, but be careful: - -WARNING: -This will **permanently** delete a file. - -```shell -rm NAME-OF-FILE -``` - -WARNING: -This will **permanently** delete a directory and **all** of its contents. - -```shell -rm -r NAME-OF-DIRECTORY -``` - -### View and Execute commands from history - -You can view the history of all the commands you executed from the command line, -and then execute any of them again, if needed. - -First, list the commands you executed previously: - -```shell -history -``` - -Then, choose a command from the list and check the number next to the command (`123`, -for example) . Execute the same full command with: - -```shell -!123 -``` - -### Carry out commands for which the account you are using lacks authority - -Not all commands can be executed from a basic user account on a computer, you may -need administrator's rights to execute commands that affect the system, or try to access -protected data, for example. You can use `sudo` to execute these commands, but you -might be asked for an administrator password. - -```shell -sudo RESTRICTED-COMMAND -``` - -WARNING: -Be careful of the commands you run with `sudo`. Certain commands may cause -damage to your data or system. - -## Sample Git task flow - -If you're completely new to Git, looking through some [sample task flows](https://rogerdudler.github.io/git-guide/) -may help you understand the best practices for using these commands as you work. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index 224d1fabb14..194647259b8 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -98,7 +98,7 @@ access on GitLab.com or any other GitLab instance. To use the repository in the examples on this page: 1. Go to [https://gitlab.com/gitlab-tests/sample-project/](https://gitlab.com/gitlab-tests/sample-project/). -1. In the upper right, select **Fork**. +1. In the upper-right corner, select **Fork**. 1. Choose a namespace for your fork. The project becomes available at `https://gitlab.com/<your-namespace>/sample-project/`. diff --git a/doc/index.md b/doc/index.md index 492320d93aa..49684705686 100644 --- a/doc/index.md +++ b/doc/index.md @@ -28,7 +28,7 @@ Welcome to the GitLab documentation! | [**New to Git and GitLab?**](tutorials/index.md)<br/>Start learning about Git and GitLab. | [**Contribute to GitLab development**](#contributing-to-gitlab)<br/>Create new GitLab functionality and documentation. | | [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Learn how to move to GitLab. | [**Build an integration with GitLab**](#build-an-integration-with-gitlab)<br/>Integrate with Jira and other common applications. | | [**Choose a subscription**](subscriptions/index.md)<br/>Determine which subscription tier makes sense for you. | [**Install GitLab**](https://about.gitlab.com/install/)<br/>Install GitLab on different platforms. | -| [**Reference architectures**](administration/reference_architectures/index.md)<br/>View recommended deployments at scale. | [**Update GitLab**](update/index.md)<br/>Update your GitLab self-managed instance to the latest version. | +| [**Reference architectures**](administration/reference_architectures/index.md)<br/>View recommended deployments at scale. | [**Upgrade GitLab**](update/index.md)<br/>Upgrade your GitLab self-managed instance to the latest version. | | [**GitLab releases**](https://about.gitlab.com/releases/)<br/>See what's new in GitLab. | | ## Popular topics @@ -43,7 +43,7 @@ Have a look at some of our most popular topics: | [Activate GitLab EE with a license](user/admin_area/license.md) | Activate GitLab Enterprise Edition functionality with a license. | | [Back up and restore GitLab](raketasks/backup_restore.md) | Rake tasks for backing up and restoring GitLab self-managed instances. | | [GitLab release and maintenance policy](policy/maintenance.md) | Policies for version naming and cadence, and also upgrade recommendations. | -| [Elasticsearch integration](integration/advanced_search/elasticsearch.md) | Integrate Elasticsearch with GitLab to enable advanced searching. | +| [Elasticsearch integration](integration/advanced_search/elasticsearch.md) | Integrate Elasticsearch with GitLab to enable advanced search. | | [Omnibus GitLab database settings](https://docs.gitlab.com/omnibus/settings/database.html) | Database settings for Omnibus GitLab self-managed instances. | | [Omnibus GitLab NGINX settings](https://docs.gitlab.com/omnibus/settings/nginx.html) | NGINX settings for Omnibus GitLab self-managed instances. | | [Omnibus GitLab SSL configuration](https://docs.gitlab.com/omnibus/settings/ssl.html) | SSL settings for Omnibus GitLab self-managed instances. | diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 99ba05925a5..8f540691a45 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -87,7 +87,7 @@ Because any given GitLab upgrade might involve data disk updates or database sch - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. - [GitLab Premium or Ultimate Marketplace (pre-licensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. -1. AMI IDs are unique per region, so after you've loaded one of the above, select the desired target region in the upper right of the console to see the appropriate AMIs. +1. AMI IDs are unique per region. After you've loaded any of these editions, in the upper-right corner, select the desired target region of the console to see the appropriate AMIs. 1. After the console is loaded, you can add additional search criteria to narrow further. For instance, type `13.` to find only 13.x versions. 1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select **Launch** near the top of left of the page. diff --git a/doc/install/installation.md b/doc/install/installation.md index eda9c503e28..1cd6a4fc5d2 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -11,8 +11,7 @@ using the source files. To set up a **development installation** or for many other installation options, see the [main installation page](index.md). It was created for and tested on **Debian/Ubuntu** operating systems. Read [requirements.md](requirements.md) for hardware and operating system requirements. -If you want to install on RHEL/CentOS, we recommend using the -[Omnibus packages](https://about.gitlab.com/install/). +If you want to install on RHEL/CentOS, you should use the [Omnibus packages](https://about.gitlab.com/install/). This guide is long because it covers many cases and includes all commands you need, this is [one of the few installation scripts that actually work out of the box](https://twitter.com/robinvdvleuten/status/424163226532986880). @@ -46,17 +45,16 @@ If the highest number stable branch is unclear, check the [GitLab blog](https:// ## Software requirements -| Software | Minimum version | Notes | -| ------------------ | --------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Ruby](#2-ruby) | `2.7` | From GitLab 13.6, Ruby 2.7 is required. Ruby 3.0 is not supported yet (see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/5149) for the current status). You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | -| [Go](#3-go) | `1.18` | From GitLab 15.6, Go 1.18 or later is required. | -| [Git](#git) | `2.38.x` | From GitLab 15.8, Git 2.38.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | -| [Node.js](#4-node) | `16.15.0` | From GitLab 15.7, Node.js 16.15.0 or later is required. | +| Software | Minimum version | Notes | +|:-------------------|:----------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Ruby](#2-ruby) | `3.0.x` | From GitLab 15.10, Ruby 3.0 is required. You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | +| [Go](#3-go) | `1.18.x` | From GitLab 15.6, Go 1.18 or later is required. | +| [Git](#git) | `2.38.x` | From GitLab 15.8, Git 2.38.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | +| [Node.js](#4-node) | `16.15.0` | From GitLab 15.7, Node.js 16.15.0 or later is required. | ## GitLab directory structure -This is the main directory structure you end up with following the instructions -of this page: +When following the instructions on this page, you create this directory structure: ```plaintext |-- home @@ -224,8 +222,8 @@ Download Ruby and compile it: ```shell mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.6.tar.gz" -echo 'e7203b0cc09442ed2c08936d483f8ac140ec1c72e37bb5c401646b7866cb5d10 ruby-2.7.6.tar.gz' | sha256sum -c - && tar xzf ruby-2.7.6.tar.gz +curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/3.0/ruby-3.0.5.tar.gz" +echo '9afc6380a027a4fe1ae1a3e2eccb6b497b9c5ac0631c12ca56f9b7beb4848776 ruby-3.0.5.tar.gz' | sha256sum -c - && tar xzf ruby-3.0.5.tar.gz cd ruby-2.7.6 ./configure --disable-install-rdoc --enable-shared diff --git a/doc/install/next_steps.md b/doc/install/next_steps.md index c718d895c8a..70b6101b1eb 100644 --- a/doc/install/next_steps.md +++ b/doc/install/next_steps.md @@ -56,7 +56,7 @@ installation. ## Cross-repository Code Search -- [Advanced Search](../integration/advanced_search/elasticsearch.md): Leverage [Elasticsearch](https://www.elastic.co/) or [OpenSearch](https://opensearch.org/) for +- [Advanced search](../integration/advanced_search/elasticsearch.md): Leverage [Elasticsearch](https://www.elastic.co/) or [OpenSearch](https://opensearch.org/) for faster, more advanced code search across your entire GitLab instance. ## Scaling and replication diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md index 6a2c562377f..389c79a281c 100644 --- a/doc/integration/advanced_search/elasticsearch.md +++ b/doc/integration/advanced_search/elasticsearch.md @@ -7,8 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Elasticsearch **(PREMIUM SELF)** -This page describes how to enable Advanced Search. When enabled, -Advanced Search provides faster search response times and [improved search features](../../user/search/advanced_search.md). +This page describes how to enable advanced search. When enabled, +advanced search provides faster search response times and [improved search features](../../user/search/advanced_search.md). ## Version requirements @@ -16,7 +16,7 @@ Advanced Search provides faster search response times and [improved search featu > Support for Elasticsearch 6.8 was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350275) in GitLab 15.0. -Advanced Search works with the following versions of Elasticsearch. +Advanced search works with the following versions of Elasticsearch. | GitLab version | Elasticsearch version | |-----------------------|--------------------------| @@ -25,13 +25,13 @@ Advanced Search works with the following versions of Elasticsearch. | GitLab 13.3 - 13.8 | Elasticsearch 6.4 - 7.x | | GitLab 12.7 - 13.2 | Elasticsearch 6.x - 7.x | -Advanced Search follows the [Elasticsearch end-of-life policy](https://www.elastic.co/support/eol). +Advanced search follows the [Elasticsearch end-of-life policy](https://www.elastic.co/support/eol). When we change Elasticsearch supported versions in GitLab, we announce them in [deprecation notes](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations) in monthly release posts before we remove them. ### OpenSearch version requirements -| GitLab version | Elasticsearch version | +| GitLab version | OpenSearch version | |-----------------------|--------------------------| | GitLab 15.0 or later | OpenSearch 1.x or later | @@ -46,7 +46,7 @@ If you are using a compatible version and after connecting to OpenSearch, you ge Elasticsearch requires additional resources to those documented in the [GitLab system requirements](../../install/requirements.md). -Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221177) in GitLab 13.10) uses the total repository size to estimate the Advanced Search storage requirements. +Memory, CPU, and storage resource amounts vary depending on the amount of data you index into the Elasticsearch cluster. Heavily used Elasticsearch clusters may require more resources. The [`estimate_cluster_size`](#gitlab-advanced-search-rake-tasks) Rake task ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221177) in GitLab 13.10) uses the total repository size to estimate the advanced search storage requirements. ## Install Elasticsearch @@ -163,20 +163,22 @@ Errors from the [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/git the [`elasticsearch.log`](../../administration/logs/index.md#elasticsearchlog) file and the [`sidekiq.log`](../../administration/logs/index.md#sidekiqlog) file with a `json.exception.class` of `Gitlab::Elastic::Indexer::Error`. These errors may occur when indexing Git repository data. -## Enable Advanced Search +## Enable advanced search + +Prerequisite: -For GitLab instances with more than 50 GB repository data you can follow the instructions for [how to index large instances efficiently](#how-to-index-large-instances-efficiently) below. +- You must have administrator access to the instance. -To enable Advanced Search, you must have administrator access to GitLab: +To enable advanced search: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Advanced Search**. NOTE: - To see the Advanced Search section, you need an active GitLab Premium + To see the **Advanced Search** section, you need an active GitLab Premium [license](../../user/admin_area/license.md). -1. Configure the [Advanced Search settings](#advanced-search-configuration) for +1. Configure the [advanced search settings](#advanced-search-configuration) for your Elasticsearch cluster. Do not enable **Search with Elasticsearch enabled** yet. 1. Enable **Elasticsearch indexing** and select **Save changes**. This creates @@ -202,7 +204,9 @@ you might have problems updating documents such as issues because your instance queues a job to index the change, but cannot find a valid Elasticsearch cluster. -### Advanced Search configuration +For GitLab instances with more than 50 GB of repository data, see [How to index large instances efficiently](#how-to-index-large-instances-efficiently). + +### Advanced search configuration The following Elasticsearch settings are available: @@ -223,8 +227,8 @@ The following Elasticsearch settings are available: | `AWS Secret Access Key` | The AWS secret access key. | | `Maximum file size indexed` | See [the explanation in instance limits.](../../administration/instance_limits.md#maximum-file-size-indexed). | | `Maximum field length` | See [the explanation in instance limits.](../../administration/instance_limits.md#maximum-field-length). | -| `Maximum bulk request size (MiB)` | Used by the GitLab Ruby and Golang-based indexer processes. This setting indicates how much data must be collected (and stored in memory) in a given indexing process before submitting the payload to the Elasticsearch Bulk API. For the GitLab Golang-based indexer, you should use this setting with `Bulk request concurrency`. `Maximum bulk request size (MiB)` must accommodate the resource constraints of both the Elasticsearch hosts and the hosts running the GitLab Golang-based indexer from either the `gitlab-rake` command or the Sidekiq tasks. | -| `Bulk request concurrency` | The Bulk request concurrency indicates how many of the GitLab Golang-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to the Elasticsearch Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch hosts and the hosts running the GitLab Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | +| `Maximum bulk request size (MiB)` | Used by the GitLab Ruby and Go-based indexer processes. This setting indicates how much data must be collected (and stored in memory) in a given indexing process before submitting the payload to the Elasticsearch Bulk API. For the GitLab Go-based indexer, you should use this setting with `Bulk request concurrency`. `Maximum bulk request size (MiB)` must accommodate the resource constraints of both the Elasticsearch hosts and the hosts running the GitLab Go-based indexer from either the `gitlab-rake` command or the Sidekiq tasks. | +| `Bulk request concurrency` | The Bulk request concurrency indicates how many of the GitLab Go-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to the Elasticsearch Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch hosts and the hosts running the GitLab Go-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. | | `Client request timeout` | Elasticsearch HTTP client request timeout value in seconds. `0` means using the system default timeout value, which depends on the libraries that GitLab application is built upon. | WARNING: @@ -294,16 +298,16 @@ For more information, see [Identity and Access Management in Amazon OpenSearch S When using fine-grained access control with a user in the internal database, you should use HTTP basic authentication to connect to OpenSearch. You can provide the master username and password as part of the -OpenSearch URL or in the **Username** and **Password** text boxes in the Advanced Search settings. See +OpenSearch URL or in the **Username** and **Password** text boxes in the advanced search settings. See [Tutorial: Internal user database and HTTP basic authentication](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-walkthrough-basic.html) for details. ##### Connecting with an IAM user -When using fine-grained access control with IAM credentials, you can provide the credentials in the **AWS OpenSearch IAM credentials** section in the Advanced Search settings. +When using fine-grained access control with IAM credentials, you can provide the credentials in the **AWS OpenSearch IAM credentials** section in the advanced search settings. ##### Permissions for fine-grained access control -The following permissions are required for Advanced Search. See [Creating roles](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html#fgac-roles) for details. +The following permissions are required for advanced search. See [Creating roles](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html#fgac-roles) for details. ```json { @@ -338,7 +342,7 @@ The following permissions are required for Advanced Search. See [Creating roles] } ``` -The index pattern `*` requires a few permissions for Advanced Search to work. +The index pattern `*` requires a few permissions for advanced search to work. ### Limit the number of namespaces and projects that can be indexed @@ -350,14 +354,14 @@ under **Elasticsearch indexing restrictions** more options become available. You can select namespaces and projects to index exclusively. If the namespace is a group, it includes any subgroups and projects belonging to those subgroups to be indexed as well. -Advanced Search only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search does not provide a code or commit scope. This is possible only in the scope of an indexed namespace. There is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. +Advanced search only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search does not provide a code or commit scope. This is possible only in the scope of an indexed namespace. There is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second. You can filter the selection dropdown list by writing part of the namespace or project name you're interested in.  NOTE: -If no namespaces or projects are selected, no Advanced Search indexing takes place. +If no namespaces or projects are selected, no advanced search indexing takes place. WARNING: If you have already indexed your instance, you must regenerate the index to delete all existing data @@ -385,11 +389,11 @@ For guidance on what to install, see the following Elasticsearch language plugin | Parameter | Description | |-------------------------------------------------------|-------------| | `Enable Chinese (smartcn) custom analyzer: Indexing` | Enables or disables Chinese language support using [`smartcn`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html) custom analyzer for newly created indices.| -| `Enable Chinese (smartcn) custom analyzer: Search` | Enables or disables using [`smartcn`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html) fields for Advanced Search. Only enable this after [installing the plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html), enabling custom analyzer indexing and recreating the index.| +| `Enable Chinese (smartcn) custom analyzer: Search` | Enables or disables using [`smartcn`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html) fields for advanced search. Only enable this after [installing the plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html), enabling custom analyzer indexing and recreating the index.| | `Enable Japanese (kuromoji) custom analyzer: Indexing` | Enables or disables Japanese language support using [`kuromoji`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html) custom analyzer for newly created indices.| -| `Enable Japanese (kuromoji) custom analyzer: Search` | Enables or disables using [`kuromoji`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html) fields for Advanced Search. Only enable this after [installing the plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html), enabling custom analyzer indexing and recreating the index.| +| `Enable Japanese (kuromoji) custom analyzer: Search` | Enables or disables using [`kuromoji`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html) fields for advanced search. Only enable this after [installing the plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html), enabling custom analyzer indexing and recreating the index.| -## Disable Advanced Search +## Disable advanced search To disable the Elasticsearch integration: @@ -423,7 +427,7 @@ the writes to the `primary` index. Then, we create another index and invoke the index data onto the new index. After the reindexing job is complete, we switch to the new index by connecting the index alias to it which becomes the new `primary` index. At the end, we resume the writes and typical operation resumes. -### Trigger the reindex via the Advanced Search administration +### Trigger the reindex via the advanced search administration > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34069) in GitLab 13.2. > - A scheduled index deletion and the ability to cancel it was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38914) in GitLab 13.3. @@ -502,15 +506,15 @@ Sometimes, you might want to abandon the unfinished reindex job and resume the i 1. Expand **Advanced Search**. 1. Clear the **Pause Elasticsearch indexing** checkbox. -## Advanced Search migrations +## Advanced search migrations > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6. With reindex migrations running in the background, there's no need for a manual intervention. This usually happens in situations where new features are added to -Advanced Search, which means adding or changing the way content is indexed. +advanced search, which means adding or changing the way content is indexed. -To confirm that the Advanced Search migrations ran, you can check with: +To confirm that the advanced search migrations ran, you can check with: ```shell curl "$CLUSTER_URL/gitlab-production-migrations/_search?q=*" | jq . @@ -554,7 +558,7 @@ To debug issues with the migrations you can check the [`elasticsearch.log` file] ### Retry a halted migration Some migrations are built with a retry limit. If the migration cannot finish within the retry limit, -it is halted and a notification is displayed in the Advanced Search integration settings. +it is halted and a notification is displayed in the advanced search integration settings. It is recommended to check the [`elasticsearch.log` file](../../administration/logs/index.md#elasticsearchlog) to debug why the migration was halted and make any changes before retrying the migration. Once you believe you've fixed the cause of the failure, select "Retry migration", and the migration is scheduled to be retried @@ -575,7 +579,7 @@ version. If you have halted migrations, these need to be resolved and [retried](#retry-a-halted-migration) before proceeding with a major version upgrade. Read more about [upgrading to a new major version](../../update/index.md#upgrading-to-a-new-major-version). -## GitLab Advanced Search Rake tasks +## GitLab advanced search Rake tasks Rake tasks are available to: @@ -587,7 +591,7 @@ The following are some available Rake tasks: | Task | Description | |:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`sudo gitlab-rake gitlab:elastic:info`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Outputs debugging information for the Advanced Search integration. | +| [`sudo gitlab-rake gitlab:elastic:info`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Outputs debugging information for the advanced search integration. | | [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, `gitlab:elastic:index_snippets`, and `gitlab:elastic:index_users`. | | [`sudo gitlab-rake gitlab:elastic:pause_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Pauses Elasticsearch indexing. Changes are still tracked. Useful for cluster/index migrations. | | [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Resumes Elasticsearch indexing. | @@ -604,7 +608,7 @@ The following are some available Rake tasks: | [`sudo gitlab-rake gitlab:elastic:mark_reindex_failed`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Mark the most recent re-index job as failed. | | [`sudo gitlab-rake gitlab:elastic:list_pending_migrations`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | List pending migrations. Pending migrations include those that have not yet started, have started but not finished, and those that are halted. | | [`sudo gitlab-rake gitlab:elastic:estimate_cluster_size`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Get an estimate of cluster size based on the total repository size. | -| [`sudo gitlab-rake gitlab:elastic:enable_search_with_elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enable advanced search with Elasticsearch. | +| [`sudo gitlab-rake gitlab:elastic:enable_search_with_elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables advanced search with Elasticsearch. | | [`sudo gitlab-rake gitlab:elastic:disable_search_with_elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Disables advanced search with Elasticsearch. | ### Environment variables @@ -633,7 +637,7 @@ Indexing project repositories...I, [2019-03-04T21:27:03.083410 #3384] INFO -- : I, [2019-03-04T21:27:05.215266 #3384] INFO -- : Indexing GitLab User / test (ID=33) is done! ``` -## Advanced Search index scopes +## Advanced search index scopes When performing a search, the GitLab index uses the following scopes: @@ -658,6 +662,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic - Generally, you want to use at least a 2-node cluster configuration with one replica, which allows you to have resilience. If your storage usage is growing quickly, you may want to plan horizontal scaling (adding more nodes) beforehand. - It's not recommended to use HDD storage with the search cluster, because it takes a hit on performance. It's better to use SSD storage (NVMe or SATA SSD drives for example). +- You should not use [coordinating-only nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-only-node) with large instances. Coordinating-only nodes are smaller than [data nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-node), which can impact performance and advanced search migrations. - You can use the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) to benchmark search performance with different search cluster sizes and configurations. - `Heap size` should be set to no more than 50% of your physical RAM. Additionally, it shouldn't be set to more than the threshold for zero-based compressed oops. The exact threshold varies, but 26 GB is safe on most systems, but can also be as large as 30 GB on some systems. See [Heap size settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#heap-size-settings) and [Setting JVM options](https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html) for more details. - Number of CPUs (CPU cores) per node usually corresponds to the `Number of Elasticsearch shards` setting described below. @@ -671,7 +676,7 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic - `refresh_interval` is a per index setting. You may want to adjust that from default `1s` to a bigger value if you don't need data in real-time. This changes how soon you see fresh results. If that's important for you, you should leave it as close as possible to the default value. - You might want to raise [`indices.memory.index_buffer_size`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html) to 30% or 40% if you have a lot of heavy indexing operations. -### Advanced Search integration settings guidance +### Advanced search integration settings guidance - The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you benefit from having at least 3*4=12 shards in the cluster. It's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards. - The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard has 1 replica). Using `0` is not recommended, because losing one node corrupts the index. @@ -862,6 +867,10 @@ However, some larger installations may wish to tune the merge policy settings: ## Index large instances with dedicated Sidekiq nodes or processes +WARNING: +Most instances should not need to configure this. The steps below use an advanced setting of Sidekiq called [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules). +Be sure to fully understand about the implication of using routing rules to avoid losing jobs entirely. + Indexing a large instance can be a lengthy and resource-intensive process that has the potential of overwhelming Sidekiq nodes and processes. This negatively affects the GitLab performance and availability. @@ -871,18 +880,20 @@ additional process dedicated to indexing a set of queues (or queue group). This ensure that indexing queues always have a dedicated worker, while the rest of the queues have another dedicated worker to avoid contention. -For this purpose, use the [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors) -option that allows a more general selection of queue groups using a [worker matching query](../../administration/sidekiq/processing_specific_job_classes.md#worker-matching-query). +For this purpose, use the [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) +option that allows Sidekiq to route jobs to a specific queue based on [worker matching query](../../administration/sidekiq/processing_specific_job_classes.md#worker-matching-query). -To handle these two queue groups, we generally recommend one of the following two options. You can either: +To handle this, we generally recommend one of the following two options. You can either: - [Use two queue groups on one single node](#single-node-two-processes). - [Use two queue groups, one on each node](#two-nodes-one-process-for-each). -For the steps below, consider: +For the steps below, consider the entry of `sidekiq['routing_rules']`: + +- `["feature_category=global_search", "global_search"]` as all indexing jobs are routed to the `global_search` queue. +- `["*", "default"]` as all other non-indexing jobs are routed to the `default` queue. -- `feature_category=global_search` as an indexing queue group with its own Sidekiq process. -- `feature_category!=global_search` as a non-indexing queue group that has its own Sidekiq process. +Note that at least one process in `sidekiq['queue_groups']` has to include the `mailers` queue, otherwise mailers jobs are not processed at all. ### Single node, two processes @@ -892,11 +903,20 @@ To create both an indexing and a non-indexing Sidekiq process in one node: ```ruby sidekiq['enable'] = true - sidekiq['queue_selector'] = true + sidekiq['queue_selector'] = false + + sidekiq['routing_rules'] = [ + ["feature_category=global_search", "global_search"], + ["*", "default"], + ] + sidekiq['queue_groups'] = [ - "feature_category=global_search", - "feature_category!=global_search" - ] + "global_search", # process that listens to global_search queue + "default,mailers" # process that listens to default and mailers queue + ] + + sidekiq['min_concurrency'] = 20 + sidekiq['max_concurrency'] = 20 ``` 1. Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) @@ -914,26 +934,42 @@ To handle these queue groups on two nodes: 1. To set up the indexing Sidekiq process, on your indexing Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: - ```ruby - sidekiq['enable'] = true - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category=global_search" - ] - ``` + ```ruby + sidekiq['enable'] = true + sidekiq['queue_selector'] = false + + sidekiq['routing_rules'] = [ + ["feature_category=global_search", "global_search"], + ["*", "default"], + ] + + sidekiq['queue_groups'] = [ + "global_search", # process that listens to global_search queue + ] + + sidekiq['min_concurrency'] = 20 + sidekiq['max_concurrency'] = 20 + ``` 1. Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) for the changes to take effect. 1. To set up the non-indexing Sidekiq process, on your non-indexing Sidekiq node, change the `/etc/gitlab/gitlab.rb` file to: - ```ruby - sidekiq['enable'] = true - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category!=global_search" - ] - ``` + ```ruby + sidekiq['enable'] = true + sidekiq['routing_rules'] = [ + ["feature_category=global_search", "global_search"], + ["*", "default"], + ] + + sidekiq['queue_groups'] = [ + "default,mailers" # process that listens to default and mailers queue + ] + + sidekiq['min_concurrency'] = 20 + sidekiq['max_concurrency'] = 20 + ``` to set up a non-indexing Sidekiq process. @@ -945,7 +981,7 @@ for the changes to take effect. Sometimes there may be issues with your Elasticsearch index data and as such GitLab allows you to revert to "basic search" when there are no search results and assuming that basic search is supported in that scope. This "basic -search" behaves as though you don't have Advanced Search enabled at all for +search" behaves as though you don't have advanced search enabled at all for your instance and search using other data sources (such as PostgreSQL data and Git data). diff --git a/doc/integration/advanced_search/elasticsearch_troubleshooting.md b/doc/integration/advanced_search/elasticsearch_troubleshooting.md index c8d11286a3c..0c4895f34fa 100644 --- a/doc/integration/advanced_search/elasticsearch_troubleshooting.md +++ b/doc/integration/advanced_search/elasticsearch_troubleshooting.md @@ -101,12 +101,12 @@ Here are some common pitfalls and how to overcome them. There are a couple of ways to achieve that: - When you perform a search, in the upper-right corner of the search results page, - **Advanced search functionality is enabled** is displayed. + **Advanced search is enabled** is displayed. This is always correctly identifying whether the current project/namespace being searched is using Elasticsearch. - From the Admin Area under **Settings > Advanced Search** check that the - Advanced Search settings are checked. + advanced search settings are checked. Those same settings there can be obtained from the Rails console if necessary: @@ -212,7 +212,7 @@ More [complex Elasticsearch API calls](https://www.elastic.co/guide/en/elasticse If the results: -- Sync up, check that you are using [supported syntax](../../user/search/advanced_search.md#syntax). Advanced Search does not support [exact substring matching](https://gitlab.com/gitlab-org/gitlab/-/issues/325234). +- Sync up, check that you are using [supported syntax](../../user/search/advanced_search.md#syntax). Advanced search does not support [exact substring matching](https://gitlab.com/gitlab-org/gitlab/-/issues/325234). - Do not match up, this indicates a problem with the documents generated from the project. It is best to [re-index that project](../advanced_search/elasticsearch.md#indexing-a-range-of-projects-or-a-specific-project). NOTE: @@ -247,8 +247,8 @@ sudo gitlab-rake gitlab:elastic:clear_locked_projects If `ElasticCommitIndexerWorker` Sidekiq workers are failing with this error during indexing, it usually means that Elasticsearch is unable to keep up with the concurrency of indexing request. To address change the following settings: -- To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced Search settings](elasticsearch.md#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations. -- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests. +- To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced search settings](elasticsearch.md#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations. +- If changing `Bulk request concurrency` didn't help, you can use the [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests. ### Indexing is very slow or fails with `rejected execution of coordinating operation` messages @@ -437,11 +437,11 @@ Improvements to the `code_analyzer` pattern and filters are being discussed in [ In GitLab 13.9, a change was made where [binary file names are being indexed](https://gitlab.com/gitlab-org/gitlab/-/issues/301083). However, without indexing all projects' data from scratch, only binary files that are added or updated after the GitLab 13.9 release are searchable. -## How does Advanced Search handle private projects? +## How does advanced search handle private projects? -Advanced Search stores all the projects in the same Elasticsearch indices, +Advanced search stores all the projects in the same Elasticsearch indices, however, searches only surface results that can be viewed by the user. -Advanced Search honors all permission checks in the application by +Advanced search honors all permission checks in the application by filtering out projects that a user does not have access to at search time. ### Role mapping when using fine-grained access control with AWS Elasticsearch or OpenSearch diff --git a/doc/integration/datadog.md b/doc/integration/datadog.md index 8b64e3898f9..2c86b7a8ed3 100644 --- a/doc/integration/datadog.md +++ b/doc/integration/datadog.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270123) in GitLab 14.1 -This integration enables you to send CI/CD pipeline and job information to +The Datadog integration enables you to send CI/CD pipeline and job information to [Datadog](https://www.datadoghq.com/). The [Datadog CI Visibility](https://app.datadoghq.com/ci) product helps you monitor for job failures and performance issues, then troubleshoot them. It's based on [Webhooks](../user/project/integrations/webhooks.md), diff --git a/doc/integration/ding_talk.md b/doc/integration/ding_talk.md index ca939dc9f9a..cb7ba6849b9 100644 --- a/doc/integration/ding_talk.md +++ b/doc/integration/ding_talk.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # DingTalk OAuth 2.0 OmniAuth provider **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341898) in GitLab 14.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341898) in GitLab 14.5. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390855) in GitLab 15.10. You can sign in to GitLab using your DingTalk account. Sign in to DingTalk Open Platform and create an application on it. DingTalk generates a client ID and secret key for you to use. diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index a3c206176b9..355be006fe8 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# External issue tracker **(FREE)** +# External issue trackers **(FREE)** GitLab has an [issue tracker](../user/project/issues/index.md), but you can configure an external issue tracker per GitLab project. diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index 8b7bdeaa177..b35e3c585e0 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -31,7 +31,7 @@ Facebook. Facebook generates an app ID and secret key for you to use. 1. Choose "Next" -1. Choose "Skip Quick Start" in the upper right corner +1. In the upper-right corner, select **Skip Quick Start**. 1. Choose "Settings" in the menu on the left diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index ee8f46e73df..6df36cff638 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -69,6 +69,6 @@ You can launch Gitpod directly from GitLab in one of these ways: 1. Select **Open in Gitpod**. - *From a merge request:* 1. Go to your merge request. - 1. In the upper right corner, select **Code**, then select **Open in Gitpod**. + 1. In the upper-right corner, select **Code**, then select **Open in Gitpod**. Gitpod builds your development environment for your branch. diff --git a/doc/integration/glab/index.md b/doc/integration/glab/index.md index 621472a2083..427751a0297 100644 --- a/doc/integration/glab/index.md +++ b/doc/integration/glab/index.md @@ -43,19 +43,24 @@ glab mr merge ## Core commands -- `glab alias` -- `glab api` -- `glab auth` -- `glab ci` -- `glab issue` -- `glab label` -- `glab mr` -- `glab project` -- `glab release` -- `glab snippet` -- `glab ssh-key` -- `glab user` -- `glab variable` +- [`glab alias`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/alias) +- [`glab api`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/api) +- [`glab auth`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/auth) +- [`glab check-update`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/check-update) +- [`glab ci`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/ci) +- [`glab completion`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/completion) +- [`glab config`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/config) +- [`glab incident`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/incident) +- [`glab issue`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/ci) +- [`glab label`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/label) +- [`glab mr`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/mr) +- [`glab release`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/release) +- [`glab repo`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/repo) +- [`glab schedule`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/schedule) +- [`glab snippet`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/snippet) +- [`glab ssh-key`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/ssh-key) +- [`glab user`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/user) +- [`glab variable`](https://gitlab.com/gitlab-org/cli/-/tree/main/docs/source/variable) ## Install the CLI @@ -69,7 +74,7 @@ To authenticate with your GitLab account, run `glab auth login`. ## Report issues -Open an issue in the [`gitlab-org/cli` repository](https://gitlab.com/gitlab-org/cli/issues/new) +Open an issue in the [`gitlab-org/cli` repository](https://gitlab.com/gitlab-org/cli/-/issues/new) to send us feedback. ## Related topics diff --git a/doc/integration/gmail_action_buttons_for_gitlab.md b/doc/integration/gmail_action_buttons_for_gitlab.md index 4a233df3899..6de7e7ab17c 100644 --- a/doc/integration/gmail_action_buttons_for_gitlab.md +++ b/doc/integration/gmail_action_buttons_for_gitlab.md @@ -7,8 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Gmail actions **(FREE)** GitLab supports [Google actions in email](https://developers.google.com/gmail/markup/actions/actions-overview). - -If correctly set up, emails that require an action are marked in Gmail. +When you configure this integration, emails that require an action are marked in Gmail.  diff --git a/doc/integration/google.md b/doc/integration/google.md index 5eac639f119..bb76d7c8ff1 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -22,7 +22,7 @@ In Google's side: the randomly generated ID or choose a new one. 1. Refresh the page and you should see your new project in the list 1. Go to the [Google API Console](https://console.developers.google.com/apis/dashboard) -1. Select the previously created project in the upper left corner +1. In the upper-left corner, select the previously created project 1. Select **Credentials** from the sidebar 1. Select **OAuth consent screen** and fill the form with the required information 1. In the **Credentials** tab, select **Create credentials > OAuth client ID** diff --git a/doc/integration/index.md b/doc/integration/index.md index 195890ea4d8..02860339b6f 100644 --- a/doc/integration/index.md +++ b/doc/integration/index.md @@ -73,7 +73,7 @@ You can integrate GitLab with the following feature enhancements: or [Kroki](../administration/integration/kroki.md) to use diagrams in AsciiDoc and Markdown documents. - Attach merge requests to [Trello](trello_power_up.md) cards. - Enable integrated code intelligence powered by [Sourcegraph](sourcegraph.md). -- Add [Elasticsearch](advanced_search/elasticsearch.md) for [Advanced Search](../user/search/advanced_search.md). +- Add [Elasticsearch](advanced_search/elasticsearch.md) for [advanced search](../user/search/advanced_search.md). ## Troubleshooting diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 983ff3c54bc..668038ef386 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -166,7 +166,7 @@ If you get this error message while configuring GitLab, the following are possib - GitLab is unable to reach your Jenkins instance at the address. If your GitLab instance is self-managed, try pinging the Jenkins instance at the domain provided on the GitLab instance. - The Jenkins instance is at a local address and is not included in the - [GitLab installation's allowlist](../security/webhooks.md#create-an-allowlist-for-local-requests). + [GitLab installation's allowlist](../security/webhooks.md#allow-outbound-requests-to-certain-ip-addresses-and-domains). - The credentials for the Jenkins instance do not have sufficient access or are invalid. - The **Enable authentication for `/project` end-point** checkbox is not selected in your [Jenkins plugin configuration](#configure-the-jenkins-server). diff --git a/doc/integration/jira/configure.md b/doc/integration/jira/configure.md index 03d742703a1..3ef4dfac3f4 100644 --- a/doc/integration/jira/configure.md +++ b/doc/integration/jira/configure.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Configure the Jira integration **(FREE)** +# Configure Jira **(FREE)** You can set up the [Jira integration](index.md#jira-integration) by configuring your project settings in GitLab. @@ -13,10 +13,10 @@ and for self-managed GitLab, at an [instance level](../../user/admin_area/settin Prerequisites: -- Ensure your GitLab installation does not use a [relative URL](development_panel.md#limitations). +- Ensure your GitLab installation does not use a [relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configure-a-relative-url-for-gitlab). - For **Jira Server**, ensure you have a Jira username and password. See [authentication in Jira](index.md#authentication-in-jira). -- For **Jira on Atlassian cloud**, ensure you have an API token +- For **Jira Cloud**, ensure you have an API token and the email address you used to create the token. See [authentication in Jira](index.md#authentication-in-jira). diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 402efc409cb..8bbac021849 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -27,8 +27,8 @@ at least the Maintainer role in the GitLab.com namespace. This integration method supports [Smart Commits](dvcs/index.md#smart-commits). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a walkthrough of the integration with GitLab for Jira Cloud app, watch -[Configure GitLab.com Jira Could Integration using Marketplace App](https://youtu.be/SwR-g1s1zTo) on YouTube. +For an overview, see +[Configure the GitLab for Jira Cloud app from the Atlassian Marketplace](https://youtu.be/SwR-g1s1zTo). To install the GitLab for Jira Cloud app: @@ -117,7 +117,7 @@ Jira apps can only link to one URL per marketplace listing. The official listing If your instance doesn't meet the prerequisites or you don't want to use the official marketplace listing, you can [install the app manually](#install-the-gitlab-for-jira-cloud-app-manually). -It's not possible to create branches from Jira for self-managed instances. +It's not possible to create branches from Jira for self-managed instances. For more information, see [issue 391432](https://gitlab.com/gitlab-org/gitlab/-/issues/391432). ### Set up your instance @@ -285,3 +285,29 @@ To resolve this issue on GitLab self-managed, follow one of the solutions below, - Contact the [Jira Software Cloud support](https://support.atlassian.com/jira-software-cloud/) and ask to trigger a new installed lifecycle event for the GitLab for Jira Cloud app in your namespace. - In all GitLab versions: - Re-install the GitLab for Jira Cloud app. This might remove all already synced development panel data. + +## Security considerations + +The GitLab for Jira Cloud app connects GitLab and Jira, as data must be shared between the two applications and access must be granted in both directions. + +## Access to GitLab through OAuth + +GitLab does not share an access token with Jira. However, users must authenticate via OAuth to configure the app. + +An access token is retrieved via [PKCE](https://www.rfc-editor.org/rfc/rfc7636) OAuth flow, and stored only on the client side. +The app-frontend that initializes the OAuth flow is a JavaScript application, which is loaded from GitLab through an iframe on Jira. + +The OAuth application requires the `api` scope that grants complete read/write access to the API, including to all groups and projects, the container registry, and the package registry. +However, the GitLab for Jira Cloud app only uses this access to: + +- Display namespaces to be linked. +- Link namespaces. + +Access through OAuth is only needed for the time a user configures the GitLab for Jira Cloud app. For more information, see [Access token expiration](../oauth_provider.md#access-token-expiration). + +## Access to Jira through access token + +Jira shares an access token with GitLab to authenticate and authorize data pushes to Jira. +As part of the app installation process, Jira sends a handshake request to GitLab containing the access token. +The handshake is signed with an [asymmetric JWT](https://developer.atlassian.com/cloud/jira/platform/understanding-jwt-for-connect-apps/) +and the access token is stored encrypted with `AES256-GCM` on GitLab. diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md index f36b9f2c4c1..57308c9a19e 100644 --- a/doc/integration/jira/development_panel.md +++ b/doc/integration/jira/development_panel.md @@ -4,99 +4,94 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Jira development panel integration **(FREE)** +# Jira development panel **(FREE)** > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) from GitLab Premium to GitLab Free in 13.4. -With the Jira development panel integration, you can reference Jira issues in GitLab. -When configured, activity (such as pipeline, deployment, and feature flags) displays in the Jira issue's -[development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). -From the development panel, you can open a detailed view and -[take various actions](#use-the-integration), including creating a new merge request from a branch: +You can view GitLab activity from the Jira development panel. - +When you're in GitLab, you can refer to a Jira issue by ID. +The [activity for that issue](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/) +is displayed in the Jira development panel. -The information displayed in the Jira development panel depends on where you mention the Jira issue ID: +In the Jira development panel, you can create a GitLab merge request from a branch. +You can also create a GitLab branch from a Jira issue in the GitLab for Jira Cloud app +([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66032) in GitLab 14.2). -| Your mention of Jira issue ID in GitLab context | Automated effect in Jira issue | -|---------------------------------------------------|--------------------------------------------------------------------------------------------------------| -| In a merge request title or description | Link to the MR is displayed in the development panel. | -| In a branch name | Link to the branch is displayed in the development panel. | -| In a commit message | Link to the commit is displayed in the development panel. | -| In a commit message with Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html) | Displays your custom comment or logged time spent and/or performs specified issue transition on merge. | +## Connected projects in GitLab -This integration connects all GitLab projects to projects in the Jira instance in either: +The Jira development panel connects to the entire Jira instance all GitLab projects in: -- A top-level GitLab group: Connects the projects in a group with no parent group, - including the projects in its subgroups. -- A personal namespace: Connects the projects in that personal namespace to Jira. +- A top-level group, including all projects in its subgroups. +- A personal namespace. -## Use the integration +These GitLab projects can interact with all Jira projects in that instance. -After the integration is [set up on GitLab and Jira](#configure-the-integration), you can: +## Information displayed in the panel -- Refer to any Jira issue by its ID (in uppercase) in GitLab branch names, - commit messages, and merge request titles. -- See the linked branches, commits, and merge requests in Jira issues. -- Create GitLab branches from Jira Cloud issues ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66032) in GitLab 14.2 for the GitLab for Jira app). +The information displayed in the Jira development panel depends on where you mention the Jira issue ID in GitLab. -At this time, merge requests are called "pull requests" in Jira issues. -This name may change in a future Jira release. +| GitLab: where you mention the Jira issue ID | Jira development panel: what information is displayed | +|------------------------------------------------|-------------------------------------------------------| +| Merge request title or description | Link to the merge request | +| Branch name | Link to the branch | +| Commit message | Link to the commit | +| [Jira Smart Commit](#jira-smart-commits) | Custom comment, logged time, or workflow transition | -Select the links to see your GitLab repository data. +## Jira Smart Commits - +Jira Smart Commits are special commands to process a Jira issue. With these commands, you can use GitLab to: - +- Add a custom comment to a Jira issue. +- Log time against a Jira issue. +- Transition a Jira issue to any status defined in the project workflow. -### Use Jira Smart Commits +For more information, see the +[Atlassian documentation](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html). -With Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html), -you can use GitLab to add Jira comments, log time spent on the issue, or apply any issue transition. - -For more information about using Jira Smart Commits to track time against an issue, specify -an issue transition, or add a custom comment, read the Atlassian page -[Using Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html). - -## Configure the integration +## Configure the panel <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of how to configure the Jira development panel integration, see -[Agile Management - GitLab Jira development panel integration](https://www.youtube.com/watch?v=VjVTOmMl85M). +For an overview, see [Jira development panel integration](https://www.youtube.com/watch?v=VjVTOmMl85M). + +### For GitLab.com -To simplify administration, we recommend that a GitLab group maintainer or group owner -(or, if possible, instance administrator in the case of self-managed GitLab) set up the integration. +Prerequisite: -| Jira usage | GitLab.com customers need | GitLab self-managed customers need | -|------------|---------------------------|------------------------------------| -| [Atlassian cloud](https://www.atlassian.com/migration/assess/why-cloud) | The [GitLab for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview) from the [Atlassian Marketplace](https://marketplace.atlassian.com). This method offers real-time sync between GitLab.com and Jira. The method requires inbound connections for the setup and then pushes data to Jira through outbound connections. For more information, see [GitLab for Jira Cloud app](connect-app.md). | The GitLab for Jira Cloud app [installed manually](connect-app.md#install-the-gitlab-for-jira-cloud-app-manually). By default, you can install the app from the [Atlassian Marketplace](https://marketplace.atlassian.com/). The method requires inbound connections for the setup and then pushes data to Jira through outbound connections. For more information, see [Connect the GitLab for Jira Cloud app for self-managed instances](connect-app.md#connect-the-gitlab-for-jira-cloud-app-for-self-managed-instances). | -| Your own server | The [Jira DVCS connector](dvcs/index.md). This method syncs data every hour and works only with inbound connections. The method tries to set up webhooks in GitLab to implement real-time data sync, which does not work without outbound connections. | The [Jira DVCS connector](dvcs/index.md). This method syncs data every hour and works only with inbound connections. The method tries to set up webhooks in GitLab to implement real-time data sync, which does not work without outbound connections. | +- You must have at least the Maintainer role for the group. -Each GitLab project can be configured to connect to an entire Jira instance. That means after -configuration, one GitLab project can interact with all Jira projects in that instance. For: +To configure the Jira development panel on GitLab.com: -- The [view Jira issues](issues.md#view-jira-issues) feature, you must associate a GitLab project with a - specific Jira project. -- Other features, you do not have to explicitly associate a GitLab project with any single Jira - project. +- **For [Jira Cloud](https://www.atlassian.com/migration/assess/why-cloud)**: + - [From the Atlassian Marketplace, install the GitLab for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-for-jira-cloud?hosting=cloud&tab=overview). + - This method syncs data between GitLab.com and Jira in real time. + - This method requires inbound connections for the setup and outbound connections to push data to Jira. + - For more information, see [GitLab for Jira Cloud app](connect-app.md). +- **For Jira Server**: + - Use the [Jira DVCS connector](dvcs/index.md). + - This method syncs data every hour and works only with inbound connections. + - This method attempts to set up webhooks in GitLab to sync data in real time, which requires outbound connections. -If you have a single Jira instance, you can pre-fill the settings. For more information, read the -documentation for [central administration of project integrations](../../user/admin_area/settings/project_integration_management.md). +### For self-managed GitLab -## Limitations +Prerequisites: -- This integration is not supported on GitLab instances under a -[relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configure-a-relative-url-for-gitlab) -(for example, `http://example.com/gitlab`). -- [Creating a branch](https://gitlab.com/gitlab-org/gitlab/-/issues/2647) is only supported by the GitLab for Jira app and is not available within the DVCS integration. See [officially supported DVCS features](https://confluence.atlassian.com/adminjiraserver/integrating-with-development-tools-938846890.html) for more information. +- You must have administrator access to the instance. +- Your GitLab installation must not use a [relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configure-a-relative-url-for-gitlab) + (for example, `https://example.com/gitlab`). -## Troubleshoot the development panel +To configure the Jira development panel on self-managed GitLab: -If you use Jira on your own server, go to the [Atlassian documentation](https://confluence.atlassian.com/jirakb/troubleshoot-the-development-panel-in-jira-server-574685212.html) -for general troubleshooting information. +- **For [Jira Cloud](https://www.atlassian.com/migration/assess/why-cloud)**: + - [Install the GitLab for Jira Cloud app manually](connect-app.md#install-the-gitlab-for-jira-cloud-app-manually). + - This method requires inbound connections for the setup and outbound connection to push data to Jira. + - For more information, see [Connect the GitLab for Jira Cloud app for self-managed instances](connect-app.md#connect-the-gitlab-for-jira-cloud-app-for-self-managed-instances). +- **For Jira Server**: + - Use the [Jira DVCS connector](dvcs/index.md). + - This method syncs data every hour and works only with inbound connections. + - This method attempts to set up webhooks in GitLab to sync data in real time, which requires outbound connections. -### Cookies for Oracle's Access Manager +## Troubleshooting -To support Oracle's Access Manager, GitLab sends additional cookies -to enable Basic Auth. The cookie being added to each request is `OBBasicAuth` with -a value of `fromDialog`. +To troubleshoot the Jira development panel on your own server, see the +[Atlassian documentation](https://confluence.atlassian.com/jirakb/troubleshoot-the-development-panel-in-jira-server-574685212.html). diff --git a/doc/integration/jira/img/jira_dev_panel_jira_setup_3.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_3.png Binary files differdeleted file mode 100644 index c58f1d5cecc..00000000000 --- a/doc/integration/jira/img/jira_dev_panel_jira_setup_3.png +++ /dev/null diff --git a/doc/integration/jira/img/jira_dev_panel_jira_setup_4.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_4.png Binary files differdeleted file mode 100644 index 81d84cb173d..00000000000 --- a/doc/integration/jira/img/jira_dev_panel_jira_setup_4.png +++ /dev/null diff --git a/doc/integration/jira/img/jira_dev_panel_jira_setup_5.png b/doc/integration/jira/img/jira_dev_panel_jira_setup_5.png Binary files differdeleted file mode 100644 index b143fc24355..00000000000 --- a/doc/integration/jira/img/jira_dev_panel_jira_setup_5.png +++ /dev/null diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index 2ad71a19cf7..3ec4c7aee87 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Jira integrations **(FREE)** +# Jira **(FREE)** If your organization uses [Jira](https://www.atlassian.com/software/jira) issues, you can [migrate your issues from Jira](../../user/project/import/jira.md) and work @@ -37,7 +37,7 @@ displays in the [development panel](https://support.atlassian.com/jira-software- To set up the Jira development panel integration, use the GitLab for Jira Cloud app or the Jira DVCS (distributed version control system) connector, -[depending on your installation](development_panel.md#configure-the-integration). +[depending on your installation](development_panel.md#configure-the-panel). ### Direct feature comparison @@ -63,8 +63,8 @@ The authentication method in Jira depends on whether you host Jira on your own s required. Connecting to Jira Server using the Central Authentication Service (CAS) is not possible. For more information, read how to [set up a user in Jira Server](jira_server_configuration.md). - **Jira on Atlassian cloud** supports authentication through an API token. When connecting to Jira on - Atlassian cloud, an email and API token are required. For more information, read - [create an API token for Jira in Atlassian cloud](jira_cloud_configuration.md). + Atlassian cloud, an email and API token are required. For more information, see + [Create a Jira Cloud API token](jira_cloud_configuration.md). ## Privacy considerations diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md index 3a5d8e66b2d..58da871926b 100644 --- a/doc/integration/jira/issues.md +++ b/doc/integration/jira/issues.md @@ -4,10 +4,10 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Jira integration issue management **(FREE)** +# Jira issue management **(FREE)** -Integrating issue management with Jira requires you to [configure Jira](index.md#jira-integration) -and [enable the Jira integration](configure.md) in GitLab. +To integrate issue management with Jira, [configure Jira](index.md#jira-integration) +and [enable the integration](configure.md) in GitLab. After you configure and enable the integration, you can reference and close Jira issues by mentioning the Jira ID in GitLab commits and merge requests. @@ -47,12 +47,11 @@ You can [disable comments](#disable-comments-on-jira-issues) on issues. ### Require associated Jira issue for merge requests to be merged **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/280766) in GitLab 13.12, disabled behind `jira_issue_association_on_merge_request` [feature flag](../../administration/feature_flags.md). -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61722) in GitLab 14.1. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/335280) in GitLab 14.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/280766) in GitLab 13.12 [with a flag](../../administration/feature_flags.md) named `jira_issue_association_on_merge_request`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/335280) in GitLab 14.2. Feature flag `jira_issue_association_on_merge_request` removed. -You can prevent merge requests from being merged if they do not refer to a Jira issue. -To enforce this: +With this integration, you can prevent merge requests from being merged if they do not refer to a Jira issue. +To enable this feature: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Merge requests**. diff --git a/doc/integration/jira/jira_server_configuration.md b/doc/integration/jira/jira_server_configuration.md index 7dee99b024e..5bac108b457 100644 --- a/doc/integration/jira/jira_server_configuration.md +++ b/doc/integration/jira/jira_server_configuration.md @@ -20,7 +20,7 @@ credentials, you must: This process creates a user named `gitlab`: 1. Sign in to your Jira instance as a Jira administrator. -1. In the upper right corner of the top menu, go to the gear icon and +1. On the top bar, in the upper-right corner, select the gear icon, then select **User Management**. 1. Create a new user account (`gitlab`) with write access to projects in Jira. @@ -43,9 +43,9 @@ group to assign permissions to the user. This process adds the `gitlab` user you created to a new group named `gitlab-developers`: 1. Sign in to your Jira instance as a Jira administrator. -1. In the upper right corner of the top menu, go to the gear icon and +1. On the top bar, in the upper-right corner, select the gear icon, then select **User Management**. -1. On the sidebar, select **Groups**. +1. On the left sidebar, select **Groups**.  @@ -70,9 +70,9 @@ Next, create a permission scheme for your group. After creating the group in Jira, grant permissions to the group by creating a permission scheme: 1. Sign in to your Jira instance as a Jira administrator. -1. In the upper right corner of the top menu, go to the gear icon and +1. On the top bar, in the upper-right corner, select the gear icon, then select **Issues**. -1. On the sidebar, select **Permission Schemes**. +1. On the left sidebar, select **Permission Schemes**. 1. Select **Add Permission Scheme**, enter a **Name** and (optionally) a **Description**, and then select **Add**. 1. In the permissions scheme list, locate your new permissions scheme, and diff --git a/doc/integration/jira/troubleshooting.md b/doc/integration/jira/troubleshooting.md index 0e679693614..bf347c4698d 100644 --- a/doc/integration/jira/troubleshooting.md +++ b/doc/integration/jira/troubleshooting.md @@ -91,16 +91,16 @@ To reset the Jira user's password for all projects with active Jira integrations run the following in a [Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session): ```ruby -p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN services s ON p.id = s.project_id WHERE s.type = 'JiraService' AND s.active = true") +p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN integrations i ON p.id = i.project_id WHERE i.type_new = 'Integrations::Jira' AND i.active = true") p.each do |project| project.jira_integration.update_attribute(:password, '<your-new-password>') end ``` -## `500 Whoops` when accessing a Jira issue in GitLab +## `500 We're sorry` when accessing a Jira issue in GitLab -When accessing a Jira issue in GitLab, you might get a `500 Whoops, something went wrong on our end` error. +When accessing a Jira issue in GitLab, you might get a `500 We're sorry. Something went wrong on our end` error. Check [`production.log`](../../administration/logs/index.md#productionlog) to see if it contains the following exception: ```plaintext diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 42592a0dff2..e1183f62225 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -10,7 +10,7 @@ NOTE: This document applies to GitLab 11.0 and later. You can run a [GitLab Mattermost](https://gitlab.com/gitlab-org/gitlab-mattermost) -service on your GitLab server. Mattermost is not part of the single application that GitLab is. There is a good integration between with GitLab, and our Omnibus installer allows you to easily install it. But it is a separate application from a separate company. +service on your GitLab server. Mattermost is not part of the single application that GitLab is. There is a good integration between [Mattermost and GitLab](https://mattermost.com/solutions/mattermost-gitlab/), and our Omnibus installer allows you to easily install it. But it is a separate application from a separate company. ## Prerequisites @@ -245,7 +245,7 @@ For local connections, the `mmctl` binary and Mattermost must be run from the sa "LocalModeSocketLocation": "/var/tmp/mattermost_local.socket", ... } - } + } ``` 1. Restart Mattermost: @@ -280,7 +280,7 @@ $ /opt/gitlab/embedded/bin/mmctl auth login http://mattermost.example.com Connection name: test Username: local-user -Password: +Password: credentials for "test": "local-user@http://mattermost.example.com" stored ``` @@ -378,6 +378,10 @@ If this is not the case, there are two options: For a complete list of upgrade notices and special considerations for older versions, see the [Mattermost documentation](https://docs.mattermost.com/administration/important-upgrade-notes.html). +## Upgrading GitLab Mattermost to 15.10 + +GitLab 15.10 ships with Mattermost 7.8. Before upgrading, [connect to the bundled PostgreSQL database](#connecting-to-the-bundled-postgresql-database) to perform the PostgreSQL maintenance described in the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) provided by Mattermost. + ## Upgrading GitLab Mattermost to 14.6 GitLab 14.6 ships with Mattermost 6.1 including potentially long running database migrations for Mattermost 6.0. For information about upgrading and for ways to reduce the downtime caused by those migrations, read the [Important Upgrade Notes](https://docs.mattermost.com/administration/important-upgrade-notes.html) for both versions. If you need to perform any manual migrations, [connect to the bundled PostgreSQL database](#connecting-to-the-bundled-postgresql-database). diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 61019915c52..66d4ccb871f 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -379,8 +379,12 @@ but we'd like to at least help those with specific needs. ## Keep OmniAuth user profiles up to date -You can enable profile syncing from selected OmniAuth providers. You can sync -all or specific user information. +You can enable profile syncing from selected OmniAuth providers. +You can sync any combination of the following user attributes: + +- `name` +- `email` +- `location` When authenticating using LDAP, the user's name and email are always synced. diff --git a/doc/integration/partner_marketplace.md b/doc/integration/partner_marketplace.md index a15457b5b0c..5ed131145f4 100644 --- a/doc/integration/partner_marketplace.md +++ b/doc/integration/partner_marketplace.md @@ -1,13 +1,13 @@ --- stage: Fulfillment -group: Commerce Integrations +group: Provision info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Marketplace partner integration guide +# Marketplace partners GitLab supports automation for selected distribution marketplaces to process sales of GitLab products to authorized -channel partners. Marketplace partners can use the GitLab Marketplace APIs to integrate their systems with GitLab to +channel partners. Marketplace partners can use the GitLab Marketplace APIs to integrate their systems with GitLab to sell GitLab subscriptions on their site. This document's target audience is third-party developers for Marketplace partners. @@ -48,17 +48,113 @@ sequenceDiagram Customers Portal ->> Marketplace partner system: Success response with order status ``` -## Prerequisites +## Marketplace API Specification + +OpenAPI specs for the Marketplace APIs are available at [Marketplace interactive API documentation](https://customers.staging.gitlab.com/openapi_docs/marketplace). + +## Access the Marketplace API + +To access the Marketplace API you need to: + +- Request access from GitLab. +- Retrieve an OAuth access token. + +Marketplace API endpoints are secured with [OAuth 2.0](https://oauth.net/2/). OAuth is an authorization framework +that grants 3rd party or client applications, like a GitLab Partner application, limited access to resources on an +HTTP service, like the Customers Portal. + +OAuth 2.0 uses _grant types_ (or _flows_) that describe how a client application gets authorization in +the form of an _access token_. An access token is a string that the client application uses to make authorized requests to +the resource server. + +The Marketplace API uses the `client_credentials` grant type. The client application uses the access token to access its +own resources, instead of accessing resources on behalf of a user. + +### Step 1: Request access -Before a marketplace partner client can use the Marketplace API, GitLab must set up the following configurations for the client: +Before you can use the Marketplace API, you must contact your GitLab Partner Manager or email [`partnerorderops`](mailto:partnerorderops@gitlab.com) +to request access. After you request access, GitLab configures the following accounts and credentials for you: -1. Client credential. Marketplace APIs are secured with OAuth 2.0. A client credential is required to get the OAuth token. +1. Client credentials. Marketplace APIs are secured with OAuth 2.0. The client credentials include the client ID and client secret + that you need to retrieve the OAuth access token. 1. Invoice owner account in Zuora system. Required for invoice processing. 1. Distributor account in Salesforce system. 1. Trading partner account in Salesforce system. -Interested GitLab Partners should contact their GitLab Partner Manager or email [`partnerorderops`](mailto:partnerorderops@gitlab.com). +### Step 2: Retrieve an access token -## Marketplace API Specification +To retrieve an access token, + +- Make a POST request to the [`/oauth/token`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/post_oauth_token) endpoint with the following required parameters: + +| Parameter | Type | Required |Description | +|-----------------|--------|----------|------------------------------------------------------------------------------------------------------------------------------------| +| `client_id` | string | yes |ID of your client application record on the Customers Portal. Received from GitLab. | +| `client_secret` | string | yes |Secret of your client application record on the Customers Portal. Received from GitLab. | +| `grant_type` | string | yes |Specifies the type of credential flow. Use `client_credentials`. | +| `scope` | string | yes |Specifies the level of access. Use `marketplace.order:read` for read-only access. Use `marketplace.order:create` for create access. | + +If the request is successful, the response body includes the access token that you can use in subsequent requests. For an example of a successful +response, see the [Marketplace interactive API documentation](https://customers.staging.gitlab.com/openapi_docs/marketplace) + +If the request is unsuccessful, the response body includes an error and error description. The errors can be: + +| Status | Description | +|--------|----------------------------------------------------------------------------------------------------------------------------------------------| +| 400 | Invalid scope. Ensure the `scope` is `marketplace.order:read` or `marketplace.order:create`. | +| 401 | Invalid client. Ensure that there are no typos or extra spaces on the client specific credentials. Incorrect `client_id` or `client_secret` | + +### Step 3: Use the access token + +To use the access token from a client application: + +1. Set the `Authorization` header of the request to `Bearer <your_access_token>`. +1. Set parameters or data needed for the endpoint and send the request. + +Example request: + +```shell +curl \ + --url "https://customers.staging.gitlab.com/api/v1/marketplace/subscriptions/:external_subscription_id" \ + --header "Authorization: Bearer NHb_VhZhPOnBTSNfBSzmCmt28lLDWb2xtwr_c3DL148" +``` + +## Create a new customer subscription + +To create a new customer subscription from a GitLab Partner client application, + +- Make an authorized POST request to the +[`/api/v1/marketplace/subscriptions`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/post_api_v1_marketplace_subscriptions) +endpoint in the Customers Portal with the following parameters in JSON format: + +| Parameter | Type | Required | Description | +|--------------------------|--------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------| +| `externalSubscriptionId` | string | yes | ID of the subscription on the GitLab Partner system. | +| `tradingPartnerId` | string | yes | ID of the GitLab Partner account on the Customers Portal. | +| `customer` | object | yes | Information about the customer. Must include company name. Contact must include `firstName`, `lastName` and `email`. Address must include `country`. | +| `orderLines` | array | yes | Specifies the product purchased. Must include `quantity` and `productId`. | + +If the request is successful, the response body includes the newly created subscription number. For an example of a full request body, +see the [Marketplace interactive API documentation](https://customers.staging.gitlab.com/openapi_docs/marketplace). + +If the subscription creation is unsuccessful, the response body includes an error message with details about the cause of the error. + +## Check the status of a subscription + +To get the status of a given subscription, + +- Make an authorized GET request to the +[`/api/v1/marketplace/subscriptions/{external_subscription_id}`](https://customers.staging.gitlab.com/openapi_docs/marketplace#/marketplace/get_api_v1_marketplace_subscriptions__external_subscription_id_) +endpoint in the Customers Portal. + +The request must include the GitLab partner system ID of the subscription to fetch the status for. + +If the request is successful, the response body contains the status of the subscription provision. The status can be: + +- Creating +- Created +- Failed +- Provisioned -OpenAPI specs for the Marketplace APIs are available upon request. The specs will be made public before the end of Q1 2023. +If the subscription cannot be found using the passed `external_subscription_id`, the response has +a 404 Not Found status. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 24b5e6152a5..888dd47a968 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -734,6 +734,10 @@ For a full list of supported assertions, see the [OmniAuth SAML gem](https://git ## Configure users based on SAML group membership +NOTE: +SAML Group Sync is only supported for the [SAML provider named `saml`](#configure-gitlab-to-use-multiple-saml-idps). +As a result, SAML Group Sync only supports a single SAML provider. For more information, see [issue 366450](https://gitlab.com/gitlab-org/gitlab/-/issues/366450). + You can: - Require users to be members of a certain group. @@ -749,7 +753,7 @@ Support for these groups depends on: - Whether you've installed [GitLab Enterprise Edition (EE)](https://about.gitlab.com/install/). - The [name of the SAML provider](#configure-saml-support-in-gitlab). Group memberships are only supported by a single SAML provider named - `saml`. For more information, see [issue 386605](https://gitlab.com/gitlab-org/gitlab/-/issues/386605). + `saml`. | Group | Tier | GitLab Enterprise Edition (EE) Only? | |------------------------------|--------------------|--------------------------------------| diff --git a/doc/integration/trello_power_up.md b/doc/integration/trello_power_up.md index 9cca8e5485d..908fc623d0e 100644 --- a/doc/integration/trello_power_up.md +++ b/doc/integration/trello_power_up.md @@ -6,14 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Trello Power-Ups **(FREE)** -You can use the Trello Power-Up for GitLab to attach +You can use Trello Power-Ups for GitLab to attach GitLab merge requests to Trello cards. - + -## Configure the Power-Up +## Configure Power-Ups -To configure a Power-Up for a Trello board: +To configure Power-Ups for a Trello board: 1. Go to your Trello board. 1. Select **Power-Ups** and find the **GitLab** row. diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 0be2f087c62..8931f3ef833 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/389991) -for use in GitLab 15.9, and is planned for removal in GitLab 16.0. We are replacing this feature with functionality in the [GitLab Observability UI](https://gitlab.com/gitlab-org/opstrace/opstrace-ui). Please also reference our direction for [Observability](https://about.gitlab.com/direction/monitor/observability/) and [data visualization](https://about.gitlab.com/direction/monitor/observability/data-visualization/). +for use in GitLab 15.9, and is planned for removal in GitLab 16.0. We are replacing this feature with functionality in the [GitLab Observability UI](https://gitlab.com/gitlab-org/opstrace/opstrace-ui). Please also reference our direction for [Observability](https://about.gitlab.com/direction/monitor/observability/) and [data visualization](https://about.gitlab.com/direction/monitor/observability/data-visualization/). Error Tracking allows developers to discover and view errors generated by their application. Because error information is surfaced where the code is being developed, efficiency and awareness are increased. @@ -34,15 +34,15 @@ For error tracking to work, you need two pieces: ## Sentry error tracking -[Sentry](https://sentry.io/) is an open source error tracking system. GitLab allows administrators to connect Sentry to GitLab, to allow users to view a list of Sentry errors in GitLab. +[Sentry](https://sentry.io/) is an open source error tracking system. GitLab allows administrators to connect Sentry to GitLab so users can view a list of Sentry errors in GitLab. ### Deploying Sentry -You can sign up to the cloud hosted [Sentry](https://sentry.io) or deploy your own [on-premise instance](https://github.com/getsentry/onpremise/). +You can sign up to the cloud-hosted [Sentry](https://sentry.io) or deploy your own [on-premise instance](https://github.com/getsentry/onpremise/). ### Enabling Sentry -GitLab provides an easy way to connect Sentry to your project. You need at +GitLab provides a way to connect Sentry to your project. You need at least Maintainer [permissions](../user/permissions.md) to enable the Sentry integration. 1. Sign up to Sentry.io or [deploy your own](#deploying-sentry) Sentry instance. @@ -84,7 +84,7 @@ DSN in **Sentry.io > Project Settings > Client Keys (DSN) > Show deprecated DSN* ERROR: Sentry failure builds=0 error=raven: dsn missing private key ``` -## Error Tracking List +## Error Tracking list Users with at least Reporter [permissions](../user/permissions.md) can find the Error Tracking list at **Monitor > Error Tracking** in your project's sidebar. @@ -92,11 +92,11 @@ Here, you can filter errors by title or by status (one of Ignored , Resolved, or  -## Error Details +## Error details -From error list, users can navigate to the error details page by selecting the title of any error. +From error list, users can go to the error details page by selecting the title of any error. -This page has: +This page includes: - A link to the Sentry issue. - A link to the GitLab commit if the Sentry [release ID/version](https://docs.sentry.io/product/releases/?platform=javascript#configure-sdk) on the Sentry Issue's first release matches a commit SHA in your GitLab hosted project. @@ -108,26 +108,26 @@ By default, a **Create issue** button is displayed:  If you create a GitLab issue from the error, the **Create issue** button changes to a **View issue** -button and a link to the GitLab issue displays within the error detail section. +button and a link to the GitLab issue displays in the error detail section. -## Taking Action on errors +## Taking action on errors -You can take action on Sentry Errors from within the GitLab UI. Marking errors ignored or resolved require at least Developer role. +You can take action on Sentry Errors in the GitLab UI. Marking errors as ignored or resolved requires at least Developer role. ### Ignoring errors > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39665) in GitLab 12.7. -From within the [Error Details](#error-details) page you can ignore a Sentry error by selecting the **Ignore** button near the top of the page. +In the [Error Details](#error-details) page you can ignore a Sentry error by selecting **Ignore** near the top of the page. -Ignoring an error prevents it from appearing in the [Error Tracking List](#error-tracking-list), and silences notifications that were set up within Sentry. +Ignoring an error prevents it from appearing in the [Error Tracking List](#error-tracking-list), and silences notifications that were set up in Sentry. ### Resolving errors > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39825) in GitLab 12.7. -From within the [Error Details](#error-details) page you can resolve a Sentry error by -selecting the **Resolve** button near the top of the page. +In the [Error Details](#error-details) page you can resolve a Sentry error by +selecting **Resolve** near the top of the page. Marking an error as resolved indicates that the error has stopped firing events. If a GitLab issue is linked to the error, then the issue closes. @@ -140,53 +140,225 @@ If another event occurs, the error reverts to unresolved. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7586) in GitLab 15.6. NOTE: -Available only on GitLab.com. This feature is currently in [Open Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#open-beta). +Available only on GitLab.com. This feature is in [Open Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#open-beta). -Integrated error tracking is a lightweight alternative to Sentry backend. -You still use Sentry SDK with your application. But you don't need to deploy Sentry -or set up for cloud-hosted Sentry. Instead, you use GitLab as a backend for it. +### Known limitations -Sentry backend automatically assigns a Data Source Name (DSN) for every project you create. -GitLab does the same. You should be able to find a DSN for your project in the GitLab error tracking -settings. By using a GitLab-provided DSN, your application connects to GitLab to report an error. -Those errors are stored in the GitLab database and rendered by the GitLab UI, in the same way as -Sentry integration. +Only basic support is provided with `capture_exception` as the holding method. +Additional features requests (see this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340178)) are added on a case-by-case basis. -In GitLab 15.6 and later, the integrated error tracking -uses a new backend based on the ClickHouse database that enables better scalability. -Integrated error tracking remains limited in comparison to the Sentry backend, as only a small subset of the -Sentry API is implemented. +### Debugging issues -Changing the GitLab error UI to use the GitLab Observability UI is tracked in epic [19](https://gitlab.com/groups/gitlab-org/opstrace/-/epics/32). +The majority of languages are supported by Sentry expose `debug` option as part of initialization. +It is also possible to output JSON before it is sent to the API. +See the [Go example](#go) below for a suggested solution. -### Project settings +### Enabling error tracking -You can find the feature configuration at **Settings > Monitor > Error Tracking**. +Regardless of the programming language, you need to enable error tracking for your project. This doc assumes you already have a project for which you want to enable error tracking. +This example uses the `gitlab.com` instance. -#### How to enable +To enable error tracking, follow these steps: -1. Select **GitLab** as the error tracking backend for your project: +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Error Tracking** tab. +1. Under **Enable error tracking**, select the **Active** checkbox. +1. Under **Error tracking backend**, select **GitLab**. +1. Select **Save changes**. +1. Copy the DSN string to use later. + +### Listing captured errors + +Once your application has emitted errors to the Error Tracking API through the Sentry SDK, they should be available under **Monitor > Error Tracking** tab/section. + +For more details, refer to the [GitLab error tracking documentation](https://gitlab.com/help/operations/error_tracking#error-tracking-list). + +This process assumes the GDK feature flag `integrated_error_tracking` is enabled. If you are running GDK locally and you do not see the option for error tracking, you can enable it by running the following commands: + +```linux +cd <PATH_TO_GDK> +gdk rails console +Feature.enable(:integrated_error_tracking) +``` + +### Emitting errors + +#### Supported Sentry types + +According to the [data model](https://develop.sentry.dev/sdk/envelopes/#data-model), the available item types are: + +- [Event](https://develop.sentry.dev/sdk/event-payloads/) +- [Transactions](https://develop.sentry.dev/sdk/event-payloads/transaction/) +- Attachment +- [Session](https://develop.sentry.dev/sdk/sessions/) +- [Sessions](https://develop.sentry.dev/sdk/sessions/) +- [User feedback](https://develop.sentry.dev/sdk/envelopes/#user-feedback) (also known as user report) +- [Client report](https://develop.sentry.dev/sdk/client-reports/) + +Items of various types can be sent to the error tracking app, using either the Store endpoint, the envelope endpoint, or both. The following table lists all event types available through Sentry SDK. It also explains which endpoint can be used for ingestion and whether it is supported by GitLab Observability Backend. + +Event item types can contain various interfaces, such as exception, message, stack trace, and template. You can read more about the core data interfaces in [Sentry documentation](https://develop.sentry.dev/sdk/event-payloads/#core-interfaces). + +| Item type | Interface | Can be sent through the Store endpoint | Can be sent through the Envelope endpoint | Supported | +| --------------- | ----------- | -------------------------------------- | ----------------------------------------- | ---------------------- | +| `event` | exception | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| `event` | message | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| `event` | stack trace | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| `event` | template | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | +| `transaction` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | +| `attachment` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | +| `session` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | +| `sessions` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | +| `user_report` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | +| `client_report` | N/A | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | + +#### Supported languages + +Each language shows a basic example of how to capture exceptions with the respective SDK. +For more in-depth documentation, see [documentation for Sentry SDK](https://docs.sentry.io/). You can also find information for additional programming languages. + +Only a subset of languages is supported. + +The following table lists them: + +| Sentry SDK | Supported? | +| ----------- | ----------- | +| Ruby | Yes | +| Go | Yes | +| JavaScript | Yes | +| Java | Yes | +| Python | Yes | +| PHP | Yes | +| .NET | Not tested | +| Android | Not tested | +| Apple | Not tested | +| Perl | Not tested | + +A more up-to-date version of [this matrix can be found in this doc](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1737). + +#### Go + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/go/` +1. Install the dependencies with the following command: + + ```shell + go mod tidy + ``` + +1. Run the following command: + + ```shell + export SENTRY_DSN="<DSN string>" + go run main.go <DSN string> + ``` + +After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document. + +#### Ruby + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/ruby/` +1. Install the dependencies with the following command: + + ```shell + gem install bundler + bundle install + ``` + +1. Execute the example with the following command: + + ```shell + export SENTRY_DSN="<DSN string>" + ruby app.rb + ``` + +After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document. + +#### PHP + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/php/` + +1. Build and run the Docker container with the following commands: + +```shell +export SENTRY_DSN="<DSN string>" +docker build -t sentry-php . +docker run -e SENTRY_DSN --rm sentry-php +``` + +After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document. + +#### Python + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/python/` + +1. Install the dependencies with the following commands: + + ```shell + virtualenv env + source env/bin/activate + pip install -r requirements.txt + ``` + +1. Run the following commands: + +```shell +export SENTRY_DSN="<DSN string>" +python send_exception.py +``` + +After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document. + +#### Java + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/python/` + +1. Run the following command: + +```shell +export SENTRY_DSN="<DSN string>" +./gradlew run +``` + +#### Node.js + +1. `chdir` into folder `docs/guides/user/error_tracking_examples/nodejs/` + +1. Install the dependencies with the following command: + + ```shell + npm install --save @sentry/node @sentry/tracing + ``` + +1. Run the following command: + + ```shell + export SENTRY_DSN="<DSN string>" + node ./test.js + ``` -  +After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document. -1. Select **Save changes**. After page reload you should see a text field with the DSN string. Copy it. +### Rotating Sentry DSN -  +The Sentry DSN (client key) is a secret and it should not be exposed to the public. If it's leaked, you can rotate the Sentry DSN with the following steps: -1. Take the DSN from the previous step and configure your Sentry SDK with it. Errors are now - reported to the GitLab collector and are visible in the [GitLab UI](#error-tracking-list). +1. [Create an access token](../user/profile/personal_access_tokens.md#create-a-personal-access-token) by clicking your profile picture in GitLab.com. Then choose Preferences,then Access Token. Make sure you add the API scope. +1. Using the [error tracking API](../api/error_tracking.md), create a new Sentry DSN with the following command: -#### Managing DSN + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys" + ``` -When you enable the feature you receive a DSN. It includes a hash used for authentication. This hash -is a client key. GitLab uses client keys to authenticate error tracking requests from your -application to the GitLab backend. +1. Get the available client keys (Sentry DSNs). Ensure that the newly created Sentry DSN is in place. Then note down the key ID of the old client key: -In some situations, you may want to create a new client key and remove an existing one. -You can do so by managing client keys with the [error tracking API](../api/error_tracking.md). + ```shell + curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys" + ``` -#### Limitations +1. Delete the old client key with the following command: -The Integrated Error Tracking feature was built and tested with Sentry SDK for Ruby on Rails. -Support for other languages and frameworks is not guaranteed. For up-to-date information, see the -[compatibility issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340178). + ```shell + curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys/<key_id>" + ``` diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 68fc0fb9499..968649e94b2 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -151,7 +151,7 @@ Enables the feature for a list of target users. It is implemented using the Unleash UserIDs (`userWithId`) activation [strategy](https://docs.getunleash.io/reference/activation-strategies#userids). Enter user IDs as a comma-separated list of values (for example, -`user@example.com, user2@example.com`, or `username1,username2,username3`, and so on). +`user@example.com, user2@example.com`, or `username1,username2,username3`, and so on). User IDs are identifiers for your application users. They do not need to be GitLab users. WARNING: @@ -295,14 +295,13 @@ Unleash currently [offers many SDKs for various languages and frameworks](https: For API content, see: - [Feature flags API](../api/feature_flags.md) -- [Feature flag specs API](../api/feature_flag_specs.md) (Deprecated and [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/213369) in GitLab 14.0.) - [Feature flag user lists API](../api/feature_flag_user_lists.md) -### Golang application example +### Go application example -Here's an example of how to integrate feature flags in a Golang application: +Here's an example of how to integrate feature flags in a Go application: -```golang +```go package main import ( diff --git a/doc/operations/img/copy-group-id.png b/doc/operations/img/copy-group-id.png Binary files differdeleted file mode 100644 index 7837f49c3e3..00000000000 --- a/doc/operations/img/copy-group-id.png +++ /dev/null diff --git a/doc/operations/img/create-gitlab-application.png b/doc/operations/img/create-gitlab-application.png Binary files differdeleted file mode 100644 index 430b830cdb2..00000000000 --- a/doc/operations/img/create-gitlab-application.png +++ /dev/null diff --git a/doc/operations/img/error_tracking_setting_dsn_v14_4.png b/doc/operations/img/error_tracking_setting_dsn_v14_4.png Binary files differdeleted file mode 100644 index b7ecaa047b2..00000000000 --- a/doc/operations/img/error_tracking_setting_dsn_v14_4.png +++ /dev/null diff --git a/doc/operations/img/error_tracking_setting_v14_3.png b/doc/operations/img/error_tracking_setting_v14_3.png Binary files differdeleted file mode 100644 index 14306130c97..00000000000 --- a/doc/operations/img/error_tracking_setting_v14_3.png +++ /dev/null diff --git a/doc/operations/img/listing_groups.png b/doc/operations/img/listing_groups.png Binary files differdeleted file mode 100644 index 1094bf4ff28..00000000000 --- a/doc/operations/img/listing_groups.png +++ /dev/null diff --git a/doc/operations/incident_management/incident_timeline_events.md b/doc/operations/incident_management/incident_timeline_events.md index e79f36884cb..7913e4668eb 100644 --- a/doc/operations/incident_management/incident_timeline_events.md +++ b/doc/operations/incident_management/incident_timeline_events.md @@ -110,12 +110,13 @@ Alternatively: ## Incident tags -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8741) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `incident_event_tags`. Disabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8741) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `incident_event_tags`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/387647) in GitLab 15.9. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/387647) in GitLab 15.10. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `incident_event_tags`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `incident_event_tags`. +On GitLab.com, this feature is available. [When creating an event using the form](#using-the-form) or editing it, you can specify incident tags to capture relevant incident timestamps. diff --git a/doc/operations/incident_management/manage_incidents.md b/doc/operations/incident_management/manage_incidents.md index ad4de8641e5..338dacda166 100644 --- a/doc/operations/incident_management/manage_incidents.md +++ b/doc/operations/incident_management/manage_incidents.md @@ -220,7 +220,7 @@ Prerequisites: - You must have at least the Reporter role for the project. -To close an incident, in the upper right, select **Close incident**. +To close an incident, in the upper-right corner, select **Close incident**. When you close an incident that is linked to an [alert](alerts.md), the linked alert's status changes to **Resolved**. diff --git a/doc/operations/index.md b/doc/operations/index.md index ff13c617ea7..d4a5f895947 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab provides a variety of tools to help operate and maintain your applications. -## Measure reliability and stability with metrics (DEPRECATED) +## Measure reliability and stability with metrics (deprecated) > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. @@ -35,14 +35,14 @@ performance degrades, and manage those alerts - all within GitLab. GitLab helps reduce alert fatigue for IT responders by providing tools to identify issues across multiple systems and aggregate alerts in a centralized place. Your -team needs a single, central interface where they can easily investigate alerts +team needs a single, central interface where they can investigate alerts and promote the critical alerts to incidents. Are your alerts too noisy? Alerts configured on GitLab metrics can configured and fine-tuned in GitLab immediately following a fire-fight. - [Manage alerts and incidents](incident_management/index.md) in GitLab. -- [Configure alerts for metrics](metrics/alerts.md) in GitLab. (DEPRECATED) +- [Configure alerts for metrics](metrics/alerts.md) in GitLab. (deprecated) - Create a [status page](incident_management/status_page.md) to communicate efficiently to your users during an incident. diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md index e22b1096023..96c1238fd34 100644 --- a/doc/operations/metrics/dashboards/default.md +++ b/doc/operations/metrics/dashboards/default.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab-defined metrics dashboards (DEPRECATED) **(FREE)** +# GitLab-defined metrics dashboards (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/develop.md b/doc/operations/metrics/dashboards/develop.md index fe14ec1dc3d..af76c09ca80 100644 --- a/doc/operations/metrics/dashboards/develop.md +++ b/doc/operations/metrics/dashboards/develop.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Developing templates for custom dashboards (DEPRECATED) **(FREE)** +# Developing templates for custom dashboards (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index 09bb8cedbf6..b2fc7397d17 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Custom dashboards (DEPRECATED) **(FREE)** +# Custom dashboards (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59974) in GitLab 12.1. > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. @@ -124,9 +124,9 @@ can manage [the settings](settings.md) for your metrics dashboard. ## Chart Context Menu -You can take action related to a chart's data by selecting the -**{ellipsis_v}** **More actions** dropdown list above the upper right corner of -any chart on a dashboard: +To take action related to a chart's data: + +- In the upper-right corner of the chart, select **More actions** (**{ellipsis_v}**).  diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index 177a55fb85b..39893f279ff 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Panel types for dashboards (DEPRECATED) **(FREE)** +# Panel types for dashboards (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md index 5fb0476e164..6ddedd98fcb 100644 --- a/doc/operations/metrics/dashboards/settings.md +++ b/doc/operations/metrics/dashboards/settings.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Dashboard settings (DEPRECATED) **(FREE)** +# Dashboard settings (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/templating_variables.md b/doc/operations/metrics/dashboards/templating_variables.md index a1c2ce063bc..a27e75aacef 100644 --- a/doc/operations/metrics/dashboards/templating_variables.md +++ b/doc/operations/metrics/dashboards/templating_variables.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Templating variables for metrics dashboards (DEPRECATED) **(FREE)** +# Templating variables for metrics dashboards (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214539) in GitLab 13.0. > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md index 2881c084115..152f7522c4e 100644 --- a/doc/operations/metrics/dashboards/variables.md +++ b/doc/operations/metrics/dashboards/variables.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Using variables (DEPRECATED) **(FREE)** +# Using variables (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index 7b0a91dd18a..e8bba4b6e42 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Dashboard YAML properties (DEPRECATED) **(FREE)** +# Dashboard YAML properties (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md index f622780530a..987f1dc019d 100644 --- a/doc/operations/metrics/embed.md +++ b/doc/operations/metrics/embed.md @@ -48,9 +48,8 @@ GitLab unfurls the link as an embedded metrics panel:  -You can also embed a single chart. To get a link to a chart, select -**{ellipsis_v}** **More actions** in the upper right corner of the chart, -and select **Copy link to chart**, as shown in this example: +You can also embed a single chart. To get a link to a chart, in the upper-right corner of the chart, +select **More actions** (**{ellipsis_v}**), then select **Copy link to chart** as shown in this example.  diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index 11350d65237..e67e7b0f97e 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitor your environment's metrics **(FREE)** +# Monitor your environment's metrics (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. @@ -124,7 +124,7 @@ After saving them, they display on the environment metrics dashboard provided th - A [connected Kubernetes cluster](../../user/clusters/agent/index.md) with the matching [environment scope](../../ci/environments/index.md#limit-the-environment-scope-of-a-cicd-variable) is used and - [Prometheus installed on the cluster](../../user/project/integrations/prometheus.md#enabling-prometheus-integration). + [Prometheus installed on the cluster](../../user/project/integrations/prometheus.md#enabling-the-prometheus-integration). - Prometheus is [manually configured](../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus).  diff --git a/doc/operations/quickstart-guide.md b/doc/operations/quickstart-guide.md deleted file mode 100644 index 91c5f25405c..00000000000 --- a/doc/operations/quickstart-guide.md +++ /dev/null @@ -1,229 +0,0 @@ ---- -stage: Monitor -group: Observability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# GitLab Observability Quickstart - -You can try GitLab Observability by [cloning or forking the project](https://gitlab.com/gitlab-org/opstrace/opstrace.git) and creating a local installation. - -## Prerequisites and dependencies - -To install GitLab Observability Platform (GOP), install and configure the following third-party dependencies. You can do this manually, or [automatically by using asdf](#install-dependencies-using-asdf): - -- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) for creating a local Kubernetes cluster. -- [Docker](https://docs.docker.com/install) - - [Docker Compose](https://docs.docker.com/compose/compose-v2/) is now part of the `docker` distribution. -- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for interacting with GitLab Observability. -- [Telepresence](https://www.telepresence.io/) allows you to code and test microservices locally against a remote Kubernetes cluster. -- [jq](https://stedolan.github.io/jq/download/) for some Makefile utilities. -- [Go 1.19](https://go.dev/doc/install). - -The current versions of these dependencies are pinned in the `.tool-versions` file in the project. - -You can run the following commands to check the availability and versions of these dependencies on your machine: - -```shell -kind --version -docker --version -kubectl version -telepresence version -jq --version -go version -``` - -### Run GOP on macOS - -If you're running GOP on macOS, ensure you have enough resources dedicated to Docker Desktop. The recommended minimum is: - -- CPUs: 4+ -- Memory: 8 GB+ -- Swap: 1 GB+ - -It's possible to run GOP with fewer resources, but this specification works. - -### Install dependencies using asdf - -If you install dependencies using [`asdf`](https://asdf-vm.com/#/core-manage-asdf), GOP manages them for you automatically. - -1. If you have not already done so, clone the `opstrace` repository into your preferred location: - - ```shell - git clone https://gitlab.com/gitlab-org/opstrace/opstrace.git - ``` - -1. Change into the project directory: - - ```shell - cd opstrace - ``` - -1. Optional. If you need to install `asdf`, run: - - ```shell - make install-asdf - ``` - -1. Install dependencies using `asdf`: - - ```shell - make bootstrap - ``` - -## Step 1: Create a local Kubernetes cluster with kind - -Make sure Docker Desktop is running. In the `opstrace` project you cloned, run the following command: - -```shell -make kind -``` - -Wait a few minutes while kind creates your Kubernetes cluster. When it's finished, you should see the following message: - -```plaintext -Traffic Manager installed successfully -``` - -Now deploy the scheduler by running the following command in the `opstrace` project: - -```shell -make deploy -``` - -This takes around 1 minute. - -## Step 2: Create a GitLab application for authentication - -You must create a GitLab application to use for authentication. - -In the GitLab instance you'd like to connect with GOP, [create an OAuth application](../integration/oauth_provider.md). -This application can be a user-owned, group-owned or instance-wide application. -In production, you would create a trusted instance-wide application so that users are explicitly authorized without the consent screen. -The following example shows how to configure the application. - -1. Select the API scope and enter `http://localhost/v1/auth/callback` as the redirect URI. - -1. Run the following command to create the secret that holds the authentication data: - - ```shell - kubectl create secret generic \ - --from-literal=gitlab_oauth_client_id=<gitlab_application_client_id> \ - --from-literal=gitlab_oauth_client_secret=<gitlab_application_client_secret> \ - --from-literal=internal_endpoint_token=<error_tracking_internal_endpoint_token> \ - dev-secret - ``` - -1. Replace `<gitlab_application_client_id>` and `<gitlab_application_client_secret>` with the values from the GitLab application you just created. -Replace `<error_tracking_internal_endpoint_token>` with any string if you do not plan to use error tracking. - -You can also view [this MR on how to get the token to test error tracking](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91928). -You must specify all the parameters when creating the secret. - -## Step 3: Create the cluster definition - -1. In your `opstrace` project, run the following command to create a `Cluster.yaml` manifest file: - - ```shell - cat <<EOF > Cluster.yaml - apiVersion: opstrace.com/v1alpha1 - kind: Cluster - metadata: - name: dev-cluster - spec: - target: kind - goui: - image: "registry.gitlab.com/gitlab-org/opstrace/opstrace-ui/ gitlab-observability-ui:c9fb6e70" - dns: - acmeEmail: "" - dns01Challenge: {} - externalDNSProvider: {} - gitlab: - groupAllowedAccess: '*' - groupAllowedSystemAccess: "6543" - instanceUrl: https://gitlab.com - authSecret: - name: dev-secret - EOF - ``` - -1. Apply the file you just created with the following command: - - ```shell - kubectl apply -f Cluster.yaml - ``` - -1. Run the following command to wait for the cluster to be ready: - - ```shell - kubectl wait --for=condition=ready cluster/dev-cluster --timeout=600s - ``` - -After the previous command exits, the cluster is ready. - -## Step 4: Enable Observability on a GitLab namespace you own - -Go to a namespace you own in the connected GitLab instance and copy the Group ID below the group name. - -GOP can only be enabled for groups you own. -To list all the groups that your user owns, go to the menu in upper left corner and select `Groups`->`View all Groups`. You then see the **Your groups** tab. - -In your browser, go to `http://localhost/-/{GroupID}`. For example, `http://localhost/-/14485840`. - -Follow the on-screen instructions to enable observability for the namespace. -This can take a couple of minutes if it's the first time observability has been enabled for the root level namespace (GitLab.org in the previous example.) - -Once your namespace has been enabled and is ready, the page automatically redirects you to the GitLab Observability UI. - -## Step 5: Send traces to GOP - -[Follow this guide to send traces to your namespace and monitor them in the UI](https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/guides/user/sending-traces-locally.md). - -## Step 6: Clean up your local GOP - -To tear down your locally running GOP instance, run the following command: - -```shell -make destroy -``` - -## Known issues - -### Incorrect architecture for `kind/node` image - -If your machine has an Apple silicon (M1/M2) chip, you might encounter an architecture problem with the `kind/node` image when running the `make kind` command. For more details, see [issue 1802](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1802). - -To fix this problem, you first need to create a Dockerfile. Then build and deploy the image: - -1. Create a new Dockerfile (without a file extension) and paste the following commands: - - ```Dockerfile - FROM --platform=arm64 kindest/node:v1.23.13 - RUN arch - ``` - -1. Save your Dockerfile, then build the image with the following command: - - ```shell - docker build -t tempkind . - ``` - - Do not forget the period at the end. - -1. Create a cluster using your new image with the following command: - - ```shell - kind create cluster --image tempkind - ``` - -### scheduler-controller-manager pod cannot start due to ImagePullBackOff - -If while executing `make deploy` in step 1, the `scheduler-controller-manager` pod cannot start due to `ImagePullBackOff`, you must set the `CI_COMMIT_TAG` to a non-dirty state. By setting the commit tag to the latest commit, you ensure the Docker image can be pulled from the container registry. - -Run the following command to set the commit tag: - -```shell -make kind -export CI_COMMIT_TAG=0.2.0-e1206acf -make deploy -``` diff --git a/doc/policy/alpha-beta-support.md b/doc/policy/alpha-beta-support.md index 1c7e9e77751..98910f9a3bb 100644 --- a/doc/policy/alpha-beta-support.md +++ b/doc/policy/alpha-beta-support.md @@ -12,7 +12,7 @@ All other features are considered to be Generally Available (GA). ## Alpha features -Support is **not** provided for Alpha features and issues with them should be opened in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/issues). +Support is **not** provided for Alpha features and issues with them should be opened in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). Characteristics of Alpha features: diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index fcb6e5c1b20..eed9f006bfa 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -110,9 +110,9 @@ The decision on whether backporting a change is performed is done at the discret [current release managers](https://about.gitlab.com/community/release-managers/), based on *all* of the following: -1. Estimated [severity](../development/contributing/issue_workflow.md#severity-labels) of the bug: +1. Estimated [severity](../development/labels/index.md#severity-labels) of the bug: Highest possible impact to users based on the current definition of severity. -1. Estimated [priority](../development/contributing/issue_workflow.md#priority-labels) of the bug: +1. Estimated [priority](../development/labels/index.md#priority-labels) of the bug: Immediate impact on all impacted users based on the above estimated severity. 1. Potentially incurring data loss and/or security breach. 1. Potentially affecting one or more strategic accounts due to a proven inability by the user to upgrade to the current stable version. @@ -122,7 +122,7 @@ the current stable release, and two previous monthly releases. In rare cases a r For instance, if we release `13.2.1` with a fix for a severe bug introduced in `13.0.0`, we could backport the fix to a new `13.0.x`, and `13.1.x` patch release. -Note that [severity](../development/contributing/issue_workflow.md#severity-labels) 3 and lower +Note that [severity](../development/labels/index.md#severity-labels) 3 and lower requests are automatically turned down. To request backporting to more than one stable release for consideration, raise an issue in the diff --git a/doc/raketasks/backup_gitlab.md b/doc/raketasks/backup_gitlab.md index d63ef61a99f..5b4df8752f4 100644 --- a/doc/raketasks/backup_gitlab.md +++ b/doc/raketasks/backup_gitlab.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Back up GitLab +# Back up GitLab **(FREE SELF)** GitLab provides a command line interface to back up your entire instance, including: @@ -262,6 +262,11 @@ For installations from source: sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production ``` +`SKIP=` is also used to: + +- [Skip creation of the tar file](#skipping-tar-creation) (`SKIP=tar`). +- [Skip uploading the backup to remote storage](#skip-uploading-backups-to-remote-storage) (`SKIP=remote`). + ### Skipping tar creation NOTE: @@ -411,12 +416,9 @@ You can let the backup script upload (using the [Fog library](https://fog.io/)) the `.tar` file it creates. In the following example, we use Amazon S3 for storage, but Fog also lets you use [other storage providers](https://fog.io/storage/). GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) -for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is +for AWS, Google, and Aliyun. A local driver is [also available](#upload-to-locally-mounted-shares). -NOTE: -Support for Openstack Swift and Rackspace APIs will be removed in GitLab 15.10. See [issue #387976](https://gitlab.com/gitlab-org/gitlab/-/issues/387976) for more information. - [Read more about using object storage with GitLab](../administration/object_storage.md). #### Using Amazon S3 diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index a13d38a199d..9d82bbafe88 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -788,7 +788,7 @@ Truncating filenames to resolve the error involves: #### Clean up remote uploaded files -A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). +A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 073a0351ec5..3f716900d17 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -154,7 +154,7 @@ These commands don't work for artifacts stored on [object storage](../administration/object_storage.md). WARNING: -Prior to GitLab 14.9, this task incorrectly deletes [pipeline artifacts](../ci/pipelines/pipeline_artifacts.md) +Prior to GitLab 14.9, this task incorrectly deletes [pipeline artifacts](../ci/pipelines/pipeline_artifacts.md). [The bug fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81022) was also back-ported to 14.6.6, 14.7.5, and 14.8.3. Upgrade to a release with the bug fix to avoid data loss. diff --git a/doc/raketasks/generate_sample_prometheus_data.md b/doc/raketasks/generate_sample_prometheus_data.md index 7445065c77e..73c49bd2599 100644 --- a/doc/raketasks/generate_sample_prometheus_data.md +++ b/doc/raketasks/generate_sample_prometheus_data.md @@ -4,9 +4,9 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Generate sample Prometheus data Rake task **(FREE SELF)** +# Sample Prometheus data Rake task **(FREE SELF)** -This command runs Prometheus queries for each of the metrics of a specific environment +The Rake task runs Prometheus queries for each of the metrics of a specific environment for a series of time intervals to now: - 30 minutes @@ -16,12 +16,12 @@ for a series of time intervals to now: - 72 hours - 7 days -The results of each of query are stored under a `sample_metrics` directory as a YAML +The results of each query are stored under a `sample_metrics` directory as a YAML file named by the metric's `identifier`. When the environmental variable `USE_SAMPLE_METRICS` -is set, the Prometheus API query is re-routed to `Projects::Environments::SampleMetricsController` -which loads the appropriate data set if it is present within the `sample_metrics` directory. +is set, the Prometheus API query is re-routed to `Projects::Environments::SampleMetricsController`, +which loads the appropriate data set if it's present in the `sample_metrics` directory. -This command requires an ID from an environment with an available Prometheus installation. +The Rake task requires an ID from an environment with an available Prometheus installation. ## Example diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index b5a778d6b74..e559605d147 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -12,7 +12,7 @@ processes. You can perform GitLab Rake tasks by using: -- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html) installations. +- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html) and [GitLab Helm chart](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) installations. - `bundle exec rake <raketask>` for [source](../install/installation.md) installations. ## Available Rake tasks @@ -28,7 +28,7 @@ The following Rake tasks are available for use with GitLab: | [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | | [Geo maintenance](../administration/raketasks/geo.md) | [Geo](../administration/geo/index.md)-related maintenance. | | [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | -| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | +| [Import large project exports](../administration/raketasks/project_import_export.md#import-large-projects) | Import large GitLab [project exports](../user/project/settings/import_export.md). | | [Incoming email](../administration/raketasks/incoming_email.md) | Incoming email-related tasks. | | [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, LDAP, and more. | | [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap/index.md)-related tasks. | @@ -56,6 +56,9 @@ To list all available Rake tasks: # Omnibus GitLab sudo gitlab-rake -vT +# GitLab Helm chart +gitlab-rake -vT + # Installations from source cd /home/git/gitlab sudo -u git -H bundle exec rake -vT RAILS_ENV=production diff --git a/doc/raketasks/restore_gitlab.md b/doc/raketasks/restore_gitlab.md index c5bbb38cc37..13422817503 100644 --- a/doc/raketasks/restore_gitlab.md +++ b/doc/raketasks/restore_gitlab.md @@ -4,7 +4,7 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Restore GitLab +# Restore GitLab **(FREE SELF)** GitLab provides a command line interface to restore your entire installation, and is flexible enough to fit your needs. @@ -273,9 +273,7 @@ project or group from there: the backed-up instance from which you want to restore. 1. [Restore the backup](#restore-gitlab) into this new instance, then export your [project](../user/project/settings/import_export.md) - or [group](../user/group/settings/import_export.md). Be sure to read the - **Important Notes** on either export feature's documentation to understand - what is and isn't exported. + or [group](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). For more information about what is and isn't exported, see the export feature's documentation. 1. After the export is complete, go to the old instance and then import it. 1. After importing the projects or groups that you wanted is complete, you may delete the new, temporary GitLab instance. diff --git a/doc/security/index.md b/doc/security/index.md index 38eb5337f5a..e447b8ffe64 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -13,7 +13,7 @@ type: index - [Generated passwords for users created through integrated authentication](passwords_for_integrated_authentication_methods.md) - [Restrict SSH key technologies and minimum length](ssh_keys_restrictions.md) - [Rate limits](rate_limits.md) -- [Webhooks and insecure internal web services](webhooks.md) +- [Filtering outbound requests](webhooks.md) - [Information exclusivity](information_exclusivity.md) - [Reset user password](reset_user_password.md) - [Unlock a locked user](unlock_user.md) diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index 38c52912d5c..f8ba3953379 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -96,7 +96,7 @@ If you know the username, user ID, or email address, you can use the Rails conso user.password = new_password user.password_confirmation = new_password ``` - + To set a specific value for the new password: ```ruby @@ -113,9 +113,9 @@ If you know the username, user ID, or email address, you can use the Rails conso 1. Save the changes: - ```ruby - user.save! - ``` + ```ruby + user.save! + ``` 1. Exit the console: @@ -145,10 +145,10 @@ attempt to fix this issue in a Rails console. For example, if a new `root` passw 1. Start a [Rails console](../administration/operations/rails_console.md). 1. Find the user and skip reconfirmation: - ```ruby - user = User.find(1) - user.skip_reconfirmation! - ``` + ```ruby + user = User.find(1) + user.skip_reconfirmation! + ``` 1. Attempt to sign in again. diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index a36d72f128d..b349eee9837 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -77,7 +77,7 @@ Project maintainers and owners can add or enable a deploy key for a project repo ## Runner registration tokens (deprecated) WARNING: -The ability to pass a runner registration token was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) in GitLab 15.6 and is +The ability to pass a runner registration token has been [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/380872) and is planned for removal in 17.0, along with support for certain configuration arguments. This change is a breaking change. GitLab plans to introduce a new [GitLab Runner token architecture](../architecture/blueprints/runner_tokens/index.md), which introduces a new method for registering runners and eliminates the diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index db2948a8bd5..80f4b1a8a2a 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -45,6 +45,30 @@ To configure authentication settings for all media files: 1. Scroll to **Project visibility** and select **Require authentication to view media files**. You cannot select this option for projects with **Public** visibility. +## Delete uploaded files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92791) in GitLab 15.3. + +You should delete an uploaded file when that file contains sensitive or confidential information. When you have deleted that file, users cannot access the file and the direct URL returns a 404 error. + +Project Owners and Maintainers can use the [interactive GraphiQL explorer](../api/graphql/index.md#graphiql) to access a [GraphQL endpoint](../api/graphql/reference/index.md#mutationuploaddelete) and delete an uploaded file. + +For example: + +```graphql +mutation{ + uploadDelete(input: { projectPath: "<path/to/project>", secret: "<32-character-id>" , filename: "<filename>" }) { + upload { + id + size + path + } + errors + } +} +``` + +Project members that do not have the Owner or Maintainer role cannot access this GraphQL endpoint. <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index eeb6720dfb7..6164fb794c2 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -5,10 +5,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts, reference, howto --- -# Webhooks and insecure internal web services **(FREE SELF)** +# Filtering outbound requests **(FREE SELF)** + +To protect against the risk of data loss and exposure, GitLab administrators can now use outbound request filtering controls to restrict certain outbound requests made by the GitLab instance. + +## Secure webhooks and integrations Users with at least the Maintainer role can set up [webhooks](../user/project/integrations/webhooks.md) that are -triggered when specific changes occur in a project. When triggered, a `POST` HTTP request is sent to a URL. A webhook is +triggered when specific changes occur in a project or group. When triggered, a `POST` HTTP request is sent to a URL. A webhook is usually configured to send data to a specific external web service, which processes the data in an appropriate way. However, a webhook can be configured with a URL for an internal web service instead of an external web service. @@ -32,9 +36,13 @@ that hosts the webhook including: Webhooks can be used to trigger destructive commands using web services that don't require authentication. These webhooks can get the GitLab server to make `POST` HTTP requests to endpoints that delete resources. -## Allow webhook and service requests to local network +### Allow requests to the local network from webhooks and integrations + +Prerequisite: -To prevent exploitation of insecure internal web services, all webhook requests to the following local network addresses are not allowed: +- You must have administrator access to the instance. + +To prevent exploitation of insecure internal web services, all webhook and integration requests to the following local network addresses are not allowed: - The current GitLab instance server address. - Private network addresses, including `127.0.0.1`, `::1`, `0.0.0.0`, `10.0.0.0/8`, `172.16.0.0/12`, @@ -45,28 +53,64 @@ To allow access to these addresses: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. -1. Select the **Allow requests to the local network from web hooks and services** checkbox. +1. Select the **Allow requests to the local network from webhooks and integrations** checkbox. + +### Prevent requests to the local network from system hooks -## Prevent system hook requests to local network +Prerequisite: -[System hooks](../administration/system_hooks.md) are permitted to make requests to local network by default because -they are set up by administrators. To prevent system hook requests to the local network: +- You must have administrator access to the instance. + +[System hooks](../administration/system_hooks.md) can make requests to the local network by default. To prevent system hook requests to the local network: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests**. 1. Clear the **Allow requests to the local network from system hooks** checkbox. -## Create an allowlist for local requests +## Filter requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377371) in GitLab 15.10 [with a flag](../administration/feature_flags.md) named `deny_all_requests_except_allowed`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `deny_all_requests_except_allowed`. +On GitLab.com, this feature is not available. + +Prerequisite: -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44496) in GitLab 12.2 +- You must have administrator access to the instance. -You can allow certain domains and IP addresses to be accessible to both system hooks and webhooks, even when local -requests are forbidden. To add these domains to the allowlist: +To filter requests by blocking many requests: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Network**. -1. Expand **Outbound requests** and add entries. +1. Expand **Outbound requests**. +1. Select the **Block all requests, except for IP addresses, IP ranges, and domain names defined in the allowlist** checkbox. + +When this checkbox is selected, requests to the following are still not blocked: + +- Core services like Geo, Git, GitLab Shell, Gitaly, PostgreSQL, and Redis. +- Object storage. +- IP addresses and domains in the [allowlist](#allow-outbound-requests-to-certain-ip-addresses-and-domains). + +This setting is respected by the main GitLab application only, so other services like Gitaly can still make requests that break the rule. +Additionally, [some areas of GitLab](https://gitlab.com/groups/gitlab-org/-/epics/8029) do not respect outbound filtering +rules. + +## Allow outbound requests to certain IP addresses and domains + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44496) in GitLab 12.2. + +Prerequisite: + +- You must have administrator access to the instance. + +To allow outbound requests to certain IP addresses and domains: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Outbound requests**. +1. In **Local IP addresses and domain names that hooks and integrations can access**, enter your IP addresses and domains. The entries can: @@ -91,14 +135,37 @@ example.com;gitlab.example.com example.com:8080 ``` -<!-- ## Troubleshooting +## Troubleshooting + +When filtering outbound requests, you might encounter the following issues. + +### Configured URLs are blocked + +You can only select the **Block all requests, except for IP addresses, IP ranges, and domain names defined in the allowlist** checkbox if no configured URLs would be blocked. Otherwise, you might get an error message that says the URL is blocked. + +If you can't enable this setting, do one of the following: + +- Disable the URL setting. +- Configure another URL, or leave the URL setting empty. +- Add the configured URL to the [allowlist](#allow-requests-to-the-local-network-from-webhooks-and-integrations). + +### Public runner releases URL is blocked + +Most GitLab instances have their `public_runner_releases_url` set to +`https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-runner/releases`. +You can't change or disable this setting in the Admin Area, which can prevent you from [filtering requests](#filter-requests). + +To enable the setting, use the [Rails console](../administration/operations/rails_console.md) to set `public_runner_releases_url` to the instance host: + +```ruby +current_settings = ApplicationSetting.find_or_create_without_cache; +ApplicationSettings::UpdateService.new(current_settings, nil, public_runner_releases_url: Gitlab.config.gitlab.base_url).execute +``` + +### GitLab subscription management is blocked -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +When you [filter requests](#filter-requests), [GitLab subscription management](../subscriptions/self_managed/index.md) +is blocked. -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +To work around this problem, add `customers.gitlab.com:443` to the +[allowlist](#allow-outbound-requests-to-certain-ip-addresses-and-domains). diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 91da875a049..accc0627a41 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -106,7 +106,7 @@ the tiers are no longer mentioned in GitLab documentation: - Search: - [Filtering merge requests](../user/project/merge_requests/index.md#filter-the-list-of-merge-requests) by approvers - [Filtering merge requests](../user/project/merge_requests/index.md#filter-the-list-of-merge-requests) by "approved by" - - [Advanced Search (Elasticsearch)](../user/search/advanced_search.md) + - [Advanced search (Elasticsearch)](../user/search/advanced_search.md) - [Service Desk](../user/project/service_desk.md) - [Storage usage statistics](../user/usage_quotas.md#storage-usage-statistics) @@ -128,7 +128,7 @@ Bronze-level subscribers: - [Group iterations API](../api/group_iterations.md) - Project milestones API: [Get all burndown chart events for a single milestone](../api/milestones.md#get-all-burndown-chart-events-for-a-single-milestone) - [Project iterations API](../api/iterations.md) - - Fields in the [Search API](../api/search.md) available only to [Advanced Search (Elasticsearch)](../integration/advanced_search/elasticsearch.md) users + - Fields in the [Search API](../api/search.md) available only to [advanced search (Elasticsearch)](../integration/advanced_search/elasticsearch.md) users - Fields in the [Merge requests API](../api/merge_requests.md) for [merge request approvals](../user/project/merge_requests/approvals/index.md) - Fields in the [Protected branches API](../api/protected_branches.md) that specify users or groups allowed to merge - [Merge request approvals API](../api/merge_request_approvals.md) diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index 632581e66d3..cd36e426852 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -70,8 +70,9 @@ The following information is displayed: A GitLab SaaS subscription uses a concurrent (_seat_) model. You pay for a subscription according to the maximum number of users assigned to the top-level group or its children during the billing period. You can -add and remove users during the subscription period, as long as the total users -at any given time doesn't exceed the subscription count. +add and remove users during the subscription period without incurring additional charges, as long as the total users +at any given time doesn't exceed the subscription count. If the total users exceeds your subscription count, you will incur an overage +which must be paid at your next [reconciliation](../quarterly_reconciliation.md). A top-level group can be [changed](../../user/group/manage.md#change-a-groups-path) like any other group. @@ -170,13 +171,13 @@ The user must not be assigned any other role, anywhere in the instance. - If your project is public, all users, including those with the Guest role can access your project. -### Add users to your subscription +### Add seats to your subscription Your subscription cost is based on the maximum number of seats you use during the billing period. Even if you reach the number of seats in your subscription, you can continue to add users. GitLab [bills you for the overage](../quarterly_reconciliation.md). -To add users to a subscription: +To add seats to a subscription: 1. Log in to the [Customers Portal](https://customers.gitlab.com/). 1. Navigate to the **Manage Purchases** page. @@ -236,7 +237,7 @@ amounts at which the alert displays. To change the namespace linked to a subscription: 1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) with a - [linked](../index.md#change-the-linked-account) GitLab.com account. + [linked](../index.md#link-a-gitlabcom-account) GitLab.com account. 1. Go to the **Manage Purchases** page. 1. Select **Change linked namespace**. 1. Select the desired group from the **This subscription is for** dropdown list. For a group to appear here, you must have the Owner role for that group. diff --git a/doc/subscriptions/gitlab_dedicated/index.md b/doc/subscriptions/gitlab_dedicated/index.md index 2d684edb992..0c2b6375d7a 100644 --- a/doc/subscriptions/gitlab_dedicated/index.md +++ b/doc/subscriptions/gitlab_dedicated/index.md @@ -20,19 +20,97 @@ It's the offering of choice for enterprises and organizations in highly regulate ## Available features -- Authentication: Support for instance-level [SAML OmniAuth](../../integration/saml.md) functionality. GitLab Dedicated acts as the service provider, and you must provide the necessary [configuration](../../integration/saml.md#configure-saml-support-in-gitlab) in order for GitLab to communicate with your IdP. This is provided during onboarding. - - SAML [request signing](../../integration/saml.md#sign-saml-authentication-requests-optional), [group sync](../../user/group/saml_sso/group_sync.md#configure-saml-group-sync), and [SAML groups](../../integration/saml.md#configure-users-based-on-saml-group-membership) are supported. -- Networking: - - Public connectivity with support for IP Allowlists. During onboarding, you can optionally specify a list of IP addresses that can access your GitLab Dedicated instance. Subsequently, when an IP not on the allowlist tries to access your instance the connection is refused. - - Optional. Private connectivity via [AWS PrivateLink](https://aws.amazon.com/privatelink/). - You can specify an AWS IAM Principal and preferred Availability Zones during onboarding to enable this functionality. Both Ingress and Egress PrivateLinks are supported. When connecting to an internal service running in your VPC over HTTPS via PrivateLink, GitLab Dedicated supports the ability to use a private SSL certificate, which can be provided during onboarding. -- Upgrades: - - Monthly upgrades tracking one release behind the latest (n-1), with the latest security release. - - Out of band security patches provided for high severity releases. -- Backups: Regular backups taken and tested. -- Choice of cloud region: Upon onboarding, choose the cloud region where you want to deploy your instance. Some AWS regions have limited features and as a result, we are not able to deploy production instances to those regions. See below for the [full list of regions](#aws-regions-not-supported) not currently supported. -- Security: Data encrypted at rest and in transit using latest encryption standards. -- Application: Self-managed [Ultimate feature set](https://about.gitlab.com/pricing/feature-comparison/) with the exception of the unsupported features [listed below](#features-that-are-not-available). +### Data residency + +GitLab Dedicated allows you to select the cloud region where your data will be stored. Upon [onboarding](../../administration/dedicated/index.md#onboarding), choose the cloud region where you want to deploy your Dedicated instance. Some AWS regions have limited features and as a result, we are not able to deploy production instances to those regions. See below for the [full list of regions](#aws-regions-not-supported) not supported. + +### Availability and scalability + +GitLab Dedicated leverages the GitLab [Cloud Native Hybrid reference architectures](../../administration/reference_architectures/index.md#cloud-native-hybrid) with high availability enabled. When [onboarding](../../administration/dedicated/index.md#onboarding), GitLab will match you to the closest reference architecture size based on your number of users. Learn about the [current Service Level Objective](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/#current-service-level-objective). + +#### Disaster Recovery + +When [onboarding](../../administration/dedicated/index.md#onboarding) to GitLab Dedicated, you can provide a Secondary AWS region in which your data is stored. This region is used to recover your GitLab Dedicated instance in case of a disaster. Regular backups of all GitLab Dedicated datastores (including Database and Git repositories) are taken and tested regularly and stored in your desired secondary region. GitLab Dedicated also provides the ability to store copies of these backups in a separate cloud region of choice for greater redundancy. + +For more information, read about the [recovery plan for GitLab Dedicated](https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/slas/#disaster-recovery-plan) as well as RPO and RTO targets. + +### Security + +#### Authentication and authorization + +GitLab Dedicated supports instance-level [SAML OmniAuth](../../integration/saml.md) functionality. Your GitLab Dedicated instance acts as the service provider, and you must provide the necessary [configuration](../../integration/saml.md#configure-saml-support-in-gitlab) in order for GitLab to communicate with your IdP. For more information, see how to [configure SAML](../../administration/dedicated/index.md#saml) for your instance. + +- SAML [request signing](../../integration/saml.md#sign-saml-authentication-requests-optional), [group sync](../../user/group/saml_sso/group_sync.md#configure-saml-group-sync), and [SAML groups](../../integration/saml.md#configure-users-based-on-saml-group-membership) are supported. + +#### Secure networking + +GitLab Dedicated offers public connectivity by default with support for IP allowlists. You can [optionally specify a list of IP addresses](../../administration/dedicated/index.md#ip-allowlist) that can access your GitLab Dedicated instance. Subsequently, when an IP not on the allowlist tries to access your instance the connection is refused. + +Private connectivity via [AWS PrivateLink](https://aws.amazon.com/privatelink/) is also offered as an option. Both [inbound](../../administration/dedicated/index.md#inbound-private-link) and [outbound](../../administration/dedicated/index.md#outbound-private-link) PrivateLinks are supported. When connecting to an internal service running in your VPC over HTTPS via PrivateLink, GitLab Dedicated supports the ability to use a private SSL certificate, which can be provided when [updating your instance configuration](../../administration/dedicated/index.md#custom-certificates). + +#### Encryption + +Data is encrypted at rest and in transit using the latest encryption standards. + +#### Bring your own key encryption + +During onboarding, you can specify an AWS KMS encryption key stored in your own AWS account that GitLab uses to encrypt the data for your Dedicated instance. This gives you full control over the data you store in GitLab. + +### Compliance + +#### Certifications + +GitLab Dedicated offers the following [compliance certifications](https://about.gitlab.com/security/): + +- SOC 2 Type 1 Report (Security and Confidentiality criteria) +- ISO/IEC 27001:2013 +- ISO/IEC 27017:2015 +- ISO/IEC 27018:2019 + +#### Isolation + +As a single-tenant SaaS service, GitLab Dedicated provides infrastructure-level isolation of your GitLab environment. Your environment is placed into a separate AWS account from other tenants. This AWS account contains all of the underlying infrastructure necessary to host the GitLab application and your data stays within the account boundary. You administer the application while GitLab manages the underlying infrastructure. Tenant environments are also completely isolated from GitLab.com. + +#### Access controls + +GitLab Dedicated adheres to the [principle of least privilege](https://about.gitlab.com/handbook/security/access-management-policy.html#principle-of-least-privilege) to control access to customer tenant environments. Tenant AWS accounts live under a top-level GitLab Dedicated AWS parent organization. Access to the AWS Organization is restricted to select GitLab team members. All user accounts within the AWS Organization follow the overall GitLab Access Management Policy [outlined here](https://about.gitlab.com/handbook/security/access-management-policy.html). Direct access to customer tenant environments is restricted to a single Hub account. The GitLab Dedicated Control Plane uses the Hub account to perform automated actions over tenant accounts when managing environments. Similarly, GitLab Dedicated engineers do not have direct access to customer tenant environments. In break glass situations, where access to resources in the tenant environment is required to address a high-severity issue, GitLab engineers must go through the Hub account to manage those resources. This is done via an approval process, and after permission is granted, the engineer will assume an IAM role on a temporary basis to access tenant resources through the Hub account. All actions within the hub account and tenant account are logged to CloudTrail. + +Inside tenant accounts, GitLab leverages Intrusion Detection and Malware Scanning capabilities from AWS GuardDuty. Infrastructure logs are monitored by the GitLab Security Incident Response Team to detect anomalous events. + +#### Audit and observability + +GitLab Dedicated provides access to [audit and system logs](../../administration/dedicated/index.md#access-to-application-logs) generated by the application. + +### Maintenance + +GitLab leverages [weekly maintenance windows](../../administration/dedicated/index.md#maintenance-window) to keep your instance up to date, fix security issues, and ensure the overall reliability and performance of your environment. + +#### Upgrades + +GitLab performs monthly upgrades to your instance with the latest security release during your preferred [maintenance window](../../administration/dedicated/index.md#maintenance-window) tracking one release behind the latest GitLab release. For example, if the latest version of GitLab available is 15.8, GitLab Dedicated runs on 15.7. + +#### Unscheduled maintenance + +GitLab may conduct unscheduled maintenance to address high-severity issues affecting the security, availability, or reliability of your instance. + +### Application + +GitLab Dedicated comes with the self-managed [Ultimate feature set](https://about.gitlab.com/pricing/feature-comparison/) with the exception of the unsupported features [listed below](#features-that-are-not-available). + +#### GitLab Runners + +With GitLab Dedicated, you must [install the GitLab Runner application](https://docs.gitlab.com/runner/install/index.html) on infrastructure that you own or manage. If hosting GitLab Runners on AWS, you can avoid having requests from the Runner fleet route through the public internet by setting up a secure connection from the Runner VPC to the GitLab Dedicated endpoint via AWS Private Link. Learn more about [networking options](#secure-networking). + +#### Migration + +To help you migrate your data to GitLab Dedicated, you can choose from the following options: + +1. When migrating from another GitLab instance, you can either: + - Use the UI, including [group import](../../user/group/import/index.md) and [project import](../../user/project/settings/import_export.md). + - Use APIs, including the [group import API](../../api/group_import_export.md) and [project import API](../../api/project_import_export.md). + - Note: Import functionality behind a feature flag (such as `bulk_import_project`) is not supported in GitLab Dedicated. +1. When migrating from third-party services, you can use [the GitLab importers](../../user/project/import/index.md#available-project-importers). +1. You can perform a fully-automated migration through the [Congregate Automation Tool](../../user/project/import/index.md#automate-group-and-project-import), which supports migrating from existing GitLab instances as well as third-party services. ## Features that are not available @@ -42,7 +120,7 @@ The following GitLab application features are not available: - LDAP, Smartcard, or Kerberos authentication - Multiple login providers -- Advanced Search +- Advanced search - GitLab Pages - FortiAuthenticator, or FortiToken 2FA - Reply-by email @@ -53,14 +131,13 @@ The following GitLab application features are not available: The following features will not be supported: - Mattermost -- Server-side Git hooks +- Server-side Git hooks. Use [push rules](../../user/project/repository/push_rules.md) instead. ### GitLab Dedicated service features The following operational features are not available: - Custom domains -- Bring Your Own Key (BYOK) encryption - Multiple Geo secondaries (Geo replicas) beyond the secondary site included by default - Self-serve purchasing and configuration - Multiple login providers diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 2c9f93886bf..f1a960a6aa9 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -73,6 +73,7 @@ With the [Customers Portal](https://customers.gitlab.com/) you can: - [Change account owner information](#change-account-owner-information) - [Change your company details](#change-your-company-details) - [Change your payment method](#change-your-payment-method) +- [Link a GitLab.com account](#link-a-gitlabcom-account) - [Change the linked account](#change-the-linked-account) - [Change the namespace the subscription is linked to](gitlab_com/index.md#change-the-linked-namespace) - [Change customers portal account password](#change-customers-portal-account-password) @@ -137,18 +138,26 @@ method as the default: 1. **Edit** the selected payment method and check the **Make default payment method** checkbox. 1. Select **Save Changes**. +### Link a GitLab.com account + +Follow this guideline if you have a legacy Customers Portal account and use an email and password to log in. + +To link a GitLab.com account to your Customers Portal account: + +1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in?legacy=true) using email and password. +1. On the Customers Portal page, select **My account > Account details**. +1. Under **Your GitLab.com account**, select **Link account**. +1. Log in to the [GitLab.com](https://gitlab.com/users/sign_in) account you want to link to the Customers Portal account. + ### Change the linked account To change the GitLab.com account linked to your Customers Portal account: -1. Log in to the - [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. In a separate browser tab, go to [GitLab.com](https://gitlab.com/users/sign_in) and ensure you - are not logged in. +1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. In a separate browser tab, go to [GitLab.com](https://gitlab.com/users/sign_in) and ensure you are not logged in. 1. On the Customers Portal page, select **My account > Account details**. 1. Under **Your GitLab.com account**, select **Change linked account**. -1. Log in to the [GitLab.com](https://gitlab.com/users/sign_in) account you want to link to the Customers Portal - account. +1. Log in to the [GitLab.com](https://gitlab.com/users/sign_in) account you want to link to the Customers Portal account. ### Change Customers Portal account password @@ -176,7 +185,7 @@ To meet GitLab for Open Source Program requirements, first add an OSI-approved o To add a license to a project: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the overview page, select **Add LICENSE**. If the license you want is not available as a license template, manually copy the entire, unaltered [text of your chosen license](https://opensource.org/licenses/alphabetical) into the `LICENSE` file. Note that GitLab defaults to **All rights reserved** if users do not perform this action. +1. On the overview page, select **Add LICENSE**. If the license you want is not available as a license template, manually copy the entire, unaltered [text of your chosen license](https://opensource.org/licenses/) into the `LICENSE` file. Note that GitLab defaults to **All rights reserved** if users do not perform this action.  diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index b5436b6cb72..52a39fb3f49 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -280,7 +280,7 @@ If you are an administrator, you can export your license usage into a CSV: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Subscription**. -1. In the upper right, select **Export license usage file**. +1. In the upper-right corner, select **Export license usage file**. This file contains the information GitLab uses to manually process quarterly reconciliations or renewals. If your instance is firewalled or an offline environment, you must provide GitLab with this information. diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index c1d0a69e1f4..55a58377b76 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -12,7 +12,7 @@ This page gathers all the resources for the topic **Authentication** within GitL - [SSH](../../user/ssh.md) - [Two-factor authentication](../../user/profile/account/two_factor_authentication.md) -- [Why do you keep getting signed out?](../../user/profile/index.md#why-do-you-keep-getting-signed-out) +- [Stay signed in indefinitely](../../user/profile/index.md#stay-signed-in-indefinitely) - **Articles:** - [Support for Universal 2nd Factor Authentication - YubiKeys](https://about.gitlab.com/blog/2016/06/22/gitlab-adds-support-for-u2f/) - [Security Webcast with Yubico](https://about.gitlab.com/blog/2016/08/31/gitlab-and-yubico-security-webcast/) diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md new file mode 100644 index 00000000000..30cbe06ab95 --- /dev/null +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md @@ -0,0 +1,301 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Use Auto DevOps to deploy an application to Amazon Elastic Kubernetes Service (EKS) + +In this tutorial, we'll help you to get started with [Auto DevOps](../index.md) +through an example of how to deploy an application to Amazon Elastic Kubernetes Service (EKS). + +The tutorial uses the GitLab native Kubernetes integration, so you don't need +to create a Kubernetes cluster manually using the AWS console. + +You can also follow this tutorial on a self-managed instance. +Ensure your own [runners are configured](../../../ci/runners/index.md). + +To deploy a project to EKS: + +1. [Configure your Amazon account](#configure-your-amazon-account) +1. [Create a Kubernetes cluster and deploy the agent](#create-a-kubernetes-cluster) +1. [Create a new project from a template](#create-an-application-project-from-a-template) +1. [Configure the agent](#configure-the-agent) +1. [Install Ingress](#install-ingress) +1. [Configure Auto DevOps](#configure-auto-devops) +1. [Enable Auto DevOps and run the pipeline](#enable-auto-devops-and-run-the-pipeline) +1. [Deploy the application](#deploy-the-application) + +## Configure your Amazon account + +Before you create and connect your Kubernetes cluster to your GitLab project, +you need an [Amazon Web Services account](https://https://aws.amazon.com/). +Sign in with an existing Amazon account or create a new one. + +## Create a Kubernetes cluster + +To create an new cluster on Amazon EKS: + +- Follow the steps in [Create an Amazon EKS cluster](../../../user/infrastructure/clusters/connect/new_eks_cluster.md). + +If you prefer, you can also create a cluster manually using `eksctl`. + +## Create an application project from a template + +Use a GitLab project template to get started. As the name suggests, +those projects provide a bare-bones application built on some well-known frameworks. + +WARNING: +Create the application project in the group hierarchy at the same level or below the project for cluster management. Otherwise, it fails to [authorize the agent](../../../user/clusters/agent/ci_cd_workflow.md#authorize-the-agent). + +1. On the top bar in GitLab, select the plus icon (**{plus-square}**), and select + **New project/repository**. +1. Select **Create from template**. +1. Select the **Ruby on Rails** template. +1. Give your project a name, optionally a description, and make it public so that + you can take advantage of the features available in the + [GitLab Ultimate plan](https://about.gitlab.com/pricing/). +1. Select **Create project**. + +Now you have an application project you are going to deploy to the EKS cluster. + +## Configure the agent + +Next, we'll configure the GitLab agent for Kubernetes so we can use it to deploy the application project. + +1. Go to the project [we created to manage the cluster](#create-a-kubernetes-cluster). +1. Navigate to the [agent configuration file](../../../user/clusters/agent/install/index.md#create-an-agent-configuration-file) (`.gitlab/agents/eks-agent/config.yaml`) and edit it. +1. Configure `ci_access:projects` attribute. Use the application project path as `id`: + +```yaml +ci_access: + projects: + - id: path/to/application-project +``` + +## Install Ingress + +After your cluster is running, you must install NGINX Ingress Controller as a +load balancer to route traffic from the internet to your application. +Install the NGINX Ingress Controller +through the GitLab [Cluster management project template](../../../user/clusters/management_project_template.md), +or manually via the command line: + +1. Ensure you have `kubectl` and Helm installed on your machine. +1. Create an IAM role to access the cluster. +1. Create an access token to access the cluster. +1. Use `kubectl` to connect to your cluster: + + ```shell + helm upgrade --install ingress-nginx ingress-nginx \ + --repo https://kubernetes.github.io/ingress-nginx \ + --namespace gitlab-managed-apps --create-namespace + + # Check that the ingress controller is installed successfully + kubectl get service ingress-nginx-controller -n gitlab-managed-apps + ``` + +## Configure Auto DevOps + +Follow these steps to configure the base domain and other settings required for Auto DevOps. + +1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can + get the external IP address with the following command: + + ```shell + kubectl get all -n gitlab-managed-apps --selector app.kubernetes.io/instance=ingress-nginx + ``` + + Replace `gitlab-managed-apps` if you have overwritten your namespace. + + Next, find the actual external IP address for your cluster with the following command: + + ```shell + nslookup [External IP] + ``` + + Where the `[External IP]` is the hostname found with the previous command. + + The IP address might be listed in the `Non-authoritative answer:` section of the response. + + Copy this IP address, as you need it in the next step. + +1. Go back to the application project. +1. On the left sidebar, select **Settings > CI/CD** and expand **Variables**. + - Add a key called `KUBE_INGRESS_BASE_DOMAIN` with the application deployment domain as the value. For this example, use the domain `<IP address>.nip.io`. + - Add a key called `KUBE_NAMESPACE` with a value of the Kubernetes namespace for your deployments to target. You can use different namespaces per environment. Configure the environment, use the environment scope. + - Add a key called `KUBE_CONTEXT` with a value like `path/to/agent/project:eks-agent`. Select the environment scope of your choice. + - Select **Save changes**. + +## Enable Auto DevOps and run the pipeline + +While Auto DevOps is enabled by default, Auto DevOps can be disabled at both +the instance level (for self-managed instances) and the group level. Complete +these steps to enable Auto DevOps if it's disabled: + +1. On the top bar, select **Main menu > Projects** and find the application project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Auto DevOps**. +1. Select **Default to Auto DevOps pipeline** to display more options. +1. In **Deployment strategy**, select your desired [continuous deployment strategy](../requirements.md#auto-devops-deployment-strategy) + to deploy the application to production after the pipeline successfully runs on the default branch. +1. Select **Save changes**. +1. Edit `.gitlab-ci.yml` file to include the Auto DevOps template and commit the change to the default branch: + + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + ``` + +The commit should trigger a pipeline. In the next section, we explain what each job does in the pipeline. + +## Deploy the application + +When your pipeline runs, what is it doing? + +To view the jobs in the pipeline, select the pipeline's status badge. The +**{status_running}** icon displays when pipeline jobs are running, and updates +without refreshing the page to **{status_success}** (for success) or +**{status_failed}** (for failure) when the jobs complete. + +The jobs are separated into stages: + + + +- **Build** - The application builds a Docker image and uploads it to your project's + [Container Registry](../../../user/packages/container_registry/index.md) ([Auto Build](../stages.md#auto-build)). +- **Test** - GitLab runs various checks on the application, but all jobs except `test` + are allowed to fail in the test stage: + + - The `test` job runs unit and integration tests by detecting the language and + framework ([Auto Test](../stages.md#auto-test-deprecated)) + - The `code_quality` job checks the code quality and is allowed to fail + ([Auto Code Quality](../stages.md#auto-code-quality)) + - The `container_scanning` job checks the Docker container if it has any + vulnerabilities and is allowed to fail ([Auto Container Scanning](../stages.md#auto-container-scanning)) + - The `dependency_scanning` job checks if the application has any dependencies + susceptible to vulnerabilities and is allowed to fail + ([Auto Dependency Scanning](../stages.md#auto-dependency-scanning)) + - Jobs suffixed with `-sast` run static analysis on the current code to check for potential + security issues, and are allowed to fail ([Auto SAST](../stages.md#auto-sast)) + - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](../stages.md#auto-secret-detection)) + - The `license_scanning` job searches the application's dependencies to determine each of their + licenses and is allowed to fail + ([Auto License Compliance](../stages.md#auto-license-compliance)) + +- **Review** - Pipelines on the default branch include this stage with a `dast_environment_deploy` job. + To learn more, see [Dynamic Application Security Testing (DAST)](../../../user/application_security/dast/index.md). + +- **Production** - After the tests and checks finish, the application deploys in + Kubernetes ([Auto Deploy](../stages.md#auto-deploy)). + +- **Performance** - Performance tests are run on the deployed application + ([Auto Browser Performance Testing](../stages.md#auto-browser-performance-testing)). + +- **Cleanup** - Pipelines on the default branch include this stage with a `stop_dast_environment` job. + +After running a pipeline, you should view your deployed website and learn how +to monitor it. + +### Monitor your project + +After successfully deploying your application, you can view its website and check +on its health on the **Environments** page by navigating to +**Deployments > Environments**. This page displays details about +the deployed applications, and the right-hand column displays icons that link +you to common environment tasks: + + + +- **Open live environment** (**{external-link}**) - Opens the URL of the application deployed in production +- **Monitoring** (**{chart}**) - Opens the metrics page where Prometheus collects data + about the Kubernetes cluster and how the application + affects it in terms of memory usage, CPU usage, and latency +- **Deploy to** (**{play}** **{chevron-lg-down}**) - Displays a list of environments you can deploy to +- **Terminal** (**{terminal}**) - Opens a [web terminal](../../../ci/environments/index.md#web-terminals-deprecated) + session inside the container where the application is running +- **Re-deploy to environment** (**{repeat}**) - For more information, see + [Retrying and rolling back](../../../ci/environments/index.md#retry-or-roll-back-a-deployment) +- **Stop environment** (**{stop}**) - For more information, see + [Stopping an environment](../../../ci/environments/index.md#stop-an-environment) + +GitLab displays the [deploy board](../../../user/project/deploy_boards.md) below the +environment's information, with squares representing pods in your +Kubernetes cluster, color-coded to show their status. Hovering over a square on +the deploy board displays the state of the deployment, and selecting the square +takes you to the pod's logs page. + +Although the example shows only one pod hosting the application at the moment, you can add +more pods by defining the [`REPLICAS` CI/CD variable](../cicd_variables.md) +in **Settings > CI/CD > Variables**. + +### Work with branches + +Following the [GitLab flow](../../gitlab_flow.md#working-with-feature-branches), +you should next create a feature branch to add content to your application: + +1. In your project's repository, go to the following file: `app/views/welcome/index.html.erb`. + This file should only contain a paragraph: `<p>You're on Rails!</p>`. +1. Open the GitLab [Web IDE](../../../user/project/web_ide/index.md) to make the change. +1. Edit the file so it contains: + + ```html + <p>You're on Rails! Powered by GitLab Auto DevOps.</p> + ``` + +1. Stage the file. Add a commit message, then create a new branch and a merge request + by selecting **Commit**. + +  + +After submitting the merge request, GitLab runs your pipeline, and all the jobs +in it, as [described previously](#deploy-the-application), in addition to +a few more that run only on branches other than the default branch. + +After a few minutes a test fails, which means a test was +'broken' by your change. Select the failed `test` job to see more information +about it: + +```plaintext +Failure: +WelcomeControllerTest#test_should_get_index [/app/test/controllers/welcome_controller_test.rb:7]: +<You're on Rails!> expected but was +<You're on Rails! Powered by GitLab Auto DevOps.>.. +Expected 0 to be >= 1. + +bin/rails test test/controllers/welcome_controller_test.rb:4 +``` + +To fix the broken test: + +1. Return to your merge request. +1. In the upper right corner, select **Code**, then select **Open in Gitpod**. +1. In the left-hand directory of files, find the `test/controllers/welcome_controller_test.rb` + file, and select it to open it. +1. Change line 7 to say `You're on Rails! Powered by GitLab Auto DevOps.` +1. Select **Commit**. +1. In the left-hand column, under **Unstaged changes**, select the checkmark icon + (**{stage-all}**) to stage the changes. +1. Write a commit message, and select **Commit**. + +Return to the **Overview** page of your merge request, and you should not only +see the test passing, but also the application deployed as a +[review application](../stages.md#auto-review-apps). You can visit it by selecting +the **View app** **{external-link}** button to see your changes deployed. + +After merging the merge request, GitLab runs the pipeline on the default branch, +and then deploys the application to production. + +## Conclusion + +After implementing this project, you should have a solid understanding of the basics of Auto DevOps. +You started from building and testing, to deploying and monitoring an application +all in GitLab. Despite its automatic nature, Auto DevOps can also be configured +and customized to fit your workflow. Here are some helpful resources for further reading: + +1. [Auto DevOps](../index.md) +1. [Multiple Kubernetes clusters](../multiple_clusters_auto_devops.md) +1. [Incremental rollout to production](../cicd_variables.md#incremental-rollout-to-production) +1. [Disable jobs you don't need with CI/CD variables](../cicd_variables.md) +1. [Use your own buildpacks to build your application](../customize.md#custom-buildpacks) +1. [Prometheus monitoring](../../../user/project/integrations/prometheus.md) diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md index 9bd1d30e1b1..06b772cc455 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md @@ -274,7 +274,7 @@ bin/rails test test/controllers/welcome_controller_test.rb:4 To fix the broken test: 1. Return to your merge request. -1. In the upper right corner, select **Code**, then select **Open in Gitpod**. +1. In the upper-right corner, select **Code**, then select **Open in Gitpod**. 1. In the left-hand directory of files, find the `test/controllers/welcome_controller_test.rb` file, and select it to open it. 1. Change line 7 to say `You're on Rails! Powered by GitLab Auto DevOps.` diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 588be855659..2a0cc7f39b1 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -185,6 +185,7 @@ and clear the **Default to Auto DevOps pipeline** checkbox. ### Deploy your app to a cloud provider - [Use Auto DevOps to deploy to a Kubernetes cluster on Google Kubernetes Engine (GKE)](cloud_deployments/auto_devops_with_gke.md) +- [Use Auto DevOps to deploy to a Kubernetes cluster on Amazon Elastic Kubernetes Service (EKS)](cloud_deployments/auto_devops_with_eks.md) - [Use Auto DevOps to deploy to EC2](cloud_deployments/auto_devops_with_ec2.md) - [Use Auto DevOps to deploy to ECS](cloud_deployments/auto_devops_with_ecs.md) diff --git a/doc/topics/autodevops/upgrading_postgresql.md b/doc/topics/autodevops/upgrading_postgresql.md index f18d5c49be5..daf1d5341d3 100644 --- a/doc/topics/autodevops/upgrading_postgresql.md +++ b/doc/topics/autodevops/upgrading_postgresql.md @@ -49,12 +49,12 @@ being modified after the database dump is created. 1. Get the Kubernetes namespace for the environment. It typically looks like `<project-name>-<project-id>-<environment>`. In our example, the namespace is called `minimal-ruby-app-4349298-production`. - ```shell - $ kubectl get ns + ```shell + $ kubectl get ns - NAME STATUS AGE - minimal-ruby-app-4349298-production Active 7d14h - ``` + NAME STATUS AGE + minimal-ruby-app-4349298-production Active 7d14h + ``` 1. For ease of use, export the namespace name: @@ -64,20 +64,20 @@ being modified after the database dump is created. 1. Get the deployment name for your application with the following command. In our example, the deployment name is `production`. - ```shell - $ kubectl get deployment --namespace "$APP_NAMESPACE" - NAME READY UP-TO-DATE AVAILABLE AGE - production 2/2 2 2 7d21h - production-postgres 1/1 1 1 7d21h - ``` + ```shell + $ kubectl get deployment --namespace "$APP_NAMESPACE" + NAME READY UP-TO-DATE AVAILABLE AGE + production 2/2 2 2 7d21h + production-postgres 1/1 1 1 7d21h + ``` 1. To prevent the database from being modified, set replicas to 0 for the deployment with the following command. We use the deployment name from the previous step (`deployments/<DEPLOYMENT_NAME>`). - ```shell - $ kubectl scale --replicas=0 deployments/production --namespace "$APP_NAMESPACE" - deployment.extensions/production scaled - ``` + ```shell + $ kubectl scale --replicas=0 deployments/production --namespace "$APP_NAMESPACE" + deployment.extensions/production scaled + ``` 1. You must also set replicas to zero for workers if you have any. @@ -85,26 +85,26 @@ being modified after the database dump is created. 1. Get the service name for PostgreSQL. The name of the service should end with `-postgres`. In our example the service name is `production-postgres`. - ```shell - $ kubectl get svc --namespace "$APP_NAMESPACE" - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - production-auto-deploy ClusterIP 10.30.13.90 <none> 5000/TCP 7d14h - production-postgres ClusterIP 10.30.4.57 <none> 5432/TCP 7d14h - ``` + ```shell + $ kubectl get svc --namespace "$APP_NAMESPACE" + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + production-auto-deploy ClusterIP 10.30.13.90 <none> 5000/TCP 7d14h + production-postgres ClusterIP 10.30.4.57 <none> 5432/TCP 7d14h + ``` 1. Get the pod name for PostgreSQL with the following command. In our example, the pod name is `production-postgres-5db86568d7-qxlxv`. - ```shell - $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=production-postgres - NAME READY STATUS RESTARTS AGE - production-postgres-5db86568d7-qxlxv 1/1 Running 0 7d14h - ``` + ```shell + $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=production-postgres + NAME READY STATUS RESTARTS AGE + production-postgres-5db86568d7-qxlxv 1/1 Running 0 7d14h + ``` 1. Connect to the pod with: - ```shell - kubectl exec -it production-postgres-5db86568d7-qxlxv --namespace "$APP_NAMESPACE" -- bash - ``` + ```shell + kubectl exec -it production-postgres-5db86568d7-qxlxv --namespace "$APP_NAMESPACE" -- bash + ``` 1. Once, connected, create a dump file with the following command. @@ -114,20 +114,20 @@ being modified after the database dump is created. - When prompted for the database password, the default is `testing-password`. - ```shell - ## Format is: - # pg_dump -h SERVICE_NAME -U USERNAME DATABASE_NAME > /tmp/backup.sql + ```shell + ## Format is: + # pg_dump -h SERVICE_NAME -U USERNAME DATABASE_NAME > /tmp/backup.sql - pg_dump -h production-postgres -U user production > /tmp/backup.sql - ``` + pg_dump -h production-postgres -U user production > /tmp/backup.sql + ``` 1. Once the backup dump is complete, exit the Kubernetes exec process with `Control-D` or `exit`. 1. Download the dump file with the following command: - ```shell - kubectl cp --namespace "$APP_NAMESPACE" production-postgres-5db86568d7-qxlxv:/tmp/backup.sql backup.sql - ``` + ```shell + kubectl cp --namespace "$APP_NAMESPACE" production-postgres-5db86568d7-qxlxv:/tmp/backup.sql backup.sql + ``` ## Retain persistent volumes @@ -184,8 +184,7 @@ You can also 1. Set `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value. This flag is a safeguard to prevent accidental deletion of databases. <!-- DO NOT REPLACE when upgrading GitLab's supported version. This is NOT related to GitLab's PostgreSQL version support, but the one deployed by Auto DevOps. --> -1. If you have a `POSTGRES_VERSION` set, make sure it is set to `9.6.16` *or -higher*. This is the +1. If you have a `POSTGRES_VERSION` set, make sure it is set to `9.6.16` *or higher*. This is the minimum PostgreSQL version supported by Auto DevOps. See also the list of [tags available](https://hub.docker.com/r/bitnami/postgresql/tags). 1. Set `PRODUCTION_REPLICAS` to `0`. For other environments, use @@ -205,11 +204,11 @@ higher*. This is the 1. Get the pod name for the new PostgreSQL, in our example, the pod name is `production-postgresql-0`: - ```shell - $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=postgresql - NAME READY STATUS RESTARTS AGE - production-postgresql-0 1/1 Running 0 19m - ```` + ```shell + $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=postgresql + NAME READY STATUS RESTARTS AGE + production-postgresql-0 1/1 Running 0 19m + ```` 1. Copy the dump file from the backup steps to the pod: diff --git a/doc/topics/awesome_co.md b/doc/topics/awesome_co.md index ff3c81a1b90..ffda564cd91 100644 --- a/doc/topics/awesome_co.md +++ b/doc/topics/awesome_co.md @@ -141,3 +141,73 @@ create(:project, name: 'Throws Error', namespace: create(:group, name: 'Some Gro create(:project, name: 'No longer throws error', owner: @owner, namespace: create(:group, name: 'Some Group')) create(:epic, group: create(:group), author: @owner) ``` + +## YAML Factories + +### Generator to generate _n_ amount of records + +### [Group Labels](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories/labels.rb) + +```yaml +group_labels: + # Group Label with Name and a Color + - name: Group Label 1 + group_id: <%= @group.id %> + color: "#FF0000" +``` + +### [Group Milestones](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories/milestones.rb) + +```yaml +group_milestones: + # Past Milestone + - name: Past Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.month.ago %> + due_date: <%= 1.day.ago %> + + # Ongoing Milestone + - name: Ongoing Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.day.ago %> + due_date: <%= 1.month.from_now %> + + # Future Milestone + - name: Ongoing Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.month.from_now %> + due_date: <%= 2.months.from_now %> +``` + +#### Quirks + +- You _must_ specify `group:` and have it be empty. This is because the Milestones factory will manipulate the factory in an `after(:build)`. If this is not present, the Milestone will not be associated properly with the Group. + +### [Epics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/factories/epics.rb) + +```yaml +epics: + # Simple Epic + - title: Simple Epic + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + + # Epic with detailed Markdown description + - title: Detailed Epic + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + description: | + # Markdown + + **Description** + + # Epic with dates + - title: Epic with dates + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + start_date: <%= 1.day.ago %> + due_date: <%= 1.month.from_now %> +``` diff --git a/doc/topics/git/bisect.md b/doc/topics/git/bisect.md index a7f8b2a846c..eaf619ce36f 100644 --- a/doc/topics/git/bisect.md +++ b/doc/topics/git/bisect.md @@ -1,76 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Bisect **(FREE)** +This document was moved to [another location](index.md). -- Find a commit that introduced a bug -- Works through a process of elimination -- Specify a known good and bad revision to begin - -## Bisect sample workflow - -1. Start the bisect process -1. Enter the bad revision (usually latest commit) -1. Enter a known good revision (commit/branch) -1. Run code to see if bug still exists -1. Tell bisect the result -1. Repeat the previous 2 items until you find the offending commit - -## Setup - -```shell - mkdir bisect-ex - cd bisect-ex - touch index.html - git add -A - git commit -m "starting out" - vi index.html - # Add all good - git add -A - git commit -m "second commit" - vi index.html - # Add all good 2 - git add -A - git commit -m "third commit" - vi index.html -``` - -```shell - # Add all good 3 - git add -A - git commit -m "fourth commit" - vi index.html - # This looks bad - git add -A - git commit -m "fifth commit" - vi index.html - # Really bad - git add -A - git commit -m "sixth commit" - vi index.html - # again just bad - git add -A - git commit -m "seventh commit" -``` - -## Commands - -```shell - git bisect start - # Test your code - git bisect bad - git bisect next - # Say yes to the warning - # Test - git bisect good - # Test - git bisect bad - # Test - git bisect good - # done - git bisect reset -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/feature_branching.md b/doc/topics/git/feature_branching.md index 73de351fde2..eaf619ce36f 100644 --- a/doc/topics/git/feature_branching.md +++ b/doc/topics/git/feature_branching.md @@ -1,31 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Feature branching **(FREE)** +This document was moved to [another location](index.md). -- Efficient parallel workflow for teams -- Develop each feature in a branch -- Keeps changes isolated -- Consider a 1-to-1 link to issues -- Push branches to the server frequently - - Hint: Pushing branches is a cheap backup for your work-in-progress code. - -## Feature branching sample workflow - -1. Create a new feature branch called `squash_some_bugs` -1. Edit '`bugs.rb`' and remove all the bugs. -1. Commit -1. Push - -```shell -git checkout -b squash_some_bugs -# Edit `bugs.rb` -git status -git add bugs.rb -git commit -m 'Fix some buggy code' -git push origin squash_some_bugs -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/git_log.md b/doc/topics/git/git_log.md index e2e9c0e8804..eaf619ce36f 100644 --- a/doc/topics/git/git_log.md +++ b/doc/topics/git/git_log.md @@ -1,60 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Git Log **(FREE)** +This document was moved to [another location](index.md). -Git log lists commit history. It allows searching and filtering. - -- Initiate log: - - ```shell - git log - ``` - -- Retrieve set number of records: - - ```shell - git log -n 2 - ``` - -- Search commits by author. Allows user name or a regular expression. - - ```shell - git log --author="user_name" - ``` - -- Search by comment message: - - ```shell - git log --grep="<pattern>" - ``` - -- Search by date: - - ```shell - git log --since=1.month.ago --until=3.weeks.ago - ``` - -## Git Log Workflow - -1. Change to workspace directory -1. Clone the multi runner projects -1. Change to project dir -1. Search by author -1. Search by date -1. Combine - -## Commands - -```shell -cd ~/workspace -git clone git@gitlab.com:gitlab-org/gitlab-runner.git -cd gitlab-runner -git log --author="Travis" -git log --since=1.month.ago --until=3.weeks.ago -git log --since=1.month.ago --until=1.day.ago --author="Travis" -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index b47a34fa7b2..fa802952b32 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -32,7 +32,6 @@ The following resources can help you get started with Git: - [How to install Git](how_to_install_git/index.md) - [Git terminology](terminology.md) - [Start using Git on the command line](../../gitlab-basics/start-using-git.md) -- [Edit files through the command line](../../gitlab-basics/command-line-commands.md) - [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) - Commits: - [Revert a commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) @@ -43,7 +42,7 @@ The following resources can help you get started with Git: - [Git stash](stash.md) - [Git file blame](../../user/project/repository/git_blame.md) - [Git file history](../../user/project/repository/git_history.md) -- [Git tags](tags.md) +- [Git tags](../../user/project/repository/tags/index.md) ### Concepts @@ -58,16 +57,11 @@ The following are resources on version control concepts: You can do many Git tasks from the command line: -- [Bisect](bisect.md). - [Cherry-pick](cherry_picking.md). -- [Feature branching](feature_branching.md). - [Getting started with Git](getting_started.md). - [Git add](git_add.md). -- [Git log](git_log.md). - [Git stash](stash.md). -- [Merge conflicts](merge_conflicts.md). - [Rollback commits](rollback_commits.md). -- [Subtree](subtree.md). - [Unstage](unstage.md). ## Git tips diff --git a/doc/topics/git/merge_conflicts.md b/doc/topics/git/merge_conflicts.md deleted file mode 100644 index ea37afe1b31..00000000000 --- a/doc/topics/git/merge_conflicts.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../user/project/merge_requests/conflicts.md#resolve-conflicts-from-the-command-line' -remove_date: '2023-02-01' ---- - -This document was moved to [another location](../../user/project/merge_requests/conflicts.md#resolve-conflicts-from-the-command-line). - -<!-- This redirect file can be deleted after <2023-02-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/partial_clone.md b/doc/topics/git/partial_clone.md index f5f11b17675..de0547ae49d 100644 --- a/doc/topics/git/partial_clone.md +++ b/doc/topics/git/partial_clone.md @@ -94,9 +94,7 @@ Updating files: 100% (28/28), done. $ cd www-gitlab-com -$ git sparse-checkout init --cone - -$ git sparse-checkout add data +$ git sparse-checkout set data --cone remote: Enumerating objects: 301, done. remote: Counting objects: 100% (301/301), done. remote: Compressing objects: 100% (292/292), done. diff --git a/doc/topics/git/subtree.md b/doc/topics/git/subtree.md index a8a665d4e13..eaf619ce36f 100644 --- a/doc/topics/git/subtree.md +++ b/doc/topics/git/subtree.md @@ -1,50 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Subtree **(FREE)** +This document was moved to [another location](index.md). -- Used when there are nested repositories. -- Not recommended when the amount of dependencies is too large. -- For these cases we need a dependency control system. -- Command are painfully long so aliases are necessary. - -## Subtree Aliases - -- Add: `git subtree add --prefix <target-folder> <url> <branch> --squash` -- Pull: `git subtree pull --prefix <target-folder> <url> <branch> --squash` -- Push: `git subtree push --prefix <target-folder> <url> <branch>` - -```shell - # Add an alias - # Add - git config alias.sba 'subtree add --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master --squash' - # Pull - git config alias.sbpl 'subtree pull --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master --squash' - # Push - git config alias.sbph 'subtree push --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master' - - # Adding this subtree adds a st dir with a readme - git sba - vi st/README.md - # Edit file - git status shows differences - -``` - -```shell - # Adding, or committing won't change the sub repo at remote - # even if we push - git add -A - git commit -m "Adding to subtree readme" - - # Push to subtree repo - git sbph - # now we can check our remote sub repo -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/tags.md b/doc/topics/git/tags.md index ab196f409f7..c66c97717f2 100644 --- a/doc/topics/git/tags.md +++ b/doc/topics/git/tags.md @@ -1,41 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../user/project/repository/tags/index.md' +remove_date: '2023-05-27' --- -# Tags **(FREE)** +This document was moved to [another location](../../user/project/repository/tags/index.md). -Tags help you mark certain deployments and releases for later -reference. Git supports two types of tags: - -- Annotated tags: An unchangeable part of Git history. -- Lightweight (soft) tags: Tags that can be set and removed as needed. - -Many projects combine an annotated release tag with a stable branch. Consider -setting deployment or release tags automatically. - -## Tags sample workflow - -1. Create a lightweight tag. -1. Create an annotated tag. -1. Push the tags to the remote repository. - -```shell -git checkout master - -# Lightweight tag -git tag my_lightweight_tag - -# Annotated tag -git tag -a v1.0 -m 'Version 1.0' - -# Show list of the existing tags -git tag - -git push origin --tags -``` - -## Related topics - -- [Tagging](https://git-scm.com/book/en/v2/Git-Basics-Tagging) Git reference page +<!-- This redirect file can be deleted after <2023-05-27>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index a116f6cc978..37450319350 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -221,11 +221,17 @@ apply more than one: 1. Modify the GitLab instance's [`gitlab.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/13.5.1+ee.0/files/gitlab-config-template/gitlab.rb.template#L1435-1455) file: - ```shell - gitaly['gitconfig'] = [ - # Set the http.postBuffer size, in bytes - {key: "http.postBuffer", value: "524288000"}, - ] + ```ruby + gitaly['configuration'] = { + # ... + git: { + # ... + config: [ + # Set the http.postBuffer size, in bytes + {key: "http.postBuffer", value: "524288000"}, + ], + }, + } ``` 1. After applying this change, apply the configuration change: diff --git a/doc/topics/git/unstage.md b/doc/topics/git/unstage.md index 142a80a9ad9..aad5d0f44f8 100644 --- a/doc/topics/git/unstage.md +++ b/doc/topics/git/unstage.md @@ -5,15 +5,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w comments: false --- -# Unstage **(FREE)** +# Unstage a file in Git **(FREE)** -- To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This unstages the file but maintain the modifications. +When you _stage_ a file in Git, you instruct Git to track changes to the file in +preparation for a commit. To instruct Git to disregard changes to a file, and not +include it in your next commit, _unstage_ the file. + +- To remove files from stage use `reset HEAD`, where HEAD is the last commit of + the current branch. This unstages the file but maintains the modifications. ```shell git reset HEAD <file> ``` -- To revert the file back to the state it was in before the changes we can use: +- To revert the file back to the state it was in before the changes: ```shell git checkout -- <file> @@ -26,7 +31,8 @@ comments: false git rm -r <dirname> ``` -- If we want to remove a file from the repository but keep it on disk, say we forgot to add it to our `.gitignore` file then use `--cache`: +- To keep a file on disk but remove it from the repository (such as a file you want + to add to `.gitignore`), use the `rm` command with the `--cache` flag: ```shell git rm <filename> --cache diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md index 235412f511a..22548be2e8b 100644 --- a/doc/topics/git/useful_git_commands.md +++ b/doc/topics/git/useful_git_commands.md @@ -130,12 +130,6 @@ Use this to check the Git history of the file: git log -- <file> ``` -### Find the tags that contain a particular SHA - -```shell -git tag --contains <sha> -``` - ### Check the content of each change to a file ```shell diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 80ce703f7db..b2767f2e5eb 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -214,3 +214,89 @@ and Praefect servers so they can use an accessible NTP server. On offline instances, the [GitLab Geo check Rake task](../../administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-site-correctly) always fails because it uses `pool.ntp.org`. This error can be ignored but you can [read more about how to work around it](../../administration/geo/replication/troubleshooting.md#message-machine-clock-is-synchronized--exception). + +## Enabling the package metadata database + +Enabling the package metadata database is required to enable [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files). +This process will require usage of the GitLab License Database, which is licensed under the [EE License](https://storage.googleapis.com/prod-export-license-bucket-1a6c642fc4de57d4/v1/LICENSE). +Please note the following in relation to use of the License Database: + +- We may change or discontinue all or any part of the License Database, at any time and without notice, at our sole discretion. +- The License Database may contain links to third-party websites or resources. We provide these links only as a convenience and are not responsible for any third-party data, content, products, or services from those websites or resources or links displayed on such websites. +- The License Database is based in part on information made available by third parties, and GitLab is not responsible for the accuracy or completeness of content made available. + +### Using the gsutil tool to download the package metadata exports + +1. Install the [`gsutil`](https://cloud.google.com/storage/docs/gsutil_install) tool. +1. Find the root of the GitLab Rails directory. + + ```shell + export GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" + echo $GITLAB_RAILS_ROOT_DIR + ``` + +1. Download the package metadata exports. + + ```shell + # To download the package metadata exports, an outbound connection to Google Cloud Storage bucket must be allowed. + mkdir $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + + # Alternatively, if the GitLab instance is not allowed to connect to the Google Cloud Storage bucket, the package metadata + # exports can be downloaded using a machine with the allowed access, and then copied to the root of the GitLab Rails directory. + rsync rsync://example_username@gitlab.example.com/package_metadata_db $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + ``` + +### Using the Google Cloud Storage REST API to download the package metadata exports + +The package metadata exports can also be downloaded using the Google Cloud Storage API. The contents are available at [https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o](https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o). The following is an example of how this can be downloaded using [cURL](https://curl.se/) and [jq](https://stedolan.github.io/jq/). + +```shell +#!/bin/bash + +set -euo pipefail + +GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" +PKG_METADATA_DIR="$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db" +PKG_METADATA_MANIFEST_OUTPUT_FILE="/tmp/license_db_export_manifest.json" +PKG_METADATA_DOWNLOADS_OUTPUT_FILE="/tmp/license_db_object_links.tsv" + +# Download the contents of the bucket +curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" + +# Parse the links and names for the bucket objects and output them into a tsv file +jq -r '.items[] | [.name, .mediaLink] | @tsv' "$PKG_METADATA_MANIFEST_OUTPUT_FILE" > "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo -e "Saving package metadata exports to $PKG_METADATA_DIR\n" + +# Track how many objects will be downloaded +INDEX=1 +TOTAL_OBJECT_COUNT="$(wc -l $PKG_METADATA_DOWNLOADS_OUTPUT_FILE | awk '{print $1}')" + +# Download the objects +while IFS= read -r line; do + FILE="$(echo -n $line | awk '{print $1}')" + URL="$(echo -n $line | awk '{print $2}')" + OUTPUT_DIR="$(dirname $PKG_METADATA_DIR/$FILE)" + OUTPUT_PATH="$PKG_METADATA_DIR/$FILE" + + echo "Downloading $FILE" + + curl --progress-bar --create-dirs --output "$OUTPUT_PATH" --request "GET" "$URL" + + echo -e "$INDEX of $TOTAL_OBJECT_COUNT objects downloaded\n" + + let INDEX=(INDEX+1) +done < "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo "All objects saved to $PKG_METADATA_DIR" +``` + +### Automatic synchronization + +Your GitLab instance is synchronized [every hour](https://gitlab.com/gitlab-org/gitlab/-/blob/d4331343d26d6e2a81fadd8f7ecd72f7cb74d04d/config/initializers/1_settings.rb#L831-832) with the contents of the `package_metadata_db` directory. +To automatically update your local copy with the upstream changes, a cron job can be added to periodically download new exports. For example, the following crontabs can be added to setup a cron job that runs every 30 minutes. + +```plaintext +*/30 * * * * gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ +``` diff --git a/doc/topics/set_up_organization.md b/doc/topics/set_up_organization.md index 855b4962960..cf0a0ae4ab7 100644 --- a/doc/topics/set_up_organization.md +++ b/doc/topics/set_up_organization.md @@ -11,7 +11,7 @@ and give everyone access to the projects they need. - [Namespaces](../user/namespace/index.md) - [Members](../user/project/members/index.md) -- [Organization](../user/workspace/index.md) _(In development)_ +- [Organization](../user/organization/index.md) _(In development)_ - [Groups](../user/group/index.md) - [User account options](../user/profile/index.md) - [SSH keys](../user/ssh.md) diff --git a/doc/topics/your_work.md b/doc/topics/your_work.md index 862f9ae8430..268a6c4df5b 100644 --- a/doc/topics/your_work.md +++ b/doc/topics/your_work.md @@ -16,3 +16,8 @@ The **Your work** left sidebar provides access to your: - [Merge requests](../user/project/merge_requests/index.md) - [To-do List](../user/todos.md) - [Milestones](../user/project/milestones/index.md) +- [Snippets](../user/snippets.md#snippets) +- [Activity](../user/profile/index.md#view-your-activity) +- [Environments dashboard](../ci/environments/environments_dashboard.md) +- [Operations dashboard](../user/operations_dashboard/index.md) +- [Security center](../user/application_security/security_dashboard/index.md#security-center) diff --git a/doc/tutorials/convert_personal_namespace_into_group.md b/doc/tutorials/convert_personal_namespace_into_group.md new file mode 100644 index 00000000000..e272d854a35 --- /dev/null +++ b/doc/tutorials/convert_personal_namespace_into_group.md @@ -0,0 +1,95 @@ +--- +stage: none +group: Tutorials +info: For assistance with this tutorial, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-other-projects-and-subjects. +--- + +# Tutorial: Convert a personal namespace into a group **(FREE SAAS)** + +If you've started out on GitLab with a personal [namespace](../user/namespace/index.md), but now find +that you've outgrown its capabilities and its limitations hinder the collaboration on your projects, +you might want to switch to a group namespace instead. +A group namespace allows you to create multiple subgroups, and manage their members and permissions. + +You don't have to start from scratch - you can create a new group +and move your existing projects to the group to get the added benefits. +To find out how, see [Tutorial: Move your personal project to a group](move_personal_project_to_a_group.md). + +But you can go one step further and convert your personal namespace into a group namespace, +so you get to keep the existing username and URL. For example, if your username is `alex`, +you can continue using the `https://gitlab.example.com/alex` URL for your group. + +This tutorial shows you how to convert your personal namespace into a group namespace +using the following steps: + +1. [Create a group](#create-a-group). +1. [Transfer projects from the personal namespace to the group](#transfer-projects-from-the-personal-namespace-to-the-group). +1. [Rename the original username](#rename-the-original-username). +1. [Rename the new group namespace to the original username](#rename-the-new-group-namespace-to-the-original-username). + +For example, if your username for a personal namespace is `alex`, first create a group namespace named `alex-group`. +Then, move all projects from the `alex` to the `alex-group` namespace. Finally, +rename the `alex` namespace to `alex-user`, and `alex-group` namespace to the now available `alex` username. + +## Create a group + +1. On the top bar, select **Main menu > Groups > View all groups**. +1. On the right of the page, select **New group**. +1. In **Group name**, enter a name for the group. +1. In **Group URL**, enter a path for the group, which is used as the namespace. + Don't worry about the actual path, this is only temporary. You'll change this URL to the username of the personal namespace in the [final step](#rename-the-new-group-namespace-to-the-original-username). +1. Choose the [visibility level](../user/public_access.md). +1. Optional. Fill in information to personalize your experience. +1. Select **Create group**. + +## Transfer projects from the personal namespace to the group + +Next, you must transfer your projects from the personal namespace to the new group. +You can transfer only one project at a time, so if you want to transfer multiple projects, +you must perform the steps below for each project. + +Before you start the transfer process, make sure you: + +- Have the Owner role for the project. +- Remove [container images](../user/packages/container_registry/index.md#move-or-rename-container-registry-repositories). + You can't transfer a project that contains container images. +- Remove npm packages. You can't update the root namespace of a project that contains npm packages. + +To transfer a project to a group: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced**. +1. Under **Transfer project**, choose the group to transfer the project to. +1. Select **Transfer project**. +1. Enter the project's name and select **Confirm**. + +## Rename the original username + +Next, rename the original username of the personal namespace, so that the username becomes available for the new group namespace. +You can keep on using the personal namespace for other personal projects, or [delete that user account](../user/profile/account/delete_account.md) + +From the moment you rename the personal namespace, the username becomes available, so it's possible that someone else registers an account with it. To avoid this, you should [rename the new group](#rename-the-new-group-namespace-to-the-original-username) as soon as possible. + +To [change a user's username](../user/profile/index.md#change-your-username): + +1. On the top bar, in the top-right corner, select your avatar. +1. Select **Edit profile**. +1. On the left sidebar, select **Account**. +1. In the **Change username** section, enter a new username as the path. +1. Select **Update username**. + +## Rename the new group namespace to the original username + +Finally, rename the new group's URL to the username of the original personal namespace. + +To [change your group path](../user/group/manage.md#change-a-groups-path) (group URL): + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General page**. +1. Expand the **Advanced** section. +1. Under **Change group URL**, enter the user's original username. +1. Select **Change group URL**. + +That's it! You have now converted a personal namespace into a group, which opens up new possibilities of +working on projects and collaborating with more members. diff --git a/doc/tutorials/index.md b/doc/tutorials/index.md index c1b538bafbe..7a94583ae69 100644 --- a/doc/tutorials/index.md +++ b/doc/tutorials/index.md @@ -20,6 +20,7 @@ and running quickly. | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Use GitLab for DevOps](https://www.youtube.com/watch?v=7q9Y1Cv-ib0) (12m 34s) | Use GitLab through the entire DevOps lifecycle, from planning to monitoring. | **{star}** | | [Use Markdown at GitLab](../user/markdown.md) | GitLab Flavored Markdown (GLFM) is used in many areas of GitLab, for example, in merge requests. | **{star}** | | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Learn GitLab project walkthrough](https://www.youtube.com/watch?v=-oaI2WEKdI4&list=PL05JrBw4t0KofkHq4GZJ05FnNGa11PQ4d) (59m 2s) | Step through the tutorial-style issues in the **Learn GitLab** project. If you don't have this project, download [the export file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/project_templates/learn_gitlab_ultimate.tar.gz) and [import it to a new project](../user/project/settings/import_export.md#import-a-project-and-its-data). | | +| <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [GitLab Continuous Delivery overview](https://www.youtube.com/watch?v=g-gO93PMZvk&list=PLFGfElNsQthYDx0A_FaNNfUm9NHsK6zED&index=134) (17m 2s) | Learn how to use GitLab features to continuously build, test, and deploy iterative code changes. | | | [Productivity tips](https://about.gitlab.com/blog/2021/02/18/improve-your-gitlab-productivity-with-these-10-tips/) | Get tips to help make you a productive GitLab user. | | ## Use Git diff --git a/doc/update/background_migrations.md b/doc/update/background_migrations.md index a55d2af8dd4..47b786f2894 100644 --- a/doc/update/background_migrations.md +++ b/doc/update/background_migrations.md @@ -124,8 +124,10 @@ If you get this error, [check the batched background migration options](#databas ### Pause batched background migrations in GitLab 14.x -To pause an ongoing batched background migration, use the `disable` command above. -This command causes the migration to complete the current batch, and then wait to start the next batch. +To pause an ongoing batched background migration, +[disable the batched background migrations feature](#enable-or-disable-background-migrations). +Disabling the feature completes the current batch of migrations, then waits to start +the next batch until after the feature is enabled again. Use the following database queries to see the state of the current batched background migration: diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index ff2a018cd23..d7bbf254f2a 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -18,14 +18,14 @@ For deprecation authors (usually Product Managers and Engineering Managers): - To add a deprecation, use the example.yml file in `/data/deprecations/templates` as a template. - For more information about authoring deprecations, check the the deprecation item guidance: - https://about.gitlab.com/handbook/marketing/blog/release-posts/#creating-a-deprecation-entry + https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-and-removals-docs For deprecation reviewers (Technical Writers only): - To update the deprecation doc, run: `bin/rake gitlab:docs:compile_deprecations` - To verify the deprecations doc is up to date, run: `bin/rake gitlab:docs:check_deprecations` - For more information about updating the deprecation doc, see the deprecation doc update guidance: - https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-doc + https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-and-removals-docs --> {::options parse_block_html="true" /} @@ -37,17 +37,89 @@ Some features cause breaking changes when they are removed. **{rss}** **To be notified of upcoming breaking changes**, add this URL to your RSS feed reader: `https://about.gitlab.com/breaking-changes.xml` -DISCLAIMER: -This page contains information related to upcoming products, features, and functionality. -It is important to note that the information presented is for informational purposes only. -Please do not rely on this information for purchasing or planning purposes. -As with all projects, the items mentioned on this page are subject to change or delay. -The development, release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +You can also view [REST API](https://docs.gitlab.com/ee/api/rest/deprecations.html) +and [GraphQL](https://docs.gitlab.com/ee/api/graphql/removed_items.html) deprecations/removals. <div class="js-deprecation-filters"></div> <div class="announcement-milestone"> +## Announced in 15.10 + +<div class="deprecation removal-160 breaking-change"> + +### Deprecated Consul http metrics + +Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The Consul provided in the GitLab Omnibus package will no longer provide older deprecated Consul metrics starting in GitLab 16.0. + +In GitLab 14.0, [Consul was updated to 1.9.6](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5344), +which deprecated some telemetry metrics from being at the `consul.http` path. In GitLab 16.0, the `consul.http` path will be removed. + +If you have monitoring that consumes Consul metrics, update them to use `consul.api.http` instead of `consul.http`. +For more information, see [the deprecation notes for Consul 1.9.0](https://github.com/hashicorp/consul/releases/tag/v1.9.0). + +</div> + +<div class="deprecation removal-170 breaking-change"> + +### DingTalk OmniAuth provider + +Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The `omniauth-dingtalk` gem that provides GitLab with the DingTalk OmniAuth provider will be removed in our next +major release, GitLab 17.0. This gem sees very little use and is better suited for JiHu edition. + +</div> + +<div class="deprecation removal-160 breaking-change"> + +### Environment search query requires at least three characters + +Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +From GitLab 16.0, when you search for environments with the API, you must use at least three characters. This change helps us ensure the scalability of the search operation. + +</div> + +<div class="deprecation removal-160 breaking-change"> + +### Legacy Gitaly configuration method + +Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +Gitaly configuration within Omnibus GitLab has been updated such that all Gitaly related configuration keys are in a single +configuration structure that matches the standard Gitaly configuration. As such, the previous configuration structure is deprecated. + +The single configuration structure is available from GitLab 15.10, though backwards compatibility is maintained. Once removed, Gitaly must be configured using the single +configuration structure. You should update the configuration of Gitaly at your earliest convenience. + +The change improves consistency between Omnibus GitLab and source installs and enables us to provide better documentation and tooling for both. + +You should update to the new configuration structure as soon as possible using +[the upgrade instructions](https://docs.gitlab.com/ee/update/#gitaly-omnibus-gitlab-configuration-structure-change). + +</div> +</div> + +<div class="announcement-milestone"> + ## Announced in 15.9 <div class="deprecation removal-170 breaking-change"> @@ -112,6 +184,21 @@ In 16.0, this inbound scope limit will be the only option available for all proj <div class="deprecation removal-160 breaking-change"> +### Deprecation and planned removal for `CI_PRE_CLONE_SCRIPT` variable on GitLab SaaS + +End of Support: GitLab <span class="removal-milestone">16.0</span> <span class="support-end-date"></span><br /> +Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The [`CI_PRE_CLONE_SCRIPT` variable](https://docs.gitlab.com/ee/ci/runners/saas/linux_saas_runner.html#pre-clone-script) supported by GitLab SaaS Runners is deprecated as of GitLab 15.9 and will be removed in 16.0. The `CI_PRE_CLONE_SCRIPT` variable enables you to run commands in your CI/CD job prior to the runner executing Git init and get fetch. For more information about how this feature works, see [Pre-clone script](https://docs.gitlab.com/ee/ci/runners/saas/linux_saas_runner.html#pre-clone-script). As an alternative, you can use the [`pre_get_sources_script`](https://docs.gitlab.com/ee/ci/yaml/#hookspre_get_sources_script). + +</div> + +<div class="deprecation removal-160 breaking-change"> + ### Development dependencies reported for PHP and Python Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> @@ -141,7 +228,7 @@ We intend to replace this feature with the ability to [embed charts](https://git <div class="deprecation removal-160 breaking-change"> -### Error Tracking UI in GitLab Rails is deprecated +### Enforced validation of CI/CD parameter character lengths Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> @@ -149,7 +236,29 @@ WARNING: This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). Review the details carefully before upgrading. -The [Error Tracking UI](https://docs.gitlab.com/ee/operations/error_tracking.html) is deprecated in 15.9 and will be removed in 16.0. In future versions, you should use the [GitLab Observability UI](https://gitlab.com/gitlab-org/opstrace/opstrace-ui/), which will gradually be made available on GitLab.com over the next few releases. +While CI/CD [job names](https://docs.gitlab.com/ee/ci/jobs/index.html#job-name-limitations) have a strict 255 character limit, other CI/CD parameters do not yet have validations ensuring they also stay under the limit. + +In GitLab 16.0, validation will be added to strictly limit the following to 255 characters as well: + +- The `stage` keyword. +- The `ref`, which is the Git branch or tag name for the pipeline. +- The `description` and `target_url` parameter, used by external CI/CD integrations. + +Users on self-managed instances should update their pipelines to ensure they do not use parameters that exceed 255 characters. Users on GitLab.com do not need to make any changes, as these are already limited in that database. + +</div> + +<div class="deprecation removal-166 breaking-change"> + +### Error Tracking UI in GitLab Rails is deprecated + +Planned removal: GitLab <span class="removal-milestone">16.6</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The [Error Tracking UI](https://docs.gitlab.com/ee/operations/error_tracking.html) is deprecated in 15.9 and will be removed in 16.6 (milestone might change) once GitLab Observability UI is made available. In future versions, you should use the [GitLab Observability UI](https://gitlab.com/gitlab-org/opstrace/opstrace-ui/), which will gradually be made available on GitLab.com over the next few releases. During the transition to the GitLab Observability UI, we will migrate the [GitLab Observability Backend](https://gitlab.com/gitlab-org/opstrace/opstrace) from a per-cluster deployment model to a per-tenant deployment model. Because [Integrated Error Tracking](https://docs.gitlab.com/ee/operations/error_tracking.html#integrated-error-tracking) is in Open Beta, we will not migrate any existing user data. For more details about the migration, see the direction pages for: @@ -268,8 +377,8 @@ Previously, Praefect configuration keys were scattered throughout the configurat Praefect configuration so the previous configuration method is deprecated. The single configuration structure available from GitLab 15.9, though backwards compatibility is maintained. Once removed, Praefect must be configured using the single -configuration structure. You should update the configuration of Praefect at your earliest convenience. See -[GitLab 15.9 upgrade information](https://docs.gitlab.com/ee/update/#1590). +configuration structure. You should update your Praefect configuration as soon as possible using +[the upgrade instructions](https://docs.gitlab.com/ee/update/#praefect-omnibus-gitlab-configuration-structure-change). This change brings Praefect configuration in Omnibus GitLab in line with the configuration structure of Praefect. Previously, the hierarchies and configuration keys didn't match. The change improves consistency between Omnibus GitLab and source installs and enables us to provide better documentation and tooling for both. @@ -346,6 +455,20 @@ Due to low customer usage, Load Performance Testing is deprecated and will be re <div class="deprecation removal-160 breaking-change"> +### Managed Licenses API + +Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The [Managed Licenses API](https://docs.gitlab.com/ee/api/managed_licenses.html) is now deprecated and is scheduled for removal in GitLab 16.0. + +</div> + +<div class="deprecation removal-160 breaking-change"> + ### Old versions of JSON web tokens are deprecated Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> @@ -379,11 +502,11 @@ WARNING: This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). Review the details carefully before upgrading. -The project deletion protection setting in the Admin Area had an option to delete projects immediately. Starting with 16.0, this option will no longer be available, and delayed project deletion will become the default behavior. +The group and project deletion protection setting in the Admin Area had an option to delete groups and projects immediately. Starting with 16.0, this option will no longer be available, and delayed group and project deletion will become the default behavior. -The option will no longer appear as a group setting. Self-managed users will still have the option to define the deletion delay period, and SaaS users have a non-adjustable default retention period of 7 days. Users can still delete the project immediately from the project settings. +The option will no longer appear as a group setting. Self-managed users will still have the option to define the deletion delay period, and SaaS users have a non-adjustable default retention period of 7 days. Users can still immediately delete the project from the project settings, and the group from the group settings. -The option to delete projects immediately by default was deprecated to prevent users from accidentally taking this action and permanently losing projects. +The option to delete groups and projects immediately by default was deprecated to prevent users from accidentally taking this action and permanently losing groups and projects. </div> @@ -638,17 +761,17 @@ We will be transitioning to a new IID as a result of moving requirements to a [w </div> -<div class="deprecation removal-160 breaking-change"> +<div class="deprecation removal-170 breaking-change"> ### Trigger jobs can mirror downstream pipeline status exactly -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> +Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). Review the details carefully before upgrading. -In some cases, like when a downstream pipeline had the `passed with warnings` status, trigger jobs that were using [`strategy: depend`](https://docs.gitlab.com/ee/ci/yaml/index.html#strategydepend) did not mirror the status of the downstream pipeline exactly. In GitLab 16.0 trigger jobs will show the exact same status as the the downstream pipeline. If your pipeline relied on this behavior, you should update your pipeline to handle the more accurate status. +In some cases, like when a downstream pipeline had the `passed with warnings` status, trigger jobs that were using [`strategy: depend`](https://docs.gitlab.com/ee/ci/yaml/index.html#strategydepend) did not mirror the status of the downstream pipeline exactly. In GitLab 17.0 trigger jobs will show the exact same status as the the downstream pipeline. If your pipeline relied on this behavior, you should update your pipeline to handle the more accurate status. </div> </div> @@ -659,20 +782,6 @@ In some cases, like when a downstream pipeline had the `passed with warnings` st <div class="deprecation removal-160 breaking-change"> -### Approvers and Approver Group fields in Merge Request Approval API - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The endpoint to get the configuration of approvals for a project returns empty arrays for `approvers` and `approval_groups`. These fields were deprecated in favor of the endpoint to [get project-level rules](https://docs.gitlab.com/ee/api/merge_request_approvals.html#get-project-level-rules) for a merge request. API users are encouraged to switch to this endpoint instead. These fields will be removed from the `get configuration` endpoint in v5 of the GitLab REST API. - -</div> - -<div class="deprecation removal-160 breaking-change"> - ### Auto DevOps no longer provisions a PostgreSQL database by default Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> @@ -1253,25 +1362,11 @@ and will be moved to the JiHu GitLab codebase. </div> -<div class="deprecation removal-160 breaking-change"> - -### Single merge request changes API endpoint - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The endpoint to get [changes from a single merge request](https://docs.gitlab.com/ee/api/merge_requests.html#get-single-merge-request-changes) has been deprecated in favor the [list merge request diffs](https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-request-diffs) endpoint. API users are encouraged to switch to the new diffs endpoint instead. The `changes from a single merge request` endpoint will be removed in v5 of the GitLab REST API. - -</div> - <div class="deprecation removal-170 breaking-change"> ### Support for REST API endpoints that reset runner registration tokens -End of Support: GitLab <span class="removal-milestone">16.6</span> <span class="support-end-date"></span><br /> +End of Support: GitLab <span class="removal-milestone">17.0</span> <span class="support-end-date"></span><br /> Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: @@ -1286,8 +1381,9 @@ The deprecated endpoints are: - `POST /projects/:id/runners/reset_registration_token` - `POST /groups/:id/runners/reset_registration_token` -In GitLab 15.8, we plan to implement a new method to bind runners to a GitLab instance, +We plan to implement a new method to bind runners to a GitLab instance as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). This new architecture introduces a new method for registering runners and will eliminate the legacy [runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens). From GitLab 17.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods. @@ -1402,14 +1498,17 @@ From GitLab 13.6, users can [specify any runner configuration in the GitLab Runn ### GitLab Runner registration token in Runner Operator -End of Support: GitLab <span class="removal-milestone">16.6</span> <span class="support-end-date"></span><br /> +End of Support: GitLab <span class="removal-milestone">17.0</span> <span class="support-end-date"></span><br /> Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). Review the details carefully before upgrading. -The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and k8s Vanilla Operator to install a runner on Kubernetes is deprecated. GitLab plans to introduce a new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/) in GitLab 15.8, which introduces a new method for registering runners and eliminates the legacy runner registration token. +The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and k8s Vanilla Operator to install a runner on Kubernetes is deprecated. +We plan to implement a new method to bind runners to a GitLab instance +as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). </div> @@ -1417,7 +1516,7 @@ The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operato ### Registration tokens and server-side runner arguments in `POST /api/v4/runners` endpoint -End of Support: GitLab <span class="removal-milestone">16.6</span> <span class="support-end-date"></span><br /> +End of Support: GitLab <span class="removal-milestone">17.0</span> <span class="support-end-date"></span><br /> Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: @@ -1429,8 +1528,9 @@ This endpoint [registers](https://docs.gitlab.com/ee/api/runners.html#register-a with a GitLab instance at the instance, group, or project level through the API. We plan to remove the support for registration tokens and certain configuration arguments in this endpoint in GitLab 17.0. -In GitLab 15.10, we plan to implement a new method to bind runners to a GitLab instance, +We plan to implement a new method to bind runners to a GitLab instance as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). This new architecture introduces a new method for registering runners and will eliminate the legacy [runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens). From GitLab 17.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods. @@ -1441,7 +1541,7 @@ From GitLab 17.0 and later, the runner registration methods implemented by the n ### Registration tokens and server-side runner arguments in `gitlab-runner register` command -End of Support: GitLab <span class="removal-milestone">16.6</span> <span class="support-end-date"></span><br /> +End of Support: GitLab <span class="removal-milestone">17.0</span> <span class="support-end-date"></span><br /> Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: @@ -1449,9 +1549,9 @@ This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_g Review the details carefully before upgrading. The support for registration tokens and certain configuration arguments in the command to [register](https://docs.gitlab.com/runner/register/) a runner, `gitlab-runner register` is deprecated. -GitLab plans to introduce a new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/) in GitLab 15.10, -which introduces a new method for registering runners and eliminates the legacy -[runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens). +We plan to implement a new method to bind runners to a GitLab instance +as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). The new method will involve creating the runner in the GitLab UI and passing the [runner authentication token](https://docs.gitlab.com/ee/security/token_overview.html#runner-authentication-tokens-also-called-runner-tokens) to the `gitlab-runner register` command. @@ -1462,7 +1562,7 @@ to the `gitlab-runner register` command. ### `runnerRegistrationToken` parameter for GitLab Runner Helm Chart -End of Support: GitLab <span class="removal-milestone">16.6</span> <span class="support-end-date"></span><br /> +End of Support: GitLab <span class="removal-milestone">17.0</span> <span class="support-end-date"></span><br /> Planned removal: GitLab <span class="removal-milestone">17.0</span> <span class="removal-date"></span> WARNING: @@ -1471,28 +1571,13 @@ Review the details carefully before upgrading. The [`runnerRegistrationToken`](https://docs.gitlab.com/runner/install/kubernetes.html#required-configuration) parameter to use the GitLab Helm Chart to install a runner on Kubernetes is deprecated. -As part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/), in GitLab 15.8 we plan to introduce: - -- A new method to bind runners to a GitLab instance leveraging `runnerToken`. -- A unique system ID saved to the `config.toml`, which will ensure traceability between jobs and runners. +We plan to implement a new method to bind runners to a GitLab instance leveraging `runnerToken` +as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/). +The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633). From GitLab 17.0 and later, the methods to register runners introduced by the new GitLab Runner token architecture will be the only supported methods. </div> - -<div class="deprecation removal-160 breaking-change"> - -### merge_status API field - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The `merge_status` field in the [merge request API](https://docs.gitlab.com/ee/api/merge_requests.html#merge-status) has been deprecated in favor of the `detailed_merge_status` field which more correctly identifies all of the potential statuses that a merge request can be in. API users are encouraged to use the new `detailed_merge_status` field instead. The `merge_status` field will be removed in v5 of the GitLab REST API. - -</div> </div> <div class="announcement-milestone"> @@ -1819,21 +1904,6 @@ The [`project_fingerprint`](https://gitlab.com/groups/gitlab-org/-/epics/2791) a </div> -<div class="deprecation removal-160 breaking-change"> - -### REST API Runner maintainer_note - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The `maintainer_note` argument in the `POST /runners` REST endpoint was deprecated in GitLab 14.8 and replaced with the `maintenance_note` argument. -The `maintainer_note` argument will be removed in GitLab 16.0. - -</div> - <div class="deprecation removal-153"> ### Vulnerability Report sort by Tool @@ -2310,29 +2380,6 @@ To align with this change, API calls to list external status checks will also re </div> -<div class="deprecation removal-160 breaking-change"> - -### GraphQL API Runner will not accept `status` filter values of `active` or `paused` - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The GitLab Runner GraphQL endpoints will stop accepting `paused` or `active` as a status value in GitLab 16.0. - -A runner's status will only relate to runner contact status, such as: `online`, `offline`. -Status values `paused` or `active` will no longer be accepted and will be replaced by the `paused` query parameter. - -When checking for paused runners, API users are advised to specify `paused: true` as the query parameter. -When checking for active runners, specify `paused: false`. - -The REST API endpoints will follow in the same direction in a future REST v5 API, however the new `paused` -status value can be used in place of `active` since GitLab 14.8. - -</div> - <div class="deprecation removal-150 breaking-change"> ### GraphQL ID and GlobalID compatibility @@ -2490,37 +2537,6 @@ The `instanceStatisticsMeasurements` GraphQL node has been renamed to `usageTren </div> -<div class="deprecation removal-160 breaking-change"> - -### REST and GraphQL API Runner usage of `active` replaced by `paused` - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -Occurrences of the `active` identifier in the GitLab Runner GraphQL API endpoints will be -renamed to `paused` in GitLab 16.0. - -- For the GraphQL API, this change affects: - - the `CiRunner` property - - the `RunnerUpdateInput` input type for the `runnerUpdate` mutation - - the `runners` and `Group.runners` queries -- In v4 of the REST API, starting in GitLab 14.8, you can use the `paused` property in place of `active` -- In v5 of the REST API, this change will affect: - - endpoints taking or returning `active` property, such as: - - `GET /runners` - - `GET /runners/all` - - `GET /runners/:id` / `PUT /runners/:id` - - `PUT --form "active=false" /runners/:runner_id` - - `GET /projects/:id/runners` / `POST /projects/:id/runners` - - `GET /groups/:id/runners` - -The 16.0 release of GitLab Runner will start using the `paused` property when registering runners. - -</div> - <div class="deprecation removal-150 breaking-change"> ### Request profiling @@ -3160,20 +3176,6 @@ Currently, test coverage visualizations in GitLab only support Cobertura reports only supported report file in 15.0, but this is the first step towards GitLab supporting other report types. </div> - -<div class="deprecation removal-160 breaking-change"> - -### merged_by API field - -Planned removal: GitLab <span class="removal-milestone">16.0</span> <span class="removal-date"></span> - -WARNING: -This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). -Review the details carefully before upgrading. - -The `merged_by` field in the [merge request API](https://docs.gitlab.com/ee/api/merge_requests.html#list-merge-requests) has been deprecated in favor of the `merge_user` field which more correctly identifies who merged a merge request when performing actions (merge when pipeline succeeds, add to merge train) other than a simple merge. API users are encouraged to use the new `merge_user` field instead. The `merged_by` field will be removed in v5 of the GitLab REST API. - -</div> </div> <div class="announcement-milestone"> @@ -3682,3 +3684,11 @@ The OAuth implicit grant authorization flow will be removed in our next major re </div> </div> + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. diff --git a/doc/update/index.md b/doc/update/index.md index d08368663da..07bd153907a 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -107,11 +107,11 @@ To address the above two scenarios, it is advised to do the following prior to u as your GitLab version. Both versions [should be the same](https://docs.gitlab.com/runner/#gitlab-runner-versions). 1. Unpause your runners and unblock new jobs from starting by reverting the previous `/etc/gitlab/gitlab.rb` change. -## Checking for pending Advanced Search migrations **(PREMIUM SELF)** +## Checking for pending advanced search migrations **(PREMIUM SELF)** This section is only applicable if you have enabled the [Elasticsearch integration](../integration/advanced_search/elasticsearch.md) **(PREMIUM SELF)**. -Major releases require all [Advanced Search migrations](../integration/advanced_search/elasticsearch.md#advanced-search-migrations) +Major releases require all [advanced search migrations](../integration/advanced_search/elasticsearch.md#advanced-search-migrations) to be finished from the most recent minor release in your current version before the major version upgrade. You can find pending migrations by running the following command: @@ -129,16 +129,16 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations ``` -### What do you do if your Advanced Search migrations are stuck? +### What do you do if your advanced search migrations are stuck? -In GitLab 15.0, an Advanced Search migration named `DeleteOrphanedCommit` can be permanently stuck +In GitLab 15.0, an advanced search migration named `DeleteOrphanedCommit` can be permanently stuck in a pending state across upgrades. This issue [is corrected in GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89539). -If you are a self-managed customer who uses GitLab 15.0 with Advanced Search, you will experience performance degradation. +If you are a self-managed customer who uses GitLab 15.0 with advanced search, you will experience performance degradation. To clean up the migration, upgrade to 15.1 or later. -For other Advanced Search migrations stuck in pending, see [how to retry a halted migration](../integration/advanced_search/elasticsearch.md#retry-a-halted-migration). +For other advanced search migrations stuck in pending, see [how to retry a halted migration](../integration/advanced_search/elasticsearch.md#retry-a-halted-migration). ### What do you do for the error `Elasticsearch version not compatible` @@ -166,7 +166,7 @@ It's also important to ensure that any [background migrations have been fully co before upgrading to a new major version. If you have enabled the [Elasticsearch integration](../integration/advanced_search/elasticsearch.md) **(PREMIUM SELF)**, then -[ensure all Advanced Search migrations are completed](#checking-for-pending-advanced-search-migrations) in the last minor version in +[ensure all advanced search migrations are completed](#checking-for-pending-advanced-search-migrations) in the last minor version in your current version before proceeding with the major version upgrade. @@ -196,11 +196,11 @@ accordingly, while also consulting the NOTE: When not explicitly specified, upgrade GitLab to the latest available patch -release rather than the first patch release, for example `13.8.8` instead of `13.8.0`. -This includes versions you must stop at on the upgrade path as there may +release of the `major`.`minor` release rather than the first patch release, for example `13.8.8` instead of `13.8.0`. +This includes `major`.`minor` versions you must stop at on the upgrade path as there may be fixes for issues relating to the upgrade process. Specifically around a [major version](#upgrading-to-a-new-major-version), -crucial database schema and migration patches are included in the latest patch releases. +crucial database schema and migration patches may be included in the latest patch releases. ## Upgrading between editions @@ -237,7 +237,7 @@ possible. ## Version-specific upgrading instructions -Each month, major, minor, or patch releases of GitLab are published along with a +Each month, major or minor as well as possibly patch releases of GitLab are published along with a [release post](https://about.gitlab.com/releases/categories/releases/). You should read the release posts for all versions you're passing over. At the end of major and minor release posts, there are three sections to look for specifically: @@ -264,10 +264,31 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 15.10.0 + +- Gitaly configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.10 while backwards compatibility is + maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#gitaly-omnibus-gitlab-configuration-structure-change). + ### 15.9.0 -- This version removes `SanitizeConfidentialTodos` background migration [added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87908/diffs) in 15.6, which removed any user inaccessible to-do items. Make sure that this migration is finished before upgrading to 15.9. -- As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. Make sure that this migration is finished before upgrading to 15.9. +- **Upgrade to patch release 15.9.3 or later**. This provides fixes for two database migration bugs: + - Patch releases 15.9.0, 15.9.1, 15.9.2 have [a bug that can cause data loss](#user-profile-data-loss-bug-in-159x) from the user profile fields. + - The second [bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/394760) ensures it is possible to upgrade directly from 15.4.x. +- As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. +- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) is now enabled by default. + + The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its + metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk. + After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica. + + Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete + deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as + unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove + the repository using `praefect remove-repository` to remove the repository's remaining records. + + You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. [Read more about repository verification, and to see an example log entry](../administration/gitaly/praefect.md#repository-verification). +- Praefect configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.9 while backwards compatibility is + maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#praefect-omnibus-gitlab-configuration-structure-change). ### 15.8.2 @@ -364,6 +385,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap consistent in our documentation and product defaults. For example, previously: + - Omnibus GitLab default (`sidekiq['max_concurrency']`): 50 - From source installation default: 50 - Helm chart default (`gitlab.sidekiq.concurrency`): 25 @@ -476,7 +498,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### 15.5.3 -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: @@ -488,7 +510,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### 15.5.2 -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: @@ -500,7 +522,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### 15.5.1 -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: @@ -512,7 +534,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### 15.5.0 -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior. - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: @@ -564,7 +586,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap - GitLab 15.4.0 includes a [batched background migration](background_migrations.md#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318). This migration might take hours or days to complete on larger GitLab instances. - By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../administration/gitaly/praefect.md#customize-time-server-setting). -- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. +- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle. - The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior. - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`: @@ -705,6 +727,8 @@ A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) Gitaly. The previous implementation in GitLab Shell was removed in GitLab 15.0. With this change, global server hooks are stored only inside a subdirectory named after the hook type. Global server hooks can no longer be a single hook file in the root of the custom hooks directory. For example, you must use `<custom_hooks_dir>/<hook_name>.d/*` rather than `<custom_hooks_dir>/<hook_name>`. + - Use `gitaly['custom_hooks_dir']` in `gitlab.rb` ([introduced in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4208)) + for Omnibus GitLab. This replaces `gitlab_shell['custom_hooks_dir']`. - [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532). - The `FF_GITLAB_REGISTRY_HELPER_IMAGE` [feature flag](../administration/feature_flags.md#enable-or-disable-the-feature) is removed and helper images are always pulled from GitLab Registry. - The `AES256-GCM-SHA384` SSL cipher is no longer allowed by NGINX. @@ -804,13 +828,13 @@ A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) [background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) that may remain stuck permanently in a **pending** state. - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): + To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments) - end - ``` + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments) + end + ``` - If upgrading from a version earlier than 14.3.0, to avoid [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade @@ -883,11 +907,11 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments) - end - ``` + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments) + end + ``` - Upgrading to 14.5 (or later) [might encounter a one hour timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/354211) owing to a long running database data change. @@ -928,13 +952,13 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo [background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. - To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): + To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): - ```ruby - Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job| - puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments) - end - ``` + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments) + end + ``` ### 14.3.0 @@ -1425,6 +1449,330 @@ After upgraded to 11.11.8 you can safely upgrade to 12.0.Z. See our [documentation on upgrade paths](../policy/maintenance.md#upgrade-recommendations) for more information. +### User profile data loss bug in 15.9.x + +There is a database migration bug in 15.9.0, 15.9.1, and 15.9.2 that can cause data loss from the user profile fields `linkedin`, `twitter`, `skype`, `website_url`, `location`, and `organization`. + +This bug is fixed in patch releases 15.9.3 and later. + +The following upgrade path also works around the bug: + +1. Upgrade to GitLab 15.6.x, 15.7.x, or 15.8.x. +1. [Ensure batched background migrations](background_migrations.md#batched-background-migrations) are complete. +1. Upgrade to an earlier GitLab 15.9 patch release that doesn't have the bug fix. + +It is not then required to upgrade to 15.9.3 or higher for this issue. + +[Read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393216) for more information. + +### Gitaly: Omnibus GitLab configuration structure change + +Gitaly configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Gitaly configuration +structure used in source installs. + +As a result of this change, a single hash under `gitaly['configuration']` holds most Gitaly +configuration. Some `gitaly['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later: + +- `enable` +- `dir` +- `log_directory` +- `bin_path` +- `env_directory` +- `env` +- `open_files_ulimit` +- `consul_service_name` +- `consul_service_meta` + +Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.10. + +The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration: + +1. Replace the `...` with the value from the old key. +1. Skip any keys you haven't configured a value for previously. +1. Remove the old keys from the configuration once migrated. +1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. + + ```ruby +gitaly['configuration'] = { + # gitaly['socket_path'] + socket_path: ..., + # gitaly['runtime_dir'] + runtime_dir: ..., + # gitaly['listen_addr'] + listen_addr: ..., + # gitaly['prometheus_listen_addr'] + prometheus_listen_addr: ..., + # gitaly['tls_listen_addr'] + tls_listen_addr: ..., + tls: { + # gitaly['certificate_path'] + certificate_path: ..., + # gitaly['key_path'] + key_path: ..., + }, + # gitaly['graceful_restart_timeout'] + graceful_restart_timeout: ..., + logging: { + # gitaly['logging_level'] + level: ..., + # gitaly['logging_format'] + format: ..., + # gitaly['logging_sentry_dsn'] + sentry_dsn: ..., + # gitaly['logging_ruby_sentry_dsn'] + ruby_sentry_dsn: ..., + # gitaly['logging_sentry_environment'] + sentry_environment: ..., + # gitaly['log_directory'] + dir: ..., + }, + prometheus: { + # gitaly['prometheus_grpc_latency_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + grpc_latency_buckets: ..., + }, + auth: { + # gitaly['auth_token'] + token: ..., + # gitaly['auth_transitioning'] + transitioning: ..., + }, + git: { + # gitaly['git_catfile_cache_size'] + catfile_cache_size: ..., + # gitaly['git_bin_path'] + bin_path: ..., + # gitaly['use_bundled_git'] + use_bundled_binaries: ..., + # gitaly['gpg_signing_key_path'] + signing_key: ..., + # gitaly['gitconfig']. This is still an array but the type of the elements have changed. + config: [ + { + # Previously the elements contained 'section', and 'subsection' in addition to 'key'. Now + # these all should be concatenated into just 'key', separated by dots. For example, + # {section: 'first', subsection: 'middle', key: 'last', value: 'value'}, should become + # {key: 'first.middle.last', value: 'value'}. + key: ..., + value: ..., + }, + ], + }, + 'gitaly-ruby': { + # gitaly['ruby_max_rss'] + max_rss: ..., + # gitaly['ruby_graceful_restart_timeout'] + graceful_restart_timeout: ..., + # gitaly['ruby_restart_delay'] + restart_delay: ..., + # gitaly['ruby_num_workers'] + num_workers: ..., + }, + # gitaly['storage']. While the structure is the same, the string keys in the array elements + # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. + storage: ..., + hooks: { + # gitaly['custom_hooks_dir'] + custom_hooks_dir: ..., + }, + daily_maintenance: { + # gitaly['daily_maintenance_disabled'] + disabled: ..., + # gitaly['daily_maintenance_start_hour'] + start_hour: ..., + # gitaly['daily_maintenance_start_minute'] + start_minute: ..., + # gitaly['daily_maintenance_duration'] + duration: ..., + # gitaly['daily_maintenance_storages'] + storages: ..., + }, + cgroups: { + # gitaly['cgroups_mountpoint'] + mountpoint: ..., + # gitaly['cgroups_hierarchy_root'] + hierarchy_root: ..., + # gitaly['cgroups_memory_bytes'] + memory_bytes: ..., + # gitaly['cgroups_cpu_shares'] + cpu_shares: ..., + repositories: { + # gitaly['cgroups_repositories_count'] + count: ..., + # gitaly['cgroups_repositories_memory_bytes'] + memory_bytes: ..., + # gitaly['cgroups_repositories_cpu_shares'] + cpu_shares: ..., + } + }, + # gitaly['concurrency']. While the structure is the same, the string keys in the array elements + # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. + concurrency: ..., + # gitaly['rate_limiting']. While the structure is the same, the string keys in the array elements + # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}. + rate_limiting: ..., + pack_objects_cache: { + # gitaly['pack_objects_cache_enabled'] + enabled: ..., + # gitaly['pack_objects_cache_dir'] + dir: ..., + # gitaly['pack_objects_cache_max_age'] + max_age: ..., + } +} +``` + +### Praefect: Omnibus GitLab configuration structure change + +Praefect configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Praefect configuration +structure used in source installs. + +As a result of this change, a single hash under `praefect['configuration']` holds most Praefect +configuration. Some `praefect['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later: + +- `enable` +- `dir` +- `log_directory` +- `env_directory` +- `env` +- `wrapper_path` +- `auto_migrate` +- `consul_service_name` + +Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.9. + +The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration: + +1. Replace the `...` with the value from the old key. +1. Skip any keys you haven't configured a value for previously. +1. Remove the old keys from the configuration once migrated. +1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added. + +```ruby +praefect['configuration'] = { + # praefect['listen_addr'] + listen_addr: ..., + # praefect['socket_path'] + socket_path: ..., + # praefect['prometheus_listen_addr'] + prometheus_listen_addr: ..., + # praefect['tls_listen_addr'] + tls_listen_addr: ..., + # praefect['separate_database_metrics'] + prometheus_exclude_database_from_default_metrics: ..., + auth: { + # praefect['auth_token'] + token: ..., + # praefect['auth_transitioning'] + transitioning: ..., + }, + logging: { + # praefect['logging_format'] + format: ..., + # praefect['logging_level'] + level: ..., + }, + failover: { + # praefect['failover_enabled'] + enabled: ..., + }, + background_verification: { + # praefect['background_verification_delete_invalid_records'] + delete_invalid_records: ..., + # praefect['background_verification_verification_interval'] + verification_interval: ..., + }, + reconciliation: { + # praefect['reconciliation_scheduling_interval'] + scheduling_interval: ..., + # praefect['reconciliation_histogram_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + histogram_buckets: ..., + }, + tls: { + # praefect['certificate_path'] + certificate_path: ..., + # praefect['key_path'] + key_path: ..., + }, + database: { + # praefect['database_host'] + host: ..., + # praefect['database_port'] + port: ..., + # praefect['database_user'] + user: ..., + # praefect['database_password'] + password: ..., + # praefect['database_dbname'] + dbname: ..., + # praefect['database_sslmode'] + sslmode: ..., + # praefect['database_sslcert'] + sslcert: ..., + # praefect['database_sslkey'] + sslkey: ..., + # praefect['database_sslrootcert'] + sslrootcert: ..., + session_pooled: { + # praefect['database_direct_host'] + host: ..., + # praefect['database_direct_port'] + port: ..., + # praefect['database_direct_user'] + user: ..., + # praefect['database_direct_password'] + password: ..., + # praefect['database_direct_dbname'] + dbname: ..., + # praefect['database_direct_sslmode'] + sslmode: ..., + # praefect['database_direct_sslcert'] + sslcert: ..., + # praefect['database_direct_sslkey'] + sslkey: ..., + # praefect['database_direct_sslrootcert'] + sslrootcert: ..., + } + }, + sentry: { + # praefect['sentry_dsn'] + sentry_dsn: ..., + # praefect['sentry_environment'] + sentry_environment: ..., + }, + prometheus: { + # praefect['prometheus_grpc_latency_buckets']. The old value was configured as a string + # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2]. + grpc_latency_buckets: ..., + }, + # praefect['graceful_stop_timeout'] + graceful_stop_timeout: ..., + + # praefect['virtual_storages']. The old value was a hash map but the new value is an array. + virtual_storage: [ + { + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]. The name was previously the key in + # the 'virtual_storages' hash. + name: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. The old value was a hash map + # but the new value is an array. + node: [ + { + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. Use NODE_NAME key as the + # storage. + storage: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['address']. + address: ..., + # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['token']. + token: ..., + }, + ], + } + ] +} +``` + ### Change to Praefect-generated replica paths in GitLab 15.3 New Git repositories created in Gitaly cluster no longer use the `@hashed` storage path. @@ -1438,19 +1786,26 @@ and pass the `@hashed` storage path to `-relative-path`. With this information, you can correctly install [server hooks](../administration/server_hooks.md). -### Maintenance mode issue in GitLab 13.9 to 14.4 +### Geo: LFS transfers redirect to primary from secondary site mid-session in GitLab 15.1.0 to 15.3.2 -When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP. +LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. -Users who were signed in before Maintenance mode was enabled, continue to be signed in. If the administrator who enabled Maintenance mode loses their session, then they can't disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/index.md#disable-maintenance-mode). +This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later: -[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. +- LFS is enabled. +- LFS objects are being replicated across Geo sites. +- Repositories are being pulled by using a Geo secondary site. -### LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2 +### Geo: Incorrect object storage LFS file deletion on secondary sites in GitLab 15.0.0 to 15.3.2 -When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects. +[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) +can occur in GitLab 15.0.0 to 15.3.2 in the following situations: -[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3. +- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled. +- GitLab-managed replication to sync object storage is enabled and subsequently disabled. + +This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites +should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites. ### PostgreSQL segmentation fault issue @@ -1466,26 +1821,19 @@ by a database engine bug that causes a segmentation fault. Read more [in the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/364763). -### Geo: Incorrect object storage LFS file deletion on secondary sites in GitLab 15.0.0 to 15.3.2 - -[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) -can occur in GitLab 15.0.0 to 15.3.2 in the following situations: +### LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2 -- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled. -- GitLab-managed replication to sync object storage is enabled and subsequently disabled. +When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects. -This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites -should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites. +[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3. -### Geo: LFS transfers redirect to primary from secondary site mid-session in GitLab 15.1.0 to 15.3.2 +### Maintenance mode issue in GitLab 13.9 to 14.4 -LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. +When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP. -This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later: +Users who were signed in before Maintenance mode was enabled, continue to be signed in. If the administrator who enabled Maintenance mode loses their session, then they can't disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/index.md#disable-maintenance-mode). -- LFS is enabled. -- LFS objects are being replicated across Geo sites. -- Repositories are being pulled by using a Geo secondary site. +[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0 and backported into 14.4.3 and 14.3.5. ## Miscellaneous diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md deleted file mode 100644 index ad36a9ff534..00000000000 --- a/doc/update/mysql_to_postgresql.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -stage: Data Stores -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-28' -redirect_to: 'index.md' ---- - -# Migrating from MySQL to PostgreSQL (removed) **(FREE SELF)** - -Support for MySQL was removed in GitLab 12.1. diff --git a/doc/update/package/convert_to_ee.md b/doc/update/package/convert_to_ee.md index 7ebb860746b..0edb08b9268 100644 --- a/doc/update/package/convert_to_ee.md +++ b/doc/update/package/convert_to_ee.md @@ -116,7 +116,7 @@ The steps can be summed up to: sudo rm /etc/yum.repos.d/gitlab_gitlab-ce.repo ``` -1. Optional. [Set up the Elasticsearch integration](../../integration/advanced_search/elasticsearch.md) to enable [Advanced Search](../../user/search/advanced_search.md). +1. Optional. [Set up the Elasticsearch integration](../../integration/advanced_search/elasticsearch.md) to enable [advanced search](../../user/search/advanced_search.md). That's it! You can now use GitLab Enterprise Edition! To update to a newer version, follow [Update using the official repositories](index.md#upgrade-using-the-official-repositories). diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 34c7c096a8d..d3bd89975d2 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -169,7 +169,7 @@ To download and install GitLab: and architecture. Next to the filename is a label indicating the distribution, as the filenames may be the same. 1. Find the package version you wish to install, and select the filename from the list. -1. Select **Download** in the upper right corner to download the package. +1. In the upper-right corner, select **Download**. 1. After the package is downloaded, install it by using one of the following commands and replacing `<package_name>` with the package name you downloaded: diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index 5b4ecb96747..0d0084cf530 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -172,7 +172,7 @@ If you have Kubernetes clusters connected with GitLab, [upgrade your GitLab agen #### Elasticsearch -Before updating GitLab, confirm Advanced Search migrations are complete by +Before updating GitLab, confirm advanced search migrations are complete by [checking for pending advanced search migrations](background_migrations.md). After updating GitLab, you may have to upgrade diff --git a/doc/update/removals.md b/doc/update/removals.md index ae42ba53fd5..248d03cbbcb 100644 --- a/doc/update/removals.md +++ b/doc/update/removals.md @@ -34,6 +34,16 @@ For removal reviewers (Technical Writers only): https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-removals-doc --> +## Removed in 15.9 + +### Live Preview no longer available in the Web IDE + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The Live Preview feature of the Web IDE was intended to provide a client-side preview of static web applications. However, complex configuration steps and a narrow set of supported project types have limited its utility. With the introduction of the Web IDE Beta in GitLab 15.7, you can now connect to a full server-side runtime environment. With upcoming support for installing extensions in the Web IDE, we’ll also support more advanced workflows than those available with Live Preview. As of GitLab 15.9, Live Preview is no longer available in the Web IDE. + ## Removed in 15.8 ### CiliumNetworkPolicy within the auto deploy Helm chart is removed @@ -45,6 +55,18 @@ If you want to preserve this functionality, you can follow one of these two path 1. Fork the [GitLab Auto Deploy Helm chart](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) into the `chart/` path within your project 1. Set `AUTO_DEPLOY_IMAGE_VERSION` and `DAST_AUTO_DEPLOY_IMAGE_VERSION` to the most recent version of the image that included the CiliumNetworkPolicy +### `artifacts:public` CI/CD keyword refactored + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/deprecation_guidelines/). +Review the details carefully before upgrading. + +The [`artifacts:public` CI/CD keyword](https://docs.gitlab.com/ee/ci/yaml/#artifactspublic) was discovered to be not working properly since GitLab 15.8 and needed to be refactored. This feature is disabled on GitLab.com, and disabled by default on self-managed instances. If an administrator for an instance running GitLab 15.8 or 15.9 enabled this feature via the `non_public_artifacts` feature flag, it is likely that artifacts created with the `public:false` setting are being treated as `public:true`. + +If you have projects that use this setting, you should delete artifacts that must not be public, or [change the visibility](https://docs.gitlab.com/ee/user/public_access.html#change-project-visibility) of affected projects to private, before updating to GitLab 15.8 or later. + +In GitLab 15.10, this feature's code was refactored. On instances with this feature enabled, new artifacts created with `public:false` are now working as expected, though still disabled by default. Avoid testing this feature with production data until it is enabled by default and made generally available. + ## Removed in 15.7 ### File Type variable expansion in `.gitlab-ci.yml` diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md deleted file mode 100644 index 92b68410dca..00000000000 --- a/doc/update/restore_after_failure.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-28' -redirect_to: '../raketasks/backup_restore.md' ---- - -# Restoring from backup after a failed upgrade (removed) **(FREE SELF)** - -This content was removed in GitLab 15.7. -Use the [backup and restore](../raketasks/backup_restore.md) documentation instead. diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index b5ce0e74100..b6efaa7ec25 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -59,18 +59,15 @@ sudo service gitlab stop ### 3. Update Ruby -NOTE: -Beginning in GitLab 13.6, we only support Ruby 2.7 or higher, and dropped -support for Ruby 2.6. Be sure to upgrade if necessary. - +From GitLab 15.10, we only support Ruby 3.0 or higher and dropped support for Ruby 2.7. Be sure to upgrade if necessary. You can check which version you are running with `ruby -v`. Download Ruby and compile it: ```shell mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.6.tar.gz" -echo 'e7203b0cc09442ed2c08936d483f8ac140ec1c72e37bb5c401646b7866cb5d10 ruby-2.7.6.tar.gz' | sha256sum -c - && tar xzf ruby-2.7.6.tar.gz +curl --remote-name --location --progress-bar "https://cache.ruby-lang.org/pub/ruby/3.0/ruby-3.0.5.tar.gz" +echo '9afc6380a027a4fe1ae1a3e2eccb6b497b9c5ac0631c12ca56f9b7beb4848776 ruby-3.0.5.tar.gz' | sha256sum -c - && tar xzf ruby-3.0.5.tar.gz cd ruby-2.7.6 ./configure --disable-install-rdoc --enable-shared diff --git a/doc/update/upgrading_postgresql_using_slony.md b/doc/update/upgrading_postgresql_using_slony.md deleted file mode 100644 index 6d2abee3fc6..00000000000 --- a/doc/update/upgrading_postgresql_using_slony.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -stage: Data Stores -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -remove_date: '2023-02-28' -redirect_to: '../administration/postgresql/replication_and_failover.md' ---- - -# Upgrading PostgreSQL Using Slony (removed) **(FREE SELF)** - -This content was removed in GitLab 15.7. -Patroni has been used for database replication since GitLab 14.0. To perform upgrades, use the [Patroni replication documentation](../administration/postgresql/replication_and_failover.md) instead. diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index bb5cd195e5d..a85ae79a61c 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -465,6 +465,8 @@ Log in to your **primary** node, executing the following: sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-rake db:migrate ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the primary site to the secondary site if they're different. The file must be the same on all of a site’s nodes. + ### Update the Geo secondary site On each **secondary** node, executing the following: @@ -665,6 +667,8 @@ sudo gitlab-ctl hup puma sudo gitlab-ctl restart sidekiq ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the primary site to the secondary site if they're different. The file must be the same on all of a site’s nodes. + ### Step 3: Update each Geo secondary multi-node deployment Only proceed if you have successfully completed all steps on the Geo **primary** multi-node deployment. diff --git a/doc/user/admin_area/analytics/index.md b/doc/user/admin_area/analytics/index.md index 4304e612e4a..2ac8941b286 100644 --- a/doc/user/admin_area/analytics/index.md +++ b/doc/user/admin_area/analytics/index.md @@ -14,7 +14,7 @@ Instance-level analytics provide insights into the feature and data usage of you Prerequisite: -- You must have administrator access for your instance. +- You must have administrator access to the instance. To view instance-level analytics: diff --git a/doc/user/admin_area/appearance.md b/doc/user/admin_area/appearance.md index a1fae7e8712..b9850048e7d 100644 --- a/doc/user/admin_area/appearance.md +++ b/doc/user/admin_area/appearance.md @@ -71,6 +71,27 @@ to review the saved appearance settings: NOTE: You can add also add a [customized help message](settings/help_page.md) below the sign in message or add [a Sign in text message](settings/sign_in_restrictions.md#sign-in-information). +## Progressive Web App + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. + +GitLab can be installed as a [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) (PWA). +Use the Progressive Web App settings to customize its appearance, including its name, +description, and icon. + +### Configure the PWA settings + +To configure the PWA settings: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > Appearance**. +1. Scroll to the **Progressive Web App (PWA)** section. +1. Complete the fields. + - **Icon**: If you use the standard GitLab icon, it is available in sizes 192x192 pixels, + 512x512 pixels, also as a maskable icon. If you use a custom icon, it must be in either size + 192x192 pixels, or 512x512 pixels. +1. Select **Update appearance settings**. + ## New project pages You can add a new project guidelines message to the **New project page** in GitLab. diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index 847f687d051..78819def5de 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -18,7 +18,7 @@ is created, based on the user's access permissions: - Public projects can be selected by any authenticated user as a template for a new project, if all enabled [project features](../project/settings/index.md#configure-project-visibility-features-and-permissions) - except for **GitLab Pages** and **Security & Compliance** are set to **Everyone With Access**. + except for **GitLab Pages** and **Security and Compliance** are set to **Everyone With Access**. The same applies to internal projects. - Private projects can be selected only by users who are members of the projects. diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 0375232334f..7227da3ce0d 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -86,6 +86,16 @@ project, the following information is listed: Projects can be edited or deleted. +To edit a project's name or description: + +1. In the Projects overview, next to the project you want to edit, select **Edit**. +1. Edit the **Project name** or **Project description**. +1. Select **Save Changes**. + +To delete a project: + +1. In the Projects overview, next to the project you want to delete, select **Delete**. + The list of projects can be sorted by: - Updated date diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 5296a918f56..823f876539f 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -78,3 +78,5 @@ You may have connectivity issues due to the following reasons: - If the curl command returns a failure, either: - [Configure a proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html) in `gitlab.rb` to point to your server. - Contact your network administrator to make changes to the proxy. + - If an SSL inspection appliance is used, you must add the appliance's root CA certificate to `/etc/gitlab/trusted-certs` on the server, then run `gitlab-ctl reconfigure`. +
\ No newline at end of file diff --git a/doc/user/admin_area/license_file.md b/doc/user/admin_area/license_file.md index 29e43476819..69edb4551da 100644 --- a/doc/user/admin_area/license_file.md +++ b/doc/user/admin_area/license_file.md @@ -185,6 +185,13 @@ License.current.license_id License.current.data ``` +#### Interaction with licenses that start in the future + +```ruby +# Future license data follows the same format as current license data it just uses a different modifier for the License prefix +License.future_dated +``` + #### Check if a project feature is available on the instance Features listed in [`features.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/gitlab_subscriptions/features.rb). diff --git a/doc/user/admin_area/reporting/git_abuse_rate_limit.md b/doc/user/admin_area/reporting/git_abuse_rate_limit.md index 66d1173058e..1dd9497fed9 100644 --- a/doc/user/admin_area/reporting/git_abuse_rate_limit.md +++ b/doc/user/admin_area/reporting/git_abuse_rate_limit.md @@ -11,13 +11,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag`. On GitLab.com, this feature is available. -Git abuse rate limiting is a feature to automatically [ban users](../moderate_users.md#ban-and-unban-users) who download or clone more than a specified number of repositories in any project in the instance within a given time frame. Banned users cannot sign in to the instance and cannot access any non-public group via HTTP or SSH. +This is the administration documentation. For information about Git abuse rate limiting at the group level, see the [group-level documentation](../../group/reporting/git_abuse_rate_limit.md). -If the `git_abuse_rate_limit_feature_flag` feature flag is enabled, all application administrators receive an email when a user is about to be banned. +Git abuse rate limiting is a feature to automatically [ban users](../moderate_users.md#ban-and-unban-users) who download or clone more than a specified number of repositories in any project in the instance in a given time frame. Banned users cannot sign in to the instance and cannot access any non-public group via HTTP or SSH. The rate limit also applies to users who authenticate with a [personal](../../../user/profile/personal_access_tokens.md) or [group access token](../../../user/group/settings/group_access_tokens.md). -If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, administrators are still notified. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. +Git abuse rate limiting does not apply to instance administrators, [deploy tokens](../../../user/project/deploy_tokens/index.md), or [deploy keys](../../../user/project/deploy_keys/index.md). -If automatic banning is enabled, administrators receive an email when a user is about to be banned, and the user is automatically banned from the GitLab instance. +## Automatic ban notifications + +If the `git_abuse_rate_limit_feature_flag` feature flag is enabled, selected users receive an email when a user is about to be banned. + +If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, notifications are still sent. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. + +If automatic banning is enabled, an email notification is sent when a user is about to be banned, and the user is automatically banned from the GitLab instance. ## Configure Git abuse rate limiting @@ -28,6 +34,7 @@ If automatic banning is enabled, administrators receive an email when a user is 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400` (10 days). This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. 1. Optional. Exclude up to `100` users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Add up to `100` users to the **Send notifications to** field. You must select at least one user. All application administrators are selected by default. 1. Optional. Turn on the **Automatically ban users from this namespace when they exceed the specified limits** toggle to enable automatic banning. 1. Select **Save changes**. diff --git a/doc/user/admin_area/reporting/spamcheck.md b/doc/user/admin_area/reporting/spamcheck.md index 5c305eff4fa..16c144d2469 100644 --- a/doc/user/admin_area/reporting/spamcheck.md +++ b/doc/user/admin_area/reporting/spamcheck.md @@ -21,15 +21,15 @@ Spamcheck is only available for package-based installations: 1. Edit `/etc/gitlab/gitlab.rb` and enable Spamcheck: - ```ruby - spamcheck['enable'] = true - ``` + ```ruby + spamcheck['enable'] = true + ``` 1. Reconfigure GitLab: - ```shell - sudo gitlab-ctl reconfigure - ``` + ```shell + sudo gitlab-ctl reconfigure + ``` 1. Verify that the new services `spamcheck` and `spam-classifier` are up and running: diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 35a4c0aeea7..a23f500172e 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -174,6 +174,16 @@ wiki, packages, or snippets. The repository size limit applies to both private a For details on manually purging files, see [reducing the repository size using Git](../../project/repository/reducing_the_repo_size_using_git.md). +## Customize the default session duration + +You can change how long users can remain signed in. + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Account and limit**. The set duration is in **Session duration (minutes)**. + +For details, see [cookies used for sign-in](../../profile/index.md#cookies-used-for-sign-in). + ## Customize session duration for Git Operations when 2FA is enabled **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296669) in GitLab 13.9. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index aa171fe4536..8f7081a909d 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -54,7 +54,7 @@ To enable a project runner for more than one project: 1. On the top bar, select **Main menu > Admin**. 1. From the left sidebar, select **CI/CD > Runners**. 1. Select the runner you want to edit. -1. In the upper right, select **Edit** (**{pencil}**). +1. In the upper-right corner, select **Edit** (**{pencil}**). 1. Under **Restrict projects for this runner**, search for a project. 1. To the left of the project, select **Enable**. 1. Repeat this process for each additional project. @@ -344,7 +344,7 @@ To restrict all users in an instance from registering runners: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD**. -1. Expand **Runner registration**. +1. Expand **Runners**. 1. Clear the checkbox if you don't want to display runner registration information in the UI for group or project members. 1. Select **Save changes**. diff --git a/doc/user/admin_area/settings/deprecated_api_rate_limits.md b/doc/user/admin_area/settings/deprecated_api_rate_limits.md index 8bf0ffd21a5..13f8bc008e3 100644 --- a/doc/user/admin_area/settings/deprecated_api_rate_limits.md +++ b/doc/user/admin_area/settings/deprecated_api_rate_limits.md @@ -28,9 +28,9 @@ the general user and IP rate limits for requests to deprecated endpoints. You ca and IP rate limits already in place, and increase or decrease the rate limits for deprecated API endpoints. No other new features are provided by this override. -Prerequisites: +Prerequisite: -- You must have administrator access for your instance. +- You must have administrator access to the instance. To override the general user and IP rate limits for requests to deprecated API endpoints: diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 94d9ec73640..32a5b0a606f 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -43,9 +43,6 @@ using Omnibus, learn to install a custom CA in the Alternatively, learn where to install custom certificates by using `openssl version -d`. -When external authorization is enabled, [deploy tokens](../../project/deploy_tokens/index.md) - and [deploy keys](../../project/deploy_keys/index.md) can't be used for Git operations. - ## Configuration The external authorization service can be enabled by an administrator: @@ -56,6 +53,26 @@ The external authorization service can be enabled by an administrator: 1. Complete the fields. 1. Select **Save changes**. +### Allow external authorization with deploy tokens and deploy keys + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386656) in GitLab 15.9. + +You can set your instance to allow external authorization for Git operations with +[deploy tokens](../../project/deploy_tokens/index.md) or [deploy keys](../../project/deploy_keys/index.md). + +Prerequisites: + +- You must be using classification labels without a service URL for external authorization. + +To allow authorization with deploy tokens and keys: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand **External authorization**, and: + - Leave the service URL field empty. + - Select **Allow deploy tokens and deploy keys to be used with external authorization**. +1. Select **Save changes**. + ## How it works When GitLab requests access, it sends a JSON POST request to the external @@ -106,7 +123,7 @@ You can use your own classification label in the project's label" box. When no classification label is specified on a project, the default label defined in the [global settings](#configuration) is used. -The label is shown on all project pages in the upper right corner. +On all project pages, in the upper-right corner, the label appears.  diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md index ef9a3674c49..8677e3d86bf 100644 --- a/doc/user/admin_area/settings/files_api_rate_limits.md +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -26,7 +26,7 @@ for the Files API. No other new features are provided by this override. Prerequisite: -- You must have administrator access for your instance. +- You must have administrator access to the instance. To override the general user and IP rate limits for requests to the Repository files API: diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 5a550f15a41..5c4d7ee4be0 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -130,7 +130,7 @@ The **Network** settings contain: - [Search rate limits](../../../administration/instance_limits.md#search-rate-limit) - Configure global search request rate limits for authenticated and unauthenticated users. - [Deprecated API Rate Limits](deprecated_api_rate_limits.md) - Configure specific limits for deprecated API requests that supersede the user and IP rate limits. -- [Outbound requests](../../../security/webhooks.md) - Allow requests to the local network from hooks and services. +- [Outbound requests](../../../security/webhooks.md) - Allow requests to the local network from webhooks and integrations, or deny all outbound requests. - [Protected Paths](protected_paths.md) - Configure paths to be protected by Rack Attack. - [Incident Management Limits](../../../operations/incident_management/index.md) - Limit the number of inbound alerts that can be sent to a project. diff --git a/doc/user/admin_area/settings/rate_limit_on_projects_api.md b/doc/user/admin_area/settings/rate_limit_on_projects_api.md new file mode 100644 index 00000000000..beed083c484 --- /dev/null +++ b/doc/user/admin_area/settings/rate_limit_on_projects_api.md @@ -0,0 +1,33 @@ +--- +type: reference +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limit on Projects API **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112283) in GitLab 15.10 behind a feature flag, disabled by default. + +You can configure the rate limit per IP address for unauthenticated requests to the [list all projects API](../../../api/projects.md#list-all-projects). + +To change the rate limit: + +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Projects API rate limit**. +1. In the **Maximum requests per 10 minutes per IP address** text box, enter the new value. +1. Select **Save changes**. + +The rate limit: + +- Applies per IP address. +- Doesn't apply to authenticated requests. +- Can be set to 0 to disable rate limiting. + +The default value of the rate limit is `400`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 400, unauthenticated requests to the `GET /projects` API endpoint that +exceed a rate of 400 within 10 minutes are blocked. Access to the endpoint is restored after ten minutes have elapsed. diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index c44901b1ad7..3bf52bfe001 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -51,17 +51,26 @@ signing up using OmniAuth or LDAP, set `block_auto_created_users` to `true` in t [OmniAuth configuration](../../../integration/omniauth.md#configure-common-settings) or [LDAP configuration](../../../administration/auth/ldap/index.md#basic-configuration-settings). -## Require email confirmation +## Confirm user email + +> - Soft email confirmation [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47003) in GitLab 12.2 [with a flag](../../../operations/feature_flags.md) named `soft_email_confirmation`. +> - Soft email confirmation [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107302/diffs) from a feature flag to an application setting in GitLab 15.9. You can send confirmation emails during sign up and require that users confirm their email address before they are allowed to sign in. -To enforce confirmation of the email address used for new sign ups: +For example, to enforce confirmation of the email address used for new sign ups: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. Under **Email confirmation settings**, select **Hard**. +The following settings are available: + +- **Hard** - Send a confirmation email during sign up. New users must confirm their email address before they can log in. +- **Soft** - Send a confirmation email during sign up. New users can log in immediately, but must confirm their email in three days. Unconfirmed accounts are deleted. +- **Off** - New users can sign up without confirming their email address. + ## User cap > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4315) in GitLab 13.7. @@ -95,22 +104,6 @@ New user sign ups are subject to the user cap restriction. New users sign ups are not subject to the user cap restriction. Users in pending approval state are automatically approved in a background job. -## Soft email confirmation - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47003) in GitLab 12.2. -> - It's [deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-soft-email-confirmation). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -The soft email confirmation improves the sign-up experience for new users by allowing -them to sign in without an immediate confirmation when an email confirmation is required. -GitLab shows the user a reminder to confirm their email address, and the user can't -create or update pipelines until their email address is confirmed. - ## Minimum password length limit > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20661) in GitLab 12.6 @@ -171,25 +164,6 @@ semicolon, comma, or a new line.  -### Enable or disable soft email confirmation - -Soft email confirmation is under development but ready for production use. -It is deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:soft_email_confirmation) -``` - -To disable it: - -```ruby -Feature.disable(:soft_email_confirmation) -``` - ## Set up LDAP user filter You can limit GitLab access to a subset of the LDAP users on your LDAP server. diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index 4f6e727f673..6037b24a294 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index acff483e4f8..cd0b461c26b 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -207,9 +207,6 @@ To enable the export of You can enable migration of groups by direct transfer using the UI. -To also migrate projects with the groups, you must enable the -[`bulk_import_projects` feature flag](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). - To enable migration of groups by direct transfer: 1. Sign in to GitLab as a user with Administrator access level. diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index ef2d3a76990..4130ff6e036 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/analytics/dora_metrics.md b/doc/user/analytics/dora_metrics.md index 9ac949f05b4..b85ec60f00e 100644 --- a/doc/user/analytics/dora_metrics.md +++ b/doc/user/analytics/dora_metrics.md @@ -29,7 +29,7 @@ For software leaders, tracking velocity alongside quality metrics ensures they'r ## DORA Metrics dashboard in Value Stream Analytics -The four DORA metrics are available out-of-the-box in the [Value Stream Analytics (VSA) overview dashboard](../group/value_stream_analytics/index.md#view-dora-metrics-and-key-metrics-for-a-group). +The four DORA metrics are available out-of-the-box in the [Value Stream Analytics (VSA) overview dashboard](../group/value_stream_analytics/index.md#view-value-stream-analytics). This helps you visualize the engineering work in the context of end-to-end value delivery. The One DevOps Platform [Value Stream Management](https://gitlab.com/gitlab-org/gitlab/-/value_stream_analytics) provides end-to-end visibility to the entire software delivery lifecycle. @@ -37,76 +37,119 @@ This enables teams and managers to understand all aspects of productivity, quali ## Deployment frequency -Deployment frequency is the frequency of successful deployments to production (hourly, daily, weekly, monthly, or yearly). -This measures how often you deliver value to end users. A higher deployment frequency means you can -get feedback sooner and iterate faster to deliver improvements and features. GitLab measures this as the number of -deployments to a production environment in the given time period. +Deployment frequency is the frequency of successful deployments to production over the given date range (hourly, daily, weekly, monthly, or yearly). -Deployment frequency displays in several charts: +Software leaders can use the deployment frequency metric to understand how often the team successfully deploys software to production, and how quickly the teams can respond to customers' requests or new market opportunities. +High deployment frequency means you can get feedback sooner and iterate faster to deliver improvements and features. -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) +### How deployment frequency is calculated -To retrieve metrics for deployment frequency, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. +In GitLab, Deployment frequency is measured by the average number of deployments per day to a given environment, based on the deployment's end time (its `finished_at` property). +GitLab calculates the deployment frequency from the number of finished deployments on the given day. + +The calculation takes into account the production `environment tier` or the environments named `production/prod`. The environment must be part of the production deployment tier for its deployment information to appear on the graphs. + +### How to improve deployment frequency + +The first step is to benchmark the cadence of code releases between groups and projects. Next, you should consider: + +- Add more automated testing. +- Add more automated code validation. +- Break the changes down into smaller iterations. ## Lead time for changes -DORA Lead time for changes measures the time to successfully deliver a commit into production. -This metric reflects the efficiency of CI/CD pipelines. +Lead time for changes is the amount of time it takes a code change to get into production. + +"Lead time for changes" is not the same as "Lead time". In the value stream, "Lead time" measures the time it takes for work on an issue to move from the moment it's requested (Issue created) to the moment it's fulfilled and delivered (Issue closed). -In GitLab, Lead time for changes calculates the median time it takes for a merge request to get merged into production. -We measure **from** code committed **to** code successfully running in production, without adding the `coding_time` to the calculation. +For software leaders, Lead time for changes reflects the efficiency of CI/CD pipelines and visualizes how quickly work is delivered to customers. +Over time, the lead time for changes should decrease, while your team's performance should increase. Low lead time for changes means more efficient CI/CD pipelines. +In GitLab, Lead time for changes is measure by the `Median time it takes for a merge request to get merged into production (from master)`. -Over time, the lead time for changes should decrease, while your team's performance should increase. +### How lead time for changes is calculated -Lead time for changes displays in several charts: +GitLab calculates Lead time for changes base on the number of seconds to successfully deliver a commit into production - **from** code committed **to** code successfully running in production, without adding the `coding_time` to the calculation. -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) +### How to improve lead time for changes -To retrieve metrics for lead time for changes, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. +The first step is to benchmark the CI/CD pipelines' efficiency between groups and projects. Next, you should consider: -- The definition of lead time for change can vary widely, which often creates confusion within the industry. -- "Lead time for changes" is not the same as "Lead time". In the value stream, "Lead time" measures the time it takes for work on an issue to move from the moment it's requested (Issue created) to the moment it's fulfilled and delivered (Issue closed). +- Using Value Stream Analytics to identify bottlenecks in the processes. +- Breaking the changes down into smaller iterations. +- Adding more automation. ## Time to restore service -Time to restore service measures how long it takes an organization to recover from a failure in production. -GitLab measures this as the average time required to close the incidents -in the given time period. This assumes: +Time to restore service is the amount of time it takes an organization to recover from a failure in production. +For software leaders, Time to restore service reflects how long it takes an organization to recover from a failure in production. +Low Time to restore service means the organization can take risks with new innovative features to drive competitive advantages and increase business results. + +### How time to restore service is calculated + +In GitLab, Time to restore service is measured as the median time an incident was open for on a production environment. +GitLab calculates the number of seconds an incident was open on a production environment in the given time period. This assumes: + +- [GitLab incidents](../../operations/incident_management/incidents.md) are tracked. - All incidents are related to a production environment. -- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only -one production deployment, and any production deployment is related to no more than one incident). +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only one production deployment, and any production deployment is related to no more than one incident. -Time to restore service displays in several charts: +### How to improve time to restore service -- [Group-level value stream analytics](../group/value_stream_analytics/index.md) -- [Project-level value stream analytics](value_stream_analytics.md) -- [CI/CD analytics](ci_cd_analytics.md) +The first step is to benchmark the team response and recover from service interruptions and outages, between groups and projects. Next, you should consider: -To retrieve metrics for time to restore service, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. +- Improving the observability into the production environment. +- Improving response workflows. ## Change failure rate -Change failure rate measures the percentage of deployments that cause a failure in production. GitLab measures this as the number -of incidents divided by the number of deployments to a -production environment in the given time period. This assumes: +Change failure rate is how often a change cause failure in production. + +Software leaders can use the change failure rate metric to gain insights into the quality of the code being shipped. +High change failure rate may indicate an inefficient deployment process or insufficient automated testing coverage. +### How change failure rate is calculated + +In GitLab, Change failure rate is measured as the percentage of deployments that cause an incident in production in the given time period. +GitLab calculates this by the number of incidents divided by the number of deployments to a production environment. This assumes: + +- [GitLab incidents](../../operations/incident_management/incidents.md) are tracked. - All incidents are related to a production environment. -- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only -one production deployment, and any production deployment is related to no +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only one production deployment, and any production deployment is related to no more than one incident. -To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. +### How to improve change failure rate + +The first step is to benchmark the quality and stability, between groups and projects. + +To improve this metric, you should consider: + +- Finding the right balance between stability and throughput (Deployment frequency and Lead time for changes), and not sacrificing quality for speed. +- Improving the efficacy of code review processes. +- Adding more automated testing. + +## DORA metrics in GitLab + +The DORA metrics are displayed on the following charts: + +- [Value Streams Dashboard](value_streams_dashboard.md), which helps you identify trends, patterns, and opportunities for improvement. +- [CI/CD analytics charts](ci_cd_analytics.md), which show pipeline success rates and duration, and the history of DORA metrics over time. +- Value stream analytics for [groups](../group/value_stream_analytics/index.md) and [projects](value_stream_analytics.md), which provide metrics about each stage of your software development process +- Insights reports for [groups](../group/insights/index.md) and [projects](value_stream_analytics.md), where you can also use [DORA query parameters](../../user/project/insights/index.md#dora-query-parameters) to create custom charts. + +The table below provides an overview of the DORA metrics' data aggregation in different charts. -### Insights: Custom DORA reporting +| Metric name | Measured values | Data aggregation in the [Value Streams Dashboard](value_streams_dashboard.md) | Data aggregation in CI/CD analytics charts | Data aggregation in Value stream analytics | Data aggregation in Custom insights reporting | +|---------------------------|-------------------|-----------------------------------------------------|------------------------|-----------------------|----------| +| Deployment frequency | Number of successful deployments | daily average per month | daily average | daily average | `day` (default) or `month` | +| Lead time for changes | Number of seconds to successfully deliver a commit into production | daily median per month | median time | median time | `day` (default) or `month` | +| Time to restore service | Number of seconds an incident was open for | daily median per month | daily median | median time | `day` (default) or `month` | +| Change failure rate | percentage of deployments that cause an incident in production | daily median per month | percentage of failed deployments | percentage of failed deployments | `day` (default) or `month` | -Custom charts to visualize DORA data with [Insights YAML-based reports](../../user/project/insights/index.md#dora-query-parameters). +## Retrieve DORA metrics data -With this new visualization, software leaders can track metrics improvements, understand patterns in their metrics trends, and compare performance between groups and projects. +To retrieve DORA data, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. ### Measure DORA metrics without using GitLab CI/CD pipelines diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 093266e8aee..f22a6a483b8 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -198,8 +198,8 @@ example, milestones have been created and CI for testing and setting environment Value stream analytics records the following times for each stage: - **Issue**: 09:00 to 11:00: 2 hrs -- **Plan**: 11:00 to 12:00: 1 hr -- **Code**: 12:00 to 14:00: 2 hrs +- **Plan**: 11:00 to 12:30: 1.5 hr +- **Code**: 12:30 to 14:00: 1.5 hrs - **Test**: 5 minutes - **Review**: 14:00 to 19:00: 5 hrs - **Staging**: 19:00 to 19:30: 30 minutes diff --git a/doc/user/analytics/value_streams_dashboard.md b/doc/user/analytics/value_streams_dashboard.md index f8cfb1bf06b..fd4cdd0d65f 100644 --- a/doc/user/analytics/value_streams_dashboard.md +++ b/doc/user/analytics/value_streams_dashboard.md @@ -4,20 +4,22 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Value Streams Dashboard **(PREMIUM)** +# Value Streams Dashboard **(ULTIMATE)** -> Introduced in GitLab 15.8 as a Closed [Beta](../../policy/alpha-beta-support.md#beta-features) feature. +> Introduced in GitLab 15.8 as a Closed [Beta](../../policy/alpha-beta-support.md#beta-features) feature [with a flag](../../administration/feature_flags.md) named `group_analytics_dashboards_page`. Disabled by default. -You can leave feedback on dashboard bugs or functionality in [issue 381787](https://gitlab.com/gitlab-org/gitlab/-/issues/381787). +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `group_analytics_dashboards_page`. +On GitLab.com, this feature is not available. This feature is not ready for production use. -This feature is not ready for production use. +You can leave feedback on dashboard bugs or functionality in [issue 381787](https://gitlab.com/gitlab-org/gitlab/-/issues/381787). -The Value Streams Dashboard is a customizable dashboard to enable decision-makers to identify trends, patterns, and opportunities for digital transformation improvements. +The Value Streams Dashboard is a customizable dashboard that enables decision-makers to identify trends, patterns, and opportunities for digital transformation improvements. This page is a work in progress, and we're updating the information as we add more features. For more information, see the [Value Stream Management category direction page](https://about.gitlab.com/direction/plan/value_stream_management/). -After the feature flag is enabled, to open the new page, append this path `/analytics/dashboards` to the group URL -(for example, `https://gitlab.com/groups/gitlab-org/-/analytics/dashboards`). +After the feature flag is enabled, to open the new page, append this path `/analytics/dashboards/value_streams_dashboard` to the group URL +(for example, `https://gitlab.com/groups/gitlab-org/-/analytics/dashboards/value_streams_dashboard`). ## Initial use case @@ -27,7 +29,7 @@ This comparison can help decision-makers understand whether projects and groups The beta version of the Value Streams Dashboard includes the following metrics: - [DORA metrics](dora_metrics.md) -- [Value Stream Analytics (VSA) - flow metrics](value_stream_analytics.md) +- [Value Stream Analytics (VSA) - flow metrics](../group/value_stream_analytics/index.md) The Value Streams Dashboard allows you to: @@ -49,7 +51,7 @@ that are the largest value contributors, overperforming, or underperforming. You can also drill down the metrics for further analysis. When you hover over a metric, a tooltip displays an explanation of the metric and a link to the related documentation page. -## Customize the dashboard widgets +## Customize the dashboard panels You can customize the Value Streams Dashboard and configure what subgroups and projects to include in the page. @@ -57,7 +59,7 @@ A view can display maximum four subgroups or projects. To display multiple subgroups and projects, specify their path as a URL parameter. -For example, the parameter `query=gitlab-org/gitlab-foss,gitlab-org/gitlab,gitlab-org/gitlab-design,gitlab-org/gitlab-docs` displays three separate widgets, one each for the: +For example, the parameter `query=gitlab-org/gitlab-foss,gitlab-org/gitlab,gitlab-org/gitlab-design,gitlab-org/gitlab-docs` displays three separate panels, one each for the: - `gitlab-org` group - `gitlab-ui` project diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index b55c5a1b299..8ae6d8ee675 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -50,6 +50,7 @@ Example projects using these methods are available: - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/postman-api-fuzzing-example) - [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/graphql-api-fuzzing-example) - [Example SOAP project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/soap-api-fuzzing-example) +- [Authentication Token using Selenium](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/auth-token-selenium) ## Enable Web API fuzzing @@ -98,7 +99,7 @@ a YAML snippet that you can paste in your GitLab CI/CD configuration. To generate an API Fuzzing configuration snippet: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **API Fuzzing** row, select **Enable API Fuzzing**. 1. Complete the fields. For details see [Available CI/CD variables](#available-cicd-variables). 1. Select **Generate code snippet**. @@ -330,6 +331,8 @@ This example is a minimal configuration for API Fuzzing. From here you can: #### API Fuzzing with a GraphQL Schema file +API Fuzzing can use a GraphQL schema file to understand and test a GraphQL endpoint that has introspection disabled. To use a GraphQL schema file, it must be in the introspection JSON format. A GraphQL schema can be converted to a the introspection JSON format using an online 3rd party tool: [https://transform.tools/graphql-to-introspection-json](https://transform.tools/graphql-to-introspection-json). + To configure API Fuzzing to use a GraphQl schema file that provides information about the target API to test: 1. [Include](../../../ci/yaml/index.md#includetemplate) @@ -1994,7 +1997,7 @@ When configured correctly, a CI/CD pipeline contains a `fuzz` stage and an `apif typical operation, the job always succeeds even if faults are identified during fuzz testing. Faults are displayed on the **Security** pipeline tab with the suite name. When testing against the -repositories default branch, the fuzzing faults are also shown on the Security & Compliance's +repositories default branch, the fuzzing faults are also shown on the Security and Compliance's Vulnerability Report page. To prevent an excessive number of reported faults, the API fuzzing scanner limits the number of @@ -2026,7 +2029,7 @@ Follow these steps to view details of a fuzzing fault: 1. You can view faults in a project, or a merge request: - - In a project, go to the project's **Security & Compliance > Vulnerability Report** + - In a project, go to the project's **Security and Compliance > Vulnerability Report** page. This page shows all vulnerabilities from the default branch only. - In a merge request, go the merge request's **Security** section and select the **Expand** button. API Fuzzing faults are available in a section labeled @@ -2702,7 +2705,7 @@ You can expedite the investigation with the [testssl.sh tool](https://testssl.sh To get support for your particular problem use the [getting help channels](https://about.gitlab.com/get-help/). The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and API Fuzzing. -Use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. +Use `~"Category:API Security"` [label](../../../development/labels/index.md) when opening a new issue regarding API fuzzing to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. [Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. diff --git a/doc/user/application_security/api_security/api_discovery/index.md b/doc/user/application_security/api_security/api_discovery/index.md new file mode 100644 index 00000000000..b77bed74c70 --- /dev/null +++ b/doc/user/application_security/api_security/api_discovery/index.md @@ -0,0 +1,169 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# API Discovery **(ULTIMATE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9302) in GitLab 15.9. The API Discovery feature is in [Beta](../../../../policy/alpha-beta-support.md). + +API Discovery analyzes your application and produces an OpenAPI document describing the web APIs it exposes. This schema document can then be used by [DAST API](../../dast_api/index.md) or [API Fuzzing](../../api_fuzzing/index.md) to perform security scans of the web API. + +## Supported frameworks + +- [Java Spring-Boot](#java-spring-boot) + +## When does API Discovery run? + +API Discovery runs as a standalone job in your pipeline. The resulting OpenAPI document is captured as a job artifact so it can be used by other jobs in later stages. + +API Discovery runs in the `test` stage by default. The `test` stage was chosen as it typically executes before the stages used by other API Security features such as DAST API and API Fuzzing. + +## Example API Discovery configurations + +The following projects demonstrate API Discovery: + +- [Example Java Spring Boot v2 Pet Store](https://gitlab.com/gitlab-org/security-products/demos/api-discovery/java-spring-boot-v2-petstore) + +## Java Spring-Boot + +[Spring Boot](https://spring.io/projects/spring-boot) is a popular framework for creating stand-alone, production-grade Spring-based applications. + +### Supported Applications + +- Spring Boot: v2.X (>= 2.1) +- Java: 11, 17 (LTS versions) +- Executable JARs + +API Discovery supports Spring Boot major version 2, minor versions 1 and higher. Versions 2.0.X are not supported due to known bugs which affect API Discovery and were fixed in 2.1. + +Major version 3 is planned to be supported in the future. Support for major version 1 is not planned. + +API Discovery is tested with and officially supports LTS versions of the Java runtime. Other versions may work also, and bug reports from non-LTS versions are welcome. + +Only applications that are built as Spring Boot [executable JARs](https://docs.spring.io/spring-boot/docs/current/reference/html/executable-jar.html#appendix.executable-jar.nested-jars.jar-structure) are supported. + +### Configure as pipeline job + +The easiest way to run API Discovery is through a pipeline job based on our CI template. +When running in this method, you provide a container image that has the required dependencies installed (such as an appropriate Java runtime). See [Image Requirements](#image-requirements) for more information. + +1. A container image that meets the [image requirements](#image-requirements) is uploaded to a container registry. If the container registry requires authentication see [this help section](/ee/ci/docker/using_docker_images.md#access-an-image-from-a-private-container-registry). +1. In a job in the `build` stage, build your application and configure the resulting Spring Boot executable JAR as a job artifact. +1. Include the API Discovery template in your `.gitlab-ci.yml` file. + + ```yaml + include: + - template: Security/API-Discovery.gitlab-ci.yml + ``` + + Only a single `include` statement is allowed per `.gitlab-ci.yml` file. If you are including other files, combine them into a single `include` statement. + + ```yaml + include: + - template: Security/API-Discovery.gitlab-ci.yml + - template: Security/DAST-API.gitlab-ci.yml + ``` + +1. Create a new job that extends from `.api_discovery_java_spring_boot`. The default stage is `test` which can be optionally changed to any value. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + ``` + +1. Configure the `image` for the job. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + image: openjdk:11-jre-slim + ``` + +1. Provide the Java classpath needed by your application. This includes your compatible build artifact from step 2, along with any additional dependencies. For this example, the build artifact is `build/libs/spring-boot-app-0.0.0.jar` and contains all needed dependencies. The variable `API_DISCOVERY_JAVA_CLASSPATH` is used to provide the classpath. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + image: openjdk:11-jre-slim + variables: + API_DISCOVERY_JAVA_CLASSPATH: build/libs/spring-boot-app-0.0.0.jar + ``` + +1. [Optional] If the image provided is missing a dependency needed by API Discovery, it can be added using a `before_script`. In this example, the `openjdk:11-jre-slim` container doesn't include `curl` which is required by API Discovery. The dependency can be installed using the Debian package manager `apt` as shown below. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + image: openjdk:11-jre-slim + variables: + API_DISCOVERY_JAVA_CLASSPATH: build/libs/spring-boot-app-0.0.0.jar + before_script: + - apt-get update && apt-get install -y curl + ``` + +1. [Optional] If the image provided doesn't automatically set the `JAVA_HOME` environment variable, or include `java` in the path, the `API_DISCOVERY_JAVA_HOME` variable can be used. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + image: openjdk:11-jre-slim + variables: + API_DISCOVERY_JAVA_CLASSPATH: build/libs/spring-boot-app-0.0.0.jar + API_DISCOVERY_JAVA_HOME: /opt/java + ``` + +1. [Optional] If the package registry at `API_DISCOVERY_PACKAGES` is not public, provide a token that has read access to the GitLab API and registry using the `API_DISCOVERY_PACKAGE_TOKEN` variable. This is not required if you are using `gitlab.com` and have not customized the `API_DISCOVERY_PACKAGES` variable. This example uses a [custom CI/CD variable](../../../../ci/variables/index.md#define-a-cicd-variable-in-the-ui) named `GITLAB_READ_TOKEN` to store the token. + + ```yaml + api_discovery: + extends: .api_discovery_java_spring_boot + image: openjdk:8-jre-alpine + variables: + API_DISCOVERY_JAVA_CLASSPATH: build/libs/spring-boot-app-0.0.0.jar + API_DISCOVERY_PACKAGE_TOKEN: $GITLAB_READ_TOKEN + ``` + +After the API Discovery job has successfully run, the OpenAPI document is available as a job artifact called `gl-api-discovery-openapi.json`. + +#### Image requirements + +- Linux container image. +- Java versions 11 or 17 are officially supported, but other versions are likely compatible as well. +- The `curl` command. +- A shell at `/bin/sh` (like busybox sh or bash). + +### Available CI/CD variables + +| CI/CD variable | Description | +|---------------------------------------------|--------------------| +| `API_DISCOVERY_DISABLED` | Disables the API Discovery job when using template job rules. | +| `API_DISCOVERY_DISABLED_FOR_DEFAULT_BRANCH` | Disables the API Discovery job for default branch pipelines when using template job rules. | +| `API_DISCOVERY_JAVA_CLASSPATH` | Java class-path that includes target Spring Boot application. (`build/libs/sample-0.0.0.jar`) | +| `API_DISCOVERY_JAVA_HOME` | If provided is used to set `JAVA_HOME`. | +| `API_DISCOVERY_PACKAGES` | GitLab Project Package API Prefix (defaults to `$CI_API_V4_URL/projects/42503323/packages`). | +| `API_DISCOVERY_PACKAGE_TOKEN` | GitLab token for calling the GitLab package API. Only needed when `API_DISCOVERY_PACKAGES` is set to a non-public project. | +| `API_DISCOVERY_VERSION` | API Discovery version to use (defaults to `1`). Can be used to pin a version by providing the full version number `1.1.0`. | + +## Get support or request an improvement + +To get support for your particular problem, use the [getting help channels](https://about.gitlab.com/get-help/). + +The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Discovery. +Use `~"Category:API Security"` [label](../../../../development/labels/index.md) when opening a new issue regarding API Discovery to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. + +[Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. + +When experiencing a behavior not working as expected, consider providing contextual information: + +- GitLab version if using a self-managed instance. +- `.gitlab-ci.yml` job definition. +- Full job console output. +- Framework in use with version (for example Spring Boot v2.3.2). +- Language runtime with version (for example OpenJDK v17.0.1). +<!-- - Scanner log file is available as a job artifact named `gl-api-discovery.log`. --> + +WARNING: +**Sanitize data attached to a support issue**. Remove sensitive information, including: credentials, passwords, tokens, keys, and secrets. diff --git a/doc/user/application_security/api_security/index.md b/doc/user/application_security/api_security/index.md new file mode 100644 index 00000000000..5c2e74bceae --- /dev/null +++ b/doc/user/application_security/api_security/index.md @@ -0,0 +1,21 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# API Security **(ULTIMATE)** + +API Security refers to the measures taken to secure and protect web Application Programming Interfaces (APIs) from unauthorized access, misuse, and attacks. +APIs are a crucial component of modern application development as they allow applications to interact with each other and exchange data. +However, this also makes them attractive to attackers and vulnerable to security threats if not properly secured. +In this section, we discuss GitLab features that can be used to ensure the security of web APIs in your application. +Some of the features discussed are specific to web APIs and others are more general solutions that are also used with web API applications. + +- [SAST](../sast) identified vulnerabilities by analyzing the application's codebase. +- [Dependency Scanning](../dependency_scanning) reviews a project 3rd party dependencies for known vulnerabilities (for example CVEs). +- [Container Scanning](../container_scanning) analyzes container images to identify known OS package vulnerabilities and installed language dependencies. +- [API Discovery](api_discovery) examines an application containing a REST API and intuits an OpenAPI specification for that API. OpenAPI specification documents are used by other GitLab security tools. +- [DAST API](../dast_api) performs dynamic analysis security testing of web APIs. It can identify various security vulnerabilities in your application, including the OWASP Top 10. +- [API Fuzzing](../api_fuzzing) performs fuzz testing of a web API. Fuzz testing looks for issues in an application that are not previously known and don't map to classic vulnerability types such as SQL Injection. diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index ea422f0b33c..380624c91ce 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -5,7 +5,7 @@ group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Security Configuration **(FREE)** +# Security configuration **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20711) in GitLab 12.6. > - SAST configuration was [enabled](https://gitlab.com/groups/gitlab-org/-/epics/3659) in 13.3 and [improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in 13.4. @@ -13,7 +13,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - A simplified version was made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/294076) in GitLab 13.10. > - [Redesigned](https://gitlab.com/gitlab-org/gitlab/-/issues/326926) in 14.2. -The Security Configuration page lists the following for the security testing and compliance tools: +The **Security configuration** page lists the following for the security testing and compliance tools: - Name, description, and a documentation link. - Whether or not it is available. @@ -41,7 +41,7 @@ all security features are configured by default. To view a project's security configuration: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. Select **Configuration history** to see the `.gitlab-ci.yml` file's history. diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 0a586a14cc4..a38f7bcb77c 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -15,6 +15,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86092) the major analyzer version from `4` to `5` in GitLab 15.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86783) from GitLab Ultimate to GitLab Free in 15.0. > - Container Scanning variables that reference Docker [renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/357264) in GitLab 15.4. +> - Container Scanning template [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/381665) from `Security/Container-Scanning.gitlab-ci.yml` to `Jobs/Container-Scanning.gitlab-ci.yml` in GitLab 15.6. Your application's Docker image may itself be based on Docker images that contain known vulnerabilities. By including an extra Container Scanning job in your pipeline that scans for those @@ -90,12 +91,12 @@ To enable container scanning in your pipeline, you need the following: ## Configuration To enable container scanning, add the -[`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Container-Scanning.gitlab-ci.yml) +[`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) to your `.gitlab-ci.yml` file: ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml ``` The included template: @@ -117,7 +118,7 @@ registry, and scans the image: ```yaml include: - template: Jobs/Build.gitlab-ci.yml - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -142,7 +143,7 @@ enables verbose output for the analyzer: ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml variables: SECURE_LOG_LEVEL: 'debug' @@ -154,7 +155,7 @@ To scan images located in a registry other than the project's, use the following ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -178,7 +179,9 @@ container_scanning: - export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml + +variables: CS_IMAGE: <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<image>:<tag> CS_REGISTRY_USER: AWS CS_REGISTRY_PASSWORD: "$AWS_ECR_PASSWORD" @@ -199,7 +202,7 @@ For example: ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -223,7 +226,7 @@ By default, the report only includes packages managed by the Operating System (O ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -326,7 +329,7 @@ To enable Container Scanning in a project, create a merge request from the Secur page: 1. In the project where you want to enable Container Scanning, go to - **Security & Compliance > Configuration**. + **Security and Compliance > Security configuration**. 1. In the **Container Scanning** row, select **Configure with a merge request**. This automatically creates a merge request with the changes necessary to enable Container Scanning. @@ -346,7 +349,7 @@ This example sets `GIT_STRATEGY` to `fetch`: ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -392,7 +395,7 @@ duplicated: ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: @@ -578,7 +581,7 @@ For details on saving and transporting Docker images as a file, see the Docker d ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: image: $CI_REGISTRY/namespace/container-scanning @@ -629,7 +632,7 @@ This example shows the configuration needed to scan images in a private [Google ```yaml include: - - template: Jobs/Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 5d2593a1bed..4731e09474c 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -40,7 +40,7 @@ You can use the following fuzzing engines to test the specified languages. | Language | Fuzzing Engine | Example | |---------------------------------------------|------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------| | C/C++ | [libFuzzer](https://llvm.org/docs/LibFuzzer.html) | [c-cpp-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/c-cpp-fuzzing-example) | -| GoLang | [go-fuzz (libFuzzer support)](https://github.com/dvyukov/go-fuzz) | [go-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/go-fuzzing-example) | +| Go | [go-fuzz (libFuzzer support)](https://github.com/dvyukov/go-fuzz) | [go-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/go-fuzzing-example) | | Swift | [libFuzzer](https://github.com/apple/swift/blob/master/docs/libFuzzerIntegration.md) | [swift-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/swift-fuzzing-example) | | Rust | [cargo-fuzz (libFuzzer support)](https://github.com/rust-fuzz/cargo-fuzz) | [rust-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/rust-fuzzing-example) | | Java | [Javafuzz](https://gitlab.com/gitlab-org/security-products/analyzers/fuzzers/javafuzz) (recommended) | [javafuzz-fuzzing-example](https://gitlab.com/gitlab-org/security-products/demos/coverage-fuzzing/javafuzz-fuzzing-example) | @@ -54,7 +54,7 @@ You can use the following fuzzing engines to test the specified languages. To confirm the status of coverage-guided fuzz testing: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Coverage Fuzzing** section the status is: - **Not configured** - **Enabled** @@ -168,7 +168,7 @@ artifacts files you can download from the CI/CD pipeline. Also, a project member To view details of the corpus registry: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Coverage Fuzzing** section, select **Manage corpus**. ### Create a corpus in the corpus registry @@ -196,7 +196,7 @@ provided by the `COVFUZZ_CORPUS_NAME` variable. The corpus is updated on every p To upload an existing corpus file: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Coverage Fuzzing** section, select **Manage corpus**. 1. Select **New corpus**. 1. Complete the fields. diff --git a/doc/user/application_security/dast/authentication.md b/doc/user/application_security/dast/authentication.md index 77732ab532c..03a273ffff9 100644 --- a/doc/user/application_security/dast/authentication.md +++ b/doc/user/application_security/dast/authentication.md @@ -45,34 +45,44 @@ To run a DAST authenticated scan: ### Prerequisites +- You have the username and password of the user you would like to authenticate as during the scan. +- You have checked the [known limitations](#known-limitations) to ensure DAST can authenticate to your application. +- You have satisfied the prerequisites depending on whether you're using [form authentication](#form-authentication) or [HTTP authentication]((#http-authentication). +- You have thought about how you can [verify](#verifying-authentication-is-successful) whether or not authentication was successful. + +#### Form authentication + - You are using either the [DAST proxy-based analyzer](proxy-based.md) or the [DAST browser-based analyzer](browser_based.md). - You know the URL of the login form of your application. Alternatively, you know how to navigate to the login form from the authentication URL (see [clicking to navigate to the login form](#clicking-to-navigate-to-the-login-form)). -- You have the username and password of the user you would like to authenticate as during the scan. - You know the [selectors](#finding-an-elements-selector) of the username and password HTML fields that DAST uses to input the respective values. - You know the element's [selector](#finding-an-elements-selector) that submits the login form when selected. -- You have thought about how you can [verify](#verifying-authentication-is-successful) whether or not authentication was successful. -- You have checked the [known limitations](#known-limitations) to ensure DAST can authenticate to your application. + +#### HTTP authentication + +- You must be using the [DAST browser-based analyzer](browser_based.md). ### Available CI/CD variables | CI/CD variable | Type | Description | |:-----------------------------------------------|:------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `DAST_AUTH_COOKIES` | string | Set to a comma-separated list of cookie names to specify which cookies are used for authentication. | -| `DAST_AUTH_REPORT` | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_REPORT` | boolean | Set to `true` to generate a report detailing steps taken during the authentication process. You must also define `gl-dast-debug-auth-report.html` as a CI job artifact to be able to access the generated report. Useful for debugging when authentication fails. | +| `DAST_AUTH_TYPE` <sup>2</sup> | string | The authentication type to use. Example: `basic-digest`. | | `DAST_AUTH_URL` <sup>1</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Example: `https://login.example.com`. | | `DAST_AUTH_VERIFICATION_LOGIN_FORM` | boolean | Verifies successful authentication by checking for the absence of a login form once the login form has been submitted. | | `DAST_AUTH_VERIFICATION_SELECTOR` | [selector](#finding-an-elements-selector) | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | | `DAST_AUTH_VERIFICATION_URL` <sup>1</sup> | URL | Verifies successful authentication by checking the URL in the browser once the login form has been submitted. Example: `"https://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | | `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1</sup> | [selector](#finding-an-elements-selector) | Comma-separated list of selectors that are selected prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | | `DAST_EXCLUDE_URLS` <sup>1</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. | -| `DAST_FIRST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_FIRST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9894) in GitLab 12.4. | | `DAST_PASSWORD` <sup>1</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | | `DAST_PASSWORD_FIELD` | string | The selector of password field at the sign-in HTML form. Example: `id:password` | -| `DAST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when selected submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9894) in GitLab 12.4. | | `DAST_USERNAME` <sup>1</sup> | string | The username to authenticate to in the website. Example: `admin` | | `DAST_USERNAME_FIELD` <sup>1</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | 1. Available to an on-demand proxy-based DAST scan. +1. Not available to proxy-based scans. ### Update the target website @@ -93,6 +103,28 @@ dast: DAST_AUTH_URL: "https://example.com/login" ``` +### Configuration for HTTP authentication + +To use an [HTTP authentication scheme](https://www.chromium.org/developers/design-documents/http-authentication/) such as Basic Authentication you can set the `DAST_AUTH_TYPE` value to `basic-digest`. +Other schemes such as Negotiate or NTLM may work but aren't officially supported due to current lack of automated test coverage. + +Configuration requires the CI/CD variables `DAST_AUTH_TYPE`, `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD` to be defined for the DAST job. If you don't have a unique login URL, please set `DAST_AUTH_URL` to the same URL as `DAST_WEBSITE`. + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_TYPE: "basic-digest" + DAST_AUTH_URL: "https://example.com" +``` + +Do **not** define `DAST_USERNAME` and `DAST_PASSWORD` in the YAML job definition file as this could present a security risk. Instead, create them as masked CI/CD variables using the GitLab UI. +See [Custom CI/CD variables](../../../ci/variables/index.md#for-a-project) for more information. +The proxy-based analyzer does not support basic authentication as an authentication mechanism. A workaround could be to set `DAST_REQUEST_HEADERS` as a masked CI/CD variable with a value containing the appropriate `Authorization` header, for example, `Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQK`. + ### Configuration for a single-step login form A single-step login form has all login form elements on a single page. @@ -114,7 +146,7 @@ dast: ``` Do **not** define `DAST_USERNAME` and `DAST_PASSWORD` in the YAML job definition file as this could present a security risk. Instead, create them as masked CI/CD variables using the GitLab UI. -See [Custom CI/CI variables](../../../ci/variables/index.md#for-a-project) for more information. +See [Custom CI/CD variables](../../../ci/variables/index.md#for-a-project) for more information. ### Configuration for a multi-step login form @@ -140,7 +172,7 @@ dast: ``` Do **not** define `DAST_USERNAME` and `DAST_PASSWORD` in the YAML job definition file as this could present a security risk. Instead, create them as masked CI/CD variables using the GitLab UI. -See [Custom CI/CI variables](../../../ci/variables/index.md#for-a-project) for more information. +See [Custom CI/CD variables](../../../ci/variables/index.md#for-a-project) for more information. ### Configuration for Single Sign-On (SSO) diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index bdc08988cc0..88be88ad00e 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -42,10 +42,10 @@ To crawl a navigation path, DAST opens a browser window and instructs it to perf When the browser has finished loading the result of the final action, DAST inspects the page for actions a user might take, creates a new navigation for each found, and adds them to the navigation path to form new navigation paths. For example: -- DAST processes navigation path `LoadURL[https://example.com]`. -- DAST finds two user actions, `LeftClick[class=menu]` and `LeftClick[id=users]`. -- DAST creates two new navigation paths, `LoadURL[https://example.com] -> LeftClick[class=menu]` and `LoadURL[https://example.com] -> LeftClick[id=users]`. -- Crawling begins on the two new navigation paths. +1. DAST processes navigation path `LoadURL[https://example.com]`. +1. DAST finds two user actions, `LeftClick[class=menu]` and `LeftClick[id=users]`. +1. DAST creates two new navigation paths, `LoadURL[https://example.com] -> LeftClick[class=menu]` and `LoadURL[https://example.com] -> LeftClick[id=users]`. +1. Crawling begins on the two new navigation paths. It's common for an HTML element to exist in multiple places in an application, such as a menu visible on every page. Duplicate elements can cause crawlers to crawl the same pages again or become stuck in a loop. @@ -170,13 +170,13 @@ For authentication CI/CD variables, see [Authentication](authentication.md). | `DAST_ADVERTISE_SCAN` | boolean | `true` | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | | `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | | `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | -| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. Headers set using `DAST_REQUEST_HEADERS` are added to every request made to these hostnames. | +| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. Headers set using `DAST_REQUEST_HEADERS` are added to every request made to these hostnames. | | `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | -| `DAST_BROWSER_CRAWL_GRAPH` | boolean | `true` | Set to `true` to generate an SVG graph of navigation paths visited during crawl phase of the scan. | +| `DAST_BROWSER_CRAWL_GRAPH` | boolean | `true` | Set to `true` to generate an SVG graph of navigation paths visited during crawl phase of the scan. You must also define `gl-dast-crawl-graph.svg` as a CI job artifact to be able to access the generated graph. | | `DAST_BROWSER_DEVTOOLS_LOG` | string | `Default:messageAndBody,truncate:2000` | Set to log protocol messages between DAST and the Chromium browser. | | | `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | | `DAST_BROWSER_EXCLUDED_ELEMENTS` | selector | `a[href='2.html'],css:.no-follow` | Comma-separated list of selectors that are ignored when scanning. | -| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | +| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | | `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | | `DAST_BROWSER_FILE_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended logging level for use in the file log. | | `DAST_BROWSER_FILE_LOG_PATH` | string | `/output/browserker.log` | Set to the path of the file log. | @@ -190,8 +190,8 @@ For authentication CI/CD variables, see [Authentication](authentication.md). | `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | | `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | | `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com, we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but are likely to produce little benefit after five to seven instances. | -| `DAST_BROWSER_PAGE_LOADING_SELECTOR` | selector | `css:#page-is-loading` | Selector that when is no longer visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Cannot be used with `DAST_BROWSER_PAGE_READY_SELECTOR` | -| `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Cannot be used with `DAST_BROWSER_PAGE_LOADING_SELECTOR` | +| `DAST_BROWSER_PAGE_LOADING_SELECTOR` | selector | `css:#page-is-loading` | Selector that when is no longer visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Cannot be used with `DAST_BROWSER_PAGE_READY_SELECTOR`. | +| `DAST_BROWSER_PAGE_READY_SELECTOR` | selector | `css:#page-is-ready` | Selector that when detected as visible on the page, indicates to the analyzer that the page has finished loading and the scan can continue. Cannot be used with `DAST_BROWSER_PAGE_LOADING_SELECTOR`. | | `DAST_BROWSER_SCAN` | boolean | `true` | Required to be `true` to run a browser-based scan. | | `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or user actions. | | `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://pkg.go.dev/time#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | @@ -201,8 +201,8 @@ For authentication CI/CD variables, see [Authentication](authentication.md). | `DAST_PATHS` | string | `/page1.html,/category1/page3.html` | Set to a comma-separated list of URL paths relative to `DAST_WEBSITE` for DAST to scan. | | `DAST_PATHS_FILE` | string | `/builds/project/urls.txt` | Set to a file path containing a list of URL paths relative to `DAST_WEBSITE` for DAST to scan. The file must be plain text with one path per line. | | `DAST_PKCS12_CERTIFICATE_BASE64` | string | `ZGZkZ2p5NGd...` | The PKCS12 certificate used for sites that require Mutual TLS. Must be encoded as base64 text. | -| `DAST_PKCS12_PASSWORD` | string | `password` | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. Create sensitive [custom CI/CI variables](../../../ci/variables/index.md#define-a-cicd-variable-in-the-ui) using the GitLab UI. | -| `DAST_REQUEST_HEADERS` | string | `Cache-control:no-cache` | Set to a comma-separated list of request header names and values. | +| `DAST_PKCS12_PASSWORD` | string | `password` | The password of the certificate used in `DAST_PKCS12_CERTIFICATE_BASE64`. Create sensitive [custom CI/CI variables](../../../ci/variables/index.md#define-a-cicd-variable-in-the-ui) using the GitLab UI. | +| `DAST_REQUEST_HEADERS` | string | `Cache-control:no-cache` | Set to a comma-separated list of request header names and values. | | `DAST_SKIP_TARGET_CHECK` | boolean | `true` | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. | | `DAST_TARGET_AVAILABILITY_TIMEOUT` | number | `60` | Time limit in seconds to wait for target availability. | | `DAST_WEBSITE` | URL | `https://example.com` | The URL of the website to scan. | @@ -275,16 +275,6 @@ dast: NOTE: Adjusting these values may impact scan time because they adjust how long each browser waits for various activities to complete. -## Artifacts - -Using the latest version of the DAST [template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml) these artifacts are exposed for download by default. - -The list of artifacts includes the following files: - -- `gl-dast-debug-auth-report.html` -- `gl-dast-debug-crawl-report.html` -- `gl-dast-crawl-graph.svg` - ## Troubleshooting See [troubleshooting](browser_based_troubleshooting.md) for more information. diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md index f70afac4c26..ae2d04ae4bb 100644 --- a/doc/user/application_security/dast/proxy-based.md +++ b/doc/user/application_security/dast/proxy-based.md @@ -80,10 +80,10 @@ To enable DAST to run automatically, either: - Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided by [Auto DevOps](../../../topics/autodevops/index.md)). -- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlabciyml-file-manually). -- [Use an automatically configured merge request](#configure-dast-using-the-ui). +- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlab-ciyml-file-manually). +- [Configure DAST using the UI](#configure-dast-using-the-ui). -#### Edit the `.gitlab.ci.yml` file manually +#### Edit the `.gitlab-ci.yml` file manually In this method you manually edit the existing `.gitlab-ci.yml` file. Use this method if your GitLab CI/CD configuration file is complex. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 0cdbb879cfc..49d5859be45 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -48,6 +48,7 @@ The following projects demonstrate DAST API scanning: - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/postman-example) - [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/graphql-example) - [Example SOAP project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/soap-example) +- [Authentication Token using Selenium](https://gitlab.com/gitlab-org/security-products/demos/api-dast/auth-token-selenium) ## Targeting API for DAST scanning @@ -97,8 +98,6 @@ The environment variables `DAST_API_OPENAPI_ALL_MEDIA_TYPES` and `DAST_API_OPENA #### Configure DAST API with an OpenAPI Specification -To configure DAST API scanning with an OpenAPI specification: - To configure DAST API scanning with an OpenAPI Specification: 1. [Include](../../../ci/yaml/index.md#includetemplate) @@ -264,6 +263,8 @@ This example is a minimal configuration for DAST API. From here you can: #### DAST API scanning with a GraphQL Schema file +DAST API can use a GraphQL schema file to understand and test a GraphQL endpoint that has introspection disabled. To use a GraphQL schema file, it must be in the introspection JSON format. A GraphQL schema can be converted to a the introspection JSON format using an online 3rd party tool: [https://transform.tools/graphql-to-introspection-json](https://transform.tools/graphql-to-introspection-json). + To configure DAST API to use a GraphQL schema file that provides information about the target API to test: 1. [Include](../../../ci/yaml/index.md#includetemplate) @@ -1924,7 +1925,7 @@ variables: When configured correctly, a CI/CD pipeline contains a `dast` stage and an `dast_api` job. The job only fails when an invalid configuration is provided. During typical operation, the job always succeeds even if vulnerabilities are identified during testing. -Vulnerabilities are displayed on the **Security** pipeline tab with the suite name. When testing against the repositories default branch, the DAST API vulnerabilities are also shown on the Security & Compliance's Vulnerability Report page. +Vulnerabilities are displayed on the **Security** pipeline tab with the suite name. When testing against the repositories default branch, the DAST API vulnerabilities are also shown on the Security and Compliance's Vulnerability Report page. To prevent an excessive number of reported vulnerabilities, the DAST API scanner limits the number of vulnerabilities it reports per operation. @@ -1941,7 +1942,7 @@ Follow these steps to view details of a vulnerability: 1. You can view vulnerabilities in a project, or a merge request: - - In a project, go to the project's **Security & Compliance > Vulnerability Report** + - In a project, go to the project's **Security and Compliance > Vulnerability Report** page. This page shows all vulnerabilities from the default branch only. - In a merge request, go the merge request's **Security** section and select the **Expand** button. DAST API vulnerabilities are available in a section labeled @@ -2558,7 +2559,7 @@ You can expedite the investigation with the [testssl.sh tool](https://testssl.sh To get support for your particular problem, use the [getting help channels](https://about.gitlab.com/get-help/). The [GitLab issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues) is the right place for bugs and feature proposals about API Security and DAST API. -Use `~"Category:API Security"` [label](../../../development/contributing/issue_workflow.md#labels) when opening a new issue regarding DAST API to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. +Use `~"Category:API Security"` [label](../../../development/labels/index.md) when opening a new issue regarding DAST API to ensure it is quickly reviewed by the right people. Refer to our [review response SLO](https://about.gitlab.com/handbook/engineering/workflow/code-review/#review-response-slo) to understand when you should receive a response. [Search the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) for similar entries before submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and or join the discussion. diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md index b86d98471ac..f46c5e5e801 100644 --- a/doc/user/application_security/dependency_list/index.md +++ b/doc/user/application_security/dependency_list/index.md @@ -13,7 +13,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Use the dependency list to review your project's dependencies and key details about those dependencies, including their known vulnerabilities. It is a collection of dependencies in your project, including existing and new findings. -To see the dependency list, go to your project and select **Security & Compliance > Dependency list**. +To see the dependency list, go to your project and select **Security and Compliance > Dependency list**. This information is sometimes referred to as a Software Bill of Materials, SBOM, or BOM. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 579dd4dfc4f..7641c39d6d6 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -187,17 +187,17 @@ table.supported-languages ul { <td>Y</td> </tr> <tr> - <td rowspan="2">Java</td> + <td rowspan="2">Java and Kotlin (not Android)<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup></td> <td rowspan="2"> 8 LTS, 11 LTS, - 13<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup>, - 14<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup>, - 15<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup>, - 16<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup>, + 13<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup>, + 14<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup>, + 15<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup>, + 16<sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup>, or 17 LTS </td> - <td><a href="https://gradle.org/">Gradle</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup></td> + <td><a href="https://gradle.org/">Gradle</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers-3">3</a></b></sup></td> <td> <ul> <li><code>build.gradle</code></li> @@ -295,15 +295,19 @@ table.supported-languages ul { <li> <a id="notes-regarding-supported-languages-and-package-managers-1"></a> <p> - Support for these versions of Java is deprecated and is planned to be removed in the GitLab 16.0 release. Additionally, these versions of Java are not supported by the FIPS-enabled image of <code>gemnasium-maven</code>. Official support is limited to LTS versions only. Although it may be possible to use Dependency Scanning with other versions by building a custom dependency scanning image, this approach is not officially supported by GitLab. + Support for Kotlin projects for Android is tracked in <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/336866">issue 336866</a>. </p> </li> <li> <a id="notes-regarding-supported-languages-and-package-managers-2"></a> <p> - Although Gradle with Java 8 is supported, there are other issues such that Android project builds are not supported at this time. - See the backlog issue <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/336866">Android support for Dependency - Scanning (gemnasium-maven)</a> for more details. Also, Gradle is not supported when <a href="https://docs.gitlab.com/ee/development/fips_compliance.html#enable-fips-mode">FIPS mode</a> is enabled. + Support for these versions of Java is deprecated and is planned to be removed in the GitLab 16.0 release. Additionally, these versions of Java are not supported by the FIPS-enabled image of <code>gemnasium-maven</code>. Official support is limited to LTS versions only. Although it may be possible to use Dependency Scanning with other versions by building a custom dependency scanning image, this approach is not officially supported by GitLab. + </p> + </li> + <li> + <a id="notes-regarding-supported-languages-and-package-managers-3"></a> + <p> + Gradle is not supported when <a href="https://docs.gitlab.com/ee/development/fips_compliance.html#enable-fips-mode">FIPS mode</a> is enabled. </p> </li> <li> @@ -553,7 +557,7 @@ always take the latest dependency scanning artifact available. To enable Dependency Scanning in a project, you can create a merge request: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Dependency Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable Dependency Scanning. @@ -626,6 +630,7 @@ The following variables allow configuration of global dependency scanning settin | `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](analyzers.md). | | `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs (see [`doublestar.Match`](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.0.2#Match) for supported patterns), or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | | `DS_IMAGE_SUFFIX` | Suffix added to the image name. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354796) in GitLab 14.10.) Automatically set to `"-fips"` when FIPS mode is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357922) in GitLab 15.0.) | +| `DS_MAX_DEPTH` | Defines how many directory levels deep that the analyzer should search for supported files to scan. A value of `-1` scans all directories regardless of depth. Default: `2`. | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | | `SECURE_LOG_LEVEL` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. Default: `info`. | @@ -926,10 +931,30 @@ include: merge cyclonedx sboms: stage: merge-cyclonedx-sboms image: - name: cyclonedx/cyclonedx-cli:0.24.0 + name: cyclonedx/cyclonedx-cli:0.24.2 entrypoint: [""] script: - - find . -name "gl-sbom-*.cdx.json" -exec /cyclonedx merge --output-file gl-sbom-all.cdx.json --input-files "{}" + + - apt-get update && apt-get install -y jq + - find . -name "gl-sbom-*.cdx.json" -exec cyclonedx merge --output-file gl-sbom-all.cdx.json --input-files "{}" + + # remove duplicates from merged file. See https://github.com/CycloneDX/cyclonedx-cli/issues/188 for details. + - | + jq '. | + { + "bomFormat": .bomFormat, + "specVersion": .specVersion, + "serialNumber": .serialNumber, + "version": .version, + "metadata": { + "tools": [ + (.metadata.tools | unique[]) + ] + }, + "components": [ + (.components | unique[]) + ] + }' "gl-sbom-all.cdx.json" > gl-sbom-all.cdx.json.tmp && mv gl-sbom-all.cdx.json.tmp gl-sbom-all.cdx.json + # optional: validate the merged sbom + - cyclonedx validate --input-version v1_4 --input-file gl-sbom-all.cdx.json artifacts: paths: - gl-sbom-all.cdx.json diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index c2f1257f989..937aaf76280 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -119,7 +119,7 @@ that you can download and analyze. To enable IaC Scanning in a project, you can create a merge request: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. 1. Review and merge the merge request to enable IaC Scanning. @@ -243,7 +243,8 @@ kics-iac-sast: ## Automatic vulnerability resolution -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.9 [with a project-level flag](../../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. Enabled by default on GitLab.com; disabled by default in self-managed. On GitLab.com, [contact Support](https://about.gitlab.com/support/) if you need to disable the flag for your project. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.9 [with a project-level flag](../../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. +> - Enabled by default in 15.10. On GitLab.com, [contact Support](https://about.gitlab.com/support/) if you need to disable the flag for your project. To help you focus on the vulnerabilities that are still relevant, GitLab IaC Scanning automatically [resolves](../vulnerabilities/index.md#vulnerability-status-values) vulnerabilities when: diff --git a/doc/user/application_security/policies/img/association_diagram.png b/doc/user/application_security/policies/img/association_diagram.png Binary files differindex d082e297c68..3a56aeba91b 100644 --- a/doc/user/application_security/policies/img/association_diagram.png +++ b/doc/user/application_security/policies/img/association_diagram.png diff --git a/doc/user/application_security/policies/img/policy_rule_mode_v14_9.png b/doc/user/application_security/policies/img/policy_rule_mode_v14_9.png Binary files differdeleted file mode 100644 index 8ca7547a33c..00000000000 --- a/doc/user/application_security/policies/img/policy_rule_mode_v14_9.png +++ /dev/null diff --git a/doc/user/application_security/policies/img/policy_rule_mode_v15_9.png b/doc/user/application_security/policies/img/policy_rule_mode_v15_9.png Binary files differnew file mode 100644 index 00000000000..8cb2e82ac05 --- /dev/null +++ b/doc/user/application_security/policies/img/policy_rule_mode_v15_9.png diff --git a/doc/user/application_security/policies/img/policy_yaml_mode_v14_9.png b/doc/user/application_security/policies/img/policy_yaml_mode_v14_9.png Binary files differdeleted file mode 100644 index 1d71e8684e9..00000000000 --- a/doc/user/application_security/policies/img/policy_yaml_mode_v14_9.png +++ /dev/null diff --git a/doc/user/application_security/policies/img/policy_yaml_mode_v15_9.png b/doc/user/application_security/policies/img/policy_yaml_mode_v15_9.png Binary files differnew file mode 100644 index 00000000000..95b637efef3 --- /dev/null +++ b/doc/user/application_security/policies/img/policy_yaml_mode_v15_9.png diff --git a/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_5.png b/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_5.png Binary files differdeleted file mode 100644 index 5ae7c2e065a..00000000000 --- a/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_5.png +++ /dev/null diff --git a/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_9.png b/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_9.png Binary files differnew file mode 100644 index 00000000000..57e729158da --- /dev/null +++ b/doc/user/application_security/policies/img/scan_execution_policy_rule_mode_v15_9.png diff --git a/doc/user/application_security/policies/img/scheduled_scan_execution_policies_diagram.png b/doc/user/application_security/policies/img/scheduled_scan_execution_policies_diagram.png Binary files differnew file mode 100644 index 00000000000..fcf7a8352fd --- /dev/null +++ b/doc/user/application_security/policies/img/scheduled_scan_execution_policies_diagram.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index a214d0d2cec..1f53b671ed1 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -13,7 +13,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Policies in GitLab provide security teams a way to require scans of their choice to be run whenever a project pipeline runs according to the configuration specified. Security teams can therefore be confident that the scans they set up have not been changed, altered, or disabled. You -can access these by navigating to your project's **Security & Compliance > Policies** page. +can access these by navigating to your project's **Security and Compliance > Policies** page. GitLab supports the following security policies: @@ -23,8 +23,8 @@ GitLab supports the following security policies: ## Security policy project All security policies are stored as YAML in a separate security policy project that gets linked to -the development project. This association can be a one-to-many relationship, allowing one security -policy project to apply to multiple development projects. Linked projects are not required to be in +the development project, group, or sub-group. This association can be a one-to-many relationship, allowing one security +policy project to apply to multiple development projects, groups, or sub-groups. Linked projects are not required to be in the same group as the development projects to which they are linked.  @@ -59,7 +59,7 @@ As a project owner, take the following steps to create or edit an association be project and a project that you would like to designate as the security policy project: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Policies**. +1. On the left sidebar, select **Security and Compliance > Policies**. 1. Select **Edit Policy Project**, and search for and select the project you would like to link from the dropdown list. 1. Select **Save**. @@ -83,7 +83,7 @@ policy's information (for example, description or enforcement status), and create and edit deployed policies: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Policies**. +1. On the left sidebar, select **Security and Compliance > Policies**.  @@ -94,7 +94,7 @@ status), and create and edit deployed policies: You can use the policy editor to create, edit, and delete policies: 1. On the top bar, select **Main menu > Projects** and find your group. -1. On the left sidebar, select **Security & Compliance > Policies**. +1. On the left sidebar, select **Security and Compliance > Policies**. - To create a new policy, select **New policy** which is located in the **Policies** page's header. You can then select which type of policy to create. - To edit an existing policy, select **Edit policy** in the selected policy drawer. @@ -104,13 +104,13 @@ The policy editor has two modes: - The visual _Rule_ mode allows you to construct and preview policy rules using rule blocks and related controls. -  +  - YAML mode allows you to enter a policy definition in `.yaml` format and is aimed at expert users and cases that the Rule mode doesn't support. -  +  You can use both modes interchangeably and switch between them at any time. If a YAML resource is incorrect or contains data not supported @@ -129,19 +129,6 @@ time that the first policy merge request is created. You can use the [Vulnerability-Check Migration](https://gitlab.com/gitlab-org/gitlab/-/snippets/2328089) script to bulk create policies or associate security policy projects with development projects. For instructions and a demonstration of how to use the Vulnerability-Check Migration script, see [this video](https://youtu.be/biU1N26DfBc). -## Scan execution policies - -See [Scan execution policies](scan-execution-policies.md). - -## Scan result policy editor - -See [Scan result policies](scan-result-policies.md). - -## Roadmap - -See the [Category Direction page](https://about.gitlab.com/direction/govern/security_policies/security_policy_management/) -for more information on the product direction of security policies within GitLab. - ## Troubleshooting ### `Branch name 'update-policy-<timestamp>' does not follow the pattern '<branch_name_regex>'` diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 26c7e1d9c77..96048bb2308 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -33,7 +33,7 @@ NOTE: Only group, subgroup, or project Owners have the [permissions](../../permissions.md#project-members-permissions) to select Security Policy Project. -Once your policy is complete, save it by selecting **Create via merge request** +Once your policy is complete, save it by selecting **Configure with a merge request** at the bottom of the editor. You are redirected to the merge request on the project's configured security policy project. If one does not link to your project, a security policy project is automatically created. Existing policies can also be @@ -44,7 +44,7 @@ Most policy changes take effect as soon as the merge request is merged. Any chan do not go through a merge request and are committed directly to the default branch may require up to 10 minutes before the policy changes take effect. - + ## Scan execution policies schema @@ -88,7 +88,7 @@ This rule enforces the defined actions and schedules a scan on the provided date |------------|------|-----------------|-------------| | `type` | `string` | `schedule` | The rule's type. | | `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). This field is required if the `agents` field is not set. | -| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | +| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. Minimum of 15 minute intervals when used together with the `branches` field. | | `agents` | `object` | | The name of the [GitLab agents](../../clusters/agent/index.md) where [Operational Container Scanning](../../clusters/agent/vulnerabilities.md) runs. The object key is the name of the Kubernetes agent configured for your project in GitLab. This field is required if the `branches` field is not set. | GitLab supports the following types of CRON syntax for the `cadence` field: @@ -99,8 +99,18 @@ GitLab supports the following types of CRON syntax for the `cadence` field: NOTE: Other elements of the [CRON syntax](https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm) may work in the cadence field if supported by the [cron](https://github.com/robfig/cron) we are using in our implementation, however, GitLab does not officially test or support them. -NOTE: -If using the `agents` field, required for `Operational Container Scanning`, the CRON expression is evaluated in [UTC](https://www.timeanddate.com/worldclock/timezone/utc) using the system-time of the Kubernetes-agent pod. If not using the `agents` field, the CRON expression is evaluated in standard [UTC](https://www.timeanddate.com/worldclock/timezone/utc) time from GitLab.com. If you have a self-managed GitLab instance and have [changed the server time zone](../../../administration/timezone.md), the CRON expression is evaluated with the new time zone. +When using the `schedule` rule type in conjunction with the `agents` field, note the following: + +- The GitLab Agent for Kubernetes checks every 30 seconds to see if there is an applicable policy. When a policy is found, the scans are executed according to the `cadence` defined. +- The CRON expression is evaluated using the system-time of the Kubernetes-agent pod. + +When using the `schedule` rule type in conjunction with the `branches` field, note the following: + +- The cron worker runs on 15 minute intervals and starts any pipelines that were scheduled to run during the previous 15 minutes. +- Based on your rule, you might expect scheduled pipelines to run with an offset of up to 15 minutes. +- The CRON expression is evaluated in standard [UTC](https://www.timeanddate.com/worldclock/timezone/utc) time from GitLab.com. If you have a self-managed GitLab instance and have [changed the server time zone](../../../administration/timezone.md), the CRON expression is evaluated with the new time zone. + + ### `agent` schema @@ -140,7 +150,7 @@ rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `scan` | `string` | `dast`, `secret_detection`, `sast`, `container_scanning`, `dependency_scanning` | The action's type. | +| `scan` | `string` | `sast`, `sast_iac`, `dast`, `secret_detection`, `container_scanning`, `dependency_scanning` | The action's type. | | `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | | `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index bc74b8bdfb1..104462529c1 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -10,8 +10,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can use scan result policies to take action based on scan results. For example, one type of scan result policy is a security approval policy that allows approval to be required based on the -findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning -job is fully executed. The following video gives you an overview of GitLab scan result policies: +findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning job is fully executed. + +NOTE: +Scan result policies are applicable only to [protected](../../project/protected_branches.md) target branches. + +The following video gives you an overview of GitLab scan result policies: <div class="video-fallback"> See the video: <a href="https://youtu.be/w5I9gcUgr9U">Overview of GitLab Scan Result Policies</a>. @@ -29,7 +33,7 @@ NOTE: Only project Owners have the [permissions](../../permissions.md#project-members-permissions) to select Security Policy Project. -Once your policy is complete, save it by selecting **Create merge request** at the bottom of the +Once your policy is complete, save it by selecting **Configure with a merge request** at the bottom of the editor. This redirects you to the merge request on the project's configured security policy project. If a security policy project doesn't link to your project, GitLab creates such a project for you. Existing policies can also be removed from the editor interface by selecting **Delete policy** at @@ -76,14 +80,14 @@ This rule enforces the defined actions based on security scan findings. |------------|------|-----------------|-------------| | `type` | `string` | `scan_finding` | The rule's type. | | `branches` | `array` of `string` | `[]` or the branch's name | Applicable only to protected target branches. An empty array, `[]`, applies the rule to all protected target branches. | -| `scanners` | `array` of `string` | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. | +| `scanners` | `array` of `string` | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. Note that `sast` includes results from both SAST and SAST IaC scanners. | | `vulnerabilities_allowed` | `integer` | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | | `severity_levels` | `array` of `string` | `info`, `unknown`, `low`, `medium`, `high`, `critical`| The severity levels for this rule to consider. | | `vulnerability_states` | `array` of `string` | `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed` | All vulnerabilities fall into two categories:<br><br>**Newly Detected Vulnerabilities** - the `newly_detected` policy option covers vulnerabilities identified in the merge request branch itself but that do not currently exist on the default branch. This policy option requires a pipeline to complete before the rule is evaluated so that it knows whether vulnerabilities are newly detected or not. Merge requests are blocked until the pipeline and necessary security scans are complete. The `newly_detected` option considers both of the following statuses:<br><br> • Detected<br> • Dismissed<br><br>**Pre-Existing Vulnerabilities** - these policy options are evaluated immediately and do not require a pipeline complete as they consider only vulnerabilities previously detected in the default branch.<br><br> • `Detected` - the policy looks for vulnerabilities in the detected state.<br> • `Confirmed` - the policy looks for vulnerabilities in the confirmed state.<br> • `Dismissed` - the policy looks for vulnerabilities in the dismissed state.<br> • `Resolved` - the policy looks for vulnerabilities in the resolved state. | ## `license_finding` rule type -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8092) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `license_scanning_policies`. Disabled by default. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8092) in GitLab 15.9 [with a flag](../../../administration/feature_flags.md) named `license_scanning_policies`. Enabled by default in GitLab 15.10. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `license_scanning_policies`. diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index b0d84e4cff9..a0c5adc89f5 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -183,7 +183,8 @@ For more information, see the confidential project `https://gitlab.com/gitlab-or ## Automatic vulnerability resolution -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.9 [with a project-level flag](../../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. Enabled by default on GitLab.com; disabled by default in self-managed. On GitLab.com, [contact Support](https://about.gitlab.com/support/) if you need to disable the flag for your project. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368284) in GitLab 15.9 [with a project-level flag](../../../administration/feature_flags.md) named `sec_mark_dropped_findings_as_resolved`. +> - Enabled by default in GitLab 15.10. On GitLab.com, [contact Support](https://about.gitlab.com/support/) if you need to disable the flag for your project. To help you focus on the vulnerabilities that are still relevant, GitLab SAST automatically [resolves](../vulnerabilities/index.md#vulnerability-status-values) vulnerabilities when: @@ -303,7 +304,7 @@ successfully, and an error may occur. To enable and configure SAST with customizations: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. If the project does not have a `.gitlab-ci.yml` file, select **Enable SAST** in the Static Application Security Testing (SAST) row, otherwise select **Configure SAST**. 1. Enter the custom SAST values. @@ -330,7 +331,7 @@ successfully, and an error may occur. To enable and configure SAST with default settings: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance** > **Configuration**. +1. On the left sidebar, select **Security and Compliance** > **Configuration**. 1. In the SAST section, select **Configure with a merge request**. 1. Review and merge the merge request to enable SAST. diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index d6aab71a2c6..6dd20ea094e 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -36,8 +36,8 @@ more information, see [Post-processing and revocation](post_processing.md). All identified secrets are reported in the: -- Merge request widget -- Pipelines' **Security** tab +- [Merge request widget](../index.md#view-security-scan-information-in-merge-requests) +- [Pipeline security report](../vulnerability_report/pipeline.md) - [Vulnerability Report](../vulnerability_report/index.md)  @@ -141,12 +141,12 @@ To enable Secret Detection, either: - Enable [Auto DevOps](../../../topics/autodevops/index.md), which includes [Auto Secret Detection](../../../topics/autodevops/stages.md#auto-secret-detection). -- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlabciyml-file-manually). Use this method if - your `.gitlab-ci.yml` file is complex. +- [Edit the `.gitlab.ci.yml` file manually](#edit-the-gitlab-ciyml-file-manually). Use this method + if your `.gitlab-ci.yml` file is complex. - [Use an automatically configured merge request](#use-an-automatically-configured-merge-request). -### Edit the `.gitlab.ci.yml` file manually +### Edit the `.gitlab-ci.yml` file manually This method requires you to manually edit the existing `.gitlab-ci.yml` file. Use this method if your GitLab CI/CD configuration file is complex. @@ -180,12 +180,12 @@ the `.gitlab-ci.yml` file. You then merge the merge request to enable Secret Det NOTE: This method works best with no existing `.gitlab-ci.yml` file, or with a minimal configuration file. If you have a complex GitLab configuration file it may not be parsed successfully, and an -error may occur. In that case, use the [manual](#edit-the-gitlabciyml-file-manually) method instead. +error may occur. In that case, use the [manual](#edit-the-gitlab-ciyml-file-manually) method instead. -To enable Secret Detection automatically: +To enable Secret Detection: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. In the **Secret Detection** row, select **Configure with a merge request**. 1. Optional. Complete the fields. 1. Select **Create merge request**. @@ -195,12 +195,13 @@ Pipelines now include a Secret Detection job. ## Responding to a leaked secret -Secrets detected by the analyzer should be immediately rotated. -[Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) -may not be effective in removing all references to the file. Additionally, the secret will remain in any existing -forks or clones of the repository. +When a secret is detected, you should rotate it immediately. GitLab attempts to +[automatically revoke](post_processing.md) some types of leaked secrets. For those that are not +automatically revoked, you must do so manually. -GitLab will attempt to [automatically revoke](post_processing.md) some types of leaked secrets. +[Purging a secret from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) +does not fully address the leak. The original secret remains in any existing forks or +clones of the repository. ## Pinning to specific analyzer version @@ -413,18 +414,16 @@ In the following example `secret-detection-ruleset.toml` file, rules are matched ### Synthesize a custom configuration -To create a custom configuration, you can use passthrough chains. Passthroughs can also be chained -to build more complex configurations. For more details, see -[SAST Customize ruleset](../sast/customize_rulesets.md). +You can use passthroughs to override the default Secret Detection ruleset. The +following passthrough types are supported by the `secrets` analyzer: -Only the following passthrough types are supported by the `secrets` analyzer: - -- `file` - `raw` +- `file` -In the `secret-detection-ruleset.toml` file, do one of the following: +To define a passthrough, add _one_ of the following to the +`secret-detection-ruleset.toml` file: -- Define a custom ruleset, for example: +- Using an inline (`raw`) value: ```toml [secrets] @@ -442,7 +441,7 @@ In the `secret-detection-ruleset.toml` file, do one of the following: """ ``` -- Provide the name of the file containing a custom ruleset, for example: +- Using an external `file` committed to the current repository: ```toml [secrets] @@ -454,6 +453,10 @@ In the `secret-detection-ruleset.toml` file, do one of the following: value = "config/gitleaks.toml" ``` +For more information on the syntax of passthroughs, see the +[passthroughs section on the SAST customize rulesets](../sast/customize_rulesets.md#the-analyzerpassthrough-section) +page. + ## Running Secret Detection in an offline environment **(PREMIUM SELF)** An offline environment has limited, restricted, or intermittent access to external resources through diff --git a/doc/user/application_security/secure_your_application.md b/doc/user/application_security/secure_your_application.md index fb10efff2c6..8dd1168a0d9 100644 --- a/doc/user/application_security/secure_your_application.md +++ b/doc/user/application_security/secure_your_application.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab can check your applications for security vulnerabilities. - [Get started](get-started-security.md) -- [Security Configuration](configuration/index.md) +- [Security configuration](configuration/index.md) - [Container Scanning](container_scanning/index.md) - [Dependency Scanning](dependency_scanning/index.md) - [Static Application Security Testing](sast/index.md) diff --git a/doc/user/application_security/security_dashboard/img/security_center_dashboard_v13_4.png b/doc/user/application_security/security_dashboard/img/security_center_dashboard_v13_4.png Binary files differdeleted file mode 100644 index 5379b5c6e5d..00000000000 --- a/doc/user/application_security/security_dashboard/img/security_center_dashboard_v13_4.png +++ /dev/null diff --git a/doc/user/application_security/security_dashboard/img/security_center_dashboard_v15_10.png b/doc/user/application_security/security_dashboard/img/security_center_dashboard_v15_10.png Binary files differnew file mode 100644 index 00000000000..c2780fce787 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/security_center_dashboard_v15_10.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 7388ef80e68..1a8b4dac10e 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -64,7 +64,7 @@ Project Security Dashboards show statistics for all vulnerabilities with a curre To view total number of vulnerabilities over time: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Security Dashboard**. +1. On the left sidebar, select **Security and Compliance > Security Dashboard**. 1. Filter and search for what you need. - To filter the chart by severity, select the legend name. - To view a specific time frame, use the time range handles (**{scroll-handle}**). @@ -77,7 +77,7 @@ To view total number of vulnerabilities over time: To download an SVG image of the vulnerabilities chart: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Security dashboard**. +1. On the left sidebar, select **Security and Compliance > Security dashboard**. 1. Select **Save chart as an image** (**{download}**). ## View vulnerabilities over time for a group @@ -133,7 +133,7 @@ The Security Center includes: - A [vulnerability report](../vulnerability_report/index.md). - A settings area to configure which projects to display. - + ### View the Security Center @@ -151,6 +151,9 @@ To add projects to the Security Center: After you add projects, the security dashboard and vulnerability report show the vulnerabilities found in those projects' default branches. +You can add a maximum of 1,000 projects, however the **Project** filter in the **Vulnerability +Report** is limited to 100 projects. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 67a1257799b..8107b9d8484 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -54,7 +54,7 @@ no longer detected, and change their status. To change a vulnerability's status from its Vulnerability Page: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Status** dropdown list select a status, then select **Change status**. 1. Optionally, at the bottom of the page, add a comment to the log entry. @@ -80,7 +80,7 @@ that when Jira integration is enabled, the GitLab issue feature is not available To create a GitLab issue for a vulnerability: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. Select **Create issue**. @@ -102,7 +102,7 @@ Prerequisites: To create a Jira issue for a vulnerability: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. Select **Create Jira issue**. 1. If you're not already logged in to Jira, sign in. @@ -135,7 +135,7 @@ Be aware of the following conditions between a vulnerability and a linked issue: To link a vulnerability to existing issues: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. In the **Linked issues** section, select the plus icon (**{plus}**). 1. For each issue to be linked, either: @@ -170,7 +170,7 @@ To resolve a vulnerability, you can either: To resolve the vulnerability with a merge request: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Resolve with merge request** dropdown list, select **Resolve with merge request**. @@ -182,7 +182,7 @@ Process the merge request according to your standard workflow. To manually apply the patch that GitLab generated for a vulnerability: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability's description. 1. From the **Resolve with merge request** dropdown list, select **Download patch to resolve**. 1. Ensure your local project has the same commit checked out that was used to generate the patch. @@ -195,17 +195,19 @@ To manually apply the patch that GitLab generated for a vulnerability: > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6176) in GitLab 14.9. +NOTE: +Security training is not available in an offline environment because it uses content from +third-party vendors. + Security training helps your developers learn how to fix vulnerabilities. Developers can view security training from selected educational providers, relevant to the detected vulnerability. To enable security training for vulnerabilities in your project: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Configuration**. +1. On the left sidebar, select **Security and Compliance > Security configuration**. 1. On the tab bar, select **Vulnerability Management**. 1. To enable a security training provider, turn on the toggle. -Security training uses content from third-party vendors. You must have an internet connection to use this feature. - ## View security training for a vulnerability > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6176) in GitLab 14.9. @@ -220,6 +222,6 @@ Vulnerabilities with a CWE are most likely to return a training result. To view the security training for a vulnerability: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. 1. Select the vulnerability for which you want to view security training. 1. Select **View training**. diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index e6353264f39..3d7565f97bc 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -45,7 +45,7 @@ At the project level, the Vulnerability Report also contains: To view the project-level vulnerability report: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability report**. ## Vulnerability Report actions @@ -242,7 +242,7 @@ Vulnerability records cannot be deleted, so a permanent record always remains. You can dismiss a vulnerability in projects and groups: 1. Select the vulnerability in the Security Dashboard. -1. In the upper right, from the **Status** selector menu, select **Dismissed**. +1. In the upper-right corner, from the **Status** dropdown list, select **Dismissed**. 1. Optional. Add a reason for the dismissal and select **Save comment**. To undo this action, select a different status from the same menu. @@ -256,7 +256,7 @@ To undo this action, select a different status from the same menu. To add a new vulnerability finding from your project level Vulnerability Report page: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Vulnerability Report**. +1. On the left sidebar, select **Security and Compliance > Vulnerability Report**. 1. Select **Submit vulnerability**. 1. Complete the fields and submit the form. diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index e1459717241..919b76e930f 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -36,7 +36,7 @@ celebrate an accomplishment or agree with an opinion. To add an award emoji: -1. In the upper right of the comment, select the smile (**{slight-smile}**). +1. In the upper-right corner of the comment, select the smile (**{slight-smile}**). 1. Select an emoji from the dropdown list. To remove an award emoji, select the emoji again. diff --git a/doc/user/clusters/agent/gitops.md b/doc/user/clusters/agent/gitops.md index 787b0062017..1d71f6ac511 100644 --- a/doc/user/clusters/agent/gitops.md +++ b/doc/user/clusters/agent/gitops.md @@ -12,6 +12,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346585) to make the `id` attribute optional in GitLab 15.7. > - Specifying a branch, tag, or commit reference to fetch the Kubernetes manifest files [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4516) in GitLab 15.7. +NOTE: +From GitLab 15.10, you should use [Flux](gitops/flux.md) for GitOps. For more information, see +[this announcement blog post](https://about.gitlab.com/blog/2023/02/08/why-did-we-choose-to-integrate-fluxcd-with-gitlab/). + With GitOps, you can manage containerized clusters and applications from a Git repository that: - Is the single source of truth of your system. diff --git a/doc/user/clusters/agent/gitops/flux.md b/doc/user/clusters/agent/gitops/flux.md new file mode 100644 index 00000000000..9a3537aacbf --- /dev/null +++ b/doc/user/clusters/agent/gitops/flux.md @@ -0,0 +1,36 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Flux (Beta) **(FREE)** + +Flux is a GitOps tool that helps you manage your Kubernetes clusters. +You can use Flux to: + +- Keep your clusters in sync with your Git repositories. +- Reconcile code changes with your deployments. +- Manage your Flux installation itself with a bootstrap. + +To get started, see the [Flux installation documentation](https://fluxcd.io/flux/installation). + +Support for Flux is in [Open Beta](../../../../policy/alpha-beta-support.md#beta-features). + +## Bootstrap installation + +Use the Flux command [`bootstrap gitlab`](https://fluxcd.io/flux/installation/#gitlab-and-gitlab-enterprise) +to configure a Kubernetes cluster to manage itself from a Git repository. + +You must authenticate your installation with either: + +- Recommended. [A project access token](../../../project/settings/project_access_tokens.md). +- A [group access token](../../../group/settings/group_access_tokens.md). +- A [personal access token](../../../profile/personal_access_tokens.md). + +Some Flux features like [automated image updates](https://fluxcd.io/flux/guides/image-update/) require +write access to the source repositories. + +## GitOps repository structure + +You should organize your repositories to meet the needs of your team. For detailed recommendations, see the Flux [repository structure documentation](https://fluxcd.io/flux/guides/repository-structure/). diff --git a/doc/user/clusters/agent/gitops/flux_tutorial.md b/doc/user/clusters/agent/gitops/flux_tutorial.md new file mode 100644 index 00000000000..5f902002f83 --- /dev/null +++ b/doc/user/clusters/agent/gitops/flux_tutorial.md @@ -0,0 +1,184 @@ +--- +stage: Configure +group: Configure +info: A tutorial using Flux with Project Access Tokens +--- + +# Tutorial: Set up Flux for GitOps **(FREE)** + +This tutorial teaches you how to set up Flux for GitOps. You'll set up a sample project, +complete a bootstrap Flux installation, and authenticate your installation with a +[project access token](../../../project/settings/project_access_tokens.md). + +You can find the fully configured tutorial project [in this GitLab repository](https://gitlab.com/gitlab-org/configure/examples/flux/flux-config). It works in conjunction with [this repository](https://gitlab.com/gitlab-org/configure/examples/flux/web-app-manifests/-/tree/main), which contains the example Kubernetes manifests. + +To set up Flux with a project access token: + +1. [Create the Flux repository](#create-the-flux-repository) +1. [Create the Kubernetes manifest repository](#create-the-kubernetes-manifest-repository) +1. [Configure Flux to sync your manifests](#configure-flux-to-sync-your-manifests) +1. [Verify your configuration](#verify-your-configuration) + +Prerequisites: + +- On GitLab SaaS, you must have the Premium or Ultimate tier to use a project access token. + On self-managed instances, you can have any tier. +- Not recommended. You can authenticate with a [personal or group access token](flux.md#bootstrap-installation) in all tiers. +- You must have a Kubernetes cluster running. + +## Create the Flux repository + +To start, create a Git repository, install Flux, and authenticate Flux with your repo: + +1. Make sure your `kubectl` is configured to access your cluster. +1. [Install the Flux CLI](https://fluxcd.io/flux/installation/#install-the-flux-cli). +1. In GitLab, create a new empty project called `flux-config`. +1. In the `flux-config` project, create a [project access token](../../../project/settings/project_access_tokens.md#create-a-project-access-token) with the following settings: + + - From the **Select a role** dropdown list, select **Maintainer**. + - Under **Select scopes**, select the **API** and **write_repository** checkboxes. + + Name the project token `flux-project-access-token`. + +1. From your shell, export a `GITLAB_TOKEN` environment variable with the value of your project access token. + For example, `export GITLAB_TOKEN=<flux-project-access-token>`. +1. Run the `bootstrap` command. The exact command depends on whether you are + creating the Flux repository under a GitLab user, group, or subgroup. For more information, + see the [Flux bootstrap documentation](https://fluxcd.io/flux/installation/#gitlab-and-gitlab-enterprise). + + In this tutorial, you're working with a public project in a subgroup. The bootstrap command looks like this: + + ```shell + flux bootstrap gitlab \ + --owner=gitlab-org/configure/examples/flux \ + --repository=flux-config \ + --branch=main \ + --path=clusters/my-cluster + ``` + + This command installs Flux on the Kubernetes cluster and configures it to manage itself from the repository `flux-config`. + +Great work! You now have a repository, a project access token, and a Flux bootstrap installation authenticated to your repo. Any updates to your repo are automatically synced to the cluster. + +## Create the Kubernetes manifest repository + +Next, create a repository for your Kubernetes manifests: + +1. In GitLab, create a new repository called `web-app-manifests`. +1. Add a file to `web-app-manifests` named `nginx-deployment.yaml` with the following contents: + + ```yaml + apiVersion: apps/v1 + + kind: Deployment + + metadata: + name: nginx-deployment + labels: + app: nginx + spec: + replicas: 3 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 + ``` + +1. In the new repository, [create a deploy token](../../../project/deploy_tokens/index.md#create-a-deploy-token) with only the **read_repository** scope. +1. Store your deploy token username and password somewhere safe. +1. In Flux CLI, create a secret with your deploy token and point the secret to the new repository. For example: + + ```shell + flux create secret git flux-deploy-authentication \ + --url=https://gitlab.com/gitlab-org/configure/examples/flux/web-app-manifests \ + --namespace=default \ + --username=<token-user-name> \ + --password=<token-password> + ``` + +1. To check if your secret was generated successfully, run: + + ```shell + kubectl -n default get secrets flux-deploy-authentication -o yaml + ``` + + Under `data`, you should see base64-encoded values associated with your token username and password. + +Congratulations! You now have a manifest repository, a deploy token, and a secret generated directly on your cluster. + +## Configure Flux to sync your manifests + +Next, tell `flux-config` to sync with the `web-app-manifests` repository. + +To do so, create a [`GitRepository`](https://fluxcd.io/flux/components/source/gitrepositories/) resource: + +1. Clone the `flux-config` repo to your machine. +1. In your local clone of `flux-config`, add the `GitRepository` file `clusters/my-cluster/web-app-manifests-source.yaml`: + + ```yaml + --- + apiVersion: source.toolkit.fluxcd.io/v1beta2 + kind: GitRepository + metadata: + name: web-app-manifests + namespace: default + spec: + interval: 1m0s + ref: + branch: main + secretRef: + name: flux-deploy-authentication + url: https://gitlab.com/gitlab-org/configure/examples/flux/web-app-manifests + ``` + + This file uses `secretRef` to refer back to the deploy token secret you created in the last step. + +1. In your local clone of `flux-config`, add the `GitRepository` file `clusters/my-cluster/web-app-manifests-kustomization.yaml`: + + ```yaml + --- + apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 + kind: Kustomization + metadata: + name: nginx-source-kustomization + namespace: default + spec: + interval: 1m0s + path: ./ + prune: true + sourceRef: + kind: GitRepository + name: web-app-manifests + namespace: default + targetNamespace: default + ``` + + This file adds a [`Kustomization`](https://fluxcd.io/flux/components/kustomize/kustomization/) resource that tells Flux to sync the manifests from + `web-app-manifests` with `kustomize`. + +1. Commit the new files and push. + +## Verify your configuration + +You should see a newly created `nginx-deployment` pod in your cluster. + +To check whether the `nginx-deployment` pod is running in the default namespace, run the following: + +```shell +kubectl -n default get pods -n default +``` + +If you want to see the deployment sync again, try updating the number of replicas in the +`nginx-deployment.yaml` file and push to your `main` branch. If all is working well, it +should sync to the cluster. + +Excellent work! You've successfully set up a complete Flux project. diff --git a/doc/user/clusters/agent/gitops/secrets_management.md b/doc/user/clusters/agent/gitops/secrets_management.md index dc1cbe3009d..8a47dcfaf72 100644 --- a/doc/user/clusters/agent/gitops/secrets_management.md +++ b/doc/user/clusters/agent/gitops/secrets_management.md @@ -50,9 +50,9 @@ To deploy containers from the GitLab Container Registry, you must configure the 1. Generate a GitLab token with at least `read-registry` rights. The token can be either a Personal or a Project Access Token. 1. Create a Kubernetes secret manifest YAML file. Update the values as needed: - ```shell - kubectl create secret docker-registry gitlab-credentials --docker-server=registry.gitlab.example.com --docker-username=<gitlab-username> --docker-password=<gitlab-token> --docker-email=<gitlab-user-email> -n <namespace> --dry-run=client -o yaml > gitlab-credentials.yaml - ``` + ```shell + kubectl create secret docker-registry gitlab-credentials --docker-server=registry.gitlab.example.com --docker-username=<gitlab-username> --docker-password=<gitlab-token> --docker-email=<gitlab-user-email> -n <namespace> --dry-run=client -o yaml > gitlab-credentials.yaml + ``` 1. Encrypt the secret into a `SealedSecret` manifest: diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 8f4855a7f93..0d84a617808 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -12,6 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the GitLab agent became available on GitLab.com. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. > - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab agent for Kubernetes" in GitLab 14.6. +> - Flux [recommended](https://gitlab.com/gitlab-org/gitlab/-/issues/357947#note_1253489000) as GitOps solution in GitLab 15.10. You can connect your Kubernetes cluster with GitLab to deploy, manage, and monitor your cloud-native solutions. @@ -33,20 +34,7 @@ You can choose from two primary workflows. The GitOps workflow is recommended. ### GitOps workflow -In a [**GitOps** workflow](gitops.md): - -- You keep your Kubernetes manifests in GitLab. -- You install a GitLab agent in your cluster. -- Any time you update your manifests, the agent updates the cluster. -- The cluster automatically cleans up unexpected changes. It uses - [server-side applies](https://kubernetes.io/docs/reference/using-api/server-side-apply/) - to fix any configuration inconsistencies that third parties introduce. - -This workflow is fully driven with Git and is considered **pull-based**, -because the cluster is pulling updates from your GitLab repository. - -GitLab recommends this workflow. We are actively investing in this workflow -so we can provide a first-class experience. +You should use Flux for GitOps. To get started, see the GitLab [Flux documentation](../../../user/clusters/agent/gitops/flux.md). ### GitLab CI/CD workflow diff --git a/doc/user/clusters/agent/troubleshooting.md b/doc/user/clusters/agent/troubleshooting.md index f5f235d2758..84cdbbcc096 100644 --- a/doc/user/clusters/agent/troubleshooting.md +++ b/doc/user/clusters/agent/troubleshooting.md @@ -107,71 +107,72 @@ This error occurs when your GitLab instance is using a certificate signed by an certificate authority that is unknown to the agent. To fix this issue, you can present the CA certificate file to the agent -by using a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it -is picked up automatically. +by [customizing the Helm installation](install/index.md#customize-the-helm-installation). +Add `--set config.caCert="$(cat ~/path/to/ca.crt)"` to the `helm install` command. Make sure to replace `~/path/to/ca.crt` +with the path to your internal CA's certificate file. The file should be a valid PEM or DER-encoded certificate. -For example, if your internal CA certificate is `myCA.pem`: +When you deploy `agentk` with a set `config.caCert` value, the certificate is added to `configmap` and the certificate file is mounted in `/etc/ssl/certs`. -```plaintext -kubectl -n gitlab-agent create configmap ca-pemstore --from-file=myCA.pem +```yaml +$ kubectl get configmap -lapp=gitlab-agent -o yaml +apiVersion: v1 +items: +- apiVersion: v1 + data: + ca.crt: |- + -----BEGIN CERTIFICATE----- + MIIFmzCCA4OgAwIBAgIUE+FvXfDpJ869UgJitjRX7HHT84cwDQYJKoZIhvcNAQEL + ...truncated certificate... + GHZCTQkbQyUwBWJOUyOxW1lro4hWqtP4xLj8Dpq1jfopH72h0qTGkX0XhFGiSaM= + -----END CERTIFICATE----- + kind: ConfigMap + metadata: + annotations: + meta.helm.sh/release-name: self-signed + meta.helm.sh/release-namespace: gitlab-agent-self-signed + creationTimestamp: "2023-03-07T20:12:26Z" + labels: + app: gitlab-agent + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: gitlab-agent + app.kubernetes.io/version: v15.9.0 + helm.sh/chart: gitlab-agent-1.11.0 + name: self-signed-gitlab-agent + resourceVersion: "263184207" +kind: List ``` -Then in `resources.yml`: +You might see a similar error in the [agent server (KAS) logs](../../../administration/logs/index.md#gitlab-agent-server) of your GitLab application server: -```yaml - spec: - serviceAccountName: gitlab-agent - containers: - - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" - args: - - --token-file=/config/token - - --kas-address - - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. - # - wss://gitlab.host.tld:443/-/kubernetes-agent/ - # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. - # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. - volumeMounts: - - name: token-volume - mountPath: /config - - name: ca-pemstore-volume - mountPath: /etc/ssl/certs/myCA.pem - subPath: myCA.pem - volumes: - - name: token-volume - secret: - secretName: gitlab-agent-token - - name: ca-pemstore-volume - configMap: - name: ca-pemstore - items: - - key: myCA.pem - path: myCA.pem +```json +{"level":"error","time":"2023-03-07T20:19:48.151Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"} ``` -Alternatively, you can mount the certificate file at a different location and specify it for the -`--ca-cert-file` agent parameter: +To fix it, [install your internal CA's public certificate](https://docs.gitlab.com/omnibus/settings/ssl/#install-custom-public-certificates) in the `/etc/gitlab/trusted-certs` directory. -```yaml - containers: - - name: agent - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>" - args: - - --ca-cert-file=/tmp/myCA.pem - - --token-file=/config/token - - --kas-address - - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. - # - wss://gitlab.host.tld:443/-/kubernetes-agent/ - # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. - # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. - volumeMounts: - - name: token-volume - mountPath: /config - - name: ca-pemstore-volume - mountPath: /tmp/myCA.pem - subPath: myCA.pem +Alternatively, you can configure the agent server (KAS) to read the certificate from a custom directory. +Add the following configuration to `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_kas['env'] = { + 'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/" + } ``` +To apply the changes: + +1. Reconfigure GitLab. + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart `gitlab-kas`. + + ```shell + gitlab-ctl restart gitlab-kas + ``` + ## Project not found ```json diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md index 74bfab49c55..910092eb4e0 100644 --- a/doc/user/clusters/cost_management.md +++ b/doc/user/clusters/cost_management.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Cluster cost management (DEPRECATED) **(ULTIMATE)** +# Cluster cost management (deprecated) **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216737) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.5. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/clusters/environments.md b/doc/user/clusters/environments.md index f4313656803..dc6898bd3d3 100644 --- a/doc/user/clusters/environments.md +++ b/doc/user/clusters/environments.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Cluster Environments (DEPRECATED) **(PREMIUM)** +# Cluster Environments (deprecated) **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13392) in GitLab 12.3 for group-level clusters. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14809) in GitLab 12.4 for instance-level clusters. diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md index c5e56fcd3a7..b5600096856 100644 --- a/doc/user/clusters/integrations.md +++ b/doc/user/clusters/integrations.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Cluster integrations (DEPRECATED) **(FREE)** +# Cluster integrations (deprecated) **(FREE)** > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. > - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/353410) in GitLab 15.0. diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index 18317ae09ed..f2eb0cfbc39 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Cluster management project (DEPRECATED) **(FREE)** +# Cluster management project (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32810) in GitLab 12.5. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/clusters/migrating_from_gma_to_project_template.md b/doc/user/clusters/migrating_from_gma_to_project_template.md index e5d81091094..42510a93495 100644 --- a/doc/user/clusters/migrating_from_gma_to_project_template.md +++ b/doc/user/clusters/migrating_from_gma_to_project_template.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrate from GitLab Managed Apps to Cluster Management Projects (DEPRECATED) **(FREE)** +# Migrate from GitLab Managed Apps to Cluster Management Projects (deprecated) **(FREE)** The GitLab Managed Apps were deprecated in GitLab 14.0 in favor of user-controlled Cluster Management projects. diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index 04609026793..d5af7edf515 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -5,38 +5,57 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Compliance report **(ULTIMATE)** +# Compliance reports **(ULTIMATE)** + +See reports about compliance violations and compliance frameworks for the group. + +## Compliance violations report > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36524) in GitLab 12.8 as Compliance Dashboard. +> - Compliance violation drawer [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299357) in GitLab 14.1. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/299360) to compliance report in GitLab 14.2. > - [Replaced](https://gitlab.com/groups/gitlab-org/-/epics/5237) by merge request violations in GitLab 14.6 [with a flag](../../../administration/feature_flags.md) named `compliance_violations_report`. Disabled by default. > - GraphQL API [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7222) in GitLab 14.9. > - [Generally available](https://gitlab.com/groups/gitlab-org/-/epics/5237) in GitLab 14.10. [Feature flag `compliance_violations_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/346266) removed. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112111) to compliance violations report in GitLab 15.9. -Compliance report gives you the ability to see a group's merge request activity. It provides a -high-level view for all projects in the group. For example, code approved for merging into -production. +With compliance violations report, you can see a high-level view of merge request activity for all projects in the group. -You can use the report to get: +When you select a row in the compliance report, a drawer appears that provides: -- A list of compliance violations from all merged merge requests within the group. -- The reason and severity of each compliance violation. -- A link to the merge request that caused each compliance violation. +- The project name and [compliance framework label](../../project/settings/index.md#add-a-compliance-framework-to-a-project), + if the project has one assigned. +- A link to the merge request that introduced the violation. +- The merge request's branch path in the format `[source] into [target]`. +- A list of users that committed changes to the merge request. +- A list of users that commented on the merge request. +- A list of users that approved the merge request. +- The user that merged the merge request. -## View the compliance report for a group +### View the compliance violations report for a group Prerequisites: - You must be an administrator or have the Owner role for the group. -To view the compliance report: +To view the compliance violations report: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance report**. +1. On the left sidebar, select **Security and Compliance > Compliance report**. + +You can sort the compliance report on: -### Severity levels scale +- Severity level. +- Type of violation. +- Merge request title. -The following is a list of available violation severity levels, ranked from most to least severe: +Select a row to see details of the compliance violation. + +#### Severity levels + +Each compliance violation has one of the following severities. + +<!-- vale gitlab.SubstitutionWarning = NO --> | Icon | Severity level | |:----------------------------------------------|:---------------| @@ -46,75 +65,69 @@ The following is a list of available violation severity levels, ranked from most | **{severity-low, 18, gl-fill-orange-300}** | Low | | **{severity-info, 18, gl-fill-blue-400}** | Info | -### Violation types +<!-- vale gitlab.SubstitutionWarning = YES --> -The following is a list of violations that are either: +#### Violation types -- Already available. -- Aren't available, but which we are tracking in issues. +From [GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870), these are the available compliance violations. -| Violation | Severity level | Category | Description | Availability | -|:-------------------------------------|:----------------|:---------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| -| Author approved merge request | High | [Separation of duties](#separation-of-duties) | The author of the merge request approved their own merge request. For more information, see [Prevent approval by author](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Committers approved merge request | High | [Separation of duties](#separation-of-duties) | The committers of the merge request approved the merge request they contributed to. For more information, see [Prevent approvals by users who add commits](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Fewer than two approvals | High | [Separation of duties](#separation-of-duties) | The merge request was merged with fewer than two approvals. For more information, see [Merge request approval rules](../../project/merge_requests/approvals/rules.md). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Pipeline failed | Medium | [Pipeline results](../../../ci/pipelines/index.md) | The merge requests pipeline failed and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Pipeline passed with warnings | Info | [Pipeline results](../../../ci/pipelines/index.md) | The merge request pipeline passed with warnings and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down more than 10% | High | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of more than 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down less than 1% | Info | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of less than 1%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Violation | Severity level | Category | Description | +|:----------------------------------|:---------------|:----------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Author approved merge request | High | [Separation of duties](#separation-of-duties) | Author of the merge request approved their own merge request. For more information, see [Prevent approval by author](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author). | +| Committers approved merge request | High | [Separation of duties](#separation-of-duties) | Committers of the merge request approved the merge request they contributed to. For more information, see [Prevent approvals by users who add commits](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits). | +| Fewer than two approvals | High | [Separation of duties](#separation-of-duties) | Merge request was merged with fewer than two approvals. For more information, see [Merge request approval rules](../../project/merge_requests/approvals/rules.md). | -## Merge request drawer +The following are unavailable compliance violations that are tracked in [epic 5237](https://gitlab.com/groups/gitlab-org/-/epics/5237). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299357) in GitLab 14.1. +<!-- vale gitlab.SubstitutionWarning = NO --> -When you select a row, a drawer is shown that provides further details about the merge -request: +| Violation | Severity level | Category | Description | +|:-------------------------------------|:---------------|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------| +| Pipeline failed | Medium | [Pipeline results](../../../ci/pipelines/index.md) | Merge requests pipeline failed and was merged. | +| Pipeline passed with warnings | Info | [Pipeline results](../../../ci/pipelines/index.md) | Merge request pipeline passed with warnings and was merged. | +| Code coverage down more than 10% | High | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of more than 10%. | +| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. | +| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. | +| Code coverage down less than 1% | Info | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of less than 1%. | -- Project name and [compliance framework label](../../project/settings/index.md#add-a-compliance-framework-to-a-project), - if the project has one assigned. -- Link to the merge request. -- The merge request's branch path in the format `[source] into [target]`. -- A list of users that committed changes to the merge request. -- A list of users that commented on the merge request. -- A list of users that approved the merge request. -- The user that merged the merge request. +<!-- vale gitlab.SubstitutionWarning = YES --> -## Separation of duties +##### Separation of duties -We support a separation of duties policy between users who create and approve merge requests. -Our criteria for the separation of duties is as follows: +GitLab supports a separation of duties policy between users who create and approve merge requests. Our criteria for the +separation of duties is: -- [A merge request author is **not** allowed to approve their merge request](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author) -- [A merge request committer is **not** allowed to approve a merge request they have added commits to](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits) -- [The minimum number of approvals required to merge a merge request is **at least** two](../../project/merge_requests/approvals/rules.md) +- [A merge request author is **not** allowed to approve their merge request](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author). +- [A merge request committer is **not** allowed to approve a merge request they have added commits to](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits). +- [The minimum number of approvals required to merge a merge request is **at least** two](../../project/merge_requests/approvals/rules.md). -## Chain of Custody report +### Chain of Custody report > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213364) in GitLab 13.3. > - Chain of Custody reports sent using email [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342594) in GitLab 15.3 with a flag named `async_chain_of_custody_report`. Disabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/370100) in GitLab 15.5. Feature flag `async_chain_of_custody_report` removed. > - Chain of Custody report includes all commits (instead of just merge commits) [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267601) in GitLab 15.9 with a flag named `all_commits_compliance_report`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112092) in GitLab 15.9. Feature flag `all_commits_compliance_report` removed. -FLAG: -On self-managed GitLab, by default the Chain of Custody report only contains information on merge commits. To make the report contain information on all commits to projects within a group, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `all_commits_compliance_report`. On GitLab.com, this feature is not available. - -The Chain of Custody report allows customers to export a list of merge commits within the group. -The data provides a comprehensive view with respect to merge commits. It includes the merge commit SHA, -merge request author, merge request ID, merge user, date merged, pipeline ID, group name, project name, and merge request approvers. -Depending on the merge strategy, the merge commit SHA can be a merge commit, squash commit, or a diff head commit. - -With the `all_commits_compliance_report` flag enabled, the Chain of Custody report provides a 1 month trailing window of any commit into a project under the group. +The Chain of Custody report provides a 1 month trailing window of all commits to a project under the group. To generate the report for all commits, GitLab: 1. Fetches all projects under the group. -1. For each project, fetches the last 1 month of commits. Each project is capped at 1024 commits. If there are more than 1024 commits in the 1-month window, they - are truncated. -1. Writes the commits to a CSV file. The file is truncated at 15 MB because the report is emailed as an attachment. +1. For each project, fetches the last 1 month of commits. Each project is capped at 1024 commits. If there are more than + 1024 commits in the 1-month window, they are truncated. +1. Writes the commits to a CSV file. The file is truncated at 15 MB because the report is emailed as an attachment + (GitLab 15.5 and later). + +The report includes: + +- Commit SHA. +- Commit author. +- Committer. +- Date committed. +- Group. +- Project. -The expanded report includes the commit SHA, commit author, committer, date committed, the group, and the project. If the commit has a related merge commit, then the following are also included: - Merge commit SHA. @@ -124,39 +137,57 @@ If the commit has a related merge commit, then the following are also included: - Pipeline ID. - Merge request approvers. +#### Generate Chain of Custody report + To generate the Chain of Custody report: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance report**. +1. On the left sidebar, select **Security and Compliance > Compliance report**. 1. Select **List of all merge commits**. -The Chain of Custody report is either: +Depending on your version of GitLab, the Chain of Custody report is either sent through email or available for download. + +#### Generate commit-specific Chain of Custody report + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267629) in GitLab 13.6. +> - Support for including all commits instead of only merge commits [added](https://gitlab.com/gitlab-org/gitlab/-/issues/393446) in GitLab 15.10. + +You can generate a commit-specific Chain of Custody report for a given commit SHA. This report provides only the +details for the provided commit SHA. -- Available for download. -- Sent through email. Requires GitLab 15.5 and later. +To generate a commit-specific Chain of Custody report: -### Commit-specific Chain of Custody report +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Security and Compliance > Compliance report**. +1. At the top of the compliance report, to the right of **List of all commits**, select the down arrow + (**{chevron-lg-down}**). +1. Enter the commit SHA, and then select **Export commit custody report**. + +Depending on your version of GitLab, the Chain of Custody report is either sent through email or available for download. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267629) in GitLab 13.6. +Alternatively, use a direct link: `https://gitlab.com/groups/<group-name>/-/security/merge_commit_reports.csv?commit_sha={optional_commit_sha}`, +passing in an optional value to the `commit_sha` query parameter. -Authenticated group owners can generate a commit-specific Chain of Custody report for a given commit SHA, either: +## Compliance frameworks report -- Using the GitLab UI: +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387910) in GitLab 15.10. - 1. On the top bar, select **Main menu > Groups** and find your group. - 1. On the left sidebar, select **Security & Compliance > Compliance report**. - 1. At the top of the compliance report, to the right of **List of all merge commits**, select the down arrow (**{chevron-lg-down}**). - 1. Enter the merge commit SHA, and then select **Export commit custody report**. - SHA and then select **Export commit custody report**. +With compliance frameworks report, you can see the compliance frameworks that are applied to projects in a group. Each row of the report shows: -The Chain of Custody report is either: +- Project name. +- Project path. +- Compliance framework label if the project has one assigned. -- Available for download. -- Sent through email. Requires GitLab 15.5 and later. +The default framework for the group has a **default** badge. -- Using a direct link: `https://gitlab.com/groups/<group-name>/-/security/merge_commit_reports.csv?commit_sha={optional_commit_sha}`, passing in an optional value to the - `commit_sha` query parameter. +### View the compliance frameworks report for a group -NOTE: -The Chain of Custody report download is a CSV file, with a maximum size of 15 MB. -The remaining records are truncated when this limit is reached. +Prerequisites: + +- You must be an administrator or have the Owner role for the group. + +To view the compliance frameworks report: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Compliance report**. +1. On the page, select the **Frameworks** tab. diff --git a/doc/user/compliance/img/license_approval_policy_v15_9.png b/doc/user/compliance/img/license_approval_policy_v15_9.png Binary files differindex 43b1f89a07c..208643b47ae 100644 --- a/doc/user/compliance/img/license_approval_policy_v15_9.png +++ b/doc/user/compliance/img/license_approval_policy_v15_9.png diff --git a/doc/user/compliance/license_approval_policies.md b/doc/user/compliance/license_approval_policies.md index 32c90a1d317..2769732c49a 100644 --- a/doc/user/compliance/license_approval_policies.md +++ b/doc/user/compliance/license_approval_policies.md @@ -11,6 +11,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w License Approval Policies allow you to specify multiple types of criteria that define when approval is required before a merge request can be merged in. +NOTE: +License Approval Policies are applicable only to [protected](../project/protected_branches.md) target branches. + +The following video provides an overview of these policies. + +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/watch?v=34qBQ9t8qO8">Overview of GitLab License Approval Policies</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/34qBQ9t8qO8" frameborder="0" allowfullscreen> </iframe> +</figure> + ## Create a new license approval policy Create a license approval policy to enforce license compliance. diff --git a/doc/user/compliance/license_check_rules.md b/doc/user/compliance/license_check_rules.md index 968cf49ffdf..4280cfa0f5b 100644 --- a/doc/user/compliance/license_check_rules.md +++ b/doc/user/compliance/license_check_rules.md @@ -5,7 +5,7 @@ group: Security Policies info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# License Check Policies (DEPRECATED) **(ULTIMATE)** +# License Check Policies (deprecated) **(ULTIMATE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390417) in GitLab 15.9. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 43e88e89c18..55aa5b2b653 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -5,7 +5,7 @@ group: Composition Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# License compliance (DEPRECATED) **(ULTIMATE)** +# License compliance (deprecated) **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5483) in GitLab 11.0. > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387561) in GitLab 15.9. diff --git a/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md b/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md index 483c15d648c..0ad73f9939d 100644 --- a/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md +++ b/doc/user/compliance/license_scanning_of_cyclonedx_files/index.md @@ -16,12 +16,12 @@ To detect the licenses in use, License Compliance relies on running the [Dependency Scanning CI Jobs](../../application_security/dependency_scanning/index.md), and analyzing the [CycloneDX](https://cyclonedx.org/) Software Bill of Materials (SBOM) generated by those jobs. Other 3rd party scanners may also be used as long as they produce a CycloneDX file with a list of dependencies for [one of our supported languages](#supported-languages-and-package-managers). -This method of scanning is also capable of parsing and identifying over 500 different types of licenses -and can extract license information from packages that are dual-licensed or have multiple different licenses that apply. +This method of scanning is also capable of parsing and identifying over 500 different types of licenses, as defined in [the SPDX list](https://spdx.org/licenses/). +Licenses not in the SPDX list are reported as "Unknown". License information can also be extracted from packages that are dual-licensed, or have multiple different licenses that apply. To enable license detection using Dependency Scanning in a project, -include the `Jobs/Dependency-Scanning.yml` template in its CI configuration, -but do not include the `Jobs/License-Scanning.yml` template. +include the `Jobs/Dependency-Scanning.gitlab-ci.yml` template in its CI configuration, +but do not include the `Jobs/License-Scanning.gitlab-ci.yml` template. ## Requirements @@ -121,3 +121,23 @@ in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6571). ## Blocking merge requests based on detected licenses Users can require approval for merge requests based on the licenses that are detected by configuring a [license approval policy](../license_approval_policies.md). + +## Running in an offline environment + +For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, some adjustments are required to successfully scan +CycloneDX reports for licenses. For more information, see the offline [quick start guide](../../../topics/offline/quick_start_guide.md#enabling-the-package-metadata-database). + +## Troubleshooting + +### A CycloneDX file is not being scanned and appears to provide no results + +Ensure that the CycloneDX file adheres to the [CycloneDX JSON specification](https://cyclonedx.org/docs/latest/json). This specification does [not permit duplicate entries](https://cyclonedx.org/docs/latest/json/#components). Projects that contain multiple SBOM files should either report each SBOM file up as individual CI report artifacts or they should ensure that duplicates are removed if the SBOMs are merged as part of the CI pipeline. + +You can validate CycloneDX SBOM files against the `CycloneDX JSON specification` as follows: + +```shell +$ docker run -it --rm -v "$PWD:/my-cyclonedx-sboms" -w /my-cyclonedx-sboms cyclonedx/cyclonedx-cli:latest cyclonedx validate --input-version v1_4 --input-file gl-sbom-all.cdx.json + +Validating JSON BOM... +BOM validated successfully. +``` diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index ebacda506b4..54e87118361 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -1,6 +1,6 @@ --- -stage: Plan -group: Certify +stage: Monitor +group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/discussions/img/index_notes_filters.png b/doc/user/discussions/img/index_notes_filters.png Binary files differdeleted file mode 100644 index 977a3770c05..00000000000 --- a/doc/user/discussions/img/index_notes_filters.png +++ /dev/null diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 9c9b5301460..c71ea6576ef 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -36,6 +36,8 @@ You can create comments in places like: - Issues - Merge requests - Snippets +- Tasks +- OKRs Each object can have as many as 5,000 comments. @@ -46,12 +48,12 @@ instance with `@username` or `@groupname`. All mentioned users are notified with Users can change this setting for themselves in the [notification settings](../profile/notifications.md). You can quickly see which comments involve you, because -mentions for yourself (the user currently signed in) are highlighted +mentions for yourself (the user who is signed in) are highlighted in a different color. Avoid mentioning `@all` in issues and merge requests. It sends an email notification -to all members of that project's parent group, not only the participants of the project, -and may be interpreted as spam. +to all members of that project's parent group, not only the participants of the project. +It might be interpreted as spam. Notifications and mentions can be disabled in [a group's settings](../group/manage.md#disable-email-notifications). @@ -89,8 +91,8 @@ The comment is not displayed on your project's **Repository > Commits** page. NOTE: When your comment contains a reference to a commit included in the merge request, -it's automatically converted to a link in the context of the current merge request. -For example, `28719b171a056960dfdc0012b625d0b47b123196` becomes +it's converted to a link in the context of the merge request. +For example, `28719b171a056960dfdc0012b625d0b47b123196` becomes `28719b17` that links to `https://gitlab.example.com/example-group/example-project/-/merge_requests/12345/diffs?commit_id=28719b171a056960dfdc0012b625d0b47b123196`. ## Add a comment to a commit @@ -203,41 +205,35 @@ You can also mark an [issue as confidential](../project/issues/confidential_issu ## Show only comments -For issues and merge requests with many comments, you can filter the page to show comments only. +In discussions with many comments, filter the discussion to show only comments or history of +changes (system notes). System notes include changes to the description, mentions in other GitLab +objects, or changes to labels, assignees, and the milestone. +GitLab saves your preference, and applies it to every issue, merge request, or epic you view. 1. Open the **Overview** tab in a merge request, issue, or epic. -1. On the right side of the page, select from the filter: +1. On the right side of the page, from the **Sort or filter** dropdown list, select a filter: - **Show all activity**: Display all user comments and system notes. - (issue updates, mentions from other issues, changes to the description, and so on). - **Show comments only**: Display only user comments. - **Show history only**: Display only activity notes. - +## Change activity sort order -GitLab saves your preference, so it persists when you visit the same page again -from any device you're logged into. +Reverse the default order and interact with the activity feed sorted by most recent items +at the top. GitLab saves your preference in local storage and applies it to every issue, +merge request, or epic you view. -## View description change history **(PREMIUM)** +To change the activity sort order: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10103) in GitLab 12.6. +1. Open the **Overview** tab in a merge request, issue, or epic. +1. On the right side of the page, from the **Sort or filter** dropdown list, select the sort order + **Newest first** or **Oldest first** (default). + +## View description change history **(PREMIUM)** You can see changes to the description listed in the history. To compare the changes, select **Compare with previous version**. -## Change activity sort order - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14588) in GitLab 12.10. - -You can reverse the default order and interact with the activity feed sorted by most recent items -at the top. Your preference is saved in local storage and automatically applies to every issue, -merge request, or epic you view. - -To change the activity sort order: - -1. Select the **Oldest first** (or **Newest first**) dropdown list. -1. Select either oldest or newest items to be shown first. - ## Assign an issue to the commenting user > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191455) in GitLab 13.1. @@ -260,7 +256,7 @@ Prerequisites: To create a thread by replying to a comment: -1. In the upper right of the comment, select **Reply to comment** (**{comment}**). +1. In the upper-right corner of the comment, select **Reply to comment** (**{comment}**).  @@ -283,7 +279,7 @@ Prerequisites: To create a thread: 1. Enter a comment. -1. Below the comment, to the right of the **Comment** button, select the down arrow (**{chevron-down}**). +1. Below the comment, to the right of **Comment**, select the down arrow (**{chevron-down}**). 1. From the list, select **Start thread**. 1. Select **Start thread** again. @@ -308,7 +304,7 @@ To resolve a thread: 1. Go to the thread. 1. Do one of the following: - - In the upper right of the original comment, select **Resolve thread** (**{check-circle}**). + - In the upper-right corner of the original comment, select **Resolve thread** (**{check-circle}**). - Below the last reply, in the **Reply** field, select **Resolve thread**. - Below the last reply, in the **Reply** field, enter text, select the **Resolve thread** checkbox, and select **Add comment now**. diff --git a/doc/user/enterprise_user/index.md b/doc/user/enterprise_user/index.md index 3daee460956..b6a823c656f 100644 --- a/doc/user/enterprise_user/index.md +++ b/doc/user/enterprise_user/index.md @@ -31,6 +31,45 @@ Although a user can be a member of more than one group, each user account can be provisioned by only one group. As a result, a user is considered an enterprise user under one top-level group only. +## Verified domains for groups + +The following automated processes use [verified domains](../project/pages/custom_domains_ssl_tls_certification/index.md) to run: + +- [Bypass email confirmation for provisioned users](#bypass-email-confirmation-for-provisioned-users). + +### Set up a verified domain + +Prerequisites: + +- A project with [GitLab Pages](../project/pages/index.md), served under the default Pages domain `*.gitlab.io`. +- A custom domain name `example.com` or subdomain `subdomain.example.com`. +- Access to your domain's server control panel to set up a DNS `TXT` record to verify your domain's ownership. + +Setting up a verified domain is similar to [setting up a custom domain on GitLab Pages](../project/pages/custom_domains_ssl_tls_certification/index.md). However, you must: + +- Only configure the DNS `TXT` record to verify the domain's ownership. +- Ignore instructions for the `A`, `CNAME`, and `ALIAS` records. + +1. [Add a custom domain](../project/pages/custom_domains_ssl_tls_certification/index.md#1-add-a-custom-domain) for the matching email domain. + - The domain must match the email domain exactly. For example, if your email is `username@example.com`, verify the `example.com` domain. +1. [Get a verification code](../project/pages/custom_domains_ssl_tls_certification/index.md#2-get-the-verification-code). +1. [Set up the DNS `TXT`](../project/pages/custom_domains_ssl_tls_certification/index.md#3-set-up-dns-records) for your custom domain. +1. [Verify the domain's ownership](../project/pages/custom_domains_ssl_tls_certification/index.md#4-verify-the-domains-ownership). +1. Optional. [Add more domain aliases](../project/pages/custom_domains_ssl_tls_certification/index.md#add-more-domain-aliases). + +### View domains in group + +To view all configured domains in your group: + +1. On the top bar, select **Main menu > Groups** and find your top-level group. +1. On the left sidebar, select **Settings > Domain Verification**. + +You then see: + +- A list of added domains. +- The domains' status of **Verified** or **Unverified**. +- The project where the domain has been configured. + ## Manage enterprise users in a namespace A top-level Owner of a namespace on a paid plan can retrieve information about and diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index fb32c64f06c..4fb16d6ea27 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -125,20 +125,21 @@ Host gitlab.com ## GitLab Pages -Below are the settings for [GitLab Pages](https://about.gitlab.com/stages-devops-lifecycle/pages/). - -| Setting | GitLab.com | Default | -|---------------------------|------------------------|------------------------| -| Domain name | `gitlab.io` | - | -| IP address | `35.185.44.232` | - | -| Custom domains support | **{check-circle}** Yes | **{dotted-circle}** No | -| TLS certificates support | **{check-circle}** Yes | **{dotted-circle}** No | -| [Maximum size](../../administration/pages/index.md#set-global-maximum-size-of-each-gitlab-pages-site) (compressed) | 1 GB | 100 MB | - -The maximum size of your Pages site is also regulated by the artifacts maximum size, +Some settings for [GitLab Pages](../project/pages/index.md) differ from the +[defaults for self-managed instances](../../administration/pages/index.md): + +| Setting | GitLab.com | +|------------------------------|------------------------| +| Domain name | `gitlab.io` | +| IP address | `35.185.44.232` | +| Support for custom domains | **{check-circle}** Yes | +| Support for TLS certificates | **{check-circle}** Yes | +| Maximum site size | 1 GB | + +The maximum size of your Pages site depends on the maximum artifact size, which is part of [GitLab CI/CD](#gitlab-cicd). -There are also [rate limits set for GitLab Pages](#gitlabcom-specific-rate-limits). +[Rate limits](#gitlabcom-specific-rate-limits) also exist for GitLab Pages. ## GitLab CI/CD @@ -175,7 +176,7 @@ varies by format: | Generic | 5 GB | | Helm | 5 MB | | Maven | 5 GB | -| npm: | 5 GB | +| npm | 5 GB | | NuGet | 5 GB | | PyPI | 5 GB | | Terraform | 1 GB | @@ -247,7 +248,7 @@ The limit varies depending on your plan and the number of seats in your subscrip | Setting | Default for GitLab.com | |----------------------|-------------------------| -| Number of webhooks | `100` per project, `50` per group | +| Number of webhooks | `100` per project, `50` per group (subgroup webhooks are not counted towards parent group limits ) | | Maximum payload size | 25 MB | | Timeout | 10 seconds | @@ -263,73 +264,6 @@ Runner SaaS is the hosted, secure, and managed build environment you can use to For more information, see [Runner SaaS](../../ci/runners/index.md). -## Sidekiq - -GitLab.com runs [Sidekiq](https://sidekiq.org) with arguments `--timeout=4 --concurrency=4` -and the following environment variables: - -| Setting | GitLab.com | Default | -|----------------------------------------|------------|-----------| -| `GITLAB_MEMORY_WATCHDOG_ENABLED` | - | `true` | -| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | - | `2000000` | -| `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` | - | - | -| `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL` | - | `3` | -| `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` | - | `900` | -| `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT` | - | `30` | -| `SIDEKIQ_LOG_ARGUMENTS` | `1` | `1` | - -For more information, see how to -[configure the environment variables](../../administration/sidekiq/sidekiq_memory_killer.md). - -NOTE: -The `SIDEKIQ_MEMORY_KILLER_MAX_RSS` setting is `16000000` on Sidekiq import -nodes and Sidekiq export nodes. - -## PostgreSQL - -GitLab.com being a fairly large installation of GitLab means we have changed -various PostgreSQL settings to better suit our needs. For example, we use -streaming replication and servers in hot-standby mode to balance queries across -different database servers. - -The list of GitLab.com specific settings (and their defaults) is as follows: - -| Setting | GitLab.com | Default | -|:--------------------------------------|:--------------------------------------------------------------------|:--------------------------------------| -| `archive_command` | `/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p` | empty | -| `archive_mode` | on | off | -| `autovacuum_analyze_scale_factor` | 0.01 | 0.01 | -| `autovacuum_max_workers` | 6 | 3 | -| `autovacuum_vacuum_cost_limit` | 1000 | -1 | -| `autovacuum_vacuum_scale_factor` | 0.01 | 0.02 | -| `checkpoint_completion_target` | 0.7 | 0.9 | -| `checkpoint_segments` | 32 | 10 | -| `effective_cache_size` | 338688 MB | Based on how much memory is available | -| `hot_standby` | on | off | -| `hot_standby_feedback` | on | off | -| `log_autovacuum_min_duration` | 0 | -1 | -| `log_checkpoints` | on | off | -| `log_line_prefix` | `%t [%p]: [%l-1]` | empty | -| `log_min_duration_statement` | 1000 | -1 | -| `log_temp_files` | 0 | -1 | -| `maintenance_work_mem` | 2048 MB | 16 MB | -| `max_replication_slots` | 5 | 0 | -| `max_wal_senders` | 32 | 0 | -| `max_wal_size` | 5 GB | 1 GB | -| `shared_buffers` | 112896 MB | Based on how much memory is available | -| `shared_preload_libraries` | `pg_stat_statements` | empty | -| `shmall` | 30146560 | Based on the server's capabilities | -| `shmmax` | 123480309760 | Based on the server's capabilities | -| `wal_buffers` | 16 MB | -1 | -| `wal_keep_segments` | 512 | 10 | -| `wal_level` | replica | minimal | -| `statement_timeout` | 15 s | 60 s | -| `idle_in_transaction_session_timeout` | 60 s | 60 s | - -Some of these settings are in the process being adjusted. For example, the value -for `shared_buffers` is quite high, and we are -[considering adjusting it](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/4985). - ## Puma GitLab.com uses the default of 60 seconds for [Puma request timeouts](../../administration/operations/puma.md#change-the-worker-timeout). @@ -497,7 +431,8 @@ and can't be configured on GitLab.com to expire. You can erase job logs In addition to the GitLab Enterprise Edition Omnibus install, GitLab.com uses the following applications and settings to achieve scale. All settings are -publicly available at [chef cookbooks](https://gitlab.com/gitlab-cookbooks). +publicly available, as [Kubernetes configuration](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) +or [Chef cookbooks](https://gitlab.com/gitlab-cookbooks). ### Elastic cluster @@ -541,3 +476,10 @@ Service discovery: High Performance TCP/HTTP Load Balancer: - [`gitlab-cookbooks` / `gitlab-haproxy` · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy) + +## Sidekiq + +GitLab.com runs [Sidekiq](https://sidekiq.org) as an [external process](../../administration/sidekiq/index.md) +for Ruby job scheduling. + +The current settings are in the [GitLab.com Kubernetes pod configuration](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). diff --git a/doc/user/group/access_and_permissions.md b/doc/user/group/access_and_permissions.md index 4aecf016e56..0299f5c1a74 100644 --- a/doc/user/group/access_and_permissions.md +++ b/doc/user/group/access_and_permissions.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -90,8 +90,8 @@ To restrict group access by IP address: Keep in mind that restricting group access by IP address has the following implications: -- Administrators and group owners can access group settings from any IP address, regardless of IP restriction. However: - - Group owners can access the subgroups, but not the projects belonging to the group or subgroups, when accessing from a disallowed IP address. +- Administrators and group Owners can access group settings from any IP address, regardless of IP restriction. However: + - Group Owners can access the subgroups, but not the projects belonging to the group or subgroups, when accessing from a disallowed IP address. - Administrators can access projects belonging to the group when accessing from a disallowed IP address. Access to projects includes cloning code from them. - Users can still see group and project names and hierarchies. Only the following are restricted: @@ -181,12 +181,12 @@ prevent a project from being shared with other groups: 1. Select **Projects in `<group_name>` cannot be shared with other groups**. 1. Select **Save changes**. -This setting applies to all subgroups unless overridden by a group owner. Groups already +This setting applies to all subgroups unless overridden by a group Owner. Groups already added to a project lose access when the setting is enabled. ## Prevent users from requesting access to a group -As a group owner, you can prevent non-members from requesting access to +As a group Owner, you can prevent non-members from requesting access to your group. 1. On the top bar, **Main menu > Groups** and find your group. @@ -221,13 +221,13 @@ Existing forks are not removed. ## Prevent members from being added to projects in a group **(PREMIUM)** -As a group owner, you can prevent any new project membership for all +As a group Owner, you can prevent any new project membership for all projects in a group, allowing tighter control over project membership. For example, if you want to lock the group for an [Audit Event](../../administration/audit_events.md), you can guarantee that project membership cannot be modified during the audit. -If group membership lock is enabled, the group owner can still: +If group membership lock is enabled, the group Owner can still: - Invite groups or add members to groups to give them access to projects in the **locked** group. - Change the role of group members. @@ -286,7 +286,15 @@ To create group links via filter: LDAP user permissions can be manually overridden by an administrator. To override a user's permissions: 1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Group information > Members**. +1. On the left sidebar, select **Group information > Members**. If LDAP synchronization + has granted a user a role with: + - More permissions than the parent group membership, that user is displayed as having + [direct membership](../project/members/index.md#display-direct-members) of the group. + - The same or fewer permissions than the parent group membership, that user is displayed as having + [inherited membership](../project/members/index.md#display-inherited-members) of the group. +1. Optional. If the user you want to edit is displayed as having inherited membership, + [filter the subgroup to show direct members](manage.md#filter-a-group) before + overriding LDAP user permissions. 1. In the row for the user you are editing, select the pencil (**{pencil}**) icon. 1. Select **Edit permissions** in the modal. @@ -302,3 +310,17 @@ If a user sees a 404 when they would usually expect access, and the problem is l - `json.allowed`: `false` In viewing the log entries, compare `remote.ip` with the list of [allowed IP addresses](#restrict-group-access-by-ip-address) for the group. + +### Cannot update permissions for a group member + +If a group Owner cannot update permissions for a group member, check which memberships +are listed. Group Owners can only update direct memberships. + +If a parent group membership has the same or higher role than a subgroup, the +[inherited membership](../project/members/index.md#inherited-membership) is +listed on the subgroup members page, even if a [direct membership](../project/members/index.md#membership-types) +on the group exists. + +To view and update direct memberships, [filter the group to show direct members](manage.md#filter-a-group). + +The need to filter members by type through a redesigned members page that lists both direct and inherited memberships is proposed in [issue 337539](https://gitlab.com/gitlab-org/gitlab/-/issues/337539#note_1277786161). diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index cb760217487..a6dba3afacd 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -5,7 +5,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Group-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE)** +# Group-level Kubernetes clusters (certificate-based) (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in GitLab 11.6. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/group/compliance_frameworks.md b/doc/user/group/compliance_frameworks.md index 84cca5800c2..4ea474069cd 100644 --- a/doc/user/group/compliance_frameworks.md +++ b/doc/user/group/compliance_frameworks.md @@ -34,6 +34,10 @@ default framework cannot be deleted. A compliance framework that is set to default has a **default** label. +NOTE: +Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/394630), only group owners can apply the default compliance framework when creating +new projects or importing projects. + ### Set and remove as default > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375038) in GitLab 15.7. @@ -94,7 +98,8 @@ mutation { > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/331231) in GitLab 14.2. Group owners can configure a compliance pipeline in a project separate to other projects. By default, the compliance -pipeline configuration (`.gitlab-ci.yml` file) is run instead of the pipeline configuration of labeled projects. +pipeline configuration (for example, `.compliance-gitlab-ci.yml`) is run instead of the pipeline configuration (for example, `.gitlab-ci.yml`) of labeled +projects. However, the compliance pipeline configuration can reference the `.gitlab-ci.yml` file of the labeled projects so that: @@ -208,16 +213,47 @@ audit trail: - "# No after scripts." include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) - project: '$CI_PROJECT_PATH' - file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_SHA' # Must be defined or MR pipelines always use the use default branch - rules: - - if: $CI_PROJECT_PATH != "my-group/project-1" # Must be the hardcoded path to the project that hosts this configuration. + - project: '$CI_PROJECT_PATH' + file: '$CI_CONFIG_PATH' + ref: '$CI_COMMIT_SHA' # Must be defined or MR pipelines always use the use default branch + rules: + - if: $CI_PROJECT_PATH != "my-group/project-1" # Must be the hardcoded path to the project that hosts this configuration. ``` The `rules` configuration in the `include` definition avoids circular inclusion in case the compliance pipeline must be able to run in the host project itself. You can leave it out if your compliance pipeline only ever runs in labeled projects. +#### Compliance pipelines and custom pipeline configuration hosted externally + +The example above assumes that all projects host their pipeline configuration in the same project. +If any projects use [configuration hosted externally to the project](../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file): + +- The `include` section in the example compliance pipeline configuration must be adjusted. + For example, using [`include:rules`](../../ci/yaml/includes.md#use-rules-with-include): + + ```yaml + include: + # If the custom path variables are defined, include the project's external config file. + - project: '$PROTECTED_PIPELINE_CI_PROJECT_PATH' + file: '$PROTECTED_PIPELINE_CI_CONFIG_PATH' + ref: '$PROTECTED_PIPELINE_CI_REF' + rules: + - if: $PROTECTED_PIPELINE_CI_PROJECT_PATH && $PROTECTED_PIPELINE_CI_CONFIG_PATH && $PROTECTED_PIPELINE_CI_REF + # If any custom path variable is not defined, include the project's internal config file as normal. + - project: '$CI_PROJECT_PATH' + file: '$CI_CONFIG_PATH' + ref: '$CI_COMMIT_SHA' + rules: + - if: $PROTECTED_PIPELINE_CI_PROJECT_PATH == null || $PROTECTED_PIPELINE_CI_CONFIG_PATH == null || $PROTECTED_PIPELINE_CI_REF == null + ``` + +- [CI/CD variables](../../ci/variables/index.md) must be added to projects with external + pipeline configuration. In this example: + + - `PROTECTED_PIPELINE_CI_PROJECT_PATH`: The path to the project hosting the configuration file, for example `group/subgroup/project`. + - `PROTECTED_PIPELINE_CI_CONFIG_PATH`: The path to the configuration file in the project, for example `path/to/.gitlab-ci.yml`. + - `PROTECTED_PIPELINE_CI_REF`: The ref to use when retrieving the configuration file, for example `main`. + #### Compliance pipelines in merge requests originating in project forks When a merge request originates in a fork, the branch to be merged usually only exists in the fork. diff --git a/doc/user/group/contribution_analytics/img/group_stats_cal.png b/doc/user/group/contribution_analytics/img/group_stats_cal.png Binary files differdeleted file mode 100644 index 0238c7bf088..00000000000 --- a/doc/user/group/contribution_analytics/img/group_stats_cal.png +++ /dev/null diff --git a/doc/user/group/contribution_analytics/img/group_stats_table.png b/doc/user/group/contribution_analytics/img/group_stats_table.png Binary files differdeleted file mode 100644 index 1f58b9717d0..00000000000 --- a/doc/user/group/contribution_analytics/img/group_stats_table.png +++ /dev/null diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 2716db27037..d205e834e60 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -42,7 +42,7 @@ Projects in nested subgroups are not included in the template list. - Public and internal projects can be selected by any authenticated user as a template for a new project, if all [project features](../project/settings/index.md#configure-project-visibility-features-and-permissions) - except for **GitLab Pages** and **Security & Compliance** are set to **Everyone With Access**. + except for **GitLab Pages** and **Security and Compliance** are set to **Everyone With Access**. - Private projects can be selected only by users who are members of the projects. ## Example structure diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index 64addd524ad..6ec56d0d510 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -18,7 +18,7 @@ To view an epic board: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Epics > Boards**. - + ## Create an epic board @@ -30,7 +30,7 @@ To create a new epic board: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Epics > Boards**. -1. In the upper left corner, select the dropdown list with the current board name. +1. In the upper-left corner, select the dropdown list with the current board name. 1. Select **Create new board**. 1. Enter the new board's title. 1. Optional. To hide the Open or Closed lists, clear the **Show the Open list** and @@ -54,7 +54,7 @@ Prerequisites: To delete the active epic board: -1. Select the dropdown list with the current board name in the upper left corner of the epic boards page. +1. In the upper-left corner of the epic board page, select the dropdown list. 1. Select **Delete board**. 1. Select **Delete**. @@ -115,7 +115,7 @@ To create an epic from a list in epic board: 1. Enter the new epic's title. 1. Select **Create epic**. - + ### Filter epics @@ -216,3 +216,15 @@ To edit the scope of an epic board: - Show or hide the Open and Closed columns. - Select other labels as the board's scope. 1. Select **Save changes**. + +#### Display total weight on board lists + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364503) in GitLab 15.1 [with a flag](../../../administration/feature_flags.md) named `fe_epic_board_total_weight`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/364503) in GitLab 15.9. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/364503) in GitLab 15.10. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `fe_epic_board_total_weight`. +On GitLab.com, this feature is available. + +When enabled, this feature displays total weight of all epics at the top of each list. diff --git a/doc/user/group/epics/img/epic_board_epic_create_v14_1.png b/doc/user/group/epics/img/epic_board_epic_create_v14_1.png Binary files differdeleted file mode 100644 index 04017014885..00000000000 --- a/doc/user/group/epics/img/epic_board_epic_create_v14_1.png +++ /dev/null diff --git a/doc/user/group/epics/img/epic_board_epic_create_v15_10.png b/doc/user/group/epics/img/epic_board_epic_create_v15_10.png Binary files differnew file mode 100644 index 00000000000..71d1fc0a9fc --- /dev/null +++ b/doc/user/group/epics/img/epic_board_epic_create_v15_10.png diff --git a/doc/user/group/epics/img/epic_board_v14_1.png b/doc/user/group/epics/img/epic_board_v14_1.png Binary files differdeleted file mode 100644 index ccf1ef9559e..00000000000 --- a/doc/user/group/epics/img/epic_board_v14_1.png +++ /dev/null diff --git a/doc/user/group/epics/img/epic_board_v15_10.png b/doc/user/group/epics/img/epic_board_v15_10.png Binary files differnew file mode 100644 index 00000000000..d836abdbb59 --- /dev/null +++ b/doc/user/group/epics/img/epic_board_v15_10.png diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 74cfa2bd6ed..3a0fc86e803 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -41,8 +41,6 @@ The newly created epic opens. ### Start and due date inheritance -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7332) in GitLab 12.5 to replace **From milestones**. - If you select **Inherited**: - For the **start date**: GitLab scans all child epics and issues assigned to the epic, @@ -52,7 +50,7 @@ If you select **Inherited**: and sets the due date to match the latest due date found in the child epics or the milestone assigned to the issues. -These are dynamic dates and recalculated if any of the following occur: +These dates are dynamic and recalculated if any of the following occur: - A child epic's dates change. - Milestones are reassigned to an issue. @@ -123,8 +121,6 @@ To reorder list items, when viewing an epic: ## Bulk edit epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7250) in GitLab 12.2. - Users with at least the Reporter role can manage epics. When bulk editing epics in a group, you can edit their labels. @@ -233,7 +229,6 @@ than 1000. The cached value is rounded to thousands or millions and updated ever ## Filter the list of epics -> - Filtering by epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/195704) in GitLab 12.9. > - Filtering by child epics was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9029) in GitLab 13.0. > - Filtering by the user's reaction emoji [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325630) in GitLab 13.11. > - Sorting by epic titles [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.1. @@ -341,7 +336,7 @@ in relation to epics. ### View issues assigned to an epic -On the **Epics and Issues** tab, you can see epics and issues assigned to this epic. +On the **Child issues and epics** section, you can see epics and issues assigned to this epic. Only epics and issues that you can access show on the list. You can always view the issues assigned to the epic if they are in the group's child project. @@ -350,7 +345,7 @@ of its parent group. ### View count of issues in an epic -On the **Epics and Issues** tab, under each epic name, hover over the total counts. +On the **Child issues and epics** section, under each epic name, hover over the total counts. The number indicates all epics associated with the project, including issues you might not have permission to. @@ -365,7 +360,7 @@ automatically added to the epic. > Minimum required role for the project [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382506) from Reporter to Guest in GitLab 15.8. You can add existing issues to an epic, including issues in a project from a [different group hierarchy](index.md#child-issues-from-different-group-hierarchies). -Newly added issues appear at the top of the list of issues in the **Epics and Issues** tab. +Newly added issues appear at the top of the list of issues in the **Child issues and epics** section. An epic contains a list of issues and an issue can be associated with at most one epic. When you add a new issue that's already linked to an epic, the issue is automatically unlinked from its @@ -377,13 +372,13 @@ Prerequisites: To add an existing issue to an epic: -1. On the epic's page, under **Epics and Issues**, select **Add**. +1. On the epic's page, under **Child issues and epics**, select **Add**. 1. Select **Add an existing issue**. 1. Identify the issue to be added, using either of the following methods: - Paste the link of the issue. - Search for the desired issue by entering part of the issue's title, then selecting the desired - match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). Issues - from different group hierarchies do not appear in search results. To add such an issue, enter its full URL. + match. Issues from different group hierarchies do not appear in search results. + To add such an issue, enter its full URL. If there are multiple issues to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. @@ -401,7 +396,7 @@ Prerequisites: To create an issue from an epic: -1. On the epic's page, under **Epics and Issues**, select **Add**. +1. On the epic's page, under **Child issues and epics**, select **Add**. 1. Select **Add a new issue**. 1. Under **Title**, enter the title for the new issue. 1. From the **Project** dropdown list, select the project in which the issue should be created. @@ -430,10 +425,9 @@ To remove an issue from an epic: ### Reorder issues assigned to an epic -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9367) in GitLab 12.5. -> - Minimum required role for the project [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382506) from Reporter to Guest in GitLab 15.8. +> Minimum required role for the project [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382506) from Reporter to Guest in GitLab 15.8. -New issues appear at the top of the list in the **Epics and Issues** tab. +New issues appear at the top of the list in the **Child issues and epics** section. You can reorder the list of issues by dragging them. Prerequisites: @@ -442,7 +436,7 @@ Prerequisites: To reorder issues assigned to an epic: -1. Go to the **Epics and Issues** tab. +1. Go to the **Child issues and epics** section. 1. Drag issues into the desired order. ### Move issues between epics **(ULTIMATE)** @@ -450,7 +444,7 @@ To reorder issues assigned to an epic: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. > - Minimum required role for the project [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382506) from Reporter to Guest in GitLab 15.8. -New issues appear at the top of the list in the **Epics and Issues** +New issues appear at the top of the list in the **Child issues and epics** tab. You can move issues from one epic to another. Prerequisites: @@ -459,13 +453,12 @@ Prerequisites: To move an issue to another epic: -1. Go to the **Epics and Issues** tab. +1. Go to the **Child issues and epics** section. 1. Drag issues into the desired parent epic in the visible hierarchy. ### Promote an issue to an epic -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab 11.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. Prerequisites: @@ -504,12 +497,12 @@ You can create a spreadsheet template to manage a pattern of consistently repeat <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an introduction to epic templates, see [GitLab Epics and Epic Template Tip](https://www.youtube.com/watch?v=D74xKFNw8vg). -For more on epic templates, see [Epic Templates - Repeatable sets of issues](https://about.gitlab.com/handbook/marketing/strategic-marketing/getting-started/104/). +For more on epic templates, see [Epic Templates - Repeatable sets of issues](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/getting-started/104/). ## Multi-level child epics **(ULTIMATE)** You can add any epic that belongs to a group or subgroup of the parent epic's group. -New child epics appear at the top of the list of epics in the **Epics and Issues** tab. +New child epics appear at the top of the list of epics in the **Child issues and epics** section. When you add an epic that's already linked to a parent epic, the link to its current parent is removed. @@ -522,11 +515,7 @@ The maximum number of direct child epics is 100. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8502) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. Disabled by default. > - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. > - Cross-group child epics [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/375622) in GitLab 15.9. Enabled by default. - -FLAG: -On self-managed GitLab, by default this feature is available. To disable it, -ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `child_epics_from_different_hierarchies`. -On GitLab.com, this feature is available. +> - [Feature flag `child_epics_from_different_hierarchies`](https://gitlab.com/gitlab-org/gitlab/-/issues/382719) removed in GitLab 15.10. You can add a child epic that belongs to a group that is different from the parent epic's group. @@ -548,7 +537,7 @@ Prerequisites: To add a new epic as child epic: 1. In an epic, in the **Child issues and epics** section, select **Add > Add a new epic**. -1. Select a group from the dropdown. The epic's group is selected by default. +1. Select a group from the dropdown list. The epic's group is selected by default. 1. Enter a title for the new epic. 1. Select **Create epic**. @@ -557,7 +546,7 @@ To add an existing epic as child epic: 1. In an epic, in the **Child issues and epics** section, select **Add > Add an existing epic**. 1. Identify the epic to be added, using either of the following methods: - Paste the link of the epic. - - Search for the desired issue by entering part of the epic's title, then selecting the desired match. This search is only available for epics within the same group hierarchy. + - Search for the desired issue by entering part of the epic's title, then selecting the desired match. This search is only available for epics in the same group hierarchy. If there are multiple epics to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. @@ -567,7 +556,7 @@ To add an existing epic as child epic: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. > - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. -New child epics appear at the top of the list in the **Epics and Issues** tab. +New child epics appear at the top of the list in the **Child issues and epics** section. You can move child epics from one epic to another. When you add a new epic that's already linked to a parent epic, the link to its current parent is removed. Issues and child epics cannot be intermingled. @@ -578,15 +567,14 @@ Prerequisites: To move child epics to another epic: -1. Go to the **Epics and Issues** tab. +1. Go to the **Child issues and epics** section. 1. Drag epics into the desired parent epic. ### Reorder child epics assigned to an epic -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9367) in GitLab 12.5. -> - Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. +> Minimum required role for the group [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/382503) from Reporter to Guest in GitLab 15.7. -New child epics appear at the top of the list in the **Epics and Issues** tab. +New child epics appear at the top of the list in the **Child issues and epics** section. You can reorder the list of child epics. Prerequisites: @@ -595,7 +583,7 @@ Prerequisites: To reorder child epics assigned to an epic: -1. Go to the **Epics and Issues** tab. +1. Go to the **Child issues and epics** section. 1. Drag epics into the desired order. ### Remove a child epic from a parent epic diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index eb32856ff79..b0297076a43 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -27,42 +27,55 @@ If you migrate from GitLab.com to self-managed GitLab, an administrator can crea > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 for project resources [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. > - New application setting `bulk_import_enabled` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. `bulk_import` feature flag removed. +> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. -FLAG: On self-managed GitLab, by default [migrating group items](#migrated-group-items) is not available. To show the feature, ask an administrator to [enable it in application settings](../../admin_area/settings/visibility_and_access_controls.md#enable-migration-of-groups-and-projects-by-direct-transfer). -Also on self-managed GitLab, by default [migrating project items](#migrated-project-items-beta) is not available. To show -this feature, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named -`bulk_import_projects`. The feature is not ready for production use. On GitLab.com, migration of both groups and projects is available. Migrating groups by direct transfer copies the groups from one place to another. You can: - Copy many groups at once. -- Copy top-level groups to: +- In the GitLab UI, copy top-level groups to: - Another top-level group. - The subgroup of any existing top-level group. - Another GitLab instance, including GitLab.com. +- In the [API](../../../api/bulk_imports.md), copy top-level groups and subgroups to these locations. - Copy groups with projects (in [beta](../../../policy/alpha-beta-support.md#beta-features) and not ready for production use) or without projects. Copying projects with groups is available: - On GitLab.com by default. - - On self-managed GitLab instances after an administrator first [enables the feature flag](../../../administration/feature_flags.md) named `bulk_import_projects`. Not all group and project resources are copied. See list of copied resources below: - [Migrated group items](#migrated-group-items). - [Migrated project items](#migrated-project-items-beta). +WARNING: +Importing groups with projects is in [beta](../../../policy/alpha-beta-support.md#beta-features). This feature is not +ready for production use. + We invite you to leave your feedback about migrating by direct transfer in [the feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/284495). If you want to move groups instead of copying groups, you can [transfer groups](../manage.md#transfer-a-group) if the groups are in the same GitLab instance. Transferring groups is a faster and more complete option. -### Rate limit +### Known issues -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386452) in GitLab 15.9. +See [epic 6629](https://gitlab.com/groups/gitlab-org/-/epics/6629) for a list of known issues for migrating by direct +transfer. -Each user can perform up to six migrations per minute. +### Limits + +| Limit | Description | +|:------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 6 | Maximum number of migrations permitted by a destination GitLab instance per minute per user. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386452) in GitLab 15.9. | +| 5 GB | Maximum relation size that can be downloaded from the source instance. | +| 10 GB | Maximum size of a decompressed archive. | +| 210 seconds | Maximum number of seconds to wait for decompressing an archive file. | +| 50 MB | Maximum length an NDJSON row can have. | +| 5 minutes | Maximum number of seconds until an empty export status on source instance is raised. | +| 8 hours | Time until migration times out. | +| 90 minutes | Time the destination is waiting for export to complete. | ### Visibility rules @@ -100,16 +113,15 @@ To migrate groups by direct transfer: To ensure GitLab maps users and their contributions correctly: -1. Create the required users on the destination GitLab instance. When migrating to GitLab.com, you must create users - manually unless [SCIM](../../group/saml_sso/scim_setup.md) is used. Creating users with the API is only available to - self-managed instances because it requires administrator access. -1. Check that users have a public email on the source GitLab instance that matches their primary email on the - destination GitLab instance. -1. Ensure that users confirm their primary email addresses on the destination GitLab instance. Most users receive an - email asking them to confirm their email address. -1. If using an OmniAuth provider like SAML, link GitLab and SAML accounts of users on GitLab. All users on the - destination GitLab instance must sign in and verify their account on the destination GitLab instance. If using - [SAML SSO for GitLab.com groups](../../group/saml_sso/index.md), users must +1. Create the required users on the destination GitLab instance. You can create users with the API only on self-managed instances because it requires + administrator access. When migrating to GitLab.com or a self-managed GitLab instance you can: + - Create users manually. + - Set up or use your existing [SAML SSO provider](../saml_sso/index.md) and leverage user synchronization of SAML SSO groups supported through + [SCIM](../../group/saml_sso/scim_setup.md). You can + [bypass the GitLab user account verification with verified email domains](../saml_sso/index.md#bypass-user-email-confirmation-with-verified-domains). +1. Ensure that users have a public email on the source GitLab instance that matches any confirmed email address on the destination GitLab instance. Most + users receive an email asking them to confirm their email address. +1. If users already exist on the destination instance and you use [SAML SSO for GitLab.com groups](../../group/saml_sso/index.md), all users must [link their SAML identity to their GitLab.com account](../../group/saml_sso/index.md#linking-saml-to-your-existing-gitlabcom-account). ### Connect the source GitLab instance @@ -135,12 +147,15 @@ role. 1. By default, the proposed group namespaces match the names as they exist in source instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. 1. Next to the groups you want to import, select either: - - **Import with projects**. Importing groups with projects is in [Beta](../../../policy/alpha-beta-support.md#beta-features). This feature is not ready for production use. + - **Import with projects**. - **Import without projects**. - - **Import** on self-managed GitLab, when the `bulk_import_projects` feature flag is disabled and the feature is not available. 1. The **Status** column shows the import status of each group. If you leave the page open, it updates in real-time. 1. After a group has been imported, select its GitLab path to open its GitLab URL. +WARNING: +Importing groups with projects is in [beta](../../../policy/alpha-beta-support.md#beta-features). This feature is not +ready for production use. + ### Group import history You can view all groups migrated by you by direct transfer listed on the group import history page. This list includes: @@ -157,108 +172,154 @@ To view group import history: 1. On the top bar, select **Create new…** (**{plus-square}**). 1. Select **New group**. 1. Select **Import group**. -1. Select **History** in the upper right corner. +1. In the upper-right corner, select **History**. 1. If there are any errors for a particular import, you can see them by selecting **Details**. ### Migrated group items -The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) -file for groups lists many of the items imported when migrating groups by direct transfer. View this file in the branch -for your version of GitLab to see the list of items relevant to you. For example, -[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). +The group items that are migrated depend on the version of GitLab you use on the destination. To determine if a +specific group item is migrated: + +1. Check the [`groups/stage.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/bulk_imports/groups/stage.rb) + file for all editions and the + [`groups/stage.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/bulk_imports/groups/stage.rb) file + for Enterprise Edition for your version on the destination. For example, for version 15.9: + - <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/lib/bulk_imports/groups/stage.rb> (all editions). + - <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/ee/lib/ee/bulk_imports/groups/stage.rb> (Enterprise + Edition). +1. Check the + [`group/import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) + file for groups for your version on the destination. For example, for version 15.9: + <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/lib/gitlab/import_export/group/import_export.yml>. + +Any other group items are **not** migrated. Group items that are migrated to the destination GitLab instance include: -- Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292431) in 13.11) -- Board Lists -- Boards -- Epics ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250281) in 13.7) - - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) -- Finisher -- Group Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) in 13.9) -- Iterations ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292428) in 13.10) -- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96570) in 15.4) -- Members ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299415) in 13.9) - Group members are associated with the imported group if: - - The user already exists in the destination GitLab instance and - - The user has a public email in the source GitLab instance that matches a - confirmed email in the destination GitLab instance -- Milestones ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292427) in 13.10) -- Namespace Settings -- Releases - - Milestones ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339422) in GitLab 15.0). -- Subgroups -- Uploads - -Any other items are **not** migrated. +| Group item | Introduced in | +|:---------------------|:----------------------------------------------------------------------------| +| Badges | [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/292431) | +| Boards | [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18938) | +| Board lists | [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24863) | +| Epics <sup>1</sup> | [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/250281) | +| Group labels | [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) | +| Iterations | [GitLab 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/292428) | +| Iteration cadences | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96570) | +| Members <sup>2</sup> | [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/299415) | +| Group milestones | [GitLab 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/292427) | +| Namespace settings | [GitLab 14.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85128) | +| Release milestones | [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/339422) | +| Subgroups | [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18938) | +| Uploads | [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18938) | + +1. Epic resource state events [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4, label + associations [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62074) in GitLab 13.12, state and + state ID [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28203) in GitLab 13.7, and system note + metadata [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63551) in GitLab 14.0. +1. Group members are associated with the imported group if the user: + - Already exists in the destination GitLab instance. + - Has a public email in the source GitLab instance that matches a confirmed email in the destination GitLab instance. ### Migrated project items (beta) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. +> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. -FLAG: -On self-managed GitLab, migrating project resources when migrating groups is not available by default. -To make it available ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named -`bulk_import_projects`. On GitLab.com, groups are migrated with all their projects by default. +The project items that are migrated depends on the version of GitLab you use on the destination. To determine if a +specific project item is migrated: -The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) -file for projects lists many of the items imported when migrating projects using group migration. View this file in the branch -for your version of GitLab to see the list of items relevant to you. For example, -[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/project/import_export.yml). +1. Check the [`projects/stage.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/bulk_imports/projects/stage.rb) + file for all editions and the + [`projects/stage.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/bulk_imports/projects/stage.rb) + file for Enterprise Edition for your version on the destination. For example, for version 15.9: + - <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/lib/bulk_imports/projects/stage.rb> (all editions). + - <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/ee/lib/ee/bulk_imports/projects/stage.rb> (Enterprise + Edition). +1. Check the + [`project/import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) + file for projects for your version on the destination. For example, for version 15.9: + <https://gitlab.com/gitlab-org/gitlab/-/blob/15-9-stable-ee/lib/gitlab/import_export/project/import_export.yml>. + +Any other project items are **not** migrated. WARNING: -Migrating projects when migrating groups by direct transfer is in [Beta](../../../policy/alpha-beta-support.md#beta-features) +Migrating projects when migrating groups by direct transfer is in [beta](../../../policy/alpha-beta-support.md#beta-features) and is not ready for production use. Project items that are migrated to the destination GitLab instance include: -- Projects ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4) - - Auto DevOps ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339410) in GitLab 14.6) - - Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75029) in GitLab 14.6) - - Branches (including protected branches) ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339414) in GitLab 14.7) - - CI Pipelines ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339407) in GitLab 14.6) - - Designs ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339421) in GitLab 15.1) - - Issues ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) in GitLab 14.4) - - Issue iteration ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96184) in 15.4) - - Issue resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - - Issue resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - - Issue resource iteration events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - - Merge request URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) - - Time Tracking ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) in GitLab 14.4) - - Issue boards ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71661) in GitLab 14.4) - - Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339419) in GitLab 14.4) - - LFS Objects ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339405) in GitLab 14.8) - - Members ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341886) in GitLab 14.8) - - Merge Requests ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.5) - - Multiple merge request assignees ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) in GitLab 15.3) - - Merge request reviewers ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) in GitLab 15.3) - - Merge request approvers ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) in GitLab 15.3) - - Merge request resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - - Merge request resource milestone events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) - - Issue URL references ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) in GitLab 15.6) - - Time Tracking ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.5) - - Push Rules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) in GitLab 14.6) - - Milestones ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339417) in GitLab 14.5) - - External Pull Requests ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339409) in GitLab 14.5) - - Pipeline History ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339412) in GitLab 14.6) - - Pipeline Schedules ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339408) in GitLab 14.8) - - Project Features ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74722) in GitLab 14.6) - - Releases ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339422) in GitLab 15.1) - - Release Evidences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360567) in GitLab 15.1) - - Repositories ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4) - - Snippets ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343438) in GitLab 14.6) - - Settings ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339416) in GitLab 14.6) - - Avatar ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75249) in GitLab 14.6) - - Container Expiration Policy ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) in GitLab 14.6) - - Project Properties ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75898) in GitLab 14.6) - - Service Desk ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) in GitLab 14.6) - - Uploads ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339401) in GitLab 14.5) - - Wikis ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345923) in GitLab 14.6) - -Items excluded from migration, because they contain sensitive information: - -- Pipeline Triggers. +| Project item | Introduced in | +|:----------------------------------------|:---------------------------------------------------------------------------| +| Projects | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) | +| Auto DevOps | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339410) | +| Badges | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75029) | +| Branches (including protected branches) | [GitLab 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/339414) | +| CI Pipelines | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339407) | +| Designs | [GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/339421) | +| Issues | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) | +| Issue boards | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71661) | +| Labels | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/339419) | +| LFS Objects | [GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/339405) | +| Members | [GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/341886) | +| Merge requests | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) | +| Push rules | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) | +| Milestones | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339417) | +| External pull requests | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339409) | +| Pipeline history | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339412) | +| Pipeline schedules | [GitLab 14.8](https://gitlab.com/gitlab-org/gitlab/-/issues/339408) | +| Project features | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74722) | +| Releases | [GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/339422) | +| Release evidences | [GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/360567) | +| Repositories | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) | +| Snippets | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/343438) | +| Settings | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/339416) | +| Uploads | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339401) | +| Wikis | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/issues/345923) | + +#### Issue-related items + +Issue-related project items that are migrated to the destination GitLab instance include: + +| Issue-related project item | Introduced in | +|:--------------------------------|:---------------------------------------------------------------------------| +| Issue iterations | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96184) | +| Issue resource state events | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) | +| Issue resource milestone events | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) | +| Issue resource iteration events | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) | +| Merge request URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | +| Time tracking | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) | + +#### Merge request-related items + +Merge request-related project items that are migrated to the destination GitLab instance include: + +| Merge request-related project item | Introduced in | +|:----------------------------------------|:--------------------------------------------------------------------| +| Multiple merge request assignees | [GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) | +| Merge request reviewers | [GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) | +| Merge request approvers | [GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/issues/339520) | +| Merge request resource state events | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) | +| Merge request resource milestone events | [GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) | +| Issue URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | +| Time tracking | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) | + +#### Setting-related items + +Setting-related project items that are migrated to the destination GitLab instance include: + +| Setting-related project item | Introduced in | +|:-----------------------------|:---------------------------------------------------------------------------| +| Avatar | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75249) | +| Container expiration policy | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) | +| Project properties | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75898) | +| Service Desk | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) | + +#### Excluded items + +Project items excluded from migration because they contain sensitive information: + +- Pipeline triggers. ### Troubleshooting @@ -316,6 +377,18 @@ To solve this, you must change the source group path to include a non-numerical - The [Groups API](../../../api/groups.md#update-group). +#### Other `404` errors + +You can receive other `404` errors when importing a group, for example: + +```json +"exception_message": "Unsuccessful response 404 from [FILTERED] Bo...", +"exception_class": "BulkImports::NetworkError", +``` + +This error indicates a problem transferring from the _source_ instance. To solve this, check that you have met the [prerequisites](#prerequisites) on the source +instance. + ## Migrate groups by uploading an export file (deprecated) > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. @@ -323,8 +396,8 @@ To solve this, you must change the source group path to include a non-numerical WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by -[migrating groups by direct transfer](#migrate-groups-by-direct-transfer-recommended). To follow progress on a solution for -[offline environments](../../application_security/offline_deployments/index.md), see +[migrating groups by direct transfer](#migrate-groups-by-direct-transfer-recommended). However, this feature is still recommended for migrating groups between +offline systems. To follow progress on an alternative solution for [offline environments](../../application_security/offline_deployments/index.md), see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). Prerequisites: @@ -377,7 +450,7 @@ For example: The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch -for your version of GitLab to see the list of items relevant to you. For example, +for your version of GitLab to check which items can be imported to the destination GitLab instance. For example, [`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). Group items that are exported include: @@ -404,7 +477,7 @@ Items that are **not** exported include: - To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make sure these users exist before importing the desired groups. -- Users must set a public email in the source GitLab instance that matches one of their verified emails in the target GitLab instance. +- Users must set a public email in the source GitLab instance that matches their confirmed primary email in the destination GitLab instance. Most users receive an email asking them to confirm their email address. ### Enable export for a group diff --git a/doc/user/group/index.md b/doc/user/group/index.md index db01358d899..d35a0920cf2 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index d21dbe357da..622112d9735 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -155,7 +155,7 @@ You can sort members by **Account**, **Access granted**, **Max role**, or **Last 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Group information > Members**. -1. Above the list of members, in the upper right, from the **Account** list, select +1. Above the list of members, in the upper-right corner, from the **Account** list, select the criteria to filter by. 1. To switch the sort between ascending and descending, to the right of the **Account** list, select the arrow (**{sort-lowest}** or **{sort-highest}**). @@ -348,7 +348,7 @@ If you need to copy a group to a different GitLab instance, When transferring groups, note: - Changing a group's parent can have unintended side effects. See [what happens when a repository path changes](../project/repository/index.md#what-happens-when-a-repository-path-changes). -- You can only transfer groups to groups you manage. +- You must have the Owner role in the source and target group. - You must update your local repositories to point to the new location. - If the immediate parent group's visibility is lower than the group's current visibility, visibility levels for subgroups and projects change to match the new parent group's visibility. - Only explicit group membership is transferred, not inherited membership. If the group's owners have only inherited membership, this leaves the group without an owner. In this case, the user transferring the group becomes the group's owner. diff --git a/doc/user/group/reporting/git_abuse_rate_limit.md b/doc/user/group/reporting/git_abuse_rate_limit.md index 14b188e1204..d96f950edcd 100644 --- a/doc/user/group/reporting/git_abuse_rate_limit.md +++ b/doc/user/group/reporting/git_abuse_rate_limit.md @@ -11,13 +11,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `limit_unique_project_downloads_per_namespace_user`. On GitLab.com, this feature is available. -Git abuse rate limiting is a feature to automatically ban users who download or clone more than a specified number of repositories in a group or any of its subgroups within a given time frame. Banned users cannot access the main group or any of its non-public subgroups via HTTP or SSH. Access to unrelated groups is unaffected. +This is the group-level documentation. For self-managed instances, see the [administration documentation](../../admin_area/reporting/git_abuse_rate_limit.md). -If the `limit_unique_project_downloads_per_namespace_user` feature flag is enabled, all users with the Owner role for the main group receive an email when a user is about to be banned. +Git abuse rate limiting is a feature to automatically [ban users](#unban-a-user) who download or clone more than a specified number of repositories of a group in a given time frame. Banned users cannot access the top-level group or any of its non-public subgroups via HTTP or SSH. The rate limit also applies to users who authenticate with a [personal](../../../user/profile/personal_access_tokens.md) or [group access token](../../../user/group/settings/group_access_tokens.md). Access to unrelated groups is unaffected. -If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, users with the Owner role for the main group are still notified. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. +Git abuse rate limiting does not apply to top-level group owners, [deploy tokens](../../../user/project/deploy_tokens/index.md), or [deploy keys](../../../user/project/deploy_keys/index.md). -If automatic banning is enabled, users with the Owner role for the main group receive an email when a user is about to be banned, and the user is automatically banned from the group and its subgroups. +## Automatic ban notifications + +If the `limit_unique_project_downloads_per_namespace_user` feature flag is enabled, selected users receive an email when a user is about to be banned. + +If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, notifications are still sent. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. + +If automatic banning is enabled, an email notification is sent when a user is about to be banned, and the user is automatically banned from the group and its subgroups. ## Configure Git abuse rate limiting @@ -26,6 +32,7 @@ If automatic banning is enabled, users with the Owner role for the main group re 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400` (10 days). This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. 1. Optional. Exclude up to `100` users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Add up to `100` users to the **Send notifications to** field. You must select at least one user. All users with the Owner role for the main group are selected by default. 1. Optional. Turn on the **Automatically ban users from this namespace when they exceed the specified limits** toggle to enable automatic banning. 1. Select **Save changes**. diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 9971457f2ac..edb3e1b5079 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Insights +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/group/saml_sso/group_sync.md b/doc/user/group/saml_sso/group_sync.md index 27482893bd6..7c9b6effc43 100644 --- a/doc/user/group/saml_sso/group_sync.md +++ b/doc/user/group/saml_sso/group_sync.md @@ -22,10 +22,7 @@ For a demo of Group Sync using Azure, see [Demo: SAML Group Sync](https://youtu. ## Configure SAML Group Sync NOTE: -You must include the SAML configuration block on all Sidekiq nodes in addition to Rails application nodes if you: - -- Use SAML Group Sync. -- Have multiple GitLab nodes, for example in a distributed or highly available architecture. +You must include the SAML configuration block on all Sidekiq nodes in addition to Rails application nodes if you use SAML Group Sync and have multiple GitLab nodes, for example in a distributed or highly available architecture. NOTE: SAML Group Sync is only supported for the [SAML provider named `saml`](../../../integration/saml.md#configure-gitlab-to-use-multiple-saml-idps). @@ -107,6 +104,32 @@ Users granted: - A lower or the same role with Group Sync are displayed as having [inherited membership](../../project/members/index.md#display-inherited-members) of the group. +SAML group membership is evaluated each time a user signs in. + +### Global SAML group memberships lock **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386390) in GitLab 15.10. + +GitLab administrators can use the global SAML group memberships lock to prevent group members from inviting new members to subgroups that have their membership synchronized with SAML Group Links. + +Global group memberships lock only applies to subgroups of a top-level group where SAML Group Links synchronization is configured. No user can modify the +membership of a top-level group configured for SAML Group Links synchronization. + +When global group memberships lock is enabled: + +- Only an administrator can manage memberships of any group including access levels. +- Users cannot: + - Share a project with other groups. + - Invite members to a project created in a group. + +To enable global group memberships lock: + +1. [Configure SAML](../../../integration/saml.md) for your self-managed GitLab instance. +1. On the top bar, select **Main menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Ensure the **Lock memberships to SAML synchronization** checkbox is selected. + ### Automatic member removal After a group sync, users who are not members of a mapped SAML group are removed from the group. diff --git a/doc/user/group/saml_sso/img/Azure-manage-group-claims.png b/doc/user/group/saml_sso/img/Azure-manage-group-claims.png Binary files differindex 2ff24733282..b08b6ab4907 100644 --- a/doc/user/group/saml_sso/img/Azure-manage-group-claims.png +++ b/doc/user/group/saml_sso/img/Azure-manage-group-claims.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index e650b2dd130..04dfdbc6892 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -8,17 +8,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab 11.0. -This page describes SAML for groups. For instance-wide SAML on self-managed GitLab instances, see [SAML SSO for self-managed GitLab instances](../../../integration/saml.md). -[View the differences between SaaS and Self-Managed Authentication and Authorization Options](../../../administration/auth/index.md#saas-vs-self-managed-comparison). +Users can sign in to GitLab through their SAML identity provider. -SAML on GitLab.com allows users to sign in through their SAML identity provider. If the user is not already a member, the sign-in process automatically adds the user to the appropriate group. +[SCIM](scim_setup.md) synchronizes users with the group on GitLab.com. -User synchronization of SAML SSO groups is supported through [SCIM](scim_setup.md). SCIM supports adding and removing users from the GitLab group automatically. -For example, if you remove a user from the SCIM app, SCIM removes that same user from the GitLab group. +- When you add or remove a user from the SCIM app, SCIM adds or removes the user + from the GitLab group. +- If the user is not already a group member, the user is added to the group as part of the sign-in process. -SAML SSO is only configurable at the top-level group. - -If required, you can find [a glossary of common terms](../../../integration/saml.md#glossary-of-common-terms). +You can configure SAML SSO for the top-level group only. ## Configure your identity provider @@ -27,8 +25,8 @@ If required, you can find [a glossary of common terms](../../../integration/saml 1. On the left sidebar, select **Settings > SAML SSO**. 1. Note the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. 1. Configure your SAML identity provider app using the noted details. - Alternatively, GitLab provides a [metadata XML configuration](#metadata-configuration). - See [specific identity provider documentation](#providers) for more details. + Alternatively, GitLab provides a [metadata XML configuration](#set-up-identity-provider-using-metadata). + See [specific identity provider documentation](#set-up-identity-provider) for more details. 1. Configure the SAML response to include a [NameID](#nameid) that uniquely identifies each user. 1. Configure the required [user attributes](#user-attributes), ensuring you include the user's email address. 1. While the default is enabled for most SAML providers, ensure the app is set to have service provider @@ -40,6 +38,137 @@ If required, you can find [a glossary of common terms](../../../integration/saml If your account is the only owner in the group after SAML is set up, you can't unlink the account. To [unlink the account](#unlinking-accounts), set up another user as a group owner. +## Set up identity provider + +The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab. + +When [configuring your identity provider](#configure-your-identity-provider), consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. + +For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#configure-saml-on-your-idp) +for additional guidance on information your identity provider may require. + +GitLab provides the following information for guidance only. +If you have any questions on configuring the SAML app, contact your provider's support. + +### Set up Azure + +To set up SSO with Azure as your identification provider: + +1. In GitLab, on the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. +1. Note the information on this page. +1. Go to Azure and [follow the instructions for configuring SSO for an application](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/add-application-portal-setup-sso). The following GitLab settings correspond to the Azure fields. + + | GitLab setting | Azure field | + | -----------------------------------------| ---------------------------------------------- | + | **Identifier** | **Identifier (Entity ID)** | + | **Assertion consumer service URL** | **Reply URL (Assertion Consumer Service URL)** | + | **GitLab single sign-on URL** | **Sign on URL** | + | **Identity provider single sign-on URL** | **Login URL** | + | **Certificate fingerprint** | **Thumbprint** | + +1. You should set the following attributes: + - **Unique User Identifier (Name identifier)** to `user.objectID`. + - **nameid-format** to `persistent`. + - **Additional claims** to [supported attributes](#user-attributes). + +1. Optional. If you use [Group Sync](#group-sync), customize the name of the + group claim to match the required attribute. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +View a demo of [SCIM provisioning on Azure using SAML SSO for groups](https://youtu.be/24-ZxmTeEBU). The `objectID` mapping is outdated in this video. Follow the [SCIM documentation](scim_setup.md#configure-azure-active-directory) instead. + +View an [example configuration page](example_saml_config.md#azure-active-directory). + +### Set up Google Workspace + +1. [Set up SSO with Google as your identity provider](https://support.google.com/a/answer/6087519?hl=en). + The following GitLab settings correspond to the Google Workspace fields. + + | GitLab setting | Google Workspace field | + |:-------------------------------------|:-----------------------| + | Identifier | **Entity ID** | + | Assertion consumer service URL | **ACS URL** | + | GitLab single sign-on URL | **Start URL** | + | Identity provider single sign-on URL | **SSO URL** | + +1. Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint + required by GitLab to [configure SAML](#configure-gitlab): + 1. Download the certificate. + 1. Run this command: + + ```shell + openssl x509 -noout -fingerprint -sha1 -inform pem -in "GoogleIDPCertificate-domain.com.pem" + ``` + +1. Set these values: + - For **Primary email**: `email` + - For **First name**: `first_name` + - For **Last name**: `last_name` + - For **Name ID format**: `EMAIL` + - For **NameID**: `Basic Information > Primary email` + +On the GitLab SAML SSO page, when you select **Verify SAML Configuration**, disregard +the warning that recommends setting the **NameID** format to `persistent`. + +For details, see the [example configuration page](example_saml_config.md#google-workspace). + +### Set up Okta + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a demo of the Okta SAML setup including SCIM, see [Demo: Okta Group SAML & SCIM setup](https://youtu.be/0ES9HsZq0AQ). + +1. [Set up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/main/). + The following GitLab settings correspond to the Okta fields. + + | GitLab setting | Okta field | + | ------------------------------------ | ---------------------------------------------------------- | + | Identifier | **Audience URI** | + | Assertion consumer service URL | **Single sign-on URL** | + | GitLab single sign-on URL | **Login page URL** (under **Application Login Page** settings) | + | Identity provider single sign-on URL | **Identity Provider Single Sign-On URL** | + +1. Under the Okta **Single sign-on URL** field, select the **Use this for Recipient URL and Destination URL** checkbox. + +1. Set these values: + - For **Application username (NameID)**: **Custom** `user.getInternalProperty("id")` + - For **Name ID Format**: `Persistent` + +The Okta GitLab application available in the App Catalog only supports [SCIM](scim_setup.md). Support +for SAML is proposed in [issue 216173](https://gitlab.com/gitlab-org/gitlab/-/issues/216173). + +### Set up OneLogin + +OneLogin supports its own [GitLab (SaaS) application](https://onelogin.service-now.com/support?id=kb_article&sys_id=92e4160adbf16cd0ca1c400e0b961923&kb_category=50984e84db738300d5505eea4b961913). + +1. If you use the OneLogin generic + [SAML Test Connector (Advanced)](https://onelogin.service-now.com/support?id=kb_article&sys_id=b2c19353dbde7b8024c780c74b9619fb&kb_category=93e869b0db185340d5505eea4b961934), + you should [use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f). The following GitLab settings correspond + to the OneLogin fields: + + | GitLab setting | OneLogin field | + | ------------------------------------------------ | -------------------------------- | + | Identifier | **Audience** | + | Assertion consumer service URL | **Recipient** | + | Assertion consumer service URL | **ACS (Consumer) URL** | + | Assertion consumer service URL (escaped version) | **ACS (Consumer) URL Validator** | + | GitLab single sign-on URL | **Login URL** | + | Identity provider single sign-on URL | **SAML 2.0 Endpoint** | + +1. For **NameID**, use `OneLogin ID`. + +### Set up identity provider using metadata + +To configure some identity providers, you need a GitLab metadata URL. +To find this URL: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. +1. Copy the provided **GitLab metadata URL**. +1. Follow your identity provider's documentation and paste the metadata URL when it's requested. + +Check your identity provider's documentation to see if it supports the GitLab metadata URL. + ### NameID GitLab.com uses the SAML NameID to identify users. The NameID element: @@ -52,7 +181,7 @@ GitLab.com uses the SAML NameID to identify users. The NameID element: guarantee it doesn't ever change, for example, when a person's name changes. Email addresses are also case-insensitive, which can result in users being unable to sign in. -The relevant field name and recommended value for supported providers are in the [provider specific notes](#providers). +The relevant field name and recommended value for supported providers are in the [provider specific notes](#set-up-identity-provider). WARNING: Once users have signed into GitLab using the SSO SAML setup, changing the `NameID` breaks the configuration and potentially locks users out of the GitLab group. @@ -73,15 +202,6 @@ You can configure the following attributes with GitLab.com Group SAML: - `username` or `nickname`. We recommend you configure only one of these. - The [attributes available](../../../integration/saml.md#configure-assertions) to self-managed GitLab instances. -### Metadata configuration - -GitLab provides metadata XML that can be used to configure your identity provider. - -1. On the top bar, select **Main menu > Groups** and find your group. -1. On the left sidebar, select **Settings > SAML SSO**. -1. Copy the provided **GitLab metadata URL**. -1. Follow your identity provider's documentation and paste the metadata URL when it's requested. - ## Configure GitLab After you set up your identity provider to work with GitLab, you must configure GitLab to use it for authentication: @@ -122,38 +242,24 @@ It can also help to compare the XML response from your provider with our [exampl > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/339888) in GitLab 14.7 to not enforce SSO checks for Git activity originating from CI/CD jobs. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/215155) in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `transparent_sso_enforcement` to include transparent enforcement even when SSO enforcement is not enabled. Disabled on GitLab.com. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/375788) in GitLab 15.8 by enabling transparent SSO by default on GitLab.com. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/389562) in GitLab 15.10. Feature flag `transparent_sso_enforcement` removed. -FLAG: -On self-managed GitLab, transparent SSO enforcement is unavailable. An -[issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/382917) to add -transparent SSO enforcement to self-managed GitLab. -On GitLab.com, transparent SSO enforcement is available by default. To turn off -transparent SSO, ask a support or production team to enable the -`transparent_sso_enforcement_override` feature flag for a specific customer -group. - -#### Transparent SSO enforcement - -By default, transparent SSO enforcement is enabled in GitLab.com. This means SSO is enforced: - -- When users access groups and projects in the organization's - group hierarchy. Users can view other groups and projects without SSO sign in. -- For each user with an existing SAML identity. - -When transparent SSO enforcement is enabled, users: +On GitLab.com, SSO is enforced: -- Are not prompted to sign in through SSO on each visit. GitLab checks - whether a user has authenticated through SSO. If the user last signed in more - than 24 hours ago, GitLab prompts the user to sign in again through SSO. -- Without SAML identities are not required to use SSO unless **Enforce - SSO-only authentication for web activity for this group** is enabled. +- When SAML SSO is enabled. +- For users with an existing SAML identity when accessing groups and projects in the organization's + group hierarchy. Users can view other groups and projects as well as their user settings without SSO sign in by using their GitLab.com credentials. A user has a SAML identity if one or both of the following are true: - They have signed in to GitLab by using their GitLab group's single sign-on URL. - They were provisioned by SCIM. -With transparent SSO enabled, SSO is enforced as follows: +Users are not prompted to sign in through SSO on each visit. GitLab checks +whether a user has authenticated through SSO. If the user last signed in more +than 24 hours ago, GitLab prompts the user to sign in again through SSO. + +SSO is enforced as follows: | Project/Group visibility | Enforce SSO setting | Member with identity | Member without identity | Non-member or not signed in | |--------------------------|---------------------|--------------------| ------ |------------------------------| @@ -181,9 +287,6 @@ SSO enforcement for web activity has the following effects when enabled: - For groups, users cannot share a project in the group outside the top-level group, even if the project is forked. -- For Git activity over SSH and HTTPS, users must have at least one active - session signed-in through SSO before they can push to or - pull from a GitLab repository. - Git activity originating from CI/CD jobs do not have the SSO check enforced. - Credentials that are not tied to regular users (for example, project and group access tokens, and deploy keys) do not have the SSO check enforced. @@ -192,7 +295,9 @@ SSO enforcement for web activity has the following effects when enabled: - When the **Enforce SSO-only authentication for Git and Dependency Proxy activity for this group** option is enabled, any API endpoint that involves Git activity is under SSO enforcement. For example, creating or deleting a - branch, commit, or tag. + branch, commit, or tag. For Git activity over SSH and HTTPS, users must + have at least one active session signed-in through SSO before they can push to or + pull from a GitLab repository. When SSO for web activity is enforced, non-SSO group members do not lose access immediately. If the user: @@ -202,149 +307,40 @@ immediately. If the user: - Is signed out, they cannot access the group after being removed from the identity provider. -## Providers - -The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab. - -When [configuring your identity provider](#configure-your-identity-provider), consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. - -For providers not listed below, you can refer to the [instance SAML notes on configuring an identity provider](../../../integration/saml.md#configure-saml-on-your-idp) -for additional guidance on information your identity provider may require. - -GitLab provides the following information for guidance only. -If you have any questions on configuring the SAML app, contact your provider's support. - -### Set up Azure - -Follow the Azure documentation on [configuring single sign-on to applications](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/add-application-portal-setup-sso), and use the following notes when needed. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a demo of the Azure SAML setup including SCIM, see [SCIM Provisioning on Azure Using SAML SSO for Groups Demo](https://youtu.be/24-ZxmTeEBU). -The video is outdated in regard to objectID mapping and you should follow the [SCIM documentation](scim_setup.md#configure-azure-active-directory). - -| GitLab Setting | Azure Field | -| ------------------------------------ | ------------------------------------------ | -| Identifier | Identifier (Entity ID) | -| Assertion consumer service URL | Reply URL (Assertion Consumer Service URL) | -| GitLab single sign-on URL | Sign on URL | -| Identity provider single sign-on URL | Login URL | -| Certificate fingerprint | Thumbprint | - -You should set the following attributes: - -- **Unique User Identifier (Name identifier)** to `user.objectID`. -- **nameid-format** to persistent. -- Additional claims to [supported attributes](#user-attributes). - -If using [Group Sync](#group-sync), customize the name of the group claim to match the required attribute. - -See our [example configuration page](example_saml_config.md#azure-active-directory). - -### Set up Google Workspace - -1. [Set up SSO with Google as your identity provider](https://support.google.com/a/answer/6087519?hl=en). - The following GitLab settings correspond to the Google Workspace fields. - - | GitLab setting | Google Workspace field | - |:-------------------------------------|:-----------------------| - | Identifier | **Entity ID** | - | Assertion consumer service URL | **ACS URL** | - | GitLab single sign-on URL | **Start URL** | - | Identity provider single sign-on URL | **SSO URL** | - -1. Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint - required by GitLab to [configure SAML](#configure-gitlab): - 1. Download the certificate. - 1. Run this command: - - ```shell - openssl x509 -noout -fingerprint -sha1 -inform pem -in "GoogleIDPCertificate-domain.com.pem" - ``` - -1. Set these values: - - For **Primary email**: `email` - - For **First name**: `first_name` - - For **Last name**: `last_name` - - For **Name ID format**: `EMAIL` - - For **NameID**: `Basic Information > Primary email` - -On the GitLab SAML SSO page, when you select **Verify SAML Configuration**, disregard -the warning that recommends setting the **NameID** format to `persistent`. - -For details, see the [example configuration page](example_saml_config.md#google-workspace). - -### Set up Okta - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a demo of the Okta SAML setup including SCIM, see [Demo: Okta Group SAML & SCIM setup](https://youtu.be/0ES9HsZq0AQ). - -1. [Set up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/main/). - The following GitLab settings correspond to the Okta fields. - - | GitLab setting | Okta field | - | ------------------------------------ | ---------------------------------------------------------- | - | Identifier | **Audience URI** | - | Assertion consumer service URL | **Single sign-on URL** | - | GitLab single sign-on URL | **Login page URL** (under **Application Login Page** settings) | - | Identity provider single sign-on URL | **Identity Provider Single Sign-On URL** | - -1. Under the Okta **Single sign-on URL** field, select the **Use this for Recipient URL and Destination URL** checkbox. - -1. Set these values: - - For **Application username (NameID)**: **Custom** `user.getInternalProperty("id")` - - For **Name ID Format**: `Persistent` - -The Okta GitLab application available in the App Catalog only supports [SCIM](scim_setup.md). Support -for SAML is proposed in [issue 216173](https://gitlab.com/gitlab-org/gitlab/-/issues/216173). - -### Set up OneLogin - -OneLogin supports its own [GitLab (SaaS) application](https://onelogin.service-now.com/support?id=kb_article&sys_id=92e4160adbf16cd0ca1c400e0b961923&kb_category=50984e84db738300d5505eea4b961913). - -1. If you use the OneLogin generic - [SAML Test Connector (Advanced)](https://onelogin.service-now.com/support?id=kb_article&sys_id=b2c19353dbde7b8024c780c74b9619fb&kb_category=93e869b0db185340d5505eea4b961934), - you should [use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f). The following GitLab settings correspond - to the OneLogin fields: +### Change the SAML app - | GitLab setting | OneLogin field | - | ------------------------------------------------ | -------------------------------- | - | Identifier | **Audience** | - | Assertion consumer service URL | **Recipient** | - | Assertion consumer service URL | **ACS (Consumer) URL** | - | Assertion consumer service URL (escaped version) | **ACS (Consumer) URL Validator** | - | GitLab single sign-on URL | **Login URL** | - | Identity provider single sign-on URL | **SAML 2.0 Endpoint** | +After you have configured your identity provider, you can: -1. For **NameID**, use `OneLogin ID`. +- Change the identity provider users sign in with. +- Migrate to a different identity provider. +- Change email domains. -### Change the SAML app +### Change the identity provider -To change the SAML app used for sign in: +To change the identity provider: -- If the NameID is not identical in both the existing and new SAML apps, users must: - 1. [Unlink the current SAML identity](#unlinking-accounts). - 1. [Link their identity](#user-access-and-management) to the new SAML app. -- If the NameID is identical, no change is required. +- If the `NameID` is not identical in the existing and new identity providers, [change the NameID for users](#change-nameid-for-one-or-more-users). +- If the `NameID` is identical, users do not have to make any changes. -### Migrate to a different SAML provider +### Migrate to a different identity provider -You can migrate to a different SAML provider. During the migration process users will not be able to access any of the SAML groups. -To mitigate this, you can disable [SSO enforcement](#sso-enforcement). +You can migrate to a different identity provider. During the migration process, +users cannot access any of the SAML groups. To mitigate this, you can disable +[SSO enforcement](#sso-enforcement). -To migrate SAML providers: +To migrate identity providers: -1. [Configure](#configure-your-identity-provider) the group with the new identity provider SAML app. -1. Ask users to [unlink their account from the group](#unlinking-accounts). -1. Ask users to [link their account to the new SAML app](#linking-saml-to-your-existing-gitlabcom-account). +1. [Configure](#configure-your-identity-provider) the group with the new identity provider. +1. [Change the NameID for users](#change-nameid-for-one-or-more-users). ### Change email domains -To migrate users to a new email domain, users must: +To migrate users to a new email domain, tell users to: 1. Add their new email as the primary email to their accounts and verify it. -1. [Unlink their account from the group](#unlinking-accounts). -1. [Link their account to the group](#linking-saml-to-your-existing-gitlabcom-account). -1. (Optional) Remove their old email from the account. +1. Optional. Remove their old email from the account. + +If the NameID is configured with the email address, [change the NameID for users](#change-nameid-for-one-or-more-users). ## User access and management @@ -388,7 +384,8 @@ On subsequent visits, you should be able to go [sign in to GitLab.com with SAML] > Update of SAML identities using the SAML API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227841) in GitLab 15.5. -Group owners can update the SAML identities for their group members using the [SAML API](../../../api/saml.md). +Group owners can update the SAML identities for their group members using the [SAML API](../../../api/saml.md#update-extern_uid-field-for-a-saml-identity). +If [SCIM](scim_setup.md) is configured, group owners can update the SCIM identities using the [SCIM API](../../../api/scim.md#update-extern_uid-field-for-a-scim-identity). Alternatively, ask the users to reconnect their SAML account. @@ -446,7 +443,7 @@ convert the information to XML. An example SAML response is shown here. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238461) in GitLab 15.4. By default, users provisioned with SAML or SCIM are sent a verification email to verify their identity. Instead, you can -[configure GitLab with a custom domain](../../project/pages/custom_domains_ssl_tls_certification/index.md) and GitLab +[configure GitLab with a custom domain](../../enterprise_user/index.md#set-up-a-verified-domain) and GitLab automatically confirms user accounts. Users still receive an [enterprise user](../../enterprise_user/index.md) welcome email. Confirmation is bypassed for users: @@ -473,7 +470,7 @@ To rescind a user's access to the group when only SAML SSO is configured, either - Remove (in order) the user from: 1. The user data store on the identity provider or the list of users on the specific app. 1. The GitLab.com group. -- Use Group Sync at the top-level of your group to [automatically remove the user](group_sync.md#automatic-member-removal). +- Use [Group Sync](group_sync.md#automatic-member-removal) at the top-level of your group with the [default role](#role) set to [minimal access](../../permissions.md#users-with-minimal-access) to automatically block access to all resources within the group. Users may continue to [use a seat](../../permissions.md#minimal-access-users-take-license-seats). To rescind a user's access to the group when also using SCIM, refer to [Remove access](scim_setup.md#remove-access). @@ -507,6 +504,17 @@ For information on automatically managing GitLab group membership, see [SAML Gro The [Generated passwords for users created through integrated authentication](../../../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via SAML SSO for Groups. +## Related topics + +For more information on: + +- Setting up SAML on self-managed GitLab instances, see + [SAML SSO for self-managed GitLab instances](../../../integration/saml.md). +- Commonly-used terms, see the + [glossary of common terms](../../../integration/saml.md#glossary-of-common-terms). +- The differences between SaaS and self-managed authentication and authorization, + see the [SaaS vs. Self-Managed comparison](../../../administration/auth/index.md#saas-vs-self-managed-comparison). + ## Troubleshooting See our [troubleshooting SAML guide](troubleshooting.md). diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index c6ff5dc63c3..a61615104c1 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -139,7 +139,7 @@ Prerequisites: To configure Okta for SCIM: 1. Sign in to Okta. -1. Ensure you are in the Admin Area by selecting the **Admin** button located in the upper right. The button is not visible from the Admin Area. +1. In the upper-right corner, select **Admin**. The button is not visible from the Admin Area. 1. In the **Application** tab, select **Browse App Catalog**. 1. Search for **GitLab**, find and select the **GitLab** application. 1. On the GitLab application overview page, select **Add**. @@ -200,6 +200,10 @@ On subsequent visits, new and existing users can access groups either: For role information, see the [Group SAML](index.md#user-access-and-management) page. +### Passwords for users created through SCIM for GitLab groups + +GitLab requires passwords for all user accounts. For more information on how GitLab generates passwords for users created through SCIM for GitLab groups, see [generated passwords for users created through integrated authentication](../../../security/passwords_for_integrated_authentication_methods.md). + ### Link SCIM and SAML identities If [group SAML](index.md) is configured and you have an existing GitLab.com account, users can link their SCIM and SAML @@ -222,6 +226,8 @@ Remove or deactivate a user on the identity provider to remove their access to: After the identity provider performs a sync based on its configured schedule, the user's membership is revoked and they lose access. +When you enable SCIM, this does not automatically remove existing users who do not have a SAML identity. + NOTE: Deprovisioning does not delete the GitLab user account. diff --git a/doc/user/group/saml_sso/troubleshooting_scim.md b/doc/user/group/saml_sso/troubleshooting_scim.md index 939ed804a99..12144c7c080 100644 --- a/doc/user/group/saml_sso/troubleshooting_scim.md +++ b/doc/user/group/saml_sso/troubleshooting_scim.md @@ -79,7 +79,7 @@ You must not: When the SCIM app changes: -- Users can follow the instructions in the [Change the SAML app](index.md#change-the-saml-app) section. +- Users can follow the instructions in the [Change the SAML app](index.md#change-the-identity-provider) section. - Administrators of the identity provider can: 1. Remove users from the SCIM app, which unlinks all removed users. 1. Turn on sync for the new SCIM app to [link existing users](scim_setup.md#link-scim-and-saml-identities). diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index cd50c209b0d..07e0fb4b616 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -48,9 +48,6 @@ You cannot use group access tokens to create other group, project, or personal a Group access tokens inherit the [default prefix setting](../../admin_area/settings/account_and_limit_settings.md#personal-access-token-prefix) configured for personal access tokens. -NOTE: -Group access tokens are not FIPS compliant and creation and use are disabled when [FIPS mode](../../../development/fips_compliance.md) is enabled. - ## Create a group access token using UI > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214045) in GitLab 14.7. @@ -87,8 +84,9 @@ or API. However, administrators can use a workaround: # Set the group group you want to create a token for. For example, group with ID 109. group = Group.find(109) - # Create the group bot user. For further group access tokens, the username should be group_#{group.id}_bot#{bot_count}. For example, group_109_bot2 and email address group_109_bot2@example.com. - bot = Users::CreateService.new(admin, { name: 'group_token', username: "group_#{group.id}_bot", email: "group_#{group.id}_bot@example.com", user_type: :project_bot }).execute + # Create the group bot user. For further group access tokens, the username should be `group_{group_id}_bot_{random_string}` and email address `group_{group_id}_bot_{random_string}@noreply.{Gitlab.config.gitlab.host}`. + random_string = SecureRandom.hex(16) + bot = Users::CreateService.new(admin, { name: 'group_token', username: "group_#{group.id}_bot_#{random_string}", email: "group_#{group.id}_bot_#{random_string}@noreply.#{Gitlab.config.gitlab.host}", user_type: :project_bot }).execute # Confirm the group bot. bot.confirm @@ -172,7 +170,11 @@ to groups instead of projects. Bot users for groups: - Do not count as licensed seats. - Can have a maximum role of Owner for a group. For more information, see [Create a group access token](../../../api/group_access_tokens.md#create-a-group-access-token). -- Have a username set to `group_{group_id}_bot` for the first access token. For example, `group_123_bot`. -- Have an email set to `group{group_id}_bot@noreply.{Gitlab.config.gitlab.host}`. For example, `group123_bot@noreply.example.com`. +- Have a username set to `group_{group_id}_bot_{random_string}`. For example, `group_123_bot_4ffca233d8298ea1`. +- Have an email set to `group_{group_id}_bot_{random_string}@noreply.{Gitlab.config.gitlab.host}`. For example, `group_123_bot_4ffca233d8298ea1@noreply.example.com`. All other properties are similar to [bot users for projects](../../project/settings/project_access_tokens.md#bot-users-for-projects). + +## Token availability + +Group access tokens are only available in paid subscriptions, and not available in trial subscriptions. For more information, see the ["What is included" section of the GitLab Trial FAQ](https://about.gitlab.com/free-trial/#what-is-included-in-my-free-trial-what-is-excluded). diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md deleted file mode 100644 index ff64a7dcd54..00000000000 --- a/doc/user/group/settings/import_export.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../import/index.md' -remove_date: '2023-03-08' ---- - -This document was moved to [another location](../import/index.md). - -<!-- This redirect file can be deleted after <2023-03-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 6c7baa848e7..63ffbf62981 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -85,7 +85,7 @@ You cannot host a GitLab Pages subgroup website with a top-level domain name. Fo To create a subgroup: 1. On the top bar, select **Main menu > Groups** and find and select the parent group to add a subgroup to. -1. On the parent group's overview page, in the upper right, select **New subgroup**. +1. On the parent group's overview page, in the upper-right corner, select **New subgroup**. 1. Select **Create group**. 1. Fill in the fields. View a list of [reserved names](../../reserved_names.md) that cannot be used as group names. 1. Select **Create group**. diff --git a/doc/user/group/value_stream_analytics/img/object_hierarchy_example_V14_10.png b/doc/user/group/value_stream_analytics/img/object_hierarchy_example_V14_10.png Binary files differnew file mode 100644 index 00000000000..9c4d4ae7718 --- /dev/null +++ b/doc/user/group/value_stream_analytics/img/object_hierarchy_example_V14_10.png diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index c5a95779087..57d4fd513ff 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -7,7 +7,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Value stream analytics for groups **(PREMIUM)** -Value stream analytics provides metrics about each stage of your software development process. +Value stream analytics measures the time it takes to go from an idea to production. It also helps businesses: + +- Visualize their end-to-end DevSecOps workstreams. +- Identify and solve inefficiencies. +- Optimize their workstreams to deliver more value, faster. A **value stream** is the entire work process that delivers value to customers. For example, the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts @@ -23,6 +27,105 @@ Use value stream analytics to identify: Value stream analytics is also available for [projects](../../analytics/value_stream_analytics.md). +## How value stream analytics works + +### How value stream analytics aggregates data + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.5. +> - Filter by stop date toggle [added](https://gitlab.com/gitlab-org/gitlab/-/issues/352428) in GitLab 14.9 +> - Data refresh badge [added](https://gitlab.com/gitlab-org/gitlab/-/issues/341739) in GitLab 14.9 +> - Filter by stop date toggle [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84356) in GitLab 14.9 +> - Enable filtering by stop date [added](https://gitlab.com/gitlab-org/gitlab/-/issues/355000) in GitLab 15.0 + +Value stream analytics uses a backend process to collect and aggregate stage-level data, which +ensures it can scale for large groups with a high number of issues and merge requests. Due to this process, +there may be a slight delay between when an action is taken (for example, closing an issue) and when the data +displays on the value stream analytics page. + +It may take up to 10 minutes to process the data and display results. Data collection may take +longer than 10 minutes in the following cases: + +- If this is the first time you are viewing value stream analytics and have not yet [created a value stream](#create-a-value-stream-with-gitlab-default-stages). +- If the group hierarchy has been re-arranged. +- If there have been bulk updates on issues and merge requests. + +To view when the data was most recently updated, in the right corner next to **Edit**, hover over the **Last updated** badge. + +### How value stream analytics measures stages + +Value stream analytics measures each stage from its start event to its end event. + +For example, a stage might start when a user adds a label to an issue, and ends when they add another label. +Items aren't included in the stage time calculation if they have not reached the end event. + +Value stream analytics allows you to customize your stages based on pre-defined events. To make the +configuration easier, GitLab provides a pre-defined list of stages that can be used as a template + +Each pre-defined stages of value stream analytics is further described in the table below. + +| Stage | Measurement method | +| ------- | -------------------- | +| Issue | The median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whichever comes first. The label is tracked only if it already has an [issue board list](../../project/issue_board.md) created for it. | +| Plan | The median time between the action you took for the previous stage, and pushing the first commit to the branch. The first commit on the branch triggers the separation between **Plan** and **Code**. At least one of the commits in the branch must contain the related issue number (for example, `#42`). If none of the commits in the branch mention the related issue number, it is not considered in the measurement time of the stage. | +| Code | The median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../../project/issues/managing_issues.md#default-closing-pattern) in the description of the merge request. For example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request. If the closing pattern is not present, then the calculation uses the creation time of the first commit in the merge request as the start time. | +| Test | The median time to run the entire pipeline for that project. It's related to the time GitLab CI/CD takes to run every job for the commits pushed to that merge request. It is basically the start->finish time for all pipelines. | +| Review | The median time taken to review a merge request that has a closing issue pattern, between its creation and until it's merged. | +| Staging | The median time between merging a merge request that has a closing issue pattern until the very first deployment to a [production environment](#how-value-stream-analytics-identifies-the-production-environment). If there isn't a production environment, this is not tracked. | + +For information about how value stream analytics calculates each stage, see the [Value stream analytics development guide](../../../development/value_stream_analytics.md). + +#### Example workflow + +This example shows a workflow through all seven stages in one day. + +If a stage does not include a start and a stop time, its data is not included in the median time. +In this example, milestones have been created and CI/CD for testing and setting environments is configured. + +- 09:00: Create issue. **Issue** stage starts. +- 11:00: Add issue to a milestone, start work on the issue, and create a branch locally. + **Issue** stage stops and **Plan** stage starts. +- 12:00: Make the first commit. +- 12:30: Make the second commit to the branch that mentions the issue number. + **Plan** stage stops and **Code** stage starts. +- 14:00: Push branch and create a merge request that contains the + [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically). + **Code** stage stops and **Test** and **Review** stages start. +- GitLab CI/CD takes 5 minutes to run scripts defined in [`.gitlab-ci.yml`](../../../ci/yaml/index.md). +- 19:00: Merge the merge request. **Review** stage stops and **Staging** stage starts. +- 19:30: Deployment to the `production` environment finishes. **Staging** stops. + +Value stream analytics records the following times for each stage: + +- **Issue**: 09:00 to 11:00: 2 hrs +- **Plan**: 11:00 to 12:00: 1 hr +- **Code**: 12:00 to 14:00: 2 hrs +- **Test**: 5 minutes +- **Review**: 14:00 to 19:00: 5 hrs +- **Staging**: 19:00 to 19:30: 30 minutes + +Keep in mind the following observations related to this example: + +- This example demonstrates that it doesn't matter if your first + commit doesn't mention the issue number, you can do this later in any commit + on the branch you are working on. +- The **Test** stage is used in the calculation for the overall time of + the cycle. It is included in the **Review** process, as every MR should be + tested. +- This example illustrates only **one cycle** of the seven stages. The value stream analytics dashboard + shows the median time for multiple cycles. + +### How value stream analytics identifies the production environment + +Value stream analytics identifies [production environments](../../../ci/environments/index.md#deployment-tier-of-environments) by looking for project +[environments](../../../ci/yaml/index.md#environment) with a name matching any of these patterns: + +- `prod` or `prod/*` +- `production` or `production/*` + +These patterns are not case-sensitive. + +You can change the name of a project environment in your GitLab CI/CD configuration. + ## View value stream analytics > - Filtering [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13216) in GitLab 13.3 @@ -55,23 +158,21 @@ completed during the selected stage. The table shows a list of related workflow items for the selected stage. Based on the stage you select, this can be: -- CI/CD jobs - Issues - Merge requests -- Pipelines -## View DORA metrics and key metrics for a group +## Value stream analytics metrics > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323982) in GitLab 13.12. -The **Overview** dashboard in value stream analytics shows key metrics and DORA metrics of group performance. Based on the filter you select, +The **Overview** page in value stream analytics shows key metrics and DORA metrics of group performance. Based on the filter you select, the dashboard automatically aggregates DORA metrics and displays the current status of the value stream. Select a DORA metric to view its chart. Prerequisite: - To view deployment metrics, you must have a -[production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). +[production environment configured](#how-value-stream-analytics-identifies-the-production-environment). To view the DORA metrics and key metrics: @@ -86,16 +187,16 @@ To view the DORA metrics and key metrics: - In the **To** field, select an end date. Key metrics and DORA metrics display below the **Filter results** text box. -### Key metrics in the value stream +### Key metrics -The **Overview** dashboard shows the following key metrics that measure team performance: +The **Overview** page shows the following key metrics that measure team performance: -- Lead time: Median time from when the issue was created to when it was closed. -- Cycle time: Median time from first commit to issue closed. GitLab measures cycle time from the earliest commit of a +- **Lead time**: Median time from when the issue was created to when it was closed. +- **Cycle time**: Median time from first commit to issue closed. GitLab measures cycle time from the earliest commit of a [linked issue's merge request](../../project/issues/crosslinking_issues.md#from-commit-messages) to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation is always later than commit time. -- New issues: Number of new issues created. -- Deploys: Total number of deployments to production. +- **New issues**: Number of new issues created. +- **Deploys**: Total number of deployments to production. ### DORA metrics **(ULTIMATE)** @@ -118,35 +219,6 @@ NOTE: In GitLab 13.9 and later, deployment frequency metrics are calculated based on when the deployment was finished. In GitLab 13.8 and earlier, deployment frequency metrics are calculated based on when the deployment was created. -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=wQU-mWvNSiI">DORA metrics and value stream analytics</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/wQU-mWvNSiI" frameborder="0" allowfullscreen> </iframe> -</figure> - -### How value stream analytics aggregates data - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.5. -> - Filter by stop date toggle [added](https://gitlab.com/gitlab-org/gitlab/-/issues/352428) in GitLab 14.9 -> - Data refresh badge [added](https://gitlab.com/gitlab-org/gitlab/-/issues/341739) in GitLab 14.9 -> - Filter by stop date toggle [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84356) in GitLab 14.9 -> - Enable filtering by stop date [added](https://gitlab.com/gitlab-org/gitlab/-/issues/355000) in GitLab 15.0 - -Value stream analytics uses a backend process to collect and aggregate stage-level data, which -ensures it can scale for large groups with a high number of issues and merge requests. Due to this process, -there may be a slight delay between when an action is taken (for example, closing an issue) and when the data -displays on the value stream analytics page. - -It may take up to 10 minutes to process the data and display results. Data collection may take -longer than 10 minutes in the following cases: - -- If this is the first time you are viewing value stream analytics and have not yet [created a value stream](#create-a-value-stream-with-gitlab-default-stages). -- If the group hierarchy has been re-arranged. -- If there have been bulk updates on issues and merge requests. - -To view when the data was most recently updated, in the right corner next to **Edit**, hover over the **Last updated** badge. - ## View metrics for each development stage > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. @@ -171,82 +243,9 @@ NOTE: The date range selector filters items by the event time. The event time is when the selected stage finished for the given item. -## How value stream analytics measures stages - -Value stream analytics measures each stage from its start event to its end event. - -For example, a stage might start when a user adds a label to an issue, and ends when they add another label. -Items aren't included in the stage time calculation if they have not reached the end event. - -Value stream analytics allows you to customize your stages based on pre-defined events. To make the -configuration easier, GitLab provides a pre-defined list of stages that can be used as a template - -Each pre-defined stages of value stream analytics is further described in the table below. - -| Stage | Measurement method | -| ------- | -------------------- | -| Issue | The median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whichever comes first. The label is tracked only if it already has an [issue board list](../../project/issue_board.md) created for it. | -| Plan | The median time between the action you took for the previous stage, and pushing the first commit to the branch. The first commit on the branch triggers the separation between **Plan** and **Code**. At least one of the commits in the branch must contain the related issue number (for example, `#42`). If none of the commits in the branch mention the related issue number, it is not considered in the measurement time of the stage. | -| Code | The median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../../project/issues/managing_issues.md#default-closing-pattern) in the description of the merge request. For example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request. If the closing pattern is not present, then the calculation uses the creation time of the first commit in the merge request as the start time. | -| Test | The median time to run the entire pipeline for that project. It's related to the time GitLab CI/CD takes to run every job for the commits pushed to that merge request. It is basically the start->finish time for all pipelines. | -| Review | The median time taken to review a merge request that has a closing issue pattern, between its creation and until it's merged. | -| Staging | The median time between merging a merge request that has a closing issue pattern until the very first deployment to a [production environment](#how-value-stream-analytics-identifies-the-production-environment). If there isn't a production environment, this is not tracked. | - -For information about how value stream analytics calculates each stage, see the [Value stream analytics development guide](../../../development/value_stream_analytics.md). - -### Example workflow - -This example shows a workflow through all seven stages in one day. - -If a stage does not include a start and a stop time, its data is not included in the median time. -In this example, milestones have been created and CI/CD for testing and setting environments is configured. - -- 09:00: Create issue. **Issue** stage starts. -- 11:00: Add issue to a milestone, start work on the issue, and create a branch locally. - **Issue** stage stops and **Plan** stage starts. -- 12:00: Make the first commit. -- 12:30: Make the second commit to the branch that mentions the issue number. - **Plan** stage stops and **Code** stage starts. -- 14:00: Push branch and create a merge request that contains the - [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically). - **Code** stage stops and **Test** and **Review** stages start. -- GitLab CI/CD takes 5 minutes to run scripts defined in [`.gitlab-ci.yml`](../../../ci/yaml/index.md). -- 19:00: Merge the merge request. **Review** stage stops and **Staging** stage starts. -- 19:30: Deployment to the `production` environment finishes. **Staging** stops. - -Value stream analytics records the following times for each stage: - -- **Issue**: 09:00 to 11:00: 2 hrs -- **Plan**: 11:00 to 12:00: 1 hr -- **Code**: 12:00 to 14:00: 2 hrs -- **Test**: 5 minutes -- **Review**: 14:00 to 19:00: 5 hrs -- **Staging**: 19:00 to 19:30: 30 minutes - -Keep in mind the following observations related to this example: - -- This example demonstrates that it doesn't matter if your first - commit doesn't mention the issue number, you can do this later in any commit - on the branch you are working on. -- The **Test** stage is used in the calculation for the overall time of - the cycle. It is included in the **Review** process, as every MR should be - tested. -- This example illustrates only **one cycle** of the seven stages. The value stream analytics dashboard - shows the median time for multiple cycles. - -## How value stream analytics identifies the production environment - -Value stream analytics identifies production environments by looking for project -[environments](../../../ci/yaml/index.md#environment) with a name matching any of these patterns: - -- `prod` or `prod/*` -- `production` or `production/*` - -These patterns are not case-sensitive. +## Create a value stream -You can change the name of a project environment in your GitLab CI/CD configuration. - -## Create a value stream with GitLab default stages +### Create a value stream with GitLab default stages > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221202) in GitLab 13.3 @@ -269,7 +268,7 @@ create custom stages in addition to those provided in the default template. NOTE: If you have recently upgraded to GitLab Premium, it can take up to 30 minutes for data to collect and display. -## Create a value stream with custom stages +### Create a value stream with custom stages > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50229) in GitLab 13.7. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55572) in GitLab 13.10. @@ -287,7 +286,7 @@ When you create a value stream, you can create and add custom stages that align 1. To re-order the stages, select the up or down arrows. 1. Select **Create value stream**. -### Label-based stages for custom value streams +#### Label-based stages for custom value streams To measure complex workflows, you can use [scoped labels](../../project/labels.md#scoped-labels). For example, to measure deployment time from a staging environment to production, you could use the following labels: @@ -297,6 +296,14 @@ time from a staging environment to production, you could use the following label  +#### Example for custom value stream configuration + + + +In the example above, two independent value streams are set up for two teams that are using different development workflows in the **Test Group** (top-level namespace). + +The first value stream uses standard timestamp-based events for defining the stages. The second value stream uses label events. + ## Edit a value stream > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267537) in GitLab 13.10. @@ -305,7 +312,7 @@ After you create a value stream, you can customize it to suit your purposes. To 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value Stream**. -1. In the upper right, select the dropdown list, and select a value stream. +1. In the upper-right corner, select the dropdown list, then select a value stream. 1. Next to the value stream dropdown list, select **Edit**. 1. Optional: - Rename the value stream. @@ -324,7 +331,7 @@ To delete a custom value stream: 1. On the top bar, select **Main menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value Stream**. -1. In the upper right, select the dropdown list and then select the value stream you would like to delete. +1. In the upper-right corner, select the dropdown list, then select the value stream you would like to delete. 1. Select **Delete (name of value stream)**. 1. To confirm, select **Delete**. diff --git a/doc/user/img/observability_copy_shortened_link.png b/doc/user/img/observability_copy_shortened_link.png Binary files differnew file mode 100644 index 00000000000..217e56972cc --- /dev/null +++ b/doc/user/img/observability_copy_shortened_link.png diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index 9325f11b0c7..655bc774477 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -10,7 +10,7 @@ The [certificate-based Kubernetes integration with GitLab](../index.md) was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect your clusters, use the [GitLab agent](../../../clusters/agent/index.md). -## Cluster levels (DEPRECATED) +## Cluster levels (deprecated) > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md index df785cce406..569f876f36c 100644 --- a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md @@ -117,18 +117,18 @@ To remove all resources: 1. Add the following to your `.gitlab-ci.yml` file: - ```yaml - stages: - - init - - validate - - build - - deploy - - cleanup - - destroy: - extends: .destroy - needs: [] - ``` + ```yaml + stages: + - init + - validate + - build + - deploy + - cleanup + + destroy: + extends: .destroy + needs: [] + ``` 1. On the left sidebar, select **CI/CD > Pipelines** and select the most recent pipeline. 1. For the `destroy` job, select **Play** (**{play}**). diff --git a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md index cefa8885bfe..befd793c949 100644 --- a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md @@ -67,42 +67,42 @@ Set up your AWS credentials when you want to authenticate AWS with GitLab. 1. Create an [IAM User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) or [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). 1. Make sure that your IAM user or role has the appropriate permissions for your project. For this example project, you must have the permissions shown below. You can expand this when you set up your own project. - ```json - // IAM custom Policy definition - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": [ - "ec2:*", - "eks:*", - "elasticloadbalancing:*", - "autoscaling:*", - "cloudwatch:*", - "logs:*", - "kms:DescribeKey", - "iam:AddRoleToInstanceProfile", - "iam:AttachRolePolicy", - "iam:CreateInstanceProfile", - "iam:CreateRole", - "iam:CreateServiceLinkedRole", - "iam:GetRole", - "iam:ListAttachedRolePolicies", - "iam:ListRolePolicies", - "iam:ListRoles", - "iam:PassRole", - // required for destroy step - "iam:DetachRolePolicy", - "iam:ListInstanceProfilesForRole", - "iam:DeleteRole" - ], - "Resource": "*" - } - ] - } - ``` + ```json + // IAM custom Policy definition + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "ec2:*", + "eks:*", + "elasticloadbalancing:*", + "autoscaling:*", + "cloudwatch:*", + "logs:*", + "kms:DescribeKey", + "iam:AddRoleToInstanceProfile", + "iam:AttachRolePolicy", + "iam:CreateInstanceProfile", + "iam:CreateRole", + "iam:CreateServiceLinkedRole", + "iam:GetRole", + "iam:ListAttachedRolePolicies", + "iam:ListRolePolicies", + "iam:ListRoles", + "iam:PassRole", + // required for destroy step + "iam:DetachRolePolicy", + "iam:ListInstanceProfilesForRole", + "iam:DeleteRole" + ], + "Resource": "*" + } + ] + } + ``` 1. [Create an access key for the user or role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). 1. Save your access key and secret. You need these to authenticate AWS with GitLab. @@ -165,19 +165,19 @@ To remove all resources: 1. Add the following to your `.gitlab-ci.yml` file: - ```yaml - stages: - - init - - validate - - test - - build - - deploy - - cleanup - - destroy: - extends: .terraform:destroy - needs: [] - ``` + ```yaml + stages: + - init + - validate + - test + - build + - deploy + - cleanup + + destroy: + extends: .terraform:destroy + needs: [] + ``` 1. On the left sidebar, select **CI/CD > Pipelines** and select the most recent pipeline. 1. For the `destroy` job, select **Play** (**{play}**). diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 7e6a0495d90..ecfd086d254 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -143,19 +143,19 @@ To remove all resources: 1. Add the following to your `.gitlab-ci.yml` file: - ```yaml - stages: - - init - - validate - - build - - test - - deploy - - cleanup - - destroy: - extends: .terraform:destroy - needs: [] - ``` + ```yaml + stages: + - init + - validate + - build + - test + - deploy + - cleanup + + destroy: + extends: .terraform:destroy + needs: [] + ``` 1. On the left sidebar, select **CI/CD > Pipelines** and select the most recent pipeline. 1. For the `destroy` job, select **Play** (**{play}**). diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md index 14fd4917141..9f56b92dabc 100644 --- a/doc/user/infrastructure/clusters/index.md +++ b/doc/user/infrastructure/clusters/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To connect clusters to GitLab, use the [GitLab agent](../../clusters/agent/index.md). -## Certificate-based Kubernetes integration (DEPRECATED) +## Certificate-based Kubernetes integration (deprecated) WARNING: In GitLab 14.5, the certificate-based method to connect Kubernetes clusters diff --git a/doc/user/infrastructure/clusters/manage/clusters_health.md b/doc/user/infrastructure/clusters/manage/clusters_health.md index 4f63f3e4e2a..f522cb5bff6 100644 --- a/doc/user/infrastructure/clusters/manage/clusters_health.md +++ b/doc/user/infrastructure/clusters/manage/clusters_health.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Clusters health (DEPRECATED) **(FREE)** +# Clusters health (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in GitLab 10.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) from GitLab Ultimate to GitLab Free in 13.2. diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index 93a82023480..62772c5b379 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -42,7 +42,7 @@ For self-managed GitLab, before you can use GitLab for your Terraform state file - An administrator must [set up Terraform state storage](../../../administration/terraform_state.md). - You must enable the **Infrastructure** menu for your project. Go to **Settings > General**, - expand **Visibility, project features, permissions**, and under **Operations**, turn on the toggle. + expand **Visibility, project features, permissions**, and under **Infrastructure**, turn on the toggle. ## Initialize a Terraform state as a backend by using GitLab CI/CD @@ -58,7 +58,7 @@ WARNING: Like any other job artifact, Terraform plan data is viewable by anyone with the Guest role on the repository. Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform plan includes sensitive data, like passwords, access tokens, or certificates, you should -encrypt plan output or modify the project visibility settings. We also strongly recommend that you **disable** +encrypt plan output or modify the project visibility settings. We also strongly recommend that you **disable** [public pipelines](../../../ci/pipelines/settings.md#change-pipeline-visibility-for-non-project-members-in-public-projects) by setting the artifact's public flag to false (`public: false`). This setting ensures artifacts are accessible only to GitLab Administrators and project members with the Reporter role and above. diff --git a/doc/user/infrastructure/iac/terraform_template_recipes.md b/doc/user/infrastructure/iac/terraform_template_recipes.md index 89a97f305e4..a79f2758441 100644 --- a/doc/user/infrastructure/iac/terraform_template_recipes.md +++ b/doc/user/infrastructure/iac/terraform_template_recipes.md @@ -36,7 +36,7 @@ include: - template: Terraform.latest.gitlab-ci.yml deploy: - envrionment: + environment: name: $TF_STATE_NAME action: start on_stop: destroy @@ -65,7 +65,7 @@ build: - when: on_success deploy: - envrionment: + environment: name: $TF_STATE_NAME action: start on_stop: destroy diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 88430df0d67..dc5d402d04b 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Instance-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE SELF)** +# Instance-level Kubernetes clusters (certificate-based) (deprecated) **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index eaceeccc148..30432b8b492 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -124,8 +124,13 @@ to see the color chips next to the color code: ### Diagrams and flowcharts -You can generate diagrams and flowcharts from text by using [Mermaid](https://mermaidjs.github.io/) or [PlantUML](https://plantuml.com). -You can also use [Kroki](https://kroki.io) to create a wide variety of diagrams. +You can generate diagrams from text by using: + +- [Mermaid](https://mermaidjs.github.io/) +- [PlantUML](https://plantuml.com) +- [Kroki](https://kroki.io) to create a wide variety of diagrams. + +In wikis, you can also add and edit diagrams created with the [diagrams.net editor](#diagramsnet-editor). #### Mermaid @@ -543,6 +548,58 @@ This example links to `<wiki_root>/miscellaneous.md`: [Link to Related Page](/miscellaneous.md) ``` +#### diagrams.net editor + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322174) in GitLab 15.10. + +NOTE: +Use of the diagrams.net editor is not available on offline environments. + +In wikis, you can use the [diagrams.net](https://www.diagrams.net/) editor to create diagrams. You +can also edit diagrams created with the diagrams.net editor. The diagram editor is available in both +the Markdown editor and the content editor. + +##### Markdown editor + +To create a diagram in the Markdown editor: + +1. In the editor's toolbar, select **Insert or edit diagram** (**{diagram}**). +1. Use the diagrams.net editor to create the diagram. +1. Select **Save & exit**. + +A Markdown image reference to the diagram is inserted in the wiki content. + +To edit a diagram in the Markdown editor: + +1. Place the Markdown editor's text field cursor in a Markdown image reference +that contains the diagram. +1. Select **Insert or edit diagram** (**{diagram}**) in the Markdown editor. +1. Use the diagrams.net editor to edit the diagram. +1. Select **Save & exit**. + +A Markdown image reference to the diagram is inserted in the wiki content, +replacing the previous diagram. + +##### Content editor + +To create a diagram in the content editor: + +1. In the editor's toolbar, select **More options** (**{plus}**). +1. In the dropdown list, select **Create or edit diagram**. +1. Use the diagrams.net editor to create the diagram. +1. Select **Save & exit**. + +The diagram as visualized in the diagrams.net editor is inserted in the wiki content. + +To edit a diagram in the content editor: + +1. Select the diagram that you want to edit. +1. In the floating toolbar, select **Edit diagram** (**{diagram}**). +1. Use the diagrams.net editor to edit the diagram. +1. Select **Save & exit**. + +The selected diagram is replaced with an updated version. + ## GitLab-specific references GitLab Flavored Markdown renders GitLab-specific references. For example, you can reference @@ -605,11 +662,34 @@ at the end of the reference. For example, a reference like `#123+` is rendered a URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+` are also expanded. -### Embedding metrics in GitLab Flavored Markdown +### Show the issue or merge request summary in the reference + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386937) in GitLab 15.10. + +To include an extended summary in the rendered link of an issue or merge request, add a `+s` +at the end of the reference. Summary includes information about **assignees**, **milestone** +and **health status** of referenced item. + +For example, a reference like `#123+s` is rendered as +`The issue title (#123) • First Assignee, Second Assignee+ • v15.10 • Needs attention`. + +URL references like `https://gitlab.com/gitlab-org/gitlab/-/issues/1234+s` are also expanded. + +### Embedding metrics Metric charts can be embedded in GitLab Flavored Markdown. Read [Embedding Metrics in GitLab flavored Markdown](../operations/metrics/embed.md) for more details. +### Embedding Observability dashboards + +You can embed GitLab Observability UI dashboards descriptions and comments, for example in epics, issues, and MRs. + +To embed an Observability dashboard URL: + +1. In GitLab Observability UI, copy the URL in the address bar. + +1. Paste your link wherever you want to embed your dashboard. GitLab Flavored Markdown recognizes the URL and displays the source. + ## Features extended from standard Markdown All standard Markdown formatting should work as expected in GitLab. Some standard diff --git a/doc/user/namespace/index.md b/doc/user/namespace/index.md index 7463b8f1099..00eb63da3ff 100644 --- a/doc/user/namespace/index.md +++ b/doc/user/namespace/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/okrs.md b/doc/user/okrs.md index 0d3be8474fe..7025dc916ac 100644 --- a/doc/user/okrs.md +++ b/doc/user/okrs.md @@ -17,12 +17,11 @@ On self-managed GitLab, by default this feature is not available. To make it ava On GitLab.com, this feature is not available. The feature is not ready for production use. -Use objectives and key results to align your workforce towards common goals and track the progress. -Set a big goal with an objective and use [child objectives and key results](#child-objectives-and-key-results) -to measure the big goal's completion. +[Objectives and key results](https:://en.wikipedia.org/wiki/OKR) (OKRs) are a framework for setting +and tracking goals that are aligned with your organization's overall strategy and vision. The objective and the key result in GitLab share many features. In the documentation, the term -**OKR** refers to either an objective or a key result. +**OKRs** refers to both objectives and key results. OKRs are a type of work item, a step towards [default issue types](https://gitlab.com/gitlab-org/gitlab/-/issues/323404) in GitLab. @@ -31,6 +30,29 @@ to work items and adding custom work item types, see [epic 6033](https://gitlab.com/groups/gitlab-org/-/epics/6033) or the [Plan direction page](https://about.gitlab.com/direction/plan/). +## Designing effective OKRs + +Use objectives and key results to align your workforce towards common goals and track the progress. +Set a big goal with an objective and use [child objectives and key results](#child-objectives-and-key-results) +to measure the big goal's completion. + +**Objectives** are aspirational goals to be achieved and define **what you're aiming to do**. +They show how an individual's, team's, or department's work impacts overall direction of the +organization by connecting their work to overall company strategy. + +**Key results** are measures of progress against aligned objectives. They express +**how you know if you have reached your goal** (objective). +By achieving a specific outcome (key result), you create progress for the linked objective. + +To know if your OKR makes sense, you can use this sentence: + +<!-- vale gitlab.FutureTense = NO --> +> I/we will accomplish (objective) by (date) through attaining and achieving the following metrics (key results). +<!-- vale gitlab.FutureTense = YES --> + +To learn how to create better OKRs and how we use them at GitLab, see the +[Objectives and Key Results handbook page](https://about.gitlab.com/company/okrs/). + ## Create an objective Prerequisites: @@ -93,6 +115,26 @@ To edit an OKR: 1. Optional. To edit the description, select the edit icon (**{pencil}**), make your changes, and select **Save**. +## View OKR system notes + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) to feature flag named `work_items_mvc` in GitLab 15.8. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 15.10. +> - Changing activity sort order [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) in GitLab 15.8. +> - Filtering activity [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/389971) in GitLab 15.10. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 15.10. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +You can view all the system notes related to the task. By default they are sorted by **Oldest first**. +You can always change the sorting order to **Newest first**, which is remembered across sessions. + +## Comments and threads + +You can add [comments](discussions/index.md) and reply to threads in tasks. + ## Assign users To show who is responsible for an OKR, you can assign users to it. diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 2911a700e14..493771fa30a 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/organization/index.md b/doc/user/organization/index.md new file mode 100644 index 00000000000..f5af26250f6 --- /dev/null +++ b/doc/user/organization/index.md @@ -0,0 +1,42 @@ +--- +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Organization + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +NOTE: +Organization is in development. + +Organization will be above the [top-level namespaces](../namespace/index.md) for you to manage +everything you do as a GitLab administrator, including: + +- Defining and applying settings to all of your groups, subgroups, and projects. +- Aggregating data from all your groups, subgroups, and projects. + +Our goal is to reach feature parity between SaaS and self-managed installations, with all +[Admin Area settings](/ee/user/admin_area/settings/index.md) moving to either: + +- Groups. Available in the Organization, and subgroups. +- Hardware Controls. For functionality that does not apply to groups, Hardware Controls are only + applicable to self-managed installations. There is one Hardware Controls section per installation. + +For more information about the state of organization development, +see [epic 9265](https://gitlab.com/groups/gitlab-org/-/epics/9265). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For a video introduction to the new hierarchy concept for groups and projects for epics, see +[Consolidating groups and projects update (August 2021)](https://www.youtube.com/watch?v=fE74lsG_8yM). + +## Related topics + +- [Organization developer documentation](../../development/organization/index.md) diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 9229ea61821..9419451fd5d 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -281,7 +281,7 @@ Here are some other options you can use to reduce the Container Registry storage If you see this error message, check the regex patterns to ensure they are valid. -GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/). +GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/) using the `Golang` flavor. View some common [regex pattern examples](#regex-pattern-examples). ### The cleanup policy doesn't delete any tags diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index 7ec20e3d036..05574c54112 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -23,6 +23,12 @@ Project and Group packages are supported. For documentation of the specific API endpoints that Debian package manager clients use, see the [Debian API documentation](../../../api/packages/debian.md). +Prerequisites: + +- The `dpkg-deb` binary must be installed on the GitLab instance. + This binary is usually provided by the [`dpkg` package](https://wiki.debian.org/Teams/Dpkg/Downstream), + installed by default on Debian and derivatives. + ## Enable the Debian API **(FREE SELF)** Debian repository support is still a work in progress. It's gated behind a feature flag that's @@ -135,6 +141,7 @@ Once built, several files are created: - `.deb` files: the binary packages - `.udeb` files: lightened .deb files, used for Debian-Installer (if needed) +- `.ddeb` files: Ubuntu debug .deb files (if needed) - `.tar.{gz,bz2,xz,...}` files: Source files - `.dsc` file: Source metadata, and list of source files (with hashes) - `.buildinfo` file: Used for Reproducible builds (optional) @@ -176,39 +183,39 @@ To install a package: 1. Configure the repository: - If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: + If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: - ```shell - echo 'machine gitlab.example.com login <username> password <password>' \ - | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf - ``` + ```shell + echo 'machine gitlab.example.com login <username> password <password>' \ + | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf + ``` - Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): + Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): - ```shell - sudo mkdir -p /usr/local/share/keyrings - curl --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions/<codename>/key.asc" \ - | \ - gpg --dearmor \ - | \ - sudo tee /usr/local/share/keyrings/<codename>-archive-keyring.gpg \ - > /dev/null - ``` + ```shell + sudo mkdir -p /usr/local/share/keyrings + curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions/<codename>/key.asc" \ + | \ + gpg --dearmor \ + | \ + sudo tee /usr/local/share/keyrings/<codename>-archive-keyring.gpg \ + > /dev/null + ``` - Add your project as a source: + Add your project as a source: - ```shell - echo 'deb [ signed-by=/usr/local/share/keyrings/<codename>-archive-keyring.gpg ] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ - | sudo tee /etc/apt/sources.list.d/gitlab_project.list - sudo apt-get update - ``` + ```shell + echo 'deb [ signed-by=/usr/local/share/keyrings/<codename>-archive-keyring.gpg ] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ + | sudo tee /etc/apt/sources.list.d/gitlab_project.list + sudo apt-get update + ``` 1. Install the package: - ```shell - sudo apt-get -y install -t <codename> <package-name> - ``` + ```shell + sudo apt-get -y install -t <codename> <package-name> + ``` ## Download a source package @@ -216,36 +223,36 @@ To download a source package: 1. Configure the repository: - If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: + If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: - ```shell - echo 'machine gitlab.example.com login <username> password <password>' \ - | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf - ``` + ```shell + echo 'machine gitlab.example.com login <username> password <password>' \ + | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf + ``` - Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): + Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): - ```shell - sudo mkdir -p /usr/local/share/keyrings - curl --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions/<codename>/key.asc" \ - | \ - gpg --dearmor \ - | \ - sudo tee /usr/local/share/keyrings/<codename>-archive-keyring.gpg \ - > /dev/null - ``` + ```shell + sudo mkdir -p /usr/local/share/keyrings + curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions/<codename>/key.asc" \ + | \ + gpg --dearmor \ + | \ + sudo tee /usr/local/share/keyrings/<codename>-archive-keyring.gpg \ + > /dev/null + ``` - Add your project as a source: + Add your project as a source: - ```shell - echo 'deb-src [ signed-by=/usr/local/share/keyrings/<codename>-archive-keyring.gpg ] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ - | sudo tee /etc/apt/sources.list.d/gitlab_project-sources.list - sudo apt-get update - ``` + ```shell + echo 'deb-src [ signed-by=/usr/local/share/keyrings/<codename>-archive-keyring.gpg ] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ + | sudo tee /etc/apt/sources.list.d/gitlab_project-sources.list + sudo apt-get update + ``` 1. Download the source package: - ```shell - sudo apt-get source -t <codename> <package-name> - ``` + ```shell + sudo apt-get source -t <codename> <package-name> + ``` diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 1a089cd82be..f0e10d73d08 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -74,7 +74,7 @@ go env -w GOPROXY='https://gitlab.example.com/api/v4/projects/1234/packages/go,h With this configuration, Go fetches dependencies in this order: 1. Go attempts to fetch from the project-specific Go proxy. -1. Go attempts to fetch from [proxy.golang.org](https://proxy.golang.org). +1. Go attempts to fetch from [`proxy.golang.org`](https://proxy.golang.org). 1. Go fetches directly with version control system operations (like `git clone`, `svn checkout`, and so on). diff --git a/doc/user/packages/harbor_container_registry/index.md b/doc/user/packages/harbor_container_registry/index.md index 1ad5e2c2f05..6cea541a55d 100644 --- a/doc/user/packages/harbor_container_registry/index.md +++ b/doc/user/packages/harbor_container_registry/index.md @@ -18,7 +18,7 @@ You can view the Harbor Registry for a project or group. You can search, sort, and filter images on this page. You can share a filtered view by copying the URL from your browser. -At the project level, you can see **CLI Commands** in the upper right corner, where you can copy +At the project level, in the upper-right corner, you can see **CLI Commands** where you can copy corresponding commands to sign in, build images, and push images. **CLI Commands** is not shown at the group level. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 11e3d0e5131..6ce4aab930e 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -244,8 +244,19 @@ npm dist-tag rm @scope/package@version my-tag # Delete a tag from the package npm install @scope/package@my-tag # Install a specific tag ``` -You cannot use your `CI_JOB_TOKEN` or deploy token with the `npm dist-tag` commands. -View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) for details. +#### From CI/CD + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) in GitLab 15.10. + +You can use a [`CI_JOB_TOKEN`](../../../ci/jobs/ci_job_token.md) or [deploy token](../../project/deploy_tokens/index.md) +to run `npm dist-tag` commands in a GitLab CI/CD job. For example: + +```yaml +npm-deploy-job: + script: + - echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc + - npm dist-tag add @scope/package@version my-tag +``` Due to a bug in npm 6.9.0, deleting distribution tags fails. Make sure your npm version is 6.9.1 or later. @@ -266,19 +277,28 @@ The GitLab npm repository supports the following commands for the npm CLI (`npm` ### `404 Not Found` errors are happening on `npm install` or `yarn` -Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project gives you 404 Not Found errors. A fix for this problem is proposed in [issue 352962](https://gitlab.com/gitlab-org/gitlab/-/issues/352962). +Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project gives you 404 Not Found errors. You need to authenticate with a token that has access to the package and all its dependencies. -As a workaround, you can: +If the package and its dependencies are in separate projects but in the same group, you can use a +[group deploy token](../../project/deploy_tokens/index.md#create-a-deploy-token): -1. Create a [personal access token](../../profile/personal_access_tokens.md). -1. Authenticate at both the instance level and project level for each package: +```ini +//gitlab.example.com/api/v4/packages/npm/:_authToken=<group-token> +@group-scope:registry=https://gitlab.example.com/api/v4/packages/npm/ +``` - ```ini - @foo:registry=https://gitlab.example.com/api/v4/packages/npm/ - //gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} - //gitlab.example.com/api/v4/projects/<your_project_id_a>/packages/npm/:_authToken=${MY_TOKEN} - //gitlab.example.com/api/v4/projects/<your_project_id_b>/packages/npm/:_authToken=${MY_TOKEN} - ``` +If the package and its dependencies are spread across multiple groups, you can use a [personal access token](../../profile/personal_access_tokens.md) +from a user that has access to all the groups or individual projects: + +```ini +//gitlab.example.com/api/v4/packages/npm/:_authToken=<personal-access-token> +@group-1:registry=https://gitlab.example.com/api/v4/packages/npm/ +@group-2:registry=https://gitlab.example.com/api/v4/packages/npm/ +``` + +WARNING: +Personal access tokens must be treated carefully. Read our [token security considerations](../../../security/token_overview.md#security-considerations) +for guidance on managing personal access tokens (for example, setting a short expiry and using minimal scopes). ### `npm publish` targets default npm registry (`registry.npmjs.org`) diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 6710c147c87..4595e0eebb9 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -472,4 +472,4 @@ nuget locals all -clear ### `Error publishing` or `Invalid Package: Failed metadata extraction error` messages when trying to publish NuGet packages in a Docker-based GitLab installation -Webhook requests to local network addresses are blocked to prevent the exploitation of internal web services. If you get `Error publishing` or `Invalid Package` messages when you try to publish NuGet packages, change your network settings to [allow webhook and service requests to the local network](../../../security/webhooks.md#allow-webhook-and-service-requests-to-local-network). +Webhook requests to local network addresses are blocked to prevent exploitation of internal web services. If you get `Error publishing` or `Invalid Package` messages when you try to publish NuGet packages, change your network settings to [allow webhook and integration requests to the local network](../../../security/webhooks.md#allow-requests-to-the-local-network-from-webhooks-and-integrations). diff --git a/doc/user/packages/package_registry/reduce_package_registry_storage.md b/doc/user/packages/package_registry/reduce_package_registry_storage.md index 673196ebad5..020cad5ac14 100644 --- a/doc/user/packages/package_registry/reduce_package_registry_storage.md +++ b/doc/user/packages/package_registry/reduce_package_registry_storage.md @@ -35,6 +35,9 @@ To delete a package in the UI, from your group or project: The package is permanently deleted. +If [request forwarding](supported_functionality.md#forwarding-requests) is enabled, +deleting a package can introduce a [dependency confusion risk](supported_functionality.md#deleting-packages). + ## Delete assets associated with a package To delete package assets, you must have suitable [permissions](../../permissions.md). diff --git a/doc/user/packages/package_registry/supported_functionality.md b/doc/user/packages/package_registry/supported_functionality.md index e56ae88872a..0c7813beae0 100644 --- a/doc/user/packages/package_registry/supported_functionality.md +++ b/doc/user/packages/package_registry/supported_functionality.md @@ -53,10 +53,10 @@ Requests for packages not found in your GitLab project are forwarded to the publ | Package type | Supports request forwarding | |-----------------------------------------------------|-----------------------------| -| [Maven](../maven_repository/index.md) | Y | -| [npm](../npm_registry/index.md) | Y | +| [Maven](../maven_repository/index.md) | [Yes (disabled by default)](../../admin_area/settings/continuous_integration.md#maven-forwarding) | +| [npm](../npm_registry/index.md) | [Yes](../../admin_area/settings/continuous_integration.md#npm-forwarding) | | [NuGet](../nuget_repository/index.md) | N | -| [PyPI](../pypi_repository/index.md) | Y | +| [PyPI](../pypi_repository/index.md) | [Yes](../../admin_area/settings/continuous_integration.md#pypi-forwarding) | | [Generic packages](../generic_packages/index.md) | N | | [Terraform](../terraform_module_registry/index.md) | N | | [Composer](../composer_repository/index.md) | N | @@ -66,6 +66,23 @@ Requests for packages not found in your GitLab project are forwarded to the publ | [Go](../go_proxy/index.md) | N | | [Ruby gems](../rubygems_registry/index.md) | N | +### Deleting packages + +When package requests are forwarded to a public registry, deleting packages can +be a [dependency confusion vulnerability](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610). + +If a system tries to pull a deleted package, the request is forwarded to the public +registry. If a package with the same name and version is found in the public registry, that package +is pulled instead. There is a risk that the package pulled from the registry might not be +what is expected, and could even be malicious. + +To reduce the associated security risks, before deleting a package you can: + +- Verify the package is not being actively used. +- Disable request forwarding: + - Instance administrators can disable forwarding in the [**Continuous Integration** section](../../admin_area/settings/continuous_integration.md#package-registry-configuration) of the Admin Area. + - Group owners can disable forwarding in the **Packages and Registries** section of the group settings. + ## Allow or prevent duplicates **(FREE)** By default, the GitLab package registry either allows or prevents duplicates based on the default of that specific package manager format. diff --git a/doc/user/packages/yarn_repository/index.md b/doc/user/packages/yarn_repository/index.md index 7e2f45019cd..e756a912928 100644 --- a/doc/user/packages/yarn_repository/index.md +++ b/doc/user/packages/yarn_repository/index.md @@ -6,220 +6,312 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Publish packages with Yarn -Publish npm packages in your project's Package Registry using Yarn. Then install the -packages whenever you need to use them as a dependency. +You can publish packages with [Yarn 1 (Classic)](https://classic.yarnpkg.com) and [Yarn 2+](https://yarnpkg.com). -Learn how to build a [yarn](../workflows/build_packages.md#yarn) package. +To find the Yarn version used in the deployment container, run `yarn --version` in the `script` block of the CI +script job block that is responsible for calling `yarn publish`**`. The Yarn version is shown in the pipeline output. -You can get started with Yarn 2 by following the [Yarn documentation](https://yarnpkg.com/getting-started/install/). +Learn how to build a [Yarn](../workflows/build_packages.md#yarn) package. + +You can use the Yarn documentation to get started with +[Yarn Classic](https://classic.yarnpkg.com/en/docs/getting-started) and +[Yarn 2+](https://yarnpkg.com/getting-started/). ## Publish to GitLab Package Registry +You can use Yarn to publish to the GitLab Package Registry. + ### Authentication to the Package Registry You need a token to publish a package. Different tokens are available depending on what you're trying to achieve. For more information, review the [guidance on tokens](../../../user/packages/package_registry/index.md#authenticate-with-the-registry). -- If your organization uses two-factor authentication (2FA), you must use a personal access token with the scope set to `api`. -- If you publish a package via CI/CD pipelines, you must use a CI job token. +- If your organization uses two-factor authentication (2FA), you must use a + personal access token with the scope set to `api`. +- If you publish a package via CI/CD pipelines, you can use a CI job token in + private runners or you can register a variable for shared runners. -Create a token and save it to use later in the process. +### Publish configuration -### Naming convention +To publish, set the following configuration in `.yarnrc.yml`. This file should be +located in the root directory of your package project source where `package.json` is found. -Depending on how you install the package, you may need to adhere to the naming convention. +```yaml +npmScopes: + <my-org>: + npmPublishRegistry: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' + npmAlwaysAuth: true + npmAuthToken: '<your_token>' +``` -You can use one of two API endpoints to install packages: +In this configuration: -- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. -- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. +- Replace `<my-org>` with your organization scope, exclude the `@` symbol. +- Replace `<your_domain>` with your domain name. +- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. +- Replace `<your_token>` with a deployment token, group access token, project access token, or personal access token. -If you plan to install a package through the [project level](#install-from-the-project-level), you do not have to -adhere to the naming convention. +Scoped registry does not work in Yarn Classic in `package.json` file, based on +this [issue](https://github.com/yarnpkg/yarn/pull/7829). +Therefore, under `publishConfig` there should be `registry` and not `@scope:registry` for Yarn Classic. +You can publish using your command line or a CI/CD pipeline to the GitLab Package Registry. -If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name -your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` and have the -`@owner/package-name` format. You can set up the scope for your package in the `.yarnrc.yml` file and by using the -`publishConfig` option in the `package.json`. +### Publishing via the command line - Manual Publish -- The value used for the `@scope` is the root of the project that hosts the packages and not the root - of the project with the package's source code. The scope should be lowercase. -- The package name can be anything you want +```shell +# Yarn 1 (Classic) +yarn publish -| Project URL | Package Registry in | Scope | Full package name | -| ------------------------------------------------------- | ------------------- | --------- | ---------------------- | -| `https://gitlab.com/my-org/engineering-group/analytics` | Analytics | `@my-org` | `@my-org/package-name` | +# Yarn 2+ +yarn npm publish +``` -### Configuring `.yarnrc.yml` to publish from the project level +Your package should now publish to the Package Registry. -To publish with the project-level npm endpoint, set the following configuration in -`.yarnrc.yml`: +### Publishing via a CI/CD pipeline - Automated Publish -```yaml -npmScopes: - foo: - npmRegistryServer: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' - npmPublishRegistry: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' +You can use pipeline variables when you use this method. -npmRegistries: - //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: - npmAlwaysAuth: true - npmAuthToken: '<your_token>' -``` +You can use **Shared Runners** *(Default)* or **Private Runners** (Advanced). -In this configuration: +#### Shared runners -- Replace `<your_domain>` with your domain name. -- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. -- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. +Third party images such as `node:latest` or `node:current` do not have direct access +to the `CI_JOB_TOKEN` when operating in a shared runner. You must configure an +authentication token or use a private runner. + +To create a authentication token: -### Configuring `.yarnrc.yml` to publish from the instance level +1. On the top bar, select **Main menu**, and: + - For a project, select **Projects** and find your project. + - For a group, select **Groups** and find your group. +1. On the left sidebar, select **Settings > Repository > Deploy Tokens**. +1. Create a deployment token with `read_package_registry` and `write_package_registry` scopes and copy the generated token. +1. On the left sidebar, select **Settings > CI/CD > Variables**. +1. Select `Add variable` and use the following settings: -For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: +| Field | Value | +|--------------------|------------------------------| +| key | `NPM_AUTH_TOKEN` | +| value | `<DEPLOY-TOKEN-FROM-STEP-3>` | +| type | Variable | +| Protected variable | `CHECKED` | +| Mask variable | `CHECKED` | +| Expand variable | `CHECKED` | + +To use any **Protected variable**: + + 1. Go to the repository that contains the Yarn package source code. + 1. On the left sidebar, select **Settings > Repository**. + - If you are building from branches with tags, select **Protected Tags** and add `v*` (wildcard) for semantic versioning. + - If you are building from branches without tags, select **Protected Branches**. + +Then add the `NPM_AUTH_TOKEN` created above, to the `.yarnrc.yml` configuration +in your package project root directory where `package.json` is found: ```yaml npmScopes: - <scope>: - npmRegistryServer: 'https://<your_domain>/api/v4/packages/npm/' - -npmRegistries: - //gitlab.example.com/api/v4/packages/npm/: + esp-code: + npmPublishRegistry: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/" npmAlwaysAuth: true - npmAuthToken: '<your_token>' + npmAuthToken: "${NPM_AUTH_TOKEN}" ``` -In this configuration: +#### Private runners -- Replace `<your_domain>` with your domain name. -- Your scope is `<scope>`, without `@`. -- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. +Add the `CI_JOB_TOKEN` to the `.yarnrc.yml` configuration in your package project +root directory where `package.json` is found: -### Publishing a package via the command line +```yaml +npmScopes: + esp-code: + npmPublishRegistry: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/" + npmAlwaysAuth: true + npmAuthToken: "${CI_JOB_TOKEN}" +``` -Publish a package: +To publish the package using CI/CD pipeline, In the GitLab project that houses +your `yarnrc.yml`, edit or create a `.gitlab-ci.yml` file. For example to trigger +only on any tag push: -```shell -npm publish -``` +```yaml +# Yarn 1 +image: node:lts -Your package should now publish to the Package Registry. +stages: + - deploy -### Publishing via a CI/CD pipeline +rules: +- if: $CI_COMMIT_TAG -In the GitLab project that houses your `yarnrc.yml`, edit or create a `.gitlab-ci.yml` file. For example: +deploy: + stage: deploy + script: + - yarn publish +``` ```yaml -image: node:latest +# Yarn 2+ +image: node:lts stages: - deploy +rules: + - if: $CI_COMMIT_TAG + deploy: stage: deploy + before_script: + - corepack enable + - yarn set version stable script: - - npm publish + - yarn npm publish ``` Your package should now publish to the Package Registry when the pipeline runs. ## Install a package -If multiple packages have the same name and version, the most recently-published package is retrieved when you install a package. +NOTE: +If multiple packages have the same name and version, the most recently-published +package is retrieved when you install a package. -You can install a package from a GitLab project or instance: +You can use one of two API endpoints to install packages: -- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. -- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. +- **Instance-level**: Best used when working with many packages in an organization scope. -### Install from the instance level +- If you plan to install a package through the [instance level](#install-from-the-instance-level), + then you must name your package with a [scope](https://docs.npmjs.com/misc/scope/). + Scoped packages begin with a `@` and have the `@owner/package-name` format. You can set up + the scope for your package in the `.yarnrc.yml` file and by using the `publishConfig` + option in the `package.json`. -WARNING: -You must use packages published with the scoped [naming convention](#naming-convention) when you install a package from the instance level. +- The value used for the `@scope` is the organization root (top-level project) `...com/my-org` + *(@my-org)* that hosts the packages, not the root of the project with the package's source code. +- The scope is always lowercase. +- The package name can be anything you want `@my-org/any-name`. -1. Authenticate to the Package Registry +- **Project-level**: For when you have a one-off package. - If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. +If you plan to install a package through the [project level](#install-from-the-project-level), +you do not have to adhere to the naming convention. - ```shell - npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token - ``` +| Project URL | Package Registry | Organization Scope | Full package name | +|-------------------------------------------------------------------|----------------------|--------------------|-----------------------------| +| `https://gitlab.com/<my-org>/<group-name>/<package-name-example>` | Package Name Example | `@my-org` | `@my-org/package-name` | +| `https://gitlab.com/<example-org>/<group-name>/<project-name>` | Project Name | `@example-org` | `@example-org/project-name` | - - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. - - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. +You can install from the instance level or from the project level. -1. Set the registry +The configurations for `.yarnrc.yml` can be added per package consuming project +root where `package.json` is located, or you can use a global +configuration located in your system user home directory. - ```shell - npm config set @scope:registry https://your_domain_name.com/api/v4/packages/npm/ - ``` +### Install from the instance level - - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. - - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. - - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. +Use these steps for global configuration in the `.yarnrc.yml` file: -1. Install the package +1. [Configure organization scope](#configure-organization-scope). +1. [Set the registry](#set-the-registry). - ```shell - yarn add @scope/my-package - ``` +#### Configure organization scope + +```yaml +npmScopes: + <my-org>: + npmRegistryServer: "https://<your_domain_name>/api/v4/packages/npm" +``` + +- Replace `<my-org>` with the root level group of the project you're installing to the package from excluding the `@` symbol. +- Replace `<your_domain_name>` with your domain name, for example, `gitlab.com`. + +#### Set the registry + +Skip this step if your package is public not private. + +```yaml + npmRegistries: + //<your_domain_name>/api/v4/packages/npm: + npmAlwaysAuth: true + npmAuthToken: "<your_token>" +``` + +- Replace `<your_domain_name>` with your domain name, for example, `gitlab.com`. +- Replace `<your_token>` with a deployment token (recommended), group access token, project access token, or personal access token. ### Install from the project level -1. Authenticate to the Package Registry +Use these steps for each project in the `.yarnrc.yml` file: + +1. [Configure project scope](#configure-project-scope). +1. [Set the registry](#set-the-registry-project-level). - If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. +#### Configure project scope - ```shell - npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token - ``` + ```yaml + npmScopes: + <my-org>: + npmRegistryServer: "https://<your_domain_name>/api/v4/projects/<your_project_id>/packages/npm" +``` - - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. - - Replace `your_project_id` is your project ID, found on the project's home page. - - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. +- Replace `<my-org>` with the root level group of the project you're installing to the package from excluding the `@` symbol. +- Replace `<your_domain_name>` with your domain name, for example, `gitlab.com`. +- Replace `<your_project_id>` with your project ID, found on the project's home page. -1. Set the registry +#### Set the registry (project level) - ```shell - npm config set @scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ - ``` +Skip this step if your package is public not private. - - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. - - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. - - Replace `your_project_id` is your project ID, found on the project's home page. +```yaml +npmRegistries: + //<your_domain_name>/api/v4/projects/<your_project_id>/packages/npm: + npmAlwaysAuth: true + npmAuthToken: "<your_token>" +``` -1. Install the package +- Replace `<your_domain_name>` with your domain name, for example, `gitlab.com`. +- Replace `<your_token>` with a deployment token (recommended), group access token, project access token, or personal access token. +- Replace `<your_project_id>` with your project ID, found on the project's home page. - ```shell - yarn add @scope/my-package - ``` +### Install the package -## Helpful hints +For Yarn 2+, use `yarn add` either in the command line or in the CI/CD pipelines to install your packages: -For full helpful hints information, refer to the [npm documentation](../npm_registry/index.md#helpful-hints). +```shell +yarn add @scope/my-package +``` -### Supported CLI commands +#### For Yarn Classic -The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI -(`yarn`): +The Yarn Classic setup, requires both `.npmrc` and `.yarnrc` files as +[mentioned in issue](https://github.com/yarnpkg/yarn/issues/4451#issuecomment-753670295): -- `npm install`: Install npm packages. -- `npm publish`: Publish an npm package to the registry. -- `npm dist-tag add`: Add a dist-tag to an npm package. -- `npm dist-tag ls`: List dist-tags for a package. -- `npm dist-tag rm`: Delete a dist-tag. -- `npm ci`: Install npm packages directly from your `package-lock.json` file. -- `npm view`: Show package metadata. -- `yarn add`: Install an npm package. -- `yarn update`: Update your dependencies. +- Place credentials in the `.npmrc` file. +- Place the scoped registry in the `.yarnrc` file. -## Troubleshooting +```shell +# .npmrc +//<your_domain_name>/api/v4/projects/<your_project_id>/packages/npm/:_authToken="<your_token>" + +# .yarnrc +"@scope:registry" "https://<your_domain_name>/api/v4/projects/<your_project_id>/packages/npm/" +``` + +Then you can use `yarn add` to install your packages. + +## Related topics -For full troubleshooting information, refer to the [npm documentation](../npm_registry/index.md#troubleshooting). +- For full helpful hints information, see the + [npm documentation](../npm_registry/index.md#helpful-hints). +- For Yarn 1 to Yarn 2+ migration information see the + [Yarn Migration Guide](https://yarnpkg.com/getting-started/migration). + +## Troubleshooting ### Error running Yarn with the Package Registry for the npm registry -If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get -an error message like: +If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get an error message like: ```shell yarn install v1.15.2 @@ -233,14 +325,7 @@ info If you think this is a bug, please open a bug report with the information p info Visit https://classic.yarnpkg.com/en/docs/cli/install for documentation about this command ``` -In this case, try adding this to your `.npmrc` file (and replace `<your_token>` -with your personal access token or deploy token): - -```plaintext -//gitlab.example.com/api/v4/projects/:_authToken=<your_token> -``` - -You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: +In this case, the following commands creates a file called `.yarnrc` in the current directory. Make sure to be in either your user home directory for global configuration or your project root for per-project configuration: ```shell yarn config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 8e736b6d83e..43029e37047 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -69,8 +69,8 @@ The following table lists project permissions available for each role: | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>View clusters | | | ✓ | ✓ | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | +| [GitLab Agent for Kubernetes](clusters/agent/index.md):<br>View agents | | | ✓ | ✓ | ✓ | +| [GitLab Agent for Kubernetes](clusters/agent/index.md):<br>Manage agents | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete [cleanup policies](packages/container_registry/delete_container_registry_images.md#use-a-cleanup-policy) | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Push an image to the Container Registry | | | ✓ | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (19) | ✓ (19) | ✓ | ✓ | ✓ | @@ -220,7 +220,7 @@ The following table lists project permissions available for each role: <!-- markdownlint-disable MD029 --> -1. On self-managed GitLab instances, users with the Guest role are able to perform this action only on public and internal projects (not on private projects). [External users](admin_area/external_users.md) must be given explicit access even if the project is internal. Users with the Guest role on GitLab.com are only able to perform this action on public projects because internal visibility is not available. In GitLab 15.9 and later, this restriction only applies to users with the non-custom Guest role on self-managed GitLab instances and GitLab.com. +1. On self-managed GitLab instances, users with the Guest role are able to perform this action only on public and internal projects (not on private projects). [External users](admin_area/external_users.md) must be given explicit access even if the project is internal. Users with the Guest role on GitLab.com are only able to perform this action on public projects because internal visibility is not available. 2. Guest users can only view the [confidential issues](project/issues/confidential_issues.md) they created themselves or are assigned to. 3. Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [protected branches](project/protected_branches.md). 4. If the [branch is protected](project/protected_branches.md), this depends on the access given to Developers and Maintainers. @@ -271,8 +271,7 @@ More details about the permissions for some project-level features follow. | View and download artifacts | ✓ (1) | ✓ (2) | ✓ | ✓ | ✓ | ✓ | | View [environments](../ci/environments/index.md) | ✓ (3) | ✓ (3) | ✓ | ✓ | ✓ | ✓ | | View job logs and job details page | ✓ (1) | ✓ (2) | ✓ | ✓ | ✓ | ✓ | -| View pipeline details page | ✓ (1) | ✓ (2) | ✓ | ✓ | ✓ | ✓ | -| View pipelines page | ✓ (1) | ✓ (2) | ✓ | ✓ | ✓ | ✓ | +| View pipelines and pipeline details pages | ✓ (1) | ✓ (2) | ✓ | ✓ | ✓ | ✓ | | View pipelines tab in MR | ✓ (3) | ✓ (3) | ✓ | ✓ | ✓ | ✓ | | [View vulnerabilities in a pipeline](application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline) | | ✓ (2) | ✓ | ✓ | ✓ | ✓ | | View and download project-level [Secure Files](../api/secure_files.md) | | | | ✓ | ✓ | ✓ | @@ -467,9 +466,10 @@ subscriptions. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106256) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `customizable_roles`. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110810) in GitLab 15.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114524) in GitLab 15.10. Custom roles allow group members who are assigned the Owner role to create roles -specific to the needs of their organization. +specific to the needs of their organization. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a demo of the custom roles feature, see [[Demo] Ultimate Guest can view code on private repositories via custom role](https://www.youtube.com/watch?v=46cp_-Rtxps). @@ -481,29 +481,36 @@ To enable custom roles for your group, a group member with the Owner role: 1. Makes sure that there is at least one private project in this group or one of its subgroups, so that you can see the effect of giving a Guest a custom role. 1. Creates a personal access token with the API scope. -1. Uses [the API](../api/member_roles.md#add-a-member-role-to-a-group) to create the Guest+1 role for the group. +1. Uses [the API](../api/member_roles.md#add-a-member-role-to-a-group) to create the Guest+1 role for the root group. ### Associate a custom role with an existing group member To associate a custom role with an existing group member, a group member with the Owner role: -1. Invites a test user account to the root group as a Guest. - At this point, this Guest user cannot see any code on the projects in the group. +1. Invites a user to the root group or any subgroup or project in the root + group's hierarchy as a Guest. At this point, this Guest user cannot see any + code on the projects in the group or subgroup. 1. Optional. If the Owner does not know the `ID` of the Guest user receiving a custom role, finds that `ID` by making an [API request](../api/member_roles.md#list-all-member-roles-of-a-group). -1. Associates the group member with the Guest+1 role using the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project) +1. Associates the member with the Guest+1 role using the [Group and Project Members API endpoint](../api/members.md#edit-a-member-of-a-group-or-project) - ```shell - curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": '$MEMBER_ROLE_ID', "access_level": 10}' "https://example.gitlab.com/api/v4/groups/$GROUP_PATH/members/$GUEST_USER_ID" - ``` + ```shell + # to update a project membership + curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": '$MEMBER_ROLE_ID', "access_level": 10}' "https://example.gitlab.com/api/v4/projects/$ID/members/$GUEST_USER_ID" + + # to update a group membership + curl --request PUT --header "Content-Type: application/json" --header "Authorization: Bearer $YOUR_ACCESS_TOKEN" --data '{"member_role_id": '$MEMBER_ROLE_ID', "access_level": 10}' "https://example.gitlab.com/api/v4/groups/$ID/members/$GUEST_USER_ID" + ``` Where: + + - `$ID`: The `ID` or [URL-encoded path of the project or group](../api/rest/index.md#namespaced-path-encoding) associated with the membership receiving the custom role. - `$MEMBER_ROLE_ID`: The `ID` of the member role created in the previous section. - `$GUEST_USER_ID`: The `ID` of the Guest user receiving a custom role. - Now the Guest+1 user can view code on all projects in the root group. + Now the Guest+1 user can view code on all projects associated with this membership. ### Remove a custom role from a group member diff --git a/doc/user/product_analytics/index.md b/doc/user/product_analytics/index.md index 6d6a609618b..1a6ad4edf02 100644 --- a/doc/user/product_analytics/index.md +++ b/doc/user/product_analytics/index.md @@ -8,9 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Introduced in GitLab 15.4 as an [Alpha](../../policy/alpha-beta-support.md#alpha-features) feature [with a flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. > - `cube_api_proxy` revised to only reference the [Product Analytics API](../../api/product_analytics.md) in GitLab 15.6. +> - `cube_api_proxy` removed and replaced with `product_analytics_internal_preview` in GitLab 15.10. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `cube_api_proxy`. +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_internal_preview`. On GitLab.com, this feature is not available. This feature is not ready for production use. @@ -50,6 +51,7 @@ Product Analytics uses several tools: > - Introduced in GitLab 15.6 behind the [feature flag](../../administration/feature_flags.md) named `cube_api_proxy`. Disabled by default. > - Moved to be behind the [feature flag](../../administration/feature_flags.md) named `product_analytics_admin_settings` in GitLab 15.7. Disabled by default. +> - `cube_api_proxy` removed and replaced with `product_analytics_internal_preview` in GitLab 15.10. FLAG: On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `product_analytics_admin_settings`. @@ -205,3 +207,47 @@ The `afterDate` filter is not supported. Please use `beforeDate` or `inDateRange } } ``` + +## Raw data export + +Exporting the raw event data from the underlying storage engine can help you debug and create datasets for data analysis. + +### Export raw data with Cube queries + +You can [query the raw data with the REST API](../../api/product_analytics.md#send-query-request-to-cube) and convert the JSON output to any required format. + +You can export the raw data for a specific dimension by passing a list of dimensions to the `dimensions` key. For example, the following query outputs the raw data for the attributes listed: + +```json +POST /api/v4/projects/PROJECT_ID/product_analytics/request/load?queryType=multi + +{ + "dimensions": [ + "TrackedEvents.docEncoding", + "TrackedEvents.docHost", + "TrackedEvents.docPath", + "TrackedEvents.docSearch", + "TrackedEvents.eventType", + "TrackedEvents.idsAjsAnonymousId", + "TrackedEvents.localTzOffset", + "TrackedEvents.pageTitle", + "TrackedEvents.src", + "TrackedEvents.utcTime", + "TrackedEvents.vpSize" + ], + "order": { + "TrackedEvents.apiKey": "asc" + } +} +``` + +If the request is successful, the returned JSON includes an array of rows of results. + +### Caveats + +Because Cube acts as an abstraction layer between the raw data and the API, the exported raw data has some caveats: + +- Data is grouped by the selected dimensions. Therefore, the exported data might be incomplete, unless including both `utcTime` and `userAnonymousId`. +- Data is by default limited to 10,000 rows, but you can increase the limit to maximum 50,000 rows. If your dataset has more than 50,000 rows, you need to paginate through the results by using the `limit` and `offset` parameters. +- Data is always returned in JSON format. If you need it in a different format, you need to convert the JSON to the required format using a scripting language of your choice. +- [Issue 391683](https://gitlab.com/gitlab-org/gitlab/-/issues/391683) tracks the implementation of a more scalable export solution. diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 60b8fe8ab88..08d23902095 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -16,9 +16,9 @@ You can create users: ## Create users on sign-in page -Prerequisites: +Prerequisite: -- [Sign-up enabled](../../admin_area/settings/sign_up_restrictions.md) +- [Sign-up must be enabled](../../admin_area/settings/sign_up_restrictions.md). Users can create their own accounts by either: @@ -27,9 +27,9 @@ Users can create their own accounts by either: ## Create users in Admin Area -Prerequisites: +Prerequisite: -- You must have administrator access for the instance. +- You must have administrator access to the instance. To create a user manually: diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index a74fd8d5215..56aa545408f 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -48,6 +48,7 @@ When deleting users, you can either: - Delete just the user. Not all associated records are deleted with the user. Instead of being deleted, these records are moved to a system-wide user with the username Ghost User. The Ghost User's purpose is to act as a container for such records. Any commits made by a deleted user still display the username of the original user. + The user's personal projects are deleted, not moved to the Ghost User. - Delete the user and their contributions, including: - Abuse reports. - Award emojis. diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index b76b99b5242..fe2e2acaae3 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -14,7 +14,7 @@ GitLab supports as a second factor of authentication: - Time-based one-time passwords ([TOTP](https://datatracker.ietf.org/doc/html/rfc6238)). When enabled, GitLab prompts you for a code when you sign in. Codes are generated by your one-time password authenticator (for example, a password manager on one of your devices). -- U2F or WebAuthn devices. You're prompted to activate your U2F or WebAuthn device (usually by pressing a button on it) when +- WebAuthn devices. You're prompted to activate your WebAuthn device (usually by pressing a button on it) when you supply your username and password to sign in. This performs secure authentication on your behalf. If you set up a device, also set up a TOTP so you can still access your account if you lose the device. @@ -42,10 +42,10 @@ Git Credential Manager is developed primarily by GitHub, Inc. It is an open-sour > - Account email confirmation requirement [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35102) in GitLab 14.3. [Deployed behind the `ensure_verified_primary_email_for_2fa` flag](../../../administration/feature_flags.md), enabled by default. > - Account email confirmation requirement generally available and [feature flag `ensure_verified_primary_email_for_2fa` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340151) in GitLab 14.4. -You can enable 2FA: +You can enable 2FA using a: -- Using a one-time password authenticator. After you enable 2FA, back up your [recovery codes](#recovery-codes). -- Using a U2F or WebAuthn device. +- One-time password authenticator. After you enable 2FA, back up your [recovery codes](#recovery-codes). +- WebAuthn device. In GitLab 14.3 and later, your account email must be confirmed to enable 2FA. @@ -60,10 +60,11 @@ To enable 2FA with a one-time password: 1. **On your device (usually your phone):** 1. Install a compatible application. For example: - Cloud-based (recommended because you can restore access if you lose the hardware device): - - [Authy](https://authy.com/) + - [Authy](https://authy.com/). + - [Duo](https://duo.com/). - Other: - - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en) - - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app) + - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en). + - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app). 1. In the application, add a new entry in one of two ways: - Scan the code displayed by GitLab with your device's camera to add the entry automatically. - Enter the details provided to add the entry manually. @@ -72,9 +73,6 @@ To enable 2FA with a one-time password: 1. Enter your current password. 1. Select **Submit**. -NOTE: -DUO [cannot be used for 2FA](https://gitlab.com/gitlab-org/gitlab/-/issues/15760). - If you entered the correct pin, GitLab displays a list of [recovery codes](#recovery-codes). Download them and keep them in a safe place. @@ -141,6 +139,70 @@ Configure FortiAuthenticator in GitLab. On your GitLab server: 1. [Reconfigure](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) (Omnibus GitLab) or [restart](../../../administration/restart_gitlab.md#installations-from-source) (GitLab installed from source). +### Enable one-time password using Duo + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15760) in GitLab 15.10. + +FLAG: +On self-managed GitLab, by default this feature is available. On GitLab.com this feature is not available. + +You can use Duo as an OTP provider in GitLab. + +#### Prerequisites + +To use Duo as an OTP provider: + +- Your account must exist in both Duo and GitLab, with the same username in both applications. +- You must have [configured Duo](https://admin.duosecurity.com/) and have an integration key, secret key, and API hostname. + +For more information, see the [Duo API documentation](https://duo.com/docs/authapi). + +GitLab 15.10 has been tested with Duo version D261.14 + +#### Configure Duo in GitLab + +On your GitLab server: + +1. Open the configuration file. + + For Omnibus GitLab: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```shell + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. Add the provider configuration: + + For Omnibus package: + + ```ruby + gitlab_rails['duo_auth_enabled'] = false + gitlab_rails['duo_auth_integration_key'] = '<duo_integration_key_value>' + gitlab_rails['duo_auth_secret_key'] = '<duo_secret_key_value>' + gitlab_rails['duo_auth_hostname'] = '<duo_api_hostname>' + ``` + + For installations from source: + + ```yaml + duo_auth: + enabled: true + hostname: <duo_api_hostname> + integration_key: <duo_integration_key_value> + secret_key: <duo_secret_key_value> + ``` + +1. Save the configuration file. +1. For Omnibus GitLab, [reconfigure GitLab](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure). + For installations from source, [restart GitLab](../../../administration/restart_gitlab.md#installations-from-source). + ### Enable one-time password using FortiToken Cloud > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212313) in GitLab 13.7 [with a flag](../../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. @@ -198,68 +260,61 @@ Configure FortiToken Cloud in GitLab. On your GitLab server: 1. [Reconfigure](../../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) (Omnibus GitLab) or [restart](../../../administration/restart_gitlab.md#installations-from-source) (GitLab installed from source). -### Set up a U2F device - -GitLab officially supports [YubiKey](https://www.yubico.com/products/) U2F devices, but users have successfully used -[SoloKeys](https://solokeys.com/) and [Google Titan Security Key](https://cloud.google.com/titan-security-key). - -U2F is [supported by](https://caniuse.com/#search=U2F) the following desktop browsers: - -- Chrome -- Edge -- Opera -- Firefox 67+. For Firefox 47-66: - - 1. Enable the FIDO U2F API in [`about:config`](https://support.mozilla.org/en-US/kb/about-config-editor-firefox). - 1. Search for `security.webauth.u2f` and select it to toggle to `true`. - -To set up 2FA with a U2F device: - -1. Access your [**User settings**](../index.md#access-your-user-settings). -1. Select **Account**. -1. Select **Enable Two-Factor Authentication**. -1. Connect your U2F device. -1. Select **Set up New U2F Device**. -1. A light begins blinking on your device. Activate it by pressing its button. - -A message displays indicating that your device was successfully set up. Select **Register U2F Device** to complete the -process. Recovery codes are not generated for U2F devices. - ### Set up a WebAuthn device -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22506) in GitLab 13.4 [with a flag](../../../administration/feature_flags.md) named `webauthn`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/232671) in GitLab 14.6. +> - WebAuthn devices [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22506) in GitLab 13.4 [with a flag](../../../administration/feature_flags.md) named `webauthn`. Disabled by default. +> - WebAuthn devices [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/232671) in GitLab 14.6. +> - Optional one-time password authentication for WebAuthn devices [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378844) in GitLab 15.10 [with a flag](../../../administration/feature_flags.md) named `webauthn_without_topt`. [Enabled on GitLab.com and self-managed by default](https://gitlab.com/gitlab-org/gitlab/-/issues/232671). FLAG: -On self-managed GitLab, by default this feature is available. To disable the feature, ask an administrator to +On self-managed GitLab, by default, WebAuthn devices are available. To disable the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `webauthn`. If you disable the WebAuthn feature flag after WebAuthn devices have been registered, these devices are not usable until you re-enable this feature. +On GitLab.com, WebAuthn devices are available. + +FLAG: +On self-managed GitLab, by default, optional one-time password authentication for WebAuthn devices is available. To hide the feature, ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `webauthn_without_topt`. On GitLab.com, this feature is available. -WebAuthn [supported by](https://caniuse.com/#search=webauthn): +WebAuthn is [supported by](https://caniuse.com/#search=webauthn) the following: -- The following desktop browsers: +- Desktop browsers: - Chrome - Edge - Firefox - Opera - Safari -- The following mobile browsers: +- Mobile browsers: - Chrome for Android - Firefox for Android - iOS Safari (since iOS 13.3) To set up 2FA with a WebAuthn-compatible device: +1. Optional. [Set up a one-time password](#enable-one-time-password). 1. Access your [**User settings**](../index.md#access-your-user-settings). 1. Select **Account**. 1. Select **Enable Two-Factor Authentication**. 1. Plug in your WebAuthn device. +1. Enter a device name and in GitLab 15.10 and later, your GitLab account password. + You might not need to enter this password if you have signed in through your + identity provider. 1. Select **Set up New WebAuthn Device**. 1. Depending on your device, you might have to press a button or touch a sensor. -A message displays indicating that your device was successfully set up. Recovery codes are not generated for WebAuthn -devices. +You should receive a message indicating that you successfully set up your device. + +When you set up 2FA with a WebAuthn-compatible device, that device is linked to +a specific browser on a specific computer. Depending on the browser and WebAuthn +device, you might be able to configure settings to use the WebAuthn device on a +different browser or computer. + +If this is the first time you have set up 2FA, you +must [download recovery codes](#recovery-codes) so you can recover access to your +account if you lose access. + +WARNING: +You can lose access to your account if you clear your browser data. ## Recovery codes @@ -272,11 +327,11 @@ these recovery codes to sign in to your account. WARNING: Each code can be used only once to sign in to your account. -We recommend copying and printing them, or downloading them using the **Download codes** button for storage in a safe +You should copy and print the codes, or use **Download codes** to download them for storage in a safe place. If you choose to download them, the file is called `gitlab-recovery-codes.txt`. NOTE: -Recovery codes are not generated for U2F or WebAuthn devices. +Recovery codes are not generated for WebAuthn devices. If you lose the recovery codes, or want to generate new ones, you can use either: @@ -302,18 +357,7 @@ and you're presented with a second prompt, depending on which type of 2FA you've ### Sign in using a one-time password -When asked, enter the pin from your one time password authenticator's application or a recovery code to sign in. - -### Sign in using a U2F device - -To sign in by using a U2F device: - -1. Select **Login via U2F Device**. -1. A light begins blinking on your device. Activate it by touching/pressing - its button. - -A message displays indicating that your device responded to the authentication request, and you're automatically signed -in. +When asked, enter the pin from your one-time password authenticator's application or a recovery code to sign in. ### Sign in using a WebAuthn device @@ -333,7 +377,7 @@ To disable 2FA: 1. Under **Register Two-Factor Authenticator**, enter your current password and select **Disable two-factor authentication**. -This clears all your 2FA registrations, including mobile applications and U2F or WebAuthn devices. +This clears all your 2FA registrations, including mobile applications and WebAuthn devices. ## Recovery options @@ -412,18 +456,18 @@ a GitLab global administrator disable 2FA for your account: ## Information for GitLab administrators **(FREE SELF)** - Take care that 2FA keeps working after [restoring a GitLab backup](../../../raketasks/backup_restore.md). -- To ensure 2FA authorizes correctly with a time-based one time passwords (TOTP) server, synchronize your GitLab +- To ensure 2FA authorizes correctly with a time-based one-time password (TOTP) server, synchronize your GitLab server's time using a service like NTP. Otherwise, authorization can always fail because of time differences. -- The GitLab U2F and WebAuthn implementation does _not_ work when the GitLab instance is accessed from multiple hostnames - or FQDNs. Each U2F or WebAuthn registration is linked to the _current hostname_ at the time of registration, and +- The GitLab WebAuthn implementation does _not_ work when the GitLab instance is accessed from multiple hostnames + or FQDNs. Each WebAuthn registration is linked to the _current hostname_ at the time of registration, and cannot be used for other hostnames or FQDNs. For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: - - The user signs in by using `first.host.xyz` and registers their U2F key. - - The user signs out and attempts to sign in by using `first.host.xyz` - U2F authentication succeeds. - - The user signs out and attempts to sign in by using `second.host.xyz` - U2F authentication fails, because - the U2F key has only been registered on `first.host.xyz`. + - The user signs in by using `first.host.xyz` and registers their WebAuthn key. + - The user signs out and attempts to sign in by using `first.host.xyz` - WebAuthn authentication succeeds. + - The user signs out and attempts to sign in by using `second.host.xyz` - WebAuthn authentication fails, because + the WebAuthn key has only been registered on `first.host.xyz`. - To enforce 2FA at the system or group levels see, [Enforce two-factor authentication](../../../security/two_factor_authentication.md). diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 1f7264d740e..f22accf2ff5 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -40,8 +40,8 @@ To revoke an active session: NOTE: When any session is revoked all **Remember me** tokens for all -devices are revoked. See [Why do you keep getting signed out?](index.md#why-do-you-keep-getting-signed-out) -for more information about the **Remember me** feature. +devices are revoked. For details about **Remember me**, see +[cookies used for sign-in](index.md#cookies-used-for-sign-in). <!-- ## Troubleshooting diff --git a/doc/user/profile/contributions_calendar.md b/doc/user/profile/contributions_calendar.md index e9802cccef6..7b5691e1ee5 100644 --- a/doc/user/profile/contributions_calendar.md +++ b/doc/user/profile/contributions_calendar.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: concepts, howto --- @@ -83,7 +83,7 @@ GitLab provides RSS feeds of user activity. To subscribe to the RSS feed of a user's activity: 1. Go to the [user's profile](index.md#access-your-user-profile). -1. In the upper right, select the feed symbol **{rss}** to display the results as an RSS feed in Atom format. +1. In the upper-right corner, select the feed symbol (**{rss}**) to display the results as an RSS feed in Atom format. The URL of the result contains both a feed token, and the user's activity that you're authorized to view. diff --git a/doc/user/profile/img/saved_replies_dropdown_v15_10.png b/doc/user/profile/img/saved_replies_dropdown_v15_10.png Binary files differnew file mode 100644 index 00000000000..50313f71f4a --- /dev/null +++ b/doc/user/profile/img/saved_replies_dropdown_v15_10.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index a84d16e6d0c..d52f119ba09 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -34,9 +34,13 @@ If you do not want to update the namespace, you can create a new user or group a Prerequisites: -- Your namespace cannot contain a project with [Container Registry](../packages/container_registry/index.md) tags. -- Your namespace cannot have a project that hosts [GitLab Pages](../project/pages/index.md). For more information, - see [this procedure in the GitLab Team Handbook](https://about.gitlab.com/handbook/tools-and-tips/#change-your-username-at-gitlabcom). +- Your namespace must not: + - Contain a project with [Container Registry](../packages/container_registry/index.md) tags. + - Have a project that hosts [GitLab Pages](../project/pages/index.md). For more information, + see [changing your username in the GitLab Team Handbook](https://about.gitlab.com/handbook/tools-and-tips/#change-your-username-at-gitlabcom). +- Your username must be between 2 and 255 characters in length, and must not: + - Contain special characters or emojis. + - End with `.<reserved file extension>`, for example `jon.png`. However, `jonpng` is valid. To change your username: @@ -310,9 +314,9 @@ and configure it on your local machine by using the following command: git config --global user.email <your email address> ``` -## User activity +## Follow users -GitLab tracks [user contribution activity](contributions_calendar.md). You can follow or unfollow other users from either: +You can follow or unfollow users from either: - Their [user profiles](#access-your-user-profile). - The small popover that appears when you hover over a user's name ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76050) @@ -321,66 +325,56 @@ GitLab tracks [user contribution activity](contributions_calendar.md). You can f In [GitLab 15.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/360755), the maximum number of users you can follow is 300. -To view a user's activity in a top-level Activity view: +## View your activity + +GitLab tracks [user contribution activity](contributions_calendar.md). +To view a summary of your activity, or the activity of other users: 1. From a user's profile, select **Follow**. 1. In the GitLab menu, select **Activity**. 1. Select the **Followed users** tab. -## Troubleshooting - -### Why do you keep getting signed out? +## Stay signed in for two weeks -When you sign in to the main GitLab application, a `_gitlab_session` cookie is -set. When you close your browser, the cookie is cleared client-side -and it expires after a set duration. GitLab administrators can determine the duration: +By default, you are signed out of GitLab every seven days, or 10080 minutes. +GitLab administrators can +[change this default](../admin_area/settings/account_and_limit_settings.md#customize-the-default-session-duration). -1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > General**. -1. Expand **Account and limit**. The set duration is in **Session duration (minutes)**. +To extend the duration to two weeks: -The default is `10080`, which equals 7 days. +- On the GitLab sign-in page, select the **Remember me** checkbox. -When you sign in to the main GitLab application, you can also check the -**Remember me** option. This sets the `remember_user_token` -cookie via [`devise`](https://github.com/heartcombo/devise). -The `remember_user_token` cookie expires after -`config/initializers/devise.rb` -> `config.remember_for`. The default is 2 weeks. +## Stay signed in indefinitely -When the `_gitlab_session` expires or isn't available, GitLab uses the `remember_user_token` -to get you a new `_gitlab_session` and keep you signed in through browser restarts. +To remain signed in indefinitely: -After your `remember_user_token` expires and your `_gitlab_session` is cleared/expired, -you are asked to sign in again to verify your identity for security reasons. +1. On the GitLab sign-in page, select the **Remember me** checkbox. +1. Access GitLab at least once every two weeks, and leave your browser open. -NOTE: -When any session is signed out, or when a session is revoked -via [Active Sessions](active_sessions.md), all **Remember me** tokens are revoked. -While other sessions remain active, the **Remember me** feature doesn't restore -a session if the browser is closed or the existing session expires. +You remain signed in because, although the server sets a time-to-live (TTL) of one week on your browser session, +the server continues to reset the TTL, regardless of whether 2FA is installed. -### Increased sign-in time +### Cookies used for sign-in > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20340) in GitLab 13.1. -The `remember_user_token` lifetime of a cookie can now extend beyond the deadline set by `config.remember_for`, as the `config.extend_remember_period` flag is now set to true. +When you sign in, two cookies are set: -GitLab uses both session and persistent cookies: +- A session cookie called `_gitlab_session`. + This cookie has no set expiration date. However, it expires based on its `session_expire_delay`. +- A persistent cookie called `remember_user_token`, which is set only if you selected **Remember me** on the sign-in page. -- Session cookie: Session cookies are typically removed at the end of the browser session when - the browser is closed. The `_gitlab_session` cookie has no fixed expiration date. However, - it expires based on its [`session_expire_delay`](#why-do-you-keep-getting-signed-out). -- Persistent cookie: The `remember_user_token` is a cookie with an expiration date of two weeks. - GitLab activates this cookie if you select **Remember Me** when you sign in. +When you close your browser, the `_gitlab_session` cookie is usually cleared client-side. +When it expires or isn't available, GitLab uses the `remember_user_token`cookie to get you +a new `_gitlab_session` cookie and keep you signed in, even if you close your browser. -By default, the server sets a time-to-live (TTL) of 1-week on any session that is used. +When both cookies are gone or expired, you must sign in again. -When you close a browser, the session cookie may still remain. For example, Chrome has the "Continue where you left off" option that restores session cookies. -In other words, as long as you access GitLab at least once every 2 weeks, you could remain signed in with GitLab, as long as your browser tab is open. -The server continues to reset the TTL for that session, independent of whether 2FA is installed, -If you close your browser and open it up again, the `remember_user_token` cookie allows your user to reauthenticate itself. - -Without the `config.extend_remember_period` flag, you would be forced to sign in again after two weeks. +NOTE: +When any session is signed out, or when a session is revoked +from the [active sessions list](active_sessions.md), all **Remember me** tokens are revoked. +While other sessions remain active, the **Remember me** feature doesn't restore +a session if the browser is closed or the existing session expires. ## Related topics @@ -389,7 +383,7 @@ Without the `config.extend_remember_period` flag, you would be forced to sign in - [Change your password](user_passwords.md) - Receive emails for: - [Sign-ins from unknown IP addresses or devices](notifications.md#notifications-for-unknown-sign-ins) - - [Attempted sign-ins using wrong two-factor authentication codes](notifications.md#notifications-for-attempted-sign-in-using-wrong-two-factor-authentication-codes) + - [Attempted sign-ins using incorrect verification codes](notifications.md#notifications-for-attempted-sign-ins-using-incorrect-verification-codes) - Manage applications that can [use GitLab as an OAuth provider](../../integration/oauth_provider.md) - Manage [personal access tokens](personal_access_tokens.md) to access your account via API and authorized applications - Manage [SSH keys](../ssh.md) to access your account via SSH diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index dcc78dac05b..e6675b4eff2 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -286,7 +286,8 @@ To always receive notifications on your own issues, merge requests, and so on, t ## Notifications for unknown sign-ins -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27211) in GitLab 13.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27211) in GitLab 13.0. +> - Listing the full name and username of the signed-in user [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225183) in GitLab 15.10. NOTE: This feature is enabled by default for self-managed instances. Administrators may disable this feature @@ -295,7 +296,12 @@ The feature is always enabled on GitLab.com. When a user successfully signs in from a previously unknown IP address or device, GitLab notifies the user by email. In this way, GitLab proactively alerts users of potentially -malicious or unauthorized sign-ins. +malicious or unauthorized sign-ins. This notification email includes the: + +- Hostname. +- User's name and username. +- IP address. +- Date and time of sign-in. GitLab uses several methods to identify a known sign-in. All methods must fail for a notification email to be sent. @@ -306,7 +312,7 @@ GitLab uses several methods to identify a known sign-in. All methods must fail f - Cookie: After successful sign in, an encrypted cookie is stored in the browser. This cookie is set to expire 14 days after the last successful sign in. -## Notifications for attempted sign-in using wrong two-factor authentication codes +## Notifications for attempted sign-ins using incorrect verification codes > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/374740) in GitLab 15.5. diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 3826c602fb4..0c733b8de30 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -45,9 +45,6 @@ For examples of how you can use a personal access token to authenticate with the Alternately, GitLab administrators can use the API to create [impersonation tokens](../../api/rest/index.md#impersonation-tokens). Use impersonation tokens to automate authentication as a specific user. -NOTE: -Personal access tokens are not FIPS compliant and creation and use are disabled when [FIPS mode](../../development/fips_compliance.md) is enabled. - ## Create a personal access token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348660) in GitLab 15.3, default expiration of 30 days is populated in the UI. @@ -225,7 +222,7 @@ Remember this if you set up an automation pipeline that depends on authenticatio ### Unrevoke a personal access token **(FREE SELF)** -If a personal access token is revoked accidentally by any method, administrators can unrevoke that token. +If a personal access token is revoked accidentally by any method, administrators can unrevoke that token. By default, a daily job deletes revoked tokens at 1:00 AM system time. WARNING: Running the following commands changes data directly. This could be damaging if not done correctly, or under the right conditions. You should first run these commands in a test environment with a backup of the instance ready to be restored, just in case. diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 7e581bb7419..0f46c3d6d25 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -63,6 +63,8 @@ Dark theme only works with the **Dark** syntax highlighting theme. ## Syntax highlighting theme +> Changing the default syntax highlighting theme for new users and users who are not signed in [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25129) in GitLab 15.10. + GitLab uses the [rouge Ruby library](http://rouge.jneen.net/ "Rouge website") for syntax highlighting outside of any Editor context. The WebIDE (like Snippets) uses [Monaco Editor](https://microsoft.github.io/monaco-editor/) and it's provided @@ -89,6 +91,10 @@ The default syntax theme is White, and you can choose among 5 different themes: Introduced in GitLab 13.6, the themes [Solarized](https://gitlab.com/gitlab-org/gitlab/-/issues/221034) and [Monokai](https://gitlab.com/gitlab-org/gitlab/-/issues/221034) also apply to the [Web IDE](../project/web_ide/index.md) and [Snippets](../snippets.md). +You can use an API call to change the default syntax highlighting theme for new users and users +who are not signed in. For more information, see the `default_syntax_highlighting_theme` +in the [list of settings that can be accessed through API calls](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). + ## Diff colors A diff compares the old/removed content with the new/added content (for example, when diff --git a/doc/user/profile/saved_replies.md b/doc/user/profile/saved_replies.md new file mode 100644 index 00000000000..302daceb31d --- /dev/null +++ b/doc/user/profile/saved_replies.md @@ -0,0 +1,61 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Saved replies **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352956) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `saved_replies`. Disabled by default. + +With saved replies, create and reuse text for any text area in: + +- Merge requests, including diffs. +- Issues, including design management comments. +- Epics. +- Work items. + +Saved replies can be small, like approving a merge request and unassigning yourself from it, +or large, like chunks of boilerplate text you use frequently: + + + +## Use saved replies in a text area + +To include the text of a saved reply in your comment: + +1. In the editor toolbar for your comment, select **Saved replies** (**{symlink}**). +1. Select your desired saved reply. + +## Create saved replies + +To create a saved reply for future use: + +1. On the top bar, in the upper-right corner, select your avatar. +1. From the dropdown list, select **Preferences**. +1. On the left sidebar, select **Saved replies** (**{symlink}**). +1. Provide a **Name** for your saved reply. +1. Enter the **Content** of your reply. You can use any formatting you use in + other GitLab text areas. +1. Select **Save**, and the page reloads with your saved reply shown. + +## View your saved replies + +To go to your saved replies: + +1. On the top bar, in the upper-right corner, select your avatar. +1. From the dropdown list, select **Preferences**. +1. On the left sidebar, select **Saved replies** (**{symlink}**). +1. Scroll to **My saved replies**. + +## Edit or delete saved replies + +To edit or delete a previously saved reply: + +1. On the top bar, in the upper-right corner, select your avatar. +1. From the dropdown list, select **Preferences**. +1. On the left sidebar, select **Saved replies** (**{symlink}**). +1. Scroll to **My saved replies**, and identify the saved reply you want to edit. +1. To edit, select **Edit** (**{pencil}**). +1. To delete, select **Delete** (**{remove}**), then select **Delete** again from the modal window. diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 0ea1a80bc54..2d36a378b82 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -117,6 +117,10 @@ time with the `?order_by` query parameter. https://gitlab.example.com/<namespace>/<project>/-/badges/release.svg?order_by=release_at ``` +You can change the width of the release name field by using the `value_width` parameter ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113615) in GitLab 15.10). +The value must be between 1 and 200, and the default value is 54. +If you set an out of range value, GitLab automatically adjusts it to the default value. + ## Project badges Badges can be added to a project by Maintainers or Owners, and are visible on the project's overview page. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index 52288af101a..3ed8a4a67cf 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Connect EKS clusters through cluster certificates (DEPRECATED) **(FREE)** +# Connect EKS clusters through cluster certificates (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22392) in GitLab 12.5. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -25,7 +25,7 @@ use the [GitLab agent](../../clusters/agent/index.md). To create a new cluster from GitLab, use [Infrastructure as Code](../../infrastructure/iac/index.md). -### How to create a new cluster on EKS through cluster certificates (DEPRECATED) +### How to create a new cluster on EKS through cluster certificates (deprecated) > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 0b0555806ce..a7fac2a1cea 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Connect existing clusters through cluster certificates (DEPRECATED) **(FREE)** +# Connect existing clusters through cluster certificates (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index b1c5bbfc4f7..e00c4370da9 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Connect GKE clusters through cluster certificates (DEPRECATED) **(FREE)** +# Connect GKE clusters through cluster certificates (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index d3048158958..bf2f6e28d9a 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Add a cluster using cluster certificates (DEPRECATED) **(FREE)** +# Add a cluster using cluster certificates (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. diff --git a/doc/user/project/clusters/cluster_access.md b/doc/user/project/clusters/cluster_access.md index 529e7a6da12..0eea3ebb8cd 100644 --- a/doc/user/project/clusters/cluster_access.md +++ b/doc/user/project/clusters/cluster_access.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Access controls with cluster certificates (RBAC or ABAC) (DEPRECATED) **(FREE)** +# Access controls with cluster certificates (RBAC or ABAC) (deprecated) **(FREE)** > - Restricted service account for deployment was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51716) in GitLab 11.5. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/project/clusters/deploy_to_cluster.md b/doc/user/project/clusters/deploy_to_cluster.md index 173f1f39a66..e2d0a010293 100644 --- a/doc/user/project/clusters/deploy_to_cluster.md +++ b/doc/user/project/clusters/deploy_to_cluster.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Deploy to a Kubernetes cluster with cluster certificates (DEPRECATED) **(FREE)** +# Deploy to a Kubernetes cluster with cluster certificates (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/project/clusters/gitlab_managed_clusters.md b/doc/user/project/clusters/gitlab_managed_clusters.md index b2a4bd85de4..46e74eb8460 100644 --- a/doc/user/project/clusters/gitlab_managed_clusters.md +++ b/doc/user/project/clusters/gitlab_managed_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab-managed clusters (DEPRECATED) **(FREE)** +# GitLab-managed clusters (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 11.5. > - Became [optional](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26565) in GitLab 11.11. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 94513fc7124..490188b27b0 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Project-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE)** +# Project-level Kubernetes clusters (certificate-based) (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. > - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. diff --git a/doc/user/project/clusters/multiple_kubernetes_clusters.md b/doc/user/project/clusters/multiple_kubernetes_clusters.md index c79f250da7a..9167e6197d6 100644 --- a/doc/user/project/clusters/multiple_kubernetes_clusters.md +++ b/doc/user/project/clusters/multiple_kubernetes_clusters.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Multiple clusters per project with cluster certificates (DEPRECATED) **(FREE)** +# Multiple clusters per project with cluster certificates (deprecated) **(FREE)** > - Introduced in GitLab 10.3 > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) from GitLab Premium to GitLab Free in 13.2. diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index df0ffff8561..b1f2a9dbb79 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -107,7 +107,7 @@ the components outlined above and the pre-loaded demo runbook. ''' We set user's id, login and access token on single user image to enable repository integration for JupyterHub. - See: https://gitlab.com/gitlab-org/gitlab-foss/issues/47138#note_154294790 + See: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47138#note_154294790 ''' auth_state = await spawner.user.get_auth_state() diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index 9de9d445965..0994bff4aa2 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -8,12 +8,32 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Moved to GitLab Premium in 13.9. -Code Owners define who develops and maintains a feature, and own the resulting -files or directories in a repository. +Use the Code Owners feature to define who has expertise for specific parts of your project's codebase. +Define the owners of files and directories in a repository to: -- The users you define as Code Owners are displayed in the UI when you browse directories. -- You can set your merge requests so they must be approved by Code Owners before merge. -- You can protect a branch and allow only Code Owners to approve changes to the branch. +- **Require owners to approve changes.** Combine protected branches with Code Owners to require + experts to approve merge requests before they merge into a protected branch. +- **Identify owners.** Code Owner names are displayed on the files and directories they own: +  + +Use Code Owners in combination with merge request +[approval rules](merge_requests/approvals/rules.md) (either optional or required) +to build a flexible approval workflow: + +- Use **Code Owners** to ensure quality. Define the users who have domain expertise + for specific paths in your repository. +- Use **Approval rules** to define areas of expertise that don't correspond to specific + file paths in your repository. Approval rules help guide merge request creators to + the correct set of reviewers, such as frontend developers or a security team. + +For example: + +| Type | Name | Scope | Comment | +|------|------|--------|------------| +| Approval rule | UX | All files | A user experience (UX) team member reviews the user experience of all changes made in your project. +| Approval rule | Security | All files | A security team member reviews all changes for vulnerabilities. +| Code Owner approval rule | Frontend: Code Style | `*.css` files | A frontend engineer reviews CSS file changes for adherence to project style standards. +| Code Owner approval rule | Backend: Code Review | `*.rb` files | A backend engineer reviews the logic and code style of Ruby files. <div class="video-fallback"> Video introduction: <a href="https://www.youtube.com/watch?v=RoyBySTUSB0">Code Owners</a>. @@ -24,45 +44,20 @@ files or directories in a repository. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -Use Code Owners and approvers together with -[approval rules](merge_requests/approvals/rules.md) to build a flexible approval -workflow: - -- Use **Code Owners** to define the users who have domain expertise for specific paths in your repository. -- Use **Approvers** and **Approval rules** to define domains of expertise (such as a security team) - that are not scoped to specific file paths in your repository. - - **Approvers** define the users. - - **Approval rules** define when these users can approve work, and whether or not their approval is required. - -For example: - -| Type | Name | Scope | Comment | -|------|------|--------|------------| -| Approval rule | UX | All files | A user experience (UX) team member reviews the user experience of all changes made in your project. | -| Approval rule | Security | All files | A security team member reviews all changes for vulnerabilities. | -| Code Owner approval rule | Frontend: Code Style | `*.css` files | A frontend engineer reviews CSS file changes for adherence to project style standards. | -| Code Owner approval rule | Backend: Code Review | `*.rb` files | A backend engineer reviews the logic and code style of Ruby files. | - -## Code Owners file +## View a file's Code Owner -A `CODEOWNERS` file (with no extension) can specify users or [shared groups](members/share_project_with_groups.md) -that are responsible for specific files and directories in a repository. Each repository -can have a single `CODEOWNERS` file, and it must be found one of these three locations: +To view the Code Owners for a file: -- In the root directory of the repository. -- In the `.gitlab/` directory. -- In the `docs/` directory. +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Files**. +1. Go to the file or directory you want to see the Code Owners for. +1. Optional. Select your desired branch. -A CODEOWNERS file in any other location is ignored. +GitLab shows the Code Owners at the top of the page. ## Set up Code Owners -1. Create a file named `CODEOWNERS` (with no extension) in one of these locations: - -- In the root directory of the repository -- In the `.gitlab/` directory -- In the `docs/` directory - +1. Create a `CODEOWNERS` file in your [preferred location](#code-owners-file). 1. In the file, enter text that follows one of these patterns: ```plaintext @@ -79,17 +74,28 @@ A CODEOWNERS file in any other location is ignored. directoryname/ @groupname ``` -The Code Owners are now displayed in the UI. They apply to the current branch only. +### Code Owners file + +A `CODEOWNERS` file (with no extension) specifies the users or +[shared groups](members/share_project_with_groups.md) responsible for +specific files and directories in a repository. + +Each repository uses a single `CODEOWNERS` file. GitLab checks these locations +in your repository in this order. The first `CODEOWNERS` file found is used, and +all others are ignored: -Next steps: +1. In the root directory: `./CODEOWNERS`. +1. In the `docs` directory: `./docs/CODEOWNERS`. +1. In the `.gitlab` directory: `./.gitlab/CODEOWNERS`. + +### Make Code Owners eligible approvers or require their approval of MRs - [Add Code Owners as merge request approvers](merge_requests/approvals/rules.md#code-owners-as-eligible-approvers). - Set up [Code Owner approval on a protected branch](protected_branches.md#require-code-owner-approval-on-a-protected-branch). -## Groups as Code Owners +### Groups as Code Owners -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53182) in GitLab 12.1. -> - Group and subgroup hierarchy support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32432) in GitLab 13.0. +> Group and subgroup hierarchy support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32432) in GitLab 13.0. You can use members of groups and subgroups as Code Owners for projects: @@ -137,7 +143,7 @@ For approval to be required, groups as Code Owners must have a direct membership that inherit membership. Members in the Code Owners group also must be direct members, and not inherit membership from any parent groups. -### Add a group as a Code Owner +#### Add a group as a Code Owner To set a group as a Code Owner: @@ -154,23 +160,26 @@ file.md @group-x/subgroup-y file.md @group-x @group-x/subgroup-y ``` -## When a file matches multiple `CODEOWNERS` entries +### Define more specific owners for more specifically defined files or directories -When a file matches multiple entries in the `CODEOWNERS` file, -the users from last pattern matching the file are used. +When a file or directory matches multiple entries in the `CODEOWNERS` file, +the users from last pattern matching the file or directory are used. This enables you +to define more specific owners for more specifically defined files or directories, when +you order the entries in a sensible way. For example, in the following `CODEOWNERS` file: ```plaintext -README.md @user1 +# This line would match the file terms.md +*.md @doc-team -# This line would also match the file README.md -*.md @user2 +# This line would also match the file terms.md +terms.md @legal-team ``` -The Code Owner for `README.md` would be `@user2`. +The Code Owner for `terms.md` would be `@legal-team`. -If you use sections, the last pattern matching the file for each section is used. +If you use sections, the last pattern matching the file or directory for each section is used. For example, in a `CODEOWNERS` file using sections: ```plaintext @@ -183,9 +192,9 @@ README.md @user3 ``` The Code Owners for the `README.md` in the root directory are `@user1`, `@user2`, -and `@user3`. The Code Owners for `internal/README.md` are `@user4` and `@user3`. +and `@user3`. The Code Owners for `internal/README.md` are `@user4` and `@user3`. -Only one CODEOWNERS pattern can match per file path. +Only one CODEOWNERS pattern per section is matched to a file path. ### Organize Code Owners by putting them into sections @@ -211,7 +220,56 @@ The following image shows a **Groups** and **Documentation** section:  -### Sections with duplicate names +#### Set default owner for a section **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/371711) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `codeowners_default_owners`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `codeowners_default_owners`. +The feature is not ready for production use. + +WARNING: +To disable this feature flag after setting default owners per section, edit your +CODEOWNERS file to [list Code Owners per line](#set-up-code-owners). + +If multiple file paths inside a section share the same ownership, define a default +Code Owner for the section. All paths in that section inherit this default, unless +you override the section default on a specific line. + +Default owners are applied when specific owners are not specified for file paths. +Specific owners defined beside the file path override default owners: + +```plaintext +[Documentation] @docs-team +docs/ +README.md + +[Database] @database-team +model/db/ +config/db/database-setup.md @docs-team +``` + +In this example: + +- `@docs-team` owns all items in the `Documentation` section. +- `@database-team` owns all items in the `Database` section except + `config/db/database-setup.md`, which has an override assigning it to `@docs-team`. + +To combine the syntax for default owners with [optional sections](#make-a-code-owners-section-optional) +and required approvals, place default owners at the end: + +```plaintext +[Documentation][2] @docs-team +docs/ +README.md + +^[Database] @database-team +model/db/ +config/db/database-setup.md @docs-team +``` + +#### Sections with duplicate names If multiple sections have the same name, they are combined. Also, section headings are not case-sensitive. For example: @@ -233,7 +291,7 @@ This code results in three entries under the **Documentation** section header, a entries under **Database**. The entries defined under the sections **Documentation** and **DOCUMENTATION** are combined, using the case of the first section. -### Make a Code Owners section optional +#### Make a Code Owners section optional > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232995) in GitLab 13.8. @@ -268,6 +326,35 @@ when changes are submitted by using merge requests. If a change is submitted dir to the protected branch, approval from Code Owners is still required, even if the section is marked as optional. +### Require multiple approvals from Code Owners + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335451) in GitLab 15.9. + +You can require multiple approvals for the Code Owners sections under the Approval Rules area in merge requests. +Append the section name with a number `n` in brackets. This requires `n` approvals from the Code Owners in this section. +Please note valid entries for `n` are integers `≥ 1`. `[1]` is optional as it is the default. Invalid values for `n` are treated as `1`. + +WARNING: +[Issue #384881](https://gitlab.com/gitlab-org/gitlab/-/issues/385881) proposes changes +to the behavior of this setting. Do not intentionally set invalid values. They may +become valid in the future, and cause unexpected behavior. + +Please confirm you enabled `Require approval from code owners` in `Settings > Repository > Protected branches`, otherwise the Code Owner approvals will be optional. + +In this example, the `[Documentation]` section requires 2 approvals: + +```plaintext +[Documentation][2] +*.md @tech-writer-team + +[Ruby] +*.rb @dev-team +``` + +The `Documentation` Code Owners section under the **Approval Rules** area displays 2 approvals are required: + + + ### Allowed to Push The Code Owner approval and protected branch features do not apply to users who @@ -338,15 +425,22 @@ path\ with\ spaces/ @space-owner ee/docs @docs docs @docs -[Database] -README.md @database -model/db @database +# Use of default owners for a section. In this case, all files (*) are owned by +the dev team except the README.md and data-models which are owned by other teams. +[Development] @dev-team +* +README.md @docs-team +data-models/ @data-science-team # This section is combined with the previously defined [Documentation] section: [DOCUMENTATION] README.md @docs ``` +## Technical Resources + +[Code Owners development guidelines](../../../ee/development/code_owners/index.md) + ## Troubleshooting ### Approvals shown as optional @@ -371,3 +465,9 @@ if any of these conditions are true: Check the project [merge request approval](merge_requests/approvals/settings.md#edit-merge-request-approval-settings) settings. - A Code Owner group has a visibility of **private**, and the current user is not a member of the Code Owner group. +- Current user is an external user who does not have permission to the internal Code Owner group. + +### Approval rule is invalid. GitLab has approved this rule automatically to unblock the merge request + +This message may appear if an approval rule uses a Code Owner that is not a direct member of the project. +Check that the group or user has been invited to the project. diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index a68f9550ebf..5adc678e942 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -1,11 +1,11 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: howto, reference --- -# Deploy boards (DEPRECATED) **(FREE)** +# Deploy boards (deprecated) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1589) in GitLab 9.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212320) from GitLab Premium to GitLab Free in 13.8. diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index fc88535dc77..362d5826124 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -15,8 +15,7 @@ Depending on your needs, you might want to use a [deploy token](../deploy_tokens |------------------|-------------|--------------| | Sharing | Shareable between multiple projects, even those in different groups. | Belong to a project or group. | | Source | Public SSH key generated on an external host. | Generated on your GitLab instance, and is provided to users only at creation time. | -| Validity | Valid as long as it's registered and enabled, and the user that created it exists. | Can be given an expiration date. | -| Registry access | Cannot access a package registry. | Can read from and write to a package registry. | +| Accessible resources | Git repository over SSH | Git repository over HTTP, package registry, and container registry. | Deploy keys can't be used for Git operations if [external authorization](../../admin_area/settings/external_authorization.md) is enabled. @@ -41,10 +40,8 @@ A deploy key is given a permission level when it is created: You can change a deploy key's permission level after creating it. Changing a project deploy key's permissions only applies for the current project. -When a read-write deploy key is used to push a commit, GitLab checks if the creator of the -deploy key has permission to access the resource. - -For example: +Although a deploy key is a secret that isn't associated with a specific user, +GitLab authorizes the creator of the deploy key if the Git-command triggers additional processes. For example: - When a deploy key is used to push a commit to a [protected branch](../protected_branches.md), the _creator_ of the deploy key must have access to the branch. @@ -52,6 +49,15 @@ For example: deploy key must have access to the CI/CD resources, including protected environments and secret variables. +## Security implications + +The intended use case for deploy keys is for non-human interaction with GitLab, for example: an automated script running on a server in your organization. +As with all sensitive information, you should ensure only those who need access to the secret can read it. +For human interactions, use credentials tied to users such as Personal Access Tokens. + +To help detect a potential secret leak, you can use the +[Audit Event](../../../administration/audit_event_streaming.md#example-payloads-for-ssh-events-with-deploy-key) feature. + ## View deploy keys To view the deploy keys available to a project: @@ -88,9 +94,9 @@ name and permissions. Prerequisites: -- You must have administrator access. -- [Generate an SSH key pair](../../ssh.md#generate-an-ssh-key-pair). Put the private SSH - key on the host that requires access to the repository. +- You must have administrator access to the instance. +- You must [generate an SSH key pair](../../ssh.md#generate-an-ssh-key-pair). +- You must put the private SSH key on the host that requires access to the repository. To create a public deploy key: diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index cfb382d73e2..23cfa7cc500 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index ffbe7447aa8..eeaf5d799d2 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -87,6 +87,10 @@ For example: `https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_templat > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89810) in GitLab 15.7. +NOTE: +This feature is available only for +[the default template](#set-a-default-template-for-merge-requests-and-issues). + When you save a merge request for the first time, GitLab replaces these variables in your merge request template with their values: diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 8d3446994e8..b83feda0c96 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -209,7 +209,7 @@ requests that modify locked files. Unlock the file to allow changes. To lock a file: 1. Open the file or directory in GitLab. -1. In the upper right, above the file, select **Lock**. +1. In the upper-right corner, above the file, select **Lock**. 1. On the confirmation dialog box, select **OK**. If you do not have permission to lock the file, the button is not enabled. @@ -221,7 +221,7 @@ To view the user who locked the file (if it was not you), hover over the button. To view and remove file locks: 1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Repository > Locked Files**. +1. On the left sidebar, select **Repository > Locked files**. This list shows all the files locked either through LFS or GitLab UI. diff --git a/doc/user/project/img/codeowners_in_UI_v15_10.png b/doc/user/project/img/codeowners_in_UI_v15_10.png Binary files differnew file mode 100644 index 00000000000..c643d333791 --- /dev/null +++ b/doc/user/project/img/codeowners_in_UI_v15_10.png diff --git a/doc/user/project/img/multi_approvals_code_owners_sections_v15_9.png b/doc/user/project/img/multi_approvals_code_owners_sections_v15_9.png Binary files differnew file mode 100644 index 00000000000..a7fea76d5b1 --- /dev/null +++ b/doc/user/project/img/multi_approvals_code_owners_sections_v15_9.png diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 7114974d8db..a5d5f93f9ad 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -35,6 +35,8 @@ When importing: - [Bitbucket Cloud integration](../../../integration/bitbucket.md) must be enabled. If that integration is not enabled, ask your GitLab administrator to enable it. The Bitbucket Cloud integration is enabled by default on GitLab.com. +- [Bitbucket Cloud import source](../../admin_area/settings/visibility_and_access_controls.md#configure-allowed-import-sources) must be enabled. If not enabled, ask your + GitLab administrator to enable it. The Bitbucket Cloud import source is enabled by default on GitLab.com. - At least the Maintainer role on the destination group to import to. Using the Developer role for this purpose was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387891) in GitLab 15.8 and will be removed in GitLab 16.0. diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index e6d2e3e00b6..9937ea956c4 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -120,6 +120,16 @@ If the project import completes but LFS objects can't be downloaded or cloned, y password or personal access token containing special characters. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/337769). +### Pull requests are missing + +Importing large projects spawns a process that can consume a lot of memory. Sometimes you can see messages such as `Sidekiq worker RSS out of range` in the +[Sidekiq logs](../../../administration/logs/index.md#sidekiq-logs). This can mean that MemoryKiller is interrupting the `RepositoryImportWorker` because it's using +too much memory. + +To resolve this problem, temporarily increase the `SIDEKIQ_MEMORY_KILLER_MAX_RSS` environment variable using +[custom environment variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html) from the default `2000000` value to a larger value like `3000000`. +Consider memory constraints on the system before deciding to increase `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. + ## Related topics For information on automating user, group, and project import API calls, see diff --git a/doc/user/project/import/gitea.md b/doc/user/project/import/gitea.md index 404bb4c8600..29fe9db8887 100644 --- a/doc/user/project/import/gitea.md +++ b/doc/user/project/import/gitea.md @@ -75,10 +75,9 @@ From there, you can view the import statuses of your Gitea repositories: You also can: -- Import all of your Gitea projects in one go by selecting **Import all projects** - in the upper left corner. -- Filter projects by name. If filter is applied, selecting **Import all projects** - imports only matched projects. +- In the upper-left corner, select **Import all projects** to import all of your Gitea projects at once. +- Filter projects by name. If a filter is applied, **Import all projects** + imports only selected projects.  diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index eeebb5a166c..a1d94d81e69 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -30,10 +30,8 @@ When importing projects: imported with a naming scheme similar to `GH-SHA-username/pull-request-number/fork-name/branch`. This may lead to a discrepancy in branches compared to those of the GitHub repository. -For additional technical details, refer to the [GitHub Importer](../../../development/github_importer.md) -developer documentation. - -For an overview of the import process, see the video [Migrating from GitHub to GitLab](https://youtu.be/VYOXuOg9tQI). +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview of the import process, see [Migrating from GitHub to GitLab](https://youtu.be/VYOXuOg9tQI). ## Prerequisites @@ -62,7 +60,7 @@ prerequisites for those imports. If you are importing from GitHub Enterprise to a self-managed GitLab instance: - You must first enable the [GitHub integration](../../../integration/github.md). -- If GitLab is behind a HTTP/HTTPS proxy, you must populate the [allowlist for local requests](../../../security/webhooks.md#create-an-allowlist-for-local-requests) +- If GitLab is behind an HTTP/HTTPS proxy, you must populate the [allowlist for local requests](../../../security/webhooks.md#allow-outbound-requests-to-certain-ip-addresses-and-domains) with `github.com` and `api.github.com` to solve the hostname. For more information, read the issue [Importing a GitHub project requires DNS resolution even when behind a proxy](https://gitlab.com/gitlab-org/gitlab/-/issues/37941). @@ -209,17 +207,18 @@ The following items of a project are imported: - Repository description. - Git repository data. - Branch protection rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22650) in GitLab 15.4. +- Collaborators (members). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388716) in GitLab 15.10. - Issues. - Pull requests. - Wiki pages. - Milestones. - Labels. -- Release note descriptions. +- Release notes content. - Attachments for: - Release notes. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15620) in GitLab 15.4. - - Comments and notes. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. + - Comments. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. - Issue description. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. - - Merge Request description. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. + - Pull Request description. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18052) in GitLab 15.5. All attachment imports are disabled by default behind `github_importer_attachments_import` [feature flag](../../../administration/feature_flags.md). From GitLab 15.5, can @@ -232,7 +231,7 @@ The following items of a project are imported: - Pull request "merged by" information. - Pull request comments replies in discussions. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336596) in GitLab 14.5. -- Diff Notes suggestions. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340624) in GitLab 14.7. +- Pull request review comments suggestions. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340624) in GitLab 14.7. - Issue events and pull requests events. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7673) in GitLab 15.4 with `github_importer_issue_events_import` [feature flag](../../../administration/feature_flags.md) disabled by default. From GitLab 15.5, can be imported [as an additional item](#select-additional-items-to-import). The feature flag was @@ -267,39 +266,25 @@ Mapping GitHub rule **Require status checks to pass before merging** to into GitLab due to technical difficulties. You can still create [external status checks](../merge_requests/status_checks.md) manually. -## Alternative way to import notes and diff notes - -When GitHub Importer runs on extremely large projects not all notes & diff notes can be imported due to GitHub API `issues_comments` & `pull_requests_comments` endpoints limitation. -Not all pages can be fetched due to the following error coming from GitHub API: `In order to keep the API fast for everyone, pagination is limited for this resource. Check the rel=last link relation in the Link response header to see how far back you can traverse.`. - -An [alternative approach](#select-additional-items-to-import) for importing comments is available. - -Instead of using `issues_comments` and `pull_requests_comments`, use individual resources `issue_comments` and `pull_request_comments` instead to pull notes from one object at a time. -This allows us to carry over any missing comments, however it increases the number of network requests required to perform the import, which means its execution takes a longer time. +### Collaborators (members) -## Reduce GitHub API request objects per page +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388716) in GitLab 15.10. -Some GitHub API endpoints may return a 500 or 502 error for project imports from large repositories. -To reduce the chance of such errors, you can enable the feature flag -`github_importer_lower_per_page_limit` in the group project importing the data. This reduces the -page size from 100 to 50. +These GitHub collaborator roles are mapped to these GitLab [member roles](../../permissions.md#roles): -To enable this feature flag, start a [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -and run the following `enable` command: +| GitHub role | Mapped GitLab role | +|:------------|:-------------------| +| Read | Guest | +| Triage | Reporter | +| Write | Developer | +| Maintain | Maintainer | +| Admin | Owner | -```ruby -group = Group.find_by_full_path('my/group/fullpath') - -# Enable -Feature.enable(:github_importer_lower_per_page_limit, group) -``` +GitHub Enterprise Cloud has +[custom repository roles](https://docs.github.com/en/enterprise-cloud@latest/organizations/managing-peoples-access-to-your-organization-with-roles/about-custom-repository-roles). +These roles aren't supported and cause partial imports. -To disable the feature, run this command: - -```ruby -# Disable -Feature.disable(:github_importer_lower_per_page_limit, group) -``` +To import GitHub collaborators, you must have at least the Write role on the GitHub project. Otherwise collaborators import is skipped. ## Import from GitHub Enterprise on an internal network @@ -443,3 +428,47 @@ repository to be imported manually. Administrators can manually import the repos # Trigger import from second step Gitlab::GithubImport::Stage::ImportRepositoryWorker.perform_async(project.id) ``` + +### Errors when importing large projects + +The GitHub importer might encounter some errors when importing large projects. + +#### Alternative way to import notes and diff notes + +When the GitHub importer runs on extremely large projects, not all notes and diff notes can be imported due to the GitHub API `issues_comments` and `pull_requests_comments` endpoint limitations. + +If it's not possible to fetch all pages, the GitHub API might return the following error: + +```plaintext +In order to keep the API fast for everyone, pagination is limited for this resource. Check the rel=last link relation in the Link response header to see how far back you can traverse. +``` + +An [alternative approach](#select-additional-items-to-import) for importing comments is available. + +Instead of using `issues_comments` and `pull_requests_comments`, use individual resources to pull notes from one object at a time. This way, you can carry over any missing comments. However, execution takes longer because this method increases the number of network requests required to perform the import. + +#### Reduce GitHub API request objects per page + +Some GitHub API endpoints might return a `500` or `502` error for project imports from large repositories. +To reduce the chance of these errors, in the group project importing the data, enable the +`github_importer_lower_per_page_limit` feature flag. When enabled, the flag reduces the +page size from `100` to `50`. + +To enable this feature flag: + +1. Start a [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Run the following `enable` command: + + ```ruby + group = Group.find_by_full_path('my/group/fullpath') + + # Enable + Feature.enable(:github_importer_lower_per_page_limit, group) + ``` + +To disable the feature flag, run this command: + +```ruby +# Disable +Feature.disable(:github_importer_lower_per_page_limit, group) +``` diff --git a/doc/user/project/import/img/bitbucket_import_select_project_v12_3.png b/doc/user/project/import/img/bitbucket_import_select_project_v12_3.png Binary files differdeleted file mode 100644 index bbc72a0b4b7..00000000000 --- a/doc/user/project/import/img/bitbucket_import_select_project_v12_3.png +++ /dev/null diff --git a/doc/user/project/import/img/fogbugz_import_finished.png b/doc/user/project/import/img/fogbugz_import_finished.png Binary files differdeleted file mode 100644 index 62c5c86c9b3..00000000000 --- a/doc/user/project/import/img/fogbugz_import_finished.png +++ /dev/null diff --git a/doc/user/project/import/img/manifest_status_v13_3.png b/doc/user/project/import/img/manifest_status_v13_3.png Binary files differdeleted file mode 100644 index c1a55ba1f50..00000000000 --- a/doc/user/project/import/img/manifest_status_v13_3.png +++ /dev/null diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 24748b2e89c..8c95e68e1f2 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -109,7 +109,7 @@ To view project import history: 1. On the top bar, select **Create new...** (**{plus-square}**). 1. Select **New project/repository**. 1. Select **Import project**. -1. Select **History** in the upper right corner. +1. In the upper-right corner, select **History**. 1. If there are any errors for a particular import, you can see them by selecting **Details**. The history also includes projects created from [built-in](../index.md#create-a-project-from-a-built-in-template) diff --git a/doc/user/project/index.md b/doc/user/project/index.md index decf3b071e7..da0546b85e6 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- @@ -139,9 +139,8 @@ Prerequisites: - You must have permission to add new projects to a namespace. To check if you have permission: 1. On the top bar, select **Main menu > Groups** and find your group. - 1. Confirm that **New project** is visible in the upper right - corner. Contact your GitLab - administrator if you require permission. + 1. In the upper-right corner, confirm that **New project** is visible. + Contact your GitLab administrator if you require permission. To push your repository and create a project: diff --git a/doc/user/project/integrations/apple_app_store.md b/doc/user/project/integrations/apple_app_store.md index 1b41dd0b669..e326e7a222b 100644 --- a/doc/user/project/integrations/apple_app_store.md +++ b/doc/user/project/integrations/apple_app_store.md @@ -6,12 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Apple App Store **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104888) in GitLab 15.8 [with a flag](../../../administration/feature_flags.md) named `apple_app_store_integration`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104888) in GitLab 15.8 [with a flag](../../../administration/feature_flags.md) named `apple_app_store_integration`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/385335) in GitLab 15.10. Feature flag `apple_app_store_integration` removed. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `apple_app_store_integration`. On GitLab.com, this feature is not available. - -The Apple App Store integration makes it easy to configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com) to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. +With the Apple App Store integration, you can configure your CI/CD pipelines to connect to [App Store Connect](https://appstoreconnect.apple.com) to build and release apps for iOS, iPadOS, macOS, tvOS, and watchOS. The integration is designed to be able to work out of the box with [fastlane](http://fastlane.tools/), but can be used with other build tools as well. diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md index 8bc1a984e3d..fac09e7f071 100644 --- a/doc/user/project/integrations/asana.md +++ b/doc/user/project/integrations/asana.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Asana **(FREE)** -This integration adds commit messages as comments to Asana tasks. +The Asana integration adds commit messages as comments to Asana tasks. Once enabled, commit messages are checked for Asana task URLs (for example, `https://app.asana.com/0/123456/987654`) or task IDs starting with `#` (for example, `#987654`). Every task ID found gets the commit comment added to it. diff --git a/doc/user/project/integrations/ewm.md b/doc/user/project/integrations/ewm.md index c2371d32853..42f0ec6b0de 100644 --- a/doc/user/project/integrations/ewm.md +++ b/doc/user/project/integrations/ewm.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Engineering Workflow Management (EWM) **(FREE)** -This integration allows you to go from GitLab to EWM work items mentioned in merge request +The EWM integration allows you to go from GitLab to EWM work items mentioned in merge request descriptions and commit messages. Each work item reference is automatically converted to a link to the work item. diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 990d0839581..3b5e55b48a1 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitHub **(PREMIUM)** You can update GitHub with pipeline status updates from GitLab. -This integration can help you if you use GitLab for CI/CD. +The GitHub integration can help you if you use GitLab for CI/CD.  diff --git a/doc/user/project/integrations/google_play.md b/doc/user/project/integrations/google_play.md new file mode 100644 index 00000000000..553e82be382 --- /dev/null +++ b/doc/user/project/integrations/google_play.md @@ -0,0 +1,49 @@ +--- +stage: Manage +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Google Play **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111621) in GitLab 15.10 [with a flag](../../../administration/feature_flags.md) named `google_play_integration`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `google_play_integration`. On GitLab.com, this feature is not available. + +With the Google Play integration, you can configure your CI/CD pipelines to connect to the [Google Play Console](https://play.google.com/console) to build and release apps for Android devices. + +The Google Play integration works out of the box with [fastlane](https://fastlane.tools/). You can also use this integration with other build tools. + +## Enable the integration in GitLab + +Prerequisites: + +- You must have a [Google Play Console](https://play.google.com/console/signup) developer account. +- You must [generate a new service account key for your project](https://developers.google.com/android-publisher/getting_started) from the Google Cloud console. + +To enable the Google Play integration in GitLab: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Google Play**. +1. In **Enable Integration**, select the **Active** checkbox. +1. In **Service account key (.JSON)**, drag or upload your key file. +1. Select **Save changes**. + +After you enable the integration, the global variable `$SUPPLY_JSON_KEY_DATA` is created for CI/CD use. + +### CI/CD variable security + +Malicious code pushed to your `.gitlab-ci.yml` file could compromise your variables, including `$SUPPLY_JSON_KEY_DATA`, and send them to a third-party server. For more information, see [CI/CD variable security](../../../ci/variables/index.md#cicd-variable-security). + +## Enable the integration in fastlane + +To enable the integration in fastlane and upload the build to the given track in Google Play, you can add the following code to your app's `fastlane/Fastfile`: + +```ruby +upload_to_play_store( + track: 'internal', + aab: '../build/app/outputs/bundle/release/app-release.aab' +) +``` diff --git a/doc/user/project/integrations/hangouts_chat.md b/doc/user/project/integrations/hangouts_chat.md index 3537033902d..c9ba5b1b3aa 100644 --- a/doc/user/project/integrations/hangouts_chat.md +++ b/doc/user/project/integrations/hangouts_chat.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Google Chat **(FREE)** -Integrate your project to send notifications from GitLab to a -room of your choice in [Google Chat](https://chat.google.com/) (former Google +You can configure your project to send notifications from GitLab to a +room of your choice in [Google Chat](https://chat.google.com/) (formerly Google Hangouts). ## Integration workflow @@ -28,7 +28,7 @@ notifications to Google Chat: To enable the integration in Google Chat: 1. Enter the room where you want to receive notifications from GitLab. -1. Open the room dropdown list in the upper left and select **Manage webhooks**. +1. In the upper-left corner, from the room dropdown list, select **Manage webhooks**. 1. Enter the name for your webhook, for example "GitLab integration". 1. Optional. Add an avatar for your bot. 1. Select **Save**. diff --git a/doc/user/project/integrations/harbor.md b/doc/user/project/integrations/harbor.md index 596821ba12b..e316f6fbd9e 100644 --- a/doc/user/project/integrations/harbor.md +++ b/doc/user/project/integrations/harbor.md @@ -8,11 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80999) in GitLab 14.9. -Use Harbor as the container registry for your GitLab project. +You can use Harbor as the container registry for your GitLab project. -[Harbor](https://goharbor.io/) is an open source registry that can help you manage artifacts across cloud-native compute platforms, like Kubernetes and Docker. +[Harbor](https://goharbor.io/) is an open-source registry that can help you manage artifacts across cloud-native compute platforms like Kubernetes and Docker. -This integration can help you if you need GitLab CI/CD and a container image repository. +The Harbor integration can help you if you need GitLab CI/CD and a container image repository. ## Prerequisites diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index 57947e21736..9ff6ad2a237 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -72,13 +72,14 @@ You can configure the following integrations. | [Pipelines emails](pipeline_status_emails.md) | Send the pipeline status to a list of recipients by email. | **{dotted-circle}** No | | [Pivotal Tracker](pivotal_tracker.md) | Add commit messages as comments to Pivotal Tracker stories. | **{dotted-circle}** No | | [Prometheus](prometheus.md) | Monitor application metrics. | **{dotted-circle}** No | -| [Pumble](pumble.md) | Send event notifications to a Pumble channel. | **{dotted-circle}** No | +| [Pumble](pumble.md) | Send event notifications to a Pumble channel. | **{dotted-circle}** No | | Pushover | Get real-time notifications on your device. | **{dotted-circle}** No | | [Redmine](redmine.md) | Use Redmine as the issue tracker. | **{dotted-circle}** No | -| [Shimo Workspace](shimo.md) | Use Shimo instead of the GitLab Wiki. | **{dotted-circle}** No | +| [Shimo Workspace](shimo.md) | Use Shimo instead of the GitLab Wiki. | **{dotted-circle}** No | | [GitLab for Slack app](gitlab_slack_application.md) | Use Slack's official GitLab application. | **{dotted-circle}** No | | [Slack notifications](slack.md) | Send notifications about project events to Slack. | **{dotted-circle}** No | | [Slack slash commands](slack_slash_commands.md) | Enable slash commands in a workspace. | **{dotted-circle}** No | +| [Squash TM](squash_tm.md) | Update Squash TM requirements when GitLab issues are modified. | **{check-circle}** Yes | | [Unify Circuit](unify_circuit.md) | Send notifications about project events to Unify Circuit. | **{dotted-circle}** No | | [Webex Teams](webex_teams.md) | Receive events notifications. | **{dotted-circle}** No | | [YouTrack](youtrack.md) | Use YouTrack as the issue tracker. | **{dotted-circle}** No | diff --git a/doc/user/project/integrations/irker.md b/doc/user/project/integrations/irker.md index 23525c33e84..757d15e71b3 100644 --- a/doc/user/project/integrations/irker.md +++ b/doc/user/project/integrations/irker.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # irker (IRC gateway) **(FREE)** -GitLab provides a way to push update messages to an irker server. When -configured, pushes to a project trigger the integration to send data directly +GitLab provides a way to push update messages to an irker server. After you configure +the integration, each push to a project triggers the integration to send data directly to the irker server. See also the [irker integration API documentation](../../../api/integrations.md). diff --git a/doc/user/project/integrations/mock_ci.md b/doc/user/project/integrations/mock_ci.md index ae1737f8d3f..cbe073485e0 100644 --- a/doc/user/project/integrations/mock_ci.md +++ b/doc/user/project/integrations/mock_ci.md @@ -9,12 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w NOTE: This integration only appears if you're in a [development environment](https://gitlab.com/gitlab-org/gitlab-mock-ci-service#setup-mockci-integration). -To set up the mock CI service server, respond to the following endpoints +To set up the mock CI service server, respond to the following endpoints: - `commit_status`: `#{project.namespace.path}/#{project.path}/status/#{sha}.json` - - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success-with-warnings'|'skipped'|'not_found'] }` - - If the service returns a 404, it is interpreted as `pending` + - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success-with-warnings'|'skipped'|'not_found'] }`. + - If the service returns a 404, the service is interpreted as `pending`. - `build_page`: `#{project.namespace.path}/#{project.path}/status/#{sha}` - - Just where the build is linked to, doesn't matter if implemented + - Where the build is linked to (whether or not it's implemented). -For an example of a mock CI server, see [`gitlab-org/gitlab-mock-ci-service`](https://gitlab.com/gitlab-org/gitlab-mock-ci-service) +For an example of a mock CI server, see [`gitlab-org/gitlab-mock-ci-service`](https://gitlab.com/gitlab-org/gitlab-mock-ci-service). diff --git a/doc/user/project/integrations/pivotal_tracker.md b/doc/user/project/integrations/pivotal_tracker.md index 034be8ab3d8..c794001672b 100644 --- a/doc/user/project/integrations/pivotal_tracker.md +++ b/doc/user/project/integrations/pivotal_tracker.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pivotal Tracker **(FREE)** -This integration adds commit messages as comments to Pivotal Tracker stories. +The Pivotal Tracker integration adds commit messages as comments to Pivotal Tracker stories. Once enabled, commit messages are checked for square brackets containing a hash mark followed by the story ID (for example, `[#555]`). Every story ID found gets the commit comment added to it. diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index cd92e49cada..80c6337f934 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -23,7 +23,7 @@ Once enabled, GitLab detects metrics from known services in the [add your own metrics](../../../operations/metrics/index.md#adding-custom-metrics) and create [custom dashboards](../../../operations/metrics/dashboards/index.md). -## Enabling Prometheus Integration +## Enabling the Prometheus integration ### Prometheus cluster integration diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md index 1e9319fa7c7..4f841b7ccb3 100644 --- a/doc/user/project/integrations/prometheus_library/cloudwatch.md +++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring AWS resources (DEPRECATED) **(FREE)** +# Monitoring AWS resources (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index b4533d83acd..8fec7f68ab1 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring HAProxy (DEPRECATED) **(FREE)** +# Monitoring HAProxy (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md index afefe80271e..fe9ce323b23 100644 --- a/doc/user/project/integrations/prometheus_library/index.md +++ b/doc/user/project/integrations/prometheus_library/index.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Prometheus Metrics library (DEPRECATED) **(FREE)** +# Prometheus Metrics library (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 34a6823f007..dc6aa41d691 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring Kubernetes (DEPRECATED) **(FREE)** +# Monitoring Kubernetes (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md index f0a3b25f11a..90fe3ab8f8e 100644 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring NGINX (DEPRECATED) **(FREE)** +# Monitoring NGINX (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index 947210541f4..ba3f41e8201 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring NGINX Ingress Controller (DEPRECATED) **(FREE)** +# Monitoring NGINX Ingress Controller (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index f8057866592..89b8388b70c 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -4,7 +4,7 @@ group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Monitoring NGINX Ingress Controller with VTS metrics (DEPRECATED) **(FREE)** +# Monitoring NGINX Ingress Controller with VTS metrics (deprecated) **(FREE)** > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) in GitLab 14.7. diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index 0a399d3481f..97c3e14dd9d 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redmine **(FREE)** -Use [Redmine](https://www.redmine.org/) as the issue tracker. - +You can use [Redmine](https://www.redmine.org/) as an external issue tracker. To enable the Redmine integration in a project: 1. On the top bar, select **Main menu > Projects** and find your project. diff --git a/doc/user/project/integrations/servicenow.md b/doc/user/project/integrations/servicenow.md index a34655c8e35..149c9172896 100644 --- a/doc/user/project/integrations/servicenow.md +++ b/doc/user/project/integrations/servicenow.md @@ -9,6 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w ServiceNow offers several integrations to help centralize and automate your management of GitLab workflows. +To simplify your stack and streamline your processes, you should use GitLab [deployment approvals](../../../ci/environments/deployment_approvals.md) whenever possible. + ## GitLab spoke With the GitLab spoke in ServiceNow, you can automate actions for GitLab diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index 2563cec2b05..f9b95a0dd0c 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -15,7 +15,7 @@ working in Slack, you can use Slack slash commands. To use Slack slash commands, you must configure both Slack and GitLab. GitLab can also send events (for example, `issue created`) to Slack as notifications. -The [Slack notifications service](slack.md) is configured separately. +The [Slack notifications integration](slack.md) is configured separately. ## Configure GitLab and Slack diff --git a/doc/user/project/integrations/squash_tm.md b/doc/user/project/integrations/squash_tm.md new file mode 100644 index 00000000000..0f63b4a48db --- /dev/null +++ b/doc/user/project/integrations/squash_tm.md @@ -0,0 +1,37 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Squash TM integration **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337855) in GitLab 15.10. + +When [Squash TM](https://www.squashtest.com/squash-gitlab-integration?lang=en) (Test Management) +integration is enabled and configured in GitLab, issues (typically user stories) created in GitLab +are synchronized as requirements in Squash TM and test progress is reported in GitLab issues. + +## Configure Squash TM + +1. Optional. Ask your system administrator to [configure a token in the properties file](https://tm-en.doc.squashtest.com/latest/redirect/gitlab-integration-token.html). +1. Follow the [Squash TM documentation](https://tm-en.doc.squashtest.com/latest/redirect/gitlab-integration-configuration.html) to: + 1. Create a GitLab server. + 1. Enable the `Xsquash4GitLab` plugin + 1. Configure a synchronization. + 1. From the **Real-time synchronization** panel, copy the following fields to use later in GitLab: + + - **Webhook URL**. + - **Secret token** if your Squash TM system administrator configured one at step 1. + +## Configure GitLab + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Squash TM**. +1. Ensure that the **Active** toggle is enabled. +1. In the **Trigger** section, indicate which type of issue is concerned by the real-time synchronization. +1. Complete the fields: + + - Enter the **Squash TM webhook URL**, + - Enter the **secret token** if your Squash TM system administrator configured it earlier. diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index d465b4e50f0..9c38cec0a01 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The Unify Circuit integration sends notifications from GitLab to a Circuit conversation. -## Set up Unify Circuit service +## Set up Unify Circuit In Unify Circuit, [add a webhook](https://www.circuit.com/unifyportalfaqdetail?articleId=164448) and copy its URL. diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 3d971af5c2a..1511e37e31e 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -118,11 +118,16 @@ Endpoints should follow these best practices: - Never return `500` server error status responses if the event has been handled as this can cause the webhook to be [temporarily disabled](#failing-webhooks). - Invalid HTTP responses are treated as failed requests. -### Failing webhooks +## Failing webhooks -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60837) in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 15.7. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 15.7. Feature flag `web_hooks_disable_failed` removed. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60837) for project webhooks in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) for project webhooks in GitLab 15.7. Feature flag `web_hooks_disable_failed` removed. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385902) for group webhooks in GitLab 15.10. +> - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/390157) in GitLab 15.10 [with a flag](../../../administration/feature_flags.md) named `auto_disabling_web_hooks`. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `auto_disabling_web_hooks`. +On GitLab.com, this feature is available. If a webhook fails repeatedly, it may be disabled automatically. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 234faa893eb..98017fb5542 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -74,7 +74,7 @@ Prerequisites: To create a new issue board: -1. Select the dropdown list with the current board name in the upper left corner of the issue boards page. +1. In the upper-left corner of the issue board page, select the dropdown list with the current board name. 1. Select **Create new board**. 1. Enter the new board's name and select its scope: milestone, labels, assignee, or weight. @@ -86,7 +86,7 @@ Prerequisites: To delete the currently active issue board: -1. Select the dropdown list with the current board name in the upper left corner of the issue boards page. +1. In the upper-left corner of the issue board page, select the dropdown list with the current board name. 1. Select **Delete board**. 1. Select **Delete** to confirm. @@ -394,11 +394,11 @@ You can also [drag issues](#move-issues-and-lists) to change their position and  -## Work In Progress limits **(PREMIUM)** +## Work in progress limits **(PREMIUM)** > Moved to GitLab Premium in 13.9. -You can set a Work In Progress (WIP) limit for each issue list on an issue board. When a limit is +You can set a work in progress (WIP) limit for each issue list on an issue board. When a limit is set, the list's header shows the number of issues in the list and the soft limit of issues. You cannot set a WIP limit on the default lists (**Open** and **Closed**). @@ -413,11 +413,11 @@ Prerequisites: - You must have at least the Reporter role for the project. -To set a WIP limit for a list: +To set a WIP limit for a list, in an issue board: -1. Navigate to a Project or Group board of which you're a member. -1. Select the settings icon in a list's header. -1. Next to **Work In Progress Limit**, select **Edit**. +1. On the top of the list you want to edit, select **List actions** (**{ellipsis_v}**) **> Edit list settings**. + The list settings sidebar opens on the right. +1. Next to **Work in progress limit**, select **Edit**. 1. Enter the maximum number of issues. 1. Press <kbd>Enter</kbd> to save. @@ -475,7 +475,7 @@ Additionally, you can also see the time tracking value. ### Create a new list -Create a new list by selecting the **Create** button in the upper right corner of the issue board. +To create a new list, in the upper-right corner of the issue board, select **Create**.  @@ -493,10 +493,10 @@ Prerequisites: To remove a list from an issue board: -1. On the top of the list you want to remove, select the **List settings** icon (**{settings}**). +1. On the top of the list you want to remove, select **List actions** (**{ellipsis_v}**). The list settings sidebar opens on the right. 1. Select **Remove list**. A confirmation dialog appears. -1. Select **OK**. +1. Select **Remove list** again. ### Add issues to a list diff --git a/doc/user/project/issues/create_issues.md b/doc/user/project/issues/create_issues.md index 5ebb2fc2e1c..b6931149ede 100644 --- a/doc/user/project/issues/create_issues.md +++ b/doc/user/project/issues/create_issues.md @@ -100,7 +100,7 @@ To create an issue from a project issue board: 1. On the top bar, select **Main menu > Projects** and find your project. 1. Select **Issues > Boards**. -1. At the top of a board list, select **New issue** (**{plus-square}**). +1. At the top of a board list, select **List actions** (**{ellipsis_v}**) **> Create new issue**. 1. Enter the issue's title. 1. Select **Create issue**. @@ -108,7 +108,7 @@ To create an issue from a group issue board: 1. On the top bar, select **Main menu > Groups** and find your group. 1. Select **Issues > Boards**. -1. At the top of a board list, select **New issue** (**{plus-square}**). +1. At the top of a board list, select **List actions** (**{ellipsis_v}**) **> Create new issue**. 1. Enter the issue's title. 1. Under **Projects**, select the project in the group that the issue should belong to. 1. Select **Create issue**. @@ -118,9 +118,6 @@ example, if the list is scoped to a label `Frontend`, the new issue also has thi ## By sending an email -> - Generated email address format changed in GitLab 11.7. -> - The older format is still supported, so existing aliases and contacts still work. - You can send an email to create an issue in a project on the project's **Issues List** page. diff --git a/doc/user/project/issues/crosslinking_issues.md b/doc/user/project/issues/crosslinking_issues.md index 52da1acd32a..f1f41de7cb4 100644 --- a/doc/user/project/issues/crosslinking_issues.md +++ b/doc/user/project/issues/crosslinking_issues.md @@ -39,10 +39,10 @@ git commit -m "this is my commit message. Ref projectname#xxx" ``` If they are not in the same group, you can add the full URL to the issue -(`https://gitlab.com/<username>/<projectname>/issues/<xxx>`). +(`https://gitlab.com/<username>/<projectname>/-/issues/<xxx>`). ```shell -git commit -m "this is my commit message. Related to https://gitlab.com/<username>/<projectname>/issues/<xxx>" +git commit -m "this is my commit message. Related to https://gitlab.com/<username>/<projectname>/-/issues/<xxx>" ``` Of course, you can replace `gitlab.com` with the URL of your own GitLab instance. diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 8a6683a55b5..ac7d8c99fa8 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -36,8 +36,8 @@ To import issues: 1. Go to your project's Issues list page. 1. Open the import feature, depending if the project has issues: - - Existing issues are present: Select the import icon in the upper right, next to **Edit issues**. - - Project has no issues: Select **Import CSV** in the middle of the page. + - The project has existing issues: in the upper-right corner, next to **Edit issues**, select the import icon (**{import}**). + - The project has no issues: in the middle of the page, select **Import CSV**. 1. Select the file you want to import, and then select **Import issues**. The file is processed in the background, and a notification email is sent diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index f43f87119a6..10b29feb072 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -227,6 +227,19 @@ New discussion threads get different pin numbers, which you can use to refer to In GitLab 12.5 and later, new discussions are output to the issue activity, so that everyone involved can participate in the discussion. +## Delete a comment from a design + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385100) in GitLab 15.9. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To delete a comment from a design: + +1. On the comment you want to delete, select **More actions** **{ellipsis_v}** **> Delete comment**. +1. On the confirmation dialog box, select **Delete comment**. + ## Resolve a discussion thread on a design > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13049) in GitLab 13.1. diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index c16074ea1d8..e0966909b1e 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -400,7 +400,7 @@ To view all issues assigned to you: Or: - To use a [keyboard shortcut](../../shortcuts.md), press <kbd>Shift</kbd> + <kbd>i</kbd>. -- On the top bar, in the upper right, select **{issues}** **Issues**. +- On the top bar, in the upper-right corner, select **Issues** (**{issues}**). ## Filter the list of issues diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index c7d45b0bd15..6e20492db05 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 4dda68a6d08..356e4e1b194 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -70,17 +70,21 @@ are given access to the project. In addition: ## Maximum role +When you invite a group to a project, the maximum role is the highest level of access the invited group members are allowed to have in the project. + When multiple groups contain the same members, and the groups have access to the same project, the group members are -given the most restrictive role for the project. - -This most restrictive role is called the *maximum role*, or **Max role**. +given the highest access level of the two for the project. The member's **Max role** is the more restrictive of: - The role the user is assigned for the group. - The role you chose when you invited the group to the project. +NOTE: +The Max role does not elevate the privileges of users. +For example, if a group member has the role of Developer, and the group is invited to a project with a Max role of Maintainer, the member's role is not elevated to Maintainer. + ### View the member's Max role To view the maximum role assigned to a member: diff --git a/doc/user/project/merge_requests/allow_collaboration.md b/doc/user/project/merge_requests/allow_collaboration.md index 0729e024df4..4a22cc71349 100644 --- a/doc/user/project/merge_requests/allow_collaboration.md +++ b/doc/user/project/merge_requests/allow_collaboration.md @@ -61,7 +61,7 @@ In the following example: To change or add a commit to the contributor's merge request: 1. Go to the merge request. -1. In the upper right corner, select **Code**, then select **Check out branch**. +1. In the upper-right corner, select **Code**, then select **Check out branch**. 1. In the modal window, select **Copy** (**{copy-to-clipboard}**). 1. In your terminal, go to your cloned version of the repository, and paste the commands. For example: diff --git a/doc/user/project/merge_requests/cherry_pick_changes.md b/doc/user/project/merge_requests/cherry_pick_changes.md index 9fac872ac4b..d2676b1bdf0 100644 --- a/doc/user/project/merge_requests/cherry_pick_changes.md +++ b/doc/user/project/merge_requests/cherry_pick_changes.md @@ -55,7 +55,7 @@ by the merge request: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Merge requests**, and find your merge request. 1. Scroll to the merge request reports section, and find the **Merged by** report. -1. In the upper right, select **Cherry-pick**: +1. In the upper-right corner, select **Cherry-pick**:  1. In the modal window, select the project and branch to cherry-pick into. @@ -72,7 +72,8 @@ To cherry-pick a commit from the list of all commits for a project: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Repository > Commits**. -1. Select the title of the commit you want to cherry-pick. +1. Select the [title](https://git-scm.com/docs/git-commit#_discussion) of the commit you want to cherry-pick. +1. In the upper-right corner, select **Options > Cherry-pick** to show the cherry-pick modal. 1. In the modal window, select the project and branch to cherry-pick into. 1. Optional. Select **Start a new merge request with these changes**. 1. Select **Cherry-pick**. @@ -86,7 +87,7 @@ list of commits included in a merge request: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Merge requests**, and find your merge request. 1. In the merge request's secondary menu, select **Commits** to display the commit details page. -1. Select the title of the commit you want to cherry-pick. +1. Select the [title](https://git-scm.com/docs/git-commit#_discussion) of the commit you want to cherry-pick. 1. In the upper-right corner, select **Options > Cherry-pick** to show the cherry-pick modal. 1. In the modal window, select the project and branch to cherry-pick into. 1. Optional. Select **Start a new merge request with these changes**. @@ -100,7 +101,8 @@ when you view that file in your project's Git repository: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Repository > Files** and go to the file changed by the commit. -1. Select **History**, then select the title of the commit you want to cherry-pick. +1. Select **History**, then select the [title](https://git-scm.com/docs/git-commit#_discussion) + of the commit you want to cherry-pick. 1. In the upper-right corner, select **Options > Cherry-pick** to show the cherry-pick modal. 1. In the modal window, select the project and branch to cherry-pick into. 1. Optional. Select **Start a new merge request with these changes**. diff --git a/doc/user/project/merge_requests/commits.md b/doc/user/project/merge_requests/commits.md deleted file mode 100644 index a9f67c39ae8..00000000000 --- a/doc/user/project/merge_requests/commits.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../merge_requests/index.md' -remove_date: '2023-03-12' ---- - -This document was removed. - -<!-- This redirect file can be deleted after <2023-03-12>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 875312bbb7c..4ea549e5986 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -11,7 +11,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/gitlab-basics/add-merge-request.h There are many different ways to create a merge request. NOTE: -Use [branch naming patterns](../repository/branches/index.md#naming) to streamline merge request creation. +Use [branch naming patterns](../repository/branches/index.md#prefix-branch-names-with-issue-numbers) to streamline merge request creation. ## From the merge request list @@ -19,7 +19,7 @@ You can create a merge request from the list of merge requests. 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left menu, select **Merge requests**. -1. In the upper right, select **New merge request**. +1. In the upper-right corner, select **New merge request**. 1. Select a source and target branch and then **Compare branches and continue**. 1. Fill out the fields and select **Create merge request**. @@ -171,7 +171,7 @@ To create a merge request by sending an email: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left menu, select **Merge requests**. -1. In the upper right, select **Email a new merge request to this project**. +1. In the upper-right corner, select **Email a new merge request to this project**. An email address is displayed. Copy this address. Ensure you keep this address private. 1. Open an email and compose a message with the following information: diff --git a/doc/user/project/merge_requests/img/remove_source_branch_status.png b/doc/user/project/merge_requests/img/remove_source_branch_status.png Binary files differdeleted file mode 100644 index afd93207e02..00000000000 --- a/doc/user/project/merge_requests/img/remove_source_branch_status.png +++ /dev/null diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index a633366cc62..f760b1b730a 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -65,7 +65,7 @@ or: or: -1. On the top bar, in the upper right, select **{merge-request-open}** **Merge requests**. +1. On the top bar, in the upper-right corner, select **Merge requests** (**{merge-request-open}**). 1. From the dropdown list, select **Assigned to you**. ## Filter the list of merge requests @@ -257,8 +257,8 @@ FLAG: On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `moved_mr_sidebar`. On GitLab.com, this feature is enabled in the following projects: `gitlab-org/gitlab`, `gitlab-com/www-gitlab-com`, and `gitlab-org/customers-gitlab-com`. -When this feature flag is enabled, you can find the following actions in -**Merge request actions** (**{ellipsis_v}**) in the upper right: +When this feature flag is enabled, in the upper-right corner, +**Merge request actions** (**{ellipsis_v}**) contains the following actions: - The [notifications](../../profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) toggle - Mark merge request as ready or [draft](../merge_requests/drafts.md) @@ -304,7 +304,6 @@ For a web developer writing a webpage for your company's website: - [GitLab keyboard shortcuts](../../shortcuts.md) - [Comments and threads](../../discussions/index.md) - [Suggest code changes](reviews/suggestions.md) -- [Commits](commits.md) - [CI/CD pipelines](../../../ci/index.md) - [Push options](../push_options.md) for merge requests diff --git a/doc/user/project/merge_requests/methods/index.md b/doc/user/project/merge_requests/methods/index.md index 1f7e15ee982..02bd4ed0502 100644 --- a/doc/user/project/merge_requests/methods/index.md +++ b/doc/user/project/merge_requests/methods/index.md @@ -228,5 +228,4 @@ workflow that requires frequent rebases. ## Related topics -- [Commits history](../commits.md) - [Squash and merge](../squash_and_merge.md) diff --git a/doc/user/project/merge_requests/reviews/img/apply_suggestion_v13_9.png b/doc/user/project/merge_requests/reviews/img/apply_suggestion_v13_9.png Binary files differdeleted file mode 100644 index e27fa629672..00000000000 --- a/doc/user/project/merge_requests/reviews/img/apply_suggestion_v13_9.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/img/make_suggestion_v13_9.png b/doc/user/project/merge_requests/reviews/img/make_suggestion_v13_9.png Binary files differdeleted file mode 100644 index 92d5ba5ddda..00000000000 --- a/doc/user/project/merge_requests/reviews/img/make_suggestion_v13_9.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/img/suggestion_button_v13_9.png b/doc/user/project/merge_requests/reviews/img/suggestion_button_v13_9.png Binary files differdeleted file mode 100644 index 58e0508d8cf..00000000000 --- a/doc/user/project/merge_requests/reviews/img/suggestion_button_v13_9.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/img/suggestions_custom_commit_messages_v14_7.png b/doc/user/project/merge_requests/reviews/img/suggestions_custom_commit_messages_v14_7.png Binary files differdeleted file mode 100644 index 2805ef19f2d..00000000000 --- a/doc/user/project/merge_requests/reviews/img/suggestions_custom_commit_messages_v14_7.png +++ /dev/null diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 5291a845437..4bd77c7ebdd 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, reference --- -# Review a merge request **(FREE)** +# Merge request reviews **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216054) in GitLab 13.5. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245190) in GitLab 13.9. @@ -84,7 +84,7 @@ To download the changes included in a merge request as a diff: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Merge requests**. 1. Select your merge request. -1. In the upper right, select **Code > Plain diff**. +1. In the upper-right corner, select **Code > Plain diff**. If you know the URL of the merge request, you can also download the diff from the command line by appending `.diff` to the URL. This example downloads the diff @@ -107,7 +107,7 @@ To download the changes included in a merge request as a patch file: 1. On the top bar, select **Main menu > Projects** and find your project. 1. On the left sidebar, select **Merge requests**. 1. Select your merge request. -1. In the upper right, select **Code > Email patches**. +1. In the upper-right corner, select **Code > Email patches**. If you know the URL of the merge request, you can also download the patch from the command line by appending `.patch` to the URL. This example downloads the patch diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 668dece9fda..6976d4052e1 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -8,44 +8,40 @@ type: index, reference # Suggest changes **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) custom commit messages for suggestions in GitLab 13.9 [with a flag](../../../../administration/feature_flags.md) named `suggestions_custom_commit`. Disabled by default. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/297404) in GitLab 13.10. - -As a reviewer, you're able to suggest code changes with a Markdown syntax in merge request -diff threads. Then, the merge request author (or other users with appropriate -[permission](../../../permissions.md)) can apply these suggestions. -This action generates a commit in the merge request, authored by the user that suggested the changes. - -1. Choose a line of code to be changed, add a new comment, then select - the **Insert suggestion** icon in the toolbar: - -  - -1. In the comment, add your suggestion to the pre-populated code block: - -  - +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/297404) in GitLab 13.10. Feature flag `suggestions_custom_commit` removed. + +Reviewers can suggest code changes with a Markdown syntax in merge request diff threads. +The merge request author (or other users with the appropriate role) can apply any or +all of the suggestions from the GitLab UI. Applying suggestions adds a commit to the +merge request, authored by the user who suggested the changes. + +## Create suggestions + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Merge requests** and find your merge request. +1. On the secondary menu, select **Changes**. +1. Find the lines of code you want to change. + - To select a single line: + - Hover over the line number, and + select **Add a comment to this line** (**{comment}**). + - To select multiple lines: + 1. Hover over the line number, and select **Add a comment to this line** (**{comment}**). + 1. Select and drag your selection until all desired lines are included. To + learn more, see [Multi-line suggestions](#multi-line-suggestions). +1. In the comment toolbar, select **Insert suggestion** (**{doc-code}**). GitLab + inserts a pre-populated code block into your comment, like this: + + ````markdown + ```suggestion:-0+0 + The content of the line you selected is shown here. + ``` + ```` + +1. Edit the pre-populated code block to add your suggestion. 1. Select either **Start a review** or **Add to review** to add your comment to a [review](index.md), or **Add comment now** to add the comment to the thread immediately. - The suggestion in the comment can be applied by the merge request author - directly from the merge request: - -  - -1. Optionally specify a custom commit message for individual suggestions (GitLab 13.9 and later) to - describe your change. If you don't specify it, the default commit message is used. - -  - -After the author applies a suggestion: - -1. The suggestion is marked as **Applied**. -1. The thread is resolved. -1. GitLab creates a new commit with the changes. -1. If the user has the Developer role, GitLab pushes - the suggested change directly into the codebase in the merge request's branch. - -## Multi-line suggestions +### Multi-line suggestions > [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/232339) in GitLab 13.11: suggestions in multi-line comments also become multi-line. @@ -68,65 +64,83 @@ Suggestions for multiple lines are limited to 100 lines _above_ and 100 lines _below_ the commented diff line. This allows for up to 200 changed lines per suggestion. -## Code block nested in suggestions +## Apply suggestions + +The merge request author can apply suggested changes directly from the merge request: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Merge requests** and find your merge request. +1. Find the comment containing the suggestion you want to apply. + - To apply suggestions individually, select **Apply suggestion**. + - To apply multiple suggestions in a single commit, select **Add suggestion to batch**. +1. Optional. Provide a custom commit message to describe your change. If you don't provide a custom message, the default commit message is used. +1. Select **Apply**. + +After a suggestion is applied: + +- The suggestion is marked as **Applied**. +- The comment thread is resolved. +- GitLab creates a new commit with the changes. +- If the user has the Developer role, GitLab pushes + the suggested change directly into the codebase in the merge request's branch. + +## Nest code blocks in suggestions To add a suggestion that includes a [fenced code block](../../../markdown.md#code-spans-and-blocks), wrap your suggestion in four backticks instead of three: -~~~markdown +`````markdown ````suggestion:-0+2 ```shell git config --global receive.advertisepushoptions true ``` ```` -~~~ +`````  ## Configure the commit message for applied suggestions -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13086) in GitLab 12.7. +GitLab uses a default commit message when applying suggestions. This message +supports placeholders, and can be changed. For example, the default message +`Apply %{suggestions_count} suggestion(s) to %{files_count} file(s)` renders +like this if you apply three suggestions to two different files: -GitLab uses a default commit message -when applying suggestions: `Apply %{suggestions_count} suggestion(s) to %{files_count} file(s)` - -<!-- vale gitlab.BadPlurals = NO --> +```plaintext +Apply 3 suggestion(s) to 2 file(s) +``` -For example, consider that a user applied 3 suggestions to 2 different files, the -default commit message is: **Apply 3 suggestion(s) to 2 file(s)** +Merge requests created from forks use the template defined in the target project. -<!-- vale gitlab.BadPlurals = YES --> +To meet your project's needs, you can customize these messages and include other +placeholder variables: -These commit messages can be customized to follow any guidelines you might have. -To do so, expand the **Merge requests** tab within your project's **General** -settings and change the **Merge suggestions** text: +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Merge requests**. +1. Scroll to **Merge suggestions**, and alter the text to meet your needs. + See [Supported variables](#supported-variables) for a list of placeholders + you can use in this message. - +### Supported variables -You can also use following variables besides static text: +The template for commit messages for applied suggestions supports these variables: | Variable | Description | Output example | |------------------------|-------------|----------------| | `%{branch_name}` | The name of the branch to which suggestions were applied. | `my-feature-branch` | -| `%{files_count}` | The number of files to which suggestions were applied.| **2** | +| `%{files_count}` | The number of files to which suggestions were applied.| `2` | | `%{file_paths}` | The paths of the file to which suggestions were applied. Paths are separated by commas.| `docs/index.md, docs/about.md` | | `%{project_path}` | The project path. | `my-group/my-project` | -| `%{project_name}` | The human-readable name of the project. | **My Project** | -| `%{suggestions_count}` | The number of suggestions applied.| **3** | +| `%{project_name}` | The human-readable name of the project. | `My Project` | +| `%{suggestions_count}` | The number of suggestions applied.| `3` | | `%{username}` | The username of the user applying suggestions. | `user_1` | -| `%{user_full_name}` | The full name of the user applying suggestions. | **User 1** | +| `%{user_full_name}` | The full name of the user applying suggestions. | `User 1` | For example, to customize the commit message to output -**Addresses user_1's review**, set the custom text to +`Addresses user_1's review`, set the custom text to `Addresses %{username}'s review`. -For merge requests created from forks, GitLab uses the template defined in target project. - -NOTE: -Custom commit messages for each applied suggestion is -introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). - ## Batch suggestions > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](../../../../policy/alpha-beta-support.md#alpha-features) with a flag named `batch_suggestions`, disabled by default. diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index 62a2baa049b..6a791a17057 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -151,12 +151,15 @@ the status check and it **is not** recoverable. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327634) in GitLab 14.1. > - UI [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91504) in GitLab 15.2. > - Ability to retry failed external status checks [added](https://gitlab.com/gitlab-org/gitlab/-/issues/383200) in GitLab 15.8. +> - Widget [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111763) to poll for updates when there are pending status checks in GitLab 15.11. The status checks widget displays in merge requests and displays the following statuses: - **pending** (**{status-neutral}**), while GitLab waits for a response from an external status check. - **success** (**{status-success}**) or **failed** (**{status-failed}**), when GitLab receives a response from an external status check. +When there are pending status checks, the widget polls for updates every few seconds until it receives a **success** or **failed** response. + To retry a failed status check: 1. Expand the merge request widget to show the list of external status checks. diff --git a/doc/user/project/milestones/burndown_and_burnup_charts.md b/doc/user/project/milestones/burndown_and_burnup_charts.md index 81b334c0a02..c7ed6069cb6 100644 --- a/doc/user/project/milestones/burndown_and_burnup_charts.md +++ b/doc/user/project/milestones/burndown_and_burnup_charts.md @@ -137,14 +137,16 @@ To switch between the two settings, select either **Issues** or **Issue weight** When sorting by weight, make sure all your issues have weight assigned, because issues with no weight don't show on the chart. -<!-- ## Troubleshooting +## Troubleshooting -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +### Burndown and burnup charts do not show the correct issue status -Each scenario can be a third-level heading, for example `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +A limitation of these charts is that [the days are in the UTC time zone](https://gitlab.com/gitlab-org/gitlab/-/issues/267967). + +This can cause the graphs to be inaccurate in other timezones. For example: + +- All the issues in a milestone are recorded as being closed on or before the last day. +- One issue was closed on the last day at 6 PM PST (Pacific time), which is UTC-7. +- The issue activity log displays the closure time at 6 PM on the last day of the milestone. +- The charts plot the time in UTC, so for this issue, the close time is 1 AM the following day. +- The charts show the milestone as incomplete and missing one closed issue. diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 5f9a2961df5..e9de780655a 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -65,7 +65,10 @@ Improving this experience is tracked in issue [339009](https://gitlab.com/gitlab You can view all the milestones you have access to in the entire GitLab namespace. You might not see some milestones because they're in projects or groups you're not a member of. -To do so, on the top bar select **Main menu > Milestones**. +To do so: + +1. On the top bar select **Main menu > Your work**. +1. On the left sidebar, select **Milestones**. ### View milestone details diff --git a/doc/user/project/organize_work_with_projects.md b/doc/user/project/organize_work_with_projects.md index 2b4ce6d2fd0..85a1dfda679 100644 --- a/doc/user/project/organize_work_with_projects.md +++ b/doc/user/project/organize_work_with_projects.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md index e55cf337d16..a92f65b064e 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -5,10 +5,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# DNS records overview **(FREE)** - -_Read this document for a brief overview of DNS records in the scope -of GitLab Pages, for beginners in web development._ +# GitLab Pages DNS records **(FREE)** A Domain Name System (DNS) web service routes visitors to websites by translating domain names (such as `www.example.com`) into the @@ -74,7 +71,7 @@ Example: This way, visitors visiting `www.example.com` are redirected to `example.com`. -## MX record +## `MX` record MX records are used to define the mail exchanges that are used for the domain. This helps email messages arrive at your mail server correctly. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 24e9e6e15a4..3bb62921979 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -5,7 +5,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Custom domains and SSL/TLS certificates **(FREE)** +# GitLab Pages custom domains **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238461) in GitLab 15.4, you can use verified domains to [bypass user email confirmation for SAML- or SCIM-provisioned users](../../../group/saml_sso/index.md#bypass-user-email-confirmation-with-verified-domains). diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index 95ac2e50f29..34a2f221327 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -6,7 +6,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Pages integration with Let's Encrypt **(FREE)** +# GitLab Pages Let's Encrypt certificates **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28996) in GitLab 12.1. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md index 398d8dc6e1e..0ba00177659 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -5,10 +5,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# SSL/TLS certificates **(FREE)** - -_Read this document for a brief overview of SSL/TLS certificates in -the scope of GitLab Pages, for beginners in web development._ +# GitLab Pages SSL/TLS certificates **(FREE)** Every GitLab Pages project on GitLab.com is available under HTTPS for the default Pages domain (`*.gitlab.io`). Once you set diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index f596e418b02..37f74d18d4c 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -5,7 +5,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create a Pages website by using a CI/CD template **(FREE)** +# Create a GitLab Pages website from a CI/CD template **(FREE)** GitLab provides `.gitlab-ci.yml` templates for the most popular Static Site Generators (SSGs). You can create your own `.gitlab-ci.yml` file from one of these templates, and run diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index 509609e9b89..a864c114b3e 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -5,7 +5,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create a Pages website from a forked sample **(FREE)** +# Create a GitLab Pages website from a forked sample project **(FREE)** GitLab provides [sample projects for the most popular Static Site Generators (SSG)](https://gitlab.com/pages). You can fork one of the sample projects and run the CI/CD pipeline to generate a Pages website. @@ -19,7 +19,7 @@ To fork a sample project and create a Pages website: 1. View the sample projects by navigating to the [GitLab Pages examples](https://gitlab.com/pages) group. 1. Select the name of the project you want to [fork](../../repository/forking_workflow.md#creating-a-fork). -1. In the upper right, select **Fork** and then choose a namespace to fork to. +1. In the upper-right corner, select **Fork**, then choose a namespace to fork to. 1. For your project, on the left sidebar, select **CI/CD > Pipelines** and then **Run pipeline**. GitLab CI/CD builds and deploys your site. diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index a301d2b64be..7b6efaa7af4 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -4,7 +4,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create a Pages website from a template **(FREE)** +# Create a GitLab Pages website from a project template **(FREE)** GitLab provides templates for the most popular Static Site Generators (SSGs). You can create a new project from a template and run the CI/CD pipeline to generate a Pages website. diff --git a/doc/user/project/pages/getting_started/pages_ui.md b/doc/user/project/pages/getting_started/pages_ui.md index 91c2c532a9a..00635fe6db2 100644 --- a/doc/user/project/pages/getting_started/pages_ui.md +++ b/doc/user/project/pages/getting_started/pages_ui.md @@ -4,7 +4,7 @@ group: Incubation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create a Pages deployment for your static site **(FREE)** +# Create a GitLab Pages deployment for a static site **(FREE)** If you already have a GitLab project that contains your static site or framework, you can generate a GitLab Pages website from it. diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index a0c8073d6eb..b286c59916a 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -4,7 +4,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Pages domain names, URLs, and base URLs **(FREE)** +# GitLab Pages default domain names and URLs **(FREE)** On this document, learn how to name your project for GitLab Pages according to your intended website's URL. diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 51c42eeecb8..0a8348f468e 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -4,7 +4,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Exploring GitLab Pages **(FREE)** +# GitLab Pages settings **(FREE)** This document is a user guide to explore the options and settings GitLab Pages offers. diff --git a/doc/user/project/pages/public_folder.md b/doc/user/project/pages/public_folder.md index 751db136e34..3007d1a10d1 100644 --- a/doc/user/project/pages/public_folder.md +++ b/doc/user/project/pages/public_folder.md @@ -6,7 +6,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Configure the public files folder **(FREE)** +# GitLab Pages public folder **(FREE)** All the files that should be accessible by the browser must be in a root-level folder called `public`. @@ -98,7 +98,7 @@ example: next export -o public ``` -### Nuxt.js +## Nuxt.js NOTE: GitLab Pages supports only static sites. diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index f5447fd67ca..75cb1474dc2 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -4,7 +4,7 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Create redirects for GitLab Pages **(FREE)** +# GitLab Pages redirects **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/24) in GitLab Pages 1.25.0 and GitLab 13.4 behind a feature flag, disabled by default. > - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/367) in GitLab 13.5. diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index da53fe2f5e9..725818a7d25 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -36,6 +36,40 @@ When a branch is protected, the default behavior enforces these restrictions on 1. No one can delete a protected branch using Git commands, however, users with at least Maintainer role can [delete a protected branch from the UI or API](#delete-a-protected-branch). +### When a branch matches multiple rules + +When a branch matches multiple rules, the **most permissive rule** determines the +level of protection for the branch. For example, consider these rules, which include +[wildcards](#configure-multiple-protected-branches-by-using-a-wildcard): + +| Branch name pattern | Allowed to merge | Allowed to push | +|---------------------|------------------------|-----------------| +| `v1.x` | Maintainer | Maintainer | +| `v1.*` | Maintainer + Developer | Maintainer | +| `v*` | No one | No one | + +A branch named `v1.x` matches all three branch name patterns: `v1.x`, `v1.*`, and `v*`. +As the most permissive option determines the behavior, the resulting permissions for branch `v1.x` are: + +- **Allowed to merge:** Of the three settings, `Maintainer + Developer` is most permissive, + and controls branch behavior as a result. Even though the branch also matched `v1.x` and `v*` + (which each have stricter permissions), users with the Developer role can merge into the branch. +- **Allowed to push:** Of the three settings, `Maintainer` is the most permissive, and controls + branch behavior as a result. Even though branches matching `v*` are set to `No one`, branches + that _also_ match `v1.x` or `v1.*` receive the more permissive `Maintainer` permission. + +To be certain that a rule controls the behavior of a branch, +_all_ other patterns that match must apply less or equally permissive rules. + +If you want to ensure that `No one` is allowed to push to branch `v1.x`, every pattern +that matches `v1.x` must set `Allowed to push` to `No one`, like this: + +| Branch name pattern | Allowed to merge | Allowed to push | +|---------------------|------------------------|-----------------| +| `v1.x` | Maintainer | No one | +| `v1.*` | Maintainer + Developer | No one | +| `v*` | No one | No one | + ### Set the default branch protection level Administrators can set a default branch protection level in the @@ -43,6 +77,37 @@ Administrators can set a default branch protection level in the ## Configure a protected branch +Configure protected branches for all projects in a group, or just for a project. + +### For all projects in a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106532) in GitLab 15.9 behind a feature flag, disabled by default. + +Group owners can create protected branches for a group. These settings are inherited by all projects in the group and can't be overridden by project settings. + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) +named `group_protected_branches`. On GitLab.com, this feature is not available. + +Prerequisite: + +- You must have the Owner role in the group. + +To protect a branch for all the projects in a group: + +1. On the top bar, select **Main menu > Groups** and find your group. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Protected branches**. +1. In the **Branch** text box, type the branch name or a wildcard. +1. From the **Allowed to merge** list, select a role, group, or user that can merge into this branch. +1. From the **Allowed to push** list, select a role, group, or user that can push to this branch. +1. Select **Protect**. + +The protected branch is added to the list of protected branches. + +### For a project + Prerequisite: - You must have at least the Maintainer role. @@ -203,8 +268,7 @@ Members who can push to this branch can now also force push. ## Require Code Owner approval on a protected branch **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13251) in GitLab 12.4. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35097) in GitLab 13.5, users and groups who can push to protected branches do not have to use a merge request to merge their feature branches. This means they can skip merge request approval rules. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35097) in GitLab 13.5, users and groups who can push to protected branches do not have to use a merge request to merge their feature branches. This means they can skip merge request approval rules. For a protected branch, you can require at least one approval by a [Code Owner](code_owners.md). diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 152a55d24b7..1256599c521 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Protected tags **(FREE)** -Protected tags: +Protected [tags](repository/tags/index.md): - Allow control over who has permission to create tags. - Prevent accidental update or deletion once created. @@ -86,6 +86,32 @@ To prevent this problem: Users can still create branches, but not tags, with the protected names. +## Allow deploy keys to create protected tags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325415) in GitLab 15.11. + +You can permit the owner of a [deploy key](deploy_keys/index.md) to create protected tags. +The deploy key works, even if the user isn't a member of the related project. However, the owner of the deploy +key must have at least read access to the project. + +Prerequisites: + +- The deploy key must be enabled for your project. A project deploy key is enabled by default when + it is created. However, a public deploy key must be + [granted](deploy_keys/index.md#grant-project-access-to-a-public-deploy-key) access to the + project. +- The deploy key must have [write access](deploy_keys/index.md#permissions) to your project + repository. + +To allow a deploy key to create a protected tag: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Protected tags**. +1. From the **Tag** dropdown list, select the tag you want to protect. +1. From the **Allowed to create** list, select the deploy key. +1. Select **Protect**. + ## Delete a protected tag You can manually delete protected tags with the GitLab API, or the @@ -106,6 +132,11 @@ Protected tags can only be deleted by using GitLab either from the UI or API. These protections prevent you from accidentally deleting a tag through local Git commands or third-party Git clients. +## Related topics + +- [Protected Tags API](../../api/protected_tags.md) +- [Tags API](../../api/tags.md) + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 90da47f7995..3a00260770e 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -69,7 +69,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/copy_metadata <!merge_request>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Copy labels and milestone from another merge request in the project. | | `/copy_metadata <#issue>` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Copy labels and milestone from another issue in the project. | | `/create_merge_request <branch name>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Create a new merge request starting from the current issue. | -| `/done` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mark to do as done. | +| `/done` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Mark to-do item as done. | | `/draft` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Set the [draft status](merge_requests/drafts.md). Use for toggling the draft status ([deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92654) in GitLab 15.4.) | | `/due <date>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. See [Chronic](https://gitlab.com/gitlab-org/ruby/gems/gitlab-chronic#examples) for more examples. | | `/duplicate <#issue>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Close this issue. Mark as a duplicate of, and related to, issue `<#issue>`. | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index dca34af41b4..be2c923adcc 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -88,7 +88,7 @@ To create a release in the Releases page: - [Title](release_fields.md#title). - [Milestones](#associate-milestones-with-a-release). - [Release notes](release_fields.md#release-notes-description). - - Whether or not to include the [Tag message](../../../topics/git/tags.md). + - Whether or not to include the [Tag message](../repository/tags/index.md). - [Asset links](release_fields.md#links). 1. Select **Create release**. diff --git a/doc/user/project/releases/release_cicd_examples.md b/doc/user/project/releases/release_cicd_examples.md index 1ec82d6702e..36e36412cf1 100644 --- a/doc/user/project/releases/release_cicd_examples.md +++ b/doc/user/project/releases/release_cicd_examples.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/releases/release_cli.md b/doc/user/project/releases/release_cli.md index 5af19c7cced..953e1d4b082 100644 --- a/doc/user/project/releases/release_cli.md +++ b/doc/user/project/releases/release_cli.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/releases/release_fields.md b/doc/user/project/releases/release_fields.md index 120bbe32a63..93822be89b6 100644 --- a/doc/user/project/releases/release_fields.md +++ b/doc/user/project/releases/release_fields.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/project/remote_development/index.md b/doc/user/project/remote_development/index.md index 1abcca547db..58d43f64812 100644 --- a/doc/user/project/remote_development/index.md +++ b/doc/user/project/remote_development/index.md @@ -97,7 +97,7 @@ docker run -d \ -v "${CERTS_DIR}/fullchain.pem:/gitlab-rd-web-ide/certs/fullchain.pem" \ -v "${CERTS_DIR}/privkey.pem:/gitlab-rd-web-ide/certs/privkey.pem" \ -v "${PROJECTS_DIR}:/projects" \ - registry.gitlab.com/gitlab-com/create-stage/editor-poc/remote-development/gitlab-rd-web-ide-docker:0.1-alpha \ + registry.gitlab.com/gitlab-org/remote-development/gitlab-rd-web-ide-docker:0.2-alpha \ --log-level warn --domain "${DOMAIN}" --ignore-version-mismatch ``` diff --git a/doc/user/project/repository/branches/img/branch_filter_search_box_v13_12.png b/doc/user/project/repository/branches/img/branch_filter_search_box_v13_12.png Binary files differdeleted file mode 100644 index a1cf9f10122..00000000000 --- a/doc/user/project/repository/branches/img/branch_filter_search_box_v13_12.png +++ /dev/null diff --git a/doc/user/project/repository/branches/img/compare_branches_v13_12.png b/doc/user/project/repository/branches/img/compare_branches_v13_12.png Binary files differdeleted file mode 100644 index 29627406729..00000000000 --- a/doc/user/project/repository/branches/img/compare_branches_v13_12.png +++ /dev/null diff --git a/doc/user/project/repository/branches/img/repository_filter_search_box_v13_12.png b/doc/user/project/repository/branches/img/repository_filter_search_box_v13_12.png Binary files differdeleted file mode 100644 index 230abf0d875..00000000000 --- a/doc/user/project/repository/branches/img/repository_filter_search_box_v13_12.png +++ /dev/null diff --git a/doc/user/project/repository/branches/img/swap_revisions_after_v13_12.png b/doc/user/project/repository/branches/img/swap_revisions_after_v13_12.png Binary files differdeleted file mode 100644 index 7eb10d10938..00000000000 --- a/doc/user/project/repository/branches/img/swap_revisions_after_v13_12.png +++ /dev/null diff --git a/doc/user/project/repository/branches/img/swap_revisions_before_v13_12.png b/doc/user/project/repository/branches/img/swap_revisions_before_v13_12.png Binary files differdeleted file mode 100644 index f92c4279871..00000000000 --- a/doc/user/project/repository/branches/img/swap_revisions_before_v13_12.png +++ /dev/null diff --git a/doc/user/project/repository/branches/img/view_branch_protections_v15_10.png b/doc/user/project/repository/branches/img/view_branch_protections_v15_10.png Binary files differnew file mode 100644 index 00000000000..09b30af91d0 --- /dev/null +++ b/doc/user/project/repository/branches/img/view_branch_protections_v15_10.png diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index 60504b94502..ef625739956 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -6,142 +6,148 @@ info: "To determine the technical writer assigned to the Stage/Group associated # Branches **(FREE)** -A branch is a version of a project's working tree. You create a branch for each -set of related changes you make. This keeps each set of changes separate from -each other, allowing changes to be made in parallel, without affecting each -other. +Branches are versions of a project's working tree. When you create a new +[project](../../index.md), GitLab creates a [default branch](default.md) (which +cannot be deleted) for your repository. Default branch settings can be configured +at the project, subgroup, group, or instance level. -After pushing your changes to a new branch, you can: +As your project grows, your team [creates](../web_editor.md#create-a-branch) more +branches, preferably by following [branch naming patterns](#prefix-branch-names-with-issue-numbers). +Each branch represents a set of changes, which allows development work to be done +in parallel. Development work in one branch does not affect another branch. -- Create a [merge request](../../merge_requests/index.md). You can streamline this process - by following [branch naming patterns](#naming). -- Perform inline code review. -- [Discuss](../../../discussions/index.md) your implementation with your team. -- Preview changes submitted to a new branch with [Review Apps](../../../../ci/review_apps/index.md). +Branches are the foundation of development in a project: -You can also request [approval](../../merge_requests/approvals/index.md) -from your managers. +1. To get started, create a branch and add commits to it. +1. When the work is ready for review, create a [merge request](../../merge_requests/index.md) to propose + merging the changes in your branch. To streamline this process, you should follow + [branch naming patterns](#prefix-branch-names-with-issue-numbers). +1. Preview changes in a branch with a [review app](../../../../ci/review_apps/index.md). +1. After the contents of your branch are merged, [delete the merged branch](#delete-merged-branches). -For more information on managing branches using the GitLab UI, see: +## Manage and protect branches -- [Default branches](default.md): When you create a new [project](../../index.md), GitLab creates a - default branch for the repository. You can change this setting at the project, - subgroup, group, or instance level. -- [Create a branch](../web_editor.md#create-a-branch) -- [Protected branches](../../protected_branches.md#protected-branches) -- [Delete merged branches](#delete-merged-branches) -- [Branch filter search box](#branch-filter-search-box) +GitLab provides you multiple methods to protect individual branches. These methods +ensure your branches receive oversight and quality checks from their creation to their deletion: -You can also manage branches using the -[command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). +- The [default branch](default.md) in your project receives extra protection. +- Configure [protected branches](../../protected_branches.md#protected-branches) + to restrict who can commit to a branch, merge other branches into it, or merge + the branch itself into another branch. +- Configure [approval rules](../../merge_requests/approvals/rules.md) to set review + requirements, including [security-related approvals](../../merge_requests/approvals/rules.md#security-approvals), before a branch can merge. +- Integrate with third-party [status checks](../../merge_requests/status_checks.md) + to ensure your branch contents meet your standards of quality. -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>Watch the video [GitLab Flow](https://www.youtube.com/watch?v=InKNIvky2KE). +You can manage your branches: -See also: +- With the GitLab user interface. +- With the [command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). +- With the [Branches API](../../../../api/branches.md). -- [Branches API](../../../../api/branches.md), for information on operating on repository branches using the GitLab API. -- [GitLab Flow](../../../../topics/gitlab_flow.md) documentation. -- [Getting started with Git](../../../../topics/git/index.md) and GitLab. - -## Naming - -Prefix a branch name with an issue number to streamline merge request creation. -When you create a merge request for a branch with a name beginning with an issue -number, GitLab: - -- Marks the issue as related. If your project is configured with a - [default closing pattern](../../issues/managing_issues.md#default-closing-pattern), - merging this merge request [also closes](../../issues/managing_issues.md#closing-issues-automatically) - the related issue. -- Copies label and milestone metadata from the issue. - -## Compare - -To compare branches in a repository: - -1. Navigate to your project's repository. -1. Select **Repository > Compare** in the sidebar. -1. Select the target repository to compare with the [repository filter search box](#repository-filter-search-box). -1. Select branches to compare using the [branch filter search box](#branch-filter-search-box). -1. Select **Compare** to view the changes inline: - -  - -## Delete merged branches - - +### View all branches -This feature allows merged branches to be deleted in bulk. Only branches that -have been merged into the project's default branch and -[are not protected](../../protected_branches.md) are deleted as part of -this operation. +To view and manage your branches in the GitLab user interface: -It's particularly useful to clean up old branches that were not deleted -automatically when a merge request was merged. - -## Repository filter search box +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Branches**. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52967) in GitLab 13.10. +On this page, you can: -This feature allows you to search and select a repository quickly when [comparing branches](#compare). +- See all branches, active branches, or stale branches. +- Create new branches. +- [Compare branches](#compare-branches). +- Delete merged branches. - +### View branches with configured protections -Search results appear in the following order: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88279) in GitLab 15.1 with a flag named `branch_rules`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/363170) in GitLab 15.10. -- Repositories with names exactly matching the search terms. -- Other repositories with names that include search terms, sorted alphabetically. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../feature_flags.md) named `branch_rules`. +On GitLab.com, this feature is available. -## Branch filter search box +Branches in your repository can be [protected](../../protected_branches.md) in multiple ways. You can: - +- Limit who can push to the branch. +- Limit who can merge the branch. +- Require approval of all changes. +- Require external tests to pass. -This feature allows you to search and select branches quickly. Search results appear in the following order: +The **Branch rules overview** page shows all branches with any configured protections, +and their protection methods: -- Branches with names that matched search terms exactly. -- Other branches with names that include search terms, sorted alphabetically. + -Sometimes when you have hundreds of branches you may want a more flexible matching pattern. In such cases you can use the following operators: +Prerequisites: -- `^` matches beginning of branch name, for example `^feat` would match `feat/user-authentication` -- `$` matches end of branch name, for example `widget$` would match `feat/search-box-widget` -- `*` wildcard matcher, for example `branch*cache*` would match `fix/branch-search-cache-expiration` +- You must have at least the Developer role in the project. -These operators can be mixed, for example `^chore/*migration$` would match `chore/user-data-migration` +To view the **Branch rules overview** list: -## Swap revisions +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Branch Rules** to view all branches with protections. + - To add protections to a new branch: + 1. Select **Add branch rule**. + 1. Select **Create protected branch**. + - To view more information about protections on an existing branch: + 1. Identify the branch you want more information about. + 1. Select **Details** to see information about its: + - [Branch protections](../../protected_branches.md). + - [Approval rules](../../merge_requests/approvals/rules.md). + - [Status checks](../../merge_requests/status_checks.md). + +## Prefix branch names with issue numbers + +To streamline the creation of merge requests, start your branch name with an +issue number. GitLab uses the issue number to import data into the merge request: + +- The issue is marked as related. The issue and merge request display links to each other. +- If your project is configured with a + [default closing pattern](../../issues/managing_issues.md#default-closing-pattern), + merging the merge request [also closes](../../issues/managing_issues.md#closing-issues-automatically) + the related issue. +- The issue milestone and labels are copied to the merge request. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60491) in GitLab 13.12. +## Compare branches - +> - Repository filter search box [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52967) in GitLab 13.10. +> - Revision swapping [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60491) in GitLab 13.12. -The Swap revisions feature allows you to swap the Source and Target revisions. When the Swap revisions button is selected, the selected revisions for Source and Target is swapped. +To compare branches in a repository: - +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Compare revisions**. +1. Select the **Source** branch to search for your desired branch. Exact matches are + shown first. You can refine your search with operators: + - `^` matches the beginning of the branch name: `^feat` matches `feat/user-authentication`. + - `$` matches the end of the branch name: `widget$` matches `feat/search-box-widget`. + - `*` matches using a wildcard: `branch*cache*` matches `fix/branch-search-cache-expiration`. + - You can combine operators: `^chore/*migration$` matches `chore/user-data-migration`. +1. Select the **Target** repository and branch. Exact matches are shown first. +1. Select **Compare** to show the list of commits, and changed files. To reverse + the **Source** and **Target**, select **Swap revisions**. -## View branches with configured protections **(FREE SELF)** +## Delete merged branches -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88279) in GitLab 15.1 with a flag named `branch_rules`. Disabled by default. + -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../feature_flags.md) named `branch_rules`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. +This feature allows merged branches to be deleted in bulk. Only branches that +have been merged into the project's default branch and +[are not protected](../../protected_branches.md) are deleted as part of +this operation. -Branches in your repository can be [protected](../../protected_branches.md) by limiting -who can push to a branch, require approval for those pushed changes, or merge those changes. -To help you track the protections for all branches, the **Branch rules overview** -page shows your branches with their configured rules. +It's particularly useful to clean up old branches that were not deleted +automatically when a merge request was merged. -To view the **Branch rules overview** list: +## Related topics -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Repository**. -1. Expand **Branch Rules** to view all branches with protections. -1. Select **Details** next to your desired branch to show information about its: - - [Branch protections](../../protected_branches.md). - - [Approval rules](../../merge_requests/approvals/rules.md). - - [Status checks](../../merge_requests/status_checks.md). +- [Protected branches](../../protected_branches.md) user documentation. +- [Branches API](../../../../api/branches.md), for information on operating on repository branches using the GitLab API. +- [Protected Branches API](../../../../api/protected_branches.md). +- [Getting started with Git](../../../../topics/git/index.md) and GitLab. ## Troubleshooting diff --git a/doc/user/project/repository/file_finder.md b/doc/user/project/repository/file_finder.md index f9a5d4e3aa7..3fcc19db344 100644 --- a/doc/user/project/repository/file_finder.md +++ b/doc/user/project/repository/file_finder.md @@ -11,7 +11,7 @@ The file finder feature allows you to search for a file in a repository using th GitLab UI. To use it: 1. Go to your project's **Repository > Files**. -1. In the upper right corner, select **Find File**. +1. In the upper-right corner, select **Find File**. If you prefer to keep your fingers on the keyboard, use the [shortcut button](../../shortcuts.md), which you can invoke from anywhere diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 929751b7247..8c7c94613a7 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -23,7 +23,7 @@ submit them through a merge request to the repository you don't have access to. To fork an existing project in GitLab: -1. On the project's home page, in the upper right, select **{fork}** **Fork**: +1. On the project's homepage, in the upper-right corner, select **Fork** (**{fork}**):  1. Optional. Edit the **Project name**. 1. For **Project URL**, select the [namespace](../../namespace/index.md) @@ -137,3 +137,11 @@ You can unlink your fork from its upstream project in the [advanced settings](.. - GitLab blog post: [How to keep your fork up to date with its origin](https://about.gitlab.com/blog/2016/12/01/how-to-keep-your-fork-up-to-date-with-its-origin/). - GitLab community forum: [Refreshing a fork](https://forum.gitlab.com/t/refreshing-a-fork/). + +## Troubleshooting + +### An error occurred while forking the project. Please try again + +This error can be due to a mismatch in shared runner settings between the forked project +and the new namespace. See [Forks](../../../ci/runners/configure_runners.md#forks) +in the Runner documentation for more information. diff --git a/doc/user/project/repository/git_blame.md b/doc/user/project/repository/git_blame.md index 78fcdec2a0b..fbf29bddd46 100644 --- a/doc/user/project/repository/git_blame.md +++ b/doc/user/project/repository/git_blame.md @@ -14,7 +14,7 @@ commit hash. To view it for a file: 1. Go to your project's **Repository > Files**. 1. Select the file you want to review. -1. In the upper right corner, select **Blame**. +1. In the upper-right corner, select **Blame**. When you select **Blame**, this information is displayed: diff --git a/doc/user/project/repository/git_history.md b/doc/user/project/repository/git_history.md index 321b9a5eb53..a9228b8cabd 100644 --- a/doc/user/project/repository/git_history.md +++ b/doc/user/project/repository/git_history.md @@ -12,7 +12,7 @@ Git file History provides information about the commit history associated with a file. To use it: 1. Go to your project's **Repository > Files**. -1. In the upper right corner, select **History**. +1. In the upper-right corner, select **History**. When you select **History**, this information is displayed: diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 64f19d1a92c..a2dd2488961 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -43,7 +43,7 @@ To view a user's public GPG key, you can either: - Go to `https://gitlab.example.com/<USERNAME>.gpg`. GitLab displays the GPG key, if the user has configured one, or a blank page for users without a configured GPG key. -- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper right +- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper-right corner of the user's profile, select **View public GPG keys** (**{key}**). ## Configure commit signing diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 5b4e1b14489..1c64a47985b 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -25,7 +25,7 @@ You can add files to a repository: - When you create a project. - After you create a project: - By using [the web editor](web_editor.md). - - [From the command line](../../../gitlab-basics/command-line-commands.md). + - From the command line. ## Commit changes to a repository @@ -235,9 +235,9 @@ The size can differ slightly from one instance to another due to compression, ho Administrators can set a [repository size limit](../../admin_area/settings/account_and_limit_settings.md). [GitLab sets the size limits for GitLab.com](../../gitlab_com/index.md#account-and-limit-settings). -## Repository contributor graph +## Repository contributor statistics -All code contributors are displayed under your project's **Repository > Contributors**. +All code contributors are displayed under your project's **Repository > Contributor statistics**. The graph shows the contributor with the most commits to the fewest. diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md index f06dd747897..ea58c472f4a 100644 --- a/doc/user/project/repository/mirror/push.md +++ b/doc/user/project/repository/mirror/push.md @@ -26,8 +26,10 @@ When you push a change to the upstream repository, the push mirror receives it: - Within five minutes. - Within one minute, if you enabled **Only mirror protected branches**. -In the case of a diverged branch, an error displays in the **Mirroring repositories** -section. +When a branch is merged into the default branch and deleted in the source project, +it is deleted from the remote mirror on the next push. Branches with unmerged +changes are kept. If a branch diverges, the **Mirroring repositories** section +displays an error. ## Configure push mirroring diff --git a/doc/user/project/repository/tags/img/tag-display_v15_9.png b/doc/user/project/repository/tags/img/tag-display_v15_9.png Binary files differnew file mode 100644 index 00000000000..015df07d025 --- /dev/null +++ b/doc/user/project/repository/tags/img/tag-display_v15_9.png diff --git a/doc/user/project/repository/tags/img/tags_commits_view_v15_10.png b/doc/user/project/repository/tags/img/tags_commits_view_v15_10.png Binary files differnew file mode 100644 index 00000000000..247d2f368d5 --- /dev/null +++ b/doc/user/project/repository/tags/img/tags_commits_view_v15_10.png diff --git a/doc/user/project/repository/tags/index.md b/doc/user/project/repository/tags/index.md new file mode 100644 index 00000000000..706b50a916d --- /dev/null +++ b/doc/user/project/repository/tags/index.md @@ -0,0 +1,108 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tags **(FREE)** + +In Git, a tag marks an important point in a repository's history. +Git supports two types of tags: + +- **Lightweight tags** point to specific commits, and contain no other information. + Also known as soft tags. Create or remove them as needed. +- **Annotated tags** contain metadata, can be signed for verification purposes, + and can't be changed. + +The creation or deletion of a tag can be used as a trigger for automation, including: + +- Using a [webhook](../../integrations/webhook_events.md#tag-events) to automate actions + like Slack notifications. +- Signaling a [repository mirror](../mirror/index.md) to update. +- Running a CI/CD pipeline with [`if: $CI_COMMIT_TAG`](../../../../ci/jobs/job_control.md#common-if-clauses-for-rules). + +When you [create a release](../../releases/index.md), +GitLab also creates a tag to mark the release point. +Many projects combine an annotated release tag with a stable branch. Consider +setting deployment or release tags automatically. + +In the GitLab UI, each tag displays: + + + +- The tag name. (**{tag}**) +- Optional. If the tag is [protected](../../protected_tags.md), a **protected** badge. +- The commit SHA (**{commit}**), linked to the commit's contents. +- The commit's title and creation date. +- Optional. A link to the release (**{rocket}**). +- Optional. If a pipeline has been run, the current pipeline status. +- Download links to the source code and artifacts linked to the tag. +- A [**Create release**](../../releases/index.md#create-a-release) (**{pencil}**) link. +- A link to delete the tag. + +## View tags for a project + +To view all existing tags for a project: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Tags**. + +## View tagged commits in the commits list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18795) in GitLab 15.10. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Commits**. +1. Commits with a tag are labeled with a tag icon (**{tag}**) and the name of the tag. + This example shows a commit tagged `v1.26.0`: + +  + +To view the list of commits in this tag, select the tag name. + +## Create a tag + +Tags can be created from the command line, or the GitLab UI. + +### From the command line + +To create either a lightweight or annotated tag from the command line, and push it upstream: + +1. To create a lightweight tag, run the command `git tag TAG_NAME`, changing + `TAG_NAME` to your desired tag name. +1. To create an annotated tag, run one of the versions of `git tag` from the command line: + + ```shell + # In this short version, the annotated tag's name is "v1.0", + # and the message is "Version 1.0". + git tag -a v1.0 -m "Version 1.0" + + # Use this version to write a longer tag message + # for annotated tag "v1.0" in your text editor. + git tag -a v1.0 + ``` + +1. Push your tags upstream with `git push origin --tags`. + +### From the UI + +To create a tag from the GitLab UI: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Tags**. +1. Select **New tag**. +1. Provide a **Tag name**. +1. For **Create from**, select an existing branch name, tag, or commit SHA. +1. Optional. Add a **Message** to create an annotated tag, or leave blank to + create a lightweight tag. +1. Select **Create tag**. + +## Prevent tag deletion **(PREMIUM)** + +To prevent users from removing a tag with `git push`, create a [push rule](../push_rules.md). + +## Related topics + +- [Tagging](https://git-scm.com/book/en/v2/Git-Basics-Tagging) Git reference page. +- [Protected tags](../../protected_tags.md). +- [Tags API](../../../../api/tags.md). diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 80b0ada6f7b..ace7e119469 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -26,11 +26,32 @@ for any change you commit through the Web Editor. To create a text file in the Web Editor: 1. On the top bar, select **Main menu > Projects** and find your project. -1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). +1. From the project dashboard or repository, next to the branch name, + select the plus icon (**{plus}**). 1. From the dropdown list, select **New file**. -1. Complete the fields. To create a merge request with the new file, ensure the **Start a new merge request with these changes** checkbox is selected. +1. Complete the fields. +1. To create a merge request with the new file, ensure the **Start a new merge request with these changes** checkbox is selected, if you had chosen a **Target branch** other than the [default branch (such as `main`)](../../../user/project/repository/branches/default.md). 1. Select **Commit changes**. +### Create a file from a template + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Files**. +1. Next to the project name, select the plus icon (**{plus}**) to display a + dropdown list, then select **New file** from the list. +1. For **Filename**, provide one of the filenames that GitLab provides a template for: + - `.gitignore` + - `.gitlab-ci.yml` + - `LICENSE` + - `Dockerfile` +1. Select **Apply a template**, then select the template you want to apply. +1. Make your changes to the file. +1. Provide a **Commit message**. +1. Enter a **Target branch** to merge into. To create a new merge request with + your changes, enter a branch name that is not your repository's + [default branch](../../../user/project/repository/branches/default.md), +1. Select **Commit changes** to add the commit to your branch. + ## Edit a file To edit a text file in the Web Editor: @@ -87,6 +108,9 @@ To link to a single line, you can also: To upload a binary file in the Web Editor: +<!-- This list is duplicated at doc/gitlab-basics/add-file.md#from-the-ui --> +<!-- For why we duplicated the info, see https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111072#note_1267429478 --> + 1. On the top bar, select **Main menu > Projects** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **Upload file**. @@ -115,7 +139,7 @@ To create a [branch](branches/index.md) in the Web Editor: ## Create a tag -You can create [tags](../../../topics/git/tags.md) to mark milestones such as +You can create [tags](tags/index.md) to mark milestones such as production releases and release candidates. To create a tag in the Web Editor: 1. On the top bar, select **Main menu > Projects** and find your project. diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index b04905e6b34..d385daa4fa1 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -1,7 +1,7 @@ --- type: reference, howto stage: Plan -group: Certify +group: Product Planning info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -235,9 +235,9 @@ Before you import your file: To import requirements: 1. In a project, go to **Issues > Requirements**. - - If the project already has existing requirements, select the import icon (**{import}**) in the - upper right. - - For a project without any requirements, select **Import CSV** in the middle of the page. + - For a project with requirements, in the + upper-right corner, select the import icon (**{import}**). + - For a project without requirements, in the middle of the page, select **Import CSV**. 1. Select the file and select **Import requirements**. The file is processed in the background and a notification email is sent @@ -296,7 +296,7 @@ Prerequisite: To export requirements: 1. In a project, go to **Issues > Requirements**. -1. In the upper right, select the **Export as CSV** icon (**{export}**). +1. In the upper-right corner, select **Export as CSV** (**{export}**). A confirmation modal appears. diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 22297149561..c4fcdc5733e 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -8,29 +8,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Moved to GitLab Free in 13.2. -Service Desk is a module that allows your team to connect -with any external party through email, without any external tools. -An ongoing conversation in the same place as where your software -is built ensures user feedback ends up where it's needed. +With Service Desk, your customers +can email you bug reports, feature requests, or general feedback. +Service Desk provides a unique email address, so they don't need their own GitLab accounts. -With Service Desk, you can provide efficient email support to your customers. They can -email you bug reports, feature requests, or general feedback. They all end up in your -GitLab project as new issues. In turn, your team can respond directly from the project. +Service Desk emails are created in your GitLab project as new issues. +Your team can respond directly from the project, while customers interact with the thread only +through email. -As Service Desk is built right into GitLab itself, the complexity and inefficiencies -of multiple tools and external integrations are eliminated. This significantly shortens -the cycle time from feedback to software update. - -For an overview, check the video demonstration on [GitLab Service Desk](https://about.gitlab.com/blog/2017/05/09/demo-service-desk/). - -## How it works - -GitLab Service Desk enables people to create issues in your -GitLab instance without needing their own user account. - -It provides a unique email address for end users to create issues in a project. -Follow-up notes can be sent either through the GitLab interface or by email. End -users only see the thread through email. +## Service Desk workflow For example, let's assume you develop a game for iOS or Android. The codebase is hosted in your GitLab instance, built and deployed @@ -43,23 +29,24 @@ Here's how Service Desk works for you: 1. Each email they send creates an issue in the appropriate project. 1. Your team members go to the Service Desk issue tracker, where they can see new support requests and respond inside associated issues. -1. Your team communicates back and forth with the customer to understand the request. +1. Your team communicates with the customer to understand the request. 1. Your team starts working on implementing code to solve your customer's problem. -1. When your team finishes the implementation, whereupon the merge request is merged and the issue +1. When your team finishes the implementation, the merge request is merged and the issue is closed automatically. -1. The customer's requests are handled through email, without ever having access to your - GitLab instance. -1. Your team saved time by not having to leave GitLab (or setup any integrations) to follow up with - your customer. + +Meanwhile: + +- The customer interacts with your team entirely through email, without needing access to your + GitLab instance. +- Your team saves time by not having to leave GitLab (or set up integrations) to follow up with + your customer. ## Configuring Service Desk Users with Maintainer and higher access in a project can configure Service Desk. Service Desk issues are [confidential](issues/confidential_issues.md), so they are -only visible to project members. In GitLab 11.7 we updated the generated email -address format. The older format is still supported, so existing aliases or -contacts still work. +only visible to project members. If you have [templates](description_templates.md) in your repository, you can optionally select one from the selector menu to append it to all Service Desk issues. @@ -93,7 +80,6 @@ displayed in the information note. ### Using customized email templates -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2460) in GitLab 12.7. > - Moved from GitLab Premium to GitLab Free in 13.2. > - `UNSUBSCRIBE_URL`, `SYSTEM_HEADER`, `SYSTEM_FOOTER`, and `ADDITIONAL_TEXT` placeholders [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285512) in GitLab 15.9. @@ -174,8 +160,6 @@ To use a custom description template with Service Desk: ### Using a custom email display name -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7529) in GitLab 12.8. - You can customize the email display name. Emails sent from Service Desk have this name in the `From` header. The default display name is `GitLab Support Bot`. @@ -207,15 +191,12 @@ you can customize the mailbox used by Service Desk. This allows you to have a separate email address for Service Desk by also configuring a [custom suffix](#configuring-a-custom-email-address-suffix) in project settings. -The `address` must include the `+%{key}` placeholder in the 'user' -portion of the address, before the `@`. The placeholder is used to identify the project -where the issue should be created. +Prerequisites: -NOTE: -When configuring a custom mailbox, the `service_desk_email` and `incoming_email` -configurations must always use separate mailboxes. It's important, because -emails picked from `service_desk_email` mailbox are processed by a different -worker and it would not recognize `incoming_email` emails. +- The `address` must include the `+%{key}` placeholder in the `user` portion of the address, + before the `@`. The placeholder is used to identify the project where the issue should be created. +- The `service_desk_email` and `incoming_email` configurations must always use separate mailboxes + to make sure Service Desk emails are processed correctly. To configure a custom mailbox for Service Desk with IMAP, add the following snippets to your configuration file in full: @@ -504,15 +485,18 @@ You can read and write comments as you usually do in GitLab: #### Receiving files attached to comments as email attachments -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11733) in GitLab 15.8 [with a flag](../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11733) in GitLab 15.8 [with a flag](../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/386860) in GitLab 15.10. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. -On GitLab.com, this feature is not available. +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `service_desk_new_note_email_native_attachments`. +On GitLab.com, this feature is available. If a comment contains any attachments and their total size is less than or equal to 10 MB, these attachments are sent as part of the email. In other cases, the email contains links to the attachments. +In GitLab 15.9 and earlier, uploads to a comment are sent as links in the email. + #### Special HTML formatting in HTML emails <!-- When the feature flag is removed, delete this topic and add as a line in version history under one of the topics above this one.--> @@ -567,8 +551,3 @@ in both issues. They continue to receive any notifications in the old issue and Your emails might be ignored because they contain one of the [email headers that GitLab ignores](../../administration/incoming_email.md#rejected-headers). - -### Responses to a Service Desk issue do not generate emails - -Your issue might have been moved to a different project. -Moved Service Desk issues do not retain email participants. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 3715eef08e0..230795222a0 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -9,8 +9,8 @@ info: "To determine the technical writer assigned to the Stage/Group associated Existing projects on any self-managed GitLab instance or GitLab.com can be exported to a file and then imported into a new GitLab instance. You can also: -- [Migrate groups](../../group/import/index.md) using the preferred method. -- [Migrate groups using file exports](../../group/settings/import_export.md). +- Migrate projects when you [migrate groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +- [Migrate groups by using file exports](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). GitLab maps user contributions correctly when an admin access token is used to perform the import. @@ -155,7 +155,8 @@ Items that are **not** exported include: - Repository size limits - Deploy keys allowed to push to protected branches - Secure Files -- [Activity logs for Git-related events](https://gitlab.com/gitlab-org/gitlab/-/issues/214700). For example, pushing and creating tags +- [Activity logs for Git-related events](https://gitlab.com/gitlab-org/gitlab/-/issues/214700) (for example, pushing and creating tags) +- Security policies associated with your project ## Import a project and its data @@ -200,7 +201,7 @@ Deploy keys aren't imported. To use deploy keys, you must enable them in your im ### Import large projects **(FREE SELF)** -If you have a larger project, consider using a Rake task as described in the [developer documentation](../../../development/import_project.md#importing-via-a-rake-task). +If you have a larger project, consider [using a Rake task](../../../administration/raketasks/project_import_export.md#import-large-projects). ## Automate group and project import **(PREMIUM)** diff --git a/doc/user/project/settings/import_export_troubleshooting.md b/doc/user/project/settings/import_export_troubleshooting.md index 00edeef5cfa..82412a1dcbf 100644 --- a/doc/user/project/settings/import_export_troubleshooting.md +++ b/doc/user/project/settings/import_export_troubleshooting.md @@ -48,41 +48,41 @@ reduce the repository size for another import attempt: 1. Create a temporary working directory from the export: - ```shell - EXPORT=<filename-without-extension> + ```shell + EXPORT=<filename-without-extension> - mkdir "$EXPORT" - tar -xf "$EXPORT".tar.gz --directory="$EXPORT"/ - cd "$EXPORT"/ - git clone project.bundle + mkdir "$EXPORT" + tar -xf "$EXPORT".tar.gz --directory="$EXPORT"/ + cd "$EXPORT"/ + git clone project.bundle - # Prevent interference with recreating an importable file later - mv project.bundle ../"$EXPORT"-original.bundle - mv ../"$EXPORT".tar.gz ../"$EXPORT"-original.tar.gz + # Prevent interference with recreating an importable file later + mv project.bundle ../"$EXPORT"-original.bundle + mv ../"$EXPORT".tar.gz ../"$EXPORT"-original.tar.gz - git switch --create smaller-tmp-main - ``` + git switch --create smaller-tmp-main + ``` 1. To reduce the repository size, work on this `smaller-tmp-main` branch: [identify and remove large files](../repository/reducing_the_repo_size_using_git.md) or [interactively rebase and fixup](../../../topics/git/git_rebase.md#interactive-rebase) to reduce the number of commits. - ```shell - # Reduce the .git/objects/pack/ file size - cd project - git reflog expire --expire=now --all - git gc --prune=now --aggressive - - # Prepare recreating an importable file - git bundle create ../project.bundle <default-branch-name> - cd .. - mv project/ ../"$EXPORT"-project - cd .. - - # Recreate an importable file - tar -czf "$EXPORT"-smaller.tar.gz --directory="$EXPORT"/ . - ``` + ```shell + # Reduce the .git/objects/pack/ file size + cd project + git reflog expire --expire=now --all + git gc --prune=now --aggressive + + # Prepare recreating an importable file + git bundle create ../project.bundle <default-branch-name> + cd .. + mv project/ ../"$EXPORT"-project + cd .. + + # Recreate an importable file + tar -czf "$EXPORT"-smaller.tar.gz --directory="$EXPORT"/ . + ``` 1. Import this new, smaller file into GitLab. 1. In a full clone of the original repository, @@ -265,12 +265,12 @@ Marked stuck import jobs as failed. JIDs: xyz | Problem | Possible solutions | | -------- | -------- | | [Slow JSON](https://gitlab.com/gitlab-org/gitlab/-/issues/25251) loading/dumping models from the database | [split the worker](https://gitlab.com/gitlab-org/gitlab/-/issues/25252) | -| | Batch export -| | Optimize SQL -| | Move away from `ActiveRecord` callbacks (difficult) -| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857) | DB Commit sweet spot that uses less memory | +| | Batch export | +| | Optimize SQL | +| | Move away from `ActiveRecord` callbacks (difficult) | +| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857)) | DB Commit sweet spot that uses less memory | | | [Netflix Fast JSON API](https://github.com/Netflix/fast_jsonapi) may help | -| | Batch reading/writing to disk and any SQL +| | Batch reading/writing to disk and any SQL | ### Temporary solutions diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index ed4eea56514..7250406144f 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: 'To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments' type: reference, index, howto --- @@ -87,7 +87,7 @@ Use the toggles to enable or disable features in the project. | **Container Registry** | | Activates a [registry](../../packages/container_registry/index.md) for your Docker images. | | **Analytics** | ✓ | Enables [analytics](../../analytics/index.md). | | **Requirements** | ✓ | Control access to [Requirements Management](../requirements/index.md). | -| **Security & Compliance** | ✓ | Control access to [security features](../../application_security/index.md). | +| **Security and Compliance** | ✓ | Control access to [security features](../../application_security/index.md). | | **Wiki** | ✓ | Enables a separate system for [documentation](../wiki/index.md). | | **Snippets** | ✓ | Enables [sharing of code and text](../../snippets.md). | | **Pages** | ✓ | Allows you to [publish static websites](../pages/index.md). | diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index f9218a228ca..b4d48e52e46 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -48,9 +48,6 @@ You cannot use project access tokens to create other group, project, or personal Project access tokens inherit the [default prefix setting](../../admin_area/settings/account_and_limit_settings.md#personal-access-token-prefix) configured for personal access tokens. -NOTE: -Project access tokens are not FIPS compliant and creation and use are disabled when [FIPS mode](../../../development/fips_compliance.md) is enabled. - ## Create a project access token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89114) in GitLab 15.1, Owners can select Owner role for project access tokens. @@ -121,12 +118,8 @@ The bot users for projects have [permissions](../../permissions.md#project-membe selected role and [scope](#scopes-for-a-project-access-token) of the project access token. - The name is set to the name of the token. -- The username is set to `project_{project_id}_bot` for the first access token. For example, `project_123_bot`. -- The email is set to `project{project_id}_bot@noreply.{Gitlab.config.gitlab.host}`. For example, `project123_bot@noreply.example.com`. -- For additional access tokens in the same project, the username is set to `project_{project_id}_bot{bot_count}`. For - example, `project_123_bot1`. -- For additional access tokens in the same project, the email is set to `project{project_id}_bot{bot_count}@noreply.{Gitlab.config.gitlab.host}`. - For example, `project123_bot1@noreply.example.com`. +- The username is set to `project_{project_id}_bot_{random_string}`. For example, `project_123_bot_4ffca233d8298ea1`. +- The email is set to `project_{project_id}_bot_{random_string}@noreply.{Gitlab.config.gitlab.host}`. For example, `project_123_bot_4ffca233d8298ea1@noreply.example.com`. API calls made with a project access token are associated with the corresponding bot user. @@ -143,3 +136,7 @@ When the project access token is [revoked](#revoke-a-project-access-token): - All records are moved to a system-wide user with the username [Ghost User](../../profile/account/delete_account.md#associated-records). See also [Bot users for groups](../../group/settings/group_access_tokens.md#bot-users-for-groups). + +## Token availability + +Project access tokens are only available in paid subscriptions, and not available in trial subscriptions. For more information, see the ["What is included" section of the GitLab Trial FAQ](https://about.gitlab.com/free-trial/#what-is-included-in-my-free-trial-what-is-excluded). diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 5e9f648daba..e2a74f02008 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -22,7 +22,7 @@ and from merge requests: ### When viewing a file or the repository file list - 1. In the upper right corner of the page, select **Open in Web IDE** if it is visible. + 1. In the upper-right corner of the page, select **Open in Web IDE** if it is visible. 1. If **Open in Web IDE** is not visible: 1. Select the (**{chevron-lg-down}**) next to **Edit** or **Gitpod**, depending on your configuration. 1. Select **Open in Web IDE** from the list to display it as the editing option. @@ -31,7 +31,7 @@ and from merge requests: ### When viewing a merge request 1. Go to your merge request. - 1. In the upper right corner, select **Code > Open in Web IDE**. + 1. In the upper-right corner, select **Code > Open in Web IDE**. ## File finder diff --git a/doc/user/project/web_ide_beta/index.md b/doc/user/project/web_ide_beta/index.md index fe110baf578..635b5e12575 100644 --- a/doc/user/project/web_ide_beta/index.md +++ b/doc/user/project/web_ide_beta/index.md @@ -40,7 +40,7 @@ or a merge request. To open the Web IDE Beta from a file or the repository file list: -- In the upper right of the page, select **Open in Web IDE**. +- In the upper-right corner of the page, select **Open in Web IDE**. If **Open in Web IDE** is not visible: @@ -53,7 +53,7 @@ If **Open in Web IDE** is not visible: To open the Web IDE Beta from a merge request: 1. Go to your merge request. -1. In the upper right corner, select **Code > Open in Web IDE**. +1. In the upper-right corner, select **Code > Open in Web IDE**. ## Open a file in the Web IDE Beta @@ -64,6 +64,26 @@ To open any file by its name:  +## Switch branches + +The Web IDE Beta uses the currently selected branch by default. +To switch branches in the Web IDE Beta: + +1. On the status bar, in the lower-left corner, select the current branch name. +1. In the search box, start typing the branch name. +1. From the dropdown list, select the branch. + +## Create a branch + +To create a branch from the current branch in the Web IDE Beta: + +1. On the status bar, in the lower-left corner, select the current branch name. +1. From the dropdown list, select **Create new branch...**. +1. Enter the branch name. +1. Press <kbd>Enter</kbd>. + +If you don't have write access to the repository, **Create new branch...** is not visible. + ## Search across files You can use VS Code to quickly search all files in the opened folder. @@ -84,6 +104,15 @@ Your `CHANGES`, `STAGED CHANGES`, and `MERGE CHANGES` are displayed. For details, see the [VS Code documentation](https://code.visualstudio.com/docs/sourcecontrol/overview#_commit). +## Open the command palette + +In the Web IDE Beta, you can access many commands through the command palette. +To open the command palette and run a command in the Web IDE Beta: + +1. Press <kbd>F1</kbd> or <kbd>Shift</kbd>+<kbd>Command</kbd>+<kbd>P</kbd>. +1. In the search box, start typing the command name. +1. From the dropdown list, select the command. + ## Stop using the Web IDE Beta If you do not want to use the Web IDE Beta, you can change your personal preferences. diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 53aaa41319b..1513ea18ed2 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -38,8 +38,8 @@ To access a group wiki: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9. Users with the Owner role in a group can -[import and export group wikis](../../group/settings/import_export.md) when importing -or exporting a group. +[import or export a group wiki](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) when they +import or export a group. Content created in a group wiki is not deleted when an account is downgraded or a GitLab trial ends. The group wiki data is exported whenever the group owner of @@ -48,8 +48,8 @@ the wiki is exported. To access the group wiki data from the export file if the feature is no longer available, you have to: -1. Extract the [export file tarball](../../group/settings/import_export.md) with - this command, replacing `FILENAME` with your file's name: +1. Extract the [export file tarball](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) + with this command, replacing `FILENAME` with your file's name: `tar -xvzf FILENAME.tar.gz` 1. Browse to the `repositories` directory. This directory contains a [Git bundle](https://git-scm.com/docs/git-bundle) with the extension `.wiki.bundle`. diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index a0e6a78dfe2..24ca86fc93b 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- @@ -11,12 +11,9 @@ code are saved in projects, and most features are in the scope of projects. ## View projects -To view projects, on the top bar, select **Main menu > Projects > View all projects**. +To view all your projects, on the top bar, select **Main menu > Projects > View all projects**. -NOTE: -The **Explore projects** tab is visible to unauthenticated users unless the -[**Public** visibility level](../admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) -is restricted. Then the tab is visible only to authenticated users. +To browse all public projects, select **Main menu > Explore > Projects**. ### Who can view the Projects page @@ -63,7 +60,7 @@ You can add a star to projects you use frequently to make them easier to find. To add a star to a project: 1. On the top bar, select **Main menu > Projects** and find your project. -1. In the upper right corner of the page, select **Star**. +1. In the upper-right corner of the page, select **Star**. ## View starred projects @@ -241,15 +238,15 @@ Configure Git to either: - Embed credentials in the request URL: - ```shell - git config --global url."https://${user}:${personal_access_token}@gitlab.example.com".insteadOf "https://gitlab.example.com" - ``` + ```shell + git config --global url."https://${user}:${personal_access_token}@gitlab.example.com".insteadOf "https://gitlab.example.com" + ``` - Use SSH instead of HTTPS: - ```shell - git config --global url."git@gitlab.example.com:".insteadOf "https://gitlab.example.com/" - ``` + ```shell + git config --global url."git@gitlab.example.com:".insteadOf "https://gitlab.example.com/" + ``` ### Disable Go module fetching for private projects @@ -263,8 +260,8 @@ the [environment variables](../../development/go_guide/dependencies.md#proxies): To disable fetching: 1. Disable `GOPRIVATE`: - - To disable queries for one project, disable `GOPRIVATE=gitlab.example.com/my/private/project`. - - To disable queries for all projects on GitLab.com, disable `GOPRIVATE=gitlab.example.com`. + - To disable queries for one project, disable `GOPRIVATE=gitlab.example.com/my/private/project`. + - To disable queries for all projects on GitLab.com, disable `GOPRIVATE=gitlab.example.com`. 1. Disable proxy queries in `GONOPROXY`. 1. Disable checksum queries in `GONOSUMDB`. @@ -291,8 +288,8 @@ To access the Geo secondary server with SSH: git config --global url."git@gitlab-secondary.example.com".insteadOf "http://gitlab.example.com" ``` - - For `gitlab.example.com`, use the primary site domain name. - - For `gitlab-secondary.example.com`, use the secondary site domain name. + - For `gitlab.example.com`, use the primary site domain name. + - For `gitlab-secondary.example.com`, use the secondary site domain name. 1. Ensure the client is set up for SSH access to GitLab repositories. You can test this on the primary, and GitLab replicates the public key to the secondary. diff --git a/doc/user/public_access.md b/doc/user/public_access.md index 2a9a9fb84fe..87380ec324e 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- diff --git a/doc/user/report_abuse.md b/doc/user/report_abuse.md index aabc9c5dff1..a20a699e550 100644 --- a/doc/user/report_abuse.md +++ b/doc/user/report_abuse.md @@ -34,6 +34,8 @@ To report abuse from a user's profile page: ## Report abuse from a user's comment +> Reporting abuse from comments in epics [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/389992) in GitLab 15.10. + To report abuse from a user's comment: 1. In the comment, in the upper-right corner, select **More actions** (**{ellipsis_v}**). diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 372ec141112..6e8a7e7c1cf 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -1,6 +1,6 @@ --- -stage: Manage -group: Organization +stage: Data Stores +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index ed1d3b1d290..1444e5385f9 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -5,18 +5,18 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Advanced Search **(PREMIUM)** +# Advanced search **(PREMIUM)** > Moved to GitLab Premium in 13.9. -You can use Advanced Search for faster, more efficient search across the entire GitLab -instance. Advanced Search is based on Elasticsearch, a purpose-built full-text search +You can use advanced search for faster, more efficient search across the entire GitLab +instance. Advanced search is based on Elasticsearch, a purpose-built full-text search engine you can horizontally scale to get results in up to a second in most cases. You can find code you want to update in all projects at once to save maintenance time and promote innersourcing. -You can use Advanced Search in: +You can use advanced search in: - Projects - Issues @@ -29,37 +29,22 @@ You can use Advanced Search in: - Commits - Project wikis (not [group wikis](../project/wiki/group.md)) -For Advanced Search: +## Enable advanced search -- You can only search files smaller than 1 MB. - For self-managed GitLab instances, an administrator can - [change this limit](../../integration/advanced_search/elasticsearch.md#advanced-search-configuration). -- You can't use any of the following characters in the search query: - - ```plaintext - . , : ; / ` ' = ? $ & ^ | ~ < > ( ) { } [ ] @ - ``` - -- Only the default branch of a project is indexed for code search. - In a non-default branch, basic search is used. -- Search results show only the first match in a file, - but there might be more results in that file. - -## Enable Advanced Search - -- On GitLab.com, Advanced Search is enabled for groups with paid subscriptions. +- On GitLab.com, advanced search is enabled for groups with paid subscriptions. - For self-managed GitLab instances, an administrator must - [enable Advanced Search](../../integration/advanced_search/elasticsearch.md#enable-advanced-search). + [enable advanced search](../../integration/advanced_search/elasticsearch.md#enable-advanced-search). ## Syntax -Advanced Search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). The syntax supports both exact and fuzzy search queries. +Advanced search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). The syntax supports both exact and fuzzy search queries. <!-- markdownlint-disable --> | Syntax | Description | Example | |--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `"` | Exact search | [`"gem sidekiq"`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=%22gem+sidekiq%22) | +| `~` | Fuzzy search | [`J~ Doe`](https://gitlab.com/search?scope=users&search=j%7E+doe) | | <code>|</code> | Or | [<code>display | banner</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+%7C+banner) | | `+` | And | [`display +banner`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=display+%2Bbanner&snippets=) | | `-` | Exclude | [`display -banner`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=display+-banner) | @@ -68,16 +53,23 @@ Advanced Search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elas | `#` | Issue ID | [`#23456`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964) | | `!` | Merge request ID | [`!23456`](https://gitlab.com/search?snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964) | +### Refining user search + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/388409) in GitLab 15.10. + +In user search, a [fuzzy query](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/query-dsl-fuzzy-query.html) is used by default. You can refine your search with [Elasticsearch syntax](#syntax). + ### Code search | Syntax | Description | Example | |--------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `filename:` | Filename | [`filename:*spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=filename%3A*spec.rb&group_id=9970&project_id=278964) | -| `path:` | Repository location | [`path:spec/workers/`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=path%3Aspec%2Fworkers&snippets=) | -| `extension:` | File extension without `.` | [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | -| `blob:` | Git object ID | [`blob:998707*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707*&group_id=9970) | +| `path:` | Repository location <sup>1</sup> | [`path:spec/workers/`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=path%3Aspec%2Fworkers&snippets=) | +| `extension:` | File extension without `.` <sup>2</sup> | [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | +| `blob:` | Git object ID <sup>2</sup> | [`blob:998707*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707*&group_id=9970) | -`extension:` and `blob:` return exact matches only. +1. `path:` returns matches for full paths or subpaths. +1. `extension:` and `blob:` return exact matches only. ### Examples @@ -87,5 +79,21 @@ Advanced Search uses [Elasticsearch syntax](https://www.elastic.co/guide/en/elas | [`RSpec.describe Resolvers -*builder`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=RSpec.describe+Resolvers+-*builder) | Returns `RSpec.describe Resolvers` that does not start with `builder`. | | [<code>bug | (display +banner)</code>](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) | Returns `bug` or both `display` and `banner`. | | [<code>helper -extension:yml -extension:js</code>](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=helper+-extension%3Ayml+-extension%3Ajs&snippets=) | Returns `helper` in all files except files with a `.yml` or `.js` extension. | +| [<code>helper path:lib/git</code>](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=helper+path%3Alib%2Fgit) | Returns `helper` in all files with a `lib/git*` path (for example, `spec/lib/gitlab`). | + <!-- markdownlint-enable --> + +## Known issues + +- You can only search files smaller than 1 MB. + For self-managed GitLab instances, an administrator can + [change this limit](../../integration/advanced_search/elasticsearch.md#advanced-search-configuration). +- You can only use advanced search on the default branch of a project. +- The search query must not contain any of the following characters: + + ```plaintext + . , : ; / ` ' = ? $ & ^ | < > ( ) { } [ ] @ + ``` + +- Search results show only the first match in a file. diff --git a/doc/user/search/exact_code_search.md b/doc/user/search/exact_code_search.md index 14dca6d4a12..76a298a2cee 100644 --- a/doc/user/search/exact_code_search.md +++ b/doc/user/search/exact_code_search.md @@ -20,9 +20,12 @@ When performing any Code search in GitLab it will choose to use "Exact Code Search" powered by [Zoekt](https://github.com/sourcegraph/zoekt) if the project is part of an enabled Group. -The main differences between Zoekt and [Advanced Search](advanced_search.md) +The main differences between Zoekt and [advanced search](advanced_search.md) are that Zoekt provides exact substring matching as well as allows you to search for regular expressions. Since it allows searching for regular expressions, certain special characters will require escaping. Backslash can escape special characters and wrapping in double quotes can be used for phrase searches. + +To understand the possible filtering options, see the +[Zoekt query syntax](https://github.com/sourcegraph/zoekt/blob/main/doc/query_syntax.md). diff --git a/doc/user/search/global_search/advanced_search_syntax.md b/doc/user/search/global_search/advanced_search_syntax.md deleted file mode 100644 index c34c5c0c8fc..00000000000 --- a/doc/user/search/global_search/advanced_search_syntax.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../advanced_search.md' -remove_date: '2023-02-02' ---- - -This document was moved to [another location](../advanced_search.md). - -<!-- This redirect file can be deleted after <2023-02-02>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/search/index.md b/doc/user/search/index.md index dc07b4c9215..310d733800a 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -50,7 +50,7 @@ Global search only flags with an error any search that includes more than: ## Perform a search -To start a search, type your search query in the search bar in the upper right of the screen. +To start a search, in the upper-right corner of the screen, in the search bar, type your search query. You must type at least two characters.  @@ -82,7 +82,16 @@ where the results were found. 1. From the code search result, hover over the line number. 1. On the left, select **View blame**. -  + + +### Filter code search results by language + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342651) in GitLab 15.10. + +To filter code search results by one or more languages: + +1. On the code search page, on the left sidebar, select one or more languages. +1. On the left sidebar, select **Apply**. ## Search for projects by full path diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index d67c595512a..9c2925ec647 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -15,7 +15,7 @@ To display a window in GitLab that lists its keyboard shortcuts, use one of the following methods: - Press <kbd>?</kbd>. -- In the Help menu in the upper right of the application, select **Keyboard shortcuts**. +- In the Help menu, in the upper-right corner of the application, select **Keyboard shortcuts**. Although [global shortcuts](#global-shortcuts) work from any area of GitLab, you must be in specific pages for the other shortcuts to be available, as diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 0532ed27010..7ca897288e1 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -67,10 +67,10 @@ In GitLab versions 13.0 and later, snippets are [versioned by default](#versione To discover all snippets visible to you in GitLab, you can: - **View all snippets visible to you**: On the top bar of your GitLab - instance, select **Main menu > Snippets** to view your snippets dashboard. + instance, select **Main menu > Your work** and then **Snippets** to view your snippets dashboard. - **Visit [GitLab snippets](https://gitlab.com/dashboard/snippets)** for your snippets on GitLab.com. - **Explore all public snippets**: On the top bar of your GitLab - instance, select **Main menu > Snippets** and select **Explore snippets** to view + instance, select **Main menu > Explore** and select **Snippets** to view [all public snippets](https://gitlab.com/explore/snippets). - **View a project's snippets**: In your project, go to **Snippets**. diff --git a/doc/user/ssh.md b/doc/user/ssh.md index d44e6a0e071..7b6fa66b1d7 100644 --- a/doc/user/ssh.md +++ b/doc/user/ssh.md @@ -208,7 +208,7 @@ the following command: ssh-keygen -o -t rsa -b 4096 -C "<comment>" ``` -## Generate an SSH key pair for a FIDO/U2F hardware security key +## Generate an SSH key pair for a FIDO2 hardware security key To generate ED25519_SK or ECDSA_SK SSH keys, you must use OpenSSH 8.2 or later: @@ -548,7 +548,7 @@ If you receive this error, restart your terminal and try the command again. ### `Key enrollment failed: invalid format` error -You may receive the following error when [generating an SSH key pair for a FIDO/U2F hardware security key](#generate-an-ssh-key-pair-for-a-fidou2f-hardware-security-key): +You may receive the following error when [generating an SSH key pair for a FIDO2 hardware security key](#generate-an-ssh-key-pair-for-a-fido2-hardware-security-key): ```shell Key enrollment failed: invalid format @@ -557,7 +557,7 @@ Key enrollment failed: invalid format You can troubleshoot this by trying the following: - Run the `ssh-keygen` command using `sudo`. -- Verify your IDO/U2F hardware security key supports +- Verify your FIDO2 hardware security key supports the key type provided. - Verify the version of OpenSSH is 8.2 or greater by running `ssh -V`. diff --git a/doc/user/tasks.md b/doc/user/tasks.md index 0fc4c7571ab..eb0184da929 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -289,3 +289,23 @@ To add a task to an iteration: The task window opens. 1. Next to **Iteration**, select **Add to iteration**. 1. From the dropdown list, select the iteration to be associated with the task. + +## View task system notes + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `work_items_mvc_2`. Disabled by default. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) to feature flag named `work_items_mvc` in GitLab 15.8. Disabled by default. +> - Changing activity sort order [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378949) in GitLab 15.8. +> - Filtering activity [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/389971) in GitLab 15.10. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 15.10. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +You can view all the system notes related to the task. By default they are sorted by **Oldest first**. +You can always change the sorting order to **Newest first**, which is remembered across sessions. +You can also filter activity by **Comments only** and **History only** in addition to the default **All activity** which is remembered across sessions. + +## Comments and threads + +You can add [comments](discussions/index.md) and reply to threads in tasks. diff --git a/doc/user/todos.md b/doc/user/todos.md index 50b60dc5b4b..4951ba024af 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -21,7 +21,7 @@ You can use the To-Do List to track [actions](#actions-that-create-to-do-items) To access your To-Do List: -On the top bar, in the upper right, select To-Do List (**{task-done}**). +On the top bar, in the upper-right corner, select the To-Do List (**{task-done}**). ## Search the To-Do List @@ -158,7 +158,7 @@ There are two ways to do this: You can mark all your to-do items as done at the same time. -In the To-Do List, in the upper right, select **Mark all as done**. +In the To-Do List, in the upper-right corner, select **Mark all as done**. ## How a user's To-Do List is affected when their access changes diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index e4e1edd6346..ca404702d72 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -1,42 +1,11 @@ --- -stage: Manage -group: Organization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../organization/index.md' +remove_date: '2023-06-13' --- -# Organization +This document was moved to [another location](../organization/index.md). -DISCLAIMER: -This page contains information related to upcoming products, features, and functionality. -It is important to note that the information presented is for informational purposes only. -Please do not rely on this information for purchasing or planning purposes. -As with all projects, the items mentioned on this page are subject to change or delay. -The development, release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. - -NOTE: -Organization is in development. - -Organization will be above the [top-level namespaces](../namespace/index.md) for you to manage -everything you do as a GitLab administrator, including: - -- Defining and applying settings to all of your groups, subgroups, and projects. -- Aggregating data from all your groups, subgroups, and projects. - -Our goal is to reach feature parity between SaaS and self-managed installations, with all -[Admin Area settings](/ee/user/admin_area/settings/index.md) moving to either: - -- Groups. Available in the Organization, and subgroups. -- Hardware Controls. For functionality that does not apply to groups, Hardware Controls are only - applicable to self-managed installations. There is one Hardware Controls section per installation. - -For more information about the state of organization development, -see [epic 9265](https://gitlab.com/groups/gitlab-org/-/epics/9265). - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a video introduction to the new hierarchy concept for groups and projects for epics, see -[Consolidating groups and projects update (August 2021)](https://www.youtube.com/watch?v=fE74lsG_8yM). - -## Related topics - -- [Organization developer documentation](../../development/workspace/index.md) +<!-- This redirect file can be deleted after <2023-06-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> |
