diff options
Diffstat (limited to 'doc/user/project/repository')
32 files changed, 1899 insertions, 1362 deletions
diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index e123debb724..fdc822ca49f 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -42,7 +42,7 @@ Prerequisites: To update the default branch for an individual [project](../../index.md): -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Branch defaults**. For **Default branch**, select a new default branch. 1. Optional. Select the **Auto-close referenced issues on default branch** checkbox to close @@ -68,7 +68,7 @@ GitLab [administrators](../../../permissions.md) of self-managed instances can customize the initial branch for projects hosted on that instance. Individual groups and subgroups can override this instance-wide setting for their projects. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand **Default branch**. @@ -85,7 +85,7 @@ overrides it. Users with the Owner role of groups and subgroups can configure the default branch name for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Repository**. 1. Expand **Default branch**. 1. For **Initial default branch name**, select a new default branch. @@ -128,7 +128,7 @@ you must either: Administrators of self-managed instances can customize the initial default branch protection for projects hosted on that instance. Individual groups and subgroups can override this instance-wide setting for their projects. -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand **Default branch**. @@ -146,7 +146,7 @@ can be overridden on a per-group basis by the group's owner. In [GitLab Premium or Ultimate](https://about.gitlab.com/pricing/), GitLab administrators can disable this privilege for group owners, enforcing the instance-level protection rule: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > Repository**. 1. Expand the **Default branch** section. @@ -167,7 +167,7 @@ can be overridden on a per-group basis by the group's owner. In [enforce protection of initial default branches](#prevent-overrides-of-default-branch-protection) which locks this setting for group owners. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > Repository**. 1. Expand **Default branch**. 1. Select [**Initial default branch protection**](#protect-initial-default-branches). diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index f7445ffe950..f33d443cd7f 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -11,7 +11,7 @@ Branches are versions of a project's working tree. When you create a new cannot be deleted) for your repository. Default branch settings can be configured at the project, subgroup, group, or instance level. -As your project grows, your team [creates](../web_editor.md#create-a-branch) more +As your project grows, your team creates more branches, preferably by following [branch naming patterns](#prefix-branch-names-with-issue-numbers). Each branch represents a set of changes, which allows development work to be done in parallel. Development work in one branch does not affect another branch. @@ -27,9 +27,13 @@ Branches are the foundation of development in a project: ## Create branch +Prerequisites: + +- You must have at least the Developer role for the project. + To create a new branch from the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. On the top right, select **New branch**. 1. Enter a **Branch name**. @@ -44,15 +48,15 @@ you can add one. Prerequisites: -- You must have at least the Developer role in the project. -- Unless you have the Maintainer or Owner roles, the +- You must have at least the Developer role for the project. +- If you don't have the Maintainer or Owner role, the [default branch protection](../../../group/manage.md#change-the-default-branch-protection-of-a-group) must be set to `Partially protected` or `Not protected` for you to push a commit to the default branch. To add a [default branch](default.md) to an empty project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Scroll to **The repository for this project is empty** and select the type of file you want to add. 1. In the Web IDE, make any desired changes to this file, then select **Create commit**. @@ -64,7 +68,7 @@ GitLab creates a default branch and adds your file to it. Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. When viewing an issue, you can create an associated branch directly from that page. Branches created this way use the @@ -73,11 +77,11 @@ including variables. Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. To create a branch from an issue: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Plan > Issues** and find your issue. 1. Below the issue description, find the **Create merge request** dropdown list, and select **{chevron-down}** to display the dropdown list. @@ -89,7 +93,7 @@ To create a branch from an issue: ## Manage and protect branches -GitLab provides you multiple methods to protect individual branches. These methods +GitLab provides multiple methods to protect individual branches. These methods ensure your branches receive oversight and quality checks from their creation to their deletion: - The [default branch](default.md) in your project receives extra protection. @@ -104,19 +108,19 @@ ensure your branches receive oversight and quality checks from their creation to You can manage your branches: - With the GitLab user interface. -- With the [command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). +- With Git on the [command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). - With the [Branches API](../../../../api/branches.md). ### View all branches To view and manage your branches in the GitLab user interface: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. On this page, you can: -- See all branches, active branches, or stale branches. +- See all branches, or filter to see only active or stale branches. - Create new branches. - [Compare branches](#compare-branches). - Delete merged branches. @@ -145,19 +149,19 @@ and their protection methods: Prerequisites: -- You must have at least the Maintainer role in the project. +- You must have at least the Maintainer role for the project. To view the **Branch rules overview** list: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. -1. Expand **Branch Rules** to view all branches with protections. +1. Expand **Branch rules** to view all branches with protections. - To add protections to a new branch: 1. Select **Add branch rule**. 1. Select **Create protected branch**. - To view more information about protections on an existing branch: 1. Identify the branch you want more information about. - 1. Select **Details** to see information about its: + 1. Select **View details** to see information about its: - [Branch protections](../../protected_branches.md). - [Approval rules](../../merge_requests/approvals/rules.md). - [Status checks](../../merge_requests/status_checks.md). @@ -173,12 +177,17 @@ GitLab enforces these additional rules on all branches: - No spaces are allowed in branch names. - Branch names with 40 hexadecimal characters are prohibited, because they are similar to Git commit hashes. -Common software packages, like Docker, may enforce +Common software packages, like Docker, can enforce [additional branch naming restrictions](../../../../administration/packages/container_registry.md#docker-connection-error). -For best compatibility with other software packages, use only numbers, hyphens (`-`), -underscores (`_`), and lower-case letters from the ASCII standard table. You -can use forward slashes (`/`) and emoji in branch names, but compatibility with other +For the best compatibility with other software packages, use only: + +- Numbers +- Hyphens (`-`) +- Underscores (`_`) +- Lowercase letters from the ASCII standard table + +You can use forward slashes (`/`) and emoji in branch names, but compatibility with other software packages cannot be guaranteed. Branch names with specific formatting offer extra benefits: @@ -200,7 +209,7 @@ Prerequisites: To change the default pattern for branches created from issues: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Branch defaults**. 1. Scroll to **Branch name template** and enter a value. The field supports these variables: @@ -210,8 +219,13 @@ To change the default pattern for branches created from issues: ### Prefix branch names with issue numbers -To streamline the creation of merge requests, start your branch name with an -issue number. GitLab uses the issue number to import data into the merge request: +To streamline the creation of merge requests, start your Git branch name with the +issue number, followed by a hyphen. +For example, to link a branch to issue `#123`, start the branch name with `123-`. + +The issue and the branch must be in the same project. + +GitLab uses the issue number to import data into the merge request: - The issue is marked as related to the merge request. The issue and merge request display links to each other. @@ -224,21 +238,9 @@ issue number. GitLab uses the issue number to import data into the merge request ## Compare branches -> - Repository filter search box [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52967) in GitLab 13.10. -> - Revision swapping [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60491) in GitLab 13.12. - -The default compare mode uses the `git diff from...to` method, instead of the -`git diff from to` method, to compare branches. The `git diff from...to` method -provides a more human-readable diff, because it does not include unrelated changes -made to the target branch after the source branch was created. - -NOTE: -The `git diff from...to` method shows all changes from a cherry-picked commit as -new changes. It uses the merge base, not the actual commit content, to compare branches. - To compare branches in a repository: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Compare revisions**. 1. Select the **Source** branch to search for your desired branch. Exact matches are shown first. You can refine your search with operators: @@ -247,8 +249,23 @@ To compare branches in a repository: - `*` matches using a wildcard: `branch*cache*` matches `fix/branch-search-cache-expiration`. - You can combine operators: `^chore/*migration$` matches `chore/user-data-migration`. 1. Select the **Target** repository and branch. Exact matches are shown first. -1. Select **Compare** to show the list of commits, and changed files. To reverse - the **Source** and **Target**, select **Swap revisions**. +1. Below **Show changes**, select the method to compare branches: + <!-- vale gitlab.SubstitutionWarning = NO --> + <!-- Disable Vale so it doesn't flag "since" --> + - **Only incoming changes from source** (default) shows differences from the source branch since + the latest common commit on both branches. + It doesn't include unrelated changes made to the target branch after the source branch was created. + This method uses the `git diff <from>...<to>` + [Git command](https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-emgitdiffemltoptionsgtltcommitgtltcommitgt--ltpathgt82308203-1). + To compare branches, this method uses the merge base instead of the actual commit, so + changes from cherry-picked commits are shown as new changes. + - **Include changes to target since source was created** shows all the differences between the two + branches. + This method uses the `git diff <from> <to>` + [Git command](https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-emgitdiffemltoptionsgt--merge-baseltcommitgtltcommitgt--ltpathgt82308203). + <!-- vale gitlab.SubstitutionWarning = YES --> +1. Select **Compare** to show the list of commits, and changed files. +1. Optional. To reverse the **Source** and **Target**, select **Swap revisions** (**{substitute}**). ## Delete merged branches @@ -259,15 +276,15 @@ Merged branches can be deleted in bulk if they meet all of these criteria: Prerequisites: -- You must have at least the Developer role in the project. +- You must have at least the Developer role for the project. To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. On the upper right corner of the page, select **More** **{ellipsis_v}**. 1. Select **Delete merged branches**. -1. In the modal window, enter the word `delete` to confirm, then select **Delete merged branches**. +1. In the dialog, enter the word `delete` to confirm, then select **Delete merged branches**. ## Related topics @@ -323,7 +340,7 @@ Error: Could not set the default branch. Do you have a branch named 'HEAD' in yo To fix this problem: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Branches**. 1. Search for a branch named `HEAD`. 1. Make sure the branch has no uncommitted changes. diff --git a/doc/user/project/repository/code_suggestions.md b/doc/user/project/repository/code_suggestions.md deleted file mode 100644 index 6f18aabaa46..00000000000 --- a/doc/user/project/repository/code_suggestions.md +++ /dev/null @@ -1,375 +0,0 @@ ---- -stage: AI-powered -group: AI Model Validation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: index, reference ---- - -# Code Suggestions (Beta) **(FREE ALL)** - -> - [Introduced](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#code-suggestions-available-in-closed-beta) in GitLab 15.9 as [Beta](../../../policy/experiment-beta-support.md#beta) for early access Ultimate customers on GitLab.com. -> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/408104) as opt-in with GitLab 15.11 as [Beta](../../../policy/experiment-beta-support.md#beta). -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1. -> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta) on self-managed GitLab. -> - Code suggestions in the GitLab WebIDE enabled for all GitLab-hosted customers. -> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. - -WARNING: -This feature is in [Beta](../../../policy/experiment-beta-support.md#beta). -Beta users should read about the [known limitations](#known-limitations). We look forward to hearing your [feedback](#feedback). - -Write code more efficiently by using generative AI to suggest code while you're developing. - -Code Suggestions are available: - -- To users of GitLab SaaS (by default) and self-managed GitLab Enterprise Edition (when requested). Code Suggestions are not available for GitLab Community Edition. -- In VS Code, Microsoft Visual Studio, JetBrains IDEs, and Neovim. You must have the corresponding GitLab extension installed. -- In the GitLab WebIDE. - -<div class="video-fallback"> - <a href="https://www.youtube.com/watch?v=WnxBYxN2-p4">View an end-to-end demo of Code Suggestions in VS Code</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/WnxBYxN2-p4" frameborder="0" allowfullscreen> </iframe> -</figure> - -Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). -Learn about [data usage when using Code Suggestions](#code-suggestions-data-usage). - -## Supported languages - -The best results from Code Suggestions are expected [for languages the Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_coding_languages) directly support: - -- C++ -- C# -- Go -- Google SQL -- Java -- JavaScript -- Kotlin -- PHP -- Python -- Ruby -- Rust -- Scala -- Swift -- TypeScript - -### Supported code infrastructure interfaces - -Code Suggestions includes [Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_code_infrastructure_interfaces) support for the following infrastructure as code interfaces: - -- Google Cloud CLI -- Kubernetes Resource Model (KRM) -- Terraform - -Suggestion quality for other languages and using natural language code comments to request completions may not yet result in high-quality suggestions. - -### Supported languages in IDEs - -Editor support for languages is documented in the following table. - -| Language | VS Code | JetBrains IDEs | Visual Studio | Neovim | -|---------------------------------|--------------------------------------------------------------|------------------------------|---------------|--------| -| C++ | ✓ | | ✓ | | -| C# | ✓ | ✓ | ✓ | | -| Go | ✓ | ✓ (IDEA Ultimate / GoLand) | | | -| Google SQL | | | | | -| Java | ✓ | ✓ | | | -| JavaScript | ✓ | ✓ | | | -| Kotlin | ✓ | ✓ | | | -| PHP | ✓ | ✓ (IDEA Ultimate) | | | -| Python | ✓ | ✓ | | ✓ | -| Ruby | ✓ | ✓ (IDEA Ultimate / RubyMine) | | ✓ | -| Rust | ✓ | ✓ | | | -| Scala | ✓ | ✓ | | | -| Swift | ✓ | ✓ | | | -| TypeScript | ✓ | ✓ | | | -| Google Cloud CLI | | | | | -| Kubernetes Resource Model (KRM) | | | | | -| Terraform | ✓ (Requires 3rd party extension providing Terraform support) | | | | - -## Supported editor extensions - -Code Suggestions supports a variety of popular editors including: - -- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). -- [GitLab WebIDE (VS Code in the Cloud)](../../project/web_ide/index.md), with no additional configuration. -- Microsoft Visual Studio, using the [Visual Studio GitLab extension](https://marketplace.visualstudio.com/items?itemName=GitLab.GitLabExtensionForVisualStudio). -- JetBrains IDEs, using the [GitLab plugin](https://plugins.jetbrains.com/plugin/22325-gitlab). -- Neovim, using the [`gitlab.vim` plugin](https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim). - -A [GitLab Language Server for Code Suggestions](https://gitlab.com/gitlab-org/editor-extensions/gitlab-language-server-for-code-suggestions) -is also in process. -This improvement should result in: - -- Faster iteration and standardization of the IDE extensions. -- The ability to use Code Suggestions even when an official editor extension isn't available. - -## Enable Code Suggestions on GitLab SaaS **(FREE SAAS)** - -Code Suggestions can be enabled [for all members of a group](../../group/manage.md#enable-code-suggestions). - -Each individual user must also choose to enable Code Suggestions. - -### Enable Code Suggestions for an individual user - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta). - -Each user can enable Code Suggestions for themselves: - -1. On the left sidebar, select your avatar. -1. Select **Preferences**. -1. In the **Code Suggestions** section, select the **Enable Code Suggestions** checkbox. -1. Select **Save changes**. - -If Code Suggestions is enabled for the group, the group setting overrides the user setting. - -NOTE: -This setting controls Code Suggestions for all IDEs. Support for [more granular control per IDE](https://gitlab.com/groups/gitlab-org/-/epics/10624) is proposed. - -## Enable Code Suggestions on self-managed GitLab **(FREE SELF)** - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../policy/experiment-beta-support.md#beta). - -To enable Code Suggestions on a self-managed GitLab EE instance, you must: - -- Be an administrator. -- Have a [GitLab SaaS account](https://gitlab.com/users/sign_up). -You do not need to have a GitLab SaaS subscription. - -Then, you will: - -1. Enable Code Suggestions for your SaaS account. -1. Enable Code Suggestions for the instance. -1. [Request early access](#request-access-to-code-suggestions) to the Code Suggestions Beta. - -### Enable Code Suggestions for your SaaS account - -To enable Code Suggestions for your GitLab SaaS account: - -1. Create a [personal access token](../../profile/personal_access_tokens.md#create-a-personal-access-token) - with the `api` scope. -1. On the left sidebar, select your avatar. -1. Select **Preferences**. -1. In the **Code Suggestions** section, select **Enable Code Suggestions**. -1. Select **Save changes**. - -### Enable Code Suggestions for the instance - -You must enable Code Suggestions for the instance. When you do this, you: - -- Agree to the [GitLab testing agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). -- Acknowledge that GitLab: - - Sends data from the instance, including personal data, to GitLab.com infrastructure. - -To enable Code Suggestions for your self-managed GitLab instance: - -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **Admin Area**. -1. On the left sidebar, select **Settings > General**. -1. Expand **Code Suggestions** and: - - Select **Turn on Code Suggestions for this instance**. - - In **Personal access token**, enter your GitLab SaaS personal access token. -1. Select **Save changes**. - -This setting is visible only in self-managed GitLab instances. - -WARNING: -If you clear the **Turn on code suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. - -### Request access to Code Suggestions - -GitLab provisions access on a customer-by-customer basis for Code Suggestions -on self-managed instances. To request access: - -1. Sign into your GitLab SaaS account. -1. Comment on [issue 415393](https://gitlab.com/gitlab-org/gitlab/-/issues/415393) - and tag your customer success manager. - -After GitLab has provisioned access to Code Suggestions for your instance, -the users in your instance can now enable Code Suggestions. - -## Use Code Suggestions - -Prerequisites: - -- For self-managed GitLab, Code Suggestions must be enabled [for the instance](#enable-code-suggestions-on-self-managed-gitlab). -- For GitLab SaaS, Code Suggestions must be enabled [for the top-level group](../../group/manage.md#enable-code-suggestions) and [for your user account](#enable-code-suggestions-for-an-individual-user). -- Install and configure a [supported IDE editor extension](#supported-editor-extensions). -To use Code Suggestions: - -1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: - - - Provides entire code snippets, like generating functions. - - Completes the current line. - -1. To accept a suggestion, press <kbd>Tab</kbd>. - -Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. - -GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. - -This feature is currently in [Beta](../../../policy/experiment-beta-support.md#beta). -Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to -mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. - -## Code Suggestions data usage - -Code Suggestions is a generative artificial intelligence (AI) model. - -Your personal access token enables a secure API connection to GitLab.com. -This API connection securely transmits a context window from your IDE/editor to the Code Suggestions GitLab hosted service which calls Google Vertex AI Codey APIs, -and the generated suggestion is transmitted back to your IDE/editor. - -GitLab currently leverages [Google Cloud's Vertex AI Codey API models](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview). - -### Data privacy - -No new additional data is collected to enable this feature. Private non-public GitLab customer data is -not used as training data. - -Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) - -### Inference window context - -Code Suggestions currently inferences against the currently opened file and has a context window of 2,048 tokens and 8,192 character limits. This limit includes content before and after the cursor, the file name, and the extension type. -Learn more about Google Vertex AI [code-gecko](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). - -The maximum number of tokens that is generated in the response is default 64. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words. -Learn more about Google Vertex AI [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion). - -### Self-managed instance data privacy - -A self-managed GitLab instance does not generate the code suggestion. After successful -authentication to the self-managed instance, a token is generated. - -The IDE/editor then uses this token to securely transmit data directly to -GitLab.com's Code Suggestions service for processing. - -The Code Suggestion service then securely returns an AI-generated code suggestion. - -Neither GitLab nor Google Vertex AI Codey APIs have any visibility into a self-managed customer's code other than -what is sent to generate the code suggestion. - -### Training data - -Code suggestions are routed through Google Vertex AI Codey APIs. Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) and [Responsible AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/responsible-ai). - -Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data. - -Google has [shared the following](https://ai.google/discover/foundation-models/) about the data Codey models are trained on: - -> Codey is our family of foundational coding models built on PaLM 2. Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources - -## Progressive enhancement - -This feature is designed as a progressive enhancement to developer's IDEs. -Code Suggestions offer a completion if the machine learning engine can generate a recommendation. -In the event of a connection issue or model inference failure, the feature gracefully degrades. -Code Suggestions do not prevent you from writing code in your IDE. - -### Internet connectivity - -Code Suggestions does not work with offline environments. - -To use Code Suggestions: - -- On GitLab.com, you must have an internet connection and be able to access GitLab. -- In GitLab 16.1 and later, on self-managed GitLab, you must have an internet connection. - -### Model accuracy and quality - -Code Suggestions can generate low-quality, incomplete, and possibly insecure code. -We strongly encourage all beta users to leverage GitLab native -[Code Quality Scanning](../../../ci/testing/code_quality.md) and -[Security Scanning](../../application_security/index.md) capabilities. - -GitLab currently does not retrain Google Vertex AI Codey APIs. GitLab makes no claims -to the accuracy or quality of code suggestions generated by Google Vertex AI Codey API. -Read more about [Google Vertex AI foundation model capabilities](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). - -## Known limitations - -While in Beta, we are working on improving the accuracy of overall generated content. -However, Code Suggestions may generate suggestions that are: - -- Low-quality -- Incomplete -- Produce failed pipelines -- Insecure code -- Offensive or insensitive - -We are also aware of specific situations that can produce unexpected or incoherent results including: - -- Suggestions written in the middle of existing functions, or "fill in the middle." -- Suggestions based on natural language code comments. -- Suggestions that mixed programming languages in unexpected ways. - -## Feedback - -Report issues in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152). - -## Troubleshooting - -### Code Suggestions aren't displayed - -If Code Suggestions are not displayed, try the following troubleshooting steps. - -In GitLab, ensure Code Suggestions is enabled: - -- [For your user account](#enable-code-suggestions-for-an-individual-user). -- [For *all* top-level groups your account belongs to](../../group/manage.md#enable-code-suggestions). If you don't have a role that lets you view the top-level group's settings, contact a group owner. - -To confirm that your account is enabled, go to [https://gitlab.com/api/v4/ml/ai-assist](https://gitlab.com/api/v4/ml/ai-assist). A response of `user_is_allowed` should return `true`. - -#### Code Suggestions not displayed in VS Code or GitLab WebIDE - -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. - -If you are a self-managed user, ensure that Code Suggestions for the [GitLab WebIDE](../../project/web_ide/index.md) are enabled. The same settings apply to VS Code as local IDE. - -1. On the left sidebar, select **Extensions > GitLab Workflow**. -1. Select **Settings** (**{settings}**), and then select **Extension Settings**. -1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion (Beta)** - checkbox. - -If the settings are enabled, but Code Suggestions are still not displayed, try the following steps: - -1. Enable the `Debug` checkbox in the GitLab Workflow **Extension Settings**. -1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Workflow** as the log filter. The command palette command is `GitLab: Show Extension Logs`. -1. Disable and re-enable the **Enable code completion (Beta)** checkbox. -1. Verify that the debug log contains similar output: - -```shell -2023-07-14T17:29:00:763 [debug]: Disabling code completion -2023-07-14T17:29:01:802 [debug]: Enabling code completion -2023-07-14T17:29:01:802 [debug]: AI Assist: Using server: https://codesuggestions.gitlab.com/v2/completions -``` - -#### Code Suggestions not displayed in Microsoft Visual Studio - -Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. - -1. Ensure you have properly [set up the extension](https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension#setup). -1. From the **Tools > Options** menu, find the **GitLab** option. Ensure **Log Level** is set to **Debug**. -1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Extension** as the log filter. -1. Verify that the debug log contains similar output: - -```shell -14:48:21:344 GitlabProposalSource.GetCodeSuggestionAsync -14:48:21:344 LsClient.SendTextDocumentCompletionAsync("GitLab.Extension.Test\TestData.cs", 34, 0) -14:48:21:346 LS(55096): time="2023-07-17T14:48:21-05:00" level=info msg="update context" -``` - -### Authentication troubleshooting - -If the above steps do not solve your issue, the problem may be related to the recent changes in authentication, -specifically the token system. To resolve the issue: - -1. Remove the existing personal access token from your GitLab account settings. -1. Reauthorize your GitLab account in VS Code using OAuth. -1. Test the code suggestions feature with different file extensions to verify if the issue is resolved. diff --git a/doc/user/project/repository/code_suggestions/index.md b/doc/user/project/repository/code_suggestions/index.md new file mode 100644 index 00000000000..4f0c0b2c9a6 --- /dev/null +++ b/doc/user/project/repository/code_suggestions/index.md @@ -0,0 +1,192 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions **(FREE ALL BETA)** + +> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. +> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. + +WARNING: +This feature is in [Beta](../../../../policy/experiment-beta-support.md#beta). +Beta users should read about the [known limitations](#known-limitations). We look forward to hearing your [feedback](#feedback). + +Write code more efficiently by using generative AI to suggest code while you're developing. + +GitLab Duo Code Suggestions are available: + +- On [self-managed](self_managed.md) and [SaaS](saas.md). +- In VS Code, Microsoft Visual Studio, JetBrains IDEs, and Neovim. You must have the corresponding GitLab extension installed. +- In the GitLab WebIDE. + +<div class="video-fallback"> + <a href="https://www.youtube.com/watch?v=WnxBYxN2-p4">View an end-to-end demo of Code Suggestions in VS Code</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube-nocookie.com/embed/WnxBYxN2-p4" frameborder="0" allowfullscreen> </iframe> +</figure> + +Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](#code-suggestions-data-usage). + +## Supported languages + +The best results from Code Suggestions are expected [for languages the Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_coding_languages) directly support: + +- C++ +- C# +- Go +- Google SQL +- Java +- JavaScript +- Kotlin +- PHP +- Python +- Ruby +- Rust +- Scala +- Swift +- TypeScript + +### Supported code infrastructure interfaces + +Code Suggestions includes [Google Vertex AI Codey APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview#supported_code_infrastructure_interfaces) support for the following infrastructure as code interfaces: + +- Google Cloud CLI +- Kubernetes Resource Model (KRM) +- Terraform + +Suggestion quality for other languages and using natural language code comments to request completions may not yet result in high-quality suggestions. + +### Supported languages in IDEs + +Editor support for languages is documented in the following table. + +| Language | VS Code | JetBrains IDEs | Visual Studio | Neovim | +|------------------|------------------------|------------------------|------------------------|--------| +| C++ | **{check-circle}** Yes | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | +| C# | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Go | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate / GoLand) | **{check-circle}** Yes | **{check-circle}** Yes | +| Google SQL | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | **{check-circle}** Yes | +| Java | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| JavaScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Kotlin | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| PHP | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate) | **{check-circle}** Yes | **{check-circle}** Yes | +| Python | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Ruby | **{check-circle}** Yes | **{check-circle}** Yes (IDEA Ultimate / RubyMine) | **{check-circle}** Yes | **{check-circle}** Yes | +| Rust | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Scala | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Swift | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| TypeScript | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | +| Google Cloud | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | +| Kubernetes Resource Model (KRM) | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | **{dotted-circle}** No | +| Terraform | **{check-circle}** Yes (Requires third-party extension providing Terraform support) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes (Requires third-party extension providing the `terraform` file type) | + +## Supported editor extensions + +Code Suggestions supports a variety of popular editors including: + +- VS Code, using [the VS Code GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). +- [GitLab WebIDE (VS Code in the Cloud)](../../../project/web_ide/index.md), with no additional configuration. +- Microsoft Visual Studio, using the [Visual Studio GitLab extension](https://marketplace.visualstudio.com/items?itemName=GitLab.GitLabExtensionForVisualStudio). +- JetBrains IDEs, using the [GitLab plugin](https://plugins.jetbrains.com/plugin/22325-gitlab). +- Neovim, using the [`gitlab.vim` plugin](https://gitlab.com/gitlab-org/editor-extensions/gitlab.vim). + +A [GitLab Language Server for Code Suggestions](https://gitlab.com/gitlab-org/editor-extensions/gitlab-language-server-for-code-suggestions) +is also in process. +This improvement should result in: + +- Faster iteration and standardization of the IDE extensions. +- The ability to use Code Suggestions even when an official editor extension isn't available. + +## Code Suggestions data usage + +Code Suggestions is a generative artificial intelligence (AI) model. + +Your personal access token enables a secure API connection to GitLab.com. +This API connection securely transmits a context window from your IDE/editor to the Code Suggestions GitLab hosted service which calls Google Vertex AI Codey APIs, +and the generated suggestion is transmitted back to your IDE/editor. + +GitLab currently leverages [Google Cloud's Vertex AI Codey API models](https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview). Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance). + +### Telemetry + +For self-managed instances that have enabled Code Suggestions and SaaS accounts, we collect aggregated or de-identified first-party usage data through our [Snowplow collector](https://about.gitlab.com/handbook/business-technology/data-team/platform/snowplow/). This usage data includes the following metrics: + +- Language the code suggestion was in (for example, Python) +- Editor being used (for example, VS Code) +- Number of suggestions shown, accepted, rejected, or that had errors +- Duration of time that a suggestion was shown +- Prompt and suffix lengths +- Model used +- Number of unique users +- Number of unique instances + +### Inference window context + +Code Suggestions currently inferences against the currently opened file and has a context window of 2,048 tokens and 8,192 character limits. This limit includes content before and after the cursor, the file name, and the extension type. +Learn more about Google Vertex AI [code-gecko](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). + +The maximum number of tokens that is generated in the response is default 64. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words. +Learn more about Google Vertex AI [`code-gecko`](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/code-completion). + +### Training data + +Code Suggestions are routed through Google Vertex AI Codey APIs. Learn more about Google Vertex AI Codey APIs [Data Governance](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance) and [Responsible AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/responsible-ai). + +Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data. + +Google has [shared the following](https://ai.google/discover/foundation-models/) about the data Codey models are trained on: + +> Codey is our family of foundational coding models built on PaLM 2. Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources + +## Progressive enhancement + +This feature is designed as a progressive enhancement to developer's IDEs. +Code Suggestions offer a completion if the machine learning engine can generate a recommendation. +In the event of a connection issue or model inference failure, the feature gracefully degrades. +Code Suggestions do not prevent you from writing code in your IDE. + +### Internet connectivity + +Code Suggestions does not work with offline environments. + +To use Code Suggestions: + +- On GitLab.com, you must have an internet connection and be able to access GitLab. +- In GitLab 16.1 and later, on self-managed GitLab, you must have an internet connection. + +### Model accuracy and quality + +Code Suggestions can generate low-quality, incomplete, and possibly insecure code. +We strongly encourage all beta users to leverage GitLab native +[Code Quality Scanning](../../../../ci/testing/code_quality.md) and +[Security Scanning](../../../application_security/index.md) capabilities. + +GitLab currently does not retrain Google Vertex AI Codey APIs. GitLab makes no claims +to the accuracy or quality of Code Suggestions generated by Google Vertex AI Codey API. +Read more about [Google Vertex AI foundation model capabilities](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models). + +## Known limitations + +While in Beta, we are working on improving the accuracy of overall generated content. +However, Code Suggestions may generate suggestions that are: + +- Low-quality +- Incomplete +- Produce failed pipelines +- Insecure code +- Offensive or insensitive + +We are also aware of specific situations that can produce unexpected or incoherent results including: + +- Suggestions written in the middle of existing functions, or "fill in the middle." +- Suggestions based on natural language code comments. +- Suggestions that mixed programming languages in unexpected ways. + +## Feedback + +Report issues in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152). diff --git a/doc/user/project/repository/code_suggestions/saas.md b/doc/user/project/repository/code_suggestions/saas.md new file mode 100644 index 00000000000..174c227b6fe --- /dev/null +++ b/doc/user/project/repository/code_suggestions/saas.md @@ -0,0 +1,54 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions on GitLab SaaS **(FREE SAAS BETA)** + +> - [Introduced](https://about.gitlab.com/releases/2023/02/22/gitlab-15-9-released/#code-suggestions-available-in-closed-beta) in GitLab 15.9 as [Beta](../../../../policy/experiment-beta-support.md#beta) for early access Ultimate customers on GitLab.com. +> - [Enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/408104) as opt-in with GitLab 15.11 as [Beta](../../../../policy/experiment-beta-support.md#beta). +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/408158) from GitLab Ultimate to GitLab Premium in 16.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/410801) from GitLab Premium to GitLab Free in 16.0. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1. + +Write code more efficiently by using generative AI to suggest code while you're developing. + +Usage of GitLab Duo Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](index.md#code-suggestions-data-usage). + +## Enable Code Suggestions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121079) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). + +You can enable Code Suggestions for an individual or group: + +- [Enable Code Suggestions for all group members](../../../group/manage.md#enable-code-suggestions). (You must be a group owner). +- [Enable Code Suggestions for your own account](../../../profile/preferences.md#enable-code-suggestions). + +The group setting takes precedence over the user setting. + +## Use Code Suggestions + +Prerequisites: + +- Ensure Code Suggestions is enabled for your user and group. +- You must have installed and configured a [supported IDE editor extension](index.md#supported-editor-extensions). + +To use Code Suggestions: + +1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: + + - Provides entire code snippets, like generating functions. + - Completes the current line. + +1. To accept a suggestion, press <kbd>Tab</kbd>. + +Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. + +GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. + +This feature is currently in [Beta](../../../../policy/experiment-beta-support.md#beta). +Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to +mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. diff --git a/doc/user/project/repository/code_suggestions/self_managed.md b/doc/user/project/repository/code_suggestions/self_managed.md new file mode 100644 index 00000000000..3c149604086 --- /dev/null +++ b/doc/user/project/repository/code_suggestions/self_managed.md @@ -0,0 +1,187 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Code Suggestions on self-managed GitLab **(PREMIUM SELF BETA)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta) on self-managed GitLab. +> - [Introduced support for Google Vertex AI Codey APIs](https://gitlab.com/groups/gitlab-org/-/epics/10562) in GitLab 16.1. +> - [Removed support for GitLab native model](https://gitlab.com/groups/gitlab-org/-/epics/10752) in GitLab 16.2. +> - Code Suggestions in the GitLab WebIDE enabled for all GitLab-hosted customers. + +Write code more efficiently by using generative AI to suggest code while you're developing. + +GitLab Duo Code Suggestions are available on GitLab Enterprise Edition. +Cloud licensing is required for Premium and Ultimate subscription tiers. + +Code Suggestions are not available for GitLab Community Edition. + +WARNING: +In GitLab 16.3 and later, only Premium and Ultimate customers can participate in the free trial of Code Suggestions on self-managed GitLab. + +Usage of Code Suggestions is governed by the [GitLab Testing Agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +Learn about [data usage when using Code Suggestions](index.md#code-suggestions-data-usage). + +## Enable Code Suggestions on self-managed GitLab **(FREE SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10653) in GitLab 16.1 as [Beta](../../../../policy/experiment-beta-support.md#beta). + +When you enable Code Suggestions for your self-managed instance, you: + +- Agree to the [GitLab testing agreement](https://about.gitlab.com/handbook/legal/testing-agreement/). +- Acknowledge that GitLab sends data from the instance, including personal data, to GitLab.com infrastructure. + +How you enable Code Suggestions differs depending on your version of GitLab. + +### GitLab 16.3 and later + +Prerequisites: + +- You are a new Code Suggestions customer as of GitLab 16.3. +- All of the users in your instance have the latest version of their IDE extension. +- You are an administrator. + +To enable Code Suggestions for your self-managed GitLab instance: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Code Suggestions** and select **Turn on Code Suggestions for this instance**. + In GitLab 16.3, you do not need to enter anything into the **Personal access token** field. + In GitLab 16.4 and later, there is no **Personal access token** field. +1. Select **Save changes**. + +This setting is visible only in self-managed GitLab instances. + +WARNING: +In GitLab 16.2 and earlier, if you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. + +To make sure Code Suggestions works immediately, you must [manually synchronize your subscription](#manually-synchronize-your-subscription). + +The users in your instance can now use Code Suggestions. + +### GitLab 16.2 and earlier + +Prerequisites: + +- You are an administrator. +- You have a [customer success manager](https://about.gitlab.com/handbook/customer-success/csm/]). +- You have a [GitLab SaaS account](https://gitlab.com/users/sign_up). You do not need to have a GitLab SaaS subscription. + +NOTE: +If you do not have a customer success manager, you cannot participate in the free trial of Code Suggestions on self-managed GitLab. Upgrade to GitLab 16.3 to [perform self-service onboarding](#gitlab-163-and-later). + +Then, you will: + +1. Enable Code Suggestions for your SaaS account. +1. Enable Code Suggestions for the instance. +1. [Request early access](#request-access-to-code-suggestions) to the Code Suggestions Beta. + +#### Enable Code Suggestions for your SaaS account + +To enable Code Suggestions for your GitLab SaaS account: + +1. Create a [personal access token](../../../profile/personal_access_tokens.md#create-a-personal-access-token) + with the `api` scope. +1. On the left sidebar, select your avatar. +1. Select **Preferences**. +1. In the **Code Suggestions** section, select **Enable Code Suggestions**. +1. Select **Save changes**. + +#### Enable Code Suggestions for the instance + +To enable Code Suggestions for your self-managed GitLab instance: + +1. On the left sidebar, select **Search or go to**. +1. Select **Admin Area**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Code Suggestions** and: + - Select **Turn on Code Suggestions for this instance**. + - In **Personal access token**, enter your GitLab SaaS personal access token. +1. Select **Save changes**. + +This setting is visible only in self-managed GitLab instances. + +WARNING: +If you clear the **Turn on Code Suggestions for this instance** checkbox, the users in your instance can still use Code Suggestions for up to one hour, until the issued JSON web token (JWT) expires. + +#### Request access to Code Suggestions + +GitLab provisions access on a customer-by-customer basis for Code Suggestions +on self-managed instances. To request access, contact your customer success manager. + +Your customer success manager then provisions access by commenting on [issue 415393](https://gitlab.com/gitlab-org/gitlab/-/issues/415393) (internal access only). + +After GitLab has provisioned access to Code Suggestions for your instance, +the users in your instance can now enable Code Suggestions. + +### Configure network and proxy settings + +Configure any firewalls to allow outbound connections to `https://codesuggestions.gitlab.com/`. + +If your GitLab instance uses an HTTP proxy server to access the internet, ensure +the server is configured to allow outbound connections, including the +[`gitlab_workhorse` environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html). + +### Update GitLab + +In GitLab 16.3 and later, GitLab is enforcing the cloud licensing requirement for Code Suggestions: + +- The Premium and Ultimate subscription tiers support cloud Licensing. +- GitLab Free does not have cloud licensing support. + +If you have a GitLab Free subscription and update to GitLab 16.3 or later, +to continue having early access to Code Suggestions, you must: + +1. Have a [subscription that supports cloud licensing](https://about.gitlab.com/pricing/). +1. Make sure you have the latest version of your [IDE extension](index.md#supported-editor-extensions). +1. [Manually synchronize your subscription](#manually-synchronize-your-subscription). + +#### Manually synchronize your subscription + +You must [manually synchronize your subscription](../../../../subscriptions/self_managed/index.md#manually-synchronize-your-subscription-details) if either: + +- You have already updated to GitLab 16.3 and have just bought a Premium or Ultimate tier subscription. +- You already have a Premium or Ultimate tier subscription and have just updated to GitLab 16.3. + +Without the manual synchronization, it might take up to 24 hours to active Code Suggestions on your instance. + +## Use Code Suggestions + +Prerequisites: + +- Code Suggestions must be enabled [for the instance](#enable-code-suggestions-on-self-managed-gitlab). +- You must have installed and configured a [supported IDE editor extension](index.md#supported-editor-extensions). + +To use Code Suggestions: + +1. Author your code. As you type, suggestions are displayed. Depending on the cursor position, the extension either: + + - Provides entire code snippets, like generating functions. + - Completes the current line. + +1. To accept a suggestion, press <kbd>Tab</kbd>. + +Suggestions are best when writing new code. Editing existing functions or 'fill in the middle' of a function may not perform as expected. + +GitLab is making improvements to the Code Suggestions to improve the quality. AI is non-deterministic, so you may not get the same suggestion every time with the same input. + +This feature is currently in [Beta](../../../../policy/experiment-beta-support.md#beta). +Code Suggestions depends on both Google Vertex AI Codey APIs and the GitLab Code Suggestions service. We have built this feature to gracefully degrade and have controls in place to allow us to +mitigate abuse or misuse. GitLab may disable this feature for any or all customers at any time at our discretion. + +### Data privacy + +A self-managed GitLab instance does not generate the code suggestion. After successful +authentication to the self-managed instance, a token is generated. + +The IDE/editor then uses this token to securely transmit data directly to +GitLab.com's Code Suggestions service for processing. + +The Code Suggestions service then securely returns an AI-generated code suggestion. + +Neither GitLab nor Google Vertex AI Codey APIs have any visibility into a self-managed customer's code other than +what is sent to generate the code suggestion. diff --git a/doc/user/project/repository/code_suggestions/troubleshooting.md b/doc/user/project/repository/code_suggestions/troubleshooting.md new file mode 100644 index 00000000000..c0cdb3cc32d --- /dev/null +++ b/doc/user/project/repository/code_suggestions/troubleshooting.md @@ -0,0 +1,69 @@ +--- +stage: Create +group: Code Creation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index, reference +--- + +# Troubleshooting Code Suggestions **(FREE ALL BETA)** + +When working with GitLab Duo Code Suggestions, you might encounter the following issues. + +## Code Suggestions aren't displayed + +If Code Suggestions are not displayed, try the following troubleshooting steps. + +In GitLab, ensure Code Suggestions is enabled: + +- [For your user account](../../../profile/preferences.md#enable-code-suggestions). +- [For *all* top-level groups your account belongs to](../../../group/manage.md#enable-code-suggestions). If you don't have a role that lets you view the top-level group's settings, contact a group owner. + +To confirm that your account is enabled, go to [https://gitlab.com/api/v4/ml/ai-assist](https://gitlab.com/api/v4/ml/ai-assist). A response of `user_is_allowed` should return `true`. + +### Code Suggestions not displayed in VS Code or GitLab WebIDE + +Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. + +If you are a self-managed user, ensure that Code Suggestions for the [GitLab WebIDE](../../../project/web_ide/index.md) are enabled. The same settings apply to VS Code as local IDE. + +1. On the left sidebar, select **Extensions > GitLab Workflow**. +1. Select **Settings** (**{settings}**), and then select **Extension Settings**. +1. In **GitLab > AI Assisted Code Suggestions**, select the **Enable code completion (Beta)** + checkbox. + +If the settings are enabled, but Code Suggestions are still not displayed, try the following steps: + +1. Enable the `Debug` checkbox in the GitLab Workflow **Extension Settings**. +1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Workflow** as the log filter. The command palette command is `GitLab: Show Extension Logs`. +1. Disable and re-enable the **Enable code completion (Beta)** checkbox. +1. Verify that the debug log contains similar output: + +```shell +2023-07-14T17:29:00:763 [debug]: Disabling code completion +2023-07-14T17:29:01:802 [debug]: Enabling code completion +2023-07-14T17:29:01:802 [debug]: AI Assist: Using server: https://codesuggestions.gitlab.com/v2/completions +``` + +### Code Suggestions not displayed in Microsoft Visual Studio + +Check all the steps in [Code Suggestions aren't displayed](#code-suggestions-arent-displayed) first. + +1. Ensure you have properly [set up the extension](https://gitlab.com/gitlab-org/editor-extensions/gitlab-visual-studio-extension#setup). +1. From the **Tools > Options** menu, find the **GitLab** option. Ensure **Log Level** is set to **Debug**. +1. Open the extension log in **View > Output** and change the dropdown list to **GitLab Extension** as the log filter. +1. Verify that the debug log contains similar output: + +```shell +14:48:21:344 GitlabProposalSource.GetCodeSuggestionAsync +14:48:21:344 LsClient.SendTextDocumentCompletionAsync("GitLab.Extension.Test\TestData.cs", 34, 0) +14:48:21:346 LS(55096): time="2023-07-17T14:48:21-05:00" level=info msg="update context" +``` + +## Authentication troubleshooting + +If the above steps do not solve your issue, the problem may be related to the recent changes in authentication, +specifically the token system. To resolve the issue: + +1. Remove the existing personal access token from your GitLab account settings. +1. Reauthorize your GitLab account in VS Code using OAuth. +1. Test the Code Suggestions feature with different file extensions to verify if the issue is resolved. diff --git a/doc/user/project/repository/file_finder.md b/doc/user/project/repository/file_finder.md index ee2be6dee7c..5dd8ad39889 100644 --- a/doc/user/project/repository/file_finder.md +++ b/doc/user/project/repository/file_finder.md @@ -14,7 +14,7 @@ File finder is powered by the [`fuzzaldrin-plus`](https://github.com/jeancroy/fu To search for a file: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. In the upper right, select **Find file**. 1. In the search box, start typing the filename. diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 4c37b92b388..a304a8d108d 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -59,8 +59,8 @@ or the command line. GitLab Premium and Ultimate tiers can also automate updates To update your fork from the GitLab UI: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). -1. Select **View all your projects**. +1. On the left sidebar, select **Search or go to**. +1. Select **View all my projects**. 1. Select the fork you want to update. 1. Below the dropdown list for branch name, find the **Forked from** (**{fork}**) information box to determine if your fork is ahead, behind, or both. In this example, @@ -181,13 +181,13 @@ To restore the fork relationship, [use the API](../../../api/projects.md#create- To remove a fork relationship: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand **Advanced**. 1. In the **Remove fork relationship** section, select **Remove fork relationship**. 1. To confirm, enter the project path and select **Confirm**. -When you unlink a fork that uses a [hashed storage pool](../../../administration/repository_storage_types.md#hashed-object-pools) +When you unlink a fork that uses a [hashed storage pool](../../../administration/repository_storage_paths.md#hashed-object-pools) to share objects with another repository: - All objects are copied from the pool into your fork. diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 8bb6d868270..592041ef4e2 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -1,330 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../signed_commits/gpg.md' +remove_date: '2023-12-01' --- -# Sign commits with GPG **(FREE ALL)** +This document was moved to [another location](../signed_commits/gpg.md). -You can sign the commits you make in a GitLab repository with a -GPG ([GNU Privacy Guard](https://gnupg.org/)) key. When you add a cryptographic -signature to your commit, you provide extra assurance that a commit -originated from you, rather than an impersonator. If GitLab can verify a commit -author's identity with a public GPG key, the commit is marked **Verified** in the -GitLab UI. You can then configure [push rules](../push_rules.md) -for your project to reject individual commits not signed with GPG, or reject all -commits from unverified users. - -NOTE: -GitLab uses the term GPG for all OpenPGP, PGP, and GPG-related material and -implementations. - -For GitLab to consider a commit verified: - -- The committer must have a GPG public/private key pair. -- The committer's public key must be uploaded to their GitLab account. -- One of the email addresses in the GPG public key must match a **verified** email address - used by the committer in GitLab. To keep this address private, use the automatically generated - [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) - GitLab provides in your profile. -- The committer's email address must match the verified email address from the - GPG key. - -GitLab uses its own keyring to verify the GPG signature. It does not access any -public key server. - -GPG verified tags are not supported. - -For more details about GPG, refer to the [related topics list](#related-topics). - -## View a user's public GPG key - -To view a user's public GPG key, you can either: - -- Go to `https://gitlab.example.com/<USERNAME>.gpg`. GitLab displays the GPG key, - if the user has configured one, or a blank page for users without a configured GPG key. -- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper-right corner - of the user's profile, select **View public GPG keys** (**{key}**). - -## Configure commit signing - -To sign commits, you must configure both your local machine and your GitLab account: - -1. [Create a GPG key](#create-a-gpg-key). -1. [Add a GPG key to your account](#add-a-gpg-key-to-your-account). -1. [Associate your GPG key with Git](#associate-your-gpg-key-with-git). -1. [Sign your Git commits](#sign-your-git-commits). - -### Create a GPG key - -If you don't already have a GPG key, create one: - -1. [Install GPG](https://www.gnupg.org/download/) for your operating system. - If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in - the commands on this page. -1. To generate your key pair, run the command appropriate for your version of `gpg`: - - ```shell - # Use this command for the default version of GPG, including - # Gpg4win on Windows, and most macOS versions: - gpg --gen-key - - # Use this command for versions of GPG later than 2.1.17: - gpg --full-gen-key - ``` - -1. Select the algorithm your key should use, or press <kbd>Enter</kbd> to select - the default option, `RSA and RSA`. -1. Select the key length, in bits. GitLab recommends 4096-bit keys. -1. Specify the validity period of your key. This value is subjective, and the - default value is no expiration. -1. To confirm your answers, enter `y`. -1. Enter your name. -1. Enter your email address. It must match a - [verified email address](../../../profile/index.md#change-the-email-displayed-on-your-commits) - in your GitLab account. -1. Optional. Enter a comment to display in parentheses after your name. -1. GPG displays the information you've entered so far. Edit the information or press - <kbd>O</kbd> (for `Okay`) to continue. -1. Enter a strong password, then enter it again to confirm it. -1. To list your private GPG key, run this command, replacing - `<EMAIL>` with the email address you used when you generated the key: - - ```shell - gpg --list-secret-keys --keyid-format LONG <EMAIL> - ``` - -1. In the output, identify the `sec` line, and copy the GPG key ID. It begins after - the `/` character. In this example, the key ID is `30F2B65B9246B6CA`: - - ```plaintext - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` - -1. To show the associated public key, run this command, replacing `<ID>` with the - GPG key ID from the previous step: - - ```shell - gpg --armor --export <ID> - ``` - -1. Copy the public key, including the `BEGIN PGP PUBLIC KEY BLOCK` and - `END PGP PUBLIC KEY BLOCK` lines. You need this key in the next step. - -### Add a GPG key to your account - -To add a GPG key to your user settings: - -1. Sign in to GitLab. -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Add new key**. -1. In **Key**, paste your _public_ key. -1. To add the key to your account, select **Add key**. GitLab shows the key's - fingerprint, email address, and creation date: - -  - -After you add a key, you cannot edit it. Instead, remove the offending key and re-add it. - -### Associate your GPG key with Git - -After you [create your GPG key](#create-a-gpg-key) and -[add it to your account](#add-a-gpg-key-to-your-account), you must configure Git -to use this key: - -1. Run this command to list the private GPG key you just created, - replacing `<EMAIL>` with the email address for your key: - - ```shell - gpg --list-secret-keys --keyid-format LONG <EMAIL> - ``` - -1. Copy the GPG private key ID that starts with `sec`. In this example, the private key ID is - `30F2B65B9246B6CA`: - - ```plaintext - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` - -1. Run this command to configure Git to sign your commits with your key, - replacing `<KEY ID>` with your GPG key ID: - - ```shell - git config --global user.signingkey <KEY ID> - ``` - -### Sign your Git commits - -After you [add your public key to your account](#add-a-gpg-key-to-your-account), -you can sign individual commits manually, or configure Git to default to signed commits: - -- Sign individual Git commits manually: - 1. Add `-S` flag to any commit you want to sign: - - ```shell - git commit -S -m "My commit message" - ``` - - 1. Enter the passphrase of your GPG key when asked. - 1. Push to GitLab and check that your commits [are verified](#verify-commits). -- Sign all Git commits by default by running this command: - - ```shell - git config --global commit.gpgsign true - ``` - -#### Set signing key conditionally - -If you maintain signing keys for separate purposes, such as work and personal -use, use an `IncludeIf` statement in your `.gitconfig` file to set which key -you sign commits with. - -Prerequisites: - -- Requires Git version 2.13 or later. - -1. In the same directory as your main `~/.gitconfig` file, create a second file, - such as `.gitconfig-gitlab`. -1. In your main `~/.gitconfig` file, add your Git settings for work in non-GitLab projects. -1. Append this information to the end of your main `~/.gitconfig` file: - - ```ini - # The contents of this file are included only for GitLab.com URLs - [includeIf "hasconfig:remote.*.url:https://gitlab.com/**"] - - # Edit this line to point to your alternate configuration file - path = ~/.gitconfig-gitlab - ``` - -1. In your alternate `.gitconfig-gitlab` file, add the configuration overrides to - use when you're committing to a GitLab repository. All settings from your - main `~/.gitconfig` file are retained unless you explicitly override them. - In this example, - - ```ini - # Alternate ~/.gitconfig-gitlab file - # These values are used for repositories matching the string 'gitlab.com', - # and override their corresponding values in ~/.gitconfig - - [user] - email = you@example.com - signingkey = <KEY ID> - - [commit] - gpgsign = true - ``` - -## Verify commits - -You can review commits for a merge request, or for an entire project: - -1. To review commits for a project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Commits**. -1. To review commits for a merge request: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Merge requests**, then select your merge request. - 1. Select **Commits**. -1. Identify the commit you want to review. Signed commits show either a **Verified** - or **Unverified** badge, depending on the verification status of the GPG - signature. Unsigned commits do not display a badge: - -  - -1. To display the signature details for a commit, select the GPG badge: - -  - -  - -## Revoke a GPG key - -If a GPG key becomes compromised, revoke it. Revoking a key changes both future and past commits: - -- Past commits signed by this key are marked as unverified. -- Future commits signed by this key are marked as unverified. - -To revoke a GPG key: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Revoke** next to the GPG key you want to delete. - -## Remove a GPG key - -When you remove a GPG key from your GitLab account: - -- Previous commits signed with this key remain verified. -- Future commits (including any commits created but not yet pushed) that attempt - to use this key are unverified. - -To remove a GPG key from your account: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. Select **GPG Keys** (**{key}**). -1. Select **Remove** (**{remove}**) next to the GPG key you want to delete. - -If you must unverify both future and past commits, -[revoke the associated GPG key](#revoke-a-gpg-key) instead. - -## Related topics - -- [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) -- [Sign commits with SSH keys](../ssh_signed_commits/index.md) -- [Commits API](../../../../api/commits.md) -- GPG resources: - - [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) - - [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) - - [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) - - [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) - - [Review existing GPG keys in your instance](../../../../administration/credentials_inventory.md#review-existing-gpg-keys) - -## Troubleshooting - -### Fix verification problems with signed commits - -Commits can be signed with [X.509 certificates](../x509_signed_commits/index.md) -or a GPG key. The verification process for both methods can fail for multiple reasons: - -| Value | Description | Possible Fixes | -|-----------------------------|-------------|----------------| -| `UNVERIFIED` | The commit signature is not valid. | Sign the commit with a valid signature. | -| `SAME_USER_DIFFERENT_EMAIL` | The GPG key used to sign the commit does not contain the committer email, but does contain a different valid email for the committer. | Amend the commit to use an email address that matches the GPG key, or update the GPG key [to include the email address](https://security.stackexchange.com/a/261468). | -| `OTHER_USER` | The signature and GPG key are valid, but the key belongs to a different user than the committer. | Amend the commit to use the correct email address, or amend the commit to use a GPG key associated with your user. | -| `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | -| `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](#add-a-gpg-key-to-your-account) to your GitLab profile. | -| `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | - -### Secret key not available - -If you receive the errors `secret key not available` -or `gpg: signing failed: secret key not available`, try using `gpg2` instead of `gpg`: - -```shell -git config --global gpg.program gpg2 -``` - -If your GPG key is password protected and the password entry prompt does not appear, -add `export GPG_TTY=$(tty)` to your shell's `rc` file (commonly `~/.bashrc` or `~/.zshrc`) - -### GPG failed to sign the data - -If your GPG key is password protected and you receive the error: - -```shell -error: gpg failed to sign the data -fatal: failed to write commit object -``` - -If the password entry prompt does not appear, add `export GPG_TTY=$(tty)` to your shell's `rc` file -(commonly `~/.bashrc` or `~/.zshrc`) and restart your terminal. +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 3d00ceafc05..e97892458b4 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -55,7 +55,7 @@ to a branch in the repository. When you use the command line, you can commit mul [Revert a commit](../merge_requests/revert_changes.md#revert-a-commit) from the UI to a selected branch. - **Sign a commit:** - Use GPG to [sign your commits](gpg_signed_commits/index.md). + Add extra security by [signing your commits](signed_commits/index.md). ## Clone a repository @@ -73,7 +73,7 @@ into Xcode on macOS. 1. Select **Xcode**. The project is cloned onto your computer and you are -prompted to open XCode. +prompted to open Xcode. ### Clone and open in Visual Studio Code @@ -216,6 +216,11 @@ To render an OpenAPI file: 1. Go to the OpenAPI file in your repository. 1. Between the **Display source** and **Edit** buttons, select **Display OpenAPI**. When an OpenAPI file is found, it replaces the **Display rendered file** button. +1. To display the `operationId` in the operations list, add `displayOperationId=true` to the query string. + +NOTE: +When `displayOperationId` is present in the query string and has _any_ value, it +evaluates to `true`. This behavior matches the default behavior of Swagger. ## Repository size diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index 70ee841a991..2a1fc0cddf8 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -37,7 +37,7 @@ When commits include changes to Jupyter Notebook files, GitLab: - Enables switching between raw and rendered diffs on the Commit and Compare pages. (Not available on merge request pages.) - Renders images on the diffs. -Code suggestions are not available on diffs and merge requests for `.ipynb` files. +Code Suggestions are not available on diffs and merge requests for `.ipynb` files. Cleaner notebook diffs are not generated when the notebook is too large. diff --git a/doc/user/project/repository/managing_large_repositories.md b/doc/user/project/repository/managing_large_repositories.md index e5c1e4a6614..1d5127b5e08 100644 --- a/doc/user/project/repository/managing_large_repositories.md +++ b/doc/user/project/repository/managing_large_repositories.md @@ -1,53 +1,411 @@ --- stage: Systems -group: Distribution +group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -description: "Documentation on large repositories." --- -# Managing large repositories **(FREE SELF)** +# Managing monorepos -GitLab, like any Git based system, is subject to similar performance restraints when it comes to large -repositories that size into the gigabytes. +Monorepos have become a regular part of development team workflows. While they have many advantages, monorepos can present performance challenges +when using them in GitLab. Therefore, you should know: -In the following sections, we detail several best practices for improving performance with these large repositories on GitLab. +- What repository characteristics can impact performance. +- Some tools and steps to optimize monorepos. -## Large File System (LFS) +## Impact on performance -It's *strongly* recommended in any Git system that binary or blob files (for example, packages, audio, video, or graphics) are stored as Large File Storage (LFS) objects. With LFS, the objects are stored externally, such as in Object Storage, which reduces the number and size of objects in the repository. Storing objects in external Object Storage can improve performance. +Because GitLab is a Git-based system, it is subject to similar performance +constraints as Git when it comes to large repositories that are gigabytes in +size. -To analyze if a repository has large objects, you can use a tool like [`git-sizer`](https://github.com/github/git-sizer) for detailed analysis. This tool shows details about what makes up the repository, and highlights any areas of concern. If any large objects are found, you can then remove them with a tool such as [`git filter-repo`](reducing_the_repo_size_using_git.md). +Monorepos can be large for [many reasons](https://about.gitlab.com/blog/2022/09/06/speed-up-your-monorepo-workflow-in-git/#characteristics-of-monorepos). + +Large repositories pose a performance risk performance when used in GitLab, especially if a large monorepo receives many clones or pushes a day, which is common for them. + +Git itself has performance limitations when it comes to handling +monorepos. + +[Gitaly](https://gitlab.com/gitlab-org/gitaly) is our Git storage service built +on top of [Git](https://git-scm.com/). This means that any limitations of +Git are experienced in Gitaly, and in turn by end users of GitLab. + +## Profiling repositories + +Large repositories generally experience performance issues in Git. Knowing why +your repository is large can help you develop mitigation strategies to avoid +performance problems. + +You can use [`git-sizer`](https://github.com/github/git-sizer) to get a snapshot +of repository characteristics and discover problem aspects of your monorepo. + +For example: + +```shell +Processing blobs: 1652370 +Processing trees: 3396199 +Processing commits: 722647 +Matching commits to trees: 722647 +Processing annotated tags: 534 +Processing references: 539 +| Name | Value | Level of concern | +| ---------------------------- | --------- | ------------------------------ | +| Overall repository size | | | +| * Commits | | | +| * Count | 723 k | * | +| * Total size | 525 MiB | ** | +| * Trees | | | +| * Count | 3.40 M | ** | +| * Total size | 9.00 GiB | **** | +| * Total tree entries | 264 M | ***** | +| * Blobs | | | +| * Count | 1.65 M | * | +| * Total size | 55.8 GiB | ***** | +| * Annotated tags | | | +| * Count | 534 | | +| * References | | | +| * Count | 539 | | +| | | | +| Biggest objects | | | +| * Commits | | | +| * Maximum size [1] | 72.7 KiB | * | +| * Maximum parents [2] | 66 | ****** | +| * Trees | | | +| * Maximum entries [3] | 1.68 k | * | +| * Blobs | | | +| * Maximum size [4] | 13.5 MiB | * | +| | | | +| History structure | | | +| * Maximum history depth | 136 k | | +| * Maximum tag depth [5] | 1 | | +| | | | +| Biggest checkouts | | | +| * Number of directories [6] | 4.38 k | ** | +| * Maximum path depth [7] | 13 | * | +| * Maximum path length [8] | 134 B | * | +| * Number of files [9] | 62.3 k | * | +| * Total size of files [9] | 747 MiB | | +| * Number of symlinks [10] | 40 | | +| * Number of submodules | 0 | | +``` + +In this example, a few items are raised with a high level of concern. See the +following sections for information on solving: + +- A high number of references. +- Large blobs. + +### Large number of references + +A reference in Git (a branch or tag) is used to refer to a commit. Each +reference is stored as an individual file. If you are curious, you can go +to any `.git` directory and look under the `refs` directory. + +A large number of references can cause performance problems because, with more references, +object walks that Git does are larger for various operations such as clones, pushes, and +housekeeping tasks. + +#### Mitigation strategies + +To mitigate the effects of a large number of references in a monorepo: + +- Create an automated process for cleaning up old branches. +- If certain references don't need to be visible to the client, hide them using the + [`transfer.hideRefs`](https://git-scm.com/docs/git-config#Documentation/git-config.txt-transferhideRefs) + configuration setting. Because Gitaly ignores any on-server Git configuration, you must change the Gitaly configuration + itself in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['configuration'] = { + # ... + git: { + # ... + config: [ + # ... + { key: "transfer.hideRefs", value: "refs/namespace_to_hide" }, + ], + }, + } + ``` + +In Git 2.42.0 and later, different Git operations can skip over hidden references +when doing an object graph walk. + +### Using LFS for large blobs + +Because Git is built to handle text data, it doesn't handle large +binary files efficiently. + +Therefore, you should store binary or blob files (for example, packages, audio, video, or graphics) +as Large File Storage (LFS) objects. With LFS, the objects are stored externally, such as in Object +Storage, which reduces the number and size of objects in the repository. Storing +objects in external Object Storage can improve performance. + +To analyze if a repository has large objects, you can use a tool like +[`git-sizer`](https://github.com/github/git-sizer) for detailed analysis. This +tool shows details about what makes up the repository, and highlights any areas +of concern. If any large objects are found, you can then remove them with a tool +such as [`git filter-repo`](reducing_the_repo_size_using_git.md). For more information, refer to the [Git LFS documentation](../../../topics/git/lfs/index.md). -## Gitaly Pack Objects Cache +## Optimizing large repositories for GitLab + +Other than modifying your workflow and the actual repository, you can take other +steps to maximize performance of monorepos with GitLab. + +### Gitaly pack-objects cache + +For very active repositories with a large number of references and files, consider using the +[Gitaly pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). +The pack-objects cache: + +- Benefits all repositories on your GitLab server. +- Automatically works for forks. + +You should always: + +- Fetch incrementally. Do not clone in a way that recreates all of the worktree. +- Use shallow clones to reduce data transfer. Be aware that this puts more burden on GitLab instance because of higher CPU impact. + +Control the clone directory if you heavily use a fork-based workflow. Optimize +`git clean` flags to ensure that you remove or keep data that might affect or +speed-up your build. + +For more information, see [Pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). + +### Reduce concurrent clones in CI/CD + +Large repositories tend to be monorepos. This usually means that these +repositories get a lot of traffic not only from users, but from CI/CD. + +CI/CD loads tend to be concurrent because pipelines are scheduled during set times. +As a result, the Git requests against the repositories can spike notably during +these times and lead to reduced performance for both CI/CD and users alike. + +You should reduce CI/CD pipeline concurrency by staggering them to run at different times. For example, a set running at one time and another set running several +minutes later. + +#### Shallow cloning + +GitLab and GitLab Runner perform a [shallow clone](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) +by default. + +Ideally, you should always use `GIT_DEPTH` with a small number +like 10. This instructs GitLab Runner to perform shallow clones. +Shallow clones make Git request only the latest set of changes for a given branch, +up to desired number of commits as defined by the `GIT_DEPTH` variable. + +This significantly speeds up fetching of changes from Git repositories, +especially if the repository has a very long backlog consisting of a number +of big files because we effectively reduce amount of data transfer. +The following pipeline configuration example makes the runner shallow clone to fetch only a given branch. +The runner does not fetch any other branches nor tags. + +```yaml +variables: + GIT_DEPTH: 10 + +test: + script: + - ls -al +``` + +#### Git strategy + +By default, GitLab is configured to use the [`fetch` Git strategy](../../../ci/runners/configure_runners.md#git-strategy), +which is recommended for large repositories. +This strategy reduces the amount of data to transfer and +does not really impact the operations that you might do on a repository from CI/CD. + +#### Git clone path + +[`GIT_CLONE_PATH`](../../../ci/runners/configure_runners.md#custom-build-directories) allows you to +control where you clone your repositories. This can have implications if you +heavily use big repositories with a fork-based workflow. + +A fork, from the perspective of GitLab Runner, is stored as a separate repository +with a separate worktree. That means that GitLab Runner cannot optimize the usage +of worktrees and you might have to instruct GitLab Runner to use that. + +In such cases, ideally you want to make the GitLab Runner executor be used only +for the given project and not shared across different projects to make this +process more efficient. + +The [`GIT_CLONE_PATH`](../../../ci/runners/configure_runners.md#custom-build-directories) must be +in the directory set in `$CI_BUILDS_DIR`. You can't pick any path from disk. + +#### Git clean flags + +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags) allows you to control +whether or not you require the `git clean` command to be executed for each CI/CD +job. By default, GitLab ensures that: + +- You have your worktree on the given SHA. +- Your repository is clean. + +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags) is disabled when set +to `none`. On very big repositories, this might be desired because `git +clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx +-e .build/` (for example) allows you to control and disable removal of some +directories in the worktree between subsequent runs, which can speed-up +the incremental builds. This has the biggest effect if you re-use existing +machines and have an existing worktree that you can re-use for builds. + +For exact parameters accepted by +[`GIT_CLEAN_FLAGS`](../../../ci/runners/configure_runners.md#git-clean-flags), see the documentation +for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters +are dependent on the Git version. + +#### Git fetch extra flags + +[`GIT_FETCH_EXTRA_FLAGS`](../../../ci/runners/configure_runners.md#git-fetch-extra-flags) allows you +to modify `git fetch` behavior by passing extra flags. + +For example, if your project contains a large number of tags that your CI/CD jobs don't rely on, +you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags) +to the extra flags to make your fetches faster and more compact. + +Also in the case where you repository does _not_ contain a lot of +tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). +If your CI/CD builds do not depend on Git tags, setting `--no-tags` is worth trying. + +For more information, see the [`GIT_FETCH_EXTRA_FLAGS` documentation](../../../ci/runners/configure_runners.md#git-fetch-extra-flags). + +#### Fork-based workflow + +Following the guidelines above, let's imagine that we want to: + +- Optimize for a big project (more than 50k files in directory). +- Use forks-based workflow for contributing. +- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. +- Runner assigned only to project and all forks. + +Let's consider the following two examples, one using `shell` executor and +other using `docker` executor. + +##### `shell` executor example + +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "shell" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.custom_build_dir] + enabled = true +``` + +This `config.toml`: + +- Uses the `shell` executor, +- Specifies a custom `/builds` directory where all clones are stored. +- Enables the ability to specify `GIT_CLONE_PATH`, +- Runs at most 4 jobs at once. + +##### `docker` executor example + +Let's assume that you have the following [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This `config.toml`: + +- Uses the `docker` executor, +- Specifies a custom `/builds` directory on disk where all clones are stored. + We host mount the `/builds` directory to make it reusable between subsequent runs + and be allowed to override the cloning strategy. +- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. +- Runs at most 4 jobs at once. + +##### Our `.gitlab-ci.yml` + +Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. + +Our pipeline is most performant if we use the following `.gitlab-ci.yml`: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME + +build: + script: ls -al +``` + +This YAML setting configures a custom clone path. This path makes it possible to re-use worktrees +between the parent project and forks because we use the same clone path for all forks. + +Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting +between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor. +When we use it to construct the path, this directory does not conflict +with other concurrent jobs running. + +### Store custom clone options in `config.toml` + +Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. +However, sometimes it is desirable to make these schemes part of the runner's configuration. -Gitaly, the service that provides storage for Git repositories, can be configured to cache a short rolling window of Git fetch responses. This is recommended for large repositories as it can notably reduce server load when your server receives lots of fetch traffic. +In the above example of forks, making this configuration discoverable for users may be preferred, +but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. +In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it +a configuration of the runner. -Refer to the [Gitaly Pack Objects Cache for more information](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache). +We can extend our [`config.toml`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +with the following specification that is used by the runner if `.gitlab-ci.yml` does not override it: -## Reference Architectures +```toml +concurrent = 4 -Large repositories tend to be found in larger organisations with many users. The GitLab Quality and Support teams provide several [Reference Architectures](../../../administration/reference_architectures/index.md) that are the recommended way to deploy GitLab at scale. +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" -In these types of setups it's recommended that the GitLab environment used matches a Reference Architecture to improve performance. + environment = [ + "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" + ] -## Gitaly Cluster + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` -Gitaly Cluster can notably improve large repository performance as it holds multiple replicas of the repository across several nodes. As a result, Gitaly Cluster can load balance read requests against those repositories and is also fault-tolerant. +This makes the cloning configuration to be part of the given runner +and does not require us to update each `.gitlab-ci.yml`. -It's recommended for large repositories, however, Gitaly Cluster is a large solution with additional complexity of setup, and management. Refer to the [Gitaly Cluster documentation for more information](../../../administration/gitaly/index.md), specifically the [Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. +### Reference architectures -## Keep GitLab up to date +Large repositories tend to be found in larger organisations with many users. The GitLab Quality and Support teams provide several [reference architectures](../../../administration/reference_architectures/index.md) that are the recommended way to deploy GitLab at scale. -Performance improvements and fixes are added continuously in GitLab. As such, it's recommended you keep GitLab updated to the latest version where possible to benefit from these. +In these types of setups, the GitLab environment used should match a reference architecture to improve performance. -## Reduce concurrent clones in CI/CD +### Gitaly Cluster -Large repositories tend to be monorepos. This in turn typically means that these repositories get a lot of traffic not only from users, but from CI/CD. +Gitaly Cluster can notably improve large repository performance because it holds multiple replicas of the repository across several nodes. +As a result, Gitaly Cluster can load balance read requests against those replicas and is fault-tolerant. -CI/CD loads tend to be concurrent as pipelines are scheduled during set times. As a result, the Git requests against the repositories can spike notably during these times and lead to reduced performance for both CI and users alike. +Though Gitaly Cluster is recommended for large repositories, it is a large solution with additional complexity of setup and management. Refer to the +[Gitaly Cluster documentation for more information](../../../administration/gitaly/index.md), specifically the +[Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. -When designing CI/CD pipelines, it's advisable to reduce their concurrency by staggering them to run at different times, for example, a set running at one time, and another set running several minutes later. +### Keep GitLab up to date -There's several other actions that can be explored to improve CI/CD performance with large repositories. Refer to the [Runner documentation for more information](../../../ci/large_repositories/index.md). +You should keep GitLab updated to the latest version where possible to benefit from performance improvements and fixes are added continuously to GitLab. diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md index fade9e1b63c..9d97f53cb43 100644 --- a/doc/user/project/repository/mirror/bidirectional.md +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -44,7 +44,7 @@ and [pull](pull.md#pull-from-a-remote-repository) mirrors in the upstream GitLab To create the webhook in the downstream instance: 1. Create a [personal access token](../../../profile/personal_access_tokens.md) with `API` scope. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. On the left sidebar, select **Settings > Webhooks**. 1. Add the webhook **URL**, which (in this case) uses the [Pull Mirror API](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project) diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index 7ade27e556d..b16197e45b2 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -40,7 +40,7 @@ Prerequisites: - If your mirror connects with `ssh://`, the host key must be detectable on the server, or you must have a local copy of the key. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Select **Add new**. @@ -111,7 +111,7 @@ Prerequisite: - You must have at least the Maintainer role for the project. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Scroll to **Mirrored repositories** and identify the mirror to update. @@ -124,9 +124,8 @@ When you create a mirror, you must configure the authentication method for it. GitLab supports these authentication methods: - [SSH authentication](#ssh-authentication). -- Password. +- Username and password. -When using password authentication, ensure you specify the username. For a [project access token](../../settings/project_access_tokens.md) or [group access token](../../../group/settings/group_access_tokens.md), use the username (not token name) and the token as the password. @@ -154,7 +153,7 @@ When you mirror a repository and select the **SSH public key** as your authentication method, GitLab generates a public key for you. The non-GitLab server needs this key to establish trust with your GitLab repository. To copy your SSH public key: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Scroll to **Mirrored repositories**. diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md index ba54c18f8ee..5b3c76d982a 100644 --- a/doc/user/project/repository/mirror/pull.md +++ b/doc/user/project/repository/mirror/pull.md @@ -64,14 +64,13 @@ Prerequisites: token serves as your GitHub password. - [GitLab Silent Mode](../../../../administration/silent_mode/index.md) is not enabled. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. -1. Enter the **Git repository URL**. Include the username - in the URL, if required: `https://MYUSERNAME@gitlab.com/GROUPNAME/PROJECTNAME.git` +1. Enter the **Git repository URL**. NOTE: - To mirror the `gitlab` repository, use `git@gitlab.com:gitlab-org/gitlab.git` + To mirror the `gitlab` repository, use `gitlab.com:gitlab-org/gitlab.git` or `https://gitlab.com/gitlab-org/gitlab.git`. 1. In **Mirror direction**, select **Pull**. diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md index cd4fe68b01b..e18e3631d7f 100644 --- a/doc/user/project/repository/mirror/push.md +++ b/doc/user/project/repository/mirror/push.md @@ -37,7 +37,7 @@ and pulling from, remote mirrors. To set up push mirroring for an existing project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. Enter a repository URL. @@ -89,12 +89,12 @@ To configure a mirror from GitLab to GitHub: 1. Enter a **Git repository URL** with this format, changing the variables as needed: ```plaintext - https://USERNAME@github.com/GROUP/PROJECT.git + https://github.com/GROUP/PROJECT.git ``` - - `USERNAME`: The username of the owner of the personal access token. - `GROUP`: The group on GitHub. - `PROJECT`: The project on GitHub. +1. For **Username**, enter the username of the owner of the personal access token. 1. For **Password**, enter your GitHub personal access token. 1. Select **Mirror repository**. @@ -164,19 +164,17 @@ To set up a mirror from GitLab to AWS CodeCommit: 1. Open your new repository, and then select **Clone URL > Clone HTTPS** (not **Clone HTTPS (GRC)**). 1. In GitLab, open the repository to be push-mirrored. 1. Select **Settings > Repository**, and then expand **Mirroring repositories**. -1. Fill in the **Git repository URL** field using this format: +1. Fill in the **Git repository URL** field using this format, replacing + `<aws-region>` with your AWS region, and + `<your_codecommit_repo>` with the name of your repository in CodeCommit: ```plaintext - https://<your_aws_git_userid>@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> + https://git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> ``` - Replace `<your_aws_git_userid>` with the AWS **special HTTPS Git user ID** - from the IAM Git credentials created earlier. Replace `<your_codecommit_repo>` - with the name of your repository in CodeCommit. - -1. For **Mirror direction**, select **Push**. -1. For **Authentication method**, select **Password**. Fill in the **Password** box - with the special IAM Git clone user ID **password** created earlier in AWS. +1. For **Authentication method**, select **Username and Password**. +1. For **Username**, enter the AWS **special HTTPS Git user ID**. +1. For **Password**, enter the special IAM Git clone user ID password created earlier in AWS. 1. Leave the option **Only mirror protected branches** for CodeCommit. It pushes more frequently (from every five minutes to every minute). @@ -202,7 +200,8 @@ If it isn't working correctly, a red `error` tag appears, and shows the error me [personal access token](../../../profile/personal_access_tokens.md) with `write_repository` scope. 1. On the source GitLab instance: 1. Enter the **Git repository URL** using this format: - `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + `https://<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + 1. Enter the **Username** `oauth2`. 1. Enter the **Password**. Use the GitLab personal access token created on the destination GitLab instance. 1. Select **Mirror repository**. diff --git a/doc/user/project/repository/mirror/troubleshooting.md b/doc/user/project/repository/mirror/troubleshooting.md index 5817aab5fc7..a83231ceb63 100644 --- a/doc/user/project/repository/mirror/troubleshooting.md +++ b/doc/user/project/repository/mirror/troubleshooting.md @@ -67,7 +67,7 @@ If you receive this error after creating a new project using Check if the repository owner is specified in the URL of your mirrored repository: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. If no repository owner is specified, delete and add the URL again in this format, @@ -168,7 +168,7 @@ Prerequisites: To resolve the issue: 1. [Verify the host key](index.md#verify-a-host-key). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. 1. To refresh the keys, either: diff --git a/doc/user/project/repository/push_rules.md b/doc/user/project/repository/push_rules.md index 2756149b5bd..d61d09301a5 100644 --- a/doc/user/project/repository/push_rules.md +++ b/doc/user/project/repository/push_rules.md @@ -39,7 +39,7 @@ Prerequisite: To create global push rules: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Push Rules**. 1. Expand **Push rules**. @@ -52,7 +52,7 @@ The push rule of an individual project overrides the global push rule. To override global push rules for a specific project, or to update the rules for an existing project to match new global push rules: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > Repository**. 1. Expand **Push rules**. 1. Set the rule you want. @@ -126,7 +126,7 @@ Some validation examples: Use these rules to prevent unintended consequences. -- **Reject unsigned commits**: Commit must be signed through [GPG](gpg_signed_commits/index.md). This rule +- **Reject unsigned commits**: Commit must be signed through [GPG](signed_commits/gpg.md). This rule can block some legitimate commits [created in the Web IDE](#reject-unsigned-commits-push-rule-disables-web-ide), and allow [unsigned commits created in the GitLab UI](#unsigned-commits-created-in-the-gitlab-ui). - **Do not allow users to remove Git tags with `git push`**: Users cannot use `git push` to remove Git tags. @@ -274,7 +274,7 @@ to use them as standard characters in a match condition. ## Related topics - [Git server hooks](../../../administration/server_hooks.md) (previously called server hooks), to create complex custom push rules -- [Signing commits with GPG](gpg_signed_commits/index.md) +- [Signing commits with GPG](signed_commits/gpg.md) - [Protected branches](../protected_branches.md) ## Troubleshooting diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 590323bfadd..bfe8964a876 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -26,7 +26,7 @@ you begin. The best way to back up a repository is to The size of a repository is determined by computing the accumulated size of all files in the repository. It is similar to executing `du --summarize --bytes` on your repository's -[hashed storage path](../../../administration/repository_storage_types.md). +[hashed storage path](../../../administration/repository_storage_paths.md). ## Purge files from repository history @@ -53,7 +53,7 @@ visible even after they have been removed from the repository. To purge files from a GitLab repository: -1. Install either [`git filter-repo`](https://github.com/newren/git-filter-repo/blob/main/INSTALL.md) or +1. Install [`git filter-repo`](https://github.com/newren/git-filter-repo/blob/main/INSTALL.md) and optionally [`git-sizer`](https://github.com/github/git-sizer#getting-started) using a supported package manager or from source. diff --git a/doc/user/project/repository/signed_commits/gpg.md b/doc/user/project/repository/signed_commits/gpg.md new file mode 100644 index 00000000000..88f6917d4b6 --- /dev/null +++ b/doc/user/project/repository/signed_commits/gpg.md @@ -0,0 +1,284 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sign commits with GPG **(FREE ALL)** + +You can sign the commits you make in a GitLab repository with a +GPG ([GNU Privacy Guard](https://gnupg.org/)) key. + +NOTE: +GitLab uses the term GPG for all OpenPGP, PGP, and GPG-related material and +implementations. + +For GitLab to consider a commit verified: + +- The committer must have a GPG public/private key pair. +- The committer's public key must be uploaded to their GitLab account. +- One of the email addresses in the GPG public key must match a **verified** email address + used by the committer in GitLab. To keep this address private, use the automatically generated + [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) + GitLab provides in your profile. +- The committer's email address must match the verified email address from the + GPG key. + +GitLab uses its own keyring to verify the GPG signature. It does not access any +public key server. + +GPG verified tags are not supported. + +For more details about GPG, refer to the [related topics list](#related-topics). + +## View a user's public GPG key + +To view a user's public GPG key, you can either: + +- Go to `https://gitlab.example.com/<USERNAME>.gpg`. GitLab displays the GPG key, + if the user has configured one, or a blank page for users without a configured GPG key. +- Go to the user's profile (such as `https://gitlab.example.com/<USERNAME>`). In the upper-right corner + of the user's profile, select **View public GPG keys** (**{key}**). + +## Configure commit signing + +To sign commits, you must configure both your local machine and your GitLab account: + +1. [Create a GPG key](#create-a-gpg-key). +1. [Add a GPG key to your account](#add-a-gpg-key-to-your-account). +1. [Associate your GPG key with Git](#associate-your-gpg-key-with-git). +1. [Sign your Git commits](#sign-your-git-commits). + +### Create a GPG key + +If you don't already have a GPG key, create one: + +1. [Install GPG](https://www.gnupg.org/download/) for your operating system. + If your operating system has `gpg2` installed, replace `gpg` with `gpg2` in + the commands on this page. +1. To generate your key pair, run the command appropriate for your version of `gpg`: + + ```shell + # Use this command for the default version of GPG, including + # Gpg4win on Windows, and most macOS versions: + gpg --gen-key + + # Use this command for versions of GPG later than 2.1.17: + gpg --full-gen-key + ``` + +1. Select the algorithm your key should use, or press <kbd>Enter</kbd> to select + the default option, `RSA and RSA`. +1. Select the key length, in bits. GitLab recommends 4096-bit keys. +1. Specify the validity period of your key. This value is subjective, and the + default value is no expiration. +1. To confirm your answers, enter `y`. +1. Enter your name. +1. Enter your email address. It must match a + [verified email address](../../../profile/index.md#change-the-email-displayed-on-your-commits) + in your GitLab account. +1. Optional. Enter a comment to display in parentheses after your name. +1. GPG displays the information you've entered so far. Edit the information or press + <kbd>O</kbd> (for `Okay`) to continue. +1. Enter a strong password, then enter it again to confirm it. +1. To list your private GPG key, run this command, replacing + `<EMAIL>` with the email address you used when you generated the key: + + ```shell + gpg --list-secret-keys --keyid-format LONG <EMAIL> + ``` + +1. In the output, identify the `sec` line, and copy the GPG key ID. It begins after + the `/` character. In this example, the key ID is `30F2B65B9246B6CA`: + + ```plaintext + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` + +1. To show the associated public key, run this command, replacing `<ID>` with the + GPG key ID from the previous step: + + ```shell + gpg --armor --export <ID> + ``` + +1. Copy the public key, including the `BEGIN PGP PUBLIC KEY BLOCK` and + `END PGP PUBLIC KEY BLOCK` lines. You need this key in the next step. + +### Add a GPG key to your account + +To add a GPG key to your user settings: + +1. Sign in to GitLab. +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Add new key**. +1. In **Key**, paste your _public_ key. +1. To add the key to your account, select **Add key**. GitLab shows the key's + fingerprint, email address, and creation date: + +  + +After you add a key, you cannot edit it. Instead, remove the offending key and re-add it. + +### Associate your GPG key with Git + +After you [create your GPG key](#create-a-gpg-key) and +[add it to your account](#add-a-gpg-key-to-your-account), you must configure Git +to use this key: + +1. Run this command to list the private GPG key you just created, + replacing `<EMAIL>` with the email address for your key: + + ```shell + gpg --list-secret-keys --keyid-format LONG <EMAIL> + ``` + +1. Copy the GPG private key ID that starts with `sec`. In this example, the private key ID is + `30F2B65B9246B6CA`: + + ```plaintext + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` + +1. Run this command to configure Git to sign your commits with your key, + replacing `<KEY ID>` with your GPG key ID: + + ```shell + git config --global user.signingkey <KEY ID> + ``` + +### Sign your Git commits + +After you [add your public key to your account](#add-a-gpg-key-to-your-account), +you can sign individual commits manually, or configure Git to default to signed commits: + +- Sign individual Git commits manually: + 1. Add `-S` flag to any commit you want to sign: + + ```shell + git commit -S -m "My commit message" + ``` + + 1. Enter the passphrase of your GPG key when asked. + 1. Push to GitLab and check that your commits [are verified](../signed_commits/index.md#verify-commits). +- Sign all Git commits by default by running this command: + + ```shell + git config --global commit.gpgsign true + ``` + +#### Set signing key conditionally + +If you maintain signing keys for separate purposes, such as work and personal +use, use an `IncludeIf` statement in your `.gitconfig` file to set which key +you sign commits with. + +Prerequisites: + +- Requires Git version 2.13 or later. + +1. In the same directory as your main `~/.gitconfig` file, create a second file, + such as `.gitconfig-gitlab`. +1. In your main `~/.gitconfig` file, add your Git settings for work in non-GitLab projects. +1. Append this information to the end of your main `~/.gitconfig` file: + + ```ini + # The contents of this file are included only for GitLab.com URLs + [includeIf "hasconfig:remote.*.url:https://gitlab.com/**"] + + # Edit this line to point to your alternate configuration file + path = ~/.gitconfig-gitlab + ``` + +1. In your alternate `.gitconfig-gitlab` file, add the configuration overrides to + use when you're committing to a GitLab repository. All settings from your + main `~/.gitconfig` file are retained unless you explicitly override them. + In this example, + + ```ini + # Alternate ~/.gitconfig-gitlab file + # These values are used for repositories matching the string 'gitlab.com', + # and override their corresponding values in ~/.gitconfig + + [user] + email = you@example.com + signingkey = <KEY ID> + + [commit] + gpgsign = true + ``` + +## Revoke a GPG key + +If a GPG key becomes compromised, revoke it. Revoking a key changes both future and past commits: + +- Past commits signed by this key are marked as unverified. +- Future commits signed by this key are marked as unverified. + +To revoke a GPG key: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Revoke** next to the GPG key you want to delete. + +## Remove a GPG key + +When you remove a GPG key from your GitLab account: + +- Previous commits signed with this key remain verified. +- Future commits (including any commits created but not yet pushed) that attempt + to use this key are unverified. + +To remove a GPG key from your account: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. Select **GPG Keys** (**{key}**). +1. Select **Remove** (**{remove}**) next to the GPG key you want to delete. + +If you must unverify both future and past commits, +[revoke the associated GPG key](#revoke-a-gpg-key) instead. + +## Related topics + +- GPG resources: + - [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) + - [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) + - [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) + - [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) + - [Review existing GPG keys in your instance](../../../../administration/credentials_inventory.md#review-existing-gpg-keys) + +## Troubleshooting + +### Secret key not available + +If you receive the errors `secret key not available` +or `gpg: signing failed: secret key not available`, try using `gpg2` instead of `gpg`: + +```shell +git config --global gpg.program gpg2 +``` + +If your GPG key is password protected and the password entry prompt does not appear, +add `export GPG_TTY=$(tty)` to your shell's `rc` file (commonly `~/.bashrc` or `~/.zshrc`) + +### GPG failed to sign the data + +If your GPG key is password protected and you receive the error: + +```shell +error: gpg failed to sign the data +fatal: failed to write commit object +``` + +If the password entry prompt does not appear, add `export GPG_TTY=$(tty)` to your shell's `rc` file +(commonly `~/.bashrc` or `~/.zshrc`) and restart your terminal. diff --git a/doc/user/project/repository/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png b/doc/user/project/repository/signed_commits/img/profile_settings_gpg_keys_single_key.png Binary files differindex ae0a8696c6c..ae0a8696c6c 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png +++ b/doc/user/project/repository/signed_commits/img/profile_settings_gpg_keys_single_key.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_and_unsigned_commits.png b/doc/user/project/repository/signed_commits/img/project_signed_and_unsigned_commits.png Binary files differindex e1d44f15f3f..e1d44f15f3f 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_and_unsigned_commits.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_and_unsigned_commits.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_unverified_signature.png b/doc/user/project/repository/signed_commits/img/project_signed_commit_unverified_signature.png Binary files differindex 763a677f94a..763a677f94a 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_unverified_signature.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_commit_unverified_signature.png diff --git a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_verified_signature.png b/doc/user/project/repository/signed_commits/img/project_signed_commit_verified_signature.png Binary files differindex 1b6fa3fc2e2..1b6fa3fc2e2 100644 --- a/doc/user/project/repository/gpg_signed_commits/img/project_signed_commit_verified_signature.png +++ b/doc/user/project/repository/signed_commits/img/project_signed_commit_verified_signature.png diff --git a/doc/user/project/repository/signed_commits/index.md b/doc/user/project/repository/signed_commits/index.md new file mode 100644 index 00000000000..e6632ecb81a --- /dev/null +++ b/doc/user/project/repository/signed_commits/index.md @@ -0,0 +1,64 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Signed commits **(FREE ALL)** + +When you add a cryptographic signature to your commit, you provide extra assurance that a commit +originated from you, rather than an impersonator. If GitLab can verify a commit +author's identity with a public GPG key, the commit is marked **Verified** in the +GitLab UI. You can then configure [push rules](../push_rules.md) +for your project to reject individual commits not signed with GPG, or reject all +commits from unverified users. + +Sign commits with your: + +- [SSH key](ssh.md). +- [GPG key](gpg.md). +- [Personal x.509 certificate](x509.md). + +## Verify commits + +You can review commits for a merge request, or for an entire project, to confirm +they are signed: + +1. To review commits for a project: + 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. Select **Code > Commits**. +1. To review commits for a merge request: + 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. + 1. On the left sidebar, select **Merge requests**, then select your merge request. + 1. Select **Commits**. +1. Identify the commit you want to review. Signed commits show either a **Verified** + or **Unverified** badge, depending on the verification status of the signature. + Unsigned commits do not display a badge: + +  + +1. To display the signature details for a commit, select **Verified** to see + the fingerprint or key ID: + +  + +  + +You can also [use the Commits API](../../../../api/commits.md#get-gpg-signature-of-a-commit) +to check a commit's signature. + +## Troubleshooting + +### Fix verification problems with signed commits + +The verification process for commits signed with GPG keys or X.509 certificates +can fail for multiple reasons: + +| Value | Description | Possible Fixes | +|-----------------------------|-------------|----------------| +| `UNVERIFIED` | The commit signature is not valid. | Sign the commit with a valid signature. | +| `SAME_USER_DIFFERENT_EMAIL` | The GPG key used to sign the commit does not contain the committer email, but does contain a different valid email for the committer. | Amend the commit to use an email address that matches the GPG key, or update the GPG key [to include the email address](https://security.stackexchange.com/a/261468). | +| `OTHER_USER` | The signature and GPG key are valid, but the key belongs to a different user than the committer. | Amend the commit to use the correct email address, or amend the commit to use a GPG key associated with your user. | +| `UNVERIFIED_KEY` | The key associated with the GPG signature has no verified email address associated with the committer. | Add and verify the email to your GitLab profile, [update the GPG key to include the email address](https://security.stackexchange.com/a/261468), or amend the commit to use a different committer email address. | +| `UNKNOWN_KEY` | The GPG key associated with the GPG signature for this commit is unknown to GitLab. | [Add the GPG key](gpg.md#add-a-gpg-key-to-your-account) to your GitLab profile. | +| `MULTIPLE_SIGNATURES` | Multiple GPG or X.509 signatures have been found for the commit. | Amend the commit to use only one GPG or X.509 signature. | diff --git a/doc/user/project/repository/signed_commits/ssh.md b/doc/user/project/repository/signed_commits/ssh.md new file mode 100644 index 00000000000..3572e56da84 --- /dev/null +++ b/doc/user/project/repository/signed_commits/ssh.md @@ -0,0 +1,167 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sign commits with SSH keys **(FREE ALL)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343879) in GitLab 15.7 [with a flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. Enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/384202) in GitLab 15.8. Feature flag `ssh_commit_signatures` removed. + +When you sign commits with SSH keys, GitLab uses the SSH public keys associated +with your GitLab account to cryptographically verify the commit signature. +If successful, GitLab displays a **Verified** label on the commit. + +You may use the same SSH keys for `git+ssh` authentication to GitLab +and signing commit signatures as long as their usage type is **Authentication & Signing**. +It can be verified on the page for [adding an SSH key to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account). + +For more information about managing the SSH keys associated with your GitLab account, see +[Use SSH keys to communicate with GitLab](../../../ssh.md). + +## Configure Git to sign commits with your SSH key + +After you [create an SSH key](../../../ssh.md#generate-an-ssh-key-pair) and +[add it to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) +or [generate it using a password manager](../../../ssh.md#generate-an-ssh-key-pair-with-a-password-manager), +configure Git to begin using the key. + +Prerequisites: + +- Git 2.34.0 or newer. +- OpenSSH 8.0 or newer. + + NOTE: + OpenSSH 8.7 has broken signing functionality. If you are on OpenSSH 8.7, upgrade to OpenSSH 8.8. + +- A SSH key with the usage type of either **Authentication & Signing** or **Signing**. + The SSH key must be one of these types: + - [ED25519](../../../ssh.md#ed25519-ssh-keys) (recommended) + - [RSA](../../../ssh.md#rsa-ssh-keys) + +To configure Git to use your key: + +1. Configure Git to use SSH for commit signing: + + ```shell + git config --global gpg.format ssh + ``` + +1. Specify which SSH key should be used as the signing key, changing the filename + (here, `~/.ssh/examplekey`) to the location of your key. The filename may + differ, depending on how you generated your key: + + ```shell + git config --global user.signingkey ~/.ssh/examplekey + ``` + +## Sign commits with your SSH key + +Prerequisites: + +- You've [created an SSH key](../../../ssh.md#generate-an-ssh-key-pair). +- You've [added the key](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) to your GitLab account. +- You've [configured Git to sign commits](#configure-git-to-sign-commits-with-your-ssh-key) with your SSH key. + +To sign a commit: + +1. Use the `-S` flag when signing your commits: + + ```shell + git commit -S -m "My commit msg" + ``` + +1. Optional. If you don't want to type the `-S` flag every time you commit, tell + Git to sign your commits automatically: + + ```shell + git config --global commit.gpgsign true + ``` + +1. If your SSH key is protected, Git prompts you to enter your passphrase. +1. Push to GitLab. +1. Check that your commits [are verified](#verify-commits). + Signature verification uses the `allowed_signers` file to associate emails and SSH keys. + For help configuring this file, read [Verify commits locally](#verify-commits-locally). + +## Verify commits + +You can verify all types of signed commits +[in the GitLab UI](../signed_commits/index.md#verify-commits). Commits signed +with an SSH key can also be verified locally. + +### Verify commits locally + +To verify commits locally, create an +[allowed signers file](https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#ALLOWED_SIGNERS) +for Git to associate SSH public keys with users: + +1. Create an allowed signers file: + + ```shell + touch allowed_signers + ``` + +1. Configure the `allowed_signers` file in Git: + + ```shell + git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers" + ``` + +1. Add your entry to the allowed signers file. Use this command to add your + email address and public SSH key to the `allowed_signers` file. Replace `<MY_KEY>` + with the name of your key, and `~/.ssh/allowed_signers` + with the location of your project's `allowed_signers` file: + + ```shell + # Modify this line to meet your needs. + # Declaring the `git` namespace helps prevent cross-protocol attacks. + echo "$(git config --get user.email) namespaces=\"git\" $(cat ~/.ssh/<MY_KEY>.pub)" >> ~/.ssh/allowed_signers + ``` + + The resulting entry in the `allowed_signers` file contains your email address, key type, + and key contents, like this: + + ```plaintext + example@gitlab.com namespaces="git" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAmaTS47vRmsKyLyK1jlIFJn/i8wdGQ3J49LYyIYJ2hv + ``` + +1. Repeat the previous step for each user who you want to verify signatures for. + Consider checking this file in to your Git repository if you want to locally + verify signatures for many different contributors. + +1. Use `git log --show-signature` to view the signature status for the commits: + + ```shell + $ git log --show-signature + + commit e2406b6cd8ebe146835ceab67ff4a5a116e09154 (HEAD -> main, origin/main, origin/HEAD) + Good "git" signature for johndoe@example.com with ED25519 key SHA256:Ar44iySGgxic+U6Dph4Z9Rp+KDaix5SFGFawovZLAcc + Author: John Doe <johndoe@example.com> + Date: Tue Nov 29 06:54:15 2022 -0600 + + SSH signed commit + ``` + +## Revoke an SSH key for signing commits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108344) in GitLab 15.9. + +If an SSH key becomes compromised, revoke it. Revoking a key changes both future and past commits: + +- Past commits signed by this key are marked as unverified. +- Future commits signed by this key are marked as unverified. + +To revoke an SSH key: + +1. On the left sidebar, select your avatar. +1. Select **Edit profile**. +1. On the left sidebar, select (**{key}**) **SSH Keys**. +1. Select **Revoke** next to the SSH key you want to delete. + +## Related topics + +- [Sign commits and tags with X.509 certificates](../signed_commits/x509.md) +- [Sign commits with GPG](gpg.md) +- [Commits API](../../../../api/commits.md) diff --git a/doc/user/project/repository/signed_commits/x509.md b/doc/user/project/repository/signed_commits/x509.md new file mode 100644 index 00000000000..17767cbd8f4 --- /dev/null +++ b/doc/user/project/repository/signed_commits/x509.md @@ -0,0 +1,362 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sign commits and tags with X.509 certificates **(FREE ALL)** + +[X.509](https://en.wikipedia.org/wiki/X.509) is a standard format for public key +certificates issued by a public or private Public Key Infrastructure (PKI). +Personal X.509 certificates are used for authentication or signing purposes +such as S/MIME (Secure/Multipurpose Internet Mail Extensions). +However, Git also supports signing of commits and tags with X.509 certificates in a +similar way as with [GPG (GnuPG, or GNU Privacy Guard)](gpg.md). +The main difference is the way GitLab determines whether or not the developer's signature is trusted: + +- For X.509, a root certificate authority is added to the GitLab trust store. + (A trust store is a repository of trusted security certificates.) Combined with + any required intermediate certificates in the signature, the developer's certificate + can be chained back to a trusted root certificate. +- For GPG, developers [add their GPG key](gpg.md#add-a-gpg-key-to-your-account) + to their account. + +GitLab uses its own certificate store and therefore defines the +[trust chain](https://www.ssl.com/faqs/what-is-a-certificate-authority/). +For a commit or tag to be *verified* by GitLab: + +- The signing certificate email must match a verified email address in GitLab. +- The GitLab instance must be able to establish a full trust chain + from the certificate in the signature to a trusted certificate in the GitLab certificate store. + This chain may include intermediate certificates supplied in the signature. You may + need to add certificates, such as Certificate Authority root certificates, + [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). +- The signing time must be in the time range of the + [certificate validity](https://www.rfc-editor.org/rfc/rfc5280.html#section-4.1.2.5), + which is usually up to three years. +- The signing time is equal to, or later than, the commit time. + +If a commit's status has already been determined and stored in the database, +use the Rake task [to re-check the status](../../../../raketasks/x509_signatures.md). +Refer to the [Troubleshooting section](#troubleshooting). +GitLab checks certificate revocation lists on a daily basis with a background worker. + +## Limitations + +- Certificates without `authorityKeyIdentifier`, + `subjectKeyIdentifier`, and `crlDistributionPoints` display as **Unverified**. We + recommend using certificates from a PKI that are in line with + [RFC 5280](https://www.rfc-editor.org/rfc/rfc5280). +- If you have more than one email in the Subject Alternative Name list in + your signing certificate, + [only the first one is used to verify commits](https://gitlab.com/gitlab-org/gitlab/-/issues/336677). +- The `X509v3 Subject Key Identifier` (SKI) in the issuer certificate and the + signing certificate + [must be 40 characters long](https://gitlab.com/gitlab-org/gitlab/-/issues/332503). + If your SKI is shorter, commits don't show as verified in GitLab, and + short subject key identifiers may also + [cause errors when accessing the project](https://gitlab.com/gitlab-org/gitlab/-/issues/332464), + such as 'An error occurred while loading commit signatures' and + `HTTP 422 Unprocessable Entity` errors. + +## Configure for signed commits + +To sign your commits, tags, or both, you must: + +1. [Obtain an X.509 key pair](#obtain-an-x509-key-pair). +1. [Associate your X.509 certificate with Git](#associate-your-x509-certificate-with-git). +1. [Sign and verify commits](#sign-and-verify-commits). +1. [Sign and verify tags](#sign-and-verify-tags). + +### Obtain an X.509 key pair + +If your organization has Public Key Infrastructure (PKI), that PKI provides +an S/MIME key. If you do not have an S/MIME key pair from a PKI, you can either +create your own self-signed pair, or purchase a pair. + +### Associate your X.509 certificate with Git + +To take advantage of X.509 signing, you need Git 2.19.0 or later. You can +check your Git version with the command `git --version`. + +If you have the correct version, you can proceed to configure Git. + +### Linux + +Configure Git to use your key for signing: + +```shell +signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) +git config --global user.signingkey $signingkey +git config --global gpg.format x509 +``` + +#### Windows and macOS + +To configure Windows or macOS: + +1. Install [S/MIME Sign](https://github.com/github/smimesign) by either: + - Downloading the installer. + - Running `brew install smimesign` on macOS. +1. Get the ID of your certificate by running `smimesign --list-keys`. +1. Set your signing key by running `git config --global user.signingkey <ID>`, replacing `<ID>` with the certificate ID. +1. Configure X.509 with this command: + + ```shell + git config --global gpg.x509.program smimesign + git config --global gpg.format x509 + ``` + +### Sign and verify commits + +After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you +can sign your commits: + +1. When you create a Git commit, add the `-S` flag: + + ```shell + git commit -S -m "feat: x509 signed commits" + ``` + +1. Push to GitLab, and check that your commits are verified with the `--show-signature` flag: + + ```shell + git log --show-signature + ``` + +1. *If you don't want to type the `-S` flag every time you commit,* run this command + for Git to sign your commits every time: + + ```shell + git config --global commit.gpgsign true + ``` + +### Sign and verify tags + +After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you +can start signing your tags: + +1. When you create a Git tag, add the `-s` flag: + + ```shell + git tag -s v1.1.1 -m "My signed tag" + ``` + +1. Push to GitLab and verify your tags are signed with this command: + + ```shell + git tag --verify v1.1.1 + ``` + +1. *If you don't want to type the `-s` flag every time you tag,* run this command + for Git to sign your tags each time: + + ```shell + git config --global tag.gpgsign true + ``` + +## Related topics + +- [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) + +## Troubleshooting + +For committers without administrator access, review the list of +[verification problems with signed commits](../signed_commits/index.md#fix-verification-problems-with-signed-commits) +for possible fixes. The other troubleshooting suggestions on this page require +administrator access. + +### Re-verify commits + +GitLab stores the status of any checked commits in the database. You can use a +Rake task to [check the status of any previously checked commits](../../../../raketasks/x509_signatures.md). + +After you make any changes, run this command: + +```shell +sudo gitlab-rake gitlab:x509:update_signatures +``` + +### Main verification checks + +The code performs +[these key checks](https://gitlab.com/gitlab-org/gitlab/-/blob/v14.1.0-ee/lib/gitlab/x509/signature.rb#L33), +which all must return `verified`: + +- `x509_certificate.nil?` should be false. +- `x509_certificate.revoked?` should be false. +- `verified_signature` should be true. +- `user.nil?` should be false. +- `user.verified_emails.include?(@email)` should be true. +- `certificate_email == @email` should be true. + +To investigate why a commit shows as `Unverified`: + +1. [Start a Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session): + + ```shell + sudo gitlab-rails console + ``` + +1. Identify the project (either by path or ID) and full commit SHA that you're investigating. + Use this information to create `signature` to run other checks against: + + ```ruby + project = Project.find_by_full_path('group/subgroup/project') + project = Project.find_by_id('121') + commit = project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653') + signedcommit=Gitlab::X509::Commit.new(commit) + signature=Gitlab::X509::Signature.new(signedcommit.signature_text, signedcommit.signed_text, commit.committer_email, commit.created_at) + ``` + + If you make changes to address issues identified running through the checks, restart the + Rails console and run though the checks again from the start. + +1. Check the certificate on the commit: + + ```ruby + signature.x509_certificate.nil? + signature.x509_certificate.revoked? + ``` + + Both checks should return `false`: + + ```ruby + > signature.x509_certificate.nil? + => false + > signature.x509_certificate.revoked? + => false + ``` + + A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332503) causes + these checks to fail with `Validation failed: Subject key identifier is invalid`. + +1. Run a cryptographic check on the signature. The code must return `true`: + + ```ruby + signature.verified_signature + ``` + + If it returns `false` then [investigate this check further](#cryptographic-verification-checks). + +1. Confirm the email addresses match on the commit and the signature: + + - The Rails console displays the email addresses being compared. + - The final command must return `true`: + + ```ruby + sigemail=signature.__send__:certificate_email + commitemail=commit.committer_email + sigemail == commitemail + ``` + + A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336677) exists: + only the first email in the `Subject Alternative Name` list is compared. To + display the `Subject Alternative Name` list, run: + + ```ruby + signature.__send__ :get_certificate_extension,'subjectAltName' + ``` + + If the developer's email address is not the first one in the list, this check + fails, and the commit is marked `unverified`. + +1. The email address on the commit must be associated with an account in GitLab. + This check should return `false`: + + ```ruby + signature.user.nil? + ``` + +1. Check the email address is associated with a user in GitLab. This check should + return a user, such as `#<User id:1234 @user_handle>`: + + ```ruby + User.find_by_any_email(commit.committer_email) + ``` + + If it returns `nil`, the email address is not associated with a user, and the check fails. + +1. Confirm the developer's email address is verified. This check must return true: + + ```ruby + signature.user.verified_emails.include?(commit.committer_email) + ``` + + If the previous check returned `nil`, this command displays an error: + + ```plaintext + NoMethodError (undefined method `verified_emails' for nil:NilClass) + ``` + +1. The verification status is stored in the database. To display the database record: + + ```ruby + pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil + ``` + + If all the previous checks returned the correct values: + + - `verification_status: "unverified"` indicates the database record needs + updating. [Use the Rake task](#re-verify-commits). + + - `[]` indicates the database doesn't have a record yet. Locate the commit + in GitLab to check the signature and store the result. + +#### Cryptographic verification checks + +If GitLab determines that `verified_signature` is `false`, investigate the reason +in the Rails console. These checks require `signature` to exist. Refer to the `signature` +step of the previous [main verification checks](#main-verification-checks). + +1. Check the signature, without checking the issuer, returns `true`: + + ```ruby + signature.__send__ :valid_signature? + ``` + +1. Check the signing time and date. This check must return `true`: + + ```ruby + signature.__send__ :valid_signing_time? + ``` + + - The code allows for code signing certificates to expire. + - A commit must be signed during the validity period of the certificate, + and at or after the commit's datestamp. Display the commit time and + certificate details including `not_before`, `not_after` with: + + ```ruby + commit.created_at + pp signature.__send__ :cert; nil + ``` + +1. Check the signature, including that TLS trust can be established. This check must return `true`: + + ```ruby + signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text)) + ``` + + 1. If this fails, add the missing certificates required to establish trust + [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). + + 1. After adding more certificates, (if these troubleshooting steps then pass) + run the Rake task to [re-verify commits](#re-verify-commits). + + 1. Display the certificates, including in the signature: + + ```ruby + pp signature.__send__(:p7).certificates ; nil + ``` + +Ensure any additional intermediate certificates and the root certificate are added +to the certificate store. For consistency with how certificate chains are built on +web servers: + +- Git clients that are signing commits should include the certificate + and all intermediate certificates in the signature. +- The GitLab certificate store should only contain the root. + +If you remove a root certificate from the GitLab +trust store, such as when it expires, commit signatures which chain back to that +root display as `unverified`. diff --git a/doc/user/project/repository/ssh_signed_commits/index.md b/doc/user/project/repository/ssh_signed_commits/index.md index d8d798ac651..89e3d811dba 100644 --- a/doc/user/project/repository/ssh_signed_commits/index.md +++ b/doc/user/project/repository/ssh_signed_commits/index.md @@ -1,181 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../signed_commits/ssh.md' +remove_date: '2023-12-01' --- -# Sign commits with SSH keys **(FREE ALL)** +This document was moved to [another location](../signed_commits/ssh.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343879) in GitLab 15.7 [with a flag](../../../../administration/feature_flags.md) named `ssh_commit_signatures`. Enabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/384202) in GitLab 15.8. Feature flag `ssh_commit_signatures` removed. - -Use SSH keys to sign Git commits in the same manner as -[GPG signed commits](../gpg_signed_commits/index.md). When you sign commits -with SSH keys, GitLab uses the SSH public keys associated with your -GitLab account to cryptographically verify the commit signature. -If successful, GitLab displays a **Verified** label on the commit. - -You may use the same SSH keys for `git+ssh` authentication to GitLab -and signing commit signatures as long as their usage type is **Authentication & Signing**. -It can be verified on the page for [adding an SSH key to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account). - -For more information about managing the SSH keys associated with your GitLab account, see -[Use SSH keys to communicate with GitLab](../../../ssh.md). - -## Configure Git to sign commits with your SSH key - -After you [create an SSH key](../../../ssh.md#generate-an-ssh-key-pair) and -[add it to your GitLab account](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) -or [generate it using a password manager](../../../ssh.md#generate-an-ssh-key-pair-with-a-password-manager), -configure Git to begin using the key. - -Prerequisites: - -- Git 2.34.0 or newer. -- OpenSSH 8.0 or newer. - - NOTE: - OpenSSH 8.7 has broken signing functionality. If you are on OpenSSH 8.7, upgrade to OpenSSH 8.8. - -- A SSH key with the usage type of either **Authentication & Signing** or **Signing**. - The SSH key must be one of these types: - - [ED25519](../../../ssh.md#ed25519-ssh-keys) (recommended) - - [RSA](../../../ssh.md#rsa-ssh-keys) - -To configure Git to use your key: - -1. Configure Git to use SSH for commit signing: - - ```shell - git config --global gpg.format ssh - ``` - -1. Specify which SSH key should be used as the signing key, changing the filename - (here, `~/.ssh/examplekey`) to the location of your key. The filename may - differ, depending on how you generated your key: - - ```shell - git config --global user.signingkey ~/.ssh/examplekey - ``` - -## Sign commits with your SSH key - -Prerequisites: - -- You've [created an SSH key](../../../ssh.md#generate-an-ssh-key-pair). -- You've [added the key](../../../ssh.md#add-an-ssh-key-to-your-gitlab-account) to your GitLab account. -- You've [configured Git to sign commits](#configure-git-to-sign-commits-with-your-ssh-key) with your SSH key. - -To sign a commit: - -1. Use the `-S` flag when signing your commits: - - ```shell - git commit -S -m "My commit msg" - ``` - -1. Optional. If you don't want to type the `-S` flag every time you commit, tell - Git to sign your commits automatically: - - ```shell - git config --global commit.gpgsign true - ``` - -1. If your SSH key is protected, Git prompts you to enter your passphrase. -1. Push to GitLab. -1. Check that your commits [are verified](#verify-commits). - Signature verification uses the `allowed_signers` file to associate emails and SSH keys. - For help configuring this file, read [Verify commits locally](#verify-commits-locally). - -## Verify commits - -You can review commits for a merge request, or for an entire project, to confirm -they are signed: - -1. To review commits for a project: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. Select **Code > Commits**. -1. To review commits for a merge request: - 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. - 1. On the left sidebar, select **Merge requests**, then select your merge request. - 1. Select **Commits**. -1. Identify the commit you want to review. Signed commits show either a **Verified** - or **Unverified** badge, depending on the verification status of the signature. - Unsigned commits do not display a badge. -1. To display the signature details for a commit, select **Verified**. GitLab shows - the SSH key's fingerprint. - -## Verify commits locally - -To verify commits locally, create an -[allowed signers file](https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#ALLOWED_SIGNERS) -for Git to associate SSH public keys with users: - -1. Create an allowed signers file: - - ```shell - touch allowed_signers - ``` - -1. Configure the `allowed_signers` file in Git: - - ```shell - git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers" - ``` - -1. Add your entry to the allowed signers file. Use this command to add your - email address and public SSH key to the `allowed_signers` file. Replace `<MY_KEY>` - with the name of your key, and `~/.ssh/allowed_signers` - with the location of your project's `allowed_signers` file: - - ```shell - # Modify this line to meet your needs. - # Declaring the `git` namespace helps prevent cross-protocol attacks. - echo "$(git config --get user.email) namespaces=\"git\" $(cat ~/.ssh/<MY_KEY>.pub)" >> ~/.ssh/allowed_signers - ``` - - The resulting entry in the `allowed_signers` file contains your email address, key type, - and key contents, like this: - - ```plaintext - example@gitlab.com namespaces="git" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAmaTS47vRmsKyLyK1jlIFJn/i8wdGQ3J49LYyIYJ2hv - ``` - -1. Repeat the previous step for each user who you want to verify signatures for. - Consider checking this file in to your Git repository if you want to locally - verify signatures for many different contributors. - -1. Use `git log --show-signature` to view the signature status for the commits: - - ```shell - $ git log --show-signature - - commit e2406b6cd8ebe146835ceab67ff4a5a116e09154 (HEAD -> main, origin/main, origin/HEAD) - Good "git" signature for johndoe@example.com with ED25519 key SHA256:Ar44iySGgxic+U6Dph4Z9Rp+KDaix5SFGFawovZLAcc - Author: John Doe <johndoe@example.com> - Date: Tue Nov 29 06:54:15 2022 -0600 - - SSH signed commit - ``` - -## Revoke an SSH key for signing commits - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108344) in GitLab 15.9. - -If an SSH key becomes compromised, revoke it. Revoking a key changes both future and past commits: - -- Past commits signed by this key are marked as unverified. -- Future commits signed by this key are marked as unverified. - -To revoke an SSH key: - -1. On the left sidebar, select your avatar. -1. Select **Edit profile**. -1. On the left sidebar, select (**{key}**) **SSH Keys**. -1. Select **Revoke** next to the SSH key you want to delete. - -## Related topics - -- [Sign commits and tags with X.509 certificates](../x509_signed_commits/index.md) -- [Sign commits with GPG](../gpg_signed_commits/index.md) -- [Commits API](../../../../api/commits.md) +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/repository/tags/index.md b/doc/user/project/repository/tags/index.md index 5a01d6f2085..765f5539244 100644 --- a/doc/user/project/repository/tags/index.md +++ b/doc/user/project/repository/tags/index.md @@ -44,14 +44,14 @@ In the GitLab UI, each tag displays: To view all existing tags for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Tags**. ## View tagged commits in the commits list > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18795) in GitLab 15.10. -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Commits**. 1. Commits with a tag are labeled with a tag icon (**{tag}**) and the name of the tag. This example shows a commit tagged `v1.26.0`: @@ -88,7 +88,7 @@ To create either a lightweight or annotated tag from the command line, and push To create a tag from the GitLab UI: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Tags**. 1. Select **New tag**. 1. Provide a **Tag name**. diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 121a7b41f54..3acc62186a3 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -25,7 +25,7 @@ for any change you commit through the Web Editor. To create a text file in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New file**. @@ -35,7 +35,7 @@ To create a text file in the Web Editor: ### Create a file from a template -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Repository**. 1. Next to the project name, select the plus icon (**{plus}**) to display a dropdown list, then select **New file** from the list. @@ -56,7 +56,7 @@ To create a text file in the Web Editor: To edit a text file in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Go to your file. 1. In the upper right, select **Edit > Edit single file**. @@ -105,7 +105,7 @@ To upload a binary file in the Web Editor: <!-- This list is duplicated at doc/gitlab-basics/add-file.md#from-the-ui --> <!-- For why we duplicated the info, see https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111072#note_1267429478 --> -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **Upload file**. 1. Complete the fields. To create a merge request with the uploaded file, ensure the **Start a new merge request with these changes** toggle is turned on. @@ -115,7 +115,7 @@ To upload a binary file in the Web Editor: To create a directory in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New directory**. 1. Complete the fields. To create a merge request with the new directory, ensure the **Start a new merge request with these changes** toggle is turned on. @@ -125,7 +125,7 @@ To create a directory in the Web Editor: To create a [branch](branches/index.md) in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New branch**. 1. Complete the fields. @@ -136,7 +136,7 @@ To create a [branch](branches/index.md) in the Web Editor: You can create [tags](tags/index.md) to mark milestones such as production releases and release candidates. To create a tag in the Web Editor: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. From the project dashboard or repository, next to the branch name, select the plus icon (**{plus}**). 1. From the dropdown list, select **New tag**. 1. Complete the fields. diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index 20860718b43..ae418581820 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -1,366 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../signed_commits/x509.md' +remove_date: '2023-12-01' --- -# Sign commits and tags with X.509 certificates **(FREE ALL)** +This document was moved to [another location](../signed_commits/x509.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17773) in GitLab 12.8. - -[X.509](https://en.wikipedia.org/wiki/X.509) is a standard format for public key -certificates issued by a public or private Public Key Infrastructure (PKI). -Personal X.509 certificates are used for authentication or signing purposes -such as S/MIME (Secure/Multipurpose Internet Mail Extensions). -However, Git also supports signing of commits and tags with X.509 certificates in a -similar way as with [GPG (GnuPG, or GNU Privacy Guard)](../gpg_signed_commits/index.md). -The main difference is the way GitLab determines whether or not the developer's signature is trusted: - -- For X.509, a root certificate authority is added to the GitLab trust store. - (A trust store is a repository of trusted security certificates.) Combined with - any required intermediate certificates in the signature, the developer's certificate - can be chained back to a trusted root certificate. -- For GPG, developers [add their GPG key](../gpg_signed_commits/index.md#add-a-gpg-key-to-your-account) - to their account. - -GitLab uses its own certificate store and therefore defines the -[trust chain](https://www.ssl.com/faqs/what-is-a-certificate-authority/). -For a commit or tag to be *verified* by GitLab: - -- The signing certificate email must match a verified email address in GitLab. -- The GitLab instance must be able to establish a full trust chain - from the certificate in the signature to a trusted certificate in the GitLab certificate store. - This chain may include intermediate certificates supplied in the signature. You may - need to add certificates, such as Certificate Authority root certificates, - [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). -- The signing time must be in the time range of the - [certificate validity](https://www.rfc-editor.org/rfc/rfc5280.html#section-4.1.2.5), - which is usually up to three years. -- The signing time is equal to, or later than, the commit time. - -If a commit's status has already been determined and stored in the database, -use the Rake task [to re-check the status](../../../../raketasks/x509_signatures.md). -Refer to the [Troubleshooting section](#troubleshooting). -GitLab checks certificate revocation lists on a daily basis with a background worker. - -## Limitations - -- Self-signed certificates without `authorityKeyIdentifier`, - `subjectKeyIdentifier`, and `crlDistributionPoints` are not supported. We - recommend using certificates from a PKI that are in line with - [RFC 5280](https://www.rfc-editor.org/rfc/rfc5280). -- If you have more than one email in the Subject Alternative Name list in - your signing certificate, - [only the first one is used to verify commits](https://gitlab.com/gitlab-org/gitlab/-/issues/336677). -- The `X509v3 Subject Key Identifier` (SKI) in the issuer certificate and the - signing certificate - [must be 40 characters long](https://gitlab.com/gitlab-org/gitlab/-/issues/332503). - If your SKI is shorter, commits don't show as verified in GitLab, and - short subject key identifiers may also - [cause errors when accessing the project](https://gitlab.com/gitlab-org/gitlab/-/issues/332464), - such as 'An error occurred while loading commit signatures' and - `HTTP 422 Unprocessable Entity` errors. - -## Configure for signed commits - -To sign your commits, tags, or both, you must: - -1. [Obtain an X.509 key pair](#obtain-an-x509-key-pair). -1. [Associate your X.509 certificate with Git](#associate-your-x509-certificate-with-git). -1. [Sign and verify commits](#sign-and-verify-commits). -1. [Sign and verify tags](#sign-and-verify-tags). - -### Obtain an X.509 key pair - -If your organization has Public Key Infrastructure (PKI), that PKI provides -an S/MIME key. If you do not have an S/MIME key pair from a PKI, you can either -create your own self-signed pair, or purchase a pair. - -### Associate your X.509 certificate with Git - -To take advantage of X.509 signing, you need Git 2.19.0 or later. You can -check your Git version with the command `git --version`. - -If you have the correct version, you can proceed to configure Git. - -### Linux - -Configure Git to use your key for signing: - -```shell -signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) -git config --global user.signingkey $signingkey -git config --global gpg.format x509 -``` - -#### Windows and macOS - -To configure Windows or macOS: - -1. Install [S/MIME Sign](https://github.com/github/smimesign) by either: - - Downloading the installer. - - Running `brew install smimesign` on macOS. -1. Get the ID of your certificate by running `smimesign --list-keys`. -1. Set your signing key by running `git config --global user.signingkey <ID>`, replacing `<ID>` with the certificate ID. -1. Configure X.509 with this command: - - ```shell - git config --global gpg.x509.program smimesign - git config --global gpg.format x509 - ``` - -### Sign and verify commits - -After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you -can sign your commits: - -1. When you create a Git commit, add the `-S` flag: - - ```shell - git commit -S -m "feat: x509 signed commits" - ``` - -1. Push to GitLab, and check that your commits are verified with the `--show-signature` flag: - - ```shell - git log --show-signature - ``` - -1. *If you don't want to type the `-S` flag every time you commit,* run this command - for Git to sign your commits every time: - - ```shell - git config --global commit.gpgsign true - ``` - -### Sign and verify tags - -After you have [associated your X.509 certificate with Git](#associate-your-x509-certificate-with-git) you -can start signing your tags: - -1. When you create a Git tag, add the `-s` flag: - - ```shell - git tag -s v1.1.1 -m "My signed tag" - ``` - -1. Push to GitLab and verify your tags are signed with this command: - - ```shell - git tag --verify v1.1.1 - ``` - -1. *If you don't want to type the `-s` flag every time you tag,* run this command - for Git to sign your tags each time: - - ```shell - git config --global tag.gpgsign true - ``` - -## Related topics - -- [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) -- [Sign commits with GPG](../gpg_signed_commits/index.md) -- [Sign commits with SSH keys](../ssh_signed_commits/index.md) - -## Troubleshooting - -For committers without administrator access, review the list of -[verification problems with signed commits](../gpg_signed_commits/index.md#fix-verification-problems-with-signed-commits) -for possible fixes. The other troubleshooting suggestions on this page require -administrator access. - -### Re-verify commits - -GitLab stores the status of any checked commits in the database. You can use a -Rake task to [check the status of any previously checked commits](../../../../raketasks/x509_signatures.md). - -After you make any changes, run this command: - -```shell -sudo gitlab-rake gitlab:x509:update_signatures -``` - -### Main verification checks - -The code performs -[these key checks](https://gitlab.com/gitlab-org/gitlab/-/blob/v14.1.0-ee/lib/gitlab/x509/signature.rb#L33), -which all must return `verified`: - -- `x509_certificate.nil?` should be false. -- `x509_certificate.revoked?` should be false. -- `verified_signature` should be true. -- `user.nil?` should be false. -- `user.verified_emails.include?(@email)` should be true. -- `certificate_email == @email` should be true. - -To investigate why a commit shows as `Unverified`: - -1. [Start a Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session): - - ```shell - sudo gitlab-rails console - ``` - -1. Identify the project (either by path or ID) and full commit SHA that you're investigating. - Use this information to create `signature` to run other checks against: - - ```ruby - project = Project.find_by_full_path('group/subgroup/project') - project = Project.find_by_id('121') - commit = project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653') - signedcommit=Gitlab::X509::Commit.new(commit) - signature=Gitlab::X509::Signature.new(signedcommit.signature_text, signedcommit.signed_text, commit.committer_email, commit.created_at) - ``` - - If you make changes to address issues identified running through the checks, restart the - Rails console and run though the checks again from the start. - -1. Check the certificate on the commit: - - ```ruby - signature.x509_certificate.nil? - signature.x509_certificate.revoked? - ``` - - Both checks should return `false`: - - ```ruby - > signature.x509_certificate.nil? - => false - > signature.x509_certificate.revoked? - => false - ``` - - A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332503) causes - these checks to fail with `Validation failed: Subject key identifier is invalid`. - -1. Run a cryptographic check on the signature. The code must return `true`: - - ```ruby - signature.verified_signature - ``` - - If it returns `false` then [investigate this check further](#cryptographic-verification-checks). - -1. Confirm the email addresses match on the commit and the signature: - - - The Rails console displays the email addresses being compared. - - The final command must return `true`: - - ```ruby - sigemail=signature.__send__:certificate_email - commitemail=commit.committer_email - sigemail == commitemail - ``` - - A [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336677) exists: - only the first email in the `Subject Alternative Name` list is compared. To - display the `Subject Alternative Name` list, run: - - ```ruby - signature.__send__ :get_certificate_extension,'subjectAltName' - ``` - - If the developer's email address is not the first one in the list, this check - fails, and the commit is marked `unverified`. - -1. The email address on the commit must be associated with an account in GitLab. - This check should return `false`: - - ```ruby - signature.user.nil? - ``` - -1. Check the email address is associated with a user in GitLab. This check should - return a user, such as `#<User id:1234 @user_handle>`: - - ```ruby - User.find_by_any_email(commit.committer_email) - ``` - - If it returns `nil`, the email address is not associated with a user, and the check fails. - -1. Confirm the developer's email address is verified. This check must return true: - - ```ruby - signature.user.verified_emails.include?(commit.committer_email) - ``` - - If the previous check returned `nil`, this command displays an error: - - ```plaintext - NoMethodError (undefined method `verified_emails' for nil:NilClass) - ``` - -1. The verification status is stored in the database. To display the database record: - - ```ruby - pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil - ``` - - If all the previous checks returned the correct values: - - - `verification_status: "unverified"` indicates the database record needs - updating. [Use the Rake task](#re-verify-commits). - - - `[]` indicates the database doesn't have a record yet. Locate the commit - in GitLab to check the signature and store the result. - -#### Cryptographic verification checks - -If GitLab determines that `verified_signature` is `false`, investigate the reason -in the Rails console. These checks require `signature` to exist. Refer to the `signature` -step of the previous [main verification checks](#main-verification-checks). - -1. Check the signature, without checking the issuer, returns `true`: - - ```ruby - signature.__send__ :valid_signature? - ``` - -1. Check the signing time and date. This check must return `true`: - - ```ruby - signature.__send__ :valid_signing_time? - ``` - - - The code allows for code signing certificates to expire. - - A commit must be signed during the validity period of the certificate, - and at or after the commit's datestamp. Display the commit time and - certificate details including `not_before`, `not_after` with: - - ```ruby - commit.created_at - pp signature.__send__ :cert; nil - ``` - -1. Check the signature, including that TLS trust can be established. This check must return `true`: - - ```ruby - signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text)) - ``` - - 1. If this fails, add the missing certificates required to establish trust - [to the GitLab certificate store](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). - - 1. After adding more certificates, (if these troubleshooting steps then pass) - run the Rake task to [re-verify commits](#re-verify-commits). - - 1. Display the certificates, including in the signature: - - ```ruby - pp signature.__send__(:p7).certificates ; nil - ``` - -Ensure any additional intermediate certificates and the root certificate are added -to the certificate store. For consistency with how certificate chains are built on -web servers: - -- Git clients that are signing commits should include the certificate - and all intermediate certificates in the signature. -- The GitLab certificate store should only contain the root. - -If you remove a root certificate from the GitLab -trust store, such as when it expires, commit signatures which chain back to that -root display as `unverified`. +<!-- This redirect file can be deleted after <2023-12-01>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> |
