diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-27 12:14:36 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-27 12:14:36 +0300 |
| commit | 7596630be2bb8febb34e782817c8339ba6ec7b2c (patch) | |
| tree | e2cbab17c8792538df117050b09ac0a587f1f8b7 /doc | |
| parent | 0878b072b015ae4db00f4824e2f5444d66d5ae5c (diff) | |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
24 files changed, 714 insertions, 681 deletions
diff --git a/doc/administration/backup_restore/restore_gitlab.md b/doc/administration/backup_restore/restore_gitlab.md index 2dd85602f99..51ada659acd 100644 --- a/doc/administration/backup_restore/restore_gitlab.md +++ b/doc/administration/backup_restore/restore_gitlab.md @@ -282,7 +282,8 @@ project or group from there: the backed-up instance from which you want to restore. 1. [Restore the backup](#restore-gitlab) into this new instance, then export your [project](../../user/project/settings/import_export.md) - or [group](../../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). For more information about what is and isn't exported, see the export feature's documentation. + or [group](../../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated). For + more information about what is and isn't exported, see the export feature's documentation. 1. After the export is complete, go to the old instance and then import it. 1. After importing the projects or groups that you wanted is complete, you may delete the new, temporary GitLab instance. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 336ed743a47..f5a50c6e9a3 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -150,7 +150,7 @@ Backups of GitLab databases and file systems are taken every 24 hours, and are k - You can use the project export option in: - [The UI](../user/project/settings/import_export.md#export-a-project-and-its-data). - [The API](../api/project_import_export.md#schedule-an-export). -- [Group export by uploading a file export](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) +- [Group export by uploading a file export](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) does **not** export the projects in it, but does export: - Epics - Milestones diff --git a/doc/administration/settings/import_and_export_settings.md b/doc/administration/settings/import_and_export_settings.md index ddb31e49483..54995cdc686 100644 --- a/doc/administration/settings/import_and_export_settings.md +++ b/doc/administration/settings/import_and_export_settings.md @@ -121,7 +121,7 @@ To modify this setting: > - **Maximum decompressed file size for archives from imports** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130081) from **Maximum decompressed size** in GitLab 16.4. When you import a project using [file exports](../../user/project/settings/import_export.md) or -[direct transfer](../../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended), you can specify the +[direct transfer](../../user/group/import/index.md), you can specify the maximum decompressed file size for imported archives. The default value is 25 GiB. When you import a compressed file, the decompressed size cannot exceed the maximum decompressed file size limit. If the diff --git a/doc/api/bulk_imports.md b/doc/api/bulk_imports.md index 1d7556f863a..bebfdb80a35 100644 --- a/doc/api/bulk_imports.md +++ b/doc/api/bulk_imports.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Project migration [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/390515) in GitLab 15.11. With the group migration by direct transfer API, you can start and view the progress of migrations initiated with -[group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended). +[group migration by direct transfer](../user/group/import/index.md). WARNING: Migrating projects with this API is in [Beta](../policy/experiment-beta-support.md#beta). This feature is not diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index 7a3b031f18a..7e065dd87d2 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -32,7 +32,7 @@ To preserve the member list and their respective permissions on imported groups, ## Prerequisites - For information on prerequisites for group import and export API, see prerequisites for - [migrating groups by uploading an export file](../user/group/import/index.md#preparation). + [migrating groups by uploading an export file](../user/project/settings/import_export.md#preparation). ## Schedule new export diff --git a/doc/api/group_relations_export.md b/doc/api/group_relations_export.md index e1c0cd81bd6..bd48fb13ec6 100644 --- a/doc/api/group_relations_export.md +++ b/doc/api/group_relations_export.md @@ -13,7 +13,7 @@ top-level relation (for example, milestones, boards, and labels). The group relations export API is primarily used in -[group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) +[group migration by direct transfer](../user/group/import/index.md) and can't be used with the [group import and export API](group_import_export.md). diff --git a/doc/api/project_relations_export.md b/doc/api/project_relations_export.md index 5fe3923edc8..a586c940b8e 100644 --- a/doc/api/project_relations_export.md +++ b/doc/api/project_relations_export.md @@ -16,7 +16,7 @@ top-level relation (for example, milestones, issues, and labels). The project relations export API is primarily used in -[group migration](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) can't +[group migration](../user/group/import/index.md) can't be used with the [project import and export API](project_import_export.md). diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 5c64abcff81..2761658a719 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -1910,8 +1910,8 @@ to select a specific site profile and scanner profile. **Related topics**: -- [Site profile](../../user/application_security/dast/proxy-based.md#site-profile). -- [Scanner profile](../../user/application_security/dast/proxy-based.md#scanner-profile). +- [Site profile](../../user/application_security/dast/on-demand_scan.md#site-profile). +- [Scanner profile](../../user/application_security/dast/on-demand_scan.md#scanner-profile). ### `dependencies` diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 6c0bed8e204..752bcbbac19 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -12,7 +12,7 @@ NOTE: To use direct transfer, ensure your GitLab installation is accessible from [GitLab IP addresses](../user/gitlab_com/index.md#ip-range) and has a public DNS entry. -[Group migration by direct transfer](../user/group/import/index.md#migrate-groups-by-direct-transfer-recommended) is the +[Group migration by direct transfer](../user/group/import/index.md) is the evolution of migrating groups and projects using file exports. The goal is to have an easier way for the user to migrate a whole group, including projects, from one GitLab instance to another. @@ -36,8 +36,8 @@ idea is to have one ETL pipeline for each relation to be imported. ### API -The current [project](../user/project/settings/import_export.md) and -[group](../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so +The current [project](../user/project/settings/import_export.md#migrate-projects-by-uploading-an-export-file) and +[group](../user/project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) imports are file based, so they require an export step to generate the file to be imported. Group migration by direct transfer leverages the [GitLab API](../api/rest/index.md) to speed the migration. diff --git a/doc/tutorials/gitlab_navigation.md b/doc/tutorials/gitlab_navigation.md index 738518e1aa7..8b91d799f91 100644 --- a/doc/tutorials/gitlab_navigation.md +++ b/doc/tutorials/gitlab_navigation.md @@ -11,7 +11,7 @@ and running quickly. | Topic | Description | Good for beginners | |-------|-------------|--------------------| -| [GitLab with Git Essentials](https://levelup.gitlab.com/courses/gitlab-with-git-essentials) | Learn the basics of Git and GitLab in this self-paced course. | **{star}** | +| [GitLab with Git Essentials](https://levelup.gitlab.com/courses/gitlab-with-git-essentials-s2) | Learn the basics of Git and GitLab in this self-paced course. | **{star}** | | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Use GitLab for DevOps](https://www.youtube.com/watch?v=7q9Y1Cv-ib0) (12m 34s) | Use GitLab through the entire DevOps lifecycle, from planning to monitoring. | **{star}** | | [Use the left sidebar to navigate GitLab](left_sidebar/index.md) | Start navigating the GitLab UI. | **{star}** | | [Use Markdown at GitLab](../user/markdown.md) | GitLab Flavored Markdown (GLFM) is used in many areas of GitLab, for example, in merge requests. | **{star}** | diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 008b5b54cca..c367d647c6c 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -50,7 +50,7 @@ You can configure the following security controls: - [Dynamic Application Security Testing](../dast/index.md) (DAST) - Select **Enable DAST** to configure DAST for the current project. - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. - For more details, read [DAST on-demand scans](../dast/proxy-based.md#on-demand-scans). + For more details, read [DAST on-demand scans](../dast/on-demand_scan.md). - [Dependency Scanning](../dependency_scanning/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to enable Dependency Scanning. For more information, see [Use a preconfigured merge request](../dependency_scanning/index.md#use-a-preconfigured-merge-request). diff --git a/doc/user/application_security/dast/on-demand_scan.md b/doc/user/application_security/dast/on-demand_scan.md new file mode 100644 index 00000000000..e43057aea54 --- /dev/null +++ b/doc/user/application_security/dast/on-demand_scan.md @@ -0,0 +1,409 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# DAST On Demand Scan **(ULTIMATE ALL)** + +WARNING: +Do not run DAST scans against a production server. Not only can it perform *any* function that a user can, such +as clicking buttons or submitting forms, but it may also trigger bugs, leading to modification or loss of production data. +Only run DAST scans against a test server. + +## On-demand scans + +> - Auditing for DAST profile management [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. +> - Scheduled on-demand DAST scans [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3 [with a flag](../../../administration/feature_flags.md) named `dast_on_demand_scans_scheduler`. Disabled by default. +> - Scheduled on-demand DAST scans [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. Feature flag `dast_on_demand_scans_scheduler` removed. +> - Runner tags selection [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111499) in GitLab 16.3. +> - Browser based on-demand DAST scans [deployed behind the feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430212) in GitLab 16.8. + +An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger +the scan. You must either start it manually, or schedule it to run. For on-demand DAST scans, +a [site profile](#site-profile) defines **what** is to be scanned, and a +[scanner profile](#scanner-profile) defines **how** the application is to be scanned. + +An on-demand scan can be run in active or passive mode: + +- **Passive mode**: The default mode, which runs a [Passive Browser based scan](/ee/user/application_security/dast/browser_based.md#passive-scans). +- **Active mode**: Runs an [Active Browser based scan](/ee/user/application_security/dast/browser_based.md#active-scans) which is potentially harmful to the site being scanned. To + minimize the risk of accidental damage, running an active scan requires a + [validated site profile](#site-profile-validation). + +### View on-demand DAST scans + +To view on-demand scans: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Secure > On-demand scans**. + +On-demand scans are grouped by their status. The scan library contains all available on-demand +scans. + +### Run an on-demand DAST scan + +Prerequisites: + +- You must have permission to run an on-demand DAST scan against a protected branch. The default + branch is automatically protected. For more information, see + [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). + +To run an existing on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the scan's row, select **Run scan**. + + If the branch saved in the scan no longer exists, you must: + + 1. [Edit the scan](#edit-an-on-demand-scan). + 1. Select a new branch. + 1. Save the edited scan. + +The on-demand DAST scan runs, and the project's dashboard shows the results. + +#### Create an on-demand scan + +Create an on-demand scan to: + +- Run it immediately. +- Save it to be run in the future. +- Schedule it to be run at a specified schedule. + +To create an on-demand DAST scan: + +1. On the left sidebar, select **Search or go to** and find your project or group. +1. Select **Secure > On-demand scans**. +1. Select **New scan**. +1. Complete the **Scan name** and **Description** fields. +1. In the **Branch** dropdown list, select the desired branch. +1. Optional. Select the runner tags. +1. Select **Select scanner profile** or **Change scanner profile** to open the drawer, and either: + - Select a scanner profile from the drawer, **or** + - Select **New profile**, create a [scanner profile](#scanner-profile), then select **Save profile**. +1. Select **Select site profile** or **Change site profile** to open the drawer, and either: + - Select a site profile from the **Site profile library** drawer, or + - Select **New profile**, create a [site profile](#site-profile), then select **Save profile**. +1. To run the on-demand scan: + + - Immediately, select **Save and run scan**. + - In the future, select **Save scan**. + - On a schedule: + + - Turn on the **Enable scan schedule** toggle. + - Complete the schedule fields. + - Select **Save scan**. + +The on-demand DAST scan runs as specified and the project's dashboard shows the results. + +### View details of an on-demand scan + +To view details of an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. + +### Edit an on-demand scan + +To edit an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. +1. Edit the saved scan's details. +1. Select **Save scan**. + +### Delete an on-demand scan + +To delete an on-demand scan: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > On-demand scans**. +1. Select the **Scan library** tab. +1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. +1. On the confirmation dialog, select **Delete**. + +## Site profile + +> - Site profile features, scan method and file URL, were [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. +> - GraphQL endpoint path feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378692) in GitLab 15.7. + +A site profile defines the attributes and configuration details of the deployed application, +website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and +on-demand scans. + +A site profile contains: + +- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced + in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Site type**: The type of target to be scanned, either website or API scan. +- **Target URL**: The URL that DAST runs against. +- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. +- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. +- **Authentication**: + - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. + - **Username**: The username used to authenticate to the website. + - **Password**: The password used to authenticate to the website. + - **Username form field**: The name of username field at the sign-in HTML form. + - **Password form field**: The name of password field at the sign-in HTML form. + - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. + +- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, HTTP Archive (HAR), or GraphQL. + - **GraphQL endpoint path**: The path to the GraphQL endpoint. This path is concatenated with the target URL to provide the URI for the scan to test. The GraphQL endpoint must support introspection queries. + - **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. + +When an API site type is selected, a host override is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. + +When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. +This data can only be read and decrypted with a valid secrets file. + +### Site profile validation + +> Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. + +Site profile validation reduces the risk of running an active scan against the wrong website. A site +must be validated before an active scan can run against it. Each of the site validation methods are +equivalent in functionality, so use whichever is most suitable: + +- **Text file validation**: Requires a text file be uploaded to the target site. The text file is + allocated a name and content that is unique to the project. The validation process checks the + file's content. +- **Header validation**: Requires the header `Gitlab-On-Demand-DAST` be added to the target site, + with a value unique to the project. The validation process checks that the header is present, and + checks its value. +- **Meta tag validation**: Requires the meta tag named `gitlab-dast-validation` be added to the + target site, with a value unique to the project. Make sure it's added to the `<head>` section of + the page. The validation process checks that the meta tag is present, and checks its value. + +### Create a site profile + +To create a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select **New > Site profile**. +1. Complete the fields then select **Save profile**. + +The site profile is saved, for use in an on-demand scan. + +### Edit a site profile + +NOTE: +If a site profile is linked to a security policy, you cannot edit the profile from this page. See +[Scan execution policies](../policies/scan-execution-policies.md) for more information. + +NOTE: +If a site profile's Target URL or Authenticated URL is updated, the request headers and password fields associated with that profile are cleared. + +When a validated site profile's file, header, or meta tag is edited, the site's +[validation status](#site-profile-validation) is revoked. + +To edit a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the fields then select **Save profile**. + +### Delete a site profile + +NOTE: +If a site profile is linked to a security policy, a user cannot delete the profile from this page. +See [Scan execution policies](../policies/scan-execution-policies.md) for more information. + +To delete a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete** to confirm the deletion. + +### Validate a site profile + +Validating a site is required to run an active scan. + +Prerequisites: + +- A runner must be available in the project to run a validation job. +- The GitLab server's certificate must be trusted and must not use a self-signed certificate. + +To validate a site profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Validate**. +1. Select the validation method. + 1. For **Text file validation**: + 1. Download the validation file listed in **Step 2**. + 1. Upload the validation file to the host, to the location in **Step 3** or any location you + prefer. + 1. If required, edit the file location in **Step 3**. + 1. Select **Validate**. + 1. For **Header validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the header of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the header. + 1. Select **Validate**. + 1. For **Meta tag validation**: + 1. Select the clipboard icon in **Step 2**. + 1. Edit the content of the site to validate, and paste the clipboard content. + 1. Select the input field in **Step 3** and enter the location of the meta tag. + 1. Select **Validate**. + +The site is validated and an active scan can run against it. A site profile's validation status is +revoked only when it's revoked manually, or its file, header, or meta tag is edited. + +### Retry a failed validation + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. +> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. +> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. + +Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** +page. + +To retry a site profile's failed validation: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Site Profiles** tab. +1. In the profile's row, select **Retry validation**. + +### Revoke a site profile's validation status + +WARNING: +When a site profile's validation status is revoked, all site profiles that share the same URL also +have their validation status revoked. + +To revoke a site profile's validation status: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Beside the validated profile, select **Revoke validation**. + +The site profile's validation status is revoked. + +### Validated site profile headers + +The following are code samples of how you can provide the required site profile header in your +application. + +#### Ruby on Rails example for on-demand scan + +Here's how you can add a custom header in a Ruby on Rails application: + +```ruby +class DastWebsiteTargetController < ActionController::Base + def dast_website_target + response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + head :ok + end +end +``` + +#### Django example for on-demand scan + +Here's how you can add a +[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): + +```python +class DastWebsiteTargetView(View): + def head(self, *args, **kwargs): + response = HttpResponse() + response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' + + return response +``` + +#### Node (with Express) example for on-demand scan + +Here's how you can add a +[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): + +```javascript +app.get('/dast-website-target', function(req, res) { + res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') + res.send('Respond to DAST ping') +}) +``` + +## Scanner profile + +> - Deprecated AJAX Spider option with the [introduction of Browser based on-demand DAST scans behind feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430210). +> - Renamed spider timeout to crawl timeout with the [introduction of Browser based on-demand DAST scans behind feature flag `dast_ods_browser_based_scanner`](https://gitlab.com/gitlab-org/gitlab/-/issues/430210). + +A scanner profile defines the configuration details of a security scanner. A scanner profile can be +referenced in `.gitlab-ci.yml` and on-demand scans. + +A scanner profile contains: + +- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner + profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. +- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. +- **Crawl timeout:** The maximum number of minutes allowed for the crawler to traverse the site. +- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before + starting the scan. +- **Debug messages:** Include debug messages in the DAST console output. + +### Create a scanner profile + +To create a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select **New > Scanner profile**. +1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). +1. Select **Save profile**. + +### Edit a scanner profile + +NOTE: +If a scanner profile is linked to a security policy, you cannot edit the profile from this page. +For more information, see [Scan execution policies](../policies/scan-execution-policies.md). + +To edit a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Scanner profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. +1. Edit the form. +1. Select **Save profile**. + +### Delete a scanner profile + +NOTE: +If a scanner profile is linked to a security policy, a user cannot delete the profile from this +page. For more information, see [Scan execution policies](../policies/scan-execution-policies.md). + +To delete a scanner profile: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Secure > Security configuration**. +1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. +1. Select the **Scanner profiles** tab. +1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. +1. Select **Delete**. + +## Auditing + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. + +The creation, updating, and deletion of DAST profiles, DAST scanner profiles, +and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). diff --git a/doc/user/application_security/dast/proxy-based.md b/doc/user/application_security/dast/proxy-based.md index 6127866b0a9..447babb0ad4 100644 --- a/doc/user/application_security/dast/proxy-based.md +++ b/doc/user/application_security/dast/proxy-based.md @@ -168,9 +168,9 @@ To configure DAST using the UI: 1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or **Configure DAST**. 1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a - scanner profile. For more details, see [scanner profiles](#scanner-profile). + scanner profile. For more details, see [scanner profiles](../dast/on-demand_scan.md#scanner-profile). 1. Select the desired **Site profile**, or select **Create site profile** and save a site - profile. For more details, see [site profiles](#site-profile). + profile. For more details, see [site profiles](../dast/on-demand_scan.md#site-profile). 1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the options you selected. 1. Do one of the following: @@ -466,403 +466,6 @@ variables: The DAST job does not require the project's repository to be present when running, so by default [`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. -## On-demand scans - -> - Auditing for DAST profile management [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. -> - Scheduled on-demand DAST scans [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3 [with a flag](../../../administration/feature_flags.md) named `dast_on_demand_scans_scheduler`. Disabled by default. -> - Scheduled on-demand DAST scans [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. Feature flag `dast_on_demand_scans_scheduler` removed. -> - Runner tags selection [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111499) in GitLab 16.3. - -WARNING: -On-demand scans are not available when GitLab is running in FIPS mode. - -An on-demand DAST scan runs outside the DevOps life cycle. Changes in your repository don't trigger -the scan. You must either start it manually, or schedule it to run. For on-demand DAST scans, -a [site profile](#site-profile) defines **what** is to be scanned, and a -[scanner profile](#scanner-profile) defines **how** the application is to be scanned. - -An on-demand scan can be run in active or passive mode: - -- **Passive mode**: The default mode, which runs a ZAP Baseline Scan. -- **Active mode**: Runs a ZAP Full Scan which is potentially harmful to the site being scanned. To - minimize the risk of accidental damage, running an active scan requires a - [validated site profile](#site-profile-validation). - -### View on-demand DAST scans - -To view on-demand scans: - -1. On the left sidebar, select **Search or go to** and find your project or group. -1. Select **Secure > On-demand scans**. - -On-demand scans are grouped by their status. The scan library contains all available on-demand -scans. - -### Run an on-demand DAST scan - -Prerequisites: - -- You must have permission to run an on-demand DAST scan against a protected branch. The default - branch is automatically protected. For more information, see - [Pipeline security on protected branches](../../../ci/pipelines/index.md#pipeline-security-on-protected-branches). - -To run an existing on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the scan's row, select **Run scan**. - - If the branch saved in the scan no longer exists, you must: - - 1. [Edit the scan](#edit-an-on-demand-scan). - 1. Select a new branch. - 1. Save the edited scan. - -The on-demand DAST scan runs, and the project's dashboard shows the results. - -#### Create an on-demand scan - -Create an on-demand scan to: - -- Run it immediately. -- Save it to be run in the future. -- Schedule it to be run at a specified schedule. - -To create an on-demand DAST scan: - -1. On the left sidebar, select **Search or go to** and find your project or group. -1. Select **Secure > On-demand scans**. -1. Select **New scan**. -1. Complete the **Scan name** and **Description** fields. -1. In the **Branch** dropdown list, select the desired branch. -1. Optional. Select the runner tags. -1. Select **Select scanner profile** or **Change scanner profile** to open the drawer, and either: - - Select a scanner profile from the drawer, **or** - - Select **New profile**, create a [scanner profile](#scanner-profile), then select **Save profile**. -1. Select **Select site profile** or **Change site profile** to open the drawer, and either: - - Select a site profile from the **Site profile library** drawer, or - - Select **New profile**, create a [site profile](#site-profile), then select **Save profile**. -1. To run the on-demand scan: - - - Immediately, select **Save and run scan**. - - In the future, select **Save scan**. - - On a schedule: - - - Turn on the **Enable scan schedule** toggle. - - Complete the schedule fields. - - Select **Save scan**. - -The on-demand DAST scan runs as specified and the project's dashboard shows the results. - -### View details of an on-demand scan - -To view details of an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. - -### Edit an on-demand scan - -To edit an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Edit**. -1. Edit the saved scan's details. -1. Select **Save scan**. - -### Delete an on-demand scan - -To delete an on-demand scan: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > On-demand scans**. -1. Select the **Scan library** tab. -1. In the saved scan's row select **More actions** (**{ellipsis_v}**), then select **Delete**. -1. On the confirmation dialog, select **Delete**. - -## Site profile - -> - Site profile features, scan method and file URL, were [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345837) in GitLab 15.6. -> - GraphQL endpoint path feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/378692) in GitLab 15.7. - -A site profile defines the attributes and configuration details of the deployed application, -website, or API to be scanned by DAST. A site profile can be referenced in `.gitlab-ci.yml` and -on-demand scans. - -A site profile contains: - -- **Profile name**: A name you assign to the site to be scanned. While a site profile is referenced - in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Site type**: The type of target to be scanned, either website or API scan. -- **Target URL**: The URL that DAST runs against. -- **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. -- **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. -- **Authentication**: - - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. - - **Username**: The username used to authenticate to the website. - - **Password**: The password used to authenticate to the website. - - **Username form field**: The name of username field at the sign-in HTML form. - - **Password form field**: The name of password field at the sign-in HTML form. - - **Submit form field**: The `id` or `name` of the element that when selected submits the sign-in HTML form. - -- **Scan method**: A type of method to perform API testing. The supported methods are OpenAPI, Postman Collections, HTTP Archive (HAR), or GraphQL. - - **GraphQL endpoint path**: The path to the GraphQL endpoint. This path is concatenated with the target URL to provide the URI for the scan to test. The GraphQL endpoint must support introspection queries. - - **File URL**: The URL of the OpenAPI, Postman Collection, or HTTP Archive file. - -When an API site type is selected, a host override is used to ensure the API being scanned is on the same host as the target. This is done to reduce the risk of running an active scan against the wrong API. - -When configured, request headers and password fields are encrypted using [`aes-256-gcm`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) before being stored in the database. -This data can only be read and decrypted with a valid secrets file. - -### Site profile validation - -> Meta tag validation [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6460) in GitLab 14.2. - -Site profile validation reduces the risk of running an active scan against the wrong website. A site -must be validated before an active scan can run against it. Each of the site validation methods are -equivalent in functionality, so use whichever is most suitable: - -- **Text file validation**: Requires a text file be uploaded to the target site. The text file is - allocated a name and content that is unique to the project. The validation process checks the - file's content. -- **Header validation**: Requires the header `Gitlab-On-Demand-DAST` be added to the target site, - with a value unique to the project. The validation process checks that the header is present, and - checks its value. -- **Meta tag validation**: Requires the meta tag named `gitlab-dast-validation` be added to the - target site, with a value unique to the project. Make sure it's added to the `<head>` section of - the page. The validation process checks that the meta tag is present, and checks its value. - -### Create a site profile - -To create a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select **New > Site profile**. -1. Complete the fields then select **Save profile**. - -The site profile is saved, for use in an on-demand scan. - -### Edit a site profile - -NOTE: -If a site profile is linked to a security policy, you cannot edit the profile from this page. See -[Scan execution policies](../policies/scan-execution-policies.md) for more information. - -NOTE: -If a site profile's Target URL or Authenticated URL is updated, the request headers and password fields associated with that profile are cleared. - -When a validated site profile's file, header, or meta tag is edited, the site's -[validation status](#site-profile-validation) is revoked. - -To edit a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the fields then select **Save profile**. - -### Delete a site profile - -NOTE: -If a site profile is linked to a security policy, a user cannot delete the profile from this page. -See [Scan execution policies](../policies/scan-execution-policies.md) for more information. - -To delete a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete** to confirm the deletion. - -### Validate a site profile - -Validating a site is required to run an active scan. - -Prerequisites: - -- A runner must be available in the project to run a validation job. -- The GitLab server's certificate must be trusted and must not use a self-signed certificate. - -To validate a site profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Validate**. -1. Select the validation method. - 1. For **Text file validation**: - 1. Download the validation file listed in **Step 2**. - 1. Upload the validation file to the host, to the location in **Step 3** or any location you - prefer. - 1. If required, edit the file location in **Step 3**. - 1. Select **Validate**. - 1. For **Header validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the header of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the header. - 1. Select **Validate**. - 1. For **Meta tag validation**: - 1. Select the clipboard icon in **Step 2**. - 1. Edit the content of the site to validate, and paste the clipboard content. - 1. Select the input field in **Step 3** and enter the location of the meta tag. - 1. Select **Validate**. - -The site is validated and an active scan can run against it. A site profile's validation status is -revoked only when it's revoked manually, or its file, header, or meta tag is edited. - -### Retry a failed validation - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. -> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. -> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. - -Failed site validation attempts are listed on the **Site profiles** tab of the **Manage profiles** -page. - -To retry a site profile's failed validation: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Site Profiles** tab. -1. In the profile's row, select **Retry validation**. - -### Revoke a site profile's validation status - -WARNING: -When a site profile's validation status is revoked, all site profiles that share the same URL also -have their validation status revoked. - -To revoke a site profile's validation status: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Beside the validated profile, select **Revoke validation**. - -The site profile's validation status is revoked. - -### Validated site profile headers - -The following are code samples of how you can provide the required site profile header in your -application. - -#### Ruby on Rails example for on-demand scan - -Here's how you can add a custom header in a Ruby on Rails application: - -```ruby -class DastWebsiteTargetController < ActionController::Base - def dast_website_target - response.headers['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - head :ok - end -end -``` - -#### Django example for on-demand scan - -Here's how you can add a -[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): - -```python -class DastWebsiteTargetView(View): - def head(self, *args, **kwargs): - response = HttpResponse() - response['Gitlab-On-Demand-DAST'] = '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c' - - return response -``` - -#### Node (with Express) example for on-demand scan - -Here's how you can add a -[custom header in Node (with Express)](https://expressjs.com/en/5x/api.html#res.append): - -```javascript -app.get('/dast-website-target', function(req, res) { - res.append('Gitlab-On-Demand-DAST', '0dd79c9a-7b29-4e26-a815-eaaf53fcab1c') - res.send('Respond to DAST ping') -}) -``` - -## Scanner profile - -A scanner profile defines the configuration details of a security scanner. A scanner profile can be -referenced in `.gitlab-ci.yml` and on-demand scans. - -A scanner profile contains: - -- **Profile name:** A name you give the scanner profile. For example, "Spider_15". While a scanner - profile is referenced in either `.gitlab-ci.yml` or an on-demand scan, it **cannot** be renamed. -- **Scan mode:** A passive scan monitors all HTTP messages (requests and responses) sent to the target. An active scan attacks the target to find potential vulnerabilities. -- **Spider timeout:** The maximum number of minutes allowed for the spider to traverse the site. -- **Target timeout:** The maximum number of seconds DAST waits for the site to be available before - starting the scan. -- **AJAX spider:** Run the AJAX spider, in addition to the traditional spider, to crawl the target site. -- **Debug messages:** Include debug messages in the DAST console output. - -### Create a scanner profile - -To create a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select **New > Scanner profile**. -1. Complete the form. For details of each field, see [Scanner profile](#scanner-profile). -1. Select **Save profile**. - -### Edit a scanner profile - -NOTE: -If a scanner profile is linked to a security policy, you cannot edit the profile from this page. -For more information, see [Scan execution policies](../policies/scan-execution-policies.md). - -To edit a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Scanner profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Edit**. -1. Edit the form. -1. Select **Save profile**. - -### Delete a scanner profile - -NOTE: -If a scanner profile is linked to a security policy, a user cannot delete the profile from this -page. For more information, see [Scan execution policies](../policies/scan-execution-policies.md). - -To delete a scanner profile: - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Secure > Security configuration**. -1. In the **Dynamic Application Security Testing (DAST)** section, select **Manage profiles**. -1. Select the **Scanner profiles** tab. -1. In the scanner's row, select the **More actions** (**{ellipsis_v}**) menu, then select **Delete**. -1. Select **Delete**. - -## Auditing - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. - -The creation, updating, and deletion of DAST profiles, DAST scanner profiles, -and DAST site profiles are included in the [audit log](../../../administration/audit_events.md). - ## Reports The DAST tool outputs a `gl-dast-report.json` report file containing details of the scan and its results. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index df8fbff720b..acc6d30acb1 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Perform Dynamic Application Security Testing (DAST) of web APIs to help discover bugs and potential security issues that other QA processes may miss. Use DAST API tests in addition to other [GitLab Secure](../index.md) security scanners and your own test processes. You can run DAST -API tests either as part your CI/CD workflow, [on-demand](../dast/proxy-based.md#on-demand-scans), or both. +API tests either as part your CI/CD workflow, [on-demand](../dast/on-demand_scan.md), or both. WARNING: Do not run DAST API testing against a production server. Not only can it perform _any_ function that diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 9a2053d43dd..1ac6162bd49 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -214,8 +214,8 @@ rule in the defined policy are met. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| | `scan` | `string` | `sast`, `sast_iac`, `dast`, `secret_detection`, `container_scanning`, `dependency_scanning`, `custom` | The action's type. | -| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/proxy-based.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | -| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/proxy-based.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| +| `site_profile` | `string` | Name of the selected [DAST site profile](../dast/on-demand_scan.md#site-profile). | The DAST site profile to execute the DAST scan. This field should only be set if `scan` type is `dast`. | +| `scanner_profile` | `string` or `null` | Name of the selected [DAST scanner profile](../dast/on-demand_scan.md#scanner-profile). | The DAST scanner profile to execute the DAST scan. This field should only be set if `scan` type is `dast`.| | `variables` | `object` | | A set of CI variables, supplied as an array of `key: value` pairs, to apply and enforce for the selected scan. The `key` is the variable name, with its `value` provided as a string. This parameter supports any variable that the GitLab CI job supports for the specified scan. | | `tags` | `array` of `string` | | A list of runner tags for the policy. The policy jobs are run by runner with the specified tags. | | `ci_configuration` <sup>1</sup> | `string` | | GitLab CI YAML as formatted as string. | @@ -225,7 +225,7 @@ rule in the defined policy are met. Note the following: -- You must create the [site profile](../dast/proxy-based.md#site-profile) and [scanner profile](../dast/proxy-based.md#scanner-profile) +- You must create the [site profile](../dast/on-demand_scan.md#site-profile) and [scanner profile](../dast/on-demand_scan.md#scanner-profile) with selected names for each project that is assigned to the selected Security Policy Project. Otherwise, the policy is not applied and a job with an error message is created instead. - Once you associate the site profile and scanner profile by name in the policy, it is not possible diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index a7b75534800..75d0c8fdd12 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -245,7 +245,7 @@ which GitLab you use: | [Bitbucket Server](../project/import/bitbucket_server.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [FogBugz](../project/import/fogbugz.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [Gitea](../project/import/gitea.md) | **{check-circle}** Yes | **{dotted-circle}** No | -| [GitLab by direct transfer](../group/import/index.md#migrate-groups-by-direct-transfer-recommended) | **{check-circle}** Yes | **{dotted-circle}** No | +| [GitLab by direct transfer](../group/import/index.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [GitLab using file exports](../project/settings/import_export.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [GitHub](../project/import/github.md) | **{check-circle}** Yes | **{dotted-circle}** No | | [Manifest file](../project/import/manifest.md) | **{check-circle}** Yes | **{dotted-circle}** No | @@ -454,7 +454,7 @@ To help avoid abuse, the following are rate limited: For more information, see: - [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits). -- [Group import/export rate limits](../../user/group/import/index.md#rate-limits). +- [Group import/export rate limits](../../user/project/settings/import_export.md#rate-limits-1). ### Non-configurable limits diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 28878855098..dfc5332921d 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -4,7 +4,14 @@ group: Import and Integrate info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Migrating GitLab groups **(FREE ALL)** +# Migrate GitLab groups and projects by using direct transfer **(FREE ALL)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. +> - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 for project resources [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. +> - New application setting `bulk_import_enabled` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. `bulk_import` feature flag removed. +> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. You can migrate GitLab groups: @@ -16,19 +23,10 @@ You can migrate GitLab groups: You can migrate groups in two ways: - By direct transfer (recommended). -- By uploading an export file. +- By [uploading an export file](../../project/settings/import_export.md). If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance. -## Migrate groups by direct transfer (recommended) - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 for group resources [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. -> - Group items [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 for project resources [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. -> - New application setting `bulk_import_enabled` [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. `bulk_import` feature flag removed. -> - `bulk_import_projects` feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.10. - On self-managed GitLab, by default [migrating group items](#migrated-group-items) is not available. To show the feature, an administrator can [enable it in application settings](../../../administration/settings/import_and_export_settings.md#enable-migration-of-groups-and-projects-by-direct-transfer). @@ -59,12 +57,12 @@ We invite you to leave your feedback about migrating by direct transfer in If you want to move groups instead of copying groups, you can [transfer groups](../manage.md#transfer-a-group) if the groups are in the same GitLab instance. Transferring groups is a faster and more complete option. -### Known issues +## Known issues See [epic 6629](https://gitlab.com/groups/gitlab-org/-/epics/6629) for a list of known issues for migrating by direct transfer. -### Estimating migration duration +## Estimating migration duration Estimating the duration of migration by direct transfer is difficult. The following factors affect migration duration: @@ -111,7 +109,7 @@ There's no exact formula to reliably estimate a migration. However, the average If you are migrating large projects and encounter problems with timeouts or duration of the migration, see [Reducing migration duration](#reducing-migration-duration). -### Limits +## Limits > Eight hour time limit on migrations [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/429867) in GitLab 16.7. @@ -138,7 +136,7 @@ You can test the maximum relation size limit using these APIs: If either API produces files larger than the maximum relation size limit, group migration by direct transfer fails. -### Visibility rules +## Visibility rules After migration: @@ -154,7 +152,7 @@ After migration: If you used a private network on your source instance to hide content from the general public, make sure to have a similar setup on the destination instance, or to import into a private group. -### Prerequisites +## Prerequisites > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. @@ -181,7 +179,7 @@ To migrate groups by direct transfer: - [Configure `proxy_download`](../../../administration/object_storage.md#configure-the-common-parameters). - Ensure that the destination GitLab instance has access to the object storage of the source GitLab instance. -### Prepare user accounts +## Prepare user accounts To ensure GitLab maps users and their contributions correctly: @@ -196,7 +194,7 @@ To ensure GitLab maps users and their contributions correctly: 1. If users already exist on the destination instance and you use [SAML SSO for GitLab.com groups](../../group/saml_sso/index.md), all users must [link their SAML identity to their GitLab.com account](../../group/saml_sso/index.md#link-saml-to-your-existing-gitlabcom-account). -### Connect the source GitLab instance +## Connect the source GitLab instance Create the group you want to import to and connect the source GitLab instance: @@ -209,7 +207,7 @@ Create the group you want to import to and connect the source GitLab instance: 1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your source GitLab instance. 1. Select **Connect instance**. -### Select the groups and projects to import +## Select the groups and projects to import > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/385689) in GitLab 15.8, option to import groups with or without projects. @@ -228,7 +226,7 @@ WARNING: Importing groups with projects is in [Beta](../../../policy/experiment-beta-support.md#beta). This feature is not ready for production use. -### Group import history +## Group import history > **Partially completed** status [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/394727) in GitLab 16.7. @@ -248,7 +246,7 @@ To view group import history: 1. In the upper-right corner, select **History**. 1. If there are any errors for a particular import, select **See failures** to see their details. -### Review results of the import +## Review results of the import > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/429109) in GitLab 16.6 [with a flag](../../feature_flags.md) named `bulk_import_details_page`. Enabled by default. @@ -257,7 +255,7 @@ To review the results of an import: 1. Go to the [Group import history page](#group-import-history). 1. To see the details of a failed import, select the **See failures** link on any import with a **Failed** or **Partially completed** status. -### Migrated group items +## Migrated group items The group items that are migrated depend on the version of GitLab you use on the destination. To determine if a specific group item is migrated: @@ -304,14 +302,14 @@ Group items that are migrated to the destination GitLab instance include: - Already exists in the destination GitLab instance. - Has a public email in the source GitLab instance that matches a confirmed email in the destination GitLab instance. -#### Excluded items +### Excluded items Some group items are excluded from migration because they either: - May contain sensitive information: CI/CD variables, webhooks, and deploy tokens. - Are not supported: push rules. -### Migrated project items **(BETA)** +## Migrated project items **(BETA)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267945) in GitLab 14.4 [with a flag](../../feature_flags.md) named `bulk_import_projects`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/339941) in GitLab 15.6. @@ -384,7 +382,7 @@ Project items that are migrated to the destination GitLab instance include: </small> </html> -#### Issue-related items +### Issue-related items Issue-related project items that are migrated to the destination GitLab instance include: @@ -397,7 +395,7 @@ Issue-related project items that are migrated to the destination GitLab instance | Merge request URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | | Time tracking | [GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/267946) | -#### Merge request-related items +### Merge request-related items Merge request-related project items that are migrated to the destination GitLab instance include: @@ -411,7 +409,7 @@ Merge request-related project items that are migrated to the destination GitLab | Issue URL references | [GitLab 15.6](https://gitlab.com/gitlab-org/gitlab/-/issues/267947) | | Time tracking | [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/339403) | -#### Setting-related items +### Setting-related items Setting-related project items that are migrated to the destination GitLab instance include: @@ -422,7 +420,7 @@ Setting-related project items that are migrated to the destination GitLab instan | Project properties | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75898) | | Service Desk | [GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75653) | -#### Excluded items +### Excluded items Some project items are excluded from migration because they either: @@ -443,7 +441,7 @@ Some project items are excluded from migration because they either: - Pages domains - Remote mirrors -### Troubleshooting +## Troubleshooting In a [rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session), you can find the failure or error messages for the group import attempt using: @@ -468,7 +466,7 @@ entities.where(status: [-1]).pluck(:destination_name, :destination_namespace, :s You can also see all migrated entities with any failures related to them using an [API endpoint](../../../api/bulk_imports.md#list-all-group-or-project-migrations-entities). -#### Stale imports +### Stale imports > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352985) in GitLab 14.10. @@ -483,7 +481,7 @@ import = BulkImports::Entity.where(namespace_id: Group.id).map(&:bulk_import) import.status #=> 3 means that the import timed out. ``` -#### Error: `404 Group Not Found` +### Error: `404 Group Not Found` If you attempt to import a group that has a path comprised of only numbers (for example, `5000`), GitLab attempts to find the group by ID instead of the path. This causes a `404 Group Not Found` error in GitLab 15.4 and earlier. @@ -499,7 +497,7 @@ To solve this, you must change the source group path to include a non-numerical - The [Groups API](../../../api/groups.md#update-group). -#### Other `404` errors +### Other `404` errors You can receive other `404` errors when importing a group, for example: @@ -511,7 +509,7 @@ You can receive other `404` errors when importing a group, for example: This error indicates a problem transferring from the _source_ instance. To solve this, check that you have met the [prerequisites](#prerequisites) on the source instance. -#### Reducing migration duration +### Reducing migration duration A single direct transfer migration runs 5 entities (groups or projects) per import at a time, independent of the number of workers available on the destination instance. That said, having more workers on the destination instance speeds up migration by decreasing the time it takes to import each entity. @@ -529,157 +527,6 @@ Distributing projects in different groups helps to avoid timeouts. If several la The GitLab UI can only migrate top-level groups. Using the API, you can also migrate subgroups. -## Migrate groups by uploading an export file (deprecated) - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. -> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. - -WARNING: -This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by -[migrating groups by direct transfer](#migrate-groups-by-direct-transfer-recommended). However, this feature is still recommended for migrating groups between -offline systems. To follow progress on an alternative solution for [offline environments](../../application_security/offline_deployments/index.md), see -[the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). - -Prerequisites: - -- Owner role on the group to migrate. - -Using file exports, you can: - -- Export any group to a file and upload that file to another GitLab instance or to another location on the same instance. -- Use either the GitLab UI or the [API](../../../api/group_import_export.md). -- Migrate groups one by one, then export and import each project for the groups one by one. - -GitLab maps user contributions correctly when an admin access token is used to perform the import. GitLab does not map -user contributions correctly when you are importing from a self-managed instance to GitLab.com. Correct mapping of user -contributions when importing from a self-managed instance to GitLab.com can be preserved with paid involvement of -Professional Services team. - -Note the following: - -- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. -- To preserve group-level relationships from imported projects, export and import groups first so that projects can - be imported into the desired group structure. -- Imported groups are given a `private` visibility level, unless imported into a parent group. -- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. -- You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) - and vice versa. The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're - exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, - see [downgrading from EE to CE](../../../index.md). - -### Compatibility - -> Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/383682) in GitLab 15.8. - -Group file exports are in NDJSON format. - -You can import group file exports that were exported from a version of GitLab up to two -[minor](../../../policy/maintenance.md#versioning) versions behind, which is similar to our process for -[security releases](../../../policy/maintenance.md#security-releases). - -For example: - -| Destination version | Compatible source versions | -|:--------------------|:---------------------------| -| 13.0 | 13.0, 12.10, 12.9 | -| 13.1 | 13.1, 13.0, 12.10 | - -### Exported contents - -The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) -file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch -for your version of GitLab to check which items can be imported to the destination GitLab instance. For example, -[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). - -Group items that are exported include: - -- Milestones -- Group Labels (_without_ associated label priorities) -- Boards and Board Lists -- Badges -- Subgroups (including all the aforementioned data) -- Epics - - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) -- Events -- [Wikis](../../project/wiki/group.md) - ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) -- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) - -Items that are **not** exported include: - -- Projects -- Runner tokens -- SAML discovery tokens - -### Preparation - -- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make -sure these users exist before importing the desired groups. -- Users must set a public email in the source GitLab instance that matches their confirmed primary email in the destination GitLab instance. Most users receive an email asking them to confirm their email address. - -### Enable export for a group - -Prerequisites: - -- You must have the Owner role for the group. - -To enable import and export for a group: - -1. On the left sidebar, at the bottom, select **Admin Area**. -1. Select **Settings > General**. -1. Expand **Visibility and access controls**. -1. In the **Import sources** section, select the checkboxes for the sources you want. - -### Export a group - -Prerequisites: - -- You must have the Owner role for the group. - -To export the contents of a group: - -1. On the left sidebar, select **Search or go to** and find your group. -1. Select **Settings > General**. -1. In the **Advanced** section, select **Export group**. -1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) - in a compressed tar archive, with contents in NDJSON format. -1. Alternatively, you can download the export from the UI: - - 1. Return to your group's **Settings > General** page. - 1. In the **Advanced** section, select **Download export**. - You can also generate a new file by selecting **Regenerate export**. - -You can also export a group [using the API](../../../api/group_import_export.md). - -### Import the group - -1. On the left sidebar, at the top, select **Create new** (**{plus}**) and **New subgroup**. -1. Select the **import an existing group** link. -1. Enter your group name. -1. Accept or modify the associated group URL. -1. Select **Choose file...**. -1. Select the file that you exported in the [Export a group](#export-a-group) section. -1. To begin importing, select **Import**. - -Your newly imported group page appears after the operation completes. - -NOTE: -The maximum import file size can be set by the administrator, default is `0` (unlimited). -As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the -[Application settings API](../../../api/settings.md#change-application-settings) or the -[Admin Area](../../../administration/settings/account_and_limit_settings.md). -Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. - -### Rate limits - -To help avoid abuse, by default, users are rate limited to: - -| Request Type | Limit | -| ---------------- | ---------------------------------------- | -| Export | 6 groups per minute | -| Download export | 1 download per group per minute | -| Import | 6 groups per minute | - ## Automate group and project import **(PREMIUM ALL)** For information on automating user, group, and project import API calls, see diff --git a/doc/user/group/manage.md b/doc/user/group/manage.md index 8da2c9677b2..58c3f837e26 100644 --- a/doc/user/group/manage.md +++ b/doc/user/group/manage.md @@ -165,7 +165,7 @@ Transferring groups moves them from one place to another in the same GitLab inst - Convert a subgroup into a top-level group by transferring it out of its current group. If you need to copy a group to a different GitLab instance, -[migrate the group by direct transfer](import/index.md#migrate-groups-by-direct-transfer-recommended). +[migrate the group by direct transfer](import/index.md). When transferring groups, note: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 8c15d696760..05633cac3b0 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -66,7 +66,7 @@ The following table lists project permissions available for each role: | [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/proxy-based.md#on-demand-scans) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/on-demand_scan.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index e604d9d871b..135b51bf81a 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The GitLab.com importer was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108502) in GitLab 15.8 and removed in GitLab 16.0. To import GitLab projects from GitLab.com to a self-managed GitLab instance use -[migrating groups and projects by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +[migrating groups and projects by direct transfer](../../group/import/index.md). ## Related topics diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index efc74b7ee1e..8c9ba408799 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -15,7 +15,7 @@ To bring existing projects to GitLab, or copy GitLab groups and projects to a di ## Migrate from GitLab to GitLab by using direct transfer The best way to migrate GitLab groups and projects between GitLab instances, or in the same GitLab instance, is -[by using direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +[by using direct transfer](../../group/import/index.md). You can also migrate GitLab projects by using a GitLab file export, which is a supported import source. diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 951c848dee1..d2c5b0af339 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -155,23 +155,50 @@ You can create a merge request by running Git commands on your local machine. You can create a merge request from your fork to contribute back to the main project. -1. On the left sidebar, select **Search or go to** and find your project. -1. Select your fork of the repository. -1. On the left sidebar, select **Code > Merge requests**, and select **New merge request**. -1. In the **Source branch** dropdown list box, select the branch in your forked repository as the source branch. -1. In the **Target branch** dropdown list box, select the branch from the upstream repository as the target branch. - You can set a [default target project](#set-the-default-target-project) to - change the default target branch (which can be useful if you are working in a - forked project). +1. On the left sidebar, select **Search or go to** and find your fork. +1. Select **Code > Merge requests**, and select **New merge request**. +1. For **Source branch**, select the branch in your fork that contains your changes. +1. For **Target branch**: + + 1. Select the target project. (Make sure to select the upstream project, rather than your fork.) + 1. Select a branch from the upstream repository. + + NOTE: + If you contribute changes upstream frequently, consider setting a + [default target project](#set-the-default-target-project) for your fork. + 1. Select **Compare branches and continue**. -1. Select **Create merge request**. +1. Select **Create merge request**. The merge request is created in the target project, + not your fork. -After your work is merged, if you don't intend to -make any other contributions to the upstream project, you can -[unlink your fork](../repository/forking_workflow.md#unlink-a-fork) from its upstream project. +After your work merges, [unlink your fork](../repository/forking_workflow.md#unlink-a-fork) +from its upstream project if you don't intend to make more contributions. For more information, [see the forking workflow documentation](../repository/forking_workflow.md). +### Set the default target project + +By default, merge requests originating from a fork target your fork, not the upstream project. +If you frequently contribute back to the upstream project, and want to target it +by default, change the default target for your fork. + +Prerequisites: + +- You're working in a fork. +- You must have at least the Developer role, or be allowed to create merge requests in the project. +- The upstream project allows merge requests to be created. +- The [visibility settings](../../public_access.md#change-project-visibility) for + the fork must match, or be less strict than, the upstream repository. For example: + this setting isn't shown if your fork is private, but the upstream is public. + +To do this: + +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > Merge requests**. +1. In the **Target project** section, select the option you want to use for + your default target project. +1. Select **Save changes**. + ## By sending an email You can create a merge request by sending an email message to GitLab. @@ -179,60 +206,41 @@ The merge request target branch is the project's default branch. Prerequisites: +- The merge request must target the current project, not an upstream project. - A GitLab administrator must configure [incoming email](../../../administration/incoming_email.md). - A GitLab administrator must configure [Reply by email](../../../administration/reply_by_email.md). +- You must have at least the Developer role, or be allowed to create merge requests in the project. To create a merge request by sending an email: 1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Code > Merge requests**. -1. In the upper-right corner, select **Email a new merge request to this project**. - An email address is displayed. Copy this address. - Ensure you keep this address private. +1. If the project contains any merge requests, select **Email a new merge request to this project**. +1. In the dialog, copy the email address shown. Keep this address private. Anyone who + has it can create issues or merge requests as if they were you. 1. Open an email and compose a message with the following information: - The **To** line is the email address you copied. - - The subject line is the source branch name. - - The message body is the merge request description. + - The **Subject** is the source branch name. + - The body of the email is the merge request description. -1. Send the email message. +1. To add commits, attach `.patch` files to the message. +1. Send the email. A merge request is created. ### Add attachments when creating a merge request by email -You can add commits to a merge request by adding -patches as attachments to the email. All attachments with a file name ending in `.patch` are considered patches and are processed -ordered by name. +Add commits to a merge request by adding patches as attachments to the email. -The combined size of the patches can be 2 MB. - -If the source branch from the subject does not exist, it is -created from the repository's HEAD or the specified target branch. -You can specify the target branch by using the -[`/target_branch` quick action](../quick_actions.md). If the source -branch already exists, the patches are applied on top of it. - -## Set the default target project - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58093) in GitLab 13.11. - -Merge requests have a source and a target project that are the same, unless -forking is involved. Creating a fork of the project can cause either of these -scenarios when you create a new merge request: - -- You target an upstream project (the project you forked, and the default - option). -- You target your own fork. - -To have merge requests from a fork by default target your own fork -(instead of the upstream project), you can change the default. - -1. On the left sidebar, select **Search or go to** and find your project. -1. Select **Settings > Merge requests**. -1. In the **Target project** section, select the option you want to use for - your default target project. -1. Select **Save changes**. +- The combined size of the patches must be 2 MB or less. +- To be considered a patch, the attachment's file name must end in `.patch`. +- Patches are processed in order by name. +- If the source branch from the subject does not exist, it is + created from the repository's `HEAD`, or the default target branch. + To change the target branch manually, use the + [`/target_branch` quick action](../quick_actions.md). +- If the source branch already exists, patches are applied on top of it. ## Troubleshooting @@ -249,3 +257,14 @@ To make this button appear, one possible workaround is to [remove your project's fork relationship](../repository/forking_workflow.md#unlink-a-fork). After removal, the fork relationship cannot be restored. This project can no longer be able to receive or send merge requests to the source project, or other forks. + +### Email message could not be processed + +When sending an email to create a merge request, and you attempt to target an +upstream project, GitLab responds with this error: + +```plaintext +Unfortunately, your email message to GitLab could not be processed. + +You are not allowed to perform this action. If you believe this is in error, contact a staff member. +``` diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index fc9b24362e0..dea50ecc408 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -4,27 +4,31 @@ group: Import and Integrate info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# Migrating projects using file exports **(FREE ALL)** +# Migrate projects and groups by using file exports **(FREE ALL)** + +You can migrate projects and groups by using file exports. However, using +[direct transfer](../../group/import/index.md) is recommended if possible. + +## Migrate projects by uploading an export file Existing projects on any self-managed GitLab instance or GitLab.com can be exported to a file and -then imported into another GitLab instance. You can also copy GitLab projects to another location with more automation by -[migrating groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +then imported into another GitLab instance. -## Preserving user contributions +### Preserving user contributions Preserving user contribution depends on meeting the following requirements: -### Migrating from GitLab self-managed to GitLab.com +#### Migrating from GitLab self-managed to GitLab.com When migrating projects by using file exports, an administrator's access token is required for user contributions to map correctly. Therefore, user contributions never map correctly when importing file exports from a self-managed instance to GitLab.com. Instead, all GitLab user associations (such as comment author) are changed to the user importing the project. To preserve contribution history, do one of the following: -- [Migrate by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). +- [Migrate by direct transfer](../../group/import/index.md). - Consider paid GitLab [migration services](https://about.gitlab.com/services/migration/). -### Migrating to GitLab self-managed +#### Migrating to GitLab self-managed To ensure GitLab maps users and their contributions correctly: @@ -43,7 +47,7 @@ That user becomes an author of merge requests created by other users. Supplement - Added for comments, merge request approvals, linked tasks, and items. - Not added for the merge request or issue creator, added or removed labels, and merged-by information. -## Edit project export files +### Edit project export files You can add or remove data from export files. For example, you can: @@ -58,7 +62,7 @@ To edit a project export file: You can also make sure that all members were exported by checking the `project_members.ndjson` file. -## Compatibility +### Compatibility > Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/389888) in GitLab 15.11. @@ -75,7 +79,7 @@ For example: | 13.0 | 13.0, 12.10, 12.9 | | 13.1 | 13.1, 13.0, 12.10 | -## Configure file exports as an import source **(FREE SELF)** +### Configure file exports as an import source **(FREE SELF)** Before you can migrate projects on a self-managed GitLab instance using file exports, GitLab administrators must: @@ -92,7 +96,7 @@ To enable file exports as an import source for the destination instance: 1. Scroll to **Import sources**. 1. Select the **GitLab export** checkbox. -## Between CE and EE +### Between CE and EE You can export projects from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) and vice versa, assuming [compatibility](#compatibility) is met. @@ -101,7 +105,7 @@ If you're exporting a project from the Enterprise Edition to the Community Editi data that is retained only in the Enterprise Edition. For more information, see [downgrading from EE to CE](../../../index.md). -## Export a project and its data +### Export a project and its data Before you can import a project, you must export it. @@ -124,7 +128,7 @@ To export a project and its data, follow these steps: The export is generated in your configured `shared_path`, a temporary shared directory, and then moved to your configured `uploads_directory`. Every 24 hours, a worker deletes these export files. -### Items that are exported +#### Items that are exported Exported project items depend on the version of GitLab you use. To determine if a specific project item is exported: @@ -166,7 +170,7 @@ For a quick overview, items that are exported include: - Project and inherited group members, as long as the user has the Maintainer role in the exported project's group or is an administrator -### Items that are not exported +#### Items that are not exported Items that are **not** exported include: @@ -189,7 +193,7 @@ Items that are **not** exported include: Migrating projects with file exports uses the same export and import mechanisms as creating projects from templates at the [group](../../group/custom_project_templates.md) and [instance](../../../administration/custom_project_templates.md) levels. Therefore, the list of exported items is the same. -## Import a project and its data +### Import a project and its data > Default maximum import file size [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. Administrators of self-managed instances can [set maximum import file size](#set-maximum-import-file-size). On GitLab.com, the value is [set to 5 GB](../../gitlab_com/index.md#account-and-limit-settings). @@ -199,7 +203,7 @@ WARNING: Only import projects from sources you trust. If you import a project from an untrusted source, it may be possible for an attacker to steal your sensitive data. -### Prerequisites +#### Prerequisites > Requirement for Maintainer role instead of Developer role introduced in GitLab 16.0 and backported to GitLab 15.11.1 and GitLab 15.10.5. @@ -209,7 +213,7 @@ may be possible for an attacker to steal your sensitive data. - Review [compatibility](#compatibility) for any issues. - At least the Maintainer role on the destination group to migrate to. -### Import a project +#### Import a project To import a project: @@ -222,7 +226,7 @@ To import a project: To get the status of an import, you can query it through the [API](../../../api/project_import_export.md#import-status). As described in the API documentation, the query may return an import error or exceptions. -### Changes to imported items +#### Changes to imported items Exported items are imported with the following changes: @@ -235,11 +239,11 @@ Exported items are imported with the following changes: Deploy keys aren't imported. To use deploy keys, you must enable them in your imported project and update protected branches. -### Import large projects **(FREE SELF)** +#### Import large projects **(FREE SELF)** If you have a larger project, consider [using a Rake task](../../../administration/raketasks/project_import_export.md#import-large-projects). -## Set maximum import file size **(FREE SELF)** +### Set maximum import file size **(FREE SELF)** Administrators can set the maximum import file size one of two ways: @@ -248,7 +252,7 @@ Administrators can set the maximum import file size one of two ways: The default is `0` (unlimited). -## Rate limits +### Rate limits To help avoid abuse, by default, users are rate limited to: @@ -258,11 +262,161 @@ To help avoid abuse, by default, users are rate limited to: | Download export | 1 download per group per minute | | Import | 6 projects per minute | +## Migrate groups by uploading an export file (deprecated) + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) in GitLab 14.6 and replaced by +[migrating groups by direct transfer](../../group/import/index.md). However, this feature is still recommended for migrating groups between +offline systems. To follow progress on an alternative solution for [offline environments](../../application_security/offline_deployments/index.md), see +[the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/8985). + +Prerequisites: + +- Owner role on the group to migrate. + +Using file exports, you can: + +- Export any group to a file and upload that file to another GitLab instance or to another location on the same instance. +- Use either the GitLab UI or the [API](../../../api/group_import_export.md). +- Migrate groups one by one, then export and import each project for the groups one by one. + +GitLab maps user contributions correctly when an admin access token is used to perform the import. GitLab does not map +user contributions correctly when you are importing from a self-managed instance to GitLab.com. Correct mapping of user +contributions when importing from a self-managed instance to GitLab.com can be preserved with paid involvement of +Professional Services team. + +Note the following: + +- Exports are stored in a temporary directory and are deleted every 24 hours by a specific worker. +- To preserve group-level relationships from imported projects, export and import groups first so that projects can + be imported into the desired group structure. +- Imported groups are given a `private` visibility level, unless imported into a parent group. +- If imported into a parent group, a subgroup inherits the same level of visibility unless otherwise restricted. +- You can export groups from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) + and vice versa. The Enterprise Edition retains some group data that isn't part of the Community Edition. If you're + exporting a group from the Enterprise Edition to the Community Edition, you may lose this data. For more information, + see [downgrading from EE to CE](../../../index.md). + +### Compatibility + +> Support for JSON-formatted project file exports [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/383682) in GitLab 15.8. + +Group file exports are in NDJSON format. + +You can import group file exports that were exported from a version of GitLab up to two +[minor](../../../policy/maintenance.md#versioning) versions behind, which is similar to our process for +[security releases](../../../policy/maintenance.md#security-releases). + +For example: + +| Destination version | Compatible source versions | +|:--------------------|:---------------------------| +| 13.0 | 13.0, 12.10, 12.9 | +| 13.1 | 13.1, 13.0, 12.10 | + +### Exported contents + +The [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) +file for groups lists items exported and imported when migrating groups using file exports. View this file in the branch +for your version of GitLab to check which items can be imported to the destination GitLab instance. For example, +[`import_export.yml` on the `14-10-stable-ee` branch](https://gitlab.com/gitlab-org/gitlab/-/blob/14-10-stable-ee/lib/gitlab/import_export/group/import_export.yml). + +Group items that are exported include: + +- Milestones +- Group Labels (_without_ associated label priorities) +- Boards and Board Lists +- Badges +- Subgroups (including all the aforementioned data) +- Epics + - Epic resource state events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/291983) in GitLab 15.4) +- Events +- [Wikis](../../project/wiki/group.md) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) +- Iterations cadences ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95372) in 15.4) + +Items that are **not** exported include: + +- Projects +- Runner tokens +- SAML discovery tokens + +### Preparation + +- To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make +sure these users exist before importing the desired groups. +- Users must set a public email in the source GitLab instance that matches their confirmed primary email in the destination GitLab instance. Most users receive an email asking them to confirm their email address. + +### Enable export for a group + +Prerequisites: + +- You must have the Owner role for the group. + +To enable import and export for a group: + +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Visibility and access controls**. +1. In the **Import sources** section, select the checkboxes for the sources you want. + +### Export a group + +Prerequisites: + +- You must have the Owner role for the group. + +To export the contents of a group: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > General**. +1. In the **Advanced** section, select **Export group**. +1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) + in a compressed tar archive, with contents in NDJSON format. +1. Alternatively, you can download the export from the UI: + + 1. Return to your group's **Settings > General** page. + 1. In the **Advanced** section, select **Download export**. + You can also generate a new file by selecting **Regenerate export**. + +You can also export a group [using the API](../../../api/group_import_export.md). + +### Import the group + +1. On the left sidebar, at the top, select **Create new** (**{plus}**) and **New subgroup**. +1. Select the **import an existing group** link. +1. Enter your group name. +1. Accept or modify the associated group URL. +1. Select **Choose file...**. +1. Select the file that you exported in the [Export a group](#export-a-group) section. +1. To begin importing, select **Import**. + +Your newly imported group page appears after the operation completes. + +NOTE: +The maximum import file size can be set by the administrator, default is `0` (unlimited). +As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the +[Application settings API](../../../api/settings.md#change-application-settings) or the +[Admin Area](../../../administration/settings/account_and_limit_settings.md). +Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to 0 in GitLab 13.8. + +### Rate limits + +To help avoid abuse, by default, users are rate limited to: + +| Request Type | Limit | +| ---------------- | ---------------------------------------- | +| Export | 6 groups per minute | +| Download export | 1 download per group per minute | +| Import | 6 groups per minute | + ## Related topics - [Project import and export API](../../../api/project_import_export.md) - [Project import and export administration Rake tasks](../../../administration/raketasks/project_import_export.md) - [Migrating GitLab groups](../../group/import/index.md) - [Group import and export API](../../../api/group_import_export.md) -- [Migrate groups by direct transfer](../../group/import/index.md#migrate-groups-by-direct-transfer-recommended). -- [Migrate groups by using file exports](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). +- [Migrate groups by direct transfer](../../group/import/index.md). diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md index 64f6fa2c75e..59e949c5218 100644 --- a/doc/user/project/wiki/group.md +++ b/doc/user/project/wiki/group.md @@ -37,7 +37,7 @@ To access a group wiki: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9. Users with the Owner role in a group can -[import or export a group wiki](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) when they +[import or export a group wiki](../../project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) when they import or export a group. Content created in a group wiki is not deleted when an account is downgraded or a @@ -47,7 +47,7 @@ the wiki is exported. To access the group wiki data from the export file if the feature is no longer available, you have to: -1. Extract the [export file tarball](../../group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated) +1. Extract the [export file tarball](../../project/settings/import_export.md#migrate-groups-by-uploading-an-export-file-deprecated) with this command, replacing `FILENAME` with your file's name: `tar -xvzf FILENAME.tar.gz` 1. Browse to the `repositories` directory. This directory contains a |