diff options
Diffstat (limited to 'doc/user/application_security/api_fuzzing/index.md')
| -rw-r--r-- | doc/user/application_security/api_fuzzing/index.md | 475 |
1 files changed, 249 insertions, 226 deletions
diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index dfb7e12a8be..8511c919c14 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -7,17 +7,51 @@ type: reference, howto # Web API Fuzz Testing **(ULTIMATE)** -You can add web API fuzzing to your [GitLab CI/CD](../../../ci/README.md) -pipelines. This helps you discover bugs and potential security issues that other QA processes may -miss. API fuzzing performs fuzz testing of API operation parameters. Fuzz testing sets operation +Web API fuzzing performs fuzz testing of API operation parameters. Fuzz testing sets operation parameters to unexpected values in an effort to cause unexpected behavior and errors in the API -backend. +backend. This helps you discover bugs and potential security issues that other QA processes may +miss. We recommend that you use fuzz testing in addition to [GitLab Secure](../index.md)'s other security scanners and your own test processes. If you're using [GitLab CI/CD](../../../ci/README.md), you can run fuzz tests as part your CI/CD workflow. -## Requirements +## When Web API fuzzing runs + +Web API fuzzing runs in the `fuzz` stage of the CI/CD pipeline. To ensure API fuzzing scans the +latest code, your CI/CD pipeline should deploy changes to a test environment in one of the stages +preceding the `fuzz` stage. + +Note the following changes have been made to the API fuzzing template: + +- In GitLab 14.0 and later, you must define a `fuzz` stage in your `.gitlab-ci.yml` file. +- In GitLab 13.12 and earlier, the API fuzzing template defines `build`, `test`, `deploy`, and + `fuzz` stages. The `fuzz` stage runs last by default. The predefined stages were deprecated, and removed from the `API-Fuzzing.latest.gitlab-ci.yml` template. They will be removed in a future GitLab + version. + +If your pipeline is configured to deploy to the same web server on each run, running a +pipeline while another is still running could cause a race condition in which one pipeline +overwrites the code from another. The API to scan should be excluded from changes for the duration +of a fuzzing scan. The only changes to the API should be from the fuzzing scanner. Any changes made +to the API (for example, by users, scheduled tasks, database changes, code changes, other pipelines, +or other scanners) during a scan could cause inaccurate results. + +You can run a Web API fuzzing scan using the following methods: + +- [OpenAPI Specification](#openapi-specification) - version 2.0 or 3.0 +- [HTTP Archive](#http-archive-har) (HAR) +- [Postman Collection](#postman-collection) - version 2.0 or 2.1 + +Example projects using these methods are available: + +- [Example OpenAPI v2 Specification project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/openapi) +- [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/har) +- [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/postman-api-fuzzing-example) +- [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/graphql-api-fuzzing-example) + +## Enable Web API fuzzing + +Requirements: - One of the following web API types: - REST API @@ -29,49 +63,26 @@ you can run fuzz tests as part your CI/CD workflow. - HTTP Archive (HAR) of API requests to test - Postman Collection v2.0 or v2.1 -## When fuzzing scans run - -When using the `API-Fuzzing.gitlab-ci.yml` template, the `fuzz` job runs last, as shown here. To -ensure API fuzzing scans the latest code, your CI pipeline should deploy changes to a test -environment in one of the jobs preceding the `fuzz` job: - -```yaml -stages: - - build - - test - - deploy - - fuzz -``` - -Note that if your pipeline is configured to deploy to the same web server on each run, running a -pipeline while another is still running could cause a race condition in which one pipeline -overwrites the code from another. The API to scan should be excluded from changes for the duration -of a fuzzing scan. The only changes to the API should be from the fuzzing scanner. Be aware that -any changes made to the API (for example, by users, scheduled tasks, database changes, code -changes, other pipelines, or other scanners) during a scan could cause inaccurate results. - -## Configuration + WARNING: + **NEVER** run fuzz testing against a production server. Not only can it perform *any* function that + the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting + data. Only run fuzzing against a test server. -There are three ways to perform scans. See the configuration section for the one you wish to use: +To enable Web API fuzzing: -- [OpenAPI v2 or v3 specification](#openapi-specification) -- [HTTP Archive (HAR)](#http-archive-har) -- [Postman Collection v2.0 or v2.1](#postman-collection) +- Include the API fuzzing template in your `.gitlab-ci.yml` file. +- From GitLab 13.10 and later, use the Web API fuzzing configuration form. -Examples of both configurations can be found here: +- For manual configuration instructions, see the respective section, depending on the API type: + - [OpenAPI Specification](#openapi-specification) + - [HTTP Archive (HAR)](#http-archive-har) + - [Postman Collection](#postman-collection) +- Otherwise, see [Web API fuzzing configuration form](#web-api-fuzzing-configuration-form). -- [Example OpenAPI v2 specification project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/openapi) -- [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/har) -- [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/postman-api-fuzzing-example) -- [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/graphql-api-fuzzing-example) - -WARNING: -GitLab 14.0 will require that you place API fuzzing configuration files (for example, -`gitlab-api-fuzzing-config.yml`) in your repository's `.gitlab` directory instead of your -repository's root. You can continue using your existing configuration files as they are, but -starting in GitLab 14.0, GitLab will not check your repository's root for configuration files. +In GitLab 14.0 and later, API fuzzing configuration files must be in your repository's +`.gitlab` directory instead of your repository's root. -### Configuration form +### Web API fuzzing configuration form > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299234) in GitLab 13.10. @@ -82,6 +93,8 @@ The API fuzzing configuration form helps you create or modify your project's API configuration. The form lets you choose values for the most common API fuzzing options and builds a YAML snippet that you can paste in your GitLab CI/CD configuration. +#### Configure Web API fuzzing with the configuration form + To generate an API Fuzzing configuration snippet: 1. From your project's home page, go to **Security & Compliance > Configuration** in the left @@ -89,25 +102,23 @@ To generate an API Fuzzing configuration snippet: 1. Select **Configure** in the **API Fuzzing** row. 1. Complete the form as needed. Read below for more information on available configuration options. 1. Select **Generate code snippet**. - -A modal opens with the YAML snippet corresponding to the options you've selected in the form. - - - -Select **Copy code and open `.gitlab-ci.yml` file** to copy the snippet to your clipboard and be redirected -to your project's `.gitlab-ci.yml` file where you can paste the YAML configuration. - -Select **Copy code only** to copy the snippet to your clipboard and close the modal. + A modal opens with the YAML snippet corresponding to the options you've selected in the form. +1. Choose one of the following actions: + 1. Select **Copy code and open `.gitlab-ci.yml` file** to copy the snippet to your clipboard and + be redirected to your project's `.gitlab-ci.yml` file where you can paste the YAML + configuration. + 1. Select **Copy code only** to copy the snippet to your clipboard and close the modal. ### OpenAPI Specification -> Support for OpenAPI Specification v3 was +> Support for OpenAPI Specification using YAML format was +> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> Support for OpenAPI Specification v3.0 was > [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228652) in GitLab 13.9. -The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an -API description format for REST APIs. This section shows you how to configure API fuzzing by using -an OpenAPI specification to provide information about the target API to test. OpenAPI specifications -are provided as a file system resource or URL. +The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. +This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. +OpenAPI Specifications are provided as a file system resource or URL. Both JSON and YAML OpenAPI formats are supported. API fuzzing uses an OpenAPI document to generate the request body. When a request body is required, the body generation is limited to these body types: @@ -116,59 +127,42 @@ the body generation is limited to these body types: - `multipart/form-data` - `application/json` -Follow these steps to configure API fuzzing in GitLab with an OpenAPI specification: +#### Configure Web API fuzzing with an OpenAPI Specification -1. To use API fuzzing, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) - that's provided as part of your GitLab installation. To do so, add the following to your - `.gitlab-ci.yml` file: - - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - ``` +To configure API fuzzing in GitLab with an OpenAPI Specification: -1. Add the configuration file [`gitlab-api-fuzzing-config.yml`](https://gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing/-/blob/master/gitlab-api-fuzzing-config.yml) to your repository's root as `.gitlab-api-fuzzing.yml`. +1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. -1. The [configuration file](#configuration-files) has several testing profiles defined with varying - amounts of fuzzing. We recommend that you start with the `Quick-10` profile. Testing with this - profile completes quickly, allowing for easier configuration validation. +1. [Include](../../../ci/yaml/README.md#includetemplate) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + in your `.gitlab-ci.yml` file. - Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick-10` for the profile you choose: +1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. + The profile specifies how many tests are run. Substitute `Quick-10` for the profile you choose. For more details, see [API fuzzing profiles](#api-fuzzing-profiles). ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - variables: FUZZAPI_PROFILE: Quick-10 ``` -1. Provide the location of the OpenAPI specification. You can provide the specification as a file - or URL. Specify the location by adding the `FUZZAPI_OPENAPI` variable: +1. Provide the location of the OpenAPI Specification. You can provide the specification as a file + or URL. Specify the location by adding the `FUZZAPI_OPENAPI` variable. - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - - variables: - FUZZAPI_PROFILE: Quick-10 - FUZZAPI_OPENAPI: test-api-specification.json - ``` - -1. The target API instance's base URL is also required. Provide it by using the `FUZZAPI_TARGET_URL` - variable or an `environment_url.txt` file. +1. Provide the target API instance's base URL. Use either the `FUZZAPI_TARGET_URL` variable or an + `environment_url.txt` file. Adding the URL in an `environment_url.txt` file at your project's root is great for testing in - dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD - pipeline, have the app persist its domain in an `environment_url.txt` file. API fuzzing - automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + dynamic environments. To run API fuzzing against an application dynamically created during a + GitLab CI/CD pipeline, have the application persist its URL in an `environment_url.txt` file. + API fuzzing automatically parses that file to find its scan target. You can see an + example of this in the [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `FUZZAPI_TARGET_URL`: +Example `.gitlab-ci.yml` file using an OpenAPI Specification: ```yaml + stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -184,10 +178,7 @@ This is a minimal configuration for API Fuzzing. From here you can: - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run fuzz testing against a production server. Not only can it perform *any* function that -the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting -data. Only run fuzzing against a test server. +For details of API fuzzing configuration options, see [Available CI/CD variables](#available-cicd-variables). ### HTTP Archive (HAR) @@ -196,59 +187,33 @@ is an archive file format for logging HTTP transactions. When used with the GitL must contain records of calling the web API to test. The API fuzzer extracts all the requests and uses them to perform testing. -You can use various tools to generate HAR files: - -- [Fiddler](https://www.telerik.com/fiddler): Web debugging proxy -- [Insomnia Core](https://insomnia.rest/): API client -- [Chrome](https://www.google.com/chrome/): Browser -- [Firefox](https://www.mozilla.org/en-US/firefox/): Browser -- [GitLab HAR Recorder](https://gitlab.com/gitlab-org/security-products/har-recorder): Command line +For more details, including how to create a HAR file, see [HTTP Archive format](create_har_files.md). WARNING: HAR files may contain sensitive information such as authentication tokens, API keys, and session cookies. We recommend that you review the HAR file contents before adding them to a repository. -Follow these steps to configure API fuzzing to use a HAR file that provides information about the -target API to test: +#### Configure Web API fuzzing with a HAR file -1. To use API fuzzing, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) - that's provided as part of your GitLab installation. To do so, add the following to your - `.gitlab-ci.yml` file: +To configure API fuzzing to use a HAR file: - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - ``` +1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. -1. Add the configuration file [`gitlab-api-fuzzing-config.yml`](https://gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing/-/blob/master/gitlab-api-fuzzing-config.yml) to your repository's root as `.gitlab-api-fuzzing.yml`. - -1. The [configuration file](#configuration-files) has several testing profiles defined with varying - amounts of fuzzing. We recommend that you start with the `Quick-10` profile. Testing with this - profile completes quickly, allowing for easier configuration validation. +1. [Include](../../../ci/yaml/README.md#includetemplate) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + in your `.gitlab-ci.yml` file. - Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick-10` for the profile you choose: +1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. + The profile specifies how many tests are run. Substitute `Quick-10` for the profile you choose. For more details, see [API fuzzing profiles](#api-fuzzing-profiles). ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - variables: FUZZAPI_PROFILE: Quick-10 ``` 1. Provide the location of the HAR specification. You can provide the specification as a file - or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_HAR` variable: - - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - - variables: - FUZZAPI_PROFILE: Quick-10 - FUZZAPI_HAR: test-api-recording.har - ``` + or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) + in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_HAR` variable. 1. The target API instance's base URL is also required. Provide it by using the `FUZZAPI_TARGET_URL` variable or an `environment_url.txt` file. @@ -259,9 +224,12 @@ target API to test: automatically parses that file to find its scan target. You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `FUZZAPI_TARGET_URL`: +Example `.gitlab-ci.yml` file using a HAR file: ```yaml + stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -271,16 +239,13 @@ target API to test: FUZZAPI_TARGET_URL: http://test-deployment/ ``` -This is a minimal configuration for API Fuzzing. From here you can: +This is a minimal configuration for API fuzzing. From here you can: - [Run your first scan](#running-your-first-scan). - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run fuzz testing against a production server. Not only can it perform *any* function that -the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting -data. Only run fuzzing against a test server. +For details of API fuzzing configuration options, see [Available CI/CD variables](#available-cicd-variables). ### Postman Collection @@ -299,51 +264,31 @@ Postman Collection files may contain sensitive information such as authenticatio and session cookies. We recommend that you review the Postman Collection file contents before adding them to a repository. -Follow these steps to configure API fuzzing to use a Postman Collection file that provides -information about the target API to test: +#### Configure Web API fuzzing with a Postman Collection file -1. To use API fuzzing, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) - that's provided as part of your GitLab installation. To do so, add the following to your - `.gitlab-ci.yml` file: +To configure API fuzzing to use a Postman Collection file: - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - ``` +1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. -1. Add the configuration file [`gitlab-api-fuzzing-config.yml`](https://gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing/-/blob/master/gitlab-api-fuzzing-config.yml) - to your repository's root as `.gitlab-api-fuzzing.yml`. - -1. The [configuration file](#configuration-files) has several testing profiles defined with varying - amounts of fuzzing. We recommend that you start with the `Quick-10` profile. Testing with this - profile completes quickly, allowing for easier configuration validation. +1. [Include](../../../ci/yaml/README.md#includetemplate) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + in your `.gitlab-ci.yml` file. - Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick-10` for the profile you choose: +1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. + The profile specifies how many tests are run. Substitute `Quick-10` for the profile you choose. For more details, see [API fuzzing profiles](#api-fuzzing-profiles). ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - variables: FUZZAPI_PROFILE: Quick-10 ``` -1. Provide the location of the Postman Collection specification. You can provide the specification as a file - or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_POSTMAN_COLLECTION` variable: - - ```yaml - include: - - template: API-Fuzzing.gitlab-ci.yml - - variables: - FUZZAPI_PROFILE: Quick-10 - FUZZAPI_POSTMAN_COLLECTION: postman-collection_serviceA.json - ``` +1. Provide the location of the Postman Collection specification. You can provide the specification + as a file or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) + in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_POSTMAN_COLLECTION` + variable. -1. The target API instance's base URL is also required. Provide it by using the `FUZZAPI_TARGET_URL` - variable or an `environment_url.txt` file. +1. Provide the target API instance's base URL. Use either the `FUZZAPI_TARGET_URL` variable or an + `environment_url.txt` file. Adding the URL in an `environment_url.txt` file at your project's root is great for testing in dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD @@ -351,9 +296,12 @@ information about the target API to test: automatically parses that file to find its scan target. You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `FUZZAPI_TARGET_URL`: +Example `.gitlab-ci.yml` file using a Postman Collection file: ```yaml + stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -369,15 +317,13 @@ This is a minimal configuration for API Fuzzing. From here you can: - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run fuzz testing against a production server. Not only can it perform *any* function that -the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting -data. Only run fuzzing against a test server. +For details of API fuzzing configuration options, see [Available CI/CD variables](#available-cicd-variables). #### Postman variables Postman allows the developer to define placeholders that can be used in different parts of the -requests. These placeholders are called variables, as explained in [Using variables](https://learning.postman.com/docs/sending-requests/variables/). +requests. These placeholders are called variables, as explained in the Postman documentation, +[Using variables](https://learning.postman.com/docs/sending-requests/variables/). You can use variables to store and reuse values in your requests and scripts. For example, you can edit the collection to add variables to the document: @@ -398,7 +344,7 @@ Postman file. For example, Postman does not export environment-scoped variables file. By default, the API fuzzer uses the Postman file to resolve Postman variable values. If a JSON file -is set in a GitLab CI environment variable `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`, then the JSON +is set in a GitLab CI/CD variable `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`, then the JSON file takes precedence to get Postman variable values. Although Postman can export environment variables into a JSON file, the format is not compatible @@ -407,6 +353,9 @@ with the JSON expected by `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`. Here is an example of using `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -428,6 +377,13 @@ values. For example: } ``` +## API fuzzing configuration + +The API fuzzing behavior can be changed through CI/CD variables. + +From GitLab 13.12 and later, the default API fuzzing configuration file is `.gitlab/gitlab-api-fuzzing-config.yml`. In GitLab 14.0 and later, API fuzzing configuration files must be in your repository's +`.gitlab` directory instead of your repository's root. + ### Authentication Authentication is handled by providing the authentication token as a header or cookie. You can @@ -449,6 +405,9 @@ GitLab projects page at **Settings > CI/CD**, in the **Variables** section. Use as the value for `FUZZAPI_HTTP_PASSWORD`: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -487,6 +446,9 @@ Follow these steps to provide the bearer token with `FUZZAPI_OVERRIDES_ENV`: 1. In your `.gitlab-ci.yml` file, set `FUZZAPI_OVERRIDES_ENV` to the variable you just created: ```yaml + stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -522,6 +484,9 @@ This file can be generated by a prior stage and provided to API fuzzing through Set `FUZZAPI_OVERRIDES_FILE` in your `.gitlab-ci.yml` file: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -561,6 +526,9 @@ You must provide three CI/CD variables, each set for correct operation: For example: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -576,16 +544,15 @@ variables: To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API's application logs. -### Configuration files +### API fuzzing profiles -To get you started quickly, GitLab provides the configuration file +GitLab provides the configuration file [`gitlab-api-fuzzing-config.yml`](https://gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing/-/blob/master/gitlab-api-fuzzing-config.yml). -This file has several testing profiles that perform various numbers of tests. The run time of each -profile increases as the test numbers go up. To use a configuration file, add it to your -repository's root as `.gitlab-api-fuzzing.yml`. +It contains several testing profiles that perform a specific numbers of tests. The runtime of each +profile increases as the number of tests increases. -| Profile | Fuzz Tests (per parameter) | -|:---------|:-----------| +| Profile | Fuzz Tests (per parameter) | +|:----------|:---------------------------| | Quick-10 | 10 | | Medium-20 | 20 | | Medium-50 | 50 | @@ -593,35 +560,23 @@ repository's root as `.gitlab-api-fuzzing.yml`. ### Available CI/CD variables -| CI/CD variable | Description | -|------------------------------------------------------|--------------------| -| `FUZZAPI_VERSION` | Specify API Fuzzing container version. Defaults to `latest`. | -| `FUZZAPI_TARGET_URL` | Base URL of API testing target. | -|[`FUZZAPI_CONFIG`](#configuration-files) | API Fuzzing configuration file. Defaults to `.gitlab-apifuzzer.yml`. | -|[`FUZZAPI_PROFILE`](#configuration-files) | Configuration profile to use during testing. Defaults to `Quick`. | -| `FUZZAPI_REPORT` | Scan report filename. Defaults to `gl-api_fuzzing-report.xml`. | -|[`FUZZAPI_OPENAPI`](#openapi-specification) | OpenAPI specification file or URL. | -|[`FUZZAPI_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | -|[`FUZZAPI_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | -|[`FUZZAPI_POSTMAN_COLLECTION_VARIABLES`](#postman-variables) | Path to a JSON file to extract postman variable values. | -|[`FUZZAPI_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. | -|[`FUZZAPI_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. | -|[`FUZZAPI_OVERRIDES_CMD`](#overrides) | Overrides command. | -|[`FUZZAPI_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). | -|[`FUZZAPI_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. | -|[`FUZZAPI_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. | - -<!--|[`FUZZAPI_D_TARGET_IMAGE`](#target-container) |API target docker image | -|[`FUZZAPI_D_TARGET_ENV`](#target-container) |Docker environment options | -|[`FUZZAPI_D_TARGET_VOLUME`](#target-container) | Docker volume options | -|[`FUZZAPI_D_TARGET_PORTS`](#target-container) |Docker port options | -| `FUZZAPI_D_WORKER_IMAGE` |Custom worker docker image | -| `FUZZAPI_D_WORKER_ENV` |Custom worker docker environment options | -| `FUZZAPI_D_WORKER_VOLUME` |Custom worker docker volume options | -| `FUZZAPI_D_WORKER_PORTS` |Custom worker docker port options | -| `FUZZAPI_D_NETWORK` |Name of docker network, defaults to "testing-net"| -| `FUZZAPI_D_PRE_SCRIPT` |Pre script runs after docker login and docker network create, but before starting the scanning image container.| -| `FUZZAPI_D_POST_SCRIPT` |Post script runs after scanning image container is started. This is the place to start your target(s) and kick off scanning when using an advanced configuration.| --> +| CI/CD variable | Description | +|-------------------------------------------------------------|-------------| +| `SECURE_ANALYZERS_PREFIX` | Specify the Docker registry base address from which to download the analyzer. | +| `FUZZAPI_VERSION` | Specify API Fuzzing container version. Defaults to `latest`. | +| `FUZZAPI_TARGET_URL` | Base URL of API testing target. | +| `FUZZAPI_CONFIG` | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/276395) in GitLab 13.12, replaced with default `.gitlab/gitlab-api-fuzzing-config.yml`. API Fuzzing configuration file. | +|[`FUZZAPI_PROFILE`](#api-fuzzing-profiles) | Configuration profile to use during testing. Defaults to `Quick-10`. | +|[`FUZZAPI_OPENAPI`](#openapi-specification) | OpenAPI Specification file or URL. | +|[`FUZZAPI_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | +|[`FUZZAPI_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | +|[`FUZZAPI_POSTMAN_COLLECTION_VARIABLES`](#postman-variables) | Path to a JSON file to extract Postman variable values. | +|[`FUZZAPI_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. | +|[`FUZZAPI_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. | +|[`FUZZAPI_OVERRIDES_CMD`](#overrides) | Overrides command. | +|[`FUZZAPI_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). | +|[`FUZZAPI_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. | +|[`FUZZAPI_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. | ### Overrides @@ -788,6 +743,9 @@ To provide the overrides JSON as a file, the `FUZZAPI_OVERRIDES_FILE` CI/CD vari Here's an example `.gitlab-ci.yml`: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -806,6 +764,9 @@ This allows you to place the JSON as variables that can be masked and protected. In this example `.gitlab-ci.yml`, the `FUZZAPI_OVERRIDES_ENV` variable is set directly to the JSON: ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -820,6 +781,9 @@ In this example `.gitlab-ci.yml`, the `SECRET_OVERRIDES` variable provides the J [group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#instance-cicd-variables): ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -845,6 +809,9 @@ You must provide three CI/CD variables, each set for correct operation: - `FUZZAPI_OVERRIDES_INTERVAL`: Interval in seconds to run command. ```yaml +stages: + - fuzz + include: - template: API-Fuzzing.gitlab-ci.yml @@ -1009,7 +976,7 @@ pipelines. For more information, see the [Security Dashboard documentation](../s Fuzzing faults show up as vulnerabilities with a severity of Unknown. Once a fault is found, you can interact with it. Read more on how to -[address the vulnerabilities](../index.md#addressing-vulnerabilities). +[address the vulnerabilities](../vulnerabilities/index.md). ## Handling False Positives @@ -1026,7 +993,7 @@ False positives can be handled in two ways: ### Turn off a Check Checks perform testing of a specific type and can be turned on and off for specific configuration -profiles. The provided [configuration files](#configuration-files) define several profiles that you +profiles. The default configuration file defines several profiles that you can use. The profile definition in the configuration file lists all the checks that are active during a scan. To turn off a specific check, remove it from the profile definition in the configuration file. The profiles are defined in the `Profiles` section of the configuration file. @@ -1143,13 +1110,19 @@ Profiles: UnicodeFuzzing: true ``` +## Running API fuzzing in an offline environment + +For self-managed GitLab instances in an environment with limited, restricted, or intermittent access +to external resources through the internet, some adjustments are required for the Web API Fuzz testing job to +successfully run. For more information, see [Offline environments](../offline_deployments/index.md). + ## Troubleshooting ### Error, the OpenAPI document is not valid. Errors were found during validation of the document using the published OpenAPI schema -At the start of an API Fuzzing job the OpenAPI specification is validated against the [published schema](https://github.com/OAI/OpenAPI-Specification/tree/master/schemas). This error is shown when the provided OpenAPI specification has validation errors. Errors can be introduced when creating an OpenAPI specification manually, and also when the schema is generated. +At the start of an API Fuzzing job the OpenAPI Specification is validated against the [published schema](https://github.com/OAI/OpenAPI-Specification/tree/master/schemas). This error is shown when the provided OpenAPI Specification has validation errors. Errors can be introduced when creating an OpenAPI Specification manually, and also when the schema is generated. -For OpenAPI specifications that are generated automatically validation errors are often the result of missing code annotations. +For OpenAPI Specifications that are generated automatically validation errors are often the result of missing code annotations. **Error message** @@ -1157,9 +1130,9 @@ For OpenAPI specifications that are generated automatically validation errors ar - `OpenAPI 2.0 schema validation error ...` - `OpenAPI 3.0.x schema validation error ...` -**Solution** +**Solution** -**For generated OpenAPI specifications** +**For generated OpenAPI Specifications** 1. Identify the validation errors. 1. Use the [Swagger Editor](https://editor.swagger.io/) to identify validation problems in your specification. The visual nature of the Swagger Editor makes it easier to understand what needs to change. @@ -1170,7 +1143,7 @@ For OpenAPI specifications that are generated automatically validation errors ar **For manually created OpenAPI Specifications** 1. Identify the validation errors. - 1. The simplest solution is to use a visual tool to edit and validate the OpenAPI document. For example the [Swagger Editor](https://editor.swagger.io/) will point out schema errors and possible solutions. + 1. The simplest solution is to use a visual tool to edit and validate the OpenAPI document. For example the [Swagger Editor](https://editor.swagger.io/) highlights schema errors and possible solutions. 1. Alternatively, you can check the log output and look for schema validation warnings. They are prefixed with messages such as `OpenAPI 2.0 schema validation error` or `OpenAPI 3.0.x schema validation error`. Each failed validation provides extra information about `location` and `description`. Correct each of the validation failures and then resubmit the OpenAPI doc. Note that JSON Schema validation message might not be easy to understand. This is why we recommend the use of editors to validate document. 1. Once the validation issues are resolved, re-run your pipeline. @@ -1183,11 +1156,61 @@ The API Fuzzing engine outputs an error message when it cannot establish a conne - In [GitLab 13.11 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/323939), `Failed to start scanner session (version header not found).` - In GitLab 13.10 and earlier, `API Security version header not found. Are you sure that you are connecting to the API Security server?`. -**Solution** +**Solution** - Remove the `FUZZAPI_API` variable from the `.gitlab-ci.yml` file. The value will be inherited from the API Fuzzing CI/CD template. We recommend this method instead of manually setting a value. - If removing the variable is not possible, check to see if this value has changed in the latest version of the [API Fuzzing CI/CD template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. +### Application cannot determine the base URL for the target API + +The API Fuzzing analyzer outputs an error message when it cannot determine the target API after inspecting the OpenAPI document. This error message is shown when the target API has not been set in the `.gitlab-ci.yml`file, it is not available in the `environment_url.txt` file, and it could not be computed using the OpenAPI document. + +There is an order of precedence in which the API Fuzzing analyzer tries to get the target API when checking the different sources. First, it will try to use the `FUZZAPI_TARGET_URL`. If the environment variable has not been set, then the API Fuzzing analyzer will attempt to use the `environment_url.txt` file. If there is no file `environment_url.txt`, the API Fuzzing analyzer will then use the OpenAPI document contents and the URL provided in `FUZZAPI_OPENAPI` (if a URL is provided) to try to compute the target API. + +The best-suited solution will depend on whether or not your target API changes for each deployment. In static environments, the target API is the same for each deployment, in this case please refer to the [static environment solution](#static-environment-solution). If the target API changes for each deployment a [dynamic environment solution](#dynamic-environment-solutions) should be applied. + +#### Static environment solution + +This solution is for pipelines in which the target API URL doesn't change (is static). + +**Add environmental variable** + +For environments where the target API remains the same, we recommend you specify the target URL by using the `FUZZAPI_TARGET_URL` environment variable. In your `.gitlab-ci.yml` file, add a variable `FUZZAPI_TARGET_URL`. The variable must be set to the base URL of API testing target. For example: + +```yaml +include: + - template: API-Fuzzing.gitlab-ci.yml + + variables: + FUZZAPI_TARGET_URL: http://test-deployment/ + FUZZAPI_OPENAPI: test-api-specification.json +``` + +#### Dynamic environment solutions + +In a dynamic environment your target API changes for each different deployment. In this case, there is more than one possible solution, we recommend to use the `environment_url.txt` file when dealing with dynamic environments. + +**Use environment_url.txt** + +To support dynamic environments in which the target API URL changes during each pipeline, API Fuzzing supports the use of an `environment_url.txt` file that contains the URL to use. This file is not checked into the repository, instead it's created during the pipeline by the job that deploys the test target and collected as an artifact that can be used by later jobs in the pipeline. The job that creates the `environment_url.txt` file must run before the API Fuzzing job. + +1. Modify the test target deployment job adding the base URL in an `environment_url.txt` file at the root of your project. +1. Modify the test target deployment job collecting the `environment_url.txt` as an artifact. + +Example: + +```yaml +deploy-test-target: + script: + # Perform deployment steps + # Create environment_url.txt (example) + - echo http://${CI_PROJECT_ID}-${CI_ENVIRONMENT_SLUG}.example.org > environment_url.txt + + artifacts: + paths: + - environment_url.txt +``` + <!-- ### Target Container |