diff options
Diffstat (limited to 'doc/user/application_security/dast_api/index.md')
| -rw-r--r-- | doc/user/application_security/dast_api/index.md | 200 |
1 files changed, 187 insertions, 13 deletions
diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index f1eba505589..cc20b49764f 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -69,8 +69,8 @@ starting in GitLab 14.0, GitLab will not check your repository's root for config ### OpenAPI Specification -> Support for OpenAPI Specification using YAML format was -> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> - Support to generate media type `application/xml` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.8. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. @@ -82,6 +82,7 @@ the body generation is limited to these body types: - `application/x-www-form-urlencoded` - `multipart/form-data` - `application/json` +- `application/xml` Follow these steps to configure DAST API in GitLab with an OpenAPI specification: @@ -478,6 +479,9 @@ Follow these steps to provide the bearer token with `DAST_API_OVERRIDES_ENV`: `{"headers":{"Authorization":"Bearer dXNlcm5hbWU6cGFzc3dvcmQ="}}` (substitute your token). You can create CI/CD variables from the GitLab projects page at **Settings > CI/CD**, in the **Variables** section. + Due to the format of `TEST_API_BEARERAUTH` it's not possible to mask the variable. + To mask the token's value, you can create a second variable with the token value's, and define + `TEST_API_BEARERAUTH` with the value `{"headers":{"Authorization":"Bearer $MASKED_VARIABLE"}}`. 1. In your `.gitlab-ci.yml` file, set `DAST_API_OVERRIDES_ENV` to the variable you just created: @@ -529,7 +533,7 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json ``` To validate that authentication is working, run an DAST API test and review the job logs and @@ -571,13 +575,12 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json DAST_API_OVERRIDES_CMD: renew_token.py DAST_API_OVERRIDES_INTERVAL: 300 ``` -To validate that authentication is working, run an DAST API test and review the job logs and -the test API's application logs. +To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. See the [overrides section](#overrides) for more information about override commands. ### Configuration files @@ -644,6 +647,9 @@ can be added, removed, and modified by creating a custom configuration. |[`DAST_API_OVERRIDES_FILE`](#overrides) | Path to a JSON file containing overrides. | |[`DAST_API_OVERRIDES_ENV`](#overrides) | JSON string containing headers to override. | |[`DAST_API_OVERRIDES_CMD`](#overrides) | Overrides command. | +|[`DAST_API_OVERRIDES_CMD_VERBOSE`](#overrides) | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.6. | +|`DAST_API_PRE_SCRIPT` | Run user command or script before scan session starts. | +|`DAST_API_POST_SCRIPT` | Run user command or script after scan session has finished. | |[`DAST_API_OVERRIDES_INTERVAL`](#overrides) | How often to run overrides command in seconds. Defaults to `0` (once). | |[`DAST_API_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. | |[`DAST_API_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. | @@ -825,7 +831,7 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json ``` #### Using a CI/CD variable @@ -869,17 +875,29 @@ variables: #### Using a command If the value must be generated or regenerated on expiration, you can provide a program or script for -the DAST API scanner to execute on a specified interval. The provided script runs in an Alpine Linux -container that has Python 3 and Bash installed. If the Python script requires additional packages, -it must detect this and install the packages at runtime. The script creates the overrides JSON file -as defined above. +the DAST API scanner to execute on a specified interval. The provided command runs in an Alpine Linux +container that has Python 3 and Bash installed. + +You have to set the environment variable `DAST_API_OVERRIDES_CMD` to the program or script you would like +to execute. The provided command creates the overrides JSON file as defined previously. + +You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for +your overrides command. In this case, we recommend setting the `DAST_API_PRE_SCRIPT` to the file path of a script which +provides those prerequisites. The script provided by `DAST_API_PRE_SCRIPT` is executed once, before the analyzer starts. + +See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) +page for information about installing Alpine Linux packages. You must provide three CI/CD variables, each set for correct operation: - `DAST_API_OVERRIDES_FILE`: File generated by the provided command. -- `DAST_API_OVERRIDES_CMD`: Command to generate JSON file. +- `DAST_API_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically. - `DAST_API_OVERRIDES_INTERVAL`: Interval in seconds to run command. +Optionally: + +- `DAST_API_PRE_SCRIPT`: Script to install runtimes or dependencies before the scan starts. + ```yaml stages: - dast @@ -891,11 +909,167 @@ variables: DAST_API_PROFILE: Quick DAST_API_OPENAPI: test-api-specification.json DAST_API_TARGET_URL: http://test-deployment/ - DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json + DAST_API_OVERRIDES_FILE: dast-api-overrides.json DAST_API_OVERRIDES_CMD: renew_token.py DAST_API_OVERRIDES_INTERVAL: 300 ``` +#### Debugging overrides + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. + +By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `DAST_API_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing. + +It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and following a naming convention. + +Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished. + +Following our example, we provided `renew_token.py` in the environment variable `DAST_API_OVERRIDES_CMD`. Please notice two things in the script: + +- Log file is saved in the location indicated by the environmental variable `CI_PROJECT_DIR`. +- Log file name should match `gl-*.log`. + +```python +#!/usr/bin/env python + +# Example of an overrides command + +# Override commands can update the overrides json file +# with new values to be used. This is a great way to +# update an authentication token that will expire +# during testing. + +import logging +import json +import os +import requests +import backoff + +# [1] Store log file in directory indicated by env var CI_PROJECT_DIR +working_directory = os.environ['CI_PROJECT_DIR'] + +# [2] File name should match the pattern: gl-*.log +log_file_path = os.path.join(working_directory, 'gl-user-overrides.log') + +# Set up logger +logging.basicConfig(filename=log_file_path, level=logging.DEBUG) + +# Use `backoff` decorator to retry in case of transient errors. +@backoff.on_exception(backoff.expo, + (requests.exceptions.Timeout, + requests.exceptions.ConnectionError), + max_time=30) +def get_auth_response(): + return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD'])) + +# In our example, access token is retrieved from a given endpoint +try: + + # Performs a http request, response sample: + # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" } + response = get_auth_response() + + # Check that the request is successful. may raise `requests.exceptions.HTTPError` + response.raise_for_status() + + # Gets JSON data + response_body = response.json() + +# If needed specific exceptions can be caught +# requests.ConnectionError : A network connection error problem occurred +# requests.HTTPError : HTTP request returned an unsuccessful status code. [Response.raise_for_status()] +# requests.ConnectTimeout : The request timed out while trying to connect to the remote server +# requests.ReadTimeout : The server did not send any data in the allotted amount of time. +# requests.TooManyRedirects : The request exceeds the configured number of maximum redirections +# requests.exceptions.RequestException : All exceptions that related to Requests +except requests.exceptions.RequestException as requests_error: + # logs exceptions related to `Requests` + logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}') + raise +except requests.exceptions.JSONDecodeError as json_decode_error: + # logs errors related decoding JSON response + logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') + raise +except Exception as e: + # logs any other error + logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') + raise + +# computes object that holds overrides file content. +# It uses data fetched from request +overrides_data = { + "headers": { + "Authorization": f"Token {response_body['Token']}" + } +} + +# log entry informing about the file override computation +# the location of the overrides json file is also CI_PROJECT_DIR +overrides_file_path = os.path.join( + working_directory, "dast-api-overrides.json") +logging.info("Creating overrides file: %s" % overrides_file_path) + +# attempts to overwrite the file +try: + if os.path.exists(overrides_file_path): + os.unlink(overrides_file_path) + + # overwrites the file with our updated dictionary + with open(overrides_file_path, "wb+") as fd: + fd.write(json.dumps(overrides_data).encode('utf-8')) +except Exception as e: + # logs any other error + logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}') + raise + +# logs informing override has finished successfully +logging.info("Override file has been updated") + +# end +``` + +In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `DAST_API_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command. +As for example, the following script `user-pre-scan-set-up.sh` + +```shell +#!/bin/bash + +# user-pre-scan-set-up.sh +# Ensures python dependencies are installed + +echo "**** install python dependencies ****" + +python3 -m ensurepip +pip3 install --no-cache --upgrade \ + pip \ + backoff + +echo "**** python dependencies installed ****" + +# end +``` + +You have to update your configuration to set the `DAST_API_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_PRE_SCRIPT: user-pre-scan-set-up.sh + DAST_API_OVERRIDES_FILE: dast-api-overrides.json + DAST_API_OVERRIDES_CMD: renew_token.py + DAST_API_OVERRIDES_INTERVAL: 300 +``` + +In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command. + ### Exclude Paths > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. |