diff options
Diffstat (limited to 'doc/user/application_security/api_fuzzing/create_har_files.md')
| -rw-r--r-- | doc/user/application_security/api_fuzzing/create_har_files.md | 234 |
1 files changed, 234 insertions, 0 deletions
diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md new file mode 100644 index 00000000000..220d00adc7b --- /dev/null +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -0,0 +1,234 @@ +--- +stage: Secure +group: Fuzz Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +# HTTP Archive format + +HTTP Archive (HAR) format files are an industry standard for exchanging information about HTTP +requests and HTTP responses. A HAR file's content is JSON formatted, containing browser interactions +with a web site. The file extension `.har` is commonly used. + +The HAR files can be used to perform [web API Fuzz Testing](index.md#http-archive-har) as part of +your [GitLab CI/CD](../../../ci/README.md) pipelines. + +WARNING: +**DANGER** A HAR file stores information exchanged between web client and web server. It could also +store sensitive information such as authentication tokens, API keys, and session cookies. We +recommend that you review the HAR file contents before adding them to a repository. + +## HAR file creation + +You can create HAR files manually or by using a specialized tool for recording web sessions. We +recommend using a specialized tool. However, it is important to make sure files created by these +tools do not expose sensitive information, and can be safely used. + +The following tools can be used generate a HAR file based on your network activity. They +automatically record your network activity and generate the HAR file: + +1. [GitLab HAR Recorder](#gitlab-har-recorder). +1. [Insomnia API Client](#insomnia-api-client). +1. [Fiddler debugging proxy](#fiddler-debugging-proxy). +1. [Safari web browser](#safari-web-browser). +1. [Chrome web browser](#chrome-web-browser). +1. [Firefox web browser](#firefox-web-browser). + +WARNING: +**DANGER** HAR files may contain sensitive information such as authentication tokens, API keys, and +session cookies. We recommend that you review the HAR file contents before adding them to a +repository. + +### GitLab HAR Recorder + +[GitLab HAR Recorder](https://gitlab.com/gitlab-org/security-products/har-recorder) is a command +line tool for recording HTTP messages and saving them to HTTP Archive (HAR) files. For more details +about the GitLab HAR Recorder, see the [homepage](https://gitlab.com/gitlab-org/security-products/har-recorder). + +#### Install GitLab HAR Recorder + +Prerequisites: + +- Install Python 3.6 or greater. +- For Microsoft Windows, you must also install `Microsoft Visual C++ 14.0`. It's included with + *Build Tools for Visual Studio* from [Visual Studio Downloads page](https://visualstudio.microsoft.com/downloads/). +- Install HAR Recorder. + +Install GitLab HAR Recorder: + + ```shell + pip install gitlab-har-recorder --extra-index-url https://gitlab.com/api/v4/projects/22441624/packages/pypi/simple + ``` + +#### Create a HAR file with GitLab HAR Recorder + +1. Start recorder with the proxy port and HAR filename. +1. Complete the browser actions, using the proxy. + 1. Make sure proxy is used! +1. Stop the recorder. + +To verify the HAR contains all requests, use the [HAR Viewer (online)](http://www.softwareishard.com/har/viewer/). +[Google Admin Toolbox HAR Analyzer](https://toolbox.googleapps.com/apps/har_analyzer/) + +### Insomnia API Client + +[Insomnia API Client](https://insomnia.rest/) is an API design tool that among many uses, helps +you to design, describe, and test your API. You can also use it to generate HAR files that can be +used in [Web API Fuzz Testing](index.md#http-archive-har). + +#### Create a HAR file with the Insomnia API Client + +1. Define or import your API. + - Postman v2. + - Curl. + - OpenAPI v2, v3. +1. Verify each API call works. + - If you imported an OpenAPI specification, go through and add working data. +1. Select **API > Import/Export**. +1. Select **Export Data > Current Workspace**. +1. Select requests to include in the HAR file. +1. Select **Export**. +1. In the **Select Export Type** dropdown select **HAR -- HTTP Archive Format**. +1. Select **Done**. +1. Enter a location and filename for the HAR file. + +### Fiddler debugging proxy + +[Fiddler](https://www.telerik.com/fiddler) is a web debugger tool. It captures HTTP and HTTP(S) +network traffic and allows you to examine each request. It also lets you export the requests and +responses in HAR format. + +#### Create a HAR file with Fiddler + +1. Go to the [Fiddler home page](https://www.telerik.com/fiddler) and sign in. If you don't already +have an account, first create an account. +1. Browse pages that call an API. Fiddler automatically captures the requests. +1. Select one or more requests, then from the context menu, select **Export > Selected Sessions**. +1. In the **Choose Format** dropdown select **HTTPArchive v1.2**. +1. Enter a filename and select **Save**. + +Fiddler shows a popup message confirming the export has succeeded. + +### Safari web browser + +[Safari](https://www.apple.com/safari/) is a web browser maintained by Apple. As web development +evolves, browsers support new capabilities. With Safari you can explore network traffic and +export it as a HAR file. + +#### Create a HAR file with Safari + +Prerequisites: + +- Enable the `Develop` menu item. + 1. Open Safari's preferences. Press <kbd>Command</kbd>+<kbd>,</kbd> or from the menu, select + **Safari > Preferences...**. + 1. Select **Advanced** tab, then select `Show Develop menu item in menu bar`. + 1. Close the **Preferences** window. + +1. Open the **Web Inspector**. Press <kbd>Option</kbd>+<kbd>Command</kbd>+<kbd>i</kbd>, or from the + menu, select **Develop > Show Web Inspector**. +1. Select the **Network** tab, and select **Preserve Log**. +1. Browse pages that call the API. +1. Open the **Web Inspector** and select the **Network** tab +1. Right-click on the request to export and select **Export HAR**. +1. Enter a filename and select **Save**. + +### Chrome web browser + +[Chrome](https://www.google.com/chrome/) is a web browser maintained by Google. As web development +evolves, browsers support new capabilities. With Chrome you can explore network traffic and +export it as a HAR file. + +#### Create a HAR file with Chrome + +1. From the Chrome context menu, select **Inspect**. +1. Select the **Network** tab. +1. Select **Preserve log**. +1. Browse pages that call the API. +1. Select one or more requests. +1. Right click and select **Save all as HAR with content**. +1. Enter a filename and select **Save**. +1. To append additional requests, select and save them to the same file. + +### Firefox Web Browser + +[Firefox](https://www.mozilla.org/en-US/firefox/new/) is a web browser maintained by Mozilla. As web +development evolves, browsers support new capabilities. With Firefox you can explore network traffic +and export it as a HAR file. + +#### Create a HAR file with Firefox + +1. From the Firefox context menu, select **Inspect**. +1. Select the **Network** tab. +1. Browse pages that call the API. +1. Check the **Network** tab and confirm requests are being recorded. If there is a message + `Perform a request or Reload the page to see detailed information about network activity`, + select **Reload** to start recording requests. +1. Select one or more requests. +1. Right click and select **Save All As HAR**. +1. Enter a filename and select **Save**. +1. To append additional requests, select and save them to the same file. + +## HAR verification + +Before using HAR files it's important to make sure they don't expose any sensitive information. + +For each HAR file you should: + +- View the HAR file's content +- Review the HAR file for sensitive information +- Edit or remove sensitive information + +### View HAR file contents + +We recommend viewing a HAR file's content in a tool that can present its content in a structured +way. Several HAR file viewers are available online. If you would prefer not to upload the HAR file, +you can use a tool installed on your computer. HAR files used JSON format, so can also be viewed in +a text editor. + +Tools recommended for viewing HAR files include: + +- [HAR Viewer](http://www.softwareishard.com/har/viewer/) - (online) +- [Google Admin Toolbox HAR Analyzer](https://toolbox.googleapps.com/apps/har_analyzer/) - (online) +- [Fiddler](https://www.telerik.com/fiddler) - local +- [Insomnia API Client](https://insomnia.rest/) - local + +## Review HAR file content + +Review the HAR file for any of the following: + +- Information that could help to grant access to your application, for example: authentication + tokens, authentication tokens, cookies, API keys. +- [Personally Identifiable Information (PII)](https://en.wikipedia.org/wiki/Personal_data). + +We strongly recommended that you [edit or remove it](#edit-or-remove-sensitive-information) any +sensitive information. + +Use the following as a checklist to start with. Note that it's not an exhaustive list. + +- Look for secrets. For example: if your application requires authentication, check common locations + or authentication information: + - Authentication related headers. For example: cookies, authorization. These headers could contain + valid information. + - A request related to authentication. The body of these requests might contain information such + as user credentials or tokens. + - Session tokens. Session tokens could grant access to your application. The location of these + token could vary. They could be in headers, query parameters or body. +- Look for Personally Identifiable Information + - For example, if your application retrieves a list of users and their personal data: phones, + names, emails. + - Authentication information might also contain personal information. + +## Edit or remove sensitive information + +Edit or remove sensitive information found during the [HAR file content review](#review-har-file-content). +HAR files are JSON files and can be edited in any text editor. + +After editing the HAR file, open it in a HAR file viewer to verify its formatting and structure are +intact. + +The following example demonstrates use of [Visual Studio Code](https://code.visualstudio.com/) text +editor to edit an Authorization token found in a header. + + |