Add latest changes from gitlab-org/gitlab@master

5 files changed, 223 insertions, 37 deletions

diff --git a/doc/install/next_steps.md b/doc/install/next_steps.md
index 3e73da123fb..0a2a1c250d6 100644
--- a/doc/install/next_steps.md
+++ b/doc/install/next_steps.md
@@ -9,11 +9,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 Here are a few resources you might want to check out after completing the
 installation.
 
-## License
+## Email and notifications
 
-- [Upload a license](../user/admin_area/license.md) or [start a free trial](https://about.gitlab.com/free-trial/):
-  Activate all GitLab Enterprise Edition functionality with a license.
-- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers.
+- [SMTP](https://docs.gitlab.com/omnibus/settings/smtp.html): Configure SMTP
+  for proper email notifications support.
+
+## CI/CD
+
+- [Set up runners](https://docs.gitlab.com/runner/): Set up one or more GitLab
+  Runners, the agents that are responsible for all of the GitLab CI/CD features.
+- [GitLab Pages](../administration/pages/index.md): Configure GitLab Pages to
+  allow hosting of static sites.
+- [GitLab Registry](../administration/packages/container_registry.md): With the
+  GitLab Container Registry, every project can have its own space to store Docker
+  images.
 
 ## Security
 
@@ -24,11 +33,7 @@ installation.
 
 - [LDAP](../administration/auth/ldap/index.md): Configure LDAP to be used as
   an authentication mechanism for GitLab.
-
-## Email and notifications
-
-- [SMTP](https://docs.gitlab.com/omnibus/settings/smtp.html): Configure SMTP
-  for proper email notifications support.
+- [SAML and OAuth](../integration/omniauth.md): Authenticate via online services like Okta, Google, Azure AD, and more.
 
 ## Backup and upgrade
 
@@ -40,15 +45,16 @@ installation.
   policies governing version naming, as well as release pace for major, minor, patch,
   and security releases.
 
-## CI/CD
+## License
 
-- [Set up runners](https://docs.gitlab.com/runner/): Set up one or more GitLab
-  Runners, the agents that are responsible for all of the GitLab CI/CD features.
-- [GitLab Pages](../administration/pages/index.md): Configure GitLab Pages to
-  allow hosting of static sites.
-- [GitLab Registry](../administration/packages/container_registry.md): With the
-  GitLab Container Registry, every project can have its own space to store Docker
-  images.
+- [Upload a license](../user/admin_area/license.md) or [start a free trial](https://about.gitlab.com/free-trial/):
+  Activate all GitLab Enterprise Edition functionality with a license.
+- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers.
+
+## Cross-repo Code Search
+
+- [Advanced Search](../integration/elasticsearch.md): Leverage Elasticsearch for
+  faster, more advanced code search across your entire GitLab instance.
 
 ## Scaling and replication
 
@@ -56,8 +62,3 @@ installation.
   GitLab supports several different types of clustering.
 - [Geo replication](../administration/geo/index.md):
   Geo is the solution for widely distributed development teams.
-
-## Search
-
-- [Advanced Search](../integration/elasticsearch.md): Leverage Elasticsearch for
-  faster, more advanced code search across your entire GitLab instance.
diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md
index b8c3529225c..aaf4496076f 100644
--- a/doc/user/application_security/dast/browser_based.md
+++ b/doc/user/application_security/dast/browser_based.md
@@ -38,9 +38,10 @@ An example configuration might look like the following:
 include:
   - template: DAST.gitlab-ci.yml
 
-variables:
-  DAST_WEBSITE: "https://example.com"
-  DAST_BROWSER_SCAN: "true"
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
 ```
 
 ### Available variables
@@ -64,8 +65,12 @@ The browser-based crawler can be configured using CI/CD variables.
 | `DAST_PASSWORD`                      | string          | `p@55w0rd`                        | The password to enter into the password field on the sign-in HTML form. |
 | `DAST_USERNAME_FIELD`                | selector        | `id:user`                         | A selector describing the username field on the sign-in HTML form. |
 | `DAST_PASSWORD_FIELD`                | selector        | `css:.password-field`             | A selector describing the password field on the sign-in HTML form. | 
-| `DAST_SUBMIT_FIELD`                  | selector        | `xpath://input[@value='Login']`   | A selector describing the element that when clicked submits the login form or the password form of a multi-page login process. |
+| `DAST_SUBMIT_FIELD`                  | selector        | `xpath://input[@value='Login']`   | A selector describing the element that when clicked submits the login form, or the password form of a multi-page login process. |
 | `DAST_FIRST_SUBMIT_FIELD`            | selector        | `.submit`                         | A selector describing the element that when clicked submits the username form of a multi-page login process. |
+| `DAST_BROWSER_AUTH_REPORT`           | boolean         | `true`                            | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. |
+| `DAST_AUTH_VERIFICATION_URL`                | URL      | `http://site.com/welcome`         | Verifies successful authentication by checking the URL once the login form has been submitted. |
+| `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`   | selector | `css:.user-photo`                | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. |
+| `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM` | boolean  | `true`                            | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. |
 
 The [DAST variables](index.md#available-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`,
 `DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans.   
@@ -75,13 +80,47 @@ The [DAST variables](index.md#available-variables) `SECURE_ANALYZERS_PREFIX`, `D
 Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser.
 Selectors have the format `type`:`search string`. The crawler will search for the selector using the search string based on the type.
 
-| Selector type | Example                        | Description |
-| ------------- | ------------------------------ | ----------- |
-| `css`         | `css:.password-field`          | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. |
-| `id`          | `id:element`                   | Searches for an HTML element with the provided element ID. |
-| `name`        | `name:element`                 | Searches for an HTML element with the provided element name. |
-| `xpath`       | `xpath://*[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. |
-| None provided | `a.click-me`                   | Defaults to searching using a CSS selector. |
+| Selector type | Example                            | Description |
+| ------------- | ---------------------------------- | ----------- |
+| `css`         | `css:.password-field`              | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. |
+| `id`          | `id:element`                       | Searches for an HTML element with the provided element ID. |
+| `name`        | `name:element`                     | Searches for an HTML element with the provided element name. |
+| `xpath`       | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. |
+| None provided | `a.click-me`                       | Defaults to searching using a CSS selector. |
+
+##### Find selectors with Google Chrome
+
+Chrome DevTools element selector tool is an effective way to find a selector. 
+
+1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site.
+1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows.
+1. Select the `Select an element in the page to select it` tool.
+   ![search-elements](img/dast_auth_browser_scan_search_elements.png)
+1. Select the field on your page that you would like to know the selector for.
+1. Once the tool is active, highlight a field you wish to view the details of.
+   ![highlight](img/dast_auth_browser_scan_highlight.png)
+1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector.
+
+In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting `DAST_USERNAME_FIELD: "css:[id=user_login]"`, or more simply, `DAST_USERNAME_FIELD: "id:user_login"`.
+
+##### Choose the right selector
+
+Judicious choice of selector leads to a scan that is resilient to the application changing. 
+
+In order of preference, it is recommended to choose as selectors:
+
+- `id` fields. These are generally unique on a page, and rarely change.
+- `name` fields. These are generally unique on a page, and rarely change.
+- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field.   
+- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field.
+- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`.   
+
+When using selectors to locate specific fields we recommend you avoid searching on:
+
+- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated.
+- Generic class names, such as `column-10` and `dark-grey`.
+- XPath searches as they are less performant than other selector searches.
+- Unscoped searches, such as those beginning with `css:*` and `xpath://*`.
 
 ## Vulnerability detection
 
@@ -93,6 +132,151 @@ When running a full scan, active vulnerability checks executed by DAST/ZAP do no
 For example, for a target website that contains forms with Anti-CSRF tokens, a passive scan will scan as intended because the browser displays pages/forms as if a user is viewing the page.
 However, active vulnerability checks run in a full scan will not be able to submit forms containing Anti-CSRF tokens. In such cases we recommend you disable Anti-CSRF tokens when running a full scan.
 
+## Authentication
+
+If your application hosts content only available to logged in users then you will likely get much higher crawl coverage of your application by configuring authentication.
+We recommended you periodically confirm that authentication is still working as this tends to break over
+time due to changes to the application.
+
+Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL.
+
+An example configuration that authenticates a user might look like the following:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    DAST_AUTH_URL: "https://login.example.com/"
+    DAST_USERNAME: "admin"
+    DAST_PASSWORD: "P@55w0rd!"
+    DAST_USERNAME_FIELD: "name:username"
+    DAST_PASSWORD_FIELD: "name:password"
+    DAST_SUBMIT_FIELD: "css:button[type='submit']"
+```  
+
+### Log in using automatic detection of the login form
+
+By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, the browser-based crawler will attempt to authenticate to the
+target application by locating the login form based on a determination about whether or not the form contains username or password fields.
+
+Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user.
+
+Login process:
+
+1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located.
+   1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is clicked and the user is logged in.
+   1. If a form contains only a username field, it is assumed that the login form is multi-step.
+      1. The `DAST_USERNAME` is inputted into the username field and the form submit button is clicked. 
+      1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is clicked and the user is logged in. 
+
+### Log in using explicit selection of the login form
+
+By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login,
+the browser-based crawler will attempt to authenticate to the target application by locating the login form based on the selectors provided.
+Most applications will benefit from this approach to authentication.
+
+Login process:
+
+1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located.
+   1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is clicked and the user is logged in.
+   1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step.
+      1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is clicked.
+      1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is clicked and the user is logged in.
+      
+### Verifying successful login
+
+Once the login form has been submitted, the browser-based crawler determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt.
+
+Following the submission of the login form, authentication is determined to be unsuccessful when:
+ 
+- A `400` or `500` series HTTP response status code is returned.
+- A new cookie/browser storage value determined to be sufficiently random has not been set.
+
+In addition to these checks, the user can configure their own verification checks.
+Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked.
+
+#### Verifying based on the URL
+
+When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable.
+If these are not exactly the same, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome"
+```  
+
+#### Verify based on presence of an element
+
+When `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable.
+If no element is found, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast: 
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user"
+```  
+
+#### Verify based on presence of a login form
+
+When `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form.
+If any such form is found, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM: "true"
+```  
+
+### Configure the authentication debug output
+
+It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline.
+To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact.
+This HTML report contains all steps the browser crawler took, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots.
+
+![dast-auth-report](img/dast_auth_report.jpg)
+
+An example configuration where the authentication debug report is exported may look like the following:
+
+```yaml
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_REPORT: "true"
+  artifacts:
+    paths: [gl-dast-debug-auth-report.html]
+```
+
 ## Managing scan time
 
 It is expected that running the browser-based crawler will result in better coverage for many web applications, when compared to the normal GitLab DAST solution.
@@ -116,10 +300,11 @@ For example, the following job definition enables the browsing module and the au
 include:
   - template: DAST.gitlab-ci.yml
 
-variables:
-  DAST_WEBSITE: "https://my.site.com"
-  DAST_BROWSER_SCAN: "true"
-  DAST_BROWSER_LOG: "brows:debug,auth:debug"
+dast:
+  variables:
+    DAST_WEBSITE: "https://my.site.com"
+    DAST_BROWSER_SCAN: "true"
+    DAST_BROWSER_LOG: "brows:debug,auth:debug"
 ```
 
 ### Log message format
diff --git a/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png b/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png
new file mode 100644
index 00000000000..132c9f9c991
--- /dev/null
+++ b/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png
diff --git a/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png b/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png
new file mode 100644
index 00000000000..4e1dca626bc
--- /dev/null
+++ b/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png
diff --git a/doc/user/application_security/dast/img/dast_auth_report.jpg b/doc/user/application_security/dast/img/dast_auth_report.jpg
new file mode 100644
index 00000000000..5d9d98045ef
--- /dev/null
+++ b/doc/user/application_security/dast/img/dast_auth_report.jpg