diff options
Diffstat (limited to 'doc/user/application_security/secret_detection')
| -rw-r--r-- | doc/user/application_security/secret_detection/index.md | 117 | ||||
| -rw-r--r-- | doc/user/application_security/secret_detection/post_processing.md | 179 |
2 files changed, 268 insertions, 28 deletions
diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index c5761a5743f..2ce2d59898f 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -63,7 +63,7 @@ as shown in the following table: | [Configure Secret Detection Scanners](#configuration) | **{check-circle}** | **{check-circle}** | | [Customize Secret Detection Settings](#customizing-settings) | **{check-circle}** | **{check-circle}** | | View [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** | **{check-circle}** | -| Presentation of JSON Report in Merge Request | **{dotted-circle}** | **{check-circle}** | +| Presentation of JSON Report in merge request | **{dotted-circle}** | **{check-circle}** | | View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** | **{check-circle}** | | [Interaction with Vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | | [Access to Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | @@ -182,14 +182,89 @@ Secret Detection can be customized by defining available CI/CD variables: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5. > - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/339614) support for > passthrough chains. Expanded to include additional passthrough types of `file`, `git`, and `url` in GitLab 14.6. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/235359) support for overriding rules in GitLab 14.8. You can customize the default secret detection rules provided with GitLab. +Ruleset customization supports the following capabilities that can be used +simultaneously: + +- [Disabling predefined rules](index.md#disable-predefined-analyzer-rules). +- [Overriding predefined rules](index.md#override-predefined-analyzer-rules). +- Modifying the default behavior of the Secret Detection analyzer by [synthesizing and passing a custom configuration](index.md#synthesize-a-custom-configuration). Available for only `nodejs-scan`, `gosec`, and `semgrep`. + Customization allows replacing the default secret detection rules with rules that you define. To create a custom ruleset: 1. Create a `.gitlab` directory at the root of your project, if one doesn't already exist. 1. Create a custom ruleset file named `secret-detection-ruleset.toml` in the `.gitlab` directory. + +#### Disable predefined analyzer rules + +To disable analyzer rules: + +1. Set the `disabled` flag to `true` in the context of a `ruleset` section. + +1. In one or more `ruleset.identifier` subsections, list the rules that you want disabled. Every `ruleset.identifier` section has: + + - a `type` field, to name the predefined rule identifier. + - a `value` field, to name the rule to be disabled. + +##### Example: Disable predefined rules of Secret Detection analyzer + +In the following example, the disabled rules is assigned to `secrets` +by matching the `type` and `value` of identifiers: + +```toml +[secrets] + [[secrets.ruleset]] + disable = true + [secrets.ruleset.identifier] + type = "gitleaks_rule_id" + value = "RSA private key" +``` + +#### Override predefined analyzer rules + +To override rules: + +1. In one or more `ruleset.identifier` subsections, list the rules that you want to override. Every `ruleset.identifier` section has: + + - a `type` field, to name the predefined rule identifier that the Secret Detection analyzer uses. + - a `value` field, to name the rule to be overridden. + +1. In the `ruleset.override` context of a `ruleset` section, + provide the keys to override. Any combination of keys can be + overridden. Valid keys are: + + - description + - message + - name + - severity (valid options are: Critical, High, Medium, Low, Unknown, Info) + +##### Example: Override predefined rules of Secret Detection analyzer + +In the following example, rules +are matched by the `type` and `value` of identifiers and +then overridden: + +```toml +[secrets] + [[secrets.ruleset]] + [secrets.ruleset.identifier] + type = "gitleaks_rule_id" + value = "RSA private key" + [secrets.ruleset.override] + description = "OVERRIDDEN description" + message = "OVERRIDDEN message" + name = "OVERRIDDEN name" + severity = "Info" +``` + +#### Synthesize a custom configuration + +To create a custom configuration, you can use passthrough chains. + 1. In the `secret-detection-ruleset.toml` file, do one of the following: - Define a custom ruleset: @@ -239,31 +314,8 @@ From highest to lowest severity, the logging levels are: ## Post-processing and revocation -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. - -Upon detection of a secret, GitLab supports post-processing hooks. These can be used to take actions like notifying the cloud service who issued the secret. The cloud provider can confirm the credentials and take remediation actions like revoking or reissuing a new secret and notifying the creator of the secret. Post-processing workflows vary by supported cloud providers. - -GitLab currently supports post-processing for following service providers: - -- Amazon Web Services (AWS) - -Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). Learn more about the [technical details of post-processing secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). - -NOTE: -Post-processing is currently limited to a project's default branch, see the above epic for future efforts to support additional branches. - -```mermaid -sequenceDiagram - autonumber - Rails->>+Sidekiq: gl-secret-detection-report.json - Sidekiq-->+Sidekiq: Ci::BuildFinishedWorker - Sidekiq-->+RevocationAPI: GET revocable keys types - RevocationAPI-->>-Sidekiq: OK - Sidekiq->>+RevocationAPI: POST revoke revocable keys - RevocationAPI-->>-Sidekiq: ACCEPTED - RevocationAPI-->>+Cloud Vendor: revoke revocable keys - Cloud Vendor-->>+RevocationAPI: ACCEPTED -``` +Upon detection of a secret, GitLab SaaS supports post-processing hooks. +For more information, see [Post-processing and revocation](post_processing.md). ## Full History Secret Detection @@ -316,7 +368,7 @@ registry.gitlab.com/security-products/secret-detection:3 The process for importing Docker images into a local offline Docker registry depends on **your network security policy**. Please consult your IT staff to find an accepted and approved -process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../vulnerabilities/index.md#vulnerability-scanner-maintenance) +process by which external resources can be imported or temporarily accessed. These scanners are [periodically updated](../index.md#vulnerability-scanner-maintenance) with new definitions, and you may be able to make occasional updates on your own. For details on saving and transporting Docker images as a file, see Docker's documentation on @@ -370,7 +422,7 @@ For information on this, see the [general Application Security troubleshooting s ### Error: `Couldn't run the gitleaks command: exit status 2` -If a pipeline is triggered from a Merge Request containing 60 commits while the `GIT_DEPTH` variable's +If a pipeline is triggered from a merge request containing 60 commits while the `GIT_DEPTH` variable's value is less than that, the Secret Detection job fails as the clone is not deep enough to contain all of the relevant commits. For information on the current default value, see the [pipeline configuration documentation](../../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). @@ -395,3 +447,12 @@ secret_detection: variables: GIT_DEPTH: 100 ``` + +### `secret-detection` job fails with `ERR fatal: ambiguous argument` message + +Your `secret-detection` job can fail with `ERR fatal: ambiguous argument` error if your +repository's default branch is unrelated to the branch the job was triggered for. +See issue [!352014](https://gitlab.com/gitlab-org/gitlab/-/issues/352014) for more details. + +To resolve the issue, make sure to correctly [set your default branch](../../project/repository/branches/default.md#change-the-default-branch-name-for-a-project) on your repository. You should set it to a branch +that has related history with the branch you run the `secret-detection` job on. diff --git a/doc/user/application_security/secret_detection/post_processing.md b/doc/user/application_security/secret_detection/post_processing.md new file mode 100644 index 00000000000..972558c3b95 --- /dev/null +++ b/doc/user/application_security/secret_detection/post_processing.md @@ -0,0 +1,179 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Secret Detection post-processing and revocation **(FREE SAAS)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4639) in GitLab 13.6. + +GitLab supports running post-processing hooks after detecting a secret. These +hooks can perform actions, like notifying the cloud service that issued the secret. +The cloud provider can then confirm the credentials and take remediation actions, like: + +- Revoking a secret. +- Reissuing a secret. +- Notifying the creator of the secret. + +GitLab SaaS supports post-processing for Amazon Web Services (AWS). +Post-processing workflows vary by supported cloud providers. + +Post-processing is limited to a project's default branch. The epic +[Post-processing of leaked secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). +contains: + +- Technical details of post-processing secrets. +- Discussions of efforts to support additional branches. + +NOTE: +Post-processing is currently limited to a project's default branch + +## High-level architecture + +This diagram describes how a post-processing hook revokes a secret within the GitLab application: + +```mermaid +sequenceDiagram + autonumber + GitLab Rails->>+Sidekiq: gl-secret-detection-report.json + Sidekiq-->+Sidekiq: StoreSecurityReportsWorker + Sidekiq-->+RevocationAPI: GET revocable keys types + RevocationAPI-->>-Sidekiq: OK + Sidekiq->>+RevocationAPI: POST revoke revocable keys + RevocationAPI-->>-Sidekiq: ACCEPTED + RevocationAPI-->>+Cloud Vendor: revoke revocable keys + Cloud Vendor-->>+RevocationAPI: ACCEPTED +``` + +## Integrate your cloud provider service with GitLab Saas + +Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). + +### Implement a vendor revocation receiver service + +A vendor revocation receiver service integrates with a GitLab instance to receive +a web notification and respond to leaked token requests. + +To implement a receiver service to revoke leaked tokens: + +1. Create a publicly accessible HTTP service matching the corresponding API contract + below. Your service should be idempotent and rate-limited. +1. When a pipeline corresponding to its revocable token type (in the example, `my_api_token`) + completes, GitLab sends a request to your receiver service. +1. The included URL should be publicly accessible, and contain the commit where the + leaked token can be found. For example: + + ```plaintext + POST / HTTP/2 + Accept: */* + Content-Type: application/json + X-Gitlab-Token: MYSECRETTOKEN + + [ + {"type": "my_api_token", "token":"XXXXXXXXXXXXXXXX","url": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java"} + ] + ``` + +## Implement a revocation service for self-managed + +Self-managed instances interested in using the revocation capabilities must: + +- Deploy the [RevocationAPI](#high-level-architecture). +- Configure the GitLab instance to use the RevocationAPI. + +A RevocationAPI must: + +- Match a minimal API specification. +- Provide two endpoints: + - Fetching revocable token types. + - Revoking leaked tokens. +- Be rate-limited and idempotent. + +Requests to the documented endpoints are authenticated via API tokens passed in +the `Authorization` header. Request and response bodies, if present, are +expected to have the content type `application/json`. + +All endpoints may return these responses: + +- `401 Unauthorized` +- `405 Method Not Allowed` +- `500 Internal Server Error` + +### `GET /v1/revocable_token_types` + +Returns the valid `type` values for use in the `revoke_tokens` endpoint. + +NOTE: +These values match the concatenation of [the `secrets` analyzer's](index.md) +[primary identifier](../../../development/integrations/secure.md#identifiers) by means +of concatenating the `primary_identifier.type` and `primary_identifier.value`. +In the case below, a finding identifier matches: + +```json +{"type": "gitleaks_rule_id", "name": "Gitleaks rule ID GitLab Personal Access Token", "value": "GitLab Personal Access Token"} +``` + +| Status Code | Description | +| ----- | --- | +| `200` | The response body contains the valid token `type` values. | + +Example response body: + +```json +{ + "types": ["gitleaks_rule_id_gitlab_personal_access_token"] +} +``` + +### `POST /v1/revoke_tokens` + +Accepts a list of tokens to be revoked by the appropriate provider. + +| Status Code | Description | +| ----- | --- | +| `204` | All submitted tokens have been accepted for eventual revocation. | +| `400` | The request body is invalid or one of the submitted token types is not supported. The request should not be retried. | +| `429` | The provider has received too many requests. The request should be retried later. | + +Example request body: + +```json +[{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--8GMtG8Mf4EnMJzmAWDU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java" +}, +{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--tG84EGK33nMLLDE70zU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile2.java" +}] +``` + +### Configure GitLab to interface with RevocationAPI + +You must configure the following database settings in the GitLab instance: + +- `secret_detection_token_revocation_enabled` +- `secret_detection_token_revocation_url` +- `secret_detection_token_revocation_token` +- `secret_detection_revocation_token_types_url` + +For example, to configure these values in the +[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + +```ruby +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens') +::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true) +``` + +After you configure these values, completing a pipeline performs these actions: + +1. The revocation service is triggered once. +1. A request is made to `secret_detection_revocation_token_types_url` to fetch a + list of revocable tokens. +1. Any Secret Detection findings matching the results of the `token_types` request + are included in the subsequent revocation request. |