diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-06 06:07:35 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-06 06:07:35 +0300 |
commit | bdb97ece6809b945927d50effcdf9d284634d4d5 (patch) | |
tree | a5bf5284a9c233f47ca65ce5790a8276b3fade3b /doc | |
parent | bd9860f6911096e8ffe269797b90edd611f9b40f (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/development/architecture.md | 1 | ||||
-rw-r--r-- | doc/development/sec/token_revocation_api.md | 118 | ||||
-rw-r--r-- | doc/user/application_security/secret_detection/index.md | 7 | ||||
-rw-r--r-- | doc/user/application_security/secret_detection/post_processing.md | 203 |
4 files changed, 182 insertions, 147 deletions
diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 2bb5ab6bca6..984ac49c514 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -378,6 +378,7 @@ Component statuses are linked to configuration documentation for each component. | [Runner](#gitlab-runner) | Executes GitLab CI/CD jobs | ⤓ | ⤓ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE | | [Sentry integration](#sentry) | Error tracking for deployed apps | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | CE & EE | +| [Token Revocation API](sec/token_revocation_api.md) | Receives and revokes leaked secrets | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | EE Only | ### Component details diff --git a/doc/development/sec/token_revocation_api.md b/doc/development/sec/token_revocation_api.md new file mode 100644 index 00000000000..15d1d2d0ef3 --- /dev/null +++ b/doc/development/sec/token_revocation_api.md @@ -0,0 +1,118 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Token Revocation API + +The Token Revocation API is an externally-deployed HTTP API that interfaces with GitLab +to receive and revoke API tokens and other secrets detected by GitLab Secret Detection. +See the [high-level architecture](../../user/application_security/secret_detection/post_processing.md) +to understand the Secret Detection post-processing and revocation flow. + +GitLab.com uses the internally-maintained [Secret Revocation Service](https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/secret-revocation-service) +(team-members only) as its Token Revocation API. For GitLab self-managed, you can create +your own API and configure GitLab to use it. + +## Implement a Token Revocation API for self-managed + +GitLab self-managed instances interested in using the revocation capabilities must: + +- Implement and deploy your own Token Revocation API. +- Configure the GitLab instance to use the Token Revocation API. + +Your service must: + +- Match the API specification below. +- Provide two endpoints: + - Fetching revocable token types. + - Revoking leaked tokens. +- Be rate-limited and idempotent. + +Requests to the documented endpoints are authenticated using API tokens passed in +the `Authorization` header. Request and response bodies, if present, are +expected to have the content type `application/json`. + +All endpoints may return these responses: + +- `401 Unauthorized` +- `405 Method Not Allowed` +- `500 Internal Server Error` + +### `GET /v1/revocable_token_types` + +Returns the valid `type` values for use in the `revoke_tokens` endpoint. + +NOTE: +These values match the concatenation of [the `secrets` analyzer's](../../user/application_security/secret_detection/index.md) +[primary identifier](../integrations/secure.md#identifiers) by means +of concatenating the `primary_identifier.type` and `primary_identifier.value`. +For example, the value `gitleaks_rule_id_gitlab_personal_access_token` matches the following finding identifier: + +```json +{"type": "gitleaks_rule_id", "name": "Gitleaks rule ID GitLab Personal Access Token", "value": "GitLab Personal Access Token"} +``` + +| Status Code | Description | +| ----- | --- | +| `200` | The response body contains the valid token `type` values. | + +Example response body: + +```json +{ + "types": ["gitleaks_rule_id_gitlab_personal_access_token"] +} +``` + +### `POST /v1/revoke_tokens` + +Accepts a list of tokens to be revoked by the appropriate provider. Your service is responsible for communicating +with each provider to revoke the token. + +| Status Code | Description | +| ----- | --- | +| `204` | All submitted tokens have been accepted for eventual revocation. | +| `400` | The request body is invalid or one of the submitted token types is not supported. The request should not be retried. | +| `429` | The provider has received too many requests. The request should be retried later. | + +Example request body: + +```json +[{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--8GMtG8Mf4EnMJzmAWDU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java" +}, +{ + "type": "gitleaks_rule_id_gitlab_personal_access_token", + "token": "glpat--tG84EGK33nMLLDE70zU", + "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile2.java" +}] +``` + +### Configure GitLab to interface with the Token Revocation API + +You must configure the following database settings in the GitLab instance: + +| Setting | Type | Description | +| ------- | ---- | ----------- | +| `secret_detection_token_revocation_enabled` | Boolean | Whether automatic token revocation is enabled | +| `secret_detection_token_revocation_url` | String | A fully-qualified URL to the `/v1/revoke_tokens` endpoint of the Token Revocation API | +| `secret_detection_revocation_token_types_url` | String | A fully-qualified URL to the `/v1/revocable_token_types` endpoint of the Token Revocation API | +| `secret_detection_token_revocation_token` | String | A pre-shared token to authenticate requests to the Token Revocation API | + +For example, to configure these values in the +[Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session): + +```ruby +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens') +::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types') +::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true) +``` + +After you configure these values, the Token Revocation API will be called according to the +[high-level architecture](../../user/application_security/secret_detection/post_processing.md#high-level-architecture) +diagram. diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index d316ed7a9de..d6aab71a2c6 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -195,7 +195,12 @@ Pipelines now include a Secret Detection job. ## Responding to a leaked secret -If the scanner detects a secret you should rotate it immediately. [Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) may not be effective in removing all references to the file. Also, the secret remains in any forks of the repository. +Secrets detected by the analyzer should be immediately rotated. +[Purging a file from the repository's history](../../project/repository/reducing_the_repo_size_using_git.md#purge-files-from-repository-history) +may not be effective in removing all references to the file. Additionally, the secret will remain in any existing +forks or clones of the repository. + +GitLab will attempt to [automatically revoke](post_processing.md) some types of leaked secrets. ## Pinning to specific analyzer version diff --git a/doc/user/application_security/secret_detection/post_processing.md b/doc/user/application_security/secret_detection/post_processing.md index 6e9b77426ad..10742114b90 100644 --- a/doc/user/application_security/secret_detection/post_processing.md +++ b/doc/user/application_security/secret_detection/post_processing.md @@ -10,26 +10,28 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Disabled by default for GitLab personal access tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/371658) in GitLab 15.6 [with a flag](../../../administration/feature_flags.md) named `gitlab_pat_auto_revocation`. Available to GitLab.com only. > - [Enabled by default for GitLab personal access tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/371658) in GitLab 15.9 -GitLab supports running post-processing hooks after detecting a secret. These -hooks can perform actions, like notifying the cloud service that issued the secret. -The cloud provider can then confirm the credentials and take remediation actions, like: +GitLab.com and self-managed supports running post-processing hooks after detecting a secret. These +hooks can perform actions, like notifying the vendor that issued the secret. +The vendor can then confirm the credentials and take remediation actions, like: - Revoking a secret. - Reissuing a secret. - Notifying the creator of the secret. -GitLab SaaS supports post-processing for [GitLab personal access tokens](../../profile/personal_access_tokens.md) and Amazon Web Services (AWS). -Post-processing workflows vary by supported cloud providers. +GitLab supports post-processing for the following vendors and secrets: -Post-processing is limited to a project's default branch. The epic -[Post-processing of leaked secrets](https://gitlab.com/groups/gitlab-org/-/epics/4639). -contains: +| Vendor | Secret | GitLab.com | Self-managed | +| ----- | --- | --- | --- | +| GitLab | [Personal access tokens](../../profile/personal_access_tokens.md) | ✅ | ✅ 15.9 and later | +| Amazon Web Services (AWS) | [IAM access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) | ✅ | ⚙ | -- Technical details of post-processing secrets. -- Discussions of efforts to support additional branches. +**Component legend** + +- ✅ - Available by default +- ⚙ - Requires manual integration using a [Token Revocation API](../../../development/sec/token_revocation_api.md) NOTE: -Post-processing is currently limited to a project's default branch +Post-processing is limited to a project's default branch. ## High-level architecture @@ -40,142 +42,51 @@ sequenceDiagram autonumber GitLab Rails->>+Sidekiq: gl-secret-detection-report.json Sidekiq-->+Sidekiq: StoreSecurityReportsWorker - Sidekiq-->+RevocationAPI: GET revocable keys types - RevocationAPI-->>-Sidekiq: OK - Sidekiq->>+RevocationAPI: POST revoke revocable keys - RevocationAPI-->>-Sidekiq: ACCEPTED - RevocationAPI-->>+Cloud Vendor: revoke revocable keys - Cloud Vendor-->>+RevocationAPI: ACCEPTED -``` - -## Integrate your cloud provider service with GitLab SaaS - -Third party cloud and SaaS providers can [express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). - -### Implement a vendor revocation receiver service - -A vendor revocation receiver service integrates with a GitLab instance to receive -a web notification and respond to leaked token requests. - -To implement a receiver service to revoke leaked tokens: - -1. Create a publicly accessible HTTP service matching the corresponding API contract - below. Your service should be idempotent and rate-limited. -1. When a pipeline corresponding to its revocable token type (in the example, `my_api_token`) - completes, GitLab sends a request to your receiver service. -1. The included URL should be publicly accessible, and contain the commit where the - leaked token can be found. For example: - - ```plaintext - POST / HTTP/2 - Accept: */* - Content-Type: application/json - X-Gitlab-Token: MYSECRETTOKEN - - [ - {"type": "my_api_token", "token":"XXXXXXXXXXXXXXXX","url": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java"} - ] - ``` - -## Implement a revocation service for self-managed - -Self-managed instances interested in using the revocation capabilities must: - -- Deploy the [RevocationAPI](#high-level-architecture). -- Configure the GitLab instance to use the RevocationAPI. - -A RevocationAPI must: - -- Match a minimal API specification. -- Provide two endpoints: - - Fetching revocable token types. - - Revoking leaked tokens. -- Be rate-limited and idempotent. - -Requests to the documented endpoints are authenticated via API tokens passed in -the `Authorization` header. Request and response bodies, if present, are -expected to have the content type `application/json`. - -All endpoints may return these responses: - -- `401 Unauthorized` -- `405 Method Not Allowed` -- `500 Internal Server Error` - -### `GET /v1/revocable_token_types` - -Returns the valid `type` values for use in the `revoke_tokens` endpoint. - -NOTE: -These values match the concatenation of [the `secrets` analyzer's](index.md) -[primary identifier](../../../development/integrations/secure.md#identifiers) by means -of concatenating the `primary_identifier.type` and `primary_identifier.value`. -In the case below, a finding identifier matches: - -```json -{"type": "gitleaks_rule_id", "name": "Gitleaks rule ID GitLab Personal Access Token", "value": "GitLab Personal Access Token"} + Sidekiq-->+Token Revocation API: GET revocable keys types + Token Revocation API-->>-Sidekiq: OK + Sidekiq->>+Token Revocation API: POST revoke revocable keys + Token Revocation API-->>-Sidekiq: ACCEPTED + Token Revocation API-->>+Receiver Service: revoke revocable keys + Receiver Service-->>+Token Revocation API: ACCEPTED ``` -| Status Code | Description | -| ----- | --- | -| `200` | The response body contains the valid token `type` values. | - -Example response body: - -```json -{ - "types": ["gitleaks_rule_id_gitlab_personal_access_token"] -} +1. A pipeline with a Secret Detection job completes on the project's default branch, producing a scan + report (**1**). +1. The report is processed (**2**) by an asynchronous worker, which communicates with an externally + deployed HTTP service (**3** and **4**) to determine which kinds of secrets can be automatically + revoked. +1. The worker sends (**5** and **6**) the list of detected secrets which the Token Revocation API is able to + revoke. +1. The Token Revocation API sends (**7** and **8**) each revocable token to their respective vendor's [receiver service](#integrate-your-cloud-provider-service-with-gitlabcom). + +See the [Token Revocation API](../../../development/sec/token_revocation_api.md) documentation for more +information. + +## Integrate your cloud provider service with GitLab.com + +Third-party cloud and SaaS vendors interested in automated token revocation can +[express integration interest by filling out this form](https://forms.gle/wWpvrtLRK21Q2WJL9). +Vendors must [implement a revocation receiver service](#implement-a-revocation-receiver-service) +which will be called by the Token Revocation API. + +### Implement a revocation receiver service + +A revocation receiver service integrates with a GitLab instance's Token Revocation API to receive and respond +to leaked token revocation requests. The service should be a publicly accessible HTTP API that is +idempotent and rate-limited. Requests to your service from the Token Revocation API will follow the example +below: + +```plaintext +POST / HTTP/2 +Accept: */* +Content-Type: application/json +X-Gitlab-Token: MYSECRETTOKEN + +[ + {"type": "my_api_token", "token":"XXXXXXXXXXXXXXXX","url": "https://example.com/some-repo/~/raw/abcdefghijklmnop/compromisedfile1.java"} +] ``` -### `POST /v1/revoke_tokens` - -Accepts a list of tokens to be revoked by the appropriate provider. - -| Status Code | Description | -| ----- | --- | -| `204` | All submitted tokens have been accepted for eventual revocation. | -| `400` | The request body is invalid or one of the submitted token types is not supported. The request should not be retried. | -| `429` | The provider has received too many requests. The request should be retried later. | - -Example request body: - -```json -[{ - "type": "gitleaks_rule_id_gitlab_personal_access_token", - "token": "glpat--8GMtG8Mf4EnMJzmAWDU", - "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile1.java" -}, -{ - "type": "gitleaks_rule_id_gitlab_personal_access_token", - "token": "glpat--tG84EGK33nMLLDE70zU", - "location": "https://example.com/some-repo/blob/abcdefghijklmnop/compromisedfile2.java" -}] -``` - -### Configure GitLab to interface with RevocationAPI - -You must configure the following database settings in the GitLab instance: - -- `secret_detection_token_revocation_enabled` -- `secret_detection_token_revocation_url` -- `secret_detection_token_revocation_token` -- `secret_detection_revocation_token_types_url` - -For example, to configure these values in the -[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session): - -```ruby -::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_token: 'MYSECRETTOKEN') -::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_url: 'https://example.gitlab.com/revocation_service/v1/revoke_tokens') -::Gitlab::CurrentSettings.update!(secret_detection_revocation_token_types_url: 'https://example.gitlab.com/revocation_service/v1/revocable_token_types') -::Gitlab::CurrentSettings.update!(secret_detection_token_revocation_enabled: true) -``` - -After you configure these values, completing a pipeline performs these actions: - -1. The revocation service is triggered once. -1. A request is made to `secret_detection_revocation_token_types_url` to fetch a - list of revocable tokens. -1. Any Secret Detection findings matching the results of the `token_types` request - are included in the subsequent revocation request. +In this example, Secret Detection has determined that an instance of `my_api_token` has been leaked. The +value of the token is provided to you, in addition to a publicly accessible URL to the raw content of the +file containing the leaked token. |