diff options
Diffstat (limited to 'doc/administration/audit_event_streaming.md')
-rw-r--r-- | doc/administration/audit_event_streaming.md | 344 |
1 files changed, 203 insertions, 141 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 817f22debbc..3bb0ce41861 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -6,44 +6,53 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit event streaming **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. -> - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. -> - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - API [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - API [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. +> - API [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - UI [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. > - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. - -Users can set an HTTP endpoint for a top-level group to receive all audit events about the group, its subgroups, and +> - Custom HTTP headers API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - Custom HTTP headers API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. +> - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. +> - Custom HTTP headers UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. +> - Custom HTTP headers UI [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) in GitLab 15.3. [Feature flag `custom_headers_streaming_audit_events_ui`](https://gitlab.com/gitlab-org/gitlab/-/issues/365259) removed. +> - [Improved user experience](https://gitlab.com/gitlab-org/gitlab/-/issues/367963) in GitLab 15.4. + +Users can set a streaming destination for a top-level group to receive all audit events about the group, its subgroups, and projects as structured JSON. Top-level group owners can manage their audit logs in third-party systems. Any service that can receive -structured JSON data can be used as the endpoint. +structured JSON data can be used as the streaming destination. + +Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. NOTE: GitLab can stream a single event more than once to the same destination. Use the `id` key in the payload to deduplicate incoming data. -## Add a new event streaming destination +## Add a new streaming destination WARNING: -Event streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. +Streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the streaming destination. ### Use the GitLab UI -Users with at least the Owner role for a group can add event streaming destinations for it: +Users with the Owner role for a group can add streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** to show the section for adding destinations. - - When the destination list is not empty, select **{plus}** to show the section for adding destinations. -1. Enter the destination URL to add and select **Add**. - -Event streaming is enabled if: - -- No warning is shown. -- The added endpoint is displayed in the UI. +1. Select **Add streaming destination** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Optional. Locate the **Custom HTTP headers** table. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. After all headers have been filled out, select **Add** to add the new streaming destination. ### Use the API -To enable event streaming and add a destination, users with at least the Owner role for a group must use the +To enable streaming and add a destination, users with the Owner role for a group must use the `externalAuditEventDestinationCreate` mutation in the GraphQL API. ```graphql @@ -51,6 +60,7 @@ mutation { externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group" } ) { errors externalAuditEventDestination { + id destinationUrl verificationToken group { @@ -66,23 +76,35 @@ Event streaming is enabled if: - The returned `errors` object is empty. - The API responds with `200 OK`. +Group owners can add an HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID +by [listing all the streaming destinations](#use-the-api-1) for the group or from the mutation above. + +```graphql +mutation { + auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + errors + } +} +``` + +The header is created if the returned `errors` object is empty. + ## List streaming destinations -Users with at least the Owner role for a group can list event streaming destinations. +Users with the Owner role for a group can list streaming destinations. ### Use the GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. - -Users with at least the Owner role for a group can list event streaming destinations: +To list the streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. +1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. ### Use the API -Users with at least the Owner role for a group can view a list of event streaming destinations at any time using the +Users with the Owner role for a group can view a list of streaming destinations at any time using the `externalAuditEventDestinations` query type. ```graphql @@ -94,42 +116,48 @@ query { destinationUrl verificationToken id + headers { + nodes { + key + value + id + } + } } } } } ``` -If the resulting list is empty, then audit event streaming is not enabled for that group. +If the resulting list is empty, then audit streaming is not enabled for that group. -## Delete streaming destinations +You need the ID values returned by this query for the update and delete mutations. -Users with at least the Owner role for a group can delete event streaming destinations using the -`deleteAuditEventDestinations` mutation type. +## Update streaming destinations -When the last destination is successfully deleted, event streaming is disabled for the group. +Users with the Owner role for a group can update streaming destinations. ### Use the GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. - -Users with at least the Owner role for a group can delete event streaming destinations. +To update a streaming destinations custom HTTP headers: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. -1. Select **{remove}** at the right side of each item. - -The external streaming destination is deleted when: - -- No warning is shown. -- The deleted endpoint is not displayed in the UI. +1. To the right of the item, select **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to update. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. Select **Save** to update the streaming destination. ### Use the API -Delete an event streaming destination by specifying an ID. Get the required ID by -[listing the details](audit_event_streaming.md#use-the-api-1) of event -streaming destinations. +Users with the Owner role for a group can update streaming destinations custom HTTP headers using the +`auditEventsStreamingHeadersUpdate` mutation type. You can retrieve the custom HTTP headers ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { @@ -139,89 +167,71 @@ mutation { } ``` -Destination is deleted if: +Streaming destination is updated if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## Custom HTTP headers - -> - API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. -> - API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. -> - UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. - -FLAG: -On self-managed GitLab, by default the API for this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. -On GitLab.com, the API for this feature is available. - -Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. - -### Adding custom HTTP headers - -Add custom HTTP headers with the API or GitLab UI. - -#### Use the API - -Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID -by [listing the external audit destinations](#list-streaming-destinations) on the group. +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { - auditEventsStreamingHeadersCreate(input: { destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601", key: "foo", value: "bar" }) { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { errors } } ``` -The header is created if the returned `errors` object is empty. +The header is deleted if the returned `errors` object is empty. -#### Use the GitLab UI +## Delete streaming destinations -FLAG: -On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to -[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is -not available. The UI for this feature is not ready for production use. +Users with the Owner role for a group can delete streaming destinations. -Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: +When the last destination is successfully deleted, streaming is disabled for the group. + +### Use the GitLab UI + +To delete a streaming destination: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** to show the section for adding destinations. - - When the destination list is not empty, select **{plus}** to show the section for adding destinations. -1. Enter the destination URL to add. -1. Locate the **Custom HTTP headers** table. -1. In the **Header** column, add the header's name. -1. In the **Value** column, add the header's value. -1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the - [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). -1. Enter as many name and value pairs as required. When you enter a unique name and a value for a header, a new row in the table automatically appears. You can add up to - 20 headers per endpoint. -1. After all headers have been filled out, select **Add** to add the new endpoint. - -Event streaming is enabled if: +1. On the main area, select the **Streams** tab. +1. To the right of the item, select **Delete** (**{remove}**). -- No warning is shown. -- The added endpoint is displayed in the UI. +To delete only the custom HTTP headers for a streaming destination: -### Updating custom HTTP headers +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to remove. +1. To the right of the header, select **Delete** (**{remove}**). +1. Select **Save** to update the streaming destination. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361964) in GitLab 15.2. +### Use the API -Group owners can update a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. +Users with the Owner role for a group can delete streaming destinations using the +`externalAuditEventDestinationDestroy` mutation type. You can retrieve the destinations ID +by [listing all the streaming destinations](#use-the-api-1) for the group. ```graphql mutation { - auditEventsStreamingHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/24255", key: "new-foo", value: "new-bar" }) { + externalAuditEventDestinationDestroy(input: { id: destination }) { errors } } ``` -### Deleting custom HTTP headers +Streaming destination is deleted if: -Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID -by [listing all the custom headers](#list-all-custom-headers) on the group. +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +Group owners can remove an HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom HTTP headers](#use-the-api-1) for the group. ```graphql mutation { @@ -233,52 +243,6 @@ mutation { The header is deleted if the returned `errors` object is empty. -### List all custom headers - -List all custom HTTP headers with the API or GitLab UI. - -#### Use the API - -You can list all custom headers for a top-level group as well as their value and ID using the GraphQL `externalAuditEventDestinations` query. The ID -value returned by this query is what you need to pass to the `deletion` mutation. - -```graphql -query { - group(fullPath: "your-group") { - id - externalAuditEventDestinations { - nodes { - destinationUrl - id - headers { - nodes { - key - value - id - } - } - } - } - } -} -``` - -#### Use the GitLab UI - -FLAG: -On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to -[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is -not available. The UI for this feature is not ready for production use. - -Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams** tab. -1. Select **{pencil}** at the right side of an item. -1. A read-only view of the items custom headers is shown. To track progress on adding editing functionality, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). -1. Select **Cancel** to close the read-only view. - ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -293,18 +257,41 @@ the destination's value when [listing streaming destinations](#list-streaming-de > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. -Users with at least the Owner role for a group can list event streaming destinations and see the verification tokens: +Users with the Owner role for a group can list streaming destinations and see the verification tokens: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events**. -1. On the main area, select **Streams**. +1. On the main area, select the **Streams**. 1. View the verification token on the right side of each item. +## Payload schema + +> Documentation for an audit event streaming schema was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358149) in GitLab 15.3. + +Streamed audit events have a predictable schema in the body of the response. + +| Field | Description | Notes | +|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------| +| `author_id` | User ID of the user who triggered the event | | +| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | +| `created_at` | Timestamp when event was triggered | | +| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | +| `entity_id` | ID of the audit event's entity | | +| `entity_path` | Full path of the entity affected by the auditable event | | +| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | +| `event_type` | String representation of the type of audit event | | +| `id` | Unique identifier for the audit event | Can be used for deduplication if required | +| `ip_address` | IP address of the host used to trigger the event | | +| `target_details` | Additional details about the target | | +| `target_id` | ID of the audit event's target | | +| `target_type` | String representation of the target's type | | + ## Audit event streaming on Git operations > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. +> - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. FLAG: On self-managed GitLab, by default this feature is available. To hide the @@ -318,7 +305,7 @@ Streaming audit events can be sent when signed-in users push or pull a project's Audit events are not captured for users that are not signed in. For example, when downloading a public project. -To configure streaming audit events for Git operations, see [Add a new event streaming destination](#add-a-new-event-streaming-destination). +To configure streaming audit events for Git operations, see [Add a new streaming destination](#add-a-new-streaming-destination). ### Headers @@ -346,6 +333,7 @@ Fetch: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -377,6 +365,7 @@ Push: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -398,6 +387,42 @@ Push: } ``` +### Example payloads for SSH events with Deploy Key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. + +Fetch: + +```json +{ + "id": 1, + "author_id": -3, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "deploy-key-name", + "author_class": "DeployKey", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-key-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:43:53.662Z", + "target_type": "Project", + "target_id": 29, + "event_type": "repository_git_operation" +} +``` + ### Example payloads for HTTP and HTTPS events Fetch: @@ -410,6 +435,7 @@ Fetch: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -441,6 +467,7 @@ Push: "entity_type": "Project", "details": { "author_name": "Administrator", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-project", @@ -462,6 +489,40 @@ Push: } ``` +### Example payloads for HTTP and HTTPS events with Deploy Token + +Fetch: + +```json +{ + "id": 1, + "author_id": -2, + "entity_id": 22, + "entity_type": "Project", + "details": { + "author_name": "deploy-token-name", + "author_class": "DeployToken", + "target_id": 22, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "deploy-token-name", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-07-26T05:46:25.850Z", + "target_type": "Project", + "target_id": 22, + "event_type": "repository_git_operation" +} +``` + ### Example payloads for events from GitLab UI download button Fetch: @@ -475,6 +536,7 @@ Fetch: "details": { "custom_message": "Repository Download Started", "author_name": "example_username", + "author_class": "User", "target_id": 29, "target_type": "Project", "target_details": "example-group/example-project", |