diff options
Diffstat (limited to 'doc/operations/incident_management')
7 files changed, 206 insertions, 9 deletions
diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index d42ee237749..6d6fde45de7 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -35,7 +35,7 @@ The alert list displays the following information: - **Triggered**: Investigation has not started. - **Acknowledged**: Someone is actively investigating the problem. - **Resolved**: No further work is required. - - **Ignored**: No action will be taken on the alert. + - **Ignored**: No action is taken on the alert. ## Alert severity diff --git a/doc/operations/incident_management/incident_timeline_events.md b/doc/operations/incident_management/incident_timeline_events.md index 4af5a815929..e79f36884cb 100644 --- a/doc/operations/incident_management/incident_timeline_events.md +++ b/doc/operations/incident_management/incident_timeline_events.md @@ -107,3 +107,31 @@ Alternatively: 1. On the right of a timeline event, select **More actions** (**{ellipsis_v}**) and then select **Edit**. 1. Select **Delete**. 1. To confirm, select **Delete event**. + +## Incident tags + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8741) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `incident_event_tags`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `incident_event_tags`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + +[When creating an event using the form](#using-the-form) or editing it, +you can specify incident tags to capture relevant incident timestamps. +Timeline tags are optional. You can choose more than one tag per event. +When you create a timeline event and select the tags, the event note +is populated with a default message. +This allows for a quick event creation. If a note has already been set, it isn't changed. +Added tags are displayed next to the timestamp. + +## Formatting rules + +Incident timeline events support the following [GitLab Flavored Markdown](../../user/markdown.md) features. + +- [Code](../../user/markdown.md#code-spans-and-blocks). +- [Emojis](../../user/markdown.md#emojis). +- [Emphasis](../../user/markdown.md#emphasis). +- [GitLab-specific references](../../user/markdown.md#gitlab-specific-references). +- [Images](../../user/markdown.md#images), rendered as a link to the uploaded image. +- [Links](../../user/markdown.md#links). diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 5cfb8a77fc9..45f1e10d2f1 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -43,7 +43,7 @@ When you [view the incidents list](manage_incidents.md#view-incidents-list), it  -For an example of the incident list in action, visit this +For an example of the incident list in action, see this [demo project](https://gitlab.com/gitlab-org/monitor/monitor-sandbox/-/incidents). ### Sort the incident list diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index fdcfbe0cb2c..41fbe8fe83c 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -9,8 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in GitLab 12.4. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) from GitLab Ultimate to GitLab Free in 12.8. -GitLab can accept alerts from any source via a webhook receiver. This can be configured -generically. +GitLab can accept alerts from any source via a webhook receiver. [Alert notifications](alerts.md) +can [trigger paging](paging.md#paging) for on-call rotations or be used to [create incidents](manage_incidents.md#from-an-alert). ## Integrations list @@ -89,7 +89,7 @@ GitLab fields when you [create an HTTP endpoint](#http-endpoints): ## Customize the alert payload outside of GitLab For HTTP Endpoints without [custom mappings](#map-fields-in-custom-alerts), you can customize the payload by sending the following -parameters. All fields are optional. If the incoming alert does not contain a value for the `Title` field, a default value of `New: Alert` will be applied. +parameters. All fields are optional. If the incoming alert does not contain a value for the `Title` field, a default value of `New: Alert` is applied. | Property | Type | Description | | ------------------------- | --------------- | ----------- | @@ -138,6 +138,175 @@ Example payload: } ``` +### Prometheus endpoint + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Alerts** section, and select **Add new integration**. +1. From the **Select integration type** dropdown list, select **Prometheus**. +1. Turn on the **Active** toggle. +1. Enter the **Prometheus API base URL**. + You should enter a placeholder URL. The features which use this field are [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) and [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/379252) in GitLab 16.0. +1. Select **Save integration**. + +The URL and authorization key for the webhook configuration +are available in the **View credentials** tab. + +Enter the URL and authorization key in your external service. +You can also send a test alert from your integration's +[**Send test alert**](#triggering-test-alerts) tab. + +#### Add integration credentials to Prometheus Alertmanager + +To send Prometheus alert notifications to GitLab, copy the URL and authorization key from +your [Prometheus integration](#prometheus-endpoint) into the +[`webhook_configs`](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) +section of the Prometheus Alertmanager configuration: + +```yaml +receivers: + - name: gitlab + webhook_configs: + - http_config: + authorization: + type: Bearer + credentials: 1234567890abdcdefg + send_resolved: true + url: http://IP_ADDRESS:PORT/root/manual_prometheus/prometheus/alerts/notify.json + # Rest of configuration omitted + # ... +``` + +#### Expected request attributes + +Alerts are expected to be formatted for a Prometheus [webhook receiver](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config). + +Top-level required attributes: + +- `alerts` +- `commonAnnotations` +- `commonLabels` +- `externalURL` +- `groupKey` +- `groupLabels` +- `receiver` +- `status` +- `version` + +From `alerts` in the Prometheus payload, a GitLab alert is created for each item in the array. +You can alter the nested parameters listed below to configure the GitLab alert. + +| Attribute | Type | Required | Description | +| -------------------------------------------------------------------------- | -------- | -------- | ------------------------------------ | +| One of `annotations/title`, `annotations/summary`, or `labels/alertname` | String | Yes | The title of the alert. | +| `startsAt` | DateTime | Yes | The start time of the alert. | +| `annotations/description` | String | No | A high-level summary of the problem. | +| `annotations/gitlab_incident_markdown` | String | No | [GitLab Flavored Markdown](../../user/markdown.md) to be appended to any incident created from the alert. | +| `annotations/runbook` | String | No | Link to documentation or instructions for how to manage this alert. | +| `endsAt` | DateTime | No | The resolution time of the alert. | +| `g0.expr` query parameter in `generatorUrl` | String | No | Query of associated metric. | +| `labels/gitlab_environment_name` | String | No | The name of the associated GitLab [environment](../../ci/environments/index.md). Required to [display alerts on a dashboard](../../user/operations_dashboard/index.md#adding-a-project-to-the-dashboard). | +| `labels/severity` | String | No | Severity of the alert. Should be one of the [Prometheus severity options](#prometheus-severity-options). Defaults to `critical` if missing or value is not in this list. | +| `status` | String | No | Status of the alert in Prometheus. If value is 'resolved', the alert is resolved. | +| One of `annotations/gitlab_y_label`, `annotations/title`, `annotations/summary`, or `labels/alertname` | String | No | The Y-Axis label to be used when embedding the metrics for this alert in [GitLab Flavored Markdown](../../user/markdown.md). | + +Additional attributes included under `annotations` are available on +the [alert details page](alerts.md#alert-details-page). Any other attributes are ignored. + +Attributes aren't limited to primitive types (such as strings or numbers), but +can be a nested JSON object. For example: + +```json +{ + "target": { + "user": { + "id": 42 + } + } +} +``` + +NOTE: +Ensure your requests are smaller than the +[payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads). + +#### Prometheus severity options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871) in GitLab 13.9 + +Alerts from Prometheus can provide any of the case-insensitive follow values for [alert severity](../incident_management/alerts.md#alert-severity): + +- **Critical**: `critical`, `s1`, `p1`, `emergency`, `fatal` +- **High**: `high`, `s2`, `p2`, `major`, `page` +- **Medium**: `medium`, `s3`, `p3`, `error`, `alert` +- **Low**: `low`, `s4`, `p4`, `warn`, `warning` +- **Info**: `info`, `s5`, `p5`, `debug`, `information`, `notice` + +The severity defaults to `critical` if the value is missing or not in this list. + +#### Example Prometheus alert + +Example alerting rule: + +```yaml +groups: +- name: example + rules: + - alert: ServiceDown + expr: up == 0 + for: 5m + labels: + severity: high + annotations: + title: "Example title" + runbook: "http://example.com/my-alert-runbook" + description: "Service has been down for more than 5 minutes." + gitlab_y_label: "y-axis label" + foo: + bar: + baz: 42 +``` + +Example request payload: + +```json +{ + "version" : "4", + "groupKey": null, + "status": "firing", + "receiver": "", + "groupLabels": {}, + "commonLabels": {}, + "commonAnnotations": {}, + "externalURL": "", + "alerts": [{ + "startsAt": "2022-010-30T11:22:40Z", + "generatorURL": "http://host?g0.expr=up", + "endsAt": null, + "status": "firing", + "labels": { + "gitlab_environment_name": "production", + "severity": "high" + }, + "annotations": { + "title": "Example title", + "runbook": "http://example.com/my-alert-runbook", + "description": "Service has been down for more than 5 minutes.", + "gitlab_y_label": "y-axis label", + "foo": { + "bar": { + "baz": 42 + } + } + } + }] +} +``` + ## Authorization The following authorization methods are accepted: @@ -244,7 +413,7 @@ If the existing alert is already `resolved`, GitLab creates a new alert instead. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. -The alert in GitLab will be automatically resolved when an HTTP Endpoint +The alert in GitLab is automatically resolved when an HTTP Endpoint receives a payload with the end time of the alert set. For HTTP Endpoints without [custom mappings](#map-fields-in-custom-alerts), the expected field is `end_time`. With custom mappings, you can select the expected field. diff --git a/doc/operations/incident_management/linked_resources.md b/doc/operations/incident_management/linked_resources.md index 96edd41c12f..642a6972bcf 100644 --- a/doc/operations/incident_management/linked_resources.md +++ b/doc/operations/incident_management/linked_resources.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230852) in GitLab 15.3 [with a flag](../../administration/feature_flags.md) named `incident_resource_links_widget`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/364755) in GitLab 15.3. -> - [Generally available](ihttps://gitlab.com/gitlab-org/gitlab/-/issues/364755) in GitLab 15.5. Feature flag `incident_resource_links_widget` removed. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/364755) in GitLab 15.5. Feature flag `incident_resource_links_widget` removed. To help your team members find the important links without having to search through many comments, you can add linked resources to an incident issue. diff --git a/doc/operations/incident_management/manage_incidents.md b/doc/operations/incident_management/manage_incidents.md index 75826c2c55e..ad4de8641e5 100644 --- a/doc/operations/incident_management/manage_incidents.md +++ b/doc/operations/incident_management/manage_incidents.md @@ -220,7 +220,7 @@ Prerequisites: - You must have at least the Reporter role for the project. -To close an incident, in the top right, select **Close incident**. +To close an incident, in the upper right, select **Close incident**. When you close an incident that is linked to an [alert](alerts.md), the linked alert's status changes to **Resolved**. diff --git a/doc/operations/incident_management/slack.md b/doc/operations/incident_management/slack.md index 0e7a85142f6..434c481900c 100644 --- a/doc/operations/incident_management/slack.md +++ b/doc/operations/incident_management/slack.md @@ -41,7 +41,7 @@ Prerequisites: To start the authorization flow, try executing a non-incident [Slack slash command](../../integration/slash_commands.md), like `/gitlab <project-alias> issue show <id>`. The `<project-alias>` you select must be a project that has the GitLab for Slack app set up. - For more context, visit [issue 377548](https://gitlab.com/gitlab-org/gitlab/-/issues/377548). + For more information, see [issue 377548](https://gitlab.com/gitlab-org/gitlab/-/issues/377548). <!-- The below content is commented out until these features are implemented in https://gitlab.com/groups/gitlab-org/-/epics/8545 --> <!-- |