diff options
Diffstat (limited to 'doc/operations/incident_management/manage_incidents.md')
| -rw-r--r-- | doc/operations/incident_management/manage_incidents.md | 263 |
1 files changed, 263 insertions, 0 deletions
diff --git a/doc/operations/incident_management/manage_incidents.md b/doc/operations/incident_management/manage_incidents.md new file mode 100644 index 00000000000..75826c2c55e --- /dev/null +++ b/doc/operations/incident_management/manage_incidents.md @@ -0,0 +1,263 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Manage incidents **(FREE)** + +This page collects instructions for all the things you can do with [incidents](incidents.md) or in relation to them. + +## Create an incident + +You can create an incident manually or automatically. + +### From the incidents list + +> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. +> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. +> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To create an incident from the incidents list: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. +1. Select **Create incident**. + +### From the issues list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To create an incident from the issues list: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Issues > List**, and select **New issue**. +1. From the **Type** dropdown list, select **Incident**. Only fields relevant to + incidents are available on the page. +1. Select **Create issue**. + +### From an alert + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. + +Create an incident issue when viewing an [alert](alerts.md). +The incident description is populated from the alert. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To create an incident from an alert: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Alerts**. +1. Select your desired alert. +1. Select **Create incident**. + +After an incident is created, to view it from the alert, select **View incident**. + +When you [close an incident](#close-an-incident) linked to an alert, GitLab +[changes the alert's status](alerts.md#change-an-alerts-status) to **Resolved**. +You are then credited with the alert's status change. + +### Automatically, when an alert is triggered **(ULTIMATE)** + +In the project settings, you can turn on [creating an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts) +whenever an alert is triggered. + +### Using the PagerDuty webhook + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. +> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7. + +You can set up a webhook with PagerDuty to automatically create a GitLab incident +for each PagerDuty incident. This configuration requires you to make changes +in both PagerDuty and GitLab. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To set up a webhook with PagerDuty: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor** +1. Expand **Incidents**. +1. Select the **PagerDuty integration** tab. +1. Turn on the **Active** toggle. +1. Select **Save integration**. +1. Copy the value of **Webhook URL** for use in a later step. +1. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks#manage-v3-webhook-subscriptions). + +To confirm the integration is successful, trigger a test incident from PagerDuty to +check if a GitLab incident is created from the incident. + +## View incidents list + +To view the [incidents list](incidents.md#incidents-list): + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. + +To view an incident's [details page](incidents.md#incident-details), select it from the list. + +### Who can view an incident + +Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and +the incident's confidentiality status: + +- Public project and a non-confidential incident: You don't have to be a member of the project. +- Private project and non-confidential incident: You must have at least the Guest role for the project. +- Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project. + +## Assign to a user + +Assign incidents to users that are actively responding. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To assign a user: + +1. In an incident, on the right sidebar, next to **Assignees**, select **Edit**. +1. From the dropdown list, select one or [multiple users](../../user/project/issues/multiple_assignees_for_issues.md) to add as **assignees**. +1. Select any area outside the dropdown list. + +## Change severity + +> Editing severity on incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. + +See [incident list](incidents.md#incidents-list) for a full description of the severity levels available. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To change an incident's severity: + +1. In an incident, on the right sidebar, next to **Severity**, select **Edit**. +1. From the dropdown list, select the new severity. + +You can also change the severity using the `/severity` [quick action](../../user/project/quick_actions.md). + +## Change status + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. +> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To change the status of an incident: + +1. In an incident, on the right sidebar, next to **Status**, select **Edit**. +1. From the dropdown list, select the new severity. + +**Triggered** is the default status for new incidents. + +### As an on-call responder **(PREMIUM)** + +On-call responders can respond to [incident pages](paging.md#escalating-an-incident) +by changing the status. + +Changing the status has the following effects: + +- To **Acknowledged**: limits on-call pages based on the project's [escalation policy](escalation_policies.md). +- To **Resolved**: silences all on-call pages for the incident. +- From **Resolved** to **Triggered**: restarts the incident escalating. + +In GitLab 15.1 and earlier, changing the status of an [incident created from an alert](#from-an-alert) +also changes the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the alert status is independent and does not change when the incident status changes. + +## Change escalation policy **(PREMIUM)** + +Prerequisites: + +- You must have at least the Developer role for the project. + +To change the escalation policy of an incident: + +1. In an incident, on the right sidebar, next to **Escalation policy**, select **Edit**. +1. From the dropdown list, select the escalation policy. + +By default, new incidents do not have an escalation policy selected. + +Selecting an escalation policy [changes the incident status](#change-status) to **Triggered** and begins +[escalating the incident to on-call responders](paging.md#escalating-an-incident). + +In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](#from-an-alert) +reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the incident escalation policy is independent and can be changed. + +## Embed metrics + +You can embed metrics anywhere [GitLab Flavored Markdown](../../user/markdown.md) is +used, like descriptions or comments. Embedding +metrics helps you share them when discussing incidents or performance issues. + +To embed metrics in a Markdown text box in GitLab, +[paste the link to the dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). + +You can embed both [GitLab-hosted metrics](../metrics/embed.md) (deprecated) and +[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue +templates. + +## Close an incident + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To close an incident, in the top right, select **Close incident**. + +When you close an incident that is linked to an [alert](alerts.md), +the linked alert's status changes to **Resolved**. +You are then credited with the alert's status change. + +### Automatically close incidents via recovery alerts + +> [Introduced for HTTP integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. + +Turn on closing an incident automatically when GitLab receives a recovery alert +from a HTTP or Prometheus webhook. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To configure the setting: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Incidents** section. +1. Select the **Automatically close associated incident** checkbox. +1. Select **Save changes**. + +When GitLab receives a recovery alert, it closes the associated incident. +This action is recorded as a system note on the incident indicating that it +was closed automatically by the GitLab Alert bot. + +## Other actions + +Because incidents in GitLab are built on top of [issues](../../user/project/issues/index.md), +they have the following actions in common: + +- [Add a to-do item](../../user/todos.md#create-a-to-do-item) +- [Add labels](../../user/project/labels.md#assign-and-unassign-labels) +- [Assign a milestone](../../user/project/milestones/index.md#assign-a-milestone-to-an-issue-or-merge-request) +- [Make an incident confidential](../../user/project/issues/confidential_issues.md) +- [Set a due date](../../user/project/issues/due_dates.md) +- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) +- [Track time spent](../../user/project/time_tracking.md) |