diff options
Diffstat (limited to 'doc/operations/incident_management/incidents.md')
| -rw-r--r-- | doc/operations/incident_management/incidents.md | 347 |
1 files changed, 89 insertions, 258 deletions
diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index de9b7d63369..5cfb8a77fc9 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -6,171 +6,78 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Incidents **(FREE)** -Incidents are critical entities in incident management workflows. They represent -a service disruption or outage that needs to be restored urgently. GitLab provides -tools for the triage, response, and remediation of incidents. +An incident is a service disruption or outage that needs to be restored urgently. +Incidents are critical in incident management workflows. +Use GitLab to triage, respond, and remediate incidents. -## Incident creation +## Incidents list -You can create an incident manually or automatically. +When you [view the incidents list](manage_incidents.md#view-incidents-list), it contains the following: -### Create incidents manually - -> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. -> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. -> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8. - -If you have at least Reporter [permissions](../../user/permissions.md), -you can create an incident manually from the Incidents List or the Issues List. - -To create an incident from the Incidents List: - -1. Navigate to **Monitor > Incidents** and select **Create Incident**. -1. Create a new issue using the `incident` template. - - - -To create an incident from the Issues List: - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. - -1. Go to **Issues > List**, and select **New issue**. -1. In the **Type** dropdown list, select **Incident**. Only fields relevant to - incidents are displayed on the page. -1. Create the incident as needed, and select **Create issue** to save the - incident. - - - -### Create incidents automatically **(ULTIMATE)** - -With at least the Maintainer role, you can enable -GitLab to create incident automatically whenever an alert is triggered: - -1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. -1. Check the **Create an incident** checkbox. -1. To customize the incident, select an - [issue template](../../user/project/description_templates.md#create-an-issue-template), - to include in the [incident summary](#summary). -1. To send [an email notification](paging.md#email-notifications-for-alerts) to users - with the Developer role, select - **Send a separate email notification to Developers**. Email notifications are - also sent to users with the **Maintainer** and **Owner** roles. -1. Select **Save changes**. +- **State**: To filter incidents by their state, select **Open**, **Closed**, + or **All** above the incident list. +- **Search**: Search for incident titles and descriptions or [filter the list](#filter-the-incidents-list). +- **Severity**: Severity of a particular incident, which can be one of the following + values: -### Create incidents via the PagerDuty webhook + - **{severity-critical}** Critical - S1 + - **{severity-high}** High - S2 + - **{severity-medium}** Medium - S3 + - **{severity-low}** Low - S4 + - **{severity-unknown}** Unknown -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. -> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7. +- **Incident**: The title of the incident, which attempts to capture the + most meaningful information. +- **Status**: The status of the incident, which can be one of the following values: -You can set up a webhook with PagerDuty to automatically create a GitLab incident -for each PagerDuty incident. This configuration requires you to make changes -in both PagerDuty and GitLab: + - Triggered + - Acknowledged + - Resolved -Prerequisites: + In the Premium or Ultimate tier, this field is also linked to [on-call escalation](paging.md#escalating-an-incident) for the incident. -- You must have at least the Maintainer role for the project. +- **Date created**: How long ago the incident was created. This field uses the + standard GitLab pattern of `X time ago`. Hover over this value to see the exact date and time formatted according to your locale. +- **Assignees**: The user assigned to the incident. +- **Published**: Whether the incident is published to a [status page](status_page.md). -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Monitor** -1. Expand **Incidents**. -1. Select the **PagerDuty integration** tab: + -  +For an example of the incident list in action, visit this +[demo project](https://gitlab.com/gitlab-org/monitor/monitor-sandbox/-/incidents). -1. Activate the integration, and save the changes in GitLab. -1. Copy the value of **Webhook URL** for use in a later step. -1. Follow the steps described in the - [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks) - to add the webhook URL to a PagerDuty webhook integration. +### Sort the incident list -To confirm the integration is successful, trigger a test incident from PagerDuty to -confirm that a GitLab incident is created from the incident. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) in GitLab 13.3: incidents are sorted by created date by default. -## Incident list +The incident list shows incidents sorted by incident created date, showing the newest first. -Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and -the incident's confidentiality status: +To sort by another column, or to change the sorting order, select the column. -- Public project and a non-confidential incident: You don't have to be a member of the project. -- Private project and non-confidential incident: You must have at least the Guest role for the project. -- Confidential incident (regardless of project visibility): You must have at least the Reporter. +The columns you can sort by: -The Incident list is available at **Monitor > Incidents** -in your project's sidebar. The list contains the following metrics: +- Severity +- Status +- Time to SLA +- Published - +### Filter the incidents list -- **State** - To filter incidents by their state, select **Open**, **Closed**, - or **All** above the incident list. -- **Search** - The Incident list supports a simple free text search, which filters - on the **Title** and **Incident** fields. -- **Severity** - Severity of a particular incident, which can be one of the following - values: - - **{severity-critical}** **Critical - S1** - - **{severity-high}** **High - S2** - - **{severity-medium}** **Medium - S3** - - **{severity-low}** **Low - S4** - - **{severity-unknown}** **Unknown** - - [Editing incident severity](#change-severity) on the incident details page was - [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. - -- **Incident** - The description of the incident, which attempts to capture the - most meaningful data. -- **Status** - The status of the incident, which can be one of the following values: - - **Triggered** - - **Acknowledged** - - **Resolved** - - In GitLab Premium, this field is also linked to [on-call escalation](paging.md#escalating-an-incident) for the incident. - -- **Date created** - How long ago the incident was created. This field uses the - standard GitLab pattern of `X time ago`, but is supported by a granular date/time - tooltip depending on the user's locale. -- **Assignees** - The user assigned to the incident. -- **Published** - Displays a green check mark (**{check-circle}**) if the incident is published - to a [Status Page](status_page.md). - -The Incident list displays incidents sorted by incident created date. -([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) in GitLab 13.3.) -To see if a column is sortable, point your mouse at the header. Sortable columns -display an arrow next to the column name. - -Incidents share the [Issues API](../../api/issues.md). - -NOTE: -For a live example of the incident list in action, visit this -[demo project](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). +To filter the incident list by author or assignee, enter these values in the search box. ## Incident details -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Monitor > Incidents**. -1. Select an incident from the list. -When you take any of these actions on an incident, GitLab logs a system note and -displays it in the Incident Details view: - -- Updating the severity of an incident - ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42358) in GitLab 13.5.) - -For live examples of GitLab incidents, visit the `tanuki-inc` project's -[incident list page](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). -Select any incident in the list to display its incident details page. - ### Summary -The summary section for incidents provides both critical details about the -incident and the contents of the issue template (if applicable). The highlighted +The summary section for incidents provides critical details about the +incident and the contents of the issue template (if [selected](../metrics/alerts.md#trigger-actions-from-alerts)). The highlighted bar at the top of the incident displays from left to right: - The link to the original alert. - The alert start time. - The event count. -Beneath the highlight bar, GitLab displays a summary that includes the following fields: +Below the highlight bar, a summary includes the following fields: - Start time - Severity @@ -178,15 +85,18 @@ Beneath the highlight bar, GitLab displays a summary that includes the following - Monitoring tool The incident summary can be further customized using -[GitLab Flavored Markdown](../../user/markdown.md). If the corresponding alert -[provided Markdown for the incident](../metrics/alerts.md#trigger-actions-from-alerts), -then the Markdown is appended to the summary after the above alert fields. If an -[incident template](#create-incidents-automatically) is configured for the -project, then the template content is appended at the end. +[GitLab Flavored Markdown](../../user/markdown.md). + +If an incident is [created from an alert](../metrics/alerts.md#trigger-actions-from-alerts) +that provided Markdown for the incident, then the Markdown is appended to the summary. +If an incident template is configured for the project, then the template content is appended at the end. Comments are displayed in threads, but can be displayed chronologically [by toggling on the recent updates view](#recent-updates-view). +When you make changes to an incident, GitLab creates system notes and +displays them below the summary. + ### Metrics **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235994) in GitLab 13.8. @@ -204,6 +114,8 @@ If you add a link, you can access the original graph by selecting the hyperlink ### Alert details +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. + Incidents show the details of linked alerts in a separate tab. To populate this tab, the incident must have been created with a linked alert. Incidents created automatically from alerts have this @@ -222,141 +134,60 @@ Read more about [timeline events](incident_timeline_events.md) and how to enable > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in GitLab 13.5. -To quickly see the latest updates on an incident, select -**{history}** **Turn recent updates view on** in the comment bar to display comments -un-threaded and ordered chronologically, newest to oldest: - - +To see the latest updates on an incident, select +**Turn recent updates view on** (**{history}**) on the comment bar. Comments display +un-threaded and chronologically, newest to oldest. ### Service Level Agreement countdown timer **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241663) in GitLab 13.5. You can enable the Service Level Agreement Countdown timer on incidents to track -the Service Level Agreements (SLAs) you hold with your customers. The timer is +the Service Level Agreements (SLA) you hold with your customers. The timer is automatically started when the incident is created, and shows the time remaining before the SLA period expires. The timer is also dynamically updated every 15 minutes so you do not have to refresh the page to see the time remaining. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + To configure the timer: -1. Navigate to **Settings > Monitor**. -1. Scroll to **Incidents** and select **Expand**, then select the - **Incident settings** tab. +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Incidents** section, then select the **Incident settings** tab. 1. Select **Activate "time to SLA" countdown timer**. 1. Set a time limit in increments of 15 minutes. 1. Select **Save changes**. -After you enable the SLA countdown timer, the **Time to SLA** attribute is displayed -as a column in the Incidents List, and as a field on newly created Incidents. If +After you enable the SLA countdown timer, the **Time to SLA** column is available in the +incidents list and as a field on new incidents. If the incident isn't closed before the SLA period ends, GitLab adds a `missed::SLA` label to the incident. -## Assign incidents - -Assign incidents to users that are actively responding. Select **Edit** in the -right-hand side bar to select or clear assignees. - -## Associate a milestone - -Associate an incident to a milestone by selecting **Edit** next to the milestone feature in the right-hand side bar. - -## Change severity - -See [Incident List](#incident-list) for a full description of the severity levels available. -Select **Edit** in the right-hand side bar to change the severity of an incident. - -You can also change the severity using the [`/severity` quick action](../../user/project/quick_actions.md). - -## Add a to-do item - -Add a to-do for incidents that you want to track in your to-do list. Select -**Add a to do** at the top of the right-hand side bar to add a to-do item. - -## Change incident status - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. -> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. - -For users with the Developer role or higher, select **Edit** in the **Status** section of the -right-hand side bar of an incident, then select a status. **Triggered** is the default status for -new incidents. - -In projects with GitLab Premium, on-call responders can respond to [incident pages](paging.md#escalating-an-incident) -by changing the status. Setting the status to: - -- **Resolved** silences on-call pages for the alert. -- **Acknowledged** limits on-call pages based on the selected [escalation policy](#change-escalation-policy). -- **Triggered** from **Resolved** restarts the incident escalating from the beginning. - -In GitLab 15.1 and earlier, updating the status of an [incident created from an alert](alerts.md#create-an-incident-from-an-alert) -also updates the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), -the alert status is independent and does not update when the incident status changes. - -## Change escalation policy **(PREMIUM)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. -> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. - -For users with the Developer role or higher, select **Edit** in the **Escalation policy** section of -the right-hand side bar of an incident, then select a policy. By default, new incidents do not have -an escalation policy selected. - -Selecting an escalation policy updates the incident status to **Triggered** and begins -[escalating the incident to on-call responders](paging.md#escalating-an-incident). -Deselecting an escalation policy halts escalation. Refer to the [incident status](#change-incident-status) -to manage on-call paging once escalation has begun. - -In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) -reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), -the incident escalation policy is independent and can be changed. - -## Associate Zoom calls - -GitLab enables you to [associate a Zoom meeting with an issue](../../user/project/issues/associate_zoom_meeting.md) -for synchronous communication during incident management. After starting a Zoom -call for an incident, you can associate the conference call with an issue. Your -team members can join the Zoom call without requesting a link. - -## Linked resources - -In an incident, you can add [links to various resources](linked_resources.md), -for example: - -- The incident Slack channel -- Zoom meeting -- Resources for resolving the incidents - -## Embed metrics in incidents - -You can embed metrics anywhere [GitLab Markdown](../../user/markdown.md) is -used, such as descriptions, comments on issues, and merge requests. Embedding -metrics helps you share them when discussing incidents or performance issues. -You can output the dashboard directly into any issue, merge request, epic, or -any other Markdown text field in GitLab by -[copying and pasting the link to the metrics dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). - -You can embed both [GitLab-hosted metrics](../metrics/embed.md) and -[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue -templates. - -## Automatically close incidents via recovery alerts - -> - [Introduced for Prometheus Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13401) in GitLab 12.5. -> - [Introduced for HTTP Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. - -With at least the Maintainer role, you can enable - GitLab to close an incident automatically when a **Recovery Alert** is received: - -1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. -1. Check the **Automatically close associated Incident** checkbox. -1. Select **Save changes**. - -When GitLab receives a **Recovery Alert**, it closes the associated incident. -This action is recorded as a system message on the incident indicating that it -was closed automatically by the GitLab Alert bot. - ## Related topics +- [Create an incident](manage_incidents.md#create-an-incident) +- [Create an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts) + whenever an alert is triggered +- [View incidents list](manage_incidents.md#view-incidents-list) +- [Assign to a user](manage_incidents.md#assign-to-a-user) +- [Change incident severity](manage_incidents.md#change-severity) +- [Change incident status](manage_incidents.md#change-status) +- [Change escalation policy](manage_incidents.md#change-escalation-policy) +- [Embed metrics](manage_incidents.md#embed-metrics) +- [Close an incident](manage_incidents.md#close-an-incident) +- [Automatically close incidents via recovery alerts](manage_incidents.md#automatically-close-incidents-via-recovery-alerts) +- [Add a to-do item](../../user/todos.md#create-a-to-do-item) +- [Add labels](../../user/project/labels.md) +- [Assign a milestone](../../user/project/milestones/index.md) +- [Make an incident confidential](../../user/project/issues/confidential_issues.md) +- [Set a due date](../../user/project/issues/due_dates.md) +- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) +- [Track spent time](../../user/project/time_tracking.md) +- [Add a Zoom meeting to an incident](../../user/project/issues/associate_zoom_meeting.md) the same + way you add it to an issue. +- [Linked resources in incidents](linked_resources.md) - Create incidents and receive incident notifications [directly from Slack](slack.md). +- Use the [Issues API](../../api/issues.md) to interact with incidents. |