diff options
Diffstat (limited to 'doc/operations/incident_management')
13 files changed, 366 insertions, 285 deletions
diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index 69b74127891..0dffa15351c 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -113,7 +113,7 @@ timeline of the alert's investigation and assignment history. The following actions result in a system note: - [Updating the status of an alert](#change-an-alerts-status) -- [Creating an incident based on an alert](#create-an-incident-from-an-alert) +- [Creating an incident based on an alert](manage_incidents.md#from-an-alert) - [Assignment of an alert to a user](#assign-an-alert) - [Escalation of an alert to on-call responders](paging.md#escalating-an-alert) @@ -158,8 +158,8 @@ Prerequisites: - You must have at least the Reporter role. -When you close an [incident](incidents.md) that is linked to an alert, -the linked alert's status changes to **Resolved**. +When you [close an incident](manage_incidents.md#close-an-incident) that is linked to an alert, +GitLab [changes the alert's status](#change-an-alerts-status) to **Resolved**. You are then credited with the alert's status change. #### As an on-call responder **(PREMIUM)** @@ -173,25 +173,10 @@ Changing the status has the following effects: - To **Resolved**: silences all on-call pages for the alert. - From **Resolved** to **Triggered**: restarts the alert escalating. -In GitLab 15.1 and earlier, updating the status of an [alert with an associated incident](alerts.md#create-an-incident-from-an-alert) +In GitLab 15.1 and earlier, updating the status of an [alert with an associated incident](manage_incidents.md#from-an-alert) also updates the incident status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), the incident status is independent and does not update when the alert status changes. -### Create an incident from an alert - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. - -The Alert detail view enables you to create an issue with a -description populated from an alert. To create the issue, -select the **Create Issue** button. You can then view the issue from the -alert by selecting the **View Issue** button. - -You can also [create incidents for alerts automatically](incidents.md#create-incidents-automatically). - -Closing a GitLab issue associated with an alert [changes the alert's status](#change-an-alerts-status) to -**Resolved**. See [Alert List](#alert-list) for more details -about alert statuses. - ### Assign an alert > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. diff --git a/doc/operations/incident_management/img/incident_list_create_v13_3.png b/doc/operations/incident_management/img/incident_list_create_v13_3.png Binary files differdeleted file mode 100644 index a000c849099..00000000000 --- a/doc/operations/incident_management/img/incident_list_create_v13_3.png +++ /dev/null diff --git a/doc/operations/incident_management/img/incident_list_v14_9.png b/doc/operations/incident_management/img/incident_list_v14_9.png Binary files differdeleted file mode 100644 index 4cf6f0e6522..00000000000 --- a/doc/operations/incident_management/img/incident_list_v14_9.png +++ /dev/null diff --git a/doc/operations/incident_management/img/incident_list_v15_6.png b/doc/operations/incident_management/img/incident_list_v15_6.png Binary files differnew file mode 100644 index 00000000000..fe2a91e2eba --- /dev/null +++ b/doc/operations/incident_management/img/incident_list_v15_6.png diff --git a/doc/operations/incident_management/img/new_incident_create_v13_4.png b/doc/operations/incident_management/img/new_incident_create_v13_4.png Binary files differdeleted file mode 100644 index 458532736bd..00000000000 --- a/doc/operations/incident_management/img/new_incident_create_v13_4.png +++ /dev/null diff --git a/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png b/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png Binary files differdeleted file mode 100644 index 08a45d001c2..00000000000 --- a/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png +++ /dev/null diff --git a/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png b/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png Binary files differdeleted file mode 100644 index 90f1d205fed..00000000000 --- a/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png +++ /dev/null diff --git a/doc/operations/incident_management/incident_timeline_events.md b/doc/operations/incident_management/incident_timeline_events.md index 897fcb9efc6..4af5a815929 100644 --- a/doc/operations/incident_management/incident_timeline_events.md +++ b/doc/operations/incident_management/incident_timeline_events.md @@ -78,7 +78,7 @@ The comment is shown on the incident timeline as a timeline event. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375280) in GitLab 15.6. -A new timeline event is created when someone [changes the severity](incidents.md#change-severity) +A new timeline event is created when someone [changes the severity](manage_incidents.md#change-severity) of an incident.  diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index de9b7d63369..5cfb8a77fc9 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -6,171 +6,78 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Incidents **(FREE)** -Incidents are critical entities in incident management workflows. They represent -a service disruption or outage that needs to be restored urgently. GitLab provides -tools for the triage, response, and remediation of incidents. +An incident is a service disruption or outage that needs to be restored urgently. +Incidents are critical in incident management workflows. +Use GitLab to triage, respond, and remediate incidents. -## Incident creation +## Incidents list -You can create an incident manually or automatically. +When you [view the incidents list](manage_incidents.md#view-incidents-list), it contains the following: -### Create incidents manually - -> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. -> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. -> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8. - -If you have at least Reporter [permissions](../../user/permissions.md), -you can create an incident manually from the Incidents List or the Issues List. - -To create an incident from the Incidents List: - -1. Navigate to **Monitor > Incidents** and select **Create Incident**. -1. Create a new issue using the `incident` template. - - - -To create an incident from the Issues List: - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. - -1. Go to **Issues > List**, and select **New issue**. -1. In the **Type** dropdown list, select **Incident**. Only fields relevant to - incidents are displayed on the page. -1. Create the incident as needed, and select **Create issue** to save the - incident. - - - -### Create incidents automatically **(ULTIMATE)** - -With at least the Maintainer role, you can enable -GitLab to create incident automatically whenever an alert is triggered: - -1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. -1. Check the **Create an incident** checkbox. -1. To customize the incident, select an - [issue template](../../user/project/description_templates.md#create-an-issue-template), - to include in the [incident summary](#summary). -1. To send [an email notification](paging.md#email-notifications-for-alerts) to users - with the Developer role, select - **Send a separate email notification to Developers**. Email notifications are - also sent to users with the **Maintainer** and **Owner** roles. -1. Select **Save changes**. +- **State**: To filter incidents by their state, select **Open**, **Closed**, + or **All** above the incident list. +- **Search**: Search for incident titles and descriptions or [filter the list](#filter-the-incidents-list). +- **Severity**: Severity of a particular incident, which can be one of the following + values: -### Create incidents via the PagerDuty webhook + - **{severity-critical}** Critical - S1 + - **{severity-high}** High - S2 + - **{severity-medium}** Medium - S3 + - **{severity-low}** Low - S4 + - **{severity-unknown}** Unknown -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. -> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7. +- **Incident**: The title of the incident, which attempts to capture the + most meaningful information. +- **Status**: The status of the incident, which can be one of the following values: -You can set up a webhook with PagerDuty to automatically create a GitLab incident -for each PagerDuty incident. This configuration requires you to make changes -in both PagerDuty and GitLab: + - Triggered + - Acknowledged + - Resolved -Prerequisites: + In the Premium or Ultimate tier, this field is also linked to [on-call escalation](paging.md#escalating-an-incident) for the incident. -- You must have at least the Maintainer role for the project. +- **Date created**: How long ago the incident was created. This field uses the + standard GitLab pattern of `X time ago`. Hover over this value to see the exact date and time formatted according to your locale. +- **Assignees**: The user assigned to the incident. +- **Published**: Whether the incident is published to a [status page](status_page.md). -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Settings > Monitor** -1. Expand **Incidents**. -1. Select the **PagerDuty integration** tab: + -  +For an example of the incident list in action, visit this +[demo project](https://gitlab.com/gitlab-org/monitor/monitor-sandbox/-/incidents). -1. Activate the integration, and save the changes in GitLab. -1. Copy the value of **Webhook URL** for use in a later step. -1. Follow the steps described in the - [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks) - to add the webhook URL to a PagerDuty webhook integration. +### Sort the incident list -To confirm the integration is successful, trigger a test incident from PagerDuty to -confirm that a GitLab incident is created from the incident. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) in GitLab 13.3: incidents are sorted by created date by default. -## Incident list +The incident list shows incidents sorted by incident created date, showing the newest first. -Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and -the incident's confidentiality status: +To sort by another column, or to change the sorting order, select the column. -- Public project and a non-confidential incident: You don't have to be a member of the project. -- Private project and non-confidential incident: You must have at least the Guest role for the project. -- Confidential incident (regardless of project visibility): You must have at least the Reporter. +The columns you can sort by: -The Incident list is available at **Monitor > Incidents** -in your project's sidebar. The list contains the following metrics: +- Severity +- Status +- Time to SLA +- Published - +### Filter the incidents list -- **State** - To filter incidents by their state, select **Open**, **Closed**, - or **All** above the incident list. -- **Search** - The Incident list supports a simple free text search, which filters - on the **Title** and **Incident** fields. -- **Severity** - Severity of a particular incident, which can be one of the following - values: - - **{severity-critical}** **Critical - S1** - - **{severity-high}** **High - S2** - - **{severity-medium}** **Medium - S3** - - **{severity-low}** **Low - S4** - - **{severity-unknown}** **Unknown** - - [Editing incident severity](#change-severity) on the incident details page was - [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. - -- **Incident** - The description of the incident, which attempts to capture the - most meaningful data. -- **Status** - The status of the incident, which can be one of the following values: - - **Triggered** - - **Acknowledged** - - **Resolved** - - In GitLab Premium, this field is also linked to [on-call escalation](paging.md#escalating-an-incident) for the incident. - -- **Date created** - How long ago the incident was created. This field uses the - standard GitLab pattern of `X time ago`, but is supported by a granular date/time - tooltip depending on the user's locale. -- **Assignees** - The user assigned to the incident. -- **Published** - Displays a green check mark (**{check-circle}**) if the incident is published - to a [Status Page](status_page.md). - -The Incident list displays incidents sorted by incident created date. -([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) in GitLab 13.3.) -To see if a column is sortable, point your mouse at the header. Sortable columns -display an arrow next to the column name. - -Incidents share the [Issues API](../../api/issues.md). - -NOTE: -For a live example of the incident list in action, visit this -[demo project](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). +To filter the incident list by author or assignee, enter these values in the search box. ## Incident details -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. - -1. On the top bar, select **Main menu > Projects** and find your project. -1. On the left sidebar, select **Monitor > Incidents**. -1. Select an incident from the list. -When you take any of these actions on an incident, GitLab logs a system note and -displays it in the Incident Details view: - -- Updating the severity of an incident - ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42358) in GitLab 13.5.) - -For live examples of GitLab incidents, visit the `tanuki-inc` project's -[incident list page](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). -Select any incident in the list to display its incident details page. - ### Summary -The summary section for incidents provides both critical details about the -incident and the contents of the issue template (if applicable). The highlighted +The summary section for incidents provides critical details about the +incident and the contents of the issue template (if [selected](../metrics/alerts.md#trigger-actions-from-alerts)). The highlighted bar at the top of the incident displays from left to right: - The link to the original alert. - The alert start time. - The event count. -Beneath the highlight bar, GitLab displays a summary that includes the following fields: +Below the highlight bar, a summary includes the following fields: - Start time - Severity @@ -178,15 +85,18 @@ Beneath the highlight bar, GitLab displays a summary that includes the following - Monitoring tool The incident summary can be further customized using -[GitLab Flavored Markdown](../../user/markdown.md). If the corresponding alert -[provided Markdown for the incident](../metrics/alerts.md#trigger-actions-from-alerts), -then the Markdown is appended to the summary after the above alert fields. If an -[incident template](#create-incidents-automatically) is configured for the -project, then the template content is appended at the end. +[GitLab Flavored Markdown](../../user/markdown.md). + +If an incident is [created from an alert](../metrics/alerts.md#trigger-actions-from-alerts) +that provided Markdown for the incident, then the Markdown is appended to the summary. +If an incident template is configured for the project, then the template content is appended at the end. Comments are displayed in threads, but can be displayed chronologically [by toggling on the recent updates view](#recent-updates-view). +When you make changes to an incident, GitLab creates system notes and +displays them below the summary. + ### Metrics **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235994) in GitLab 13.8. @@ -204,6 +114,8 @@ If you add a link, you can access the original graph by selecting the hyperlink ### Alert details +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. + Incidents show the details of linked alerts in a separate tab. To populate this tab, the incident must have been created with a linked alert. Incidents created automatically from alerts have this @@ -222,141 +134,60 @@ Read more about [timeline events](incident_timeline_events.md) and how to enable > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in GitLab 13.5. -To quickly see the latest updates on an incident, select -**{history}** **Turn recent updates view on** in the comment bar to display comments -un-threaded and ordered chronologically, newest to oldest: - - +To see the latest updates on an incident, select +**Turn recent updates view on** (**{history}**) on the comment bar. Comments display +un-threaded and chronologically, newest to oldest. ### Service Level Agreement countdown timer **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241663) in GitLab 13.5. You can enable the Service Level Agreement Countdown timer on incidents to track -the Service Level Agreements (SLAs) you hold with your customers. The timer is +the Service Level Agreements (SLA) you hold with your customers. The timer is automatically started when the incident is created, and shows the time remaining before the SLA period expires. The timer is also dynamically updated every 15 minutes so you do not have to refresh the page to see the time remaining. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + To configure the timer: -1. Navigate to **Settings > Monitor**. -1. Scroll to **Incidents** and select **Expand**, then select the - **Incident settings** tab. +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Incidents** section, then select the **Incident settings** tab. 1. Select **Activate "time to SLA" countdown timer**. 1. Set a time limit in increments of 15 minutes. 1. Select **Save changes**. -After you enable the SLA countdown timer, the **Time to SLA** attribute is displayed -as a column in the Incidents List, and as a field on newly created Incidents. If +After you enable the SLA countdown timer, the **Time to SLA** column is available in the +incidents list and as a field on new incidents. If the incident isn't closed before the SLA period ends, GitLab adds a `missed::SLA` label to the incident. -## Assign incidents - -Assign incidents to users that are actively responding. Select **Edit** in the -right-hand side bar to select or clear assignees. - -## Associate a milestone - -Associate an incident to a milestone by selecting **Edit** next to the milestone feature in the right-hand side bar. - -## Change severity - -See [Incident List](#incident-list) for a full description of the severity levels available. -Select **Edit** in the right-hand side bar to change the severity of an incident. - -You can also change the severity using the [`/severity` quick action](../../user/project/quick_actions.md). - -## Add a to-do item - -Add a to-do for incidents that you want to track in your to-do list. Select -**Add a to do** at the top of the right-hand side bar to add a to-do item. - -## Change incident status - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. -> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. - -For users with the Developer role or higher, select **Edit** in the **Status** section of the -right-hand side bar of an incident, then select a status. **Triggered** is the default status for -new incidents. - -In projects with GitLab Premium, on-call responders can respond to [incident pages](paging.md#escalating-an-incident) -by changing the status. Setting the status to: - -- **Resolved** silences on-call pages for the alert. -- **Acknowledged** limits on-call pages based on the selected [escalation policy](#change-escalation-policy). -- **Triggered** from **Resolved** restarts the incident escalating from the beginning. - -In GitLab 15.1 and earlier, updating the status of an [incident created from an alert](alerts.md#create-an-incident-from-an-alert) -also updates the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), -the alert status is independent and does not update when the incident status changes. - -## Change escalation policy **(PREMIUM)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. -> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. - -For users with the Developer role or higher, select **Edit** in the **Escalation policy** section of -the right-hand side bar of an incident, then select a policy. By default, new incidents do not have -an escalation policy selected. - -Selecting an escalation policy updates the incident status to **Triggered** and begins -[escalating the incident to on-call responders](paging.md#escalating-an-incident). -Deselecting an escalation policy halts escalation. Refer to the [incident status](#change-incident-status) -to manage on-call paging once escalation has begun. - -In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) -reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), -the incident escalation policy is independent and can be changed. - -## Associate Zoom calls - -GitLab enables you to [associate a Zoom meeting with an issue](../../user/project/issues/associate_zoom_meeting.md) -for synchronous communication during incident management. After starting a Zoom -call for an incident, you can associate the conference call with an issue. Your -team members can join the Zoom call without requesting a link. - -## Linked resources - -In an incident, you can add [links to various resources](linked_resources.md), -for example: - -- The incident Slack channel -- Zoom meeting -- Resources for resolving the incidents - -## Embed metrics in incidents - -You can embed metrics anywhere [GitLab Markdown](../../user/markdown.md) is -used, such as descriptions, comments on issues, and merge requests. Embedding -metrics helps you share them when discussing incidents or performance issues. -You can output the dashboard directly into any issue, merge request, epic, or -any other Markdown text field in GitLab by -[copying and pasting the link to the metrics dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). - -You can embed both [GitLab-hosted metrics](../metrics/embed.md) and -[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue -templates. - -## Automatically close incidents via recovery alerts - -> - [Introduced for Prometheus Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13401) in GitLab 12.5. -> - [Introduced for HTTP Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. - -With at least the Maintainer role, you can enable - GitLab to close an incident automatically when a **Recovery Alert** is received: - -1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. -1. Check the **Automatically close associated Incident** checkbox. -1. Select **Save changes**. - -When GitLab receives a **Recovery Alert**, it closes the associated incident. -This action is recorded as a system message on the incident indicating that it -was closed automatically by the GitLab Alert bot. - ## Related topics +- [Create an incident](manage_incidents.md#create-an-incident) +- [Create an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts) + whenever an alert is triggered +- [View incidents list](manage_incidents.md#view-incidents-list) +- [Assign to a user](manage_incidents.md#assign-to-a-user) +- [Change incident severity](manage_incidents.md#change-severity) +- [Change incident status](manage_incidents.md#change-status) +- [Change escalation policy](manage_incidents.md#change-escalation-policy) +- [Embed metrics](manage_incidents.md#embed-metrics) +- [Close an incident](manage_incidents.md#close-an-incident) +- [Automatically close incidents via recovery alerts](manage_incidents.md#automatically-close-incidents-via-recovery-alerts) +- [Add a to-do item](../../user/todos.md#create-a-to-do-item) +- [Add labels](../../user/project/labels.md) +- [Assign a milestone](../../user/project/milestones/index.md) +- [Make an incident confidential](../../user/project/issues/confidential_issues.md) +- [Set a due date](../../user/project/issues/due_dates.md) +- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) +- [Track spent time](../../user/project/time_tracking.md) +- [Add a Zoom meeting to an incident](../../user/project/issues/associate_zoom_meeting.md) the same + way you add it to an issue. +- [Linked resources in incidents](linked_resources.md) - Create incidents and receive incident notifications [directly from Slack](slack.md). +- Use the [Issues API](../../api/issues.md) to interact with incidents. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index d0d4ee6e498..fdcfbe0cb2c 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -249,7 +249,7 @@ receives a payload with the end time of the alert set. For HTTP Endpoints without [custom mappings](#map-fields-in-custom-alerts), the expected field is `end_time`. With custom mappings, you can select the expected field. -You can also configure the associated [incident to be closed automatically](../incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) when the alert resolves. +You can also configure the associated [incident to be closed automatically](../incident_management/manage_incidents.md#automatically-close-incidents-via-recovery-alerts) when the alert resolves. ## Link to your Opsgenie Alerts **(PREMIUM)** diff --git a/doc/operations/incident_management/manage_incidents.md b/doc/operations/incident_management/manage_incidents.md new file mode 100644 index 00000000000..75826c2c55e --- /dev/null +++ b/doc/operations/incident_management/manage_incidents.md @@ -0,0 +1,263 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Manage incidents **(FREE)** + +This page collects instructions for all the things you can do with [incidents](incidents.md) or in relation to them. + +## Create an incident + +You can create an incident manually or automatically. + +### From the incidents list + +> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. +> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. +> - Automatic application of the `incident` label [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/290964) in GitLab 14.8. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To create an incident from the incidents list: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. +1. Select **Create incident**. + +### From the issues list + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To create an incident from the issues list: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Issues > List**, and select **New issue**. +1. From the **Type** dropdown list, select **Incident**. Only fields relevant to + incidents are available on the page. +1. Select **Create issue**. + +### From an alert + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. + +Create an incident issue when viewing an [alert](alerts.md). +The incident description is populated from the alert. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To create an incident from an alert: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Alerts**. +1. Select your desired alert. +1. Select **Create incident**. + +After an incident is created, to view it from the alert, select **View incident**. + +When you [close an incident](#close-an-incident) linked to an alert, GitLab +[changes the alert's status](alerts.md#change-an-alerts-status) to **Resolved**. +You are then credited with the alert's status change. + +### Automatically, when an alert is triggered **(ULTIMATE)** + +In the project settings, you can turn on [creating an incident automatically](../metrics/alerts.md#trigger-actions-from-alerts) +whenever an alert is triggered. + +### Using the PagerDuty webhook + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. +> - [PagerDuty V3 Webhook](https://support.pagerduty.com/docs/webhooks) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383029) in GitLab 15.7. + +You can set up a webhook with PagerDuty to automatically create a GitLab incident +for each PagerDuty incident. This configuration requires you to make changes +in both PagerDuty and GitLab. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To set up a webhook with PagerDuty: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor** +1. Expand **Incidents**. +1. Select the **PagerDuty integration** tab. +1. Turn on the **Active** toggle. +1. Select **Save integration**. +1. Copy the value of **Webhook URL** for use in a later step. +1. To add the webhook URL to a PagerDuty webhook integration, follow the steps described in the [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks#manage-v3-webhook-subscriptions). + +To confirm the integration is successful, trigger a test incident from PagerDuty to +check if a GitLab incident is created from the incident. + +## View incidents list + +To view the [incidents list](incidents.md#incidents-list): + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. + +To view an incident's [details page](incidents.md#incident-details), select it from the list. + +### Who can view an incident + +Whether you can view an incident depends on the [project visibility level](../../user/public_access.md) and +the incident's confidentiality status: + +- Public project and a non-confidential incident: You don't have to be a member of the project. +- Private project and non-confidential incident: You must have at least the Guest role for the project. +- Confidential incident (regardless of project visibility): You must have at least the Reporter role for the project. + +## Assign to a user + +Assign incidents to users that are actively responding. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To assign a user: + +1. In an incident, on the right sidebar, next to **Assignees**, select **Edit**. +1. From the dropdown list, select one or [multiple users](../../user/project/issues/multiple_assignees_for_issues.md) to add as **assignees**. +1. Select any area outside the dropdown list. + +## Change severity + +> Editing severity on incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. + +See [incident list](incidents.md#incidents-list) for a full description of the severity levels available. + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To change an incident's severity: + +1. In an incident, on the right sidebar, next to **Severity**, select **Edit**. +1. From the dropdown list, select the new severity. + +You can also change the severity using the `/severity` [quick action](../../user/project/quick_actions.md). + +## Change status + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. +> - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To change the status of an incident: + +1. In an incident, on the right sidebar, next to **Status**, select **Edit**. +1. From the dropdown list, select the new severity. + +**Triggered** is the default status for new incidents. + +### As an on-call responder **(PREMIUM)** + +On-call responders can respond to [incident pages](paging.md#escalating-an-incident) +by changing the status. + +Changing the status has the following effects: + +- To **Acknowledged**: limits on-call pages based on the project's [escalation policy](escalation_policies.md). +- To **Resolved**: silences all on-call pages for the incident. +- From **Resolved** to **Triggered**: restarts the incident escalating. + +In GitLab 15.1 and earlier, changing the status of an [incident created from an alert](#from-an-alert) +also changes the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the alert status is independent and does not change when the incident status changes. + +## Change escalation policy **(PREMIUM)** + +Prerequisites: + +- You must have at least the Developer role for the project. + +To change the escalation policy of an incident: + +1. In an incident, on the right sidebar, next to **Escalation policy**, select **Edit**. +1. From the dropdown list, select the escalation policy. + +By default, new incidents do not have an escalation policy selected. + +Selecting an escalation policy [changes the incident status](#change-status) to **Triggered** and begins +[escalating the incident to on-call responders](paging.md#escalating-an-incident). + +In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](#from-an-alert) +reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the incident escalation policy is independent and can be changed. + +## Embed metrics + +You can embed metrics anywhere [GitLab Flavored Markdown](../../user/markdown.md) is +used, like descriptions or comments. Embedding +metrics helps you share them when discussing incidents or performance issues. + +To embed metrics in a Markdown text box in GitLab, +[paste the link to the dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). + +You can embed both [GitLab-hosted metrics](../metrics/embed.md) (deprecated) and +[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue +templates. + +## Close an incident + +Prerequisites: + +- You must have at least the Reporter role for the project. + +To close an incident, in the top right, select **Close incident**. + +When you close an incident that is linked to an [alert](alerts.md), +the linked alert's status changes to **Resolved**. +You are then credited with the alert's status change. + +### Automatically close incidents via recovery alerts + +> [Introduced for HTTP integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. + +Turn on closing an incident automatically when GitLab receives a recovery alert +from a HTTP or Prometheus webhook. + +Prerequisites: + +- You must have at least the Maintainer role for the project. + +To configure the setting: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Monitor**. +1. Expand the **Incidents** section. +1. Select the **Automatically close associated incident** checkbox. +1. Select **Save changes**. + +When GitLab receives a recovery alert, it closes the associated incident. +This action is recorded as a system note on the incident indicating that it +was closed automatically by the GitLab Alert bot. + +## Other actions + +Because incidents in GitLab are built on top of [issues](../../user/project/issues/index.md), +they have the following actions in common: + +- [Add a to-do item](../../user/todos.md#create-a-to-do-item) +- [Add labels](../../user/project/labels.md#assign-and-unassign-labels) +- [Assign a milestone](../../user/project/milestones/index.md#assign-a-milestone-to-an-issue-or-merge-request) +- [Make an incident confidential](../../user/project/issues/confidential_issues.md) +- [Set a due date](../../user/project/issues/due_dates.md) +- [Toggle notifications](../../user/profile/notifications.md#edit-notification-settings-for-issues-merge-requests-and-epics) +- [Track time spent](../../user/project/time_tracking.md) diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 9ce87d03686..70c67977b73 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -54,12 +54,14 @@ or stop alert escalations by [updating the alert's status](alerts.md#change-an-a > - [Feature flag `incident_escalations`](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) removed in GitLab 15.1. For incidents, paging on-call responders is optional for each individual incident. -To begin escalating the incident, [set the incident's escalation policy](incidents.md#change-escalation-policy). + +To begin escalating the incident, [set the incident's escalation policy](manage_incidents.md#change-escalation-policy). + For each escalation rule, the designated on-call responders receive one email when -the rule fires. You can respond to a page or stop incident escalations by -[updating the incident's status](incidents.md#change-incident-status) or, if applicable, -[unsetting the incident's escalation policy](incidents.md#change-escalation-policy). +the rule fires. Respond to a page or stop incident escalations by +[changing the incident's status](manage_incidents.md#change-status) or +changing the incident's escalation policy back to **No escalation policy**. -In GitLab 15.1 and earlier, [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) +In GitLab 15.1 and earlier, [incidents created from alerts](manage_incidents.md#from-an-alert) do not support independent escalation. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), all incidents can be escalated independently. diff --git a/doc/operations/incident_management/slack.md b/doc/operations/incident_management/slack.md index 37cfb790d70..1ab1391ea2a 100644 --- a/doc/operations/incident_management/slack.md +++ b/doc/operations/incident_management/slack.md @@ -68,7 +68,7 @@ To declare a GitLab incident from Slack: - The project where the incident should be created. - The severity of the incident. - If there is an existing [incident template](incidents.md#create-incidents-automatically) for your + If there is an existing [incident template](../metrics/alerts.md#trigger-actions-from-alerts) for your project, that template is automatically applied to the description text box. The template is applied only if the description text box is empty. |
