Commit 7b06d27f authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'axil/incident-management' into 'master'

Icident management docs

See merge request gitlab-org/gitlab!18805
parents 899c0f13 329d65ff
...@@ -337,6 +337,7 @@ The following documentation relates to the DevOps **Monitor** stage: ...@@ -337,6 +337,7 @@ The following documentation relates to the DevOps **Monitor** stage:
| [Health check](user/admin_area/monitoring/health_check.md) | GitLab provides liveness and readiness probes to indicate service health and reachability to required services. | | [Health check](user/admin_area/monitoring/health_check.md) | GitLab provides liveness and readiness probes to indicate service health and reachability to required services. |
| [Prometheus project integration](user/project/integrations/prometheus.md) | Configure the Prometheus integration per project and monitor your CI/CD environments. | | [Prometheus project integration](user/project/integrations/prometheus.md) | Configure the Prometheus integration per project and monitor your CI/CD environments. |
| [Prometheus metrics](user/project/integrations/prometheus_library/index.md) | Let Prometheus collect metrics from various services, like Kubernetes, NGINX, NGINX Ingress controller, HAProxy, and Amazon Cloud Watch. | | [Prometheus metrics](user/project/integrations/prometheus_library/index.md) | Let Prometheus collect metrics from various services, like Kubernetes, NGINX, NGINX Ingress controller, HAProxy, and Amazon Cloud Watch. |
| [Incident management](user/incident_management/index.md) | Use GitLab to help you better respond to incidents that may occur in your systems. |
<div align="right"> <div align="right">
<a type="button" class="btn btn-default" href="#overview"> <a type="button" class="btn btn-default" href="#overview">
......
---
description: "GitLab - Incident Management. GitLab offers solutions for handling incidents in your applications and services"
---
# Incident Management
GitLab offers solutions for handling incidents in your applications and services,
from setting up an alert with Prometheus, to receiving a notification via a
monitoring tool like Slack, and automatically setting up Zoom calls with your
support team.
## Configuring incidents **(ULTIMATE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/4925) in GitLab Ultimate 11.11.
The Incident Management features can be enabled and disabled via your project's
**Settings > Operations > Incidents**.
![Incident Management Settings](img/incident_management_settings.png)
### Automatically create issues from alerts
GitLab issues can automatically be created as a result of an alert notification.
An issue created this way will contain the error information to help you further
debug it.
### Issue templates
You can create your own [issue templates](../project/description_templates.md#creating-issue-templates)
that can be [used within Incident Management](../project/integrations/prometheus.md#taking-action-on-incidents-ultimate).
To select your issue template for use within Incident Management:
1. Visit your project's **Settings > Operations > Incidents**.
1. Select the template from the **Issue Template** dropdown.
## Alerting
GitLab can react to the alerts that your applications and services may be
triggering by automatically creating issues, and alerting developers via email.
### Prometheus alerts
Prometheus alerts can be set up in both:
- [GitLab-managed Prometheus](../project/integrations/prometheus.md#setting-up-alerts-for-prometheus-metrics-ultimate) and
- [Self-managed Prometheus](../project/integrations/prometheus.md#external-prometheus-instances) installations.
### Alert endpoint
GitLab can accept alerts from any source via a generic webhook receiver. When
you set up the generic alerts integration, a unique endpoint will
be created which can receive a payload in JSON format.
[Read more on setting this up, including how to customize the payload](../project/integrations/generic_alerts.md).
### Recovery alerts
GitLab can [automatically close issues](../project/integrations/prometheus.md#taking-action-on-incidents-ultimate)
that have been automatically created when you receive notification that the
alert is resolved.
## Embedded metrics
Metrics can be embedded anywhere where GitLab Markdown is used, for example,
descriptions and comments on issues and merge requests.
TIP: **Tip:**
Both GitLab-hosted and Grafana metrics can also be
[embedded in issue templates](../project/integrations/prometheus.md#embedding-metrics-in-issue-templates).
### GitLab-hosted metrics
Learn how to embed [GitLab hosted metric charts](../project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown).
### Grafana metrics
Learn how to embed [Grafana hosted metric charts](../project/integrations/prometheus.md#embedding-live-grafana-charts).
## Slack integration
Slack slash commands allow you to control GitLab and view content right inside
Slack, without having to leave it.
Learn how to [set up Slack slash commands](../project/integrations/slack_slash_commands.md)
and how to [use them](../../integration/slash_commands.md).
### Slash commands
Please refer to a list of [available slash commands](../../integration/slash_commands.md) and associated descriptions.
## Zoom in issues
In order to communicate synchronously for incidents management, GitLab allows to
associate a Zoom meeting with an issue. Once you start a Zoom call for a fire-fight,
you need a way to associate the conference call with an issue, so that your team
members can join swiftly without requesting a link.
Read more how to [add or remove a zoom meeting](../project/issues/associate_zoom_meeting.md).
### Alerting
You can let GitLab know of alerts that may be triggering in your applications and services. GitLab can react to these by automatically creating Issues, and alerting developers via Email.
#### Prometheus Alerts
Prometheus alerts can be setup in both GitLab-managed Prometheus installs and self-managed Prometheus installs.
Documentation for each method can be found here:
- [GitLab-managed Prometheus](../project/integrations/prometheus.md#setting-up-alerts-for-prometheus-metrics-ultimate)
- [Self-managed Prometheus](../project/integrations/prometheus.md#external-prometheus-instances)
#### Alert Endpoint
GitLab can accept alerts from any source via a generic webhook receiver. When you set up the generic alerts integration, a unique endpoint will
be created which can receive a payload in JSON format.
More information on setting this up, including how to customize the payload [can be found here](../project/integrations/generic_alerts.md).
#### Recovery Alerts
Coming soon: GitLab can automatically close Issues that have been automatically created when we receive notification that the alert is resolved.
### Configuring Incidents
Incident Management features can be easily enabled & disabled via the Project settings page. Head to Project -> Settings -> Operations -> Incidents.
#### Auto-creation
GitLab Issues can automatically be created as a result of an Alert notification. An Issue created this way will contain error information to help you further debug the error.
...@@ -403,9 +403,12 @@ receivers: ...@@ -403,9 +403,12 @@ receivers:
... ...
``` ```
In order for GitLab to associate your alerts with an [environment](../../../ci/environments.md), you need to configure a `gitlab_environment_name` label on the alerts you set up in Prometheus. The value of this should match the name of your Environment in GitLab.
### Taking action on incidents **(ULTIMATE)** ### Taking action on incidents **(ULTIMATE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. >- [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11.
>- [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/issues/13401), when GitLab receives a recovery alert, it will automatically close the associated issue.
Alerts can be used to trigger actions, like open an issue automatically (enabled by default since `12.1`). To configure the actions: Alerts can be used to trigger actions, like open an issue automatically (enabled by default since `12.1`). To configure the actions:
...@@ -426,7 +429,7 @@ Once enabled, an issue will be opened automatically when an alert is triggered w ...@@ -426,7 +429,7 @@ Once enabled, an issue will be opened automatically when an alert is triggered w
- Optional list of attached annotations extracted from `annotations/*` - Optional list of attached annotations extracted from `annotations/*`
- Alert [GFM](../../markdown.md): GitLab Flavored Markdown from `annotations/gitlab_incident_markdown` - Alert [GFM](../../markdown.md): GitLab Flavored Markdown from `annotations/gitlab_incident_markdown`
When GitLab recieves a **Recovery Alert**, it will automatically close the associated issue. This action will be recorded as a system message on the issue indicated that it was closed automatically by the GitLab Alert bot. When GitLab receives a **Recovery Alert**, it will automatically close the associated issue. This action will be recorded as a system message on the issue indicated that it was closed automatically by the GitLab Alert bot.
To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template, which will apply to all incidents. To limit quick actions or other information to only specific types of alerts, use the `annotations/gitlab_incident_markdown` field. To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template, which will apply to all incidents. To limit quick actions or other information to only specific types of alerts, use the `annotations/gitlab_incident_markdown` field.
...@@ -487,6 +490,12 @@ The following requirements must be met for the metric to unfurl: ...@@ -487,6 +490,12 @@ The following requirements must be met for the metric to unfurl:
![Embedded Metrics](img/embed_metrics.png) ![Embedded Metrics](img/embed_metrics.png)
### Embedding metrics in issue templates
It is also possible to embed either a dashboard or individual metrics in issue templates. The entire dashboard can be embedded as well as individual metrics, separated by either a comma or a space.
![Embedded Metrics in issue templates](img/embed_metrics_issue_template.png)
### Embedding live Grafana charts ### Embedding live Grafana charts
It is also possible to embed live [Grafana](https://docs.gitlab.com/omnibus/settings/grafana.html) charts within issues, as a [Direct Linked Rendered Image](https://grafana.com/docs/reference/sharing/#direct-link-rendered-image). It is also possible to embed live [Grafana](https://docs.gitlab.com/omnibus/settings/grafana.html) charts within issues, as a [Direct Linked Rendered Image](https://grafana.com/docs/reference/sharing/#direct-link-rendered-image).
......
# Associate a Zoom meeting with an issue
> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/16609) in GitLab 12.4.
In order to communicate synchronously for incidents management,
GitLab allows to associate a Zoom meeting with an issue.
Once you start a Zoom call for a fire-fight, you need a way to
associate the conference call with an issue, so that your team
members can join swiftly without requesting a link.
## Adding a zoom meeting to an issue
To associate a zoom meeting with an issue, you can use GitLab's
[quick actions](../quick_actions.md#quick-actions-for-issues-merge-requests-and-epics).
In an issue, leave a comment using the `/zoom` quick action followed by a valid Zoom link:
```sh
/zoom https://zoom.us/j/123456789
```
If the Zoom meeting URL is valid and you have at least [Reporter permissions](../../permissions.md),
a system alert will notify you that the addition of the meeting URL was successful.
The issue's description will be automatically edited to include the Zoom link, and a button will
appear right under the issue's title.
![Link Zoom Call in Issue](img/zoom-quickaction-button.png)
You are only allowed to attach a single Zoom meeting to an issue. If you attempt
to add a second Zoom meeting using the `/zoom` quick action, it won't work, you
need to [remove it](#removing-an-existing-zoom-meeting-from-an-issue) first.
## Removing an existing Zoom meeting from an issue
Similarly to adding a zoom meeting, you can remove it with a quick action:
```sh
/remove_zoom
```
If you have at least [Reporter permissions](../../permissions.md),
a system alert will notify you that the meeting URL was successfully removed.
...@@ -257,4 +257,4 @@ You can attach and remove Zoom meetings to issues using the `/zoom` and `/remove ...@@ -257,4 +257,4 @@ You can attach and remove Zoom meetings to issues using the `/zoom` and `/remove
Attaching a [Zoom](https://zoom.us) call an issue Attaching a [Zoom](https://zoom.us) call an issue
results in a **Join Zoom meeting** button at the top of the issue, just under the header. results in a **Join Zoom meeting** button at the top of the issue, just under the header.
![Link Zoom Call in Issue](img/zoom-quickaction-button.png) Read more how to [add or remove a zoom meeting](associate_zoom_meeting.md).
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment