alert_integrations.md 8.78 KB
Newer Older
1 2 3
---
stage: Monitor
group: Health
4
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
5 6
---

7
# Alert integrations **(CORE)**
8 9 10 11

> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.4.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.8.

Sarah Waldner's avatar
Sarah Waldner committed
12 13
GitLab can accept alerts from any source via a webhook receiver. This can be configured
generically or, in GitLab versions 13.1 and greater, you can configure
14 15 16
[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances)
to use this endpoint.

17 18 19 20 21 22 23 24 25 26 27
## Integrations list

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in [GitLab Core](https://about.gitlab.com/pricing/) 13.5.

With Maintainer or higher [permissions](../../user/permissions.md), you can view
the list of configured alerts integrations by navigating to
**Settings > Operations** in your project's sidebar menu, and expanding **Alerts** section.
The list displays the integration name, type, and status (enabled or disabled):

![Current Integrations](img/integrations_list_v13_5.png)

28 29
## Configuration

30
GitLab can receive alerts via a HTTP endpoint that you configure,
Sarah Waldner's avatar
Sarah Waldner committed
31
or the [Prometheus integration](#external-prometheus-integration).
32

33
### Single HTTP Endpoint **CORE**
34

35
Enabling the HTTP Endpoint in a GitLab projects activates it to
Sarah Waldner's avatar
Sarah Waldner committed
36 37
receive alert payloads in JSON format. You can always
[customize the payload](#customize-the-alert-payload-outside-of-gitlab) to your liking.
38 39 40

1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md)
   for a project.
41 42
1. Navigate to **Settings > Operations** in your project.
1. Expand the **Alerts** section, and in the **Integration** dropdown menu, select **Generic**.
43 44 45
1. Toggle the **Active** alert setting to display the **URL** and **Authorization Key**
   for the webhook configuration.

Sarah Waldner's avatar
Sarah Waldner committed
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
### HTTP Endpoints **PREMIUM**

> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4442) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.6.

In [GitLab Premium](https://about.gitlab.com/pricing/), you can create multiple
unique HTTP endpoints to receive alerts from any external source in JSON format,
and you can [customize the payload](#customize-the-alert-payload-outside-of-gitlab).

1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md)
   for a project.
1. Navigate to **Settings > Operations** in your project.
1. Expand the **Alerts** section.
1. For each endpoint you want to create:

   1. In the **Integration** dropdown menu, select **HTTP Endpoint**.
   1. Name the integration.
   1. Toggle the **Active** alert setting to display the **URL** and **Authorization Key**
63
      for the webhook configuration. You must also input the URL and Authorization Key
Sarah Waldner's avatar
Sarah Waldner committed
64 65
      in your external service.
   1. _(Optional)_ To generate a test alert to test the new integration, enter a
66
      sample payload, then click **Save and test alert payload**. Valid JSON is required.
Sarah Waldner's avatar
Sarah Waldner committed
67 68 69 70 71 72
   1. Click **Save Integration**.

The new HTTP Endpoint displays in the [integrations list](#integrations-list).
You can edit the integration by selecting the **{pencil}** pencil icon on the right
side of the integrations list.

73 74
### External Prometheus integration

Sarah Waldner's avatar
Sarah Waldner committed
75 76 77
For GitLab versions 13.1 and greater, please read
[External Prometheus Instances](../metrics/alerts.md#external-prometheus-instances)
to configure alerts for this integration.
78

Sarah Waldner's avatar
Sarah Waldner committed
79
## Customize the alert payload outside of GitLab
80

Sarah Waldner's avatar
Sarah Waldner committed
81
For all integration types, you can customize the payload by sending the following
82
parameters. All fields are optional. If the incoming alert does not contain a value for the `Title` field, a default value of `New: Incident` will be applied.
83 84 85

| Property                  | Type            | Description |
| ------------------------- | --------------- | ----------- |
86
| `title`                   | String          | The title of the incident. |
87
| `description`             | String          | A high-level summary of the problem. |
Sarah Waldner's avatar
Sarah Waldner committed
88
| `start_time`              | DateTime        | The time of the incident. If none is provided, a timestamp of the issue is used. |
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
| `end_time`                | DateTime        | For existing alerts only. When provided, the alert is resolved and the associated incident is closed. |
| `service`                 | String          | The affected service. |
| `monitoring_tool`         | String          |  The name of the associated monitoring tool. |
| `hosts`                   | String or Array | One or more hosts, as to where this incident occurred. |
| `severity`                | String          | The severity of the alert. Must be one of `critical`, `high`, `medium`, `low`, `info`, `unknown`. Default is `critical`. |
| `fingerprint`             | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. |
| `gitlab_environment_name` | String          | The name of the associated GitLab [environment](../../ci/environments/index.md). This can be used to associate your alert to your environment. |

You can also add custom fields to the alert's payload. The values of extra
parameters aren't limited to primitive types (such as strings or numbers), but
can be a nested JSON object. For example:

```json
{ "foo": { "bar": { "baz": 42 } } }
```

105
NOTE:
Sarah Waldner's avatar
Sarah Waldner committed
106 107
Ensure your requests are smaller than the
[payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads).
108 109 110 111 112 113 114 115 116 117 118

Example request:

```shell
curl --request POST \
  --data '{"title": "Incident title"}' \
  --header "Authorization: Bearer <authorization_key>" \
  --header "Content-Type: application/json" \
  <url>
```

119
The `<authorization_key>` and `<url>` values can be found when configuring an alert integration.
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144

Example payload:

```json
{
  "title": "Incident title",
  "description": "Short description of the incident",
  "start_time": "2019-09-12T06:00:55Z",
  "service": "service affected",
  "monitoring_tool": "value",
  "hosts": "value",
  "severity": "high",
  "fingerprint": "d19381d4e8ebca87b55cda6e8eee7385",
  "foo": {
    "bar": {
      "baz": 42
    }
  }
}
```

## Triggering test alerts

> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Core in 13.2.

145 146
After a [project maintainer or owner](../../user/permissions.md)
configures an integration, you can trigger a test
147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163
alert to confirm your integration works properly.

1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md).
1. Navigate to **Settings > Operations** in your project.
1. Click **Alerts endpoint** to expand the section.
1. Enter a sample payload in **Alert test payload** (valid JSON is required).
1. Click **Test alert payload**.

GitLab displays an error or success message, depending on the outcome of your test.

## Automatic grouping of identical alerts **(PREMIUM)**

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.

In GitLab versions 13.2 and greater, GitLab groups alerts based on their
payload. When an incoming alert contains the same payload as another alert
(excluding the `start_time` and `hosts` attributes), GitLab groups these alerts
164
together and displays a counter on the [Alert Management List](incidents.md)
165 166 167 168
and details pages.

If the existing alert is already `resolved`, GitLab creates a new alert instead.

169
![Alert Management List](img/alert_list_v13_1.png)
170

171 172 173
## Link to your Opsgenie Alerts

> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.
174

175
WARNING:
176
We are building deeper integration with Opsgenie and other alerting tools through
177
[HTTP endpoint integrations](#single-http-endpoint) so you can see alerts in
178
the GitLab interface. As a result, the previous direct link to Opsgenie Alerts from
179 180
the GitLab alerts list is deprecated in
GitLab versions [13.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/273657).
181 182 183 184

You can monitor alerts using a GitLab integration with [Opsgenie](https://www.atlassian.com/software/opsgenie).

If you enable the Opsgenie integration, you can't have other GitLab alert
185
services
186 187 188 189 190 191 192 193 194 195 196 197 198 199
active at the same time.

To enable Opsgenie integration:

1. Sign in as a user with Maintainer or Owner [permissions](../../user/permissions.md).
1. Navigate to **Operations > Alerts**.
1. In the **Integrations** select box, select **Opsgenie**.
1. Select the **Active** toggle.
1. In the **API URL** field, enter the base URL for your Opsgenie integration,
   such as `https://app.opsgenie.com/alert/list`.
1. Select **Save changes**.

After you enable the integration, navigate to the Alerts list page at
**Operations > Alerts**, and then select **View alerts in Opsgenie**.