Commit 19c47551 authored by Rémy Coutable's avatar Rémy Coutable

Merge branch '36236-auto-detect-metrics-for-auto-deployed-environments' into 'master'

Improve Prometheus docs, include nginx ingress

Closes #36236

See merge request !13438
parents 7963e2bb 69c9c83a
- group: Response metrics (NGINX Ingress)
priority: 10
metrics:
- title: "Throughput"
y_label: "Requests / Sec"
required_metrics:
- nginx_upstream_requests_total
weight: 1
queries:
- query_range: 'sum(rate(nginx_upstream_requests_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'
label: Total
unit: req / sec
- title: "Latency"
y_label: "Latency (ms)"
required_metrics:
- nginx_upstream_response_msecs_avg
weight: 1
queries:
- query_range: 'avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"})'
label: Average
unit: ms
- title: "HTTP Error Rate"
y_label: "HTTP 500 Errors / Sec"
required_metrics:
- nginx_upstream_responses_total
weight: 1
queries:
- query_range: 'sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'
label: HTTP Errors
unit: "errors / sec"
- group: Response metrics (HA Proxy)
priority: 10
metrics:
......@@ -68,18 +98,18 @@
- nginx_upstream_response_msecs_avg
weight: 1
queries:
- query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000'
- query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}})'
label: Upstream
unit: ms
- title: "HTTP Error Rate"
y_label: "Error Rate (%)"
y_label: "HTTP 500 Errors / Sec"
required_metrics:
- nginx_responses_total
weight: 1
queries:
- query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m])) / sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m]))'
- query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m]))'
label: HTTP Errors
unit: "%"
unit: "errors / sec"
- group: System metrics (Kubernetes)
priority: 5
metrics:
......
......@@ -607,10 +607,9 @@ exist, you should see something like:
- With GitLab 9.2, all deployments to an environment are shown directly on the
monitoring dashboard
If you have enabled Prometheus for collecting metrics, you can monitor the performance behavior of your app
through the environments.
If you have enabled [Prometheus for monitoring system and response metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html), you can monitor the performance behavior of your app running in each environment.
Once configured, GitLab will attempt to retrieve performance metrics for any
Once configured, GitLab will attempt to retrieve [supported performance metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus_library/metrics.html) for any
environment which has had a successful deployment. If monitoring data was
successfully retrieved, a Monitoring button will appear on the environment's
detail page.
......
......@@ -40,7 +40,7 @@ Installing and configuring Prometheus to monitor applications is fairly straight
### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments
With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>).
version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>).
1. Read how to configure the bundled Prometheus server in the
[Administration guide][gitlab-prometheus-k8s-monitor].
......@@ -133,6 +133,8 @@ to integrate with.
Once configured, GitLab will attempt to retrieve performance metrics for any
environment which has had a successful deployment.
GitLab will automatically scan the Prometheus server for known metrics and attempt to identify the metrics for a particular environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html).
[Learn more about monitoring environments.](../../../ci/environments.md#monitoring-environments)
## Determining the performance impact of a merge
......@@ -174,7 +176,7 @@ If the "Attempting to load performance data" screen continues to appear, it coul
[prometheus-docker-image]: https://hub.docker.com/r/prom/prometheus/
[prometheus-yml]:samples/prometheus.yml
[gitlab.com-ip-range]: https://gitlab.com/gitlab-com/infrastructure/issues/434
[ci-environment-slug]: https://docs.gitlab.com/ce/ci/variables/#predefined-variables-environment-variables
[ci-environment-slug]: ../../../ci/variables/#predefined-variables-environment-variables
[ce-8935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935
[ce-10408]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10408
[promgldocs]: ../../../administration/monitoring/prometheus/index.md
......@@ -4,6 +4,7 @@
GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
* [Kubernetes](kubernetes.md)
* [NGINX](nginx.md)
* [NGINX Ingress Controller](nginx_ingress.md)
* [HAProxy](haproxy.md)
* [Amazon Cloud Watch](cloudwatch.md)
......@@ -14,10 +15,7 @@ We have tried to surface the most important metrics for each exporter, and will
GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
GitLab will look for the required metrics which have a label that
matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
GitLab uses the defined queries and fills in the environment specific variables. Typically this involves looking for the [$CI_ENVIRONMENT_SLUG](https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables), but may also include other information such as the project's Kubernetes namespace. Each search query is defined in the [exporter specific documentation](#prometheus-metrics-library).
## Adding to the library
......
......@@ -8,8 +8,8 @@ GitLab has support for automatically detecting and monitoring NGINX. This is pro
| Name | Query |
| ---- | ----- |
| Throughput (req/sec) | sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000 |
| HTTP Error Rate (%) | sum(rate(haproxy_frontend_http_responses_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_responses_total{%{environment_filter}}[2m])) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) |
| HTTP Error Rate (HTTP Errors / sec) | rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m])) |
## Configuring Prometheus to monitor for NGINX metrics
......
# Monitoring NGINX Ingress Controller
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5
GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) of the ingress.
## Metrics supported
| Name | Query |
| ---- | ----- |
| Throughput (req/sec) | sum(rate(nginx_upstream_requests_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) |
| HTTP Error Rate (HTTP Errors / sec) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) |
## Configuring Prometheus to monitor for NGINX ingress metrics
The easiest way to get started is to use at least version 0.9.0 of [NGINX ingress](https://github.com/kubernetes/ingress/tree/master/controllers/nginx). If you are using NGINX as your Kubernetes ingress, there is [direct support](https://github.com/kubernetes/ingress/pull/423) for enabling Prometheus monitoring in the 0.9.0 release.
If you have deployed with the [gitlab-omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.md) Helm chart, these metrics will be automatically enabled and annotated for Prometheus monitoring.
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will search metrics with appropriate labels. In this case, the `upstream` label must be of the form `<Kubernetes Namespace>-<CI_ENVIRONMENT_SLUG>-*`.
If you have used [Auto Deploy](https://docs.gitlab.com/ee/ci/autodeploy/index.html) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment