Commit 7ee7d3f9 authored by Marcia Ramos's avatar Marcia Ramos

Merge branch '35129-update-prometheus-docs' into 'master'

Update Prometheus documentation for 9.4

Closes #35129

See merge request !12885
parents 64aec37d b64752c2
...@@ -602,9 +602,8 @@ exist, you should see something like: ...@@ -602,9 +602,8 @@ exist, you should see something like:
>**Notes:** >**Notes:**
> >
- For the monitor dashboard to appear, you need to: - For the monitor dashboard to appear, you need to:
- Have enabled the [Kubernetes integration][kube]
- Have your app deployed on Kubernetes
- Have enabled the [Prometheus integration][prom] - Have enabled the [Prometheus integration][prom]
- Configured Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/metrics.md)
- With GitLab 9.2, all deployments to an environment are shown directly on the - With GitLab 9.2, all deployments to an environment are shown directly on the
monitoring dashboard monitoring dashboard
......
...@@ -17,32 +17,27 @@ the settings page with a default template. To configure the template, see the ...@@ -17,32 +17,27 @@ the settings page with a default template. To configure the template, see the
Integration with Prometheus requires the following: Integration with Prometheus requires the following:
1. GitLab 9.0 or higher 1. GitLab 9.0 or higher
1. The [Kubernetes integration must be enabled][kube] on your project 1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md)
1. Your app must be deployed on [Kubernetes][]
1. Prometheus must be configured to collect Kubernetes metrics
1. Each metric must be have a label to indicate the environment 1. Each metric must be have a label to indicate the environment
1. GitLab must have network connectivity to the Prometheus sever 1. GitLab must have network connectivity to the Prometheus server
There are a few steps necessary to set up integration between Prometheus and ## Getting started with Prometheus monitoring
GitLab.
## Configuring Prometheus to collect Kubernetes metrics Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring.
In order for Prometheus to collect Kubernetes metrics, you first must have a * If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes).
Prometheus server up and running. You have two options here: * If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes).
- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the ### Getting started with Prometheus outside of Kubernetes
[bundled version of Prometheus][promgldocs]. In that case, follow the info in the
[Omnibus GitLab section](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes)
below.
- If you are using GitLab.com or installed GitLab outside of Kubernetes, you
will likely need to run a Prometheus server within the Kubernetes cluster.
Once installed, the easiest way to monitor Kubernetes is to simply use
Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
In that case, follow the instructions on
[configuring your own Prometheus server within Kubernetes](#configuring-your-own-prometheus-server-within-kubernetes).
### Configuring Omnibus GitLab Prometheus to monitor Kubernetes Installing and configuring Prometheus to monitor applications is fairly straight forward.
1. [Install Prometheus](https://prometheus.io/docs/introduction/install/)
1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md)
1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config)
### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments
With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>). version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>).
...@@ -74,7 +69,7 @@ kubectl apply -f path/to/prometheus.yml ...@@ -74,7 +69,7 @@ kubectl apply -f path/to/prometheus.yml
Once deployed, you should see the Prometheus service, deployment, and Once deployed, you should see the Prometheus service, deployment, and
pod start within the `prometheus` namespace. The server will begin to collect pod start within the `prometheus` namespace. The server will begin to collect
metrics from each Kubernetes Node in the cluster, based on the configuration metrics from each Kubernetes Node in the cluster, based on the configuration
provided in the template. provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod).
Since GitLab is not running within Kubernetes, the template provides external Since GitLab is not running within Kubernetes, the template provides external
network access via a `NodePort` running on `30090`. This method allows access network access via a `NodePort` running on `30090`. This method allows access
...@@ -133,30 +128,6 @@ to integrate with. ...@@ -133,30 +128,6 @@ to integrate with.
![Configure Prometheus Service](img/prometheus_service_configuration.png) ![Configure Prometheus Service](img/prometheus_service_configuration.png)
## Metrics and Labels
GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
and `container_memory_usage_bytes`. These metrics are collected from the
Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
container or Pod running in the cluster.
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which pods are associated. To do that,
GitLab will specifically request metrics that have an `environment` tag that
matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
configuring Prometheus above, the `environment` will be automatically added.
### GitLab Prometheus queries
The queries utilized by GitLab are shown in the following table.
| Metric | Query |
| ------ | ----- |
| Average Memory (MB) | `(sum(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024` |
| Average CPU Utilization (%) | `sum(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) * 100` |
## Monitoring CI/CD Environments ## Monitoring CI/CD Environments
Once configured, GitLab will attempt to retrieve performance metrics for any Once configured, GitLab will attempt to retrieve performance metrics for any
...@@ -168,8 +139,9 @@ environment which has had a successful deployment. ...@@ -168,8 +139,9 @@ environment which has had a successful deployment.
> [Introduced][ce-10408] in GitLab 9.2. > [Introduced][ce-10408] in GitLab 9.2.
> GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages. > GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages.
> Requires [Kubernetes](prometheus_library/kubernetes.md) metrics
Developers can view the performance impact of their changes within the merge Developers can view theperformance impact of their changes within the merge
request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot
indicates when the current changes were deployed, with up to 30 minutes of indicates when the current changes were deployed, with up to 30 minutes of
performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after
......
# Monitoring AWS Resources
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4
GitLab has support for automatically detecting and monitoring AWS resources, starting with the [Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/). This is provided by leveraging the official [Cloudwatch exporter](https://github.com/prometheus/cloudwatch_exporter), which translates [Cloudwatch metrics](https://aws.amazon.com/cloudwatch/) into a Prometheus readable form.
## Metrics supported
| Name | Query |
| ---- | ----- |
| Throughput (req/sec) | sum(aws_elb_request_count_sum{%{environment_filter}}) / 60 |
| Latency (ms) | avg(aws_elb_latency_average{%{environment_filter}}) * 1000 |
| HTTP Error Rate (%) | sum(aws_elb_httpcode_backend_5_xx_sum{%{environment_filter}}) / sum(aws_elb_request_count_sum{%{environment_filter}}) |
## Configuring Prometheus to monitor for Cloudwatch metrics
To get started with Cloudwatch monitoring, you should install and configure the [Cloudwatch exporter](https://github.com/hnlq715/nginx-vts-exporter) which retrieves and parses the specified Cloudwatch metrics and translates them into a Prometheus monitoring endpoint.
Right now, the only AWS resource supported is the Elastic Load Balancer, whose Cloudwatch metrics can be found [here](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-cloudwatch-metrics.html).
A sample Cloudwatch Exporter configuration file, configured for basic AWS ELB monitoring, is [available for download](../samples/cloudwatch.yml).
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
# Monitoring Kubernetes
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor). No additional services or exporters are needed.
## Metrics supported
| Name | Query |
| ---- | ----- |
| Average Memory Usage (MB) | (sum(container_memory_usage_bytes{container_name!="POD",%{environment_filter}}) / count(container_memory_usage_bytes{container_name!="POD",%{environment_filter}})) /1024/1024 |
| Average CPU Utilization (%) | sum(rate(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}) * 100 |
## Configuring Prometheus to monitor for Kubernetes node metrics
In order for Prometheus to collect Kubernetes metrics, you first must have a
Prometheus server up and running. You have two options here:
- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes).
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
If you are using [GitLab Auto-Deploy][autodeploy] and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added.
# Prometheus Metrics library
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
* [Kubernetes](kubernetes.md)
* [NGINX](nginx.md)
* [Amazon Cloud Watch](cloudwatch.md)
We have tried to surface the most important metrics for each exporter, and will be continuing to add support for additional exporters in future releases. If you would like to add support for other official exporters, [contributions](#adding-to-the-library) are welcome.
## Identifying Environments
GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
GitLab will look for the required metrics which have a label that
matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
## Adding to the library
We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `additional_metrics.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml) file.
> Note: The library is only for monitoring public, common, system services which all customers can benefit from. Support for monitoring [customer proprietary metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/2273) will be added in a subsequent release.
# Monitoring NGINX
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4
GitLab has support for automatically detecting and monitoring NGINX. This is provided by leveraging the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter), which translates [VTS statistics](https://github.com/vozlt/nginx-module-vts) into a Prometheus readable form.
## Metrics supported
| Name | Query |
| ---- | ----- |
| Throughput (req/sec) | sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000 |
| HTTP Error Rate (%) | sum(nginx_responses_total{status_code="5xx", %{environment_filter}}) / sum(nginx_responses_total{server_zone!="*", server_zone!="_", %{environment_filter}}) |
## Configuring Prometheus to monitor for NGINX metrics
To get started with NGINX monitoring, you should first enable the [VTS statistics](https://github.com/vozlt/nginx-module-vts)) module for your NGINX server. This will capture and display statistics in an HTML readable form. Next, you should install and configure the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter) which parses these statistics and translates them into a Prometheus monitoring endpoint.
If you are using NGINX as your Kubernetes ingress, there is [upcoming direct support](https://github.com/kubernetes/ingress/pull/423) for enabling Prometheus monitoring in the 0.9.0 release.
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
region: us-east-1
metrics:
- aws_namespace: AWS/ELB
aws_metric_name: RequestCount
aws_dimensions: [AvailabilityZone, LoadBalancerName]
aws_dimension_select:
LoadBalancerName: [gitlab-ha-lb]
aws_statistics: [Sum]
- aws_namespace: AWS/ELB
aws_metric_name: Latency
aws_dimensions: [AvailabilityZone, LoadBalancerName]
aws_dimension_select:
LoadBalancerName: [gitlab-ha-lb]
aws_statistics: [Average]
- aws_namespace: AWS/ELB
aws_metric_name: HTTPCode_Backend_2XX
aws_dimensions: [AvailabilityZone, LoadBalancerName]
aws_dimension_select:
LoadBalancerName: [gitlab-ha-lb]
aws_statistics: [Sum]
- aws_namespace: AWS/ELB
aws_metric_name: HTTPCode_Backend_5XX
aws_dimensions: [AvailabilityZone, LoadBalancerName]
aws_dimension_select:
LoadBalancerName: [gitlab-ha-lb]
aws_statistics: [Sum]
...@@ -24,6 +24,44 @@ data: ...@@ -24,6 +24,44 @@ data:
target_label: environment target_label: environment
regex: (.+)-.+-.+ regex: (.+)-.+-.+
replacement: $1 replacement: $1
- job_name: kubernetes-pods
tls_config:
ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
insecure_skip_verify: true
bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
kubernetes_sd_configs:
- role: pod
api_server: https://kubernetes.default.svc:443
tls_config:
ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
action: keep
regex: 'true'
- source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
action: replace
target_label: __metrics_path__
regex: "(.+)"
- source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
action: replace
regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
replacement: "$1:$2"
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels:
- __meta_kubernetes_namespace
action: replace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_pod_name
action: replace
target_label: kubernetes_pod_name
--- ---
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment