Moved from CE: Squashed commit of the following:

commit 766a42a96393f502d439c1f0beb0b6cfb2c228d4
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 17:58:45 2018 +0100

    Move prometheus adapter to app/models/concerns

commit ca84eed49811cf3064a2e5ea611af1c947c590d7
Merge: 66702099586 cb504cedc2a
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 15:36:03 2018 +0100

    Merge remote-tracking branch 'upstream/backport_custom_metrics_ce_components' into 38783-add-cluster-metrics.yml

    + fix failing tests

    # Conflicts:
    #	app/controllers/projects/prometheus_controller.rb
    #	app/models/project_services/prometheus_service.rb
    #	lib/gitlab/prometheus/queries/query_additional_metrics.rb
    #	spec/models/project_services/prometheus_service_spec.rb

commit cb504cedc2a6e353ffb56833334681e3da09fc14
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 14:46:15 2018 +0100

    Fix prometheus_service found by find_or_initialize_service

commit 928b84c72c2c4c46e1785b9a943c1822a137de16
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 23:15:11 2018 +0100

    additional metrics and backported tests

commit 41291383b4b6976af94eaf9e1a7b2e2a172310e7
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 21:57:45 2018 +0100

    Introduce Metrics controller and retire prometheus controller

commit 6e7492e4c7ffa9d8621f09198071bc14be875976
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 21:46:37 2018 +0100

    Backport PrometheusClient::Error and all->common_metrics rename

commit 66702099586c864a7f78970a0ee0dd9f3c2beeef
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 20:23:03 2018 +0100

    fix failing tests

commit ce921ea723cde61b2441ba98c9aca97606c719c8
Author: Mike Greiling <mike@pixelcog.com>
Date:   Wed Feb 21 23:47:30 2018 -0600

    add labels to cluster metrics

commit 993830c6892e4fbf53d3f72da00002d642c7e9b2
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:20:49 2018 +0100

    Fix formatting probelms and few small tests

commit 66ee65d8e3f747c90d986bc3056178422156bc8e
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:08:03 2018 +0100

    stop using in deployment tests environment.id

commit 62c91978d15f0369988521363dae24bd7510d68d
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:06:33 2018 +0100

    fix prometheus_controller and adapter tests

commit 977b1d34c1d03c7233582e8328f85caf634895ed
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 00:19:39 2018 +0100

    finish up active? -> can_query? rename

commit e614f7daee58a9758d83ba3efe1649c8b80bc1e2
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 23:51:33 2018 +0100

    deployment prometheus adapter tests fix

commit ebd726c114a6026fef0adf3eba6ee1972530148a
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 21:40:40 2018 +0100

    Move environment dependant tests to environment

commit 6d31311cd3729c29233283dded70e03a4a9a3c97
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 20:40:24 2018 +0100

    update monitoring service and move adding dpeloyment_time to deployment model

commit 60b6bf391ab36846dce122bc6b0c5196a186267c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 20:15:46 2018 +0100

    adjust deployment spec and prometheus specs

commit 6681662cf1c028aff2ff94aa0501732cb7119ba1
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:30:20 2018 +0100

    Revert changes to reactive caching

commit e282f86c45a056889f57d3f7fd23a81c88efff6a
Merge: 5751c73df59 6844a2df
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:08:17 2018 +0100

    Merge remote-tracking branch 'upstream/master' into 38783-add-cluster-metrics.yml

commit 5751c73df59d0a03840a1b4b71b0637670f971a6
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:07:43 2018 +0100

    rename active? to can_query? and cleanup environment prometheus router

commit 3f3c6e1d33dcd9315979daf26a95f2aab83a7de9
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 21:51:43 2018 +0100

    Fix tests, and only use prometheus service if its active

commit 6345838bac584c213b665d334252ccab202cb271
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 16:24:27 2018 +0100

    Fix typo

commit 7a585d32afe8da050b5615b1d036a550e06479f5
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:29:40 2018 +0100

    Cluster id is not required

commit e6af62afb11fa380f6aff1c31a81bcc9bab3b1eb
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:15:46 2018 +0100

    Result transformation support

commit f3b1bd7c67894f44efe33591ddb70093bd620c03
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:07:07 2018 +0100

    Fix rubocop warning and exten cluster query timeframe

commit be77947cea64261a4d3dead33c3c57f413a9880c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 14:42:47 2018 +0100

    Fix additional metrics test

commit eb3922e16221abe16f59fae1c38122f227643343
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 14:34:44 2018 +0100

    rename prometheus adapter methods

commit 045476cd08b21593818b274ae8a44d19b705523f
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 13:19:36 2018 +0100

    Make prometheus adapter a module

commit f2daf050d8c689f72c4c61207930bc53c331f12e
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 12:00:44 2018 +0100

    refactoring wip

commit 52e4ef5587794e811dc10a0f2dca522342a865da
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Sun Feb 18 20:25:55 2018 +0100

    cleanup prometheus adapter concept

commit 3887365faab9dfcd9c00bcfc501d09ac62431a03
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Sun Feb 18 19:22:11 2018 +0100

    Refactor out deployment id, Rename PrometheusQuerier to PrometheusAdapter

commit aa2fc2df57bd72c9a5e94f66d1f1e23990be6c3f
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 16 23:59:54 2018 +0100

    Refactor prometheus client

commit e43c1ca9d9874d6cf1569f40fa1aca158d9d5d91
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 20:24:45 2018 +0100

    Use initial version of cluster_metrics.yml

commit 867821ce0b2609ebf8994220aa8e3a94d66a01e0
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 05:24:14 2018 +0100

    Fix querying cluster metrics

commit 1601e002a064cbb10ffe110a19433c5662858f1d
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 04:49:03 2018 +0100

    Queues for unicersal querier

commit 5db198fdc925c0223be24939b76da1d544dd569c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 03:35:17 2018 +0100

    refactor reactive caching and prometheus querying

commit b0fc00e8c9d21e961ef44b0129103e2a62928b52
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 00:12:03 2018 +0100

    Add generic query additional metrics

commit ffe76e6a9ba196bccff22c4880e0384959ad5e48
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 13 15:40:15 2018 +0100

    Cluster Metric yml initial

app/assets/javascripts/monitoring/utils/multiple_time_series.js

@@ -76,7 +76,7 @@ function queryTimeSeries(query, graphWidth, graphHeight, graphHeightOffset, xDom
       metricTag = seriesCustomizationData.value || timeSeriesMetricLabel;
       [lineColor, areaColor] = pickColor(seriesCustomizationData.color);
     } else {
-      metricTag = timeSeriesMetricLabel || `series ${timeSeriesNumber + 1}`;
+      metricTag = timeSeriesMetricLabel || query.label || `series ${timeSeriesNumber + 1}`;
       [lineColor, areaColor] = pickColor();
     }
 

app/controllers/projects/clusters_controller.rb

@@ -63,6 +63,22 @@ class Projects::ClustersController < Projects::ApplicationController
     end
   end
 
+  def metrics
+    render_404 unless prometheus_adapter&.can_query?
+
+    respond_to do |format|
+      format.json do
+        metrics = prometheus_adapter.query(:cluster) || {}
+
+        if metrics.any?
+          render json: metrics
+        else
+          head :no_content
+        end
+      end
+    end
+  end
+
   private
 
   def cluster
@@ -70,6 +86,12 @@ class Projects::ClustersController < Projects::ApplicationController
                                  .present(current_user: current_user)
   end
 
+  def prometheus_adapter
+    return unless cluster&.application_prometheus&.installed?
+
+    cluster.application_prometheus
+  end
+
   def update_params
     if cluster.managed?
       params.require(:cluster).permit(

app/controllers/projects/deployments_controller.rb

@@ -24,7 +24,7 @@ class Projects::DeploymentsController < Projects::ApplicationController
   end
 
   def additional_metrics
-    return render_404 unless deployment.has_additional_metrics?
+    return render_404 unless deployment.has_metrics?
 
     respond_to do |format|
       format.json do

app/controllers/projects/prometheus/metrics_controller.rb

+module Projects
+  module Prometheus
+    class MetricsController < Projects::ApplicationController
+      before_action :authorize_admin_project!
+      before_action :require_prometheus_metrics!
+
+      def active_common
+        respond_to do |format|
+          format.json do
+            matched_metrics = prometheus_adapter.query(:matched_metrics) || {}
+
+            if matched_metrics.any?
+              render json: matched_metrics
+            else
+              head :no_content
+            end
+          end
+        end
+      end
+
+      private
+
+      def prometheus_adapter
+        project.prometheus_service
+      end
+
+      def require_prometheus_metrics!
+        render_404 unless prometheus_adapter.can_query?
+      end
+    end
+  end
+end

app/controllers/projects/prometheus_controller.rb

@@ -5,7 +5,7 @@ class Projects::PrometheusController < Projects::ApplicationController
   def active_metrics
     respond_to do |format|
       format.json do
-        matched_metrics = project.prometheus_service.matched_metrics || {}
+        matched_metrics = prometheus_adapter.query(:matched_metrics) || {}
 
         if matched_metrics.any?
           render json: matched_metrics
@@ -18,7 +18,11 @@ class Projects::PrometheusController < Projects::ApplicationController
 
   private
 
+  def prometheus_adapter
+    project.prometheus_service
+  end
+
   def require_prometheus_metrics!
-    render_404 unless project.prometheus_service.present?
+    render_404 unless prometheus_adapter.can_query?
   end
 end

app/models/clusters/applications/prometheus.rb

 module Clusters
   module Applications
     class Prometheus < ActiveRecord::Base
+      include PrometheusAdapter
+
       VERSION = "2.0.0".freeze
 
       self.table_name = 'clusters_applications_prometheus'
@@ -38,7 +40,7 @@ module Clusters
         Gitlab::Kubernetes::Helm::InstallCommand.new(name, chart: chart, chart_values_file: chart_values_file)
       end
 
-      def proxy_client
+      def prometheus_client
         return unless kube_client
 
         proxy_url = kube_client.proxy_url('service', service_name, service_port, Gitlab::Kubernetes::Helm::NAMESPACE)

app/models/clusters/cluster.rb

@@ -52,7 +52,6 @@ module Clusters
     scope :disabled, -> { where(enabled: false) }
 
     scope :for_environment, -> (env) { where(environment_scope: ['*', '', env.slug]) }
-    scope :for_all_environments, -> { where(environment_scope: ['*', '']) }
 
     def status_name
       if provider

app/models/clusters/platforms/kubernetes.rb

@@ -143,7 +143,7 @@ module Clusters
       end
 
       def kubeclient_ssl_options
-        opts = { verify_ssl: OpenSSL::SSL::VERIFY_PEER }
+        opts = { verify_ssl: OpenSSL::SSL::VERIFY_NONE }
 
         if ca_pem.present?
           opts[:cert_store] = OpenSSL::X509::Store.new

app/models/concerns/prometheus_adapter.rb

+module PrometheusAdapter
+  extend ActiveSupport::Concern
+
+  included do
+    include ReactiveCaching
+
+    self.reactive_cache_key = ->(adapter) { [adapter.class.model_name.singular, adapter.id] }
+    self.reactive_cache_lease_timeout = 30.seconds
+    self.reactive_cache_refresh_interval = 30.seconds
+    self.reactive_cache_lifetime = 1.minute
+
+    def prometheus_client
+      raise NotImplementedError
+    end
+
+    def prometheus_client_wrapper
+      Gitlab::PrometheusClient.new(prometheus_client)
+    end
+
+    def can_query?
+      prometheus_client.present?
+    end
+
+    def query(query_name, *args)
+      return unless can_query?
+
+      query_class = Gitlab::Prometheus::Queries.const_get("#{query_name.to_s.classify}Query")
+
+      args.map! { |arg| arg.id }
+
+      with_reactive_cache(query_class.name, *args, &query_class.method(:transform_reactive_result))
+    end
+
+    # Cache metrics for specific environment
+    def calculate_reactive_cache(query_class_name, *args)
+      return unless prometheus_client
+
+      data = Kernel.const_get(query_class_name).new(prometheus_client_wrapper).query(*args)
+      {
+        success: true,
+        data: data,
+        last_update: Time.now.utc
+      }
+    rescue Gitlab::PrometheusClient::Error => err
+      { success: false, result: err.message }
+    end
+  end
+end

app/models/deployment.rb

@@ -98,28 +98,29 @@ class Deployment < ActiveRecord::Base
   end
 
   def has_metrics?
-    project.monitoring_service.present?
+    prometheus_adapter&.can_query?
   end
 
   def metrics
     return {} unless has_metrics?
 
-    project.monitoring_service.deployment_metrics(self)
-  end
-
-  def has_additional_metrics?
-    project.prometheus_service.present?
+    metrics = prometheus_adapter.query(:deployment, self)
+    metrics&.merge(deployment_time: created_at.to_i) || {}
   end
 
   def additional_metrics
-    return {} unless project.prometheus_service.present?
+    return {} unless has_metrics?
 
-    metrics = project.prometheus_service.additional_deployment_metrics(self)
+    metrics = prometheus_adapter.query(:additional_metrics_deployment, self)
     metrics&.merge(deployment_time: created_at.to_i) || {}
   end
 
   private
 
+  def prometheus_adapter
+    environment.prometheus_adapter
+  end
+
   def ref_path
     File.join(environment.ref_path, 'deployments', iid.to_s)
   end

app/models/environment.rb

@@ -154,21 +154,15 @@ class Environment < ActiveRecord::Base
   end
 
   def has_metrics?
-    project.monitoring_service.present? && available? && last_deployment.present?
+    prometheus_adapter&.can_query? && available? && last_deployment.present?
   end
 
   def metrics
-    project.monitoring_service.environment_metrics(self) if has_metrics?
-  end
-
-  def has_additional_metrics?
-    project.prometheus_service.present? && available? && last_deployment.present?
+    prometheus_adapter.query(:environment, self) if has_metrics?
   end
 
   def additional_metrics
-    if has_additional_metrics?
-      project.prometheus_service.additional_environment_metrics(self)
-    end
+    prometheus_adapter.query(:additional_metrics_environment, self) if has_metrics?
   end
 
   def slug
@@ -234,6 +228,27 @@ class Environment < ActiveRecord::Base
     self.environment_type || self.name
   end
 
+  def prometheus_adapter
+    @prometheus_adapter ||= if service_prometheus_adapter.can_query?
+                              service_prometheus_adapter
+                            else
+                              cluster_prometheus_adapter
+                            end
+  end
+
+  def service_prometheus_adapter
+    project.find_or_initialize_service('prometheus')
+  end
+
+  def cluster_prometheus_adapter
+    # sort results by descending order based on environment_scope being longer
+    # thus more closely matching environment slug
+    clusters = project.clusters.enabled.for_environment(self).sort_by { |c| c.environment_scope&.length }.reverse!
+
+    cluster = clusters&.detect { |cluster| cluster.application_prometheus&.installed? }
+    cluster&.application_prometheus
+  end
+
   private
 
   # Slugifying a name may remove the uniqueness guarantee afforded by it being

app/models/project_services/monitoring_service.rb

@@ -9,11 +9,11 @@ class MonitoringService < Service
     %w()
   end
 
-  def environment_metrics(environment)
+  def can_query?
     raise NotImplementedError
   end
 
-  def deployment_metrics(deployment)
+  def query(_, *_)
     raise NotImplementedError
   end
 end

app/models/project_services/prometheus_service.rb

 class PrometheusService < MonitoringService
-  include ReactiveService
-
-  self.reactive_cache_lease_timeout = 30.seconds
-  self.reactive_cache_refresh_interval = 30.seconds
-  self.reactive_cache_lifetime = 1.minute
+  include PrometheusAdapter
 
   #  Access to prometheus is directly through the API
   prop_accessor :api_url
@@ -66,63 +62,15 @@ class PrometheusService < MonitoringService
 
   # Check we can connect to the Prometheus API
   def test(*args)
-    client.ping
+    Gitlab::PrometheusClient.new(prometheus_client).ping
 
     { success: true, result: 'Checked API endpoint' }
-  rescue Gitlab::PrometheusError => err
+  rescue Gitlab::PrometheusClient::Error => err
     { success: false, result: err }
   end
 
-  def environment_metrics(environment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::EnvironmentQuery.name, environment.id, &method(:rename_data_to_metrics))
-  end
-
-  def deployment_metrics(deployment)
-    metrics = with_reactive_cache(Gitlab::Prometheus::Queries::DeploymentQuery.name, deployment.environment.id, deployment.id, &method(:rename_data_to_metrics))
-    metrics&.merge(deployment_time: deployment.created_at.to_i) || {}
-  end
-
-  def additional_environment_metrics(environment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::AdditionalMetricsEnvironmentQuery.name, environment.id, &:itself)
-  end
-
-  def additional_deployment_metrics(deployment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::AdditionalMetricsDeploymentQuery.name, deployment.environment.id, deployment.id, &:itself)
-  end
-
-  def matched_metrics
-    with_reactive_cache(Gitlab::Prometheus::Queries::MatchedMetricsQuery.name, &:itself)
-  end
-
-  # Cache metrics for specific environment
-  def calculate_reactive_cache(query_class_name, *args)
-    return unless active? && project && !project.pending_delete?
-
-    environment_id = args.first
-    client = client(environment_id)
-
-    data = Kernel.const_get(query_class_name).new(client).query(*args)
-    {
-      success: true,
-      data: data,
-      last_update: Time.now.utc
-    }
-  rescue Gitlab::PrometheusError => err
-    { success: false, result: err.message }
-  end
-
-  def client(environment_id = nil)
-    if manual_configuration?
-      Gitlab::PrometheusClient.new(RestClient::Resource.new(api_url))
-    else
-      cluster = cluster_with_prometheus(environment_id)
-      raise Gitlab::PrometheusError, "couldn't find cluster with Prometheus installed" unless cluster
-
-      rest_client = client_from_cluster(cluster)
-      raise Gitlab::PrometheusError, "couldn't create proxy Prometheus client" unless rest_client
-
-      Gitlab::PrometheusClient.new(rest_client)
-    end
+  def prometheus_client
+    RestClient::Resource.new(api_url) if api_url && manual_configuration? && active?
   end
 
   def prometheus_installed?
@@ -134,29 +82,6 @@ class PrometheusService < MonitoringService
 
   private
 
-  def cluster_with_prometheus(environment_id = nil)
-    clusters = if environment_id
-                 ::Environment.find_by(id: environment_id).try do |env|
-                   # sort results by descending order based on environment_scope being longer
-                   # thus more closely matching environment slug
-                   project.clusters.enabled.for_environment(env).sort_by { |c| c.environment_scope&.length }.reverse!
-                 end
-               else
-                 project.clusters.enabled.for_all_environments
-               end
-
-    clusters&.detect { |cluster| cluster.application_prometheus&.installed? }
-  end
-
-  def client_from_cluster(cluster)
-    cluster.application_prometheus.proxy_client
-  end
-
-  def rename_data_to_metrics(metrics)
-    metrics[:metrics] = metrics.delete :data
-    metrics
-  end
-
   def synchronize_service_state!
     self.active = prometheus_installed? || manual_configuration?
 

app/views/projects/services/prometheus/_show.html.haml

@@ -7,7 +7,7 @@
       = link_to s_('PrometheusService|More information'), help_page_path('user/project/integrations/prometheus')
 
   .col-lg-9
-    .panel.panel-default.js-panel-monitored-metrics{ data: { "active-metrics" => "#{project_prometheus_active_metrics_path(@project, :json)}" } }
+    .panel.panel-default.js-panel-monitored-metrics{ data: { active_metrics: active_common_project_prometheus_metrics_path(@project, :json) } }
       .panel-heading
         %h3.panel-title
           = s_('PrometheusService|Monitored')

config/prometheus/cluster_metrics.yml

+- group: Cluster Health
+  priority: 1
+  metrics:
+  - title: "CPU Usage"
+    y_label: "CPU"
+    required_metrics: ['container_cpu_usage_seconds_total']
+    weight: 1
+    queries:
+    - query_range: 'sum(rate(container_cpu_usage_seconds_total{id="/"}[15m]))'
+      label: Usage
+      unit: "cores"
+    - query_range: 'sum(kube_node_status_capacity_cpu_cores)'
+      label: Capacity
+      unit: "cores"
+  - title: "Memory usage"
+    y_label: "Memory"
+    required_metrics: ['container_memory_usage_bytes']
+    weight: 1
+    queries:
+    - query_range: 'sum(container_memory_usage_bytes{id="/"})/2^30'
+      label: Usage
+      unit: "GiB"
+    - query_range: 'sum(kube_node_status_capacity_memory_bytes)/2^30'
+      label: Capacity
+      unit: "GiB"

config/routes/project.rb

@@ -74,7 +74,9 @@ constraints(ProjectUrlConstrainer.new) do
       resource :mattermost, only: [:new, :create]
 
       namespace :prometheus do
-        get :active_metrics
+        resources :metrics, constraints: { id: %r{[^\/]+} }, only: [] do
+          get :active_common, on: :collection
+        end
       end
 
       resources :deploy_keys, constraints: { id: /\d+/ }, only: [:index, :new, :create, :edit, :update] do
@@ -237,6 +239,7 @@ constraints(ProjectUrlConstrainer.new) do
 
         member do
           get :status, format: :json
+          get :metrics, format: :json
 
           scope :applications do
             post '/:application', to: 'clusters/applications#create', as: :install_applications

lib/gitlab/prometheus/additional_metrics_parser.rb

 module Gitlab
   module Prometheus
     module AdditionalMetricsParser
+      CONFIG_ROOT = 'config/prometheus'.freeze
+      MUTEX = Mutex.new
       extend self
 
-      def load_groups_from_yaml
-        additional_metrics_raw.map(&method(:group_from_entry))
+      def load_groups_from_yaml(file_name = 'additional_metrics.yml')
+        yaml_metrics_raw(file_name).map(&method(:group_from_entry))
       end
 
       private
@@ -22,13 +24,20 @@ module Gitlab
         MetricGroup.new(entry).tap(&method(:validate!))
       end
 
-      def additional_metrics_raw
-        load_yaml_file&.map(&:deep_symbolize_keys).freeze
+      def yaml_metrics_raw(file_name)
+        load_yaml_file(file_name)&.map(&:deep_symbolize_keys).freeze
       end
 
-      def load_yaml_file
-        @loaded_yaml_file ||= YAML.load_file(Rails.root.join('config/prometheus/additional_metrics.yml'))
+      # rubocop:disable Gitlab/ModuleWithInstanceVariables
+      def load_yaml_file(file_name)
+        return YAML.load_file(Rails.root.join(CONFIG_ROOT, file_name)) if Rails.env.development?
+
+        MUTEX.synchronize do
+          @loaded_yaml_cache ||= {}
+          @loaded_yaml_cache[file_name] ||= YAML.load_file(Rails.root.join(CONFIG_ROOT, file_name))
+        end
       end
+      # rubocop:enable Gitlab/ModuleWithInstanceVariables
     end
   end
 end

lib/gitlab/prometheus/metric_group.rb

@@ -6,9 +6,14 @@ module Gitlab
       attr_accessor :name, :priority, :metrics
       validates :name, :priority, :metrics, presence: true
 
-      def self.all
+      def self.common_metrics
         AdditionalMetricsParser.load_groups_from_yaml
       end
+
+      # EE only
+      def self.for_project(_)
+        common_metrics
+      end
     end
   end
 end

lib/gitlab/prometheus/queries/additional_metrics_deployment_query.rb

@@ -4,9 +4,10 @@ module Gitlab
       class AdditionalMetricsDeploymentQuery < BaseQuery
         include QueryAdditionalMetrics
 
-        def query(environment_id, deployment_id)
+        def query(deployment_id)
           Deployment.find_by(id: deployment_id).try do |deployment|
             query_metrics(
+              deployment.project,
               common_query_context(
                 deployment.environment,
                 timeframe_start: (deployment.created_at - 30.minutes).to_f,

lib/gitlab/prometheus/queries/additional_metrics_environment_query.rb

@@ -7,6 +7,7 @@ module Gitlab
         def query(environment_id)
           ::Environment.find_by(id: environment_id).try do |environment|
             query_metrics(
+              environment.project,
               common_query_context(environment, timeframe_start: 8.hours.ago.to_f, timeframe_end: Time.now.to_f)
             )
           end

lib/gitlab/prometheus/queries/base_query.rb

@@ -20,6 +20,10 @@ module Gitlab
         def query(*args)
           raise NotImplementedError
         end
+
+        def self.transform_reactive_result(result)
+          result
+        end
       end
     end
   end

lib/gitlab/prometheus/queries/cluster_query.rb

+module Gitlab
+  module Prometheus
+    module Queries
+      class ClusterQuery < BaseQuery
+        include QueryAdditionalMetrics
+
+        def query
+          AdditionalMetricsParser.load_groups_from_yaml('cluster_metrics.yml')
+            .map(&query_group(base_query_context(8.hours.ago, Time.now)))
+        end
+      end
+    end
+  end
+end

lib/gitlab/prometheus/queries/deployment_query.rb

@@ -2,7 +2,7 @@ module Gitlab
   module Prometheus
     module Queries
       class DeploymentQuery < BaseQuery
-        def query(environment_id, deployment_id)
+        def query(deployment_id)
           Deployment.find_by(id: deployment_id).try do |deployment|
             environment_slug = deployment.environment.slug
 
@@ -25,6 +25,11 @@ module Gitlab
             }
           end
         end
+
+        def self.transform_reactive_result(result)
+          result[:metrics] = result.delete :data
+          result
+        end
       end
     end
   end

lib/gitlab/prometheus/queries/environment_query.rb

@@ -19,6 +19,11 @@ module Gitlab
             }
           end
         end
+
+        def self.transform_reactive_result(result)
+          result[:metrics] = result.delete :data
+          result
+        end
       end
     end
   end

lib/gitlab/prometheus/queries/matched_metrics_query.rb → lib/gitlab/prometheus/queries/matched_metric_query.rb

 module Gitlab
   module Prometheus
     module Queries
-      class MatchedMetricsQuery < BaseQuery
+      class MatchedMetricQuery < BaseQuery
         MAX_QUERY_ITEMS = 40.freeze
 
         def query
@@ -18,7 +18,7 @@ module Gitlab
         private
 
         def groups_data
-          metrics_groups = groups_with_active_metrics(Gitlab::Prometheus::MetricGroup.all)
+          metrics_groups = groups_with_active_metrics(Gitlab::Prometheus::MetricGroup.common_metrics)
           lookup = active_series_lookup(metrics_groups)
 
           groups = {}

lib/gitlab/prometheus/queries/query_additional_metrics.rb

@@ -2,10 +2,17 @@ module Gitlab
   module Prometheus
     module Queries
       module QueryAdditionalMetrics
-        def query_metrics(query_context)
+        def query_metrics(project, query_context)
+          matched_metrics(project).map(&query_group(query_context))
+            .select(&method(:group_with_any_metrics))
+        end
+
+        protected
+
+        def query_group(query_context)
           query_processor = method(:process_query).curry[query_context]
 
-          groups = matched_metrics.map do |group|
+          lambda do |group|
             metrics = group.metrics.map do |metric|
               {
                 title: metric.title,
@@ -21,8 +28,6 @@ module Gitlab
               metrics: metrics.select(&method(:metric_with_any_queries))
             }
           end
-
-          groups.select(&method(:group_with_any_metrics))
         end
 
         private
@@ -60,8 +65,8 @@ module Gitlab
           @available_metrics ||= client_label_values || []
         end
 
-        def matched_metrics
-          result = Gitlab::Prometheus::MetricGroup.all.map do |group|
+        def matched_metrics(project)
+          result = Gitlab::Prometheus::MetricGroup.for_project(project).map do |group|
             group.metrics.select! do |metric|
               metric.required_metrics.all?(&available_metrics.method(:include?))
             end
@@ -72,12 +77,17 @@ module Gitlab
         end
 
         def common_query_context(environment, timeframe_start:, timeframe_end:)
-          {
-            timeframe_start: timeframe_start,
-            timeframe_end: timeframe_end,
+          base_query_context(timeframe_start, timeframe_end).merge({
             ci_environment_slug: environment.slug,
             kube_namespace: environment.project.deployment_platform(environment: environment)&.actual_namespace || '',
             environment_filter: %{container_name!="POD",environment="#{environment.slug}"}
+          })
+        end
+
+        def base_query_context(timeframe_start, timeframe_end)
+          {
+            timeframe_start: timeframe_start,
+            timeframe_end: timeframe_end
           }
         end
       end

lib/gitlab/prometheus_client.rb

 module Gitlab
-  PrometheusError = Class.new(StandardError)
-
   # Helper methods to interact with Prometheus network services & resources
   class PrometheusClient
+    Error = Class.new(StandardError)
+    QueryError = Class.new(Gitlab::PrometheusClient::Error)
+
     attr_reader :rest_client, :headers
 
     def initialize(rest_client)
@@ -43,22 +44,26 @@ module Gitlab
       path = ['api', 'v1', type].join('/')
       get(path, args)
     rescue JSON::ParserError
-      raise PrometheusError, 'Parsing response failed'
+      raise PrometheusClient::Error, 'Parsing response failed'
     rescue Errno::ECONNREFUSED
-      raise PrometheusError, 'Connection refused'
+      raise PrometheusClient::Error, 'Connection refused'
     end
 
     def get(path, args)
       response = rest_client[path].get(params: args)
       handle_response(response)
     rescue SocketError
-      raise PrometheusError, "Can't connect to #{rest_client.url}"
+      raise PrometheusClient::Error, "Can't connect to #{rest_client.url}"
     rescue OpenSSL::SSL::SSLError
-      raise PrometheusError, "#{rest_client.url} contains invalid SSL data"
+      raise PrometheusClient::Error, "#{rest_client.url} contains invalid SSL data"
     rescue RestClient::ExceptionWithResponse => ex
+      if ex.response
         handle_exception_response(ex.response)
+      else
+        raise PrometheusClient::Error, "Network connection error"
+      end
     rescue RestClient::Exception
-      raise PrometheusError, "Network connection error"
+      raise PrometheusClient::Error, "Network connection error"
     end
 
     def handle_response(response)
@@ -66,16 +71,18 @@ module Gitlab
       if response.code == 200 && json_data['status'] == 'success'
         json_data['data'] || {}
       else
-        raise PrometheusError, "#{response.code} - #{response.body}"
+        raise PrometheusClient::Error, "#{response.code} - #{response.body}"
       end
     end
 
     def handle_exception_response(response)
-      if response.code == 400
+      if response.code == 200 && response['status'] == 'success'
+        response['data'] || {}
+      elsif response.code == 400
         json_data = JSON.parse(response.body)
-        raise PrometheusError, json_data['error'] || 'Bad data received'
+        raise PrometheusClient::QueryError, json_data['error'] || 'Bad data received'
       else
-        raise PrometheusError, "#{response.code} - #{response.body}"
+        raise PrometheusClient::Error, "#{response.code} - #{response.body}"
       end
     end
 

spec/controllers/projects/deployments_controller_spec.rb

@@ -129,10 +129,10 @@ describe Projects::DeploymentsController do
     end
 
     context 'when metrics are enabled' do
-      let(:prometheus_service) { double('prometheus_service') }
+      let(:prometheus_adapter) { double('prometheus_adapter', can_query?: true) }
 
       before do
-        allow(deployment.project).to receive(:prometheus_service).and_return(prometheus_service)
+        allow(deployment).to receive(:prometheus_adapter).and_return(prometheus_adapter)
       end
 
       context 'when environment has no metrics' do

spec/controllers/projects/prometheus_controller_spec.rb → spec/controllers/projects/prometheus/metrics_controller_spec.rb

-require('spec_helper')
+require 'spec_helper'
 
-describe Projects::PrometheusController do
+describe Projects::Prometheus::MetricsController do
   let(:user) { create(:user) }
-  let!(:project) { create(:project) }
+  let(:project) { create(:project) }
 
-  let(:prometheus_service) { double('prometheus_service') }
+  let(:prometheus_adapter) { double('prometheus_adapter', can_query?: true) }
 
   before do
     allow(controller).to receive(:project).and_return(project)
-    allow(project).to receive(:prometheus_service).and_return(prometheus_service)
+    allow(project).to receive(:prometheus_service).and_return(prometheus_adapter)
 
     project.add_master(user)
     sign_in(user)
   end
 
-  describe 'GET #active_metrics' do
+  describe 'GET #active_common' do
     context 'when prometheus metrics are enabled' do
       context 'when data is not present' do
         before do
-          allow(prometheus_service).to receive(:matched_metrics).and_return({})
+          allow(prometheus_adapter).to receive(:query).with(:matched_metrics).and_return({})
         end
 
         it 'returns no content response' do
-          get :active_metrics, project_params(format: :json)
+          get :active_common, project_params(format: :json)
 
           expect(response).to have_gitlab_http_status(204)
         end
@@ -32,11 +32,11 @@ describe Projects::PrometheusController do
         let(:sample_response) { { some_data: 1 } }
 
         before do
-          allow(prometheus_service).to receive(:matched_metrics).and_return(sample_response)
+          allow(prometheus_adapter).to receive(:query).with(:matched_metrics).and_return(sample_response)
         end
 
         it 'returns no content response' do
-          get :active_metrics, project_params(format: :json)
+          get :active_common, project_params(format: :json)
 
           expect(response).to have_gitlab_http_status(200)
           expect(json_response).to eq(sample_response.deep_stringify_keys)
@@ -45,7 +45,7 @@ describe Projects::PrometheusController do
 
       context 'when requesting non json response' do
         it 'returns not found response' do
-          get :active_metrics, project_params
+          get :active_common, project_params
 
           expect(response).to have_gitlab_http_status(404)
         end

spec/lib/gitlab/prometheus/queries/additional_metrics_deployment_query_spec.rb

@@ -7,7 +7,7 @@ describe Gitlab::Prometheus::Queries::AdditionalMetricsDeploymentQuery do
 
   include_examples 'additional metrics query' do
     let(:deployment) { create(:deployment, environment: environment) }
-    let(:query_params) { [environment.id, deployment.id] }
+    let(:query_params) { [deployment.id] }
 
     it 'queries using specific time' do
       expect(client).to receive(:query_range).with(anything,

spec/lib/gitlab/prometheus/queries/deployment_query_spec.rb

@@ -31,7 +31,7 @@ describe Gitlab::Prometheus::Queries::DeploymentQuery do
     expect(client).to receive(:query).with('avg(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="environment-slug"}[30m])) * 100',
                                            time: stop_time)
 
-    expect(subject.query(environment.id, deployment.id)).to eq(memory_values: nil, memory_before: nil, memory_after: nil,
+    expect(subject.query(deployment.id)).to eq(memory_values: nil, memory_before: nil, memory_after: nil,
                                                cpu_values: nil, cpu_before: nil, cpu_after: nil)
   end
 end

spec/lib/gitlab/prometheus/queries/matched_metrics_query_spec.rb → spec/lib/gitlab/prometheus/queries/matched_metric_query_spec.rb

 require 'spec_helper'
 
-describe Gitlab::Prometheus::Queries::MatchedMetricsQuery do
+describe Gitlab::Prometheus::Queries::MatchedMetricQuery do
   include Prometheus::MetricBuilders
 
   let(:metric_group_class) { Gitlab::Prometheus::MetricGroup }
@@ -24,7 +24,7 @@ describe Gitlab::Prometheus::Queries::MatchedMetricsQuery do
 
   context 'with one group where two metrics is found' do
     before do
-      allow(metric_group_class).to receive(:all).and_return([simple_metric_group])
+      allow(metric_group_class).to receive(:common_metrics).and_return([simple_metric_group])
       allow(client).to receive(:label_values).and_return(metric_names)
     end
 
@@ -70,7 +70,7 @@ describe Gitlab::Prometheus::Queries::MatchedMetricsQuery do
 
   context 'with one group where only one metric is found' do
     before do
-      allow(metric_group_class).to receive(:all).and_return([simple_metric_group])
+      allow(metric_group_class).to receive(:common_metrics).and_return([simple_metric_group])
       allow(client).to receive(:label_values).and_return('metric_a')
     end
 
@@ -99,7 +99,7 @@ describe Gitlab::Prometheus::Queries::MatchedMetricsQuery do
     let(:second_metric_group) { simple_metric_group(name: 'nameb', metrics: simple_metrics(added_metric_name: 'metric_c')) }
 
     before do
-      allow(metric_group_class).to receive(:all).and_return([simple_metric_group, second_metric_group])
+      allow(metric_group_class).to receive(:common_metrics).and_return([simple_metric_group, second_metric_group])
       allow(client).to receive(:label_values).and_return('metric_c')
     end
 

spec/lib/gitlab/prometheus_client_spec.rb

@@ -19,41 +19,41 @@ describe Gitlab::PrometheusClient do
   # - execute_query: A query call
   shared_examples 'failure response' do
     context 'when request returns 400 with an error message' do
-      it 'raises a Gitlab::PrometheusError error' do
+      it 'raises a Gitlab::PrometheusClient::Error error' do
         req_stub = stub_prometheus_request(query_url, status: 400, body: { error: 'bar!' })
 
         expect { execute_query }
-          .to raise_error(Gitlab::PrometheusError, 'bar!')
+          .to raise_error(Gitlab::PrometheusClient::Error, 'bar!')
         expect(req_stub).to have_been_requested
       end
     end
 
     context 'when request returns 400 without an error message' do
-      it 'raises a Gitlab::PrometheusError error' do
+      it 'raises a Gitlab::PrometheusClient::Error error' do
         req_stub = stub_prometheus_request(query_url, status: 400)
 
         expect { execute_query }
-          .to raise_error(Gitlab::PrometheusError, 'Bad data received')
+          .to raise_error(Gitlab::PrometheusClient::Error, 'Bad data received')
         expect(req_stub).to have_been_requested
       end
     end
 
     context 'when request returns 500' do
-      it 'raises a Gitlab::PrometheusError error' do
+      it 'raises a Gitlab::PrometheusClient::Error error' do
         req_stub = stub_prometheus_request(query_url, status: 500, body: { message: 'FAIL!' })
 
         expect { execute_query }
-          .to raise_error(Gitlab::PrometheusError, '500 - {"message":"FAIL!"}')
+          .to raise_error(Gitlab::PrometheusClient::Error, '500 - {"message":"FAIL!"}')
         expect(req_stub).to have_been_requested
       end
     end
 
     context 'when request returns non json data' do
-      it 'raises a Gitlab::PrometheusError error' do
+      it 'raises a Gitlab::PrometheusClient::Error error' do
         req_stub = stub_prometheus_request(query_url, status: 200, body: 'not json')
 
         expect { execute_query }
-          .to raise_error(Gitlab::PrometheusError, 'Parsing response failed')
+          .to raise_error(Gitlab::PrometheusClient::Error, 'Parsing response failed')
         expect(req_stub).to have_been_requested
       end
     end
@@ -65,27 +65,27 @@ describe Gitlab::PrometheusClient do
     subject { described_class.new(RestClient::Resource.new(prometheus_url)) }
 
     context 'exceptions are raised' do
-      it 'raises a Gitlab::PrometheusError error when a SocketError is rescued' do
+      it 'raises a Gitlab::PrometheusClient::Error error when a SocketError is rescued' do
         req_stub = stub_prometheus_request_with_exception(prometheus_url, SocketError)
 
         expect { subject.send(:get, '/', {}) }
-          .to raise_error(Gitlab::PrometheusError, "Can't connect to #{prometheus_url}")
+          .to raise_error(Gitlab::PrometheusClient::Error, "Can't connect to #{prometheus_url}")
         expect(req_stub).to have_been_requested
       end
 
-      it 'raises a Gitlab::PrometheusError error when a SSLError is rescued' do
+      it 'raises a Gitlab::PrometheusClient::Error error when a SSLError is rescued' do
         req_stub = stub_prometheus_request_with_exception(prometheus_url, OpenSSL::SSL::SSLError)
 
         expect { subject.send(:get, '/', {}) }
-          .to raise_error(Gitlab::PrometheusError, "#{prometheus_url} contains invalid SSL data")
+          .to raise_error(Gitlab::PrometheusClient::Error, "#{prometheus_url} contains invalid SSL data")
         expect(req_stub).to have_been_requested
       end
 
-      it 'raises a Gitlab::PrometheusError error when a RestClient::Exception is rescued' do
+      it 'raises a Gitlab::PrometheusClient::Error error when a RestClient::Exception is rescued' do
         req_stub = stub_prometheus_request_with_exception(prometheus_url, RestClient::Exception)
 
         expect { subject.send(:get, '/', {}) }
-          .to raise_error(Gitlab::PrometheusError, "Network connection error")
+          .to raise_error(Gitlab::PrometheusClient::Error, "Network connection error")
         expect(req_stub).to have_been_requested
       end
     end

spec/models/clusters/applications/prometheus_spec.rb

@@ -32,11 +32,11 @@ describe Clusters::Applications::Prometheus do
     end
   end
 
-  describe '#proxy_client' do
+  describe '#prometheus_client' do
     context 'cluster is nil' do
       it 'returns nil' do
         expect(subject.cluster).to be_nil
-        expect(subject.proxy_client).to be_nil
+        expect(subject.prometheus_client).to be_nil
       end
     end
 
@@ -45,7 +45,7 @@ describe Clusters::Applications::Prometheus do
       subject { create(:clusters_applications_prometheus, cluster: cluster) }
 
       it 'returns nil' do
-        expect(subject.proxy_client).to be_nil
+        expect(subject.prometheus_client).to be_nil
       end
     end
 
@@ -73,15 +73,15 @@ describe Clusters::Applications::Prometheus do
       end
 
       it 'creates proxy prometheus rest client' do
-        expect(subject.proxy_client).to be_instance_of(RestClient::Resource)
+        expect(subject.prometheus_client).to be_instance_of(RestClient::Resource)
       end
 
       it 'creates proper url' do
-        expect(subject.proxy_client.url).to eq('http://example.com/api/v1/proxy/namespaces/gitlab-managed-apps/service/prometheus-prometheus-server:80')
+        expect(subject.prometheus_client.url).to eq('http://example.com/api/v1/proxy/namespaces/gitlab-managed-apps/service/prometheus-prometheus-server:80')
       end
 
       it 'copies options and headers from kube client to proxy client' do
-        expect(subject.proxy_client.options).to eq(kube_client.rest_client.options.merge(headers: kube_client.headers))
+        expect(subject.prometheus_client.options).to eq(kube_client.rest_client.options.merge(headers: kube_client.headers))
       end
     end
   end

spec/models/deployment_spec.rb

@@ -64,6 +64,7 @@ describe Deployment do
 
   describe '#metrics' do
     let(:deployment) { create(:deployment) }
+    let(:prometheus_adapter) { double('prometheus_adapter', can_query?: true) }
 
     subject { deployment.metrics }
 
@@ -76,17 +77,16 @@ describe Deployment do
         {
           success: true,
           metrics: {},
-          last_update: 42,
-          deployment_time: 1494408956
+          last_update: 42
         }
       end
 
       before do
-        allow(deployment.project).to receive_message_chain(:monitoring_service, :deployment_metrics)
-                                       .with(any_args).and_return(simple_metrics)
+        allow(deployment).to receive(:prometheus_adapter).and_return(prometheus_adapter)
+        allow(prometheus_adapter).to receive(:query).with(:deployment, deployment).and_return(simple_metrics)
       end
 
-      it { is_expected.to eq(simple_metrics) }
+      it { is_expected.to eq(simple_metrics.merge({ deployment_time: deployment.created_at.to_i })) }
     end
   end
 
@@ -109,11 +109,11 @@ describe Deployment do
         }
       end
 
-      let(:prometheus_service) { double('prometheus_service') }
+      let(:prometheus_adapter) { double('prometheus_adapter', can_query?: true) }
 
       before do
-        allow(project).to receive(:prometheus_service).and_return(prometheus_service)
-        allow(prometheus_service).to receive(:additional_deployment_metrics).and_return(simple_metrics)
+        allow(deployment).to receive(:prometheus_adapter).and_return(prometheus_adapter)
+        allow(prometheus_adapter).to receive(:query).with(:additional_metrics_deployment, deployment).and_return(simple_metrics)
       end
 
       it { is_expected.to eq(simple_metrics.merge({ deployment_time: deployment.created_at.to_i })) }

spec/models/environment_spec.rb

@@ -553,8 +553,8 @@ describe Environment do
       end
 
       it 'returns the metrics from the deployment service' do
-        expect(project.monitoring_service)
-          .to receive(:environment_metrics).with(environment)
+        expect(environment.prometheus_adapter)
+          .to receive(:query).with(:environment, environment)
           .and_return(:fake_metrics)
 
         is_expected.to eq(:fake_metrics)
@@ -609,12 +609,12 @@ describe Environment do
 
     context 'when the environment has additional metrics' do
       before do
-        allow(environment).to receive(:has_additional_metrics?).and_return(true)
+        allow(environment).to receive(:has_metrics?).and_return(true)
       end
 
       it 'returns the additional metrics from the deployment service' do
-        expect(project.prometheus_service).to receive(:additional_environment_metrics)
-                                                .with(environment)
+        expect(environment.prometheus_adapter).to receive(:query)
+                                                .with(:additional_metrics_environment, environment)
                                                 .and_return(:fake_metrics)
 
         is_expected.to eq(:fake_metrics)
@@ -623,46 +623,13 @@ describe Environment do
 
     context 'when the environment does not have metrics' do
       before do
-        allow(environment).to receive(:has_additional_metrics?).and_return(false)
+        allow(environment).to receive(:has_metrics?).and_return(false)
       end
 
       it { is_expected.to be_nil }
     end
   end
 
-  describe '#has_additional_metrics??' do
-    subject { environment.has_additional_metrics? }
-
-    context 'when the enviroment is available' do
-      context 'with a deployment service' do
-        let(:project) { create(:prometheus_project) }
-
-        context 'and a deployment' do
-          let!(:deployment) { create(:deployment, environment: environment) }
-          it { is_expected.to be_truthy }
-        end
-
-        context 'but no deployments' do
-          it { is_expected.to be_falsy }
-        end
-      end
-
-      context 'without a monitoring service' do
-        it { is_expected.to be_falsy }
-      end
-    end
-
-    context 'when the environment is unavailable' do
-      let(:project) { create(:prometheus_project) }
-
-      before do
-        environment.stop
-      end
-
-      it { is_expected.to be_falsy }
-    end
-  end
-
   describe '#slug' do
     it "is automatically generated" do
       expect(environment.slug).not_to be_nil
@@ -755,4 +722,77 @@ describe Environment do
       end
     end
   end
+
+  describe '#prometheus_adapter' do
+    let!(:cluster_for_all) { create(:cluster, environment_scope: '*', projects: [project]) }
+    let!(:cluster_for_dev) { create(:cluster, environment_scope: 'dev', projects: [project]) }
+
+    let!(:prometheus_for_dev) { create(:clusters_applications_prometheus, :installed, cluster: cluster_for_dev) }
+
+    context 'prometheus service can execute queries' do
+      let(:prometheus_service) { double(:prometheus_service, can_query?: true) }
+
+      before do
+        allow(environment.project).to receive(:find_or_initialize_service).with('prometheus').and_return prometheus_service
+      end
+
+      it 'return prometheus service as prometheus adapter' do
+        expect(environment.prometheus_adapter).to eq(prometheus_service)
+      end
+    end
+
+    context "prometheus service can't execute queries" do
+      let(:prometheus_service) { double(:prometheus_service, can_query?: false) }
+
+      context 'with cluster for all environments with prometheus installed' do
+        let!(:prometheus_for_all) { create(:clusters_applications_prometheus, :installed, cluster: cluster_for_all) }
+
+        context 'without environment supplied' do
+          it 'returns application handling all environments' do
+            expect(environment.prometheus_adapter).to eq(prometheus_for_all)
+          end
+        end
+
+        context 'with dev environment supplied' do
+          let!(:environment) { create(:environment, project: project, name: 'dev') }
+
+          it 'returns dev cluster prometheus application' do
+            expect(environment.prometheus_adapter).to eq(prometheus_for_dev)
+          end
+        end
+
+        context 'with prod environment supplied' do
+          let!(:environment) { create(:environment, project: project, name: 'prod') }
+
+          it 'returns application handling all environments' do
+            expect(environment.prometheus_adapter).to eq(prometheus_for_all)
+          end
+        end
+      end
+
+      context 'with cluster for all environments without prometheus installed' do
+        context 'without environment supplied' do
+          it 'returns nil' do
+            expect(environment.prometheus_adapter).to be_nil
+          end
+        end
+
+        context 'with dev environment supplied' do
+          let!(:environment) { create(:environment, project: project, name: 'dev') }
+
+          it 'returns dev cluster prometheus application' do
+            expect(environment.prometheus_adapter).to eq(prometheus_for_dev)
+          end
+        end
+
+        context 'with prod environment supplied' do
+          let!(:environment) { create(:environment, project: project, name: 'prod') }
+
+          it 'returns nil' do
+            expect(environment.prometheus_adapter).to be_nil
+          end
+        end
+      end
+    end
+  end
 end

spec/models/project_services/prometheus_service_spec.rb

@@ -63,7 +63,7 @@ describe PrometheusService, :use_clean_rails_memory_store_caching do
     end
 
     context 'with valid data' do
-      subject { service.environment_metrics(environment) }
+      subject { service.query(:environment, environment) }
 
       before do
         stub_reactive_cache(service, prometheus_data, environment_query, environment.id)
@@ -76,14 +76,14 @@ describe PrometheusService, :use_clean_rails_memory_store_caching do
   end
 
   describe '#matched_metrics' do
-    let(:matched_metrics_query) { Gitlab::Prometheus::Queries::MatchedMetricsQuery }
-    let(:client) { double(:client, label_values: nil) }
+    let(:matched_metrics_query) { Gitlab::Prometheus::Queries::MatchedMetricQuery }
+    let(:prometheus_client_wrapper) { double(:prometheus_client_wrapper, label_values: nil) }
 
     context 'with valid data' do
-      subject { service.matched_metrics }
+      subject { service.query(:matched_metrics) }
 
       before do
-        allow(service).to receive(:client).and_return(client)
+        allow(service).to receive(:prometheus_client_wrapper).and_return(prometheus_client_wrapper)
         synchronous_reactive_cache(service)
       end
 
@@ -103,17 +103,14 @@ describe PrometheusService, :use_clean_rails_memory_store_caching do
     end
 
     context 'with valid data' do
-      subject { service.deployment_metrics(deployment) }
-      let(:fake_deployment_time) { 10 }
+      subject { service.query(:deployment, deployment) }
 
       before do
-        stub_reactive_cache(service, prometheus_data, deployment_query, deployment.environment.id, deployment.id)
+        stub_reactive_cache(service, prometheus_data, deployment_query, deployment.id)
       end
 
       it 'returns reactive data' do
-        expect(deployment).to receive(:created_at).and_return(fake_deployment_time)
-
-        expect(subject).to eq(prometheus_metrics_data.merge(deployment_time: fake_deployment_time))
+        expect(subject).to eq(prometheus_metrics_data)
       end
     end
   end
@@ -161,91 +158,31 @@ describe PrometheusService, :use_clean_rails_memory_store_caching do
     end
   end
 
-  describe '#client' do
+  describe '#prometheus_client' do
     context 'manual configuration is enabled' do
       let(:api_url) { 'http://some_url' }
+
       before do
+        subject.active = true
         subject.manual_configuration = true
         subject.api_url = api_url
       end
 
-      it 'returns simple rest client from api_url' do
-        expect(subject.client).to be_instance_of(Gitlab::PrometheusClient)
-        expect(subject.client.rest_client.url).to eq(api_url)
+      it 'returns rest client from api_url' do
+        expect(subject.prometheus_client.url).to eq(api_url)
       end
     end
 
     context 'manual configuration is disabled' do
-      let!(:cluster_for_all) { create(:cluster, environment_scope: '*', projects: [project]) }
-      let!(:cluster_for_dev) { create(:cluster, environment_scope: 'dev', projects: [project]) }
-
-      let!(:prometheus_for_dev) { create(:clusters_applications_prometheus, :installed, cluster: cluster_for_dev) }
-      let(:proxy_client) { double('proxy_client') }
+      let(:api_url) { 'http://some_url' }
 
       before do
-        service.manual_configuration = false
-      end
-
-      context 'with cluster for all environments with prometheus installed' do
-        let!(:prometheus_for_all) { create(:clusters_applications_prometheus, :installed, cluster: cluster_for_all) }
-
-        context 'without environment supplied' do
-          it 'returns client handling all environments' do
-            expect(service).to receive(:client_from_cluster).with(cluster_for_all).and_return(proxy_client).twice
-
-            expect(service.client).to be_instance_of(Gitlab::PrometheusClient)
-            expect(service.client.rest_client).to eq(proxy_client)
-          end
-        end
-
-        context 'with dev environment supplied' do
-          let!(:environment) { create(:environment, project: project, name: 'dev') }
-
-          it 'returns dev cluster client' do
-            expect(service).to receive(:client_from_cluster).with(cluster_for_dev).and_return(proxy_client).twice
-
-            expect(service.client(environment.id)).to be_instance_of(Gitlab::PrometheusClient)
-            expect(service.client(environment.id).rest_client).to eq(proxy_client)
-          end
-        end
-
-        context 'with prod environment supplied' do
-          let!(:environment) { create(:environment, project: project, name: 'prod') }
-
-          it 'returns dev cluster client' do
-            expect(service).to receive(:client_from_cluster).with(cluster_for_all).and_return(proxy_client).twice
-
-            expect(service.client(environment.id)).to be_instance_of(Gitlab::PrometheusClient)
-            expect(service.client(environment.id).rest_client).to eq(proxy_client)
-          end
-        end
-      end
-
-      context 'with cluster for all environments without prometheus installed' do
-        context 'without environment supplied' do
-          it 'raises PrometheusError because cluster was not found' do
-            expect { service.client }.to raise_error(Gitlab::PrometheusError, /couldn't find cluster with Prometheus installed/)
-          end
-        end
-
-        context 'with dev environment supplied' do
-          let!(:environment) { create(:environment, project: project, name: 'dev') }
-
-          it 'returns dev cluster client' do
-            expect(service).to receive(:client_from_cluster).with(cluster_for_dev).and_return(proxy_client).twice
-
-            expect(service.client(environment.id)).to be_instance_of(Gitlab::PrometheusClient)
-            expect(service.client(environment.id).rest_client).to eq(proxy_client)
-          end
+        subject.manual_configuration = false
+        subject.api_url = api_url
       end
 
-        context 'with prod environment supplied' do
-          let!(:environment) { create(:environment, project: project, name: 'prod') }
-
-          it 'raises PrometheusError because cluster was not found' do
-            expect { service.client }.to raise_error(Gitlab::PrometheusError, /couldn't find cluster with Prometheus installed/)
-          end
-        end
+      it 'no client provided' do
+        expect(subject.prometheus_client).to be_nil
       end
     end
   end

spec/support/prometheus/additional_metrics_shared_examples.rb

@@ -12,11 +12,12 @@ RSpec.shared_examples 'additional metrics query' do
 
   let(:client) { double('prometheus_client') }
   let(:query_result) { described_class.new(client).query(*query_params) }
-  let(:environment) { create(:environment, slug: 'environment-slug') }
+  let(:project) { create(:project) }
+  let(:environment) { create(:environment, slug: 'environment-slug', project: project) }
 
   before do
     allow(client).to receive(:label_values).and_return(metric_names)
-    allow(metric_group_class).to receive(:all).and_return([simple_metric_group(metrics: [simple_metric])])
+    allow(metric_group_class).to receive(:common_metrics).and_return([simple_metric_group(metrics: [simple_metric])])
   end
 
   context 'metrics query context' do
@@ -24,13 +25,14 @@ RSpec.shared_examples 'additional metrics query' do
 
     shared_examples 'query context containing environment slug and filter' do
       it 'contains ci_environment_slug' do
-        expect(subject).to receive(:query_metrics).with(hash_including(ci_environment_slug: environment.slug))
+        expect(subject).to receive(:query_metrics).with(project, hash_including(ci_environment_slug: environment.slug))
 
         subject.query(*query_params)
       end
 
       it 'contains environment filter' do
         expect(subject).to receive(:query_metrics).with(
+          project,
           hash_including(
             environment_filter: "container_name!=\"POD\",environment=\"#{environment.slug}\""
           )
@@ -48,7 +50,7 @@ RSpec.shared_examples 'additional metrics query' do
         it_behaves_like 'query context containing environment slug and filter'
 
         it 'query context contains kube_namespace' do
-          expect(subject).to receive(:query_metrics).with(hash_including(kube_namespace: kube_namespace))
+          expect(subject).to receive(:query_metrics).with(project, hash_including(kube_namespace: kube_namespace))
 
           subject.query(*query_params)
         end
@@ -72,7 +74,7 @@ RSpec.shared_examples 'additional metrics query' do
       it_behaves_like 'query context containing environment slug and filter'
 
       it 'query context contains empty kube_namespace' do
-        expect(subject).to receive(:query_metrics).with(hash_including(kube_namespace: ''))
+        expect(subject).to receive(:query_metrics).with(project, hash_including(kube_namespace: ''))
 
         subject.query(*query_params)
       end
@@ -81,7 +83,7 @@ RSpec.shared_examples 'additional metrics query' do
 
   context 'with one group where two metrics is found' do
     before do
-      allow(metric_group_class).to receive(:all).and_return([simple_metric_group])
+      allow(metric_group_class).to receive(:common_metrics).and_return([simple_metric_group])
     end
 
     context 'some queries return results' do
@@ -117,7 +119,7 @@ RSpec.shared_examples 'additional metrics query' do
     let(:metrics) { [simple_metric(queries: [simple_query])] }
 
     before do
-      allow(metric_group_class).to receive(:all).and_return(
+      allow(metric_group_class).to receive(:common_metrics).and_return(
         [
           simple_metric_group(name: 'group_a', metrics: [simple_metric(queries: [simple_query])]),
           simple_metric_group(name: 'group_b', metrics: [simple_metric(title: 'title_b', queries: [simple_query('b')])])