Commit a6173de9 authored by Douwe Maan's avatar Douwe Maan Committed by Ruben Davila

Merge branch 'custom-events-tracking' into 'master'

Tracking of custom events

## What does this MR do?

This MR adds the ability to track custom events such as the number of Git pushes.

## Are there points in the code the reviewer needs to double check?

The usual stuff.

## Why was this MR needed?

We want to track more business level data such as the number of Git pushes, how many repositories are imported (and from where), etc.

## What are the relevant issue numbers?

gitlab-org/gitlab-ce#13720

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] ~~API support added~~
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

cc @pcarranza 

See merge request !5830
parent 83dfaefc
...@@ -17,6 +17,7 @@ v 8.11.0 (unreleased) ...@@ -17,6 +17,7 @@ v 8.11.0 (unreleased)
- API: List access requests, request access, approve, and deny access requests to a project or a group. !4833 - API: List access requests, request access, approve, and deny access requests to a project or a group. !4833
- Use long options for curl examples in documentation !5703 (winniehell) - Use long options for curl examples in documentation !5703 (winniehell)
- Remove magic comments (`# encoding: UTF-8`) from Ruby files. !5456 (winniehell) - Remove magic comments (`# encoding: UTF-8`) from Ruby files. !5456 (winniehell)
- GitLab Performance Monitoring can now track custom events such as the number of tags pushed to a repository
- Add support for relative links starting with ./ or / to RelativeLinkFilter (winniehell) - Add support for relative links starting with ./ or / to RelativeLinkFilter (winniehell)
- Ignore URLs starting with // in Markdown links !5677 (winniehell) - Ignore URLs starting with // in Markdown links !5677 (winniehell)
- Fix CI status icon link underline (ClemMakesApps) - Fix CI status icon link underline (ClemMakesApps)
......
...@@ -391,6 +391,8 @@ class Repository ...@@ -391,6 +391,8 @@ class Repository
expire_exists_cache expire_exists_cache
expire_root_ref_cache expire_root_ref_cache
expire_emptiness_caches expire_emptiness_caches
repository_event(:create_repository)
end end
# Runs code just before a repository is deleted. # Runs code just before a repository is deleted.
...@@ -407,6 +409,8 @@ class Repository ...@@ -407,6 +409,8 @@ class Repository
expire_root_ref_cache expire_root_ref_cache
expire_emptiness_caches expire_emptiness_caches
expire_exists_cache expire_exists_cache
repository_event(:remove_repository)
end end
# Runs code just before the HEAD of a repository is changed. # Runs code just before the HEAD of a repository is changed.
...@@ -414,6 +418,8 @@ class Repository ...@@ -414,6 +418,8 @@ class Repository
# Cached divergent commit counts are based on repository head # Cached divergent commit counts are based on repository head
expire_branch_cache expire_branch_cache
expire_root_ref_cache expire_root_ref_cache
repository_event(:change_default_branch)
end end
# Runs code before pushing (= creating or removing) a tag. # Runs code before pushing (= creating or removing) a tag.
...@@ -421,12 +427,16 @@ class Repository ...@@ -421,12 +427,16 @@ class Repository
expire_cache expire_cache
expire_tags_cache expire_tags_cache
expire_tag_count_cache expire_tag_count_cache
repository_event(:push_tag)
end end
# Runs code before removing a tag. # Runs code before removing a tag.
def before_remove_tag def before_remove_tag
expire_tags_cache expire_tags_cache
expire_tag_count_cache expire_tag_count_cache
repository_event(:remove_tag)
end end
def before_import def before_import
...@@ -443,6 +453,8 @@ class Repository ...@@ -443,6 +453,8 @@ class Repository
# Runs code after a new commit has been pushed. # Runs code after a new commit has been pushed.
def after_push_commit(branch_name, revision) def after_push_commit(branch_name, revision)
expire_cache(branch_name, revision) expire_cache(branch_name, revision)
repository_event(:push_commit, branch: branch_name)
end end
# Runs code after a new branch has been created. # Runs code after a new branch has been created.
...@@ -450,11 +462,15 @@ class Repository ...@@ -450,11 +462,15 @@ class Repository
expire_branches_cache expire_branches_cache
expire_has_visible_content_cache expire_has_visible_content_cache
expire_branch_count_cache expire_branch_count_cache
repository_event(:push_branch)
end end
# Runs code before removing an existing branch. # Runs code before removing an existing branch.
def before_remove_branch def before_remove_branch
expire_branches_cache expire_branches_cache
repository_event(:remove_branch)
end end
# Runs code after an existing branch has been removed. # Runs code after an existing branch has been removed.
...@@ -1067,4 +1083,8 @@ class Repository ...@@ -1067,4 +1083,8 @@ class Repository
def keep_around_ref_name(sha) def keep_around_ref_name(sha)
"refs/keep-around/#{sha}" "refs/keep-around/#{sha}"
end end
def repository_event(event, tags = {})
Gitlab::Metrics.add_event(event, { path: path_with_namespace }.merge(tags))
end
end end
...@@ -5,6 +5,10 @@ class RepositoryForkWorker ...@@ -5,6 +5,10 @@ class RepositoryForkWorker
sidekiq_options queue: :gitlab_shell sidekiq_options queue: :gitlab_shell
def perform(project_id, forked_from_repository_storage_path, source_path, target_path) def perform(project_id, forked_from_repository_storage_path, source_path, target_path)
Gitlab::Metrics.add_event(:fork_repository,
source_path: source_path,
target_path: target_path)
project = Project.find_by_id(project_id) project = Project.find_by_id(project_id)
unless project.present? unless project.present?
......
...@@ -10,6 +10,10 @@ class RepositoryImportWorker ...@@ -10,6 +10,10 @@ class RepositoryImportWorker
@project = Project.find(project_id) @project = Project.find(project_id)
@current_user = @project.creator @current_user = @project.creator
Gitlab::Metrics.add_event(:import_repository,
import_url: @project.import_url,
path: @project.path_with_namespace)
result = Projects::ImportService.new(project, current_user).execute result = Projects::ImportService.new(project, current_user).execute
if result[:status] == :error if result[:status] == :error
......
...@@ -9,6 +9,7 @@ The following measurements are currently stored in InfluxDB: ...@@ -9,6 +9,7 @@ The following measurements are currently stored in InfluxDB:
- `PROCESS_object_counts` - `PROCESS_object_counts`
- `PROCESS_transactions` - `PROCESS_transactions`
- `PROCESS_views` - `PROCESS_views`
- `events`
Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the
process type. In all series, any form of duration is stored in milliseconds. process type. In all series, any form of duration is stored in milliseconds.
...@@ -78,6 +79,14 @@ following value fields are available: ...@@ -78,6 +79,14 @@ following value fields are available:
The `action` tag contains the action name of the transaction that rendered the The `action` tag contains the action name of the transaction that rendered the
view. view.
## events
This measurement is used to store generic events such as the number of Git
pushes, Emails sent, etc. Each point in this measurement has a single value
field called `count`. The value of this field is simply set to `1`. Each point
also has at least one tag: `event`. This tag's value is set to the event name.
Depending on the event type additional tags may be available as well.
--- ---
Read more on: Read more on:
......
...@@ -20,8 +20,13 @@ module Ci ...@@ -20,8 +20,13 @@ module Ci
build = Ci::RegisterBuildService.new.execute(current_runner) build = Ci::RegisterBuildService.new.execute(current_runner)
if build if build
Gitlab::Metrics.add_event(:build_found,
project: build.project.path_with_namespace)
present build, with: Entities::BuildDetails present build, with: Entities::BuildDetails
else else
Gitlab::Metrics.add_event(:build_not_found)
not_found! not_found!
end end
end end
...@@ -42,6 +47,9 @@ module Ci ...@@ -42,6 +47,9 @@ module Ci
build.update_attributes(trace: params[:trace]) if params[:trace] build.update_attributes(trace: params[:trace]) if params[:trace]
Gitlab::Metrics.add_event(:update_build,
project: build.project.path_with_namespace)
case params[:state].to_s case params[:state].to_s
when 'success' when 'success'
build.success build.success
......
...@@ -124,6 +124,15 @@ module Gitlab ...@@ -124,6 +124,15 @@ module Gitlab
trans.action = action if trans trans.action = action if trans
end end
# Tracks an event.
#
# See `Gitlab::Metrics::Transaction#add_event` for more details.
def self.add_event(*args)
trans = current_transaction
trans.add_event(*args) if trans
end
# Returns the prefix to use for the name of a series. # Returns the prefix to use for the name of a series.
def self.series_prefix def self.series_prefix
@series_prefix ||= Sidekiq.server? ? 'sidekiq_' : 'rails_' @series_prefix ||= Sidekiq.server? ? 'sidekiq_' : 'rails_'
......
...@@ -4,15 +4,20 @@ module Gitlab ...@@ -4,15 +4,20 @@ module Gitlab
class Metric class Metric
JITTER_RANGE = 0.000001..0.001 JITTER_RANGE = 0.000001..0.001
attr_reader :series, :values, :tags attr_reader :series, :values, :tags, :type
# series - The name of the series (as a String) to store the metric in. # series - The name of the series (as a String) to store the metric in.
# values - A Hash containing the values to store. # values - A Hash containing the values to store.
# tags - A Hash containing extra tags to add to the metrics. # tags - A Hash containing extra tags to add to the metrics.
def initialize(series, values, tags = {}) def initialize(series, values, tags = {}, type = :metric)
@values = values @values = values
@series = series @series = series
@tags = tags @tags = tags
@type = type
end
def event?
type == :event
end end
# Returns a Hash in a format that can be directly written to InfluxDB. # Returns a Hash in a format that can be directly written to InfluxDB.
......
...@@ -17,6 +17,10 @@ module Gitlab ...@@ -17,6 +17,10 @@ module Gitlab
begin begin
retval = trans.run { @app.call(env) } retval = trans.run { @app.call(env) }
rescue Exception => error # rubocop: disable Lint/RescueException
trans.add_event(:rails_exception)
raise error
# Even in the event of an error we want to submit any metrics we # Even in the event of an error we want to submit any metrics we
# might've gathered up to this point. # might've gathered up to this point.
ensure ensure
......
...@@ -11,6 +11,10 @@ module Gitlab ...@@ -11,6 +11,10 @@ module Gitlab
# Old gitlad-shell messages don't provide enqueued_at/created_at attributes # Old gitlad-shell messages don't provide enqueued_at/created_at attributes
trans.set(:sidekiq_queue_duration, Time.now.to_f - (message['enqueued_at'] || message['created_at'] || 0)) trans.set(:sidekiq_queue_duration, Time.now.to_f - (message['enqueued_at'] || message['created_at'] || 0))
trans.run { yield } trans.run { yield }
rescue Exception => error # rubocop: disable Lint/RescueException
trans.add_event(:sidekiq_exception)
raise error
ensure ensure
trans.finish trans.finish
end end
......
...@@ -4,7 +4,10 @@ module Gitlab ...@@ -4,7 +4,10 @@ module Gitlab
class Transaction class Transaction
THREAD_KEY = :_gitlab_metrics_transaction THREAD_KEY = :_gitlab_metrics_transaction
attr_reader :tags, :values, :methods # The series to store events (e.g. Git pushes) in.
EVENT_SERIES = 'events'
attr_reader :tags, :values, :method, :metrics
attr_accessor :action attr_accessor :action
...@@ -55,6 +58,20 @@ module Gitlab ...@@ -55,6 +58,20 @@ module Gitlab
@metrics << Metric.new("#{Metrics.series_prefix}#{series}", values, tags) @metrics << Metric.new("#{Metrics.series_prefix}#{series}", values, tags)
end end
# Tracks a business level event
#
# Business level events including events such as Git pushes, Emails being
# sent, etc.
#
# event_name - The name of the event (e.g. "git_push").
# tags - A set of tags to attach to the event.
def add_event(event_name, tags = {})
@metrics << Metric.new(EVENT_SERIES,
{ count: 1 },
{ event: event_name }.merge(tags),
:event)
end
# Returns a MethodCall object for the given name. # Returns a MethodCall object for the given name.
def method_call_for(name) def method_call_for(name)
unless method = @methods[name] unless method = @methods[name]
...@@ -101,7 +118,7 @@ module Gitlab ...@@ -101,7 +118,7 @@ module Gitlab
submit_hashes = submit.map do |metric| submit_hashes = submit.map do |metric|
hash = metric.to_hash hash = metric.to_hash
hash[:tags][:action] ||= @action if @action hash[:tags][:action] ||= @action if @action && !metric.event?
hash hash
end end
......
...@@ -23,6 +23,24 @@ describe Gitlab::Metrics::Metric do ...@@ -23,6 +23,24 @@ describe Gitlab::Metrics::Metric do
it { is_expected.to eq({ host: 'localtoast' }) } it { is_expected.to eq({ host: 'localtoast' }) }
end end
describe '#type' do
subject { metric.type }
it { is_expected.to eq(:metric) }
end
describe '#event?' do
it 'returns false for a regular metric' do
expect(metric.event?).to eq(false)
end
it 'returns true for an event metric' do
expect(metric).to receive(:type).and_return(:event)
expect(metric.event?).to eq(true)
end
end
describe '#to_hash' do describe '#to_hash' do
it 'returns a Hash' do it 'returns a Hash' do
expect(metric.to_hash).to be_an_instance_of(Hash) expect(metric.to_hash).to be_an_instance_of(Hash)
......
...@@ -45,6 +45,15 @@ describe Gitlab::Metrics::RackMiddleware do ...@@ -45,6 +45,15 @@ describe Gitlab::Metrics::RackMiddleware do
middleware.call(env) middleware.call(env)
end end
it 'tracks any raised exceptions' do
expect(app).to receive(:call).with(env).and_raise(RuntimeError)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:add_event).with(:rails_exception)
expect { middleware.call(env) }.to raise_error(RuntimeError)
end
end end
describe '#transaction_from_env' do describe '#transaction_from_env' do
......
...@@ -12,7 +12,9 @@ describe Gitlab::Metrics::SidekiqMiddleware do ...@@ -12,7 +12,9 @@ describe Gitlab::Metrics::SidekiqMiddleware do
with('TestWorker#perform'). with('TestWorker#perform').
and_call_original and_call_original
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).with(:sidekiq_queue_duration, instance_of(Float)) expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).
with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish) expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish)
middleware.call(worker, message, :test) { nil } middleware.call(worker, message, :test) { nil }
...@@ -25,10 +27,28 @@ describe Gitlab::Metrics::SidekiqMiddleware do ...@@ -25,10 +27,28 @@ describe Gitlab::Metrics::SidekiqMiddleware do
with('TestWorker#perform'). with('TestWorker#perform').
and_call_original and_call_original
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).with(:sidekiq_queue_duration, instance_of(Float)) expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).
with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish) expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish)
middleware.call(worker, {}, :test) { nil } middleware.call(worker, {}, :test) { nil }
end end
it 'tracks any raised exceptions' do
worker = double(:worker, class: double(:class, name: 'TestWorker'))
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:run).and_raise(RuntimeError)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:add_event).with(:sidekiq_exception)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:finish)
expect { middleware.call(worker, message, :test) }.
to raise_error(RuntimeError)
end
end end
end end
...@@ -142,5 +142,62 @@ describe Gitlab::Metrics::Transaction do ...@@ -142,5 +142,62 @@ describe Gitlab::Metrics::Transaction do
transaction.submit transaction.submit
end end
it 'does not add an action tag for events' do
transaction.action = 'Foo#bar'
transaction.add_event(:meow)
hash = {
series: 'events',
tags: { event: :meow },
values: { count: 1 },
timestamp: an_instance_of(Fixnum)
}
expect(Gitlab::Metrics).to receive(:submit_metrics).
with([hash])
transaction.submit
end
end
describe '#add_event' do
it 'adds a metric' do
transaction.add_event(:meow)
expect(transaction.metrics[0]).to be_an_instance_of(Gitlab::Metrics::Metric)
end
it "does not prefix the metric's series name" do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.series).to eq(described_class::EVENT_SERIES)
end
it 'tracks a counter for every event' do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.values).to eq(count: 1)
end
it 'tracks the event name' do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.tags).to eq(event: :meow)
end
it 'allows tracking of custom tags' do
transaction.add_event(:meow, animal: 'cat')
metric = transaction.metrics[0]
expect(metric.tags).to eq(event: :meow, animal: 'cat')
end
end end
end end
...@@ -153,4 +153,28 @@ describe Gitlab::Metrics do ...@@ -153,4 +153,28 @@ describe Gitlab::Metrics do
expect(described_class.series_prefix).to be_an_instance_of(String) expect(described_class.series_prefix).to be_an_instance_of(String)
end end
end end
describe '.add_event' do
context 'without a transaction' do
it 'does nothing' do
expect_any_instance_of(Gitlab::Metrics::Transaction).
not_to receive(:add_event)
Gitlab::Metrics.add_event(:meow)
end
end
context 'with a transaction' do
it 'adds an event' do
transaction = Gitlab::Metrics::Transaction.new
expect(transaction).to receive(:add_event).with(:meow)
expect(Gitlab::Metrics).to receive(:current_transaction).
and_return(transaction)
Gitlab::Metrics.add_event(:meow)
end
end
end
end end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment