Skip to content

G
gitlab-ce
Commit 98be24b5 authored by Kati Paizee's avatar Kati Paizee
Browse files
Options

Merge branch 'am-add-metrics-removal-policy-to-service-ping-guide' into 'master' 

Move Metric life cycle docs in a seprate page

See merge request gitlab-org/gitlab!67968
parents 0b144001 f6fdf64f
Show whitespace changes
Inline Side-by-side
Showing with 179 additions and 172 deletions
doc/development/service_ping/index.md
View file @ 98be24b5
...@@ -220,178 +220,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec ...@@ -220,178 +220,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec
] ]
``` ```
## Service Ping Metric Life cycle
### 1. New metrics addition
Please follow the [Implementing Service Ping](#implementing-service-ping) guide.
### 2. Existing metric change
Because we do not control when customers update their self-managed instances of GitLab,
we **STRONGLY DISCOURAGE** changes to the logic used to calculate any metric.
Any such changes lead to inconsistent reports from multiple GitLab instances.
If there is a problem with an existing metric, it's best to deprecate the existing metric,
and use it, side by side, with the desired new metric.
Example:
Consider following change. Before GitLab 12.6, the `example_metric` was implemented as:
```ruby
{
...
example_metric: distinct_count(Project, :creator_id)
}
```
For GitLab 12.6, the metric was changed to filter out archived projects:
```ruby
{
...
example_metric: distinct_count(Project.non_archived, :creator_id)
}
```
In this scenario all instances running up to GitLab 12.5 continue to report `example_metric`,
including all archived projects, while all instances running GitLab 12.6 and higher filters
out such projects. As Service Ping data is collected from all reporting instances, the
resulting dataset includes mixed data, which distorts any following business analysis.
The correct approach is to add a new metric for GitLab 12.6 release with updated logic:
```ruby
{
...
example_metric_without_archived: distinct_count(Project.non_archived, :creator_id)
}
```
and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric`
### 3. Deprecate a metric
If a metric is obsolete and you no longer use it, you can mark it as deprecated.
For an example of the metric deprecation process take a look at this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59899)
To deprecate a metric:
1. Check the following YAML files and verify the metric is not used in an aggregate:
- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/)
- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/)
1. Create an issue in the [GitLab Data Team
project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for
confirmation that the metric is not used by other teams, or in any of the SiSense
dashboards.
1. Verify the metric is not used to calculate the conversational index. The
conversational index is a measure that reports back to self-managed instances
to inform administrators of the progress of DevOps adoption for the instance.
You can check
[`CalculateConvIndexService`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/calculate_conv_index_service.rb)
to view the metrics that are used. The metrics are represented
as the keys that are passed as a field argument into the `get_value` method.
1. Document the deprecation in the metric's YAML definition. Set
the `status:` attribute to `deprecated`, for example:
```yaml
---
key_path: analytics_unique_visits.analytics_unique_visits_for_any_target_monthly
description: Visits to any of the pages listed above per month
product_section: dev
product_stage: manage
product_group: group::analytics
product_category:
value_type: number
status: deprecated
time_frame: 28d
data_source:
distribution:
- ce
tier:
- free
```
1. Replace the metric's instrumentation with a fixed value. This avoids wasting
resources to calculate the deprecated metric. In
[`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb)
or
[`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb),
replace the code that calculates the metric's value with a fixed value that
indicates it's deprecated:
```ruby
module Gitlab
class UsageData
DEPRECATED_VALUE = -1000
def analytics_unique_visits_data
results['analytics_unique_visits_for_any_target'] = redis_usage_data { unique_visit_service.unique_visits_for(targets: :analytics) }
results['analytics_unique_visits_for_any_target_monthly'] = DEPRECATED_VALUE
{ analytics_unique_visits: results }
end
# ...
end
end
```
### 4. Remove a metric
Only deprecated metrics can be removed from Service Ping.
For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029)
To remove a deprecated metric:
1. Verify that removing the metric from the Service Ping payload does not cause
errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com)
when the updated payload is collected and processed. Version App collects
and persists all Service Ping reports. To do that you can modify
[fixtures](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/spec/support/usage_data_helpers.rb#L540)
used to test
[`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75)
endpoint, and assure that test suite does not fail when metric that you wish to remove is not included into test payload.
1. Create an issue in the
[GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues).
Ask for confirmation that the metric is not referred to in any SiSense dashboards and
can be safely removed from Service Ping. Use this
[example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance.
This step can be skipped if verification done during [deprecation process](#3-deprecate-a-metric)
reported that metric is not required by any data transformation in Snowflake data warehouse nor it is
used by any of SiSense dashboards.
1. After you verify the metric can be safely removed,
update the attributes of the metric's YAML definition:
- Set the `status:` to `removed`.
- Set `milestone_removed:` to the number of the
milestone in which the metric was removed.
Do not remove the metric's YAML definition altogether. Some self-managed
instances might not immediately update to the latest version of GitLab, and
therefore continue to report the removed metric. The Product Intelligence team
requires a record of all removed metrics in order to identify and filter them.
For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#b01f429a54843feb22265100c0e4fec1b7da1240_10_10).
1. After you verify the metric can be safely removed,
remove the metric's instrumentation from
[`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb)
or
[`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb).
For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#6335dc533bd21df26db9de90a02dd66278c2390d_167_167).
1. Remove any other records related to the metric:
- The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags).
- The entry in the known events YAML file at [`lib/gitlab/usage_data_counters/known_events/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage_data_counters/known_events).
## Implementing Service Ping ## Implementing Service Ping
Service Ping consists of two kinds of data, counters and observations. Counters track how often a certain event Service Ping consists of two kinds of data, counters and observations. Counters track how often a certain event
......
doc/development/service_ping/metrics_lifecycle.md 0 → 100644
View file @ 98be24b5
---
stage: Growth
group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Service Ping metric lifecycle
The following guidelines explain the steps to follow at each stage of a metric's lifecycle.
## Add a new metric
Please follow the [Implementing Service Ping](index.md#implementing-service-ping) guide.
## Change an existing metric
Because we do not control when customers update their self-managed instances of GitLab,
we **STRONGLY DISCOURAGE** changes to the logic used to calculate any metric.
Any such changes lead to inconsistent reports from multiple GitLab instances.
If there is a problem with an existing metric, it's best to deprecate the existing metric,
and use it, side by side, with the desired new metric.
Example:
Consider following change. Before GitLab 12.6, the `example_metric` was implemented as:
```ruby
{
...
example_metric: distinct_count(Project, :creator_id)
}
```
For GitLab 12.6, the metric was changed to filter out archived projects:
```ruby
{
...
example_metric: distinct_count(Project.non_archived, :creator_id)
}
```
In this scenario all instances running up to GitLab 12.5 continue to report `example_metric`,
including all archived projects, while all instances running GitLab 12.6 and higher filters
out such projects. As Service Ping data is collected from all reporting instances, the
resulting dataset includes mixed data, which distorts any following business analysis.
The correct approach is to add a new metric for GitLab 12.6 release with updated logic:
```ruby
{
...
example_metric_without_archived: distinct_count(Project.non_archived, :creator_id)
}
```
and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric`
## Deprecate a metric
If a metric is obsolete and you no longer use it, you can mark it as deprecated.
For an example of the metric deprecation process take a look at this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59899)
To deprecate a metric:
1. Check the following YAML files and verify the metric is not used in an aggregate:
- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/)
- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/)
1. Create an issue in the [GitLab Data Team
project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for
confirmation that the metric is not used by other teams, or in any of the SiSense
dashboards.
1. Verify the metric is not used to calculate the conversational index. The
conversational index is a measure that reports back to self-managed instances
to inform administrators of the progress of DevOps adoption for the instance.
You can check
[`CalculateConvIndexService`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/calculate_conv_index_service.rb)
to view the metrics that are used. The metrics are represented
as the keys that are passed as a field argument into the `get_value` method.
1. Document the deprecation in the metric's YAML definition. Set
the `status:` attribute to `deprecated`, for example:
```yaml
---
key_path: analytics_unique_visits.analytics_unique_visits_for_any_target_monthly
description: Visits to any of the pages listed above per month
product_section: dev
product_stage: manage
product_group: group::analytics
product_category:
value_type: number
status: deprecated
time_frame: 28d
data_source:
distribution:
- ce
tier:
- free
```
1. Replace the metric's instrumentation with a fixed value. This avoids wasting
resources to calculate the deprecated metric. In
[`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb)
or
[`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb),
replace the code that calculates the metric's value with a fixed value that
indicates it's deprecated:
```ruby
module Gitlab
class UsageData
DEPRECATED_VALUE = -1000
def analytics_unique_visits_data
results['analytics_unique_visits_for_any_target'] = redis_usage_data { unique_visit_service.unique_visits_for(targets: :analytics) }
results['analytics_unique_visits_for_any_target_monthly'] = DEPRECATED_VALUE
{ analytics_unique_visits: results }
end
# ...
end
end
```
## Remove a metric
Only deprecated metrics can be removed from Service Ping.
For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029)
To remove a deprecated metric:
1. Verify that removing the metric from the Service Ping payload does not cause
errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com)
when the updated payload is collected and processed. Version App collects
and persists all Service Ping reports. To do that you can modify
[fixtures](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/spec/support/usage_data_helpers.rb#L540)
used to test
[`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75)
endpoint, and assure that test suite does not fail when metric that you wish to remove is not included into test payload.
1. Create an issue in the
[GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues).
Ask for confirmation that the metric is not referred to in any SiSense dashboards and
can be safely removed from Service Ping. Use this
[example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance.
This step can be skipped if verification done during [deprecation process](#deprecate-a-metric)
reported that metric is not required by any data transformation in Snowflake data warehouse nor it is
used by any of SiSense dashboards.
1. After you verify the metric can be safely removed,
update the attributes of the metric's YAML definition:
- Set the `status:` to `removed`.
- Set `milestone_removed:` to the number of the
milestone in which the metric was removed.
Do not remove the metric's YAML definition altogether. Some self-managed
instances might not immediately update to the latest version of GitLab, and
therefore continue to report the removed metric. The Product Intelligence team
requires a record of all removed metrics in order to identify and filter them.
For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#b01f429a54843feb22265100c0e4fec1b7da1240_10_10).
1. After you verify the metric can be safely removed,
remove the metric's instrumentation from
[`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb)
or
[`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb).
For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#6335dc533bd21df26db9de90a02dd66278c2390d_167_167).
1. Remove any other records related to the metric:
- The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags).
- The entry in the known events YAML file at [`lib/gitlab/usage_data_counters/known_events/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage_data_counters/known_events).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7