Commit bacab4b8 authored by Russell Dickenson's avatar Russell Dickenson

Merge branch '235697-add-basic-redis-hll-module-update-telemetry-guide' into 'master'

Update Telemetry guide with Basic Redis HLL module

Closes #235697

See merge request gitlab-org/gitlab!39812
parents d873214e 2bf1f0d1
......@@ -222,38 +222,77 @@ Examples of implementation:
#### Redis HLL Counters
With `Gitlab::Redis::HLL` we have available data structures used to count unique values.
With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values.
Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount).
Recommendations:
##### Adding new events
- Key should expire in 29 days.
- If possible, data granularity should be a week. For example a key could be composed from the metric's name and week of the year, `2020-33-{metric_name}`.
- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics.
- If possible, data granularity should be week, for example a key could be composed from metric name and week of the year, 2020-33-{metric_name}
- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics
1. Define events in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml).
Examples of implementation:
Example event:
- [`Gitlab::UsageDataCounters::TrackUniqueEvents`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/track_unique_actions.rb)
- [`Gitlab::Analytics::UniqueVisits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/unique_visits.rb)
```yaml
- name: i_compliance_credential_inventory
category: compliance
redis_slot: compliance
expiry: 42 # 6 weeks
aggregation: weekly
```
Example of usage:
Keys:
- `name`: unique event name.
- `category`: event category. Used for getting total counts for events in a category, for easier
access to a group of events.
- `redis_slot`: optional Redis slot; default value: event name. Used if needed to calculate totals
for a group of metrics. Ensure keys are in the same slot. For example:
`i_compliance_credential_inventory` with `redis_slot: 'compliance'` will build Redis key
`i_{compliance}_credential_inventory-2020-34`. If `redis_slot` is not defined the Redis key will
be `{i_compliance_credential_inventory}-2020-34`.
- `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly
aggregation.
- `aggregation`: aggregation `:daily` or `:weekly`. The argument defines how we build the Redis
keys for data storage. For `daily` we keep a key for metric per day of the year, for `weekly` we
keep a key for metric per week of the year.
1. Track event using `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`.
Arguments:
- `entity_id`: value we count. For example: user_id, visitor_id.
- `event_name`: event name.
1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date)`.
Arguments:
- `event_names`: the list of event names.
- `start_date`: start date of the period for which we want to get event data.
- `end_date`: end date of the period for which we want to get event data.
Recommendations:
- Key should expire in 29 days for daily and 42 days for weekly.
- If possible, data granularity should be a week. For example a key could be composed from the
metric's name and week of the year, `2020-33-{metric_name}`.
- Use a [feature flag](../../operations/feature_flags.md) to have a control over the impact when
adding new metrics.
Example usage:
```ruby
# Redis Counters
redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter)
redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] }
# Redis HLL counter
counter = Gitlab::UsageDataCounters::TrackUniqueEvents
redis_usage_data do
counter.count_unique_events(
event_action: Gitlab::UsageDataCounters::TrackUniqueEvents::PUSH_ACTION,
date_from: time_period[:created_at].first,
date_to: time_period[:created_at].last
)
# Define events in known_events.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml
# Tracking events
Gitlab::UsageDataCounters::HLLRedisCounter.track_event(visitor_id, 'expand_vulnerabilities')
# Get unique events for metric
redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'expand_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) }
```
### Alternative Counters
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment