Commit dcfc4b0e authored by Phil Hughes's avatar Phil Hughes

Merge branch 'rename-usageping-serviceping-docs' into 'master'

Rename usage ping to service ping

See merge request gitlab-org/gitlab!65426
parents ee68dd92 04bab7da
......@@ -22,7 +22,7 @@
- if can_be_configured
%p.mb-2= _('To help improve GitLab and its user experience, GitLab will periodically collect usage information.')
- service_ping_path = help_page_path('user/admin_area/settings/usage_statistics', anchor: 'usage-ping')
- service_ping_path = help_page_path('user/admin_area/settings/usage_statistics', anchor: 'service-ping')
- service_ping_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: service_ping_path }
%p.mb-2= s_('%{service_ping_link_start}Learn more%{service_ping_link_end} about what information is shared with GitLab Inc.').html_safe % { service_ping_link_start: service_ping_link_start, service_ping_link_end: '</a>'.html_safe }
......@@ -32,7 +32,7 @@
%pre.service-data-payload-container.js-syntax-highlight.code.highlight.mt-2.d-none{ class: payload_class, data: { endpoint: usage_data_admin_application_settings_path(format: :html) } }
- else
= _('Service ping is disabled, and cannot be configured through this form.')
- deactivating_service_ping_path = help_page_path('development/usage_ping/index.md', anchor: 'disable-usage-ping')
- deactivating_service_ping_path = help_page_path('development/usage_ping/index.md', anchor: 'disable-service-ping')
- deactivating_service_ping_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: deactivating_service_ping_path }
= s_('For more information, see the documentation on %{deactivating_service_ping_link_start}deactivating service ping%{deactivating_service_ping_link_end}.').html_safe % { deactivating_service_ping_link_start: deactivating_service_ping_link_start, deactivating_service_ping_link_end: '</a>'.html_safe }
.form-group
......
......@@ -44,7 +44,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Adjust your instance's timezone](timezone.md): Customize the default time zone of GitLab.
- [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed.
- [Security](../security/index.md): Learn what you can do to further secure your GitLab instance.
- [Usage statistics, version check, and usage ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc.
- [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc.
- [Global user settings](user_settings.md): Configure instance-wide user permissions.
- [Polling](polling.md): Configure how often the GitLab UI polls for updates.
- [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages.
......
......@@ -1172,17 +1172,17 @@ registry = Geo::SnippetRepositoryRegistry.find(registry_id)
registry.replicator.send(:sync_repository)
```
### Generate usage ping
### Generate Service Ping
#### Generate or get the cached usage ping
#### Generate or get the cached Service Ping
```ruby
Gitlab::UsageData.to_json
```
#### Generate a fresh new usage ping
#### Generate a fresh new Service Ping
This will also refresh the cached usage ping displayed in the admin area
This will also refresh the cached Service Ping displayed in the admin area
```ruby
Gitlab::UsageData.to_json(force_refresh: true)
......@@ -1190,13 +1190,13 @@ Gitlab::UsageData.to_json(force_refresh: true)
#### Generate and print
Generates usage ping data in JSON format.
Generates Service Ping data in JSON format.
```shell
rake gitlab:usage_data:generate
```
#### Generate and send usage ping
#### Generate and send Service Ping
Prints the metrics saved in `conversational_development_index_metrics`.
......
......@@ -7,7 +7,7 @@ type: reference, api
# Usage Data API **(FREE SELF)**
The Usage Data API is associated with [Usage Ping](../development/usage_ping/index.md).
The Usage Data API is associated with [Service Ping](../development/usage_ping/index.md).
## Export metric definitions as a single YAML file
......@@ -48,14 +48,14 @@ Example response:
...
```
## Export Usage Ping SQL queries
## Export Service Ping SQL queries
This action is available only for the GitLab instance [Administrator](../user/permissions.md) users.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57016) in GitLab 13.11.
> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default.
Return all of the raw SQL queries used to compute Usage Ping.
Return all of the raw SQL queries used to compute Service Ping.
```plaintext
GET /usage_data/queries
......@@ -116,7 +116,7 @@ Example response:
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57050) in GitLab 13.11.
> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default.
Return all non-SQL metrics data used in the usage ping.
Return all non-SQL metrics data used in the Service ping.
Example request:
......
......@@ -95,7 +95,7 @@ notify users that we plan to remove them.
- Define a data-informed deprecation policy that will serve our users better
- Build a dashboard showing usage frequency of deprecated GraphQL fields
- Build mechanisms required to send deprecated fields usage in usage ping
- Build mechanisms required to send deprecated fields usage in Service Ping
### Ensure consistency with the rest of the codebase
......
......@@ -271,7 +271,7 @@ See [database guidelines](database/index.md).
## Product Intelligence guides
- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
- [Usage Ping guide](usage_ping/index.md)
- [Service Ping guide](usage_ping/index.md)
- [Snowplow guide](snowplow/index.md)
## Experiment guide
......
......@@ -394,4 +394,4 @@ end
### `EachBatch` vs `BatchCount`
When adding new counters for usage ping, the preferred way to count records is using the `Gitlab::Database::BatchCount` class. The iteration logic implemented in `BatchCount` has similar performance characteristics like `EachBatch`. Most of the tips and suggestions for improving `BatchCount` mentioned above applies to `BatchCount` as well.
When adding new counters for Service Ping, the preferred way to count records is using the `Gitlab::Database::BatchCount` class. The iteration logic implemented in `BatchCount` has similar performance characteristics like `EachBatch`. Most of the tips and suggestions for improving `BatchCount` mentioned above applies to `BatchCount` as well.
......@@ -21,7 +21,7 @@ When you are optimizing your SQL queries, there are two dimensions to pay attent
| Queries in a migration | `100ms` | This is different than the total [migration time](database_review.md#timing-guidelines-for-migrations). |
| Concurrent operations in a migration | `5min` | Concurrent operations do not block the database, but they block the GitLab update. This includes operations such as `add_concurrent_index` and `add_concurrent_foreign_key`. |
| Background migrations | `1s` | |
| Usage Ping | `1s` | See the [usage ping docs](usage_ping/index.md#developing-and-testing-usage-ping) for more details. |
| Service Ping | `1s` | See the [Service Ping docs](usage_ping/index.md#developing-and-testing-service-ping) for more details. |
- When analyzing your query's performance, pay attention to if the time you are seeing is on a [cold or warm cache](#cold-and-warm-cache). These guidelines apply for both cache types.
- When working with batched queries, change the range and batch size to see how it effects the query timing and caching.
......
......@@ -11,7 +11,7 @@ This guide provides an overview of how Snowplow works, and implementation detail
For more information about Product Intelligence, see:
- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
- [Usage Ping Guide](../usage_ping/index.md)
- [Service Ping Guide](../usage_ping/index.md)
More useful links:
......
......@@ -4,11 +4,11 @@ group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Usage Ping Guide **(FREE SELF)**
# Service Ping Guide **(FREE SELF)**
> Introduced in GitLab Ultimate 11.2, more statistics.
This guide describes Usage Ping's purpose and how it's implemented.
This guide describes Service Ping's purpose and how it's implemented.
For more information about Product Intelligence, see:
......@@ -22,9 +22,9 @@ More links:
- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/)
- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/)
## What is Usage Ping?
## What is Service Ping?
Usage Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc.
Service Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc.
The payload provides important high-level data that helps our product, support,
and sales teams understand how GitLab is used. For example, the data helps to:
......@@ -34,39 +34,39 @@ and sales teams understand how GitLab is used. For example, the data helps to:
- Calculate our Stage Monthly Active Users (SMAU), which helps to measure the success of our stages
and features.
Usage Ping information is not anonymous. It's linked to the instance's hostname. However, it does
Service Ping information is not anonymous. It's linked to the instance's hostname. However, it does
not contain project names, usernames, or any other specific data.
Sending a Usage Ping payload is optional and can be [disabled](#disable-usage-ping) on any self-managed instance.
When Usage Ping is enabled, GitLab gathers data from the other instances
Sending a Service Ping payload is optional and can be [disabled](#disable-service-ping) on any self-managed instance.
When Service Ping is enabled, GitLab gathers data from the other instances
and can show your instance's usage statistics to your users.
### Terminology
We use the following terminology to describe the Usage Ping components:
We use the following terminology to describe the Service Ping components:
- **Usage Ping**: the process that collects and generates a JSON payload.
- **Usage data**: the contents of the Usage Ping JSON payload. This includes metrics.
- **Service Ping**: the process that collects and generates a JSON payload.
- **Usage data**: the contents of the Service Ping JSON payload. This includes metrics.
- **Metrics**: primarily made up of row counts for different tables in an instance's database. Each
metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation)
in a YAML file.
### Why should we enable Usage Ping?
### Why should we enable Service Ping?
- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions.
- As a benefit of having the usage ping active, GitLab lets you analyze the users' activities over time of your GitLab installation.
- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring.
- The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions.
- As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation.
- As a benefit of having Service Ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring.
- You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value)
- You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization?
- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes.
- Usage Ping is enabled by default. To disable it, see [Disable Usage Ping](#disable-usage-ping).
- When Usage Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features.
- Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping).
- When Service Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features.
#### Registration Features Program
> Introduced in GitLab 14.1.
Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Usage Ping](#what-is-usage-ping).
Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping).
The paid feature available in this offering is [Email from GitLab](../../tools/email.md).
Administrators can use this [Premium](https://about.gitlab.com/pricing/premium/) feature to streamline
......@@ -77,10 +77,10 @@ Registration is not yet required for participation, but will be added in a futur
### Limitations
- Usage Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events.
- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed.
- Service Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events.
- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Service Ping to track aggregated backend events on self-managed.
## View the Usage Ping payload **(FREE SELF)**
## View the Service Ping payload **(FREE SELF)**
You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload:
......@@ -90,32 +90,32 @@ You can view the exact JSON payload sent to GitLab Inc. in the administration pa
1. Expand the **Usage statistics** section.
1. Select **Preview payload**.
For an example payload, see [Example Usage Ping payload](#example-usage-ping-payload).
For an example payload, see [Example Service Ping payload](#example-service-ping-payload).
## Disable Usage Ping **(FREE SELF)**
## Disable Service Ping **(FREE SELF)**
NOTE:
The method to disable Usage Ping in the GitLab configuration file does not work in
GitLab versions 9.3 to 13.12.3. See the [troubleshooting section](#cannot-disable-usage-ping-using-the-configuration-file)
The method to disable Service Ping in the GitLab configuration file does not work in
GitLab versions 9.3 to 13.12.3. See the [troubleshooting section](#cannot-disable-service-ping-using-the-configuration-file)
on how to disable it.
You can disable the usage ping either using the GitLab UI, or editing the GitLab
You can disable Service Ping either using the GitLab UI, or editing the GitLab
configuration file.
### Disable Usage Ping using the UI
### Disable Service Ping using the UI
To disable Usage Ping in the GitLab UI:
To disable Service Ping in the GitLab UI:
1. Sign in as a user with [Administrator](../../user/permissions.md) permissions.
1. On the top bar, select **Menu >** **{admin}** **Admin**.
1. On the left sidebar, select **Settings > Metrics and profiling**.
1. Expand the **Usage statistics** section.
1. Clear the **Enable usage ping** checkbox.
1. Clear the **Enable service ping** checkbox.
1. Select **Save changes**.
### Disable Usage Ping using the configuration file
### Disable Service Ping using the configuration file
To disable Usage Ping and prevent it from being configured in the future through
To disable Service Ping and prevent it from being configured in the future through
the admin area:
**For installations using the Linux package:**
......@@ -150,7 +150,7 @@ the admin area:
sudo service gitlab restart
```
## Usage Ping request flow
## Service Ping request flow
The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, the GitLab S3 Bucket, the GitLab Snowflake Data Warehouse, and Sisense:
......@@ -163,7 +163,7 @@ sequenceDiagram
participant S3 Bucket
participant Snowflake DW
participant Sisense Dashboards
GitLab Instance->>Versions Application: Send Usage Ping
GitLab Instance->>Versions Application: Send Service Ping
loop Process usage data
Versions Application->>Versions Application: Parse usage data
Versions Application->>Versions Application: Write to database
......@@ -183,9 +183,9 @@ sequenceDiagram
Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index)
```
## How Usage Ping works
## How Service Ping works
1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly.
1. The Service Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly.
1. When the cron job runs, it calls [`Gitlab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22).
1. `Gitlab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls.
1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in `Gitlab::UsageData.to_json`.
......@@ -194,11 +194,11 @@ sequenceDiagram
the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`,
the required URL is <https://version.gitlab.com/>.
## Usage Ping Metric Life cycle
## Service Ping Metric Life cycle
### 1. New metrics addition
Please follow the [Implementing Usage Ping](#implementing-usage-ping) guide.
Please follow the [Implementing Service Ping](#implementing-service-ping) guide.
### 2. Existing metric change
......@@ -229,7 +229,7 @@ For GitLab 12.6, the metric was changed to filter out archived projects:
In this scenario all instances running up to GitLab 12.5 continue to report `example_metric`,
including all archived projects, while all instances running GitLab 12.6 and higher filters
out such projects. As Usage Ping data is collected from all reporting instances, the
out such projects. As Service Ping data is collected from all reporting instances, the
resulting dataset includes mixed data, which distorts any following business analysis.
The correct approach is to add a new metric for GitLab 12.6 release with updated logic:
......@@ -318,16 +318,16 @@ To deprecate a metric:
### 4. Remove a metric
Only deprecated metrics can be removed from Usage Ping.
Only deprecated metrics can be removed from Service Ping.
For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029)
To remove a deprecated metric:
1. Verify that removing the metric from the Usage Ping payload does not cause
1. Verify that removing the metric from the Service Ping payload does not cause
errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com)
when the updated payload is collected and processed. Version App collects
and persists all Usage Ping reports. To do that you can modify
and persists all Service Ping reports. To do that you can modify
[fixtures](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/spec/support/usage_data_helpers.rb#L540)
used to test
[`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75)
......@@ -336,7 +336,7 @@ To remove a deprecated metric:
1. Create an issue in the
[GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues).
Ask for confirmation that the metric is not referred to in any SiSense dashboards and
can be safely removed from Usage Ping. Use this
can be safely removed from Service Ping. Use this
[example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance.
This step can be skipped if verification done during [deprecation process](#3-deprecate-a-metric)
reported that metric is not required by any data transformation in Snowflake data warehouse nor it is
......@@ -370,9 +370,9 @@ To remove a deprecated metric:
1. Update the Metrics Dictionary following [guidelines instructions](dictionary.md).
## Implementing Usage Ping
## Implementing Service Ping
Usage Ping consists of two kinds of data, counters and observations. Counters track how often a certain event
Service Ping consists of two kinds of data, counters and observations. Counters track how often a certain event
happened over time, such as how many CI pipelines have run. They are monotonic and always trend up.
Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no
general guidelines around how to collect those, due to the individual nature of that data.
......@@ -386,7 +386,7 @@ There are several types of counters which are all found in `usage_data.rb`:
- **Redis Counters:** Used for in-memory counts.
NOTE:
Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping.
Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Service Ping.
### Why batch counting
......@@ -961,7 +961,7 @@ alt_usage_data(999)
### Adding counters to build new metrics
When adding the results of two counters, use the `add` usage data method that
handles fallback values and exceptions. It also generates a valid [SQL export](#exporting-usage-ping-sql-queries-and-definitions).
handles fallback values and exceptions. It also generates a valid [SQL export](#exporting-service-ping-sql-queries-and-definitions).
Example usage:
......@@ -971,12 +971,12 @@ add(User.active, User.bot)
### Prometheus Queries
In those cases where operational metrics should be part of Usage Ping, a database or Redis query is unlikely
In those cases where operational metrics should be part of Service Ping, a database or Redis query is unlikely
to provide useful data. Instead, Prometheus might be more appropriate, because most GitLab architectural
components publish metrics to it that can be queried back, aggregated, and included as usage data.
NOTE:
Prometheus as a data source for Usage Ping is currently only available for single-node Omnibus installations
Prometheus as a data source for Service Ping is currently only available for single-node Omnibus installations
that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance.
To query Prometheus for metrics, a helper method is available to `yield` a fully configured
......@@ -1002,7 +1002,7 @@ We return fallback values in these cases:
| Timeouts, general failures | -1 |
| Standard errors in counters | -2 |
## Developing and testing Usage Ping
## Developing and testing Service Ping
### 1. Naming and placing the metrics
......@@ -1078,7 +1078,7 @@ Check if new metrics need to be added to the Versions Application. See `usage_da
### 7. Add the feature label
Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature.
Add the `feature` label to the Merge Request for new Service Ping metrics. These are user-facing changes and are part of expanding the Service Ping feature.
### 8. Add a changelog
......@@ -1090,21 +1090,21 @@ On GitLab.com, we have DangerBot set up to monitor Product Intelligence related
### 10. Verify your metric
On GitLab.com, the Product Intelligence team regularly [monitors Usage Ping](https://gitlab.com/groups/gitlab-org/-/epics/6000).
On GitLab.com, the Product Intelligence team regularly [monitors Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/6000).
They may alert you that your metrics need further optimization to run quicker and with greater success.
The Usage Ping JSON payload for GitLab.com is shared in the
The Service Ping JSON payload for GitLab.com is shared in the
[#g_product_intelligence](https://gitlab.slack.com/archives/CL3A7GFPF) Slack channel every week.
You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "SaaS" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric.
You may also use the [Service Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "SaaS" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric.
### Usage Ping local setup
### Service Ping local setup
To set up Usage Ping locally, you must:
To set up Service Ping locally, you must:
1. [Set up local repositories](#set-up-local-repositories).
1. [Test local setup](#test-local-setup).
1. (Optional) [Test Prometheus-based usage ping](#test-prometheus-based-usage-ping).
1. (Optional) [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping).
#### Set up local repositories
......@@ -1117,20 +1117,20 @@ To set up Usage Ping locally, you must:
#### Test local setup
1. Using the `gitlab` Rails console, manually trigger a usage ping:
1. Using the `gitlab` Rails console, manually trigger Service Ping:
```ruby
SubmitUsagePingService.new.execute
```
1. Use the `versions` Rails console to check the usage ping was successfully received,
1. Use the `versions` Rails console to check the Service Ping was successfully received,
parsed, and stored in the Versions database:
```ruby
UsageData.last
```
### Test Prometheus-based usage ping
### Test Prometheus-based Service Ping
If the data submitted includes metrics [queried from Prometheus](#prometheus-queries)
you want to inspect and verify, you must:
......@@ -1139,9 +1139,9 @@ you want to inspect and verify, you must:
- Ensure the respective GitLab components are exporting metrics to the Prometheus server.
If you do not need to test data coming from Prometheus, no further action
is necessary. Usage Ping should degrade gracefully in the absence of a running Prometheus server.
is necessary. Service Ping should degrade gracefully in the absence of a running Prometheus server.
Three kinds of components may export data to Prometheus, and are included in Usage Ping:
Three kinds of components may export data to Prometheus, and are included in Service Ping:
- [`node_exporter`](https://github.com/prometheus/node_exporter): Exports node metrics
from the host machine.
......@@ -1151,7 +1151,7 @@ Three kinds of components may export data to Prometheus, and are included in Usa
#### Test with an Omnibus container
This is the recommended approach to test Prometheus based Usage Ping.
This is the recommended approach to test Prometheus based Service Ping.
The easiest way to verify your changes is to build a new Omnibus image from your code branch by using CI, then download the image
and run a local container instance:
......@@ -1172,15 +1172,15 @@ This is the less recommended approach, because it comes with a number of difficu
The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would
like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start.
The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Usage Ping.
The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Service Ping.
By default, it already comes with a fully configured Prometheus service that is set up to scrape a number of components,
but with the following limitations:
- It does not run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing.
- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it would normally report itself to not be associated
with any of the other services that are running. That is not how node metrics are reported in a production setup, where `node_exporter`
always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore
appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Usage Ping.
always runs as a process alongside other GitLab components on any given node. From Service Ping's perspective none of the node data would therefore
appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping.
## Aggregated metrics
......@@ -1189,11 +1189,11 @@ appear to be associated to any of the services running, because they all appear
WARNING:
This feature is intended solely for internal GitLab use.
To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available at Community Edition and at [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for Enterprise Edition ones.
To add data for aggregated metrics into Service Ping payload you should add corresponding definition at [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available at Community Edition and at [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for Enterprise Edition ones.
Each aggregate definition includes following parts:
- `name`: Unique name under which the aggregate metric is added to the Usage Ping payload.
- `name`: Unique name under which the aggregate metric is added to the Service Ping payload.
- `operator`: Operator that defines how the aggregated metric data is counted. Available operators are:
- `OR`: Removes duplicates and counts all entries that triggered any of listed events.
- `AND`: Removes duplicates and counts all elements that were observed triggering all of following events.
......@@ -1238,7 +1238,7 @@ Example aggregated metric entries:
feature_flag: example_aggregated_metric
```
Aggregated metrics collected in `7d` and `28d` time frames are added into Usage Ping payload under the `aggregated_metrics` sub-key in the `counts_weekly` and `counts_monthly` top level keys.
Aggregated metrics collected in `7d` and `28d` time frames are added into Service Ping payload under the `aggregated_metrics` sub-key in the `counts_weekly` and `counts_monthly` top level keys.
```ruby
{
......@@ -1320,9 +1320,9 @@ The `Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_me
method accepts the following arguments:
- `metric_name`: The name of metric to use for aggregations. Should be the same
as the key under which the metric is added into Usage Ping.
as the key under which the metric is added into Service Ping.
- `recorded_at_timestamp`: The timestamp representing the moment when a given
Usage Ping payload was collected. You should use the convenience method `recorded_at`
Service Ping payload was collected. You should use the convenience method `recorded_at`
to fill `recorded_at_timestamp` argument, like this: `recorded_at_timestamp: recorded_at`
- `time_period`: The time period used to build the `relation` argument passed into
`estimate_batch_distinct_count`. To collect the metric with all available historical
......@@ -1373,9 +1373,9 @@ Example definition:
- all
```
## Example Usage Ping payload
## Example Service Ping payload
The following is example content of the Usage Ping payload.
The following is example content of the Service Ping payload.
```json
{
......@@ -1578,9 +1578,9 @@ The following is example content of the Usage Ping payload.
In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/).
## Exporting Usage Ping SQL queries and definitions
## Exporting Service Ping SQL queries and definitions
Two Rake tasks exist to export Usage Ping definitions.
Two Rake tasks exist to export Service Ping definitions.
- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`.
- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`.
......@@ -1599,7 +1599,7 @@ bin/rake gitlab:usage_data:dump_sql_in_json
bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml
```
## Generating and troubleshooting usage ping
## Generating and troubleshooting Service Ping
This activity is to be done via a detached screen session on a remote server.
......@@ -1625,15 +1625,15 @@ with the `ssh-add` command.
## Troubleshooting
### Cannot disable Usage Ping using the configuration file
### Cannot disable Service Ping using the configuration file
The method to disable Usage Ping using the GitLab configuration file does not work in
The method to disable Service Ping using the GitLab configuration file does not work in
GitLab versions 9.3.0 to 13.12.3. To disable it, you need to use the Admin Area in
the GitLab UI instead. For more information, see
[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269).
GitLab functionality and application settings cannot override or circumvent
restrictions at the network layer. If usage ping is blocked by your firewall,
restrictions at the network layer. If Service Ping is blocked by your firewall,
you are not impacted by this bug.
#### Check if you are affected
......@@ -1646,16 +1646,16 @@ checking the configuration file of your GitLab instance:
1. On the top bar, go to the admin area (**{admin}**).
1. On the left sidebar, select **Settings > Metrics and profiling**.
1. Expand **Usage Statistics**.
1. Are you able to check/uncheck the checkbox to disable usage ping?
1. Are you able to check/uncheck the checkbox to disable Service Ping?
- If _yes_, your GitLab instance is not affected by this bug.
- If you can't check/uncheck the checkbox, you are affected by this bug.
Read below [how to fix this](#how-to-fix-the-cannot-disable-usage-ping-bug).
Read below [how to fix this](#how-to-fix-the-cannot-disable-service-ping-bug).
- Checking your GitLab instance configuration file:
To check whether you're impacted by this bug, check your instance configuration
settings. The configuration file in which Usage Ping can be disabled will depend
settings. The configuration file in which Service Ping can be disabled will depend
on your installation and deployment method, but it will typically be one of the following:
- `/etc/gitlab/gitlab.rb` for Omnibus GitLab Linux Package and Docker.
......@@ -1663,7 +1663,7 @@ checking the configuration file of your GitLab instance:
- `gitlab.yml` for GitLab installations from source.
To check the relevant configuration file for strings that indicate whether
usage ping is disabled, you can use `grep`:
Service Ping is disabled, you can use `grep`:
```shell
# Linux package
......@@ -1679,13 +1679,13 @@ checking the configuration file of your GitLab instance:
If you see any output after running the relevant command, your GitLab instance
may be affected by the bug. Otherwise, your instance is not affected.
#### How to fix the "Cannot disable Usage Ping" bug
#### How to fix the "Cannot disable Service Ping" bug
To work around this bug, you have two options:
- [Update](../../update/index.md) to GitLab 13.12.4 or newer to fix this bug.
- If you can't update to GitLab 13.12.4 or newer, enable usage ping in the
configuration file, then disable usage ping in the UI. For example, if you're
- If you can't update to GitLab 13.12.4 or newer, enable Service Ping in the
configuration file, then disable Service Ping in the UI. For example, if you're
using the Linux package:
1. Edit `/etc/gitlab/gitlab.rb`:
......@@ -1703,5 +1703,5 @@ To work around this bug, you have two options:
1. In GitLab, on the top bar, go to the admin area (**{admin}**).
1. On the left sidebar, select **Settings > Metrics and profiling**.
1. Expand **Usage Statistics**.
1. Clear the **Enable usage ping** checkbox.
1. Clear the **Enable service ping** checkbox.
1. Select **Save Changes**.
......@@ -12,7 +12,7 @@ This guide describes Metrics Dictionary and how it's implemented
We are using [JSON Schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json) to validate the metrics definition.
This process is meant to ensure consistent and valid metrics defined for Usage Ping. All metrics *must*:
This process is meant to ensure consistent and valid metrics defined for Service Ping. All metrics *must*:
- Comply with the defined [JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json).
- Have a unique `key_path` .
......@@ -26,7 +26,7 @@ Each metric is defined in a separate YAML file consisting of a number of fields:
| Field | Required | Additional information |
|---------------------|----------|----------------------------------------------------------------|
| `key_path` | yes | JSON key path for the metric, location in Usage Ping payload. |
| `key_path` | yes | JSON key path for the metric, location in Service Ping payload. |
| `name` | no | Metric name suggestion. Can replace the last part of `key_path`. |
| `description` | yes | |
| `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). |
......@@ -58,7 +58,7 @@ Metric definitions can have one of the following statuses:
- `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute.
- `not_used`: Metric is not used in any dashboard.
- `deprecated`: Metric is deprecated and possibly planned to be removed.
- `removed`: Metric was removed, but it may appear in Usage Ping payloads sent from instances running on older versions of GitLab.
- `removed`: Metric was removed, but it may appear in Service Ping payloads sent from instances running on older versions of GitLab.
### Metric value_type
......@@ -99,7 +99,7 @@ should be changed.
We use the following categories to classify a metric:
- `Operational`: Required data for operational purposes.
- `Optional`: Data that is optional to collect. This can be [enabled or disabled](../usage_ping/index.md#disable-usage-ping) in the Admin Area.
- `Optional`: Data that is optional to collect. This can be [enabled or disabled](../usage_ping/index.md#disable-service-ping) in the Admin Area.
- `Subscription`: Data related to licensing.
- `Standard`: Standard set of identifiers that are included when collecting data.
......@@ -214,9 +214,9 @@ bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --d
create ee/config/metrics/counts_7d/issues.yml
```
## Metrics added dynamic to Usage Ping payload
## Metrics added dynamic to Service Ping payload
The [Redis HLL metrics](index.md#known-events-are-added-automatically-in-usage-data-payload) are added automatically to Usage Ping payload.
The [Redis HLL metrics](index.md#known-events-are-added-automatically-in-usage-data-payload) are added automatically to Service Ping payload.
A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events.
......
......@@ -6,13 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Metrics instrumentation guide
This guide describes how to develop Usage Ping metrics using metrics instrumentation.
This guide describes how to develop Service Ping metrics using metrics instrumentation.
## Nomenclature
- **Instrumentation class**:
- Inherits one of the metric classes: `DatabaseMetric`, `RedisHLLMetric` or `GenericMetric`.
- Implements the logic that calculates the value for a Usage Ping metric.
- Implements the logic that calculates the value for a Service Ping metric.
- **Metric definition**
The Usage Data metric YAML definition.
......@@ -27,7 +27,7 @@ A metric definition has the [`instrumentation_class`](metrics_dictionary.md) fie
The defined instrumentation class should have one of the existing metric classes: `DatabaseMetric`, `RedisHLLMetric`, or `GenericMetric`.
Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire
process of Usage Ping generation.
process of Service Ping generation.
We have built a domain-specific language (DSL) to define the metrics instrumentation.
......@@ -88,7 +88,7 @@ end
## Creating a new metric instrumentation class
To create a stub instrumentation for a Usage Ping metric, you can use a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_generator.rb):
To create a stub instrumentation for a Service Ping metric, you can use a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_generator.rb):
The generator takes the class name as an argument and the following options:
......
......@@ -4,22 +4,22 @@ group: Product Intelligence
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Usage Ping review guidelines
# Service Ping review guidelines
This page includes introductory material for a
[Product Intelligence](https://about.gitlab.com/handbook/engineering/development/growth/product-intelligence/)
review, and is specific to Usage Ping related reviews. For broader advice and
review, and is specific to Service Ping related reviews. For broader advice and
general best practices for code reviews, refer to our [code review guide](../code_review.md).
## Resources for reviewers
- [Usage Ping Guide](index.md)
- [Service Ping Guide](index.md)
- [Metrics Dictionary](metrics_dictionary.md)
## Review process
We recommend a Product Intelligence review when a merge request (MR) touches
any of the following Usage Ping files:
any of the following Service Ping files:
- `usage_data*` files.
- The Metrics Dictionary, including files in:
......@@ -50,7 +50,7 @@ any of the following Usage Ping files:
- Perform a first-pass review on the merge request and suggest improvements to the author.
- Check the [metrics location](index.md#1-naming-and-placing-the-metrics) in
the Usage Ping JSON payload.
the Service Ping JSON payload.
- Add the `~database` label and ask for a [database review](../database_review.md) for
metrics that are based on Database.
- For tracking using Redis HLL (HyperLogLog):
......
......@@ -46,7 +46,7 @@ The following Rake tasks are available for use with GitLab:
| [Repository storage](../administration/raketasks/storage.md) | List and migrate existing projects and attachments from legacy storage to hashed storage. |
| [Uploads migrate](../administration/raketasks/uploads/migrate.md) | Migrate uploads between local storage and object storage. |
| [Uploads sanitize](../administration/raketasks/uploads/sanitize.md) | Remove EXIF data from images uploaded to earlier versions of GitLab. |
| [Usage data](../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping) | Generate and troubleshoot [Usage Ping](../development/usage_ping/index.md). |
| [Usage data](../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-service-ping) | Generate and troubleshoot [Service Ping](../development/usage_ping/index.md). |
| [User management](user_management.md) | Perform user management tasks. |
| [Webhooks administration](web_hooks.md) | Maintain project webhooks. |
| [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, which can be useful if the certificate store changed. |
......@@ -20,19 +20,19 @@ To see DevOps Report:
## DevOps Score
NOTE:
Your GitLab instance's [usage ping](../settings/usage_statistics.md#usage-ping) must be activated in order to use this feature.
Your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping) must be activated in order to use this feature.
The DevOps Score tab displays the usage of major GitLab features on your instance over
the last 30 days, averaged over the number of billable users in that time period. It also
provides a Lead score per feature, which is calculated based on GitLab analysis
of top-performing instances based on [usage ping data](../settings/usage_statistics.md#usage-ping) that GitLab has
of top-performing instances based on [Service Ping data](../settings/usage_statistics.md#service-ping) that GitLab has
collected. Your score is compared to the lead score of each feature and then expressed as a percentage at the bottom of said feature.
Your overall **DevOps Score** is an average of your feature scores. You can use this score to compare your DevOps status to other organizations.
The page also provides helpful links to articles and GitLab docs, to help you
improve your scores.
Usage ping data is aggregated on GitLab servers for analysis. Your usage
Service Ping data is aggregated on GitLab servers for analysis. Your usage
information is **not sent** to any other GitLab instances. If you have just started using GitLab, it may take a few weeks for data to be
collected before this feature is available.
......
......@@ -14,7 +14,7 @@ All statistics are opt-out. To enable or disable them:
1. On the top bar, select **Menu >** **{admin}** **Admin**.
1. In the left sidebar, select **Settings > Metrics and profiling**, and expand **Usage statistics**.
1. Enable or disable **Version check** and **Usage ping**.
1. Enable or disable **Version check** and **Service ping**.
1. Select **Save changes**.
## Network configuration
......@@ -67,14 +67,14 @@ sequenceDiagram
Version Application->>GitLab instance: Response (PNG/SVG)
```
## Usage Ping **(FREE SELF)**
## Service Ping **(FREE SELF)**
See [Usage Ping guide](../../../development/usage_ping/index.md).
See [Service Ping guide](../../../development/usage_ping/index.md).
## Instance-level analytics availability
After usage ping is enabled, GitLab gathers data from other instances and
enables certain [instance-level analytics features](../analytics/index.md) that are dependent on usage ping.
After Service Ping is enabled, GitLab gathers data from other instances and
enables certain [instance-level analytics features](../analytics/index.md) that are dependent on Service Ping.
<!-- ## Troubleshooting
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment