Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat (limited to 'doc/development/service_ping')
-rw-r--r--doc/development/service_ping/implement.md27
-rw-r--r--doc/development/service_ping/index.md6
-rw-r--r--doc/development/service_ping/metrics_dictionary.md6
-rw-r--r--doc/development/service_ping/metrics_instrumentation.md12
-rw-r--r--doc/development/service_ping/performance_indicator_metrics.md5
-rw-r--r--doc/development/service_ping/review_guidelines.md6
-rw-r--r--doc/development/service_ping/usage_data.md2
7 files changed, 39 insertions, 25 deletions
diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 3263ba6458e..0ebc58dd669 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Service Ping consists of two kinds of data:
- **Counters**: Track how often a certain event happened over time, such as how many CI/CD pipelines have run.
- They are monotonic and always trend up.
+ They are monotonic and usually trend up.
- **Observations**: Facts collected from one or more GitLab instances and can carry arbitrary data.
There are no general guidelines for how to collect those, due to the individual nature of that data.
@@ -94,7 +94,7 @@ add_metric('CountUsersAssociatingMilestonesToReleasesMetric', time_frame: 'all')
```
WARNING:
-Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide.
+Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../database/iterating_tables_in_batches.md) guide.
Examples:
@@ -269,9 +269,15 @@ Arguments:
#### Ordinary Redis counters
-Example of implementation:
+Example of implementation: [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb), using Redis methods [`INCR`](https://redis.io/commands/incr) and [`GET`](https://redis.io/commands/get).
-Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb)
+Events are handled by counter classes in the `Gitlab::UsageDataCounters` namespace, inheriting from `BaseCounter`, that are either:
+
+1. Listed in [`Gitlab::UsageDataCounters::COUNTERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters.rb#L5) to be then included in `Gitlab::UsageData`.
+
+1. Specified in the metric definition using the `RedisMetric` instrumentation class as a `counter_class` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation.
+
+Inheriting classes are expected to override `KNOWN_EVENTS` and `PREFIX` constants to build event names and associated metrics. For example, for prefix `issues` and events array `%w[create, update, delete]`, three metrics will be added to the Service Ping payload: `counts.issues_create`, `counts.issues_update` and `counts.issues_delete`.
##### `UsageData` API
@@ -316,7 +322,7 @@ Enabled by default in GitLab 13.7 and later.
#### Redis HLL counters
WARNING:
-HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount), data from
+HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount/), data from
used HLL implementation is "approximated with a standard error of 0.81%".
NOTE:
@@ -324,7 +330,7 @@ NOTE:
With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values.
-Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount).
+Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [PFCOUNT](https://redis.io/commands/pfcount/).
##### Add new events
@@ -371,14 +377,15 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF
- In the controller using the `RedisTracking` module and the following format:
```ruby
- track_redis_hll_event(*controller_actions, name:, if: nil, &block)
+ track_event(*controller_actions, name:, conditions: nil, destinations: [:redis_hll], &block)
```
Arguments:
- `controller_actions`: the controller actions to track.
- `name`: the event name.
- - `if`: optional custom conditions. Uses the same format as Rails callbacks.
+ - `conditions`: optional custom conditions. Uses the same format as Rails callbacks.
+ - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: [:redis_hll].
- `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`.
Example:
@@ -389,7 +396,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF
include RedisTracking
skip_before_action :authenticate_user!, only: :show
- track_redis_hll_event :index, :show, name: 'users_visiting_projects'
+ track_event :index, :show, name: 'users_visiting_projects'
def index
render html: 'index'
@@ -688,7 +695,7 @@ pry(main)> Gitlab::UsageData.count(User.active)
Paste the SQL query into `#database-lab` to see how the query performs at scale.
- GitLab.com's production database has a 15 second timeout.
-- Any single query must stay below the [1 second execution time](../query_performance.md#timing-guidelines-for-queries) with cold caches.
+- Any single query must stay below the [1 second execution time](../database/query_performance.md#timing-guidelines-for-queries) with cold caches.
- Add a specialized index on columns involved to reduce the execution time.
To understand the query's execution, we add the following information
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index cd8af3e9152..4481fe33bda 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -22,9 +22,7 @@ and sales teams understand how GitLab is used. The data helps to:
Service Ping information is not anonymous. It's linked to the instance's hostname, but does
not contain project names, usernames, or any other specific data.
-Sending a Service Ping payload is optional and you can [disable](../../user/admin_area/settings/usage_statistics.md#enable-or-disable-usage-statistics) it on any
-self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances
-and can show your instance's usage statistics to your users.
+Service Ping is enabled by default. However, you can [disable](../../user/admin_area/settings/usage_statistics.md#enable-or-disable-usage-statistics) it on any self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance's usage statistics to your users.
## Service Ping terminology
@@ -113,7 +111,7 @@ sequenceDiagram
1. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics:
- > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag(../../user/feature_flags.md), enabled by default.
+ > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag](../../user/feature_flags.md), enabled by default.
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/295289) in GitLab 15.2. [Feature flag `measure_service_ping_metric_collection`](https://gitlab.com/gitlab-org/gitlab/-/issues/358128) removed.
```ruby
diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md
index 2adba5d8095..d063c4c7601 100644
--- a/doc/development/service_ping/metrics_dictionary.md
+++ b/doc/development/service_ping/metrics_dictionary.md
@@ -205,8 +205,8 @@ instance unique identifier.
key_path: uuid
description: GitLab instance unique identifier
product_category: collection
-product_section: growth
-product_stage: growth
+product_section: analytics
+product_stage: analytics
product_group: product_intelligence
value_type: string
status: active
@@ -301,7 +301,7 @@ bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues users
## Metrics Dictionary
-[Metrics Dictionary is a separate application](https://gitlab.com/gitlab-org/growth/product-intelligence/metric-dictionary).
+[Metrics Dictionary is a separate application](https://gitlab.com/gitlab-org/analytics-section/product-intelligence/metric-dictionary).
All metrics available in Service Ping are in the [Metrics Dictionary](https://metrics.gitlab.com/).
diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md
index e1c51713f3c..9dc37386111 100644
--- a/doc/development/service_ping/metrics_instrumentation.md
+++ b/doc/development/service_ping/metrics_instrumentation.md
@@ -29,7 +29,7 @@ A metric definition has the [`instrumentation_class`](metrics_dictionary.md) fie
The defined instrumentation class should inherit one of the existing metric classes: `DatabaseMetric`, `RedisMetric`, `RedisHLLMetric`, `NumbersMetric` or `GenericMetric`.
-The current convention is that a single instrumentation class corresponds to a single metric. On a rare occasions, there are exceptions to that convention like [Redis metrics](#redis-metrics). To use a single instrumentation class for more than one metric, please reach out to one of the `@gitlab-org/growth/product-intelligence/engineers` members to consult about your case.
+The current convention is that a single instrumentation class corresponds to a single metric. On rare occasions, there are exceptions to that convention like [Redis metrics](#redis-metrics). To use a single instrumentation class for more than one metric, please reach out to one of the `@gitlab-org/analytics-section/product-intelligence/engineers` members to consult about your case.
Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire
process of Service Ping generation.
@@ -38,12 +38,15 @@ We have built a domain-specific language (DSL) to define the metrics instrumenta
## Database metrics
+You can use database metrics to track data kept in the database, for example, a count of issues that exist on a given instance.
+
- `operation`: Operations for the given `relation`, one of `count`, `distinct_count`, `sum`, and `average`.
- `relation`: `ActiveRecord::Relation` for the objects we want to perform the `operation`.
- `start`: Specifies the start value of the batch counting, by default is `relation.minimum(:id)`.
- `finish`: Specifies the end value of the batch counting, by default is `relation.maximum(:id)`.
- `cache_start_and_finish_as`: Specifies the cache key for `start` and `finish` values and sets up caching them. Use this call when `start` and `finish` are expensive queries that should be reused between different metric calculations.
- `available?`: Specifies whether the metric should be reported. The default is `true`.
+- `timestamp_column`: Optionally specifies timestamp column for metric used to filter records for time constrained metrics. The default is `created_at`.
[Example of a merge request that adds a database metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60022).
@@ -149,6 +152,8 @@ end
## Redis metrics
+You can use Redis metrics to track events not kept in the database, for example, a count of how many times the search bar has been used.
+
[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66582).
Count unique values for `source_code_pushes` event.
@@ -199,6 +204,9 @@ options:
```
## Redis HyperLogLog metrics
+
+You can use Redis HyperLogLog metrics to track events not kept in the database and incremented for unique values such as unique users,
+for example, a count of how many different users used the search bar.
[Example of a merge request that adds a `RedisHLL` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61685).
@@ -283,6 +291,8 @@ instrumentation_class: 'IssuesBoardsCountMetric'
## Generic metrics
+You can use generic metrics for other metrics, for example, an instance's database version. Observations type of data will always have a Generic metric counter type.
+
- `value`: Specifies the value of the metric.
- `available?`: Specifies whether the metric should be reported. The default is `true`.
diff --git a/doc/development/service_ping/performance_indicator_metrics.md b/doc/development/service_ping/performance_indicator_metrics.md
index bdd4c319d41..d2abc597a22 100644
--- a/doc/development/service_ping/performance_indicator_metrics.md
+++ b/doc/development/service_ping/performance_indicator_metrics.md
@@ -10,8 +10,7 @@ This guide describes how to use metrics definitions to define [performance indic
To use a metric definition to manage a performance indicator:
-1. Create a new issue and use the [Performance Indicator Metric issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Performance%20Indicator%20Metric).
+1. Create a merge request that includes related changes.
1. Use labels `~"product intelligence"`, `"~Data Warehouse::Impact Check"`.
-1. Create a merge request that includes changes related only to the metric performance indicator.
1. Update the metric definition `performance_indicator_type` [field](metrics_dictionary.md#metrics-definition-and-validation).
-1. Create an issue in GitLab Data Team project with the [Product Performance Indicator template](https://gitlab.com/gitlab-data/analytics/-/issues/new?issuable_template=Product%20Performance%20Indicator%20Template).
+1. Create an issue in GitLab Product Data Insights project with the [PI Chart Help template](https://gitlab.com/gitlab-data/product-analytics/-/issues/new?issuable_template=PI%20Chart%20Help) to have the new metric visualized.
diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md
index 4ce5b2d577c..1b00858be7e 100644
--- a/doc/development/service_ping/review_guidelines.md
+++ b/doc/development/service_ping/review_guidelines.md
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Service Ping review guidelines
This page includes introductory material for a
-[Product Intelligence](https://about.gitlab.com/handbook/engineering/development/growth/product-intelligence/)
+[Product Intelligence](https://about.gitlab.com/handbook/engineering/development/analytics/product-intelligence/)
review, and is specific to Service Ping related reviews. For broader advice and
general best practices for code reviews, refer to our [code review guide](../code_review.md).
@@ -42,7 +42,7 @@ are regular backend changes.
- Assign both the `~backend` and `~product intelligence` reviews to another Product Intelligence team member.
- Assign the maintainer review to someone outside of the Product Intelligence group.
- Assign an
- [engineer](https://gitlab.com/groups/gitlab-org/growth/product-intelligence/engineers/-/group_members?with_inherited_permissions=exclude) from the Product Intelligence team for a review.
+ [engineer](https://gitlab.com/groups/gitlab-org/analytics-section/product-intelligence/engineers/-/group_members?with_inherited_permissions=exclude) from the Product Intelligence team for a review.
- Set the correct attributes in the metric's YAML definition:
- `product_section`, `product_stage`, `product_group`, `product_category`
- Provide a clear description of the metric.
@@ -76,7 +76,7 @@ are regular backend changes.
[Danger bot](../dangerbot.md) adds the list of changed Product Intelligence files
and pings the
-[`@gitlab-org/growth/product-intelligence/engineers`](https://gitlab.com/groups/gitlab-org/growth/product-intelligence/engineers/-/group_members?with_inherited_permissions=exclude) group for merge requests
+[`@gitlab-org/analytics-section/product-intelligence/engineers`](https://gitlab.com/groups/gitlab-org/analytics-section/product-intelligence/engineers/-/group_members?with_inherited_permissions=exclude) group for merge requests
that are not drafts.
Any of the Product Intelligence engineers can be assigned for the Product Intelligence review.
diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md
index a659bbf2265..4181bd90a02 100644
--- a/doc/development/service_ping/usage_data.md
+++ b/doc/development/service_ping/usage_data.md
@@ -59,7 +59,7 @@ Arguments:
- `end`: custom end of the batch counting to avoid complex min calculations
WARNING:
-Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide.
+Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../database/iterating_tables_in_batches.md) guide.
Examples: