Add latest changes from gitlab-org/gitlab@14-9-stable-eev14.9.0-rc42

6 files changed, 191 insertions, 91 deletions

diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 3a1e4c6d87b..25e841e113b 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -16,7 +16,7 @@ Service Ping consists of two kinds of data:
 To implement a new metric in Service Ping, follow these steps:
 
 1. [Implement the required counter](#types-of-counters)
-1. [Name and place the metric](#name-and-place-the-metric)
+1. [Name and place the metric](metrics_dictionary.md#metric-key_path)
 1. [Test counters manually using your Rails console](#test-counters-manually-using-your-rails-console)
 1. [Generate the SQL query](#generate-the-sql-query)
 1. [Optimize queries with `#database-lab`](#optimize-queries-with-database-lab)
@@ -26,13 +26,12 @@ To implement a new metric in Service Ping, follow these steps:
 1. [Verify your metric](#verify-your-metric)
 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally)
 
-NOTE:
-When you add or change a Service Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md).
-For information about the progress on migrating Service ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547).
-
 ## Instrumentation classes
 
-We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible.
+NOTE:
+Implementing metrics directly in `usage_data.rb` is deprecated.
+When you add or change a Service Ping Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md).
+For information about the progress on migrating Service Ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547).
 
 For example, we have the following instrumentation class:
 `lib/gitlab/usage/metrics/instrumentations/count_boards_metric.rb`.
@@ -45,7 +44,7 @@ boards: add_metric('CountBoardsMetric', time_frame: 'all'),
 
 ## Types of counters
 
-There are several types of counters in `usage_data.rb`:
+There are several types of counters for metrics:
 
 - **[Batch counters](#batch-counters)**: Used for counts and sums.
 - **[Redis counters](#redis-counters):** Used for in-memory counts.
@@ -72,64 +71,39 @@ you must add a specialized index on the columns involved in a counter.
 
 #### Ordinary batch counters
 
-Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors.
-Handles the `ActiveRecord::StatementInvalid` error.
+Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `count` operation for a given `ActiveRecord_Relation`
 
 Method:
 
 ```ruby
-count(relation, column = nil, batch: true, start: nil, finish: nil)
+add_metric('CountIssuesMetric', time_frame: 'all')
 ```
 
-Arguments:
-
-- `relation` the ActiveRecord_Relation to perform the count
-- `column` the column to perform the count on, by default is the primary key
-- `batch`: default `true` to use batch counting
-- `start`: custom start of the batch counting to avoid complex min calculations
-- `end`: custom end of the batch counting to avoid complex min calculations
-
 Examples:
 
-```ruby
-count(User.active)
-count(::Clusters::Cluster.aws_installed.enabled, :cluster_id)
-count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id))
-```
+Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md).
 
 #### Distinct batch counters
 
-Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors.
-Handles the `ActiveRecord::StatementInvalid` error.
+Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `distinct_count` operation for a given `ActiveRecord_Relation`.
 
 Method:
 
 ```ruby
-distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)
+add_metric('CountUsersAssociatingMilestonesToReleasesMetric', time_frame: 'all')
 ```
 
-Arguments:
-
-- `relation`: the ActiveRecord_Relation to perform the count
-- `column`: the column to perform the distinct count, by default is the primary key
-- `batch`: default `true` to use batch counting
-- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter`
-- `start`: custom start of the batch counting to avoid complex min calculations
-- `end`: custom end of the batch counting to avoid complex min calculations
-
 WARNING:
 Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide.
 
 Examples:
 
-```ruby
-distinct_count(::Project, :creator_id)
-distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id))
-distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id')
-```
+Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md).
 
 #### Sum batch operation
 
+There is no support for `sum` for database metrics.
+
 Sum the values of a given ActiveRecord_Relation on given column and handles errors.
 Handles the `ActiveRecord::StatementInvalid` error
 
@@ -686,29 +660,6 @@ We return fallback values in these cases:
 | Timeouts, general failures  | -1    |
 | Standard errors in counters | -2    |
 
-## Name and place the metric
-
-Add the metric in one of the top-level keys:
-
-- `settings`: for settings related metrics.
-- `counts_weekly`: for counters that have data for the most recent 7 days.
-- `counts_monthly`: for counters that have data for the most recent 28 days.
-- `counts`: for counters that have data for all time.
-
-### How to get a metric name suggestion
-
-The metric YAML generator can suggest a metric name for you.
-To generate a metric name suggestion, first instrument the metric at the provided `key_path`.
-Then, generate the metric's YAML definition and
-return to the instrumentation and update it.
-
-1. Add the metric instrumentation to `lib/gitlab/usage_data.rb` inside one
-   of the [top-level keys](#name-and-place-the-metric), using any name you choose.
-1. Run the [metrics YAML generator](metrics_dictionary.md#metrics-definition-and-validation).
-1. Use the metric name suggestion to select a suitable metric name.
-1. Update the instrumentation you created in the first step and change the metric name to the suggested name.
-1. Update the metric's YAML definition with the correct `key_path`.
-
 ## Test counters manually using your Rails console
 
 ```ruby
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index 86e70cc8bbc..6878fd1bf28 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 # Service Ping Guide **(FREE SELF)**
 
-> Introduced in GitLab Ultimate 11.2, more statistics.
+> - Introduced in GitLab Ultimate 11.2, more statistics.
+> - In GitLab 14.1, [renamed from Usage Ping to Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/5990). In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version.
 
 Service Ping is a GitLab process that collects and sends a weekly payload to GitLab.
 The payload provides important high-level data that helps our product, support,
@@ -68,7 +69,10 @@ Because of these limitations we recommend you:
 
 > Introduced in GitLab 14.1.
 
-In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data.
+In GitLab versions 14.1 and later, GitLab Free customers with a self-managed instance running
+[GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us
+activity data through Service Ping. Features introduced here do not remove the feature from its paid
+tier. Users can continue to access the features in a paid tier without sharing usage data.
 
 #### Features available in 14.1 and later
 
@@ -209,17 +213,17 @@ sequenceDiagram
 
 - `uuid` - GitLab instance unique identifier
 - `hostname` - GitLab instance hostname
-- `version` - GitLab instance current versions 
+- `version` - GitLab instance current versions
 - `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence
 - `message` - Error message
 
 <pre>
 <code>
 {
-  "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", 
-  "hostname"=>"127.0.0.1", 
-  "version"=>"14.7.0-pre", 
-  "elapsed"=>0.006946, 
+  "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d",
+  "hostname"=>"127.0.0.1",
+  "version"=>"14.7.0-pre",
+  "elapsed"=>0.006946,
   "message"=>'PG::UndefinedColumn: ERROR:  column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...'
 }
 </code>
@@ -576,7 +580,7 @@ skip_db_write:
 ServicePing::SubmitService.new(skip_db_write: true).execute
 ```
 
-## Manually upload Service Ping payload 
+## Manually upload Service Ping payload
 
 > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default.
 
@@ -596,7 +600,7 @@ To upload payload manually:
 
 ## Monitoring
 
-Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). 
+Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health).
 
 ## Troubleshooting
 
diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md
index 93eec4efabd..6884844da3f 100644
--- a/doc/development/service_ping/metrics_dictionary.md
+++ b/doc/development/service_ping/metrics_dictionary.md
@@ -47,13 +47,58 @@ Each metric is defined in a separate YAML file consisting of a number of fields:
 | `distribution`      | yes      | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available.  |
 | `performance_indicator_type`  | no      | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, or `umau`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). |
 | `tier`              | yes      | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. |
-| `milestone`         | no       | The milestone when the metric is introduced. |
+| `milestone`         | no       | The milestone when the metric is introduced and when it's available to self-managed instances with the official GitLab release. |
 | `milestone_removed` | no       | The milestone when the metric is removed. |
-| `introduced_by_url` | no       | The URL to the merge request that introduced the metric. |
+| `introduced_by_url` | no       | The URL to the merge request that introduced the metric to be available for self-managed instances. |
 | `repair_issue_url`  | no       | The URL of the issue that was created to repair a metric with a `broken` status. |
 | `options`           | no       | `object`: options information needed to calculate the metric value. |
 | `skip_validation`   | no       | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). |
 
+### Metric key_path
+
+The `key_path` of the metric is the location in the JSON Service Ping payload.
+
+The `key_path` could be composed from multiple parts separated by `.` and it must be unique.
+
+We recommend to add the metric in one of the top-level keys:
+
+- `settings`: for settings related metrics.
+- `counts_weekly`: for counters that have data for the most recent 7 days.
+- `counts_monthly`: for counters that have data for the most recent 28 days.
+- `counts`: for counters that have data for all time.
+
+NOTE:
+We can't control what the metric's `key_path` is, because some of them are generated dynamically in `usage_data.rb`.
+For example, see [Redis HLL metrics](implement.md#redis-hll-counters).
+
+### Metric name
+
+To improve metric discoverability by a wider audience, each metric with
+instrumentation added at an appointed `key_path` receives a `name` attribute
+filled with the name suggestion, corresponding to the metric `data_source` and instrumentation.
+Metric name suggestions can contain two types of elements:
+
+1. **User input prompts**: enclosed by angle brackets (`< >`), these pieces should be replaced or
+   removed when you create a metrics YAML file.
+1. **Fixed suggestion**: plaintext parts generated according to well-defined algorithms.
+   They are based on underlying instrumentation, and must not be changed.
+
+For a metric name to be valid, it must not include any prompt, and fixed suggestions
+must not be changed.
+
+#### Generate a metric name suggestion
+
+The metric YAML generator can suggest a metric name for you.
+To generate a metric name suggestion, first instrument the metric at the provided `key_path`.
+Then, generate the metric's YAML definition and
+return to the instrumentation and update it.
+
+1. Add the metric instrumentation class to `lib/gitlab/usage/metrics/instrumentations/`.
+1. Add the metric logic in the instrumentation class.
+1. Run the [metrics YAML generator](metrics_dictionary.md#metrics-definition-and-validation).
+1. Use the metric name suggestion to select a suitable metric name.
+1. Update the metric's YAML definition with the correct `key_path`.
+
 ### Metric statuses
 
 Metric definitions can have one of the following statuses:
@@ -81,21 +126,6 @@ which has a related schema in `/config/metrics/objects_schemas/topology_schema.j
 - `all`: The metric data applies for the whole time the metric has been active (all-time interval). For example, the following metric counts all users that create issues: `/config/metrics/counts_all/20210216181115_issues.yml`.
 - `none`: The metric collects a type of data that's not tracked over time, such as settings and configuration information. Therefore, a time interval is not applicable. For example, `uuid` has no time interval applicable: `config/metrics/license/20210201124933_uuid.yml`.
 
-### Metric name
-
-To improve metric discoverability by a wider audience, each metric with
-instrumentation added at an appointed `key_path` receives a `name` attribute
-filled with the name suggestion, corresponding to the metric `data_source` and instrumentation.
-Metric name suggestions can contain two types of elements:
-
-1. **User input prompts**: Enclosed by `<>`, these pieces should be replaced or
-   removed when you create a metrics YAML file.
-1. **Fixed suggestion**: Plaintext parts generated according to well-defined algorithms.
-   They are based on underlying instrumentation, and should not be changed.
-
-For a metric name to be valid, it must not include any prompt, and no fixed suggestions
-should be changed.
-
 ### Data category
 
 We use the following categories to classify a metric:
diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md
index c98b0df92aa..c684d9d12ef 100644
--- a/doc/development/service_ping/metrics_instrumentation.md
+++ b/doc/development/service_ping/metrics_instrumentation.md
@@ -57,6 +57,50 @@ module Gitlab
 end
 ```
 
+### Ordinary batch counters Example
+
+```ruby
+module Gitlab
+  module Usage
+    module Metrics
+      module Instrumentations
+        class CountIssuesMetric < DatabaseMetric
+          operation :count
+
+          start { Issue.minimum(:id) }
+          finish { Issue.maximum(:id) }
+
+          relation { Issue }
+        end
+      end
+    end
+  end
+end
+```
+
+### Distinct batch counters Example
+
+```ruby
+# frozen_string_literal: true
+
+module Gitlab
+  module Usage
+    module Metrics
+      module Instrumentations
+        class CountUsersAssociatingMilestonesToReleasesMetric < DatabaseMetric
+          operation :distinct_count, column: :author_id
+
+          relation { Release.with_milestones }
+
+          start { Release.minimum(:author_id) }
+          finish { Release.maximum(:author_id) }
+        end
+      end
+    end
+  end
+end
+```
+
 ## Redis metrics
 
 [Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66582).
diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md
index 137e11608cf..ee2d8f4f4a1 100644
--- a/doc/development/service_ping/review_guidelines.md
+++ b/doc/development/service_ping/review_guidelines.md
@@ -51,12 +51,13 @@ are regular backend changes.
 #### The Product Intelligence **reviewer** should
 
 - Perform a first-pass review on the merge request and suggest improvements to the author.
-- Check the [metrics location](implement.md#name-and-place-the-metric) in
+- Check the [metrics location](metrics_dictionary.md#metric-key_path) in
   the Service Ping JSON payload.
-- Suggest that the author checks the [naming suggestion](implement.md#how-to-get-a-metric-name-suggestion) while
+- Suggest that the author checks the [naming suggestion](metrics_dictionary.md#generate-a-metric-name-suggestion) while
   generating the metric's YAML definition.
 - Add the `~database` label and ask for a [database review](../database_review.md) for
   metrics that are based on Database.
+- Add `~Data Warehouse::Impact Check` for any database metric that has a query change. Changes in queries can affect [data operations](https://about.gitlab.com/handbook/business-technology/data-team/how-we-work/triage/#gitlabcom-db-structure-changes).
 - For tracking using Redis HLL (HyperLogLog):
   - Check the Redis slot.
   - Check if a [feature flag is needed](implement.md#recommendations).
diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md
new file mode 100644
index 00000000000..a25ad5f62be
--- /dev/null
+++ b/doc/development/service_ping/usage_data.md
@@ -0,0 +1,70 @@
+---
+stage: Growth
+group: Product Intelligence
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Usage Data Metrics guide
+
+This guide describes deprecated usage for metrics in `usage_data.rb`.
+
+NOTE:
+Implementing metrics direct in `usage_data.rb` is deprecated, We recommend you use [instrumentation classes](metrics_instrumentation.md).
+
+## Ordinary batch counters
+
+Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors.
+Handles the `ActiveRecord::StatementInvalid` error.
+
+Method:
+
+```ruby
+count(relation, column = nil, batch: true, start: nil, finish: nil)
+```
+
+Arguments:
+
+- `relation` the ActiveRecord_Relation to perform the count
+- `column` the column to perform the count on, by default is the primary key
+- `batch`: default `true` to use batch counting
+- `start`: custom start of the batch counting to avoid complex min calculations
+- `end`: custom end of the batch counting to avoid complex min calculations
+
+Examples:
+
+```ruby
+count(User.active)
+count(::Clusters::Cluster.aws_installed.enabled, :cluster_id)
+count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id))
+```
+
+## Distinct batch counters
+
+Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors.
+Handles the `ActiveRecord::StatementInvalid` error.
+
+Method:
+
+```ruby
+distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)
+```
+
+Arguments:
+
+- `relation`: the ActiveRecord_Relation to perform the count
+- `column`: the column to perform the distinct count, by default is the primary key
+- `batch`: default `true` to use batch counting
+- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter`
+- `start`: custom start of the batch counting to avoid complex min calculations
+- `end`: custom end of the batch counting to avoid complex min calculations
+
+WARNING:
+Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide.
+
+Examples:
+
+```ruby
+distinct_count(::Project, :creator_id)
+distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id))
+distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id')
+```