Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42

5 files changed, 105 insertions, 129 deletions

diff --git a/doc/development/service_ping/dictionary.md b/doc/development/service_ping/dictionary.md
deleted file mode 100644
index 810c789bc03..00000000000
--- a/doc/development/service_ping/dictionary.md
+++ /dev/null
@@ -1,4 +0,0 @@
----
-redirect_to: 'https://metrics.gitlab.com/index.html'
-remove_date: '2021-11-10'
----
diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 65a8b4c1cad..c32789740c3 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -26,6 +26,10 @@ To implement a new metric in Service Ping, follow these steps:
 1. [Verify your metric](#verify-your-metric)
 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally)
 
+NOTE:
+When you add or change a Service Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md).
+For information about the progress on migrating Service ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547).
+
 ## Instrumentation classes
 
 We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible.
@@ -795,7 +799,7 @@ To set up Service Ping locally, you must:
 
 1. [Set up local repositories](#set-up-local-repositories).
 1. [Test local setup](#test-local-setup).
-1. (Optional) [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping).
+1. Optional. [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping).
 
 ### Set up local repositories
 
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index 6ddbe2f9646..1f751eea4d8 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -8,40 +8,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 > Introduced in GitLab Ultimate 11.2, more statistics.
 
-This guide describes Service Ping's purpose and how it's implemented.
-
-For more information about Product Intelligence, see:
-
-- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
-- [Snowplow Guide](../snowplow/index.md)
-
-More links:
-
-- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/)
-- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/)
-- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/)
-- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/)
-
-## What is Service Ping?
-
-Service Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc.
+Service Ping is a GitLab process that collects and sends a weekly payload to GitLab.
 The payload provides important high-level data that helps our product, support,
-and sales teams understand how GitLab is used. For example, the data helps to:
+and sales teams understand how GitLab is used. The data helps to:
 
 - Compare counts month over month (or week over week) to get a rough sense for how an instance uses
   different product features.
 - Collect other facts that help us classify and understand GitLab installations.
-- Calculate our Stage Monthly Active Users (SMAU), which helps to measure the success of our stages
+- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages
   and features.
 
-Service Ping information is not anonymous. It's linked to the instance's hostname. However, it does
+Service Ping information is not anonymous. It's linked to the instance's hostname, but does
 not contain project names, usernames, or any other specific data.
 
-Sending a Service Ping payload is optional and can be [disabled](#disable-service-ping) on any self-managed instance.
-When Service Ping is enabled, GitLab gathers data from the other instances
+Sending a Service Ping payload is optional and you can [disable](#disable-service-ping) it on any
+self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances
 and can show your instance's usage statistics to your users.
 
-### Terminology
+## Service Ping terminology
 
 We use the following terminology to describe the Service Ping components:
 
@@ -53,12 +37,18 @@ We use the following terminology to describe the Service Ping components:
 - **MAU**: monthly active users.
 - **WAU**: weekly active users.
 
-### Why should we enable Service Ping?
+### Why enable Service Ping?
+
+The main purpose of Service Ping is to build a better GitLab. We collect data about how GitLab is used
+to understand feature or stage adoption and usage. This data gives an insight into how GitLab adds
+value and helps our team understand the reasons why people use GitLab, and with this knowledge we're able to
+make better product decisions.
+
+There are several other benefits to enabling Service Ping:
 
-- The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions.
 - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation.
 - As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring.
-- You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value)
+- You get better, more proactive support (assuming that our TAMs and support organization used the data to deliver more value).
 - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization?
 - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes.
 - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping).
@@ -78,7 +68,7 @@ Because of these limitations we recommend you:
 
 > Introduced in GitLab 14.1.
 
-In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data.
+In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data.
 
 #### Features available in 14.1 and later
 
@@ -484,20 +474,34 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta
 
 1. Connect to bastion with agent forwarding:
 
-   `$ ssh -A lb-bastion.gprd.gitlab.com`.
+   ```shell
+   ssh -A lb-bastion.gprd.gitlab.com
+   ```
+
 1. Create named screen:
 
-   `$ screen -S <username>_usage_ping_<date>`.
+   ```shell
+   screen -S <username>_usage_ping_<date>
+   ```
+
 1. Connect to console host:
+  
+   ```shell
+   ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal
+   ```
 
-   `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`.
-1. Run `ServicePing::SubmitService.new.execute`
-1. Detach from screen:
+1. Run:
 
-   `ctrl + a, ctrl + d`
-1. Exit from bastion:
+   ```shell
+   ServicePing::SubmitService.new.execute
+   ```
 
-   `$ exit`
+1. To detach from screen, press `ctrl + A`, `ctrl + D`.
+1. Exit from bastion:
+  
+   ```shell
+   exit
+   ```
 
 ### Verification (After approx 30 hours)
 
@@ -511,22 +515,49 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta
 
 1. Reconnect to bastion:
 
-   `$ ssh -A lb-bastion.gprd.gitlab.com`
+   ```shell
+   ssh -A lb-bastion.gprd.gitlab.com
+   ```
+
 1. Find your screen session:
 
-   `$ screen -ls`
+   ```shell
+   screen -ls
+   ```
+
 1. Attach to your screen session:
 
-   `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22`
-1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload`
-1. Check the when the payload was sent: `RawUsageData.last.sent_at`
+   ```shell
+   screen -x 14226.mwawrzyniak_usage_ping_2021_01_22
+   ```
+
+1. Check the last payload in `raw_usage_data` table:
+  
+   ```shell
+   RawUsageData.last.payload
+   ```
+
+1. Check the when the payload was sent:
+
+   ```shell
+   RawUsageData.last.sent_at
+   ```
+
+### Skip database write operations
+
+To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument:
+
+```shell
+skip_db_write:
+ServicePing::SubmitService.new(skip_db_write: true).execute
+```
 
 ## Troubleshooting
 
 ### Cannot disable Service Ping using the configuration file
 
 The method to disable Service Ping using the GitLab configuration file does not work in
-GitLab versions 9.3.0 to 13.12.3. To disable it, you need to use the Admin Area in
+GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in
 the GitLab UI instead. For more information, see
 [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269).
 
@@ -603,3 +634,12 @@ To work around this bug, you have two options:
   1. Expand **Usage Statistics**.
   1. Clear the **Enable Service Ping** checkbox.
   1. Select **Save Changes**.
+
+## Related topics
+
+- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/)
+- [Snowplow Guide](../snowplow/index.md)
+- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/)
+- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/)
+- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/)
+- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/)
diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md
index b3c404de150..808c5064cf3 100644
--- a/doc/development/service_ping/metrics_dictionary.md
+++ b/doc/development/service_ping/metrics_dictionary.md
@@ -181,6 +181,7 @@ product_group: group::product intelligence
 value_type: string
 status: active
 milestone: 9.1
+instrumentation_class: UuidMetric
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521
 time_frame: none
 data_source: database
@@ -199,22 +200,23 @@ The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-o
 
 For uniqueness, the generated files include a timestamp prefix in ISO 8601 format.
 
-The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location:
+The generator takes a list of key paths and 3 options as arguments. It creates metric YAML definitions in the corresponding location:
 
 - `--ee`, `--no-ee` Indicates if metric is for EE.
 - `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`.
+- `--class_name=CLASS_NAME` Indicates the instrumentation class. For example `UsersCreatingIssuesMetric`, `UuidMetric`
 
 **Single metric example**
 
 ```shell
-bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d
+bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d --class_name=CountIssues
 create  config/metrics/counts_7d/issues.yml
 ```
 
 **Multiple metrics example**
 
 ```shell
-bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d
+bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d --class_name=CountUsersCreatingIssues
 create  config/metrics/counts_7d/issues.yml
 create  config/metrics/counts_7d/users.yml
 ```
@@ -223,7 +225,7 @@ NOTE:
 To create a metric definition used in EE, add the `--ee` flag.
 
 ```shell
-bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d
+bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d --class_name=CountUsersCreatingIssues
 create  ee/config/metrics/counts_7d/issues.yml
 ```
 
@@ -236,9 +238,9 @@ A YAML metric definition is required for each metric. A dedicated generator is p
 The generator takes `category` and `event` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for weekly and monthly time frames:
 
 ```shell
-bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed
-create  config/metrics/counts_7d/i_closed_weekly.yml
-create  config/metrics/counts_28d/i_closed_monthly.yml
+bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues
+create  config/metrics/counts_7d/count_users_closing_issues_weekly.yml
+create  config/metrics/counts_28d/count_users_closing_issues_monthly.yml
 ```
 
 To create a metric definition used in EE, add the `--ee` flag.
diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md
index 46040146de2..ebfab6341e9 100644
--- a/doc/development/service_ping/metrics_lifecycle.md
+++ b/doc/development/service_ping/metrics_lifecycle.md
@@ -55,23 +55,24 @@ The correct approach is to add a new metric for GitLab 12.6 release with updated
 
 and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric`
 
-## Deprecate a metric
+## Remove a metric
+
+WARNING:
+If a metric is not used in Sisense or any other system after 6 months, the
+Product Intelligence team marks it as inactive and assigns it to the group owner for review.
+
+We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details.
 
-If a metric is obsolete and you no longer use it, you can mark it as deprecated.
+Product Intelligence removes metrics from Service Ping if they are not used in any Sisense dashboard.
 
-For an example of the metric deprecation process take a look at this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59899)
+For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029).
 
-To deprecate a metric:
+To remove a metric:
 
 1. Check the following YAML files and verify the metric is not used in an aggregate:
    - [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/)
    - [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/)
 
-1. Create an issue in the [GitLab Data Team
-   project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for
-   confirmation that the metric is not used by other teams, or in any of the SiSense
-   dashboards.
-
 1. Verify the metric is not used to calculate the conversational index. The
    conversational index is a measure that reports back to self-managed instances
    to inform administrators of the progress of DevOps adoption for the instance.
@@ -81,70 +82,6 @@ To deprecate a metric:
    to view the metrics that are used. The metrics are represented
    as the keys that are passed as a field argument into the `get_value` method.
 
-1. Document the deprecation in the metric's YAML definition. Set
-   the `status:` attribute to `deprecated`, for example:
-
-   ```yaml
-   ---
-   key_path: analytics_unique_visits.analytics_unique_visits_for_any_target_monthly
-   description: Visits to any of the pages listed above per month
-   product_section: dev
-   product_stage: manage
-   product_group: group::analytics
-   product_category:
-   value_type: number
-   status: deprecated
-   time_frame: 28d
-   data_source:
-   distribution:
-   - ce
-   tier:
-   - free
-   ```
-
-1. Replace the metric's instrumentation with a fixed value. This avoids wasting
-   resources to calculate the deprecated metric. In
-   [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb)
-   or
-   [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb),
-   replace the code that calculates the metric's value with a fixed value that
-   indicates it's deprecated:
-
-   ```ruby
-   module Gitlab
-     class UsageData
-       DEPRECATED_VALUE = -1000
-
-       def analytics_unique_visits_data
-         results['analytics_unique_visits_for_any_target'] = redis_usage_data { unique_visit_service.unique_visits_for(targets: :analytics) }
-         results['analytics_unique_visits_for_any_target_monthly'] = DEPRECATED_VALUE
-
-         { analytics_unique_visits: results }
-       end
-     # ...
-     end
-   end
-   ```
-
-## Remove a metric
-
-### Removal policy
-
-WARNING:
-A metric that is not used in Sisense or any other system after 6 months is marked by the
-Product Intelligence team as inactive and is assigned to the group owner for review.
-
-We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details.
-
-Metrics can be removed from Service Ping if they:
-
-- Were previously [deprecated](#deprecate-a-metric).
-- Are not used in any Sisense dashboard.
-
-For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029)
-
-### To remove a deprecated metric
-
 1. Verify that removing the metric from the Service Ping payload does not cause
    errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com)
    when the updated payload is collected and processed. Version App collects
@@ -159,9 +96,6 @@ For an example of the metric removal process take a look at this [example issue]
    Ask for confirmation that the metric is not referred to in any SiSense dashboards and
    can be safely removed from Service Ping. Use this
    [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance.
-   This step can be skipped if verification done during [deprecation process](#deprecate-a-metric)
-   reported that metric is not required by any data transformation in Snowflake data warehouse nor it is
-   used by any of SiSense dashboards.
 
 1. After you verify the metric can be safely removed,
    update the attributes of the metric's YAML definition: