Add latest changes from gitlab-org/gitlab@14-5-stable-eev14.5.0-rc42

3 files changed, 111 insertions, 53 deletions

diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md
index 34a845147dc..65a8b4c1cad 100644
--- a/doc/development/service_ping/implement.md
+++ b/doc/development/service_ping/implement.md
@@ -295,19 +295,22 @@ Examples of implementation:
 - Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb)
 - Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb)
 
-##### UsageData API tracking
+##### `UsageData` API
 
-<!-- There's nearly identical content in `##### Adding new events`. If you fix errors here, you may need to fix the same errors in the other location. -->
+You can use the `UsageData` API to track events.
+To track events, the `usage_data_api` feature flag must
+be enabled (set to `default_enabled: true`).
+Enabled by default in GitLab 13.7 and later.
 
-1. Track event using `UsageData` API
+##### UsageData API tracking
 
-   Increment event count using ordinary Redis counter, for given event name.
+1. Track events using the [`UsageData` API](#usagedata-api).
 
-   Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default.
+   Increment event count using an ordinary Redis counter, for a given event name.
 
    API requests are protected by checking for a valid CSRF token.
 
-   To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled.
+   To increment the values, the related feature `usage_data_<event_name>` must be enabled.
 
    ```plaintext
    POST /usage_data/increment_counter
@@ -315,18 +318,18 @@ Examples of implementation:
 
    | Attribute | Type | Required | Description |
    | :-------- | :--- | :------- | :---------- |
-   | `event` | string | yes | The event name it should be tracked |
+   | `event` | string | yes | The event name to track. |
 
    Response:
 
-   - `200` if event was tracked
-   - `400 Bad request` if event parameter is missing
-   - `401 Unauthorized` if user is not authenticated
-   - `403 Forbidden` for invalid CSRF token provided
+   - `200` if the event was tracked.
+   - `400 Bad request` if the event parameter is missing.
+   - `401 Unauthorized` if the user is not authenticated.
+   - `403 Forbidden` if an invalid CSRF token is provided.
 
-1. Track events using JavaScript/Vue API helper which calls the API above
+1. Track events using the JavaScript/Vue API helper which calls the [`UsageData` API](#usagedata-api).
 
-   Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events
+   To track events, `usage_data_api` and `usage_data_#{event_name}` must be enabled.
 
    ```javascript
    import api from '~/api';
@@ -460,14 +463,10 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF
        track_usage_event(:incident_management_incident_created, current_user.id)
      ```
 
-   - Using the `UsageData` API.
-     <!-- There's nearly identical content in `##### UsageData API Tracking`. If you find / fix errors here, you may need to fix errors in that section too. -->
+   - Using the [`UsageData` API](#usagedata-api).
 
      Increment unique users count using Redis HLL, for a given event name.
 
-     To track events using the `UsageData` API, ensure the `usage_data_api` feature flag
-     is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later.
-
      API requests are protected by checking for a valid CSRF token.
 
      ```plaintext
@@ -485,10 +484,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF
      - `401 Unauthorized` if the user is not authenticated.
      - `403 Forbidden` if an invalid CSRF token is provided.
 
-   - Using the JavaScript/Vue API helper, which calls the `UsageData` API.
-
-     To track events using the `UsageData` API, ensure the `usage_data_api` feature flag
-     is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later.
+   - Using the JavaScript/Vue API helper, which calls the [`UsageData` API](#usagedata-api).
 
      Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/):
 
@@ -559,13 +555,18 @@ We can also disable tracking completely by using the global flag:
 
 ##### Known events are added automatically in Service Data payload
 
-All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209).
+Service Ping adds all events [`known_events/*.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events) to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209).
 For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable:
 
 - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events.
 - `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events.
 
-Redis HLL implementation calculates automatic total metrics, if there are more than one metric for the same category, aggregation, and Redis slot.
+Redis HLL implementation calculates total metrics when both of these conditions are met:
+
+- The category is manually included in [CATEGORIES_FOR_TOTALS](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/hll_redis_counter.rb#L21).
+- There is more than one metric for the same category, aggregation, and Redis slot.
+
+We add total unique counts for the weekly and monthly time frames where applicable:
 
 - `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric.
 - `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric.
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md
index 19bf7446da9..6ddbe2f9646 100644
--- a/doc/development/service_ping/index.md
+++ b/doc/development/service_ping/index.md
@@ -50,35 +50,57 @@ We use the following terminology to describe the Service Ping components:
 - **Metrics**: primarily made up of row counts for different tables in an instance's database. Each
   metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation)
   in a YAML file.
+- **MAU**: monthly active users.
+- **WAU**: weekly active users.
 
 ### Why should we enable Service Ping?
 
 - The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions.
 - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation.
-- As a benefit of having Service Ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring.
+- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring.
 - You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value)
 - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization?
 - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes.
 - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping).
 - When Service Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features.
 
-#### Registration Features Program
+### Limitations
+
+- Service Ping does not track frontend events things like page views, link clicks, or user sessions.
+- Service Ping focuses only on aggregated backend events.
+
+Because of these limitations we recommend you:
+
+- Instrument your products with Snowplow for more detailed analytics on GitLab.com.
+- Use Service Ping to track aggregated backend events on self-managed instances.
+
+### Registration Features Program
 
 > Introduced in GitLab 14.1.
 
-Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data.
+In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data.
 
-##### Features available in 14.1 and later
+#### Features available in 14.1 and later
 
 1. [Email from GitLab](../../tools/email.md).
 
+#### Features available in 14.4 and later
+
+1. [Repository size limit](../../user/admin_area/settings/account_and_limit_settings.md#repository-size-limit).
+1. [Restrict group access by IP address](../../user/group/index.md#restrict-group-access-by-ip-address).
+
 NOTE:
 Registration is not yet required for participation, but will be added in a future milestone.
 
-### Limitations
+#### Enable Registration Features
 
-- Service Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events.
-- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Service Ping to track aggregated backend events on self-managed.
+1. Sign in as a user with the [Administrator](../../user/permissions.md) role.
+1. On the top bar, select **Menu > Admin**.
+1. On the left sidebar, select **Settings > Metrics and profiling**.
+1. Expand the **Usage statistics** section.
+1. If not enabled, select the **Enable Service Ping** checkbox.
+1. Select the **Enable Registration Features** checkbox.
+1. Select **Save changes**.
 
 ## View the Service Ping payload **(FREE SELF)**
 
@@ -180,7 +202,7 @@ sequenceDiagram
     S3 Bucket->>Snowflake DW: Import data
     Snowflake DW->>Snowflake DW: Transform data using dbt
     Snowflake DW->>Sisense Dashboards: Data available for querying
-    Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index)
+    Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index)
 ```
 
 ## How Service Ping works
@@ -209,10 +231,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec
        "repositories_replication_enabled"=>true,
        "repositories_synced_count"=>24,
        "repositories_failed_count"=>0,
-       "attachments_replication_enabled"=>true,
-       "attachments_count"=>1,
-       "attachments_synced_count"=>1,
-       "attachments_failed_count"=>0,
        "git_fetch_event_count_weekly"=>nil,
        "git_push_event_count_weekly"=>nil,
        ... other geo node status fields
@@ -450,27 +468,56 @@ bin/rake gitlab:usage_data:dump_sql_in_json
 bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml
 ```
 
-## Generating and troubleshooting Service Ping
-
-This activity is to be done via a detached screen session on a remote server.
+## Generate Service Ping
 
-Before you begin these steps, make sure the key is added to the SSH agent locally
-with the `ssh-add` command.
+To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server.
 
 ### Triggering
 
-1. Connect to bastion with agent forwarding: `$ ssh -A lb-bastion.gprd.gitlab.com`
-1. Create named screen: `$ screen -S <username>_usage_ping_<date>`
-1. Connect to console host: `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`
-1. Run `SubmitUsagePingService.new.execute`
-1. Detach from screen: `ctrl + a, ctrl + d`
-1. Exit from bastion: `$ exit`
+#### Trigger Service Ping with Teleport
+
+1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment.
+1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval).
+1. Run `ServicePing::SubmitService.new.execute`.
+
+#### Trigger Service Ping with a detached screen session
+
+1. Connect to bastion with agent forwarding:
+
+   `$ ssh -A lb-bastion.gprd.gitlab.com`.
+1. Create named screen:
+
+   `$ screen -S <username>_usage_ping_<date>`.
+1. Connect to console host:
+
+   `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`.
+1. Run `ServicePing::SubmitService.new.execute`
+1. Detach from screen:
+
+   `ctrl + a, ctrl + d`
+1. Exit from bastion:
+
+   `$ exit`
 
 ### Verification (After approx 30 hours)
 
-1. Reconnect to bastion: `$ ssh -A lb-bastion.gprd.gitlab.com`
-1. Find your screen session: `$ screen -ls`
-1. Attach to your screen session: `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22`
+#### Verify with a detached screen session
+
+1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console
+1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload`
+1. Check the when the payload was sent: `RawUsageData.last.sent_at`
+
+#### Verify using detached screen session
+
+1. Reconnect to bastion:
+
+   `$ ssh -A lb-bastion.gprd.gitlab.com`
+1. Find your screen session:
+
+   `$ screen -ls`
+1. Attach to your screen session:
+
+   `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22`
 1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload`
 1. Check the when the payload was sent: `RawUsageData.last.sent_at`
 
diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md
index c1478e6290e..b3c404de150 100644
--- a/doc/development/service_ping/metrics_dictionary.md
+++ b/doc/development/service_ping/metrics_dictionary.md
@@ -197,18 +197,28 @@ tier:
 
 The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions.
 
-For uniqueness, the generated file includes a timestamp prefix, in ISO 8601 format.
+For uniqueness, the generated files include a timestamp prefix in ISO 8601 format.
 
-The generator takes the key path argument and 2 options and creates the metric YAML definition in corresponding location:
+The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location:
 
 - `--ee`, `--no-ee` Indicates if metric is for EE.
-- `--dir=DIR` indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`.
+- `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`.
+
+**Single metric example**
 
 ```shell
 bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d
 create  config/metrics/counts_7d/issues.yml
 ```
 
+**Multiple metrics example**
+
+```shell
+bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d
+create  config/metrics/counts_7d/issues.yml
+create  config/metrics/counts_7d/users.yml
+```
+
 NOTE:
 To create a metric definition used in EE, add the `--ee` flag.