diff options
Diffstat (limited to 'doc/development/telemetry')
| -rw-r--r-- | doc/development/telemetry/event_dictionary.md | 32 | ||||
| -rw-r--r-- | doc/development/telemetry/index.md | 178 | ||||
| -rw-r--r-- | doc/development/telemetry/snowplow.md | 18 | ||||
| -rw-r--r-- | doc/development/telemetry/usage_ping.md | 422 |
4 files changed, 186 insertions, 464 deletions
diff --git a/doc/development/telemetry/event_dictionary.md b/doc/development/telemetry/event_dictionary.md new file mode 100644 index 00000000000..d8cc32ea8d0 --- /dev/null +++ b/doc/development/telemetry/event_dictionary.md @@ -0,0 +1,32 @@ +--- +stage: Growth +group: Telemetry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Event Dictionary + +**Note: We've temporarily moved the Event Dictionary to a [Google Sheet](https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit?usp=sharing)**. The previous Markdown table exceeded 600 rows making it difficult to manage. In the future, our intention is to move this back into our docs using a [YAML file](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/823). + +The event dictionary is a single source of truth for the metrics and events we collect for product usage data. The Event Dictionary lists all the metrics and events we track, why we're tracking them, and where they are tracked. + +This is a living document that is updated any time a new event is planned or implemented. It includes the following information. + +- Section, stage, or group +- Description +- Implementation status +- Availability by plan type +- Code path + +We're currently focusing our Event Dictionary on [Usage Ping](usage_ping.md). In the future, we will also include [Snowplow](snowplow.md). We currently have an initiative across the entire product organization to complete the [Event Dictionary for Usage Ping](https://gitlab.com/groups/gitlab-org/-/epics/4174). + +## Instructions + +1. Open the Event Dictionary and fill in all the **PM to edit** columns highlighted in yellow. +1. Check that all the metrics and events are assigned to the correct section, stage, or group. If a metric is used across many groups, assign it to the stage. If a metric is used across many stages, assign it to the section. If a metric is incorrectly assigned to another section, stage, or group, let the PM know you have reassigned it. If your group has no assigned metrics and events, check that your metrics and events are not incorrectly assigned to another PM. +1. Add descriptions of what your metrics and events are tracking. Work with your Engineering team or the Telemetry team if you need help understanding this. +1. Add what plans this metric is available on. Work with your Engineering team or the Telemetry team if you need help understanding this. + +## Planned metrics and events + +For future metrics and events you plan to track, please add them to the Event Dictionary and note the status as `Planned`, `In Progress`, or `Implemented`. Once you have confirmed the metric has been implemented and have confirmed the metric data is in our data warehouse, change the status to **Data Available**. diff --git a/doc/development/telemetry/index.md b/doc/development/telemetry/index.md index 0000e7e9e4f..b5032ce3730 100644 --- a/doc/development/telemetry/index.md +++ b/doc/development/telemetry/index.md @@ -6,60 +6,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Telemetry Guide -At GitLab, we collect telemetry for the purpose of helping us build a better GitLab. Data about how GitLab is used is collected to better understand what parts of GitLab needs improvement and what features to build next. Telemetry also helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions. +At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions. -We also encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. By enabling tracking, users can: +We encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. + +By enabling tracking, users can: - Contribute back to the wider community. - Help GitLab improve on the product. -This documentation consists of three guides providing an overview of Telemetry at GitLab. - -Telemetry Guide: - - 1. [Our tracking tools](#our-tracking-tools) - 1. [What data can be tracked](#what-data-can-be-tracked) - 1. [Telemetry systems overview](#telemetry-systems-overview) - 1. [Snowflake data warehouse](#snowflake-data-warehouse) +## Our tracking tools -[Usage Ping Guide](usage_ping.md) +We use three methods to gather product usage data: - 1. [What is Usage Ping](usage_ping.md#what-is-usage-ping) - 1. [Usage Ping payload](usage_ping.md#usage-ping-payload) - 1. [Disable Usage Ping](usage_ping.md#disable-usage-ping) - 1. [Usage Ping request flow](usage_ping.md#usage-ping-request-flow) - 1. [How Usage Ping works](usage_ping.md#how-usage-ping-works) - 1. [Implementing Usage Ping](usage_ping.md#implementing-usage-ping) - 1. [Developing and testing Usage Ping](usage_ping.md#developing-and-testing-usage-ping) +- [Snowplow](#snowplow) +- [Usage Ping](#usage-ping) +- [Database import](#database-import) -[Snowplow Guide](snowplow.md) +### Snowplow -1. [What is Snowplow](snowplow.md#what-is-snowplow) -1. [Snowplow schema](snowplow.md#snowplow-schema) -1. [Enabling Snowplow](snowplow.md#enabling-snowplow) -1. [Snowplow request flow](snowplow.md#snowplow-request-flow) -1. [Implementing Snowplow JS (Frontend) tracking](snowplow.md#implementing-snowplow-js-frontend-tracking) -1. [Implementing Snowplow Ruby (Backend) tracking](snowplow.md#implementing-snowplow-ruby-backend-tracking) -1. [Developing and testing Snowplow](snowplow.md#developing-and-testing-snowplow) +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way +users engage with our website and application. -More useful links: +Snowplow consists of two components: -- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) - -## Our tracking tools +- [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) tracks client-side + events. +- [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) tracks server-side events. -We use several different technologies to gather product usage data. - -### Snowplow JS (Frontend) - -Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) is a frontend tracker for client-side events. - -### Snowplow Ruby (Backend) - -Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) is a backend tracker for server-side events. +For more details, read the [Snowplow](snowplow.md) guide. ### Usage Ping @@ -71,37 +46,39 @@ For more details, read the [Usage Ping](usage_ping.md) guide. Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/platform/#extract-and-load). -### Log system - -System logs are the application logs generated from running the GitLab Rails application. For more details, see the [log system](../../administration/logs.md) and [logging infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview). - ## What data can be tracked Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. -| Event Type | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Log system | -|---------------------|------------------------|-------------------------|---------------------|---------------------|---------------------| -| Database counts | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | -| Pageview events | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| UI events | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| CRUD and API events | **{dotted-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| Event funnels | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| PostgreSQL Data | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | **{dotted-circle}** | -| Logs | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | -| External services | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | +The availability of event types and their tracking tools varies by segment. For example, on Self-Managed Users, we only have reporting using Database records via Usage Ping. + +| Event Types | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|----------------------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow (JS Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (JS UI events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (Ruby Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (Ruby CRUD / API events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Usage Ping (Redis UI counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Redis Pageview counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Redis CRUD / API counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Database counters) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Usage Ping (Instance settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Usage Ping (Integration settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Database import (Database records) | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +[Source file](https://docs.google.com/spreadsheets/d/1e8Afo41Ar8x3JxAXJF3nL83UxVZ3hPIyXdt243VnNuE/edit?usp=sharing) -### Database counts +**Legend** -- Number of Projects created by unique users -- Number of users logged in the past 28 day +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible -Database counts are row counts for different tables in an instance’s database. These are SQL count queries which have been filtered, grouped, or aggregated which provide high level usage data. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). +SaaS = GitLab.com. SM = Self-Managed instance ### Pageview events - Number of sessions that visited the /dashboard/groups page -### UI Events +### UI events - Number of sessions that clicked on a button or link - Number of sessions that closed a modal @@ -116,24 +93,55 @@ UI events are any interface-driven actions from the browser including click data These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface. -### Event funnels +### Database records -- Number of sessions that performed action A, B, then C -- Conversion rate from step A to B +These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). -### PostgreSQL data +### Instance settings -These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). +These are settings of your instance such as the instance's Git version and if certain features are enabled such as `container_registry_enabled`. -### Logs +### Integration settings -These are raw logs such as the [Production logs](../../administration/logs.md#production_jsonlog), [API logs](../../administration/logs.md#api_jsonlog), or [Sidekiq logs](../../administration/logs.md#sidekiqlog). See the [overview of Logging Infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview) for more details. +These are integrations your GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. -### External services +## Reporting level -These are external services a GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. +Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Usage Ping but not on an Individual user level. -## Telemetry systems overview +| Aggregated Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|----------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✅ | 📅 | 📅 | ✅ | 📅 | +| Usage Ping | ✅ | 🔄 | 📅 | 📅 | ✅ | ✅ | ✅ | ✅ | 📅 | ✅ | +| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +| Identifiable Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | +| Usage Ping | ✅ | 🔄 | 📅 | ✖️ | ✖️ | ✅ | ✅ | ✖️ | ✖️ | ✖️ | +| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +**Legend** + +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible + +SaaS = GitLab.com. SM = Self-Managed instance + +## Reporting time period + +Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Usage Ping. + +| Reporting Time Period | All Time | 28 Days | 7 Days | Daily | +|-----------------------|----------|---------|--------|-------| +| Snowplow | ✅ | ✅ | ✅ | ✅ | +| Usage Ping | ✅ | ✅ | 📅 | ✖️ | +| Database import | ✅ | ✅ | ✅ | ✅ | + +**Legend** + +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible + +## Systems overview The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances. @@ -145,7 +153,7 @@ The systems overview is a simplified diagram showing the interactions between Gi For Telemetry purposes, GitLab Inc has three major components: -1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/) and GitLab's Versions Application. +1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/snowplow/) and GitLab's Versions Application. 1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. 1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana. @@ -162,27 +170,13 @@ As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Pi As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. -The differences between GitLab.com and self-managed are summarized below: - -| Environment | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Logs system | -|--------------|------------------------|-------------------------|--------------------|---------------------|---------------------| -| GitLab.com | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Self-Managed | **{dotted-circle}**(1) | **{dotted-circle}**(1) | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | - Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. -## Snowflake data warehouse - -The Snowflake data warehouse is where we keep all of GitLab Inc's data. - -### Data sources - -There are several data sources available in Snowflake and Sisense each representing a different view of the data along the transformation pipeline. +## Additional information -| Source | Description | Access | -| ------ | ------ | ------ | -| raw | These tables are the raw data source | Access via Snowflake | -| analytics_staging | These tables have undergone little to no data transformation, meaning they're basically clones of the raw data source | Access via Snowflake or Sisense | -| analytics | These tables have typically undergone more data transformation. They will typically end in `_xf` to represent the fact that they are transformed | Access via Snowflake or Sisense | +More useful links: -If you are a Product Manager interested in the raw data, you will likely focus on the `analytics` and `analytics_staging` sources. The raw source is limited to the data and infrastructure teams. For more information, please see [Data For Product Managers: What's the difference between analytics_staging and analytics?](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#whats-the-difference-between-analytics_staging-and-analytics) +- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md index f03742afe2d..547ba36464b 100644 --- a/doc/development/telemetry/snowplow.md +++ b/doc/development/telemetry/snowplow.md @@ -278,7 +278,7 @@ Custom event tracking and instrumentation can be added by directly calling the ` |:-----------|:-------|:--------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | | `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). These are set as empty strings if you don't provide them. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Instrumentation at GitLab](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). These are set as empty strings if you don't provide them. | Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visualize performance over time in an area or aspect of code. @@ -305,12 +305,16 @@ We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker There are several tools for developing and testing Snowplow Event -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | -| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | +|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| +| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | + +**Legend** + +**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned ### Snowplow Analytics Debugger Chrome Extension diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index 5e78e2c5f25..ea5eb6c389f 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -131,7 +131,7 @@ There are four types of counters which are all found in `usage_data.rb`: - **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation - **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column - **Alternative Counters:** Used for settings and configurations -- **Redis Counters:** Used for in-memory counts. This method is being deprecated due to data inaccuracies and will be replaced with a persistent method. +- **Redis Counters:** Used for in-memory counts. NOTE: **Note:** Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. @@ -155,7 +155,7 @@ There are two batch counting methods provided, `Ordinary Batch Counters` and `Di Handles `ActiveRecord::StatementInvalid` error -Simple count of a given ActiveRecord_Relation +Simple count of a given ActiveRecord_Relation, does a non-distinct batch count, smartly reduces batch_size and handles errors. Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` @@ -179,15 +179,16 @@ count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters: Handles `ActiveRecord::StatementInvalid` error -Distinct count of a given ActiveRecord_Relation on given column +Distinct count of a given ActiveRecord_Relation on given column, a distinct batch count, smartly reduces batch_size and handles errors. -Method: `distinct_count(relation, column = nil, batch: true, start: nil, finish: nil)` +Method: `distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)` Arguments: - `relation` the ActiveRecord_Relation to perform the count - `column` the column to perform the distinct count, by default is the primary key - `batch`: default `true` in order to use batch counting +- `batch_size`: if none set it will use default value 10000 from `Gitlab::Database::BatchCounter` - `start`: custom start of the batch counting in order to avoid complex min calculations - `end`: custom end of the batch counting in order to avoid complex min calculations @@ -212,14 +213,48 @@ Arguments: - `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented - or a `block`: which is evaluated +#### Ordinary Redis Counters + +Examples of implementation: + +- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) +- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) + +#### Redis HLL Counters + +With `Gitlab::Redis::HLL` we have available data structures used to count unique values. + +Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). + +Recommendations: + +- Key should expire in 29 days. +- If possible, data granularity should be a week. For example a key could be composed from the metric's name and week of the year, `2020-33-{metric_name}`. +- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics. +- If possible, data granularity should be week, for example a key could be composed from metric name and week of the year, 2020-33-{metric_name} +- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics + +Examples of implementation: + +- [`Gitlab::UsageDataCounters::TrackUniqueActions`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/track_unique_actions.rb) +- [`Gitlab::Analytics::UniqueVisits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/unique_visits.rb) + Example of usage: ```ruby +# Redis Counters redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } -``` -Note that Redis counters are in the [process of being deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/216330) and you should instead try to use Snowplow events instead. We're in the process of building [self-managed event tracking](https://gitlab.com/gitlab-org/telemetry/-/issues/373) and once this is available, we will convert all Redis counters into Snowplow events. +# Redis HLL counter +counter = Gitlab::UsageDataCounters::TrackUniqueActions +redis_usage_data do + counter.count_unique_events( + event_action: Gitlab::UsageDataCounters::TrackUniqueActions::PUSH_ACTION, + date_from: time_period[:created_at].first, + date_to: time_period[:created_at].last + ) +``` ### Alternative Counters @@ -313,14 +348,16 @@ In order to have an understanding of the query's execution we add in the MR desc We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). -Examples of query optimization work: +#### Optimization recommendations and examples -- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) -- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) +- Use specialized indexes [example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871), [example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445). +- Use defined `start` and `finish`, and simple queries, because these values can be memoized and reused, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). +- Avoid joins and write the queries as simply as possible, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). +- Set a custom `batch_size` for `distinct_count`, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). ### 4. Add the metric definition -When adding, changing, or updating metrics, please update the [Usage Statistics definition table](#usage-statistics-definitions). +When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](event_dictionary.md). ### 5. Add new metric to Versions Application @@ -340,6 +377,10 @@ Ensure you comply with the [Changelog entries guide](../changelog.md). On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Mention `@gitlab-org/growth/telemetry/engineers` in your MR for a review. +### 9. Verify your metric + +On GitLab.com, the Product Analytics team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "Saas" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. + ### Optional: Test Prometheus based Usage Ping If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, @@ -386,358 +427,6 @@ with any of the other services that are running. That is not how node metrics ar always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. -## Usage Statistics definitions - -| Statistic | Section | Stage | Tier | Edition | Description | -| --------------------------------------------------------- | ------------------------------------ | ------------- | ---------------- | ------- | -------------------------------------------------------------------------- | -| `uuid` | | | | | | -| `hostname` | | | | | | -| `version` | | | | | | -| `installation_type` | | | | | | -| `active_user_count` | | | | | | -| `recorded_at` | | | | | | -| `recording_ce_finished_at` | | | | CE+EE | When the core features were computed | -| `recording_ee_finished_at` | | | | EE | When the EE-specific features were computed | -| `edition` | | | | | | -| `license_md5` | | | | | | -| `license_id` | | | | | | -| `historical_max_users` | | | | | | -| `Name` | `licensee` | | | | | -| `Email` | `licensee` | | | | | -| `Company` | `licensee` | | | | | -| `license_user_count` | | | | | | -| `license_starts_at` | | | | | | -| `license_expires_at` | | | | | | -| `license_plan` | | | | | | -| `license_trial` | | | | | | -| `assignee_lists` | `counts` | | | | | -| `boards` | `counts` | | | | | -| `ci_builds` | `counts` | `verify` | | | Unique builds in project | -| `ci_internal_pipelines` | `counts` | `verify` | | | Total pipelines in GitLab repositories | -| `ci_external_pipelines` | `counts` | `verify` | | | Total pipelines in external repositories | -| `ci_pipeline_config_auto_devops` | `counts` | `verify` | | | Total pipelines from an Auto DevOps template | -| `ci_pipeline_config_repository` | `counts` | `verify` | | | Total Pipelines from templates in repository | -| `ci_runners` | `counts` | `verify` | | | Total configured Runners in project | -| `ci_triggers` | `counts` | `verify` | | | Total configured Triggers in project | -| `ci_pipeline_schedules` | `counts` | `verify` | | | Pipeline schedules in GitLab | -| `auto_devops_enabled` | `counts` | `configure` | | | Projects with Auto DevOps template enabled | -| `auto_devops_disabled` | `counts` | `configure` | | | Projects with Auto DevOps template disabled | -| `deploy_keys` | `counts` | | | | | -| `deployments` | `counts` | `release` | | | Total deployments | -| `deployments` | `counts_monthly` | `release` | | | Total deployments last 28 days | -| `dast_jobs` | `counts` | | | | | -| `successful_deployments` | `counts` | `release` | | | Total successful deployments | -| `successful_deployments` | `counts_monthly` | `release` | | | Total successful deployments last 28 days | -| `failed_deployments` | `counts` | `release` | | | Total failed deployments | -| `failed_deployments` | `counts_monthly` | `release` | | | Total failed deployments last 28 days | -| `environments` | `counts` | `release` | | | Total available and stopped environments | -| `clusters` | `counts` | `configure` | | | Total GitLab Managed clusters both enabled and disabled | -| `clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters currently enabled | -| `project_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to projects | -| `group_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to groups | -| `instance_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to the instance | -| `clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters | -| `project_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to projects | -| `group_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to groups | -| `instance_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to the instance | -| `clusters_platforms_eks` | `counts` | `configure` | | | Total GitLab Managed clusters provisioned with GitLab on AWS EKS | -| `clusters_platforms_gke` | `counts` | `configure` | | | Total GitLab Managed clusters provisioned with GitLab on GCE GKE | -| `clusters_platforms_user` | `counts` | `configure` | | | Total GitLab Managed clusters that are user provisioned | -| `clusters_applications_helm` | `counts` | `configure` | | | Total GitLab Managed clusters with Helm enabled | -| `clusters_applications_ingress` | `counts` | `configure` | | | Total GitLab Managed clusters with Ingress enabled | -| `clusters_applications_cert_managers` | `counts` | `configure` | | | Total GitLab Managed clusters with Cert Manager enabled | -| `clusters_applications_crossplane` | `counts` | `configure` | | | Total GitLab Managed clusters with Crossplane enabled | -| `clusters_applications_prometheus` | `counts` | `configure` | | | Total GitLab Managed clusters with Prometheus enabled | -| `clusters_applications_runner` | `counts` | `configure` | | | Total GitLab Managed clusters with Runner enabled | -| `clusters_applications_knative` | `counts` | `configure` | | | Total GitLab Managed clusters with Knative enabled | -| `clusters_applications_elastic_stack` | `counts` | `configure` | | | Total GitLab Managed clusters with Elastic Stack enabled | -| `clusters_applications_cilium` | `counts` | `configure` | | | Total GitLab Managed clusters with Cilium enabled | -| `clusters_management_project` | `counts` | `configure` | | | Total GitLab Managed clusters with defined cluster management project | -| `in_review_folder` | `counts` | | | | | -| `grafana_integrated_projects` | `counts` | | | | | -| `groups` | `counts` | | | | | -| `issues` | `counts` | | | | | -| `issues_created_from_gitlab_error_tracking_ui` | `counts` | `monitor` | | | | -| `issues_with_associated_zoom_link` | `counts` | `monitor` | | | | -| `issues_using_zoom_quick_actions` | `counts` | `monitor` | | | | -| `issues_with_embedded_grafana_charts_approx` | `counts` | `monitor` | | | | -| `issues_with_health_status` | `counts` | | | | | -| `keys` | `counts` | | | | | -| `label_lists` | `counts` | | | | | -| `lfs_objects` | `counts` | | | | | -| `milestone_lists` | `counts` | | | | | -| `milestones` | `counts` | | | | | -| `pages_domains` | `counts` | `release` | | | Total GitLab Pages domains | -| `pool_repositories` | `counts` | | | | | -| `projects` | `counts` | | | | | -| `projects_imported_from_github` | `counts` | | | | | -| `projects_with_repositories_enabled` | `counts` | | | | | -| `projects_with_error_tracking_enabled` | `counts` | `monitor` | | | | -| `protected_branches` | `counts` | | | | | -| `releases` | `counts` | `release` | | | Unique release tags | -| `remote_mirrors` | `counts` | | | | | -| `requirements_created` | `counts` | | | | | -| `snippets` | `counts` | 'create' | | CE+EE | | -| `snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `personal_snippets` | `counts` | 'create' | | CE+EE | | -| `personal_snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `project_snippets` | `counts` | 'create' | | CE+EE | | -| `project_snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `suggestions` | `counts` | | | | | -| `todos` | `counts` | | | | | -| `uploads` | `counts` | | | | | -| `web_hooks` | `counts` | | | | | -| `projects_alerts_active` | `counts` | | | | | -| `projects_asana_active` | `counts` | | | | | -| `projects_assembla_active` | `counts` | | | | | -| `projects_bamboo_active` | `counts` | | | | | -| `projects_bugzilla_active` | `counts` | | | | | -| `projects_buildkite_active` | `counts` | | | | | -| `projects_campfire_active` | `counts` | | | | | -| `projects_custom_issue_tracker_active` | `counts` | | | | | -| `projects_discord_active` | `counts` | | | | | -| `projects_drone_ci_active` | `counts` | | | | | -| `projects_emails_on_push_active` | `counts` | | | | | -| `projects_external_wiki_active` | `counts` | | | | | -| `projects_flowdock_active` | `counts` | | | | | -| `projects_github_active` | `counts` | | | | | -| `projects_hangouts_chat_active` | `counts` | | | | | -| `projects_hipchat_active` | `counts` | | | | | -| `projects_irker_active` | `counts` | | | | | -| `projects_jenkins_active` | `counts` | | | | | -| `projects_jira_active` | `counts` | | | | | -| `projects_mattermost_active` | `counts` | | | | | -| `projects_mattermost_slash_commands_active` | `counts` | | | | | -| `projects_microsoft_teams_active` | `counts` | | | | | -| `projects_packagist_active` | `counts` | | | | | -| `projects_pipelines_email_active` | `counts` | | | | | -| `projects_pivotaltracker_active` | `counts` | | | | | -| `projects_prometheus_active` | `counts` | | | | | -| `projects_pushover_active` | `counts` | | | | | -| `projects_redmine_active` | `counts` | | | | | -| `projects_slack_active` | `counts` | | | | | -| `projects_slack_slash_commands_active` | `counts` | | | | | -| `projects_teamcity_active` | `counts` | | | | | -| `projects_unify_circuit_active` | `counts` | | | | | -| `projects_webex_teams_active` | `counts` | | | | | -| `projects_youtrack_active` | `counts` | | | | | -| `projects_jira_server_active` | `counts` | | | | | -| `projects_jira_cloud_active` | `counts` | | | | | -| `projects_jira_dvcs_cloud_active` | `counts` | | | | | -| `projects_jira_dvcs_server_active` | `counts` | | | | | -| `projects_jira_issuelist_active` | `counts` | `create` | | EE | Total Jira Issue feature enabled | -| `labels` | `counts` | | | | | -| `merge_requests` | `counts` | | | | | -| `merge_requests_users` | `counts` | | | | | -| `notes` | `counts` | | | | | -| `wiki_pages_create` | `counts` | | | | | -| `wiki_pages_update` | `counts` | | | | | -| `wiki_pages_delete` | `counts` | | | | | -| `web_ide_commits` | `counts` | | | | | -| `web_ide_views` | `counts` | | | | | -| `web_ide_merge_requests` | `counts` | | | | | -| `web_ide_previews` | `counts` | | | | | -| `snippet_comment` | `counts` | | | | | -| `commit_comment` | `counts` | | | | | -| `merge_request_comment` | `counts` | | | | | -| `snippet_create` | `counts` | | | | | -| `snippet_update` | `counts` | | | | | -| `navbar_searches` | `counts` | | | | | -| `cycle_analytics_views` | `counts` | | | | | -| `productivity_analytics_views` | `counts` | | | | | -| `source_code_pushes` | `counts` | | | | | -| `merge_request_create` | `counts` | | | | | -| `design_management_designs_create` | `counts` | | | | | -| `design_management_designs_update` | `counts` | | | | | -| `design_management_designs_delete` | `counts` | | | | | -| `licenses_list_views` | `counts` | | | | | -| `user_preferences_group_overview_details` | `counts` | | | | | -| `user_preferences_group_overview_security_dashboard` | `counts` | | | | | -| `ingress_modsecurity_logging` | `counts` | | | | | -| `ingress_modsecurity_blocking` | `counts` | | | | | -| `ingress_modsecurity_disabled` | `counts` | | | | | -| `ingress_modsecurity_not_installed` | `counts` | | | | | -| `dependency_list_usages_total` | `counts` | | | | | -| `epics` | `counts` | | | | | -| `feature_flags` | `counts` | | | | | -| `geo_nodes` | `counts` | `geo` | | | Number of sites in a Geo deployment | -| `geo_event_log_max_id` | `counts` | `geo` | | | Number of replication events on a Geo primary | -| `incident_issues` | `counts` | `monitor` | | | Issues created by the alert bot | -| `alert_bot_incident_issues` | `counts` | `monitor` | | | Issues created by the alert bot | -| `incident_labeled_issues` | `counts` | `monitor` | | | Issues with the incident label | -| `issues_created_gitlab_alerts` | `counts` | `monitor` | | | Issues created from alerts by non-alert bot users | -| `issues_created_manually_from_alerts` | `counts` | `monitor` | | | Issues created from alerts by non-alert bot users | -| `issues_created_from_alerts` | `counts` | `monitor` | | | Issues created from Prometheus and alert management alerts | -| `ldap_group_links` | `counts` | | | | | -| `ldap_keys` | `counts` | | | | | -| `ldap_users` | `counts` | | | | | -| `pod_logs_usages_total` | `counts` | | | | | -| `projects_enforcing_code_owner_approval` | `counts` | | | | | -| `projects_mirrored_with_pipelines_enabled` | `counts` | `release` | | | Projects with repository mirroring enabled | -| `projects_reporting_ci_cd_back_to_github` | `counts` | `verify` | | | Projects with a GitHub service pipeline enabled | -| `projects_with_packages` | `counts` | `package` | | | Projects with package registry configured | -| `projects_with_prometheus_alerts` | `counts` | `monitor` | | | Projects with Prometheus alerting enabled | -| `projects_with_tracing_enabled` | `counts` | `monitor` | | | Projects with tracing enabled | -| `projects_with_alerts_service_enabled` | `counts` | `monitor` | | | Projects with alerting service enabled | -| `template_repositories` | `counts` | | | | | -| `container_scanning_jobs` | `counts` | | | | | -| `dependency_scanning_jobs` | `counts` | | | | | -| `license_management_jobs` | `counts` | | | | | -| `sast_jobs` | `counts` | | | | | -| `status_page_projects` | `counts` | `monitor` | | | Projects with status page enabled | -| `status_page_issues` | `counts` | `monitor` | | | Issues published to a Status Page | -| `status_page_incident_publishes` | `counts` | `monitor` | | | Cumulative count of usages of publish operation | -| `status_page_incident_unpublishes` | `counts` | `monitor` | | | Cumulative count of usages of unpublish operation | -| `epics_deepest_relationship_level` | `counts` | | | | | -| `operations_dashboard_default_dashboard` | `counts` | `monitor` | | | Active users with enabled operations dashboard | -| `operations_dashboard_users_with_projects_added` | `counts` | `monitor` | | | Active users with projects on operations dashboard | -| `container_registry_enabled` | | | | | | -| `dependency_proxy_enabled` | | | | | | -| `gitlab_shared_runners_enabled` | | | | | | -| `gravatar_enabled` | | | | | | -| `ldap_enabled` | | | | | | -| `mattermost_enabled` | | | | | | -| `omniauth_enabled` | | | | | | -| `prometheus_enabled` | | | | | Whether the bundled Prometheus is enabled | -| `prometheus_metrics_enabled` | | | | | | -| `reply_by_email_enabled` | | | | | | -| `average` | `avg_cycle_analytics - code` | | | | | -| `sd` | `avg_cycle_analytics - code` | | | | | -| `missing` | `avg_cycle_analytics - code` | | | | | -| `average` | `avg_cycle_analytics - test` | | | | | -| `sd` | `avg_cycle_analytics - test` | | | | | -| `missing` | `avg_cycle_analytics - test` | | | | | -| `average` | `avg_cycle_analytics - review` | | | | | -| `sd` | `avg_cycle_analytics - review` | | | | | -| `missing` | `avg_cycle_analytics - review` | | | | | -| `average` | `avg_cycle_analytics - staging` | | | | | -| `sd` | `avg_cycle_analytics - staging` | | | | | -| `missing` | `avg_cycle_analytics - staging` | | | | | -| `average` | `avg_cycle_analytics - production` | | | | | -| `sd` | `avg_cycle_analytics - production` | | | | | -| `missing` | `avg_cycle_analytics - production` | | | | | -| `total` | `avg_cycle_analytics` | | | | | -| `g_analytics_contribution` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/contribution_analytics | -| `g_analytics_insights` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/insights | -| `g_analytics_issues` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/issues_analytics | -| `g_analytics_productivity` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/analytics/productivity_analytics | -| `g_analytics_valuestream` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/analytics/value_stream_analytics | -| `p_analytics_pipelines` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/pipelines/charts | -| `p_analytics_code_reviews` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/analytics/code_reviews | -| `p_analytics_valuestream` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/value_stream_analytics | -| `p_analytics_insights` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/insights | -| `p_analytics_issues` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/analytics/issues_analytics | -| `p_analytics_repo` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/graphs/master/charts | -| `u_analytics_todos` | `analytics_unique_visits` | `manage` | | | Visits to /dashboard/todos | -| `i_analytics_cohorts` | `analytics_unique_visits` | `manage` | | | Visits to /-/instance_statistics/cohorts | -| `i_analytics_dev_ops_score` | `analytics_unique_visits` | `manage` | | | Visits to /-/instance_statistics/dev_ops_score | -| `analytics_unique_visits_for_any_target` | `analytics_unique_visits` | `manage` | | | Visits to any of the pages listed above | -| `clusters_applications_cert_managers` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with certificate managers enabled | -| `clusters_applications_helm` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Helm enabled | -| `clusters_applications_ingress` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Ingress enabled | -| `clusters_applications_knative` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Knative enabled | -| `clusters_management_project` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with project management enabled | -| `clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Total non-"GitLab Managed clusters" | -| `clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Total GitLab Managed clusters | -| `clusters_platforms_gke` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Google Cloud installed | -| `clusters_platforms_eks` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with AWS installed | -| `clusters_platforms_user` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters that are user provided | -| `instance_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on instance | -| `instance_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on instance | -| `group_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on group | -| `group_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on group | -| `project_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on project | -| `project_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on project | -| `projects_slack_notifications_active` | `usage_activity_by_stage` | `configure` | | EE | Unique projects with Slack service enabled | -| `projects_slack_slash_active` | `usage_activity_by_stage` | `configure` | | EE | Unique projects with Slack '/' commands enabled | -| `projects_with_prometheus_alerts` | `usage_activity_by_stage` | `configure` | | EE | Projects with Prometheus enabled and no alerts | -| `deploy_keys` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `keys` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `merge_requests` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `projects_with_disable_overriding_approvers_per_merge_request` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `projects_without_disable_overriding_approvers_per_merge_request` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `remote_mirrors` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `snippets` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `merge_requests_users` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who used a merge request | -| `action_monthly_active_users_project_repo` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who pushed to a project repo | -| `action_monthly_active_users_design_management` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who interacted with the design system management | -| `action_monthly_active_users_wiki_repo` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who created or updated a wiki repo | -| `projects_enforcing_code_owner_approval` | `usage_activity_by_stage` | `create` | | EE | | -| `merge_requests_with_optional_codeowners` | `usage_activity_by_stage` | `create` | | EE | | -| `merge_requests_with_required_codeowners` | `usage_activity_by_stage` | `create` | | EE | | -| `projects_imported_from_github` | `usage_activity_by_stage` | `create` | | EE | | -| `projects_with_repositories_enabled` | `usage_activity_by_stage` | `create` | | EE | | -| `protected_branches` | `usage_activity_by_stage` | `create` | | EE | | -| `suggestions` | `usage_activity_by_stage` | `create` | | EE | | -| `approval_project_rules` | `usage_activity_by_stage` | `create` | | EE | Number of project approval rules | -| `approval_project_rules_with_target_branch` | `usage_activity_by_stage` | `create` | | EE | Number of project approval rules with not default target branch | -| `merge_requests_with_added_rules` | `usage_activity_by_stage` | `create` | | EE | Merge Requests with added rules | -| `clusters` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `clusters_applications_prometheus` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `operations_dashboard_default_dashboard` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `operations_dashboard_users_with_projects_added` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_prometheus_active` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_with_error_tracking_enabled` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_with_tracing_enabled` | `usage_activity_by_stage` | `monitor` | | EE | | -| `events` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `groups` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `users_created_at` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `omniauth_providers` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `ldap_keys` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_users` | `usage_activity_by_stage` | `manage` | | EE | | -| `value_stream_management_customized_group_stages` | `usage_activity_by_stage` | `manage` | | EE | | -| `projects_with_compliance_framework` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_servers` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_group_sync_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_admin_sync_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `group_saml_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `issues` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `notes` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `projects` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `todos` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `assignee_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `epics` | `usage_activity_by_stage` | `plan` | | EE | | -| `label_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `milestone_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_dvcs_server_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_dvcs_server_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `service_desk_enabled_projects` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `service_desk_issues` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total deployments | -| `failed_deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total failed deployments | -| `projects_mirrored_with_pipelines_enabled` | `usage_activity_by_stage` | `release` | | EE | Projects with repository mirroring enabled | -| `releases` | `usage_activity_by_stage` | `release` | | CE+EE | Unique release tags in project | -| `successful_deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total successful deployments | -| `user_preferences_group_overview_security_dashboard` | `usage_activity_by_stage` | `secure` | | | | -| `ci_builds` | `usage_activity_by_stage` | `verify` | | CE+EE | Unique builds in project | -| `ci_external_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines in external repositories | -| `ci_internal_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines in GitLab repositories | -| `ci_pipeline_config_auto_devops` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines from an Auto DevOps template | -| `ci_pipeline_config_repository` | `usage_activity_by_stage` | `verify` | | CE+EE | Pipelines from templates in repository | -| `ci_pipeline_schedules` | `usage_activity_by_stage` | `verify` | | CE+EE | Pipeline schedules in GitLab | -| `ci_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines | -| `ci_triggers` | `usage_activity_by_stage` | `verify` | | CE+EE | Triggers enabled | -| `clusters_applications_runner` | `usage_activity_by_stage` | `verify` | | CE+EE | Unique clusters with Runner enabled | -| `projects_reporting_ci_cd_back_to_github` | `usage_activity_by_stage` | `verify` | | EE | Unique projects with a GitHub pipeline enabled | -| `merge_requests_users` | `usage_activity_by_stage_monthly` | `create` | | | Unique count of users who used a merge request | -| `duration_s` | `topology` | `enablement` | | | Time it took to collect topology data | -| `application_requests_per_hour` | `topology` | `enablement` | | | Number of requests to the web application per hour | -| `failures` | `topology` | `enablement` | | | Contains information about failed queries | -| `nodes` | `topology` | `enablement` | | | The list of server nodes on which GitLab components are running | -| `node_memory_total_bytes` | `topology > nodes` | `enablement` | | | The total available memory of this node | -| `node_cpus` | `topology > nodes` | `enablement` | | | The number of CPU cores of this node | -| `node_uname_info` | `topology > nodes` | `enablement` | | | The basic hardware architecture and OS release information on this node | -| `node_services` | `topology > nodes` | `enablement` | | | The list of GitLab services running on this node | -| `name` | `topology > nodes > node_services` | `enablement` | | | The name of the GitLab service running on this node | -| `process_count` | `topology > nodes > node_services` | `enablement` | | | The number of processes running for this service | -| `process_memory_rss` | `topology > nodes > node_services` | `enablement` | | | The average Resident Set Size of a service process | -| `process_memory_uss` | `topology > nodes > node_services` | `enablement` | | | The average Unique Set Size of a service process | -| `process_memory_pss` | `topology > nodes > node_services` | `enablement` | | | The average Proportional Set Size of a service process | -| `server` | `topology > nodes > node_services` | `enablement` | | | The type of web server used (Unicorn or Puma) | -| `network_policy_forwards` | `counts` | `defend` | | EE | Cumulative count of forwarded packets by Container Network | -| `network_policy_drops` | `counts` | `defend` | | EE | Cumulative count of dropped packets by Container Network | - ## Example Usage Ping payload The following is example content of the Usage Ping payload. @@ -811,13 +500,14 @@ The following is example content of the Usage Ping payload. "enabled": true, "version": "1.17.0" }, + "container_registry_server": { + "vendor": "gitlab", + "version": "2.9.1-gitlab" + }, "database": { "adapter": "postgresql", "version": "9.6.15" }, - "app_server": { - "type": "console" - }, "avg_cycle_analytics": { "issue": { "average": 999, @@ -941,7 +631,9 @@ The following is example content of the Usage Ping payload. "nodes": [ { "node_memory_total_bytes": 33269903360, + "node_memory_utilization": 0.35, "node_cpus": 16, + "node_cpu_utilization": 0.2, "node_uname_info": { "machine": "x86_64", "sysname": "Linux", |