diff options
Diffstat (limited to 'doc/development/internal_analytics/service_ping')
9 files changed, 141 insertions, 1848 deletions
diff --git a/doc/development/internal_analytics/service_ping/implement.md b/doc/development/internal_analytics/service_ping/implement.md index c6da26f86c2..8a85a310a9e 100644 --- a/doc/development/internal_analytics/service_ping/implement.md +++ b/doc/development/internal_analytics/service_ping/implement.md @@ -1,850 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../metrics/metrics_instrumentation.md' +remove_date: '2024-01-13' --- -# Implement Service Ping +This document was moved to [another location](../metrics/metrics_instrumentation.md). -Service Ping consists of two kinds of data: - -- **Counters**: Track how often a certain event happened over time, such as how many CI/CD pipelines have run. - They are monotonic and usually trend up. -- **Observations**: Facts collected from one or more GitLab instances and can carry arbitrary data. - There are no general guidelines for how to collect those, due to the individual nature of that data. - -To implement a new metric in Service Ping, follow these steps: - -1. [Implement the required counter](#types-of-counters) -1. [Name and place the metric](metrics_dictionary.md#metric-key_path) -1. [Test counters manually using your Rails console](#test-counters-manually-using-your-rails-console) -1. [Generate the SQL query](#generate-the-sql-query) -1. [Optimize queries with Database Lab](#optimize-queries-with-database-lab) -1. [Add the metric definition to the Metrics Dictionary](#add-the-metric-definition) -1. [Create a merge request](#create-a-merge-request) -1. [Verify your metric](#verify-your-metric) -1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) - -## Instrumentation classes - -NOTE: -Implementing metrics directly in `usage_data.rb` is deprecated. -When you add or change a Service Ping Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md). -For information about the progress on migrating Service Ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547). - -For example, we have the following instrumentation class: -`lib/gitlab/usage/metrics/instrumentations/count_boards_metric.rb`. - -You should add it to `usage_data.rb` as follows: - -```ruby -boards: add_metric('CountBoardsMetric', time_frame: 'all'), -``` - -## Types of counters - -There are several types of counters for metrics: - -- **[Batch counters](#batch-counters)**: Used for counts, sums, and averages. -- **[Redis counters](#redis-counters):** Used for in-memory counts. -- **[Alternative counters](#alternative-counters):** Used for settings and configurations. - -NOTE: -Only use the provided counter methods. Each counter method contains a built-in fail-safe mechanism that isolates each counter to avoid breaking the entire Service Ping process. - -### Batch counters - -For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. - -For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: - -| Table | Row counts in millions | -|------------------------------|------------------------| -| `merge_request_diff_commits` | 2280 | -| `ci_build_trace_sections` | 1764 | -| `merge_request_diff_files` | 1082 | -| `events` | 514 | - -Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, -you must add a specialized index on the columns involved in a counter. - -#### Ordinary batch counters - -Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `count` operation for a given `ActiveRecord_Relation` - -Method: - -```ruby -add_metric('CountIssuesMetric', time_frame: 'all') -``` - -Examples: - -Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md). - -#### Distinct batch counters - -Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `distinct_count` operation for a given `ActiveRecord_Relation`. - -Method: - -```ruby -add_metric('CountUsersAssociatingMilestonesToReleasesMetric', time_frame: 'all') -``` - -WARNING: -Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../../database/iterating_tables_in_batches.md) guide. - -Examples: - -Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md). - -#### Sum batch operation - -Sum the values of a given ActiveRecord_Relation on given column and handles errors. -Handles the `ActiveRecord::StatementInvalid` error - -Method: - -```ruby -add_metric('JiraImportsTotalImportedIssuesCountMetric') -``` - -#### Average batch operation - -Average the values of a given `ActiveRecord_Relation` on given column and handles errors. - -Method: - -```ruby -add_metric('CountIssuesWeightAverageMetric') -``` - -Examples: - -Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md). - -#### Grouping and batch operations - -The `count`, `distinct_count` and `sum` batch counters can accept an `ActiveRecord::Relation` -object, which groups by a specified column. With a grouped relation, the methods do batch counting, -handle errors, and returns a hash table of key-value pairs. - -Examples: - -```ruby -count(Namespace.group(:type)) -# returns => {nil=>179, "Group"=>54} - -distinct_count(Project.group(:visibility_level), :creator_id) -# returns => {0=>1, 10=>1, 20=>11} - -sum(Issue.group(:state_id), :weight)) -# returns => {1=>3542, 2=>6820} -``` - -#### Add operation - -Sum the values given as parameters. Handles the `StandardError`. -Returns `-1` if any of the arguments are `-1`. - -Method: - -```ruby -add(*args) -``` - -Examples: - -```ruby -project_imports = distinct_count(::Project.where.not(import_type: nil), :creator_id) -bulk_imports = distinct_count(::BulkImport, :user_id) - - add(project_imports, bulk_imports) -``` - -#### Estimated batch counters - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7. - -Estimated batch counter functionality handles `ActiveRecord::StatementInvalid` errors -when used through the provided `estimate_batch_distinct_count` method. -Errors return a value of `-1`. - -WARNING: -This functionality estimates a distinct count of a specific ActiveRecord_Relation in a given column, -which uses the [HyperLogLog](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf) algorithm. -As the HyperLogLog algorithm is probabilistic, the **results always include error**. -The highest encountered error rate is 4.9%. - -When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over -columns that contain non-unique values, which cannot be assured by other counters. - -##### `estimate_batch_distinct_count` method - -Method: - -```ruby -estimate_batch_distinct_count(relation, column = nil, batch_size: nil, start: nil, finish: nil) -``` - -The [method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/usage_data.rb#L63) -includes the following arguments: - -- `relation`: The ActiveRecord_Relation to perform the count. -- `column`: The column to perform the distinct count. The default is the primary key. -- `batch_size`: From `Gitlab::Database::PostgresHll::BatchDistinctCounter::DEFAULT_BATCH_SIZE`. Default value: 10,000. -- `start`: The custom start of the batch count, to avoid complex minimum calculations. -- `finish`: The custom end of the batch count to avoid complex maximum calculations. - -The method includes the following prerequisites: - -- The supplied `relation` must include the primary key defined as the numeric column. - For example: `id bigint NOT NULL`. -- The `estimate_batch_distinct_count` can handle a joined relation. To use its ability to - count non-unique columns, the joined relation **must not** have a one-to-many relationship, - such as `has_many :boards`. -- Both `start` and `finish` arguments should always represent primary key relationship values, - even if the estimated count refers to another column, for example: - - ```ruby - estimate_batch_distinct_count(::Note, :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) - ``` - -Examples: - -1. Simple execution of estimated batch counter, with only relation provided, - returned value represents estimated number of unique values in `id` column - (which is the primary key) of `Project` relation: - - ```ruby - estimate_batch_distinct_count(::Project) - ``` - -1. Execution of estimated batch counter, where provided relation has applied - additional filter (`.where(time_period)`), number of unique values estimated - in custom column (`:author_id`), and parameters: `start` and `finish` together - apply boundaries that defines range of provided relation to analyze: - - ```ruby - estimate_batch_distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) - ``` - -When instrumenting metric with usage of estimated batch counter please add -`_estimated` suffix to its name, for example: - -```ruby - "counts": { - "ci_builds_estimated": estimate_batch_distinct_count(Ci::Build), - ... -``` - -### Redis counters - -Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent`. -Returns -1 when a block is sent or hash with all values and -1 when a `counter(Gitlab::UsageDataCounters)` is sent. -The different behavior is due to 2 different implementations of the Redis counter. - -Method: - -```ruby -redis_usage_data(counter, &block) -``` - -Arguments: - -- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented -- or a `block`: which is evaluated - -#### Ordinary Redis counters - -Example of implementation: [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb), using Redis methods [`INCR`](https://redis.io/commands/incr/) and [`GET`](https://redis.io/commands/get/). - -Events are handled by counter classes in the `Gitlab::UsageDataCounters` namespace, inheriting from `BaseCounter`, that are either: - -1. Listed in [`Gitlab::UsageDataCounters::COUNTERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters.rb#L5) to be then included in `Gitlab::UsageData`. - -1. Specified in the metric definition using the `RedisMetric` instrumentation class by their `prefix` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation. - -Inheriting classes are expected to override `KNOWN_EVENTS` and `PREFIX` constants to build event names and associated metrics. For example, for prefix `issues` and events array `%w[create, update, delete]`, three metrics will be added to the Service Ping payload: `counts.issues_create`, `counts.issues_update` and `counts.issues_delete`. - -##### `UsageData` API - -You can use the `UsageData` API to track events. -To track events, the `usage_data_api` feature flag must -be enabled (set to `default_enabled: true`). -Enabled by default in GitLab 13.7 and later. - -##### UsageData API tracking - -1. Track events using the [`UsageData` API](#usagedata-api). - - Increment event count using an ordinary Redis counter, for a given event name. - - API requests are protected by checking for a valid CSRF token. - - ```plaintext - POST /usage_data/increment_counter - ``` - - | Attribute | Type | Required | Description | - | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name to track. | - - Response: - - - `200` if the event was tracked. - - `400 Bad request` if the event parameter is missing. - - `401 Unauthorized` if the user is not authenticated. - - `403 Forbidden` if an invalid CSRF token is provided. - -1. Track events using the JavaScript/Vue API helper which calls the [`UsageData` API](#usagedata-api). - - To track events, `usage_data_api` and `usage_data_#{event_name}` must be enabled. - - ```javascript - import api from '~/api'; - - api.trackRedisCounterEvent('my_already_defined_event_name'), - ``` - -#### Redis HLL counters - -WARNING: -HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount/), data from -used HLL implementation is "approximated with a standard error of 0.81%". - -NOTE: - A user's consent for `usage_stats` (`User.single_user&.requires_usage_stats_consent?`) is not checked during the data tracking stage due to performance reasons. Keys corresponding to those counters are present in Redis even if `usage_stats_consent` is still required. However, no metric is collected from Redis and reported back to GitLab as long as `usage_stats_consent` is required. - -With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. - -Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [PFCOUNT](https://redis.io/commands/pfcount/). - -##### Add new events - -1. Add an event to the required metric ([see example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20230210200054_i_ee_code_review_merge_request_widget_license_compliance_expand_weekly.yml#L17-17)) or create a metric. - -1. Use one of the following methods to track the event: - - - In the controller using the `ProductAnalyticsTracking` module and the following format: - - ```ruby - track_event(*controller_actions, name:, action:, label:, conditions: nil, destinations: [:redis_hll], &block) - ``` - - Arguments: - - - `controller_actions`: the controller actions to track. - - `name`: the event name. - - `action`: required if destination is `:snowplow. Action name for the triggered event. See [event schema](../snowplow/index.md#event-schema) for more details. - - `label`: required if destination is `:snowplow. Label for the triggered event. See [event schema](../snowplow/index.md#event-schema) for more details. - - `conditions`: optional custom conditions. Uses the same format as Rails callbacks. - - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: `:redis_hll`. - - `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`. - - Example: - - ```ruby - # controller - class ProjectsController < Projects::ApplicationController - include ProductAnalyticsTracking - - skip_before_action :authenticate_user!, only: :show - track_event :index, :show, - name: 'users_visiting_projects', - action: 'user_perform_visit', - label: 'redis_hll_counters.users_visiting_project_monthly', - destinations: %i[redis_hll snowplow] - - def index - render html: 'index' - end - - def new - render html: 'new' - end - - def show - render html: 'show' - end - end - ``` - - - In the API using the `increment_unique_values(event_name, values)` helper method. - - Arguments: - - - `event_name`: the event name. - - `values`: the values counted. Can be one value or an array of values. - - Example: - - ```ruby - get ':id/registry/repositories' do - repositories = ContainerRepositoriesFinder.new( - user: current_user, subject: user_group - ).execute - - increment_unique_values('users_listing_repositories', current_user.id) - - present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] - end - ``` - - - Using `track_usage_event(event_name, values)` in services and GraphQL. - - Increment unique values count using Redis HLL, for a given event name. - - Examples: - - - [Track usage event for an incident in a service](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/services/issues/update_service.rb#L66) - - [Track usage event for an incident in GraphQL](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/graphql/mutations/alert_management/update_alert_status.rb#L16) - - ```ruby - track_usage_event(:incident_management_incident_created, current_user.id) - ``` - - - Using the [`UsageData` API](#usagedata-api). - - Increment unique users count using Redis HLL, for a given event name. - - API requests are protected by checking for a valid CSRF token. - - ```plaintext - POST /usage_data/increment_unique_users - ``` - - | Attribute | Type | Required | Description | - | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name to track | - - Response: - - - `200` if the event was tracked, or if tracking failed for any reason. - - `400 Bad request` if an event parameter is missing. - - `401 Unauthorized` if the user is not authenticated. - - `403 Forbidden` if an invalid CSRF token is provided. - - - Using the JavaScript/Vue API helper, which calls the [`UsageData` API](#usagedata-api). - - Example for an existing event already defined in [metric fields](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20220407125907_p_ci_templates_themekit_monthly.yml#L17-17): - - ```javascript - import api from '~/api'; - - api.trackRedisHllUserEvent('my_already_defined_event_name'), - ``` - -1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date:, context: '')`. - - Arguments: - - - `event_names`: the list of event names. - - `start_date`: start date of the period for which we want to get event data. - - `end_date`: end date of the period for which we want to get event data. - - `context`: context of the event. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate`. - -1. Testing tracking and getting unique events - -Trigger events in rails console by using `track_event` method - - ```ruby - Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: 1) - Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: [2, 3]) - ``` - -Next, get the unique events for the current week. - - ```ruby - # Get unique events for metric for current_week - Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_viewing_compliance_audit_events', - start_date: Date.current.beginning_of_week, end_date: Date.current.next_week) - ``` - -##### Recommendations - -We have the following recommendations for [adding new events](#add-new-events): - -- When adding new metrics, use a [feature flag](../../../operations/feature_flags.md) to control the impact. -It's recommended to disable the new feature flag by default (set `default_enabled: false`). -- Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change - -##### Enable or disable Redis HLL tracking - -We can disable tracking completely by using the global flag: - -```shell -/chatops run feature set redis_hll_tracking true -/chatops run feature set redis_hll_tracking false -``` - -##### Known events are added automatically in Service Data payload - -Service Ping adds all events to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/db/schema.rb#L213). -For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - -- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. -- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. - -Example of `redis_hll_counters` data: - -```ruby -{:redis_hll_counters=> - {"compliance"=> - {"users_viewing_compliance_dashboard_weekly"=>0, - "users_viewing_compliance_dashboard_monthly"=>0, - "users_viewing_compliance_audit_events_weekly"=>0, - "users_viewing_audit_events_monthly"=>0, - "compliance_total_unique_counts_weekly"=>0, - "compliance_total_unique_counts_monthly"=>0}, - "analytics"=> - {"users_viewing_analytics_group_devops_adoption_weekly"=>0, - "users_viewing_analytics_group_devops_adoption_monthly"=>0, - "analytics_total_unique_counts_weekly"=>0, - "analytics_total_unique_counts_monthly"=>0}, - "ide_edit"=> - {"users_editing_by_web_ide_weekly"=>0, - "users_editing_by_web_ide_monthly"=>0, - "users_editing_by_sfe_weekly"=>0, - "users_editing_by_sfe_monthly"=>0, - "ide_edit_total_unique_counts_weekly"=>0, - "ide_edit_total_unique_counts_monthly"=>0} - } -} -``` - -Example: - -```ruby -# Redis Counters -redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) - -# Tracking events -Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_expanding_vulnerabilities', values: visitor_id) - -# Get unique events for metric -redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_expanding_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } -``` - -### Alternative counters - -Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. -Mainly used for settings and configurations. - -Method: - -```ruby -alt_usage_data(value = nil, fallback: -1, &block) -``` - -Arguments: - -- `value`: a static value in which case the value is returned. -- or a `block`: which is evaluated -- `fallback: -1`: the common value used for any metrics that are failing. - -Example: - -```ruby -alt_usage_data { Gitlab::VERSION } -alt_usage_data { Gitlab::CurrentSettings.uuid } -alt_usage_data(999) -``` - -### Add counters to build new metrics - -When adding the results of two counters, use the `add` Service Data method that -handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-data). - -Example: - -```ruby -add(User.active, User.bot) -``` - -### Prometheus queries - -In those cases where operational metrics should be part of Service Ping, a database or Redis query is unlikely -to provide useful data. Instead, Prometheus might be more appropriate, because most GitLab architectural -components publish metrics to it that can be queried back, aggregated, and included as Service Data. - -NOTE: -Prometheus as a data source for Service Ping is only available for single-node Omnibus installations -that are running the [bundled Prometheus](../../../administration/monitoring/prometheus/index.md) instance. - -To query Prometheus for metrics, a helper method is available to `yield` a fully configured -`PrometheusClient`, given it is available as per the note above: - -```ruby -with_prometheus_client do |client| - response = client.query('<your query>') - ... -end -``` - -Refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) -for how to use its API to query for data. - -### Fallback values for Service Ping - -We return fallback values in these cases: - -| Case | Value | -|-----------------------------|-------| -| Deprecated Metric ([Removed with version 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/335894)) | -1000 | -| Timeouts, general failures | -1 | -| Standard errors in counters | -2 | -| Histogram metrics failure | { '-1' => -1 } | - -## Test counters manually using your Rails console - -```ruby -# count -Gitlab::UsageData.count(User.active) -Gitlab::UsageData.count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) - -# count distinct -Gitlab::UsageData.distinct_count(::Project, :creator_id) -Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -``` - -## Generate the SQL query - -Your Rails console returns the generated SQL queries. For example: - -```ruby -pry(main)> Gitlab::UsageData.count(User.active) - (2.6ms) SELECT "features"."key" FROM "features" - (15.3ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) - (2.4ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) - (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 -``` - -## Optimize queries with Database Lab - -[Database Lab](../../database/database_lab.md) is a service that uses a production clone to test queries. - -- GitLab.com's production database has a 15 second timeout. -- Any single query must stay below the [1 second execution time](../../database/query_performance.md#timing-guidelines-for-queries) with cold caches. -- Add a specialized index on columns involved to reduce the execution time. - -To understand the query's execution, we add the following information -to a merge request description: - -- For counters that have a `time_period` test, we add information for both: - - `time_period = {}` for all time periods. - - `time_period = { created_at: 28.days.ago..Time.current }` for the last 28 days. -- Execution plan and query time before and after optimization. -- Query generated for the index and time. -- Migration output for up and down execution. - -For more details, see the [database review guide](../../database_review.md#preparation-when-adding-or-modifying-queries). - -### Optimization recommendations and examples - -- Use specialized indexes. For examples, see these merge requests: - - [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) - - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) -- Use defined `start` and `finish`. These values can be memoized and reused, as in this - [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). -- Avoid joins and unnecessary complexity in your queries. See this - [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316) as an example. -- Set a custom `batch_size` for `distinct_count`, as in this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). - -## Add the metric definition - -See the [Metrics Dictionary guide](metrics_dictionary.md) for more information. - -## Create a merge request - -Create a merge request for the new Service Ping metric, and do the following: - -- Add the `feature` label to the merge request. A metric is a user-facing change and is part of expanding the Service Ping feature. -- Add a changelog entry that complies with the [changelog entries guide](../../changelog.md). -- Ask for an Analytics Instrumentation review. - On GitLab.com, we have DangerBot set up to monitor Analytics Instrumentation related files and recommend a [Analytics Instrumentation review](review_guidelines.md). - -## Verify your metric - -On GitLab.com, the Product Intelligence team regularly [monitors Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/6000). -They may alert you that your metrics need further optimization to run quicker and with greater success. - -The Service Ping JSON payload for GitLab.com is shared in the -[#g_product_intelligence](https://gitlab.slack.com/archives/CL3A7GFPF) Slack channel every week. - -You may also use the [Service Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. -The dashboard allows filtering by GitLab version, by "Self-managed" and "SaaS", and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you can re-optimize your metric. - -Use [Metrics Dictionary](https://metrics.gitlab.com/) [copy query to clipboard feature](https://www.youtube.com/watch?v=n4o65ivta48&list=PL05JrBw4t0Krg3mbR6chU7pXtMt_es6Pb) to get a query ready to run in Sisense for a specific metric. - -## Set up and test Service Ping locally - -To set up Service Ping locally, you must: - -1. [Set up local repositories](#set-up-local-repositories). -1. [Test local setup](#test-local-setup). -1. Optional. [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). - -### Set up local repositories - -1. Clone and start [GitLab](https://gitlab.com/gitlab-org/gitlab-development-kit). -1. Clone and start [Versions Application](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com). - Make sure you run `docker-compose up` to start a PostgreSQL and Redis instance. -1. Point GitLab to the Versions Application endpoint instead of the default endpoint: - 1. Open [service_ping/submit_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L5) locally and modify `STAGING_BASE_URL`. - 1. Set it to the local Versions Application URL: `http://localhost:3000`. - -### Test local setup - -1. Using the `gitlab` Rails console, manually trigger Service Ping: - - ```ruby - GitlabServicePingWorker.new.perform('triggered_from_cron' => false) - ``` - -1. Use the `versions` Rails console to check the Service Ping was successfully received, - parsed, and stored in the Versions database: - - ```ruby - UsageData.last - ``` - -## Test Prometheus-based Service Ping - -If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) -you want to inspect and verify, you must: - -- Ensure that a Prometheus server is running locally. -- Ensure the respective GitLab components are exporting metrics to the Prometheus server. - -If you do not need to test data coming from Prometheus, no further action -is necessary. Service Ping should degrade gracefully in the absence of a running Prometheus server. - -Three kinds of components may export data to Prometheus, and are included in Service Ping: - -- [`node_exporter`](https://github.com/prometheus/node_exporter): Exports node metrics - from the host machine. -- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter): Exports process metrics - from various GitLab components. -- Other various GitLab services, such as Sidekiq and the Rails server, which export their own metrics. - -### Test with an Omnibus container - -This is the recommended approach to test Prometheus-based Service Ping. - -To verify your change, build a new Omnibus image from your code branch using CI/CD, download the image, -and run a local container instance: - -1. From your merge request, select the `qa` stage, then trigger the `e2e:package-and-test` job. This job triggers an Omnibus - build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). -1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. -1. Open the job logs and locate the full container name including the version. It takes the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. -1. On your local machine, make sure you are signed in to the GitLab Docker registry. You can find the instructions for this in - [Authenticate to the GitLab Container Registry](../../../user/packages/container_registry/authenticate_with_container_registry.md). -1. Once signed in, download the new image by using `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` -1. For more information about working with and running Omnibus GitLab containers in Docker, refer to [GitLab Docker images](../../../install/docker.md) documentation. - -### Test with GitLab development toolkits - -This is the less recommended approach, because it comes with a number of difficulties when emulating a real GitLab deployment. - -The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would -like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start. - -The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Service Ping. -By default, it comes with a fully configured Prometheus service that is set up to scrape a number of components. -However, it has the following limitations: - -- It does not run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing. -- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it usually reports itself as not associated - with any of the other running services. That is not how node metrics are reported in a production setup, where `node_exporter` - always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore - appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping. - -## Aggregated metrics - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. - -WARNING: -This feature is intended solely for internal GitLab use. - -The aggregated metrics feature provides insight into the data attributes in a collection of Service Ping metrics. -This aggregation allows you to count data attributes in events without counting each occurrence of the same data attribute in multiple events. -For example, you can aggregate the number of users who perform several actions, such as creating a new issue and opening a new merge request. -You can then count each user that performed any combination of these actions. - -### Defining aggregated metric via metric YAML definition - -To add data for aggregated metrics to the Service Ping payload, -create metric YAML definition file following [Aggregated metric instrumentation guide](metrics_instrumentation.md#aggregated-metrics). - -### Redis sourced aggregated metrics - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. - -To declare the aggregate of events collected with [Redis HLL Counters](#redis-hll-counters), make sure `time_frame` does not include the `all` value, which is unavailable for Redis-sourced aggregated metrics. - -While it is possible to aggregate EE-only events together with events that occur in all GitLab editions, it's important to remember that doing so may produce high variance between data collected from EE and CE GitLab instances. - -### Database sourced aggregated metrics - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52784) in GitLab 13.9. - -To declare an aggregate of metrics based on events collected from database, follow -these steps: - -1. [Persist the metrics for aggregation](#persist-metrics-for-aggregation). -1. [Add new aggregated metric definition](#add-new-aggregated-metric-definition). - -#### Persist metrics for aggregation - -Only metrics calculated with [Estimated Batch Counters](#estimated-batch-counters) -can be persisted for database sourced aggregated metrics. To persist a metric, -inject a Ruby block into the -[`estimate_batch_distinct_count`](#estimate_batch_distinct_count-method) method. -This block should invoke the -`Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` -[method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage/metrics/aggregates/sources/postgres_hll.rb#L21), -which stores `estimate_batch_distinct_count` results for future use in aggregated metrics. - -The `Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` -method accepts the following arguments: - -- `metric_name`: The name of metric to use for aggregations. Should be the same - as the key under which the metric is added into Service Ping. -- `recorded_at_timestamp`: The timestamp representing the moment when a given - Service Ping payload was collected. You should use the convenience method `recorded_at` - to fill `recorded_at_timestamp` argument, like this: `recorded_at_timestamp: recorded_at` -- `time_period`: The time period used to build the `relation` argument passed into - `estimate_batch_distinct_count`. To collect the metric with all available historical - data, set a `nil` value as time period: `time_period: nil`. -- `data`: HyperLogLog buckets structure representing unique entries in `relation`. - The `estimate_batch_distinct_count` method always passes the correct argument - into the block, so `data` argument must always have a value equal to block argument, - like this: `data: result` - -Example metrics persistence: - -```ruby -class UsageData - def count_secure_pipelines(time_period) - ... - relation = ::Security::Scan.by_scan_types(scan_type).where(time_period) - - pipelines_with_secure_jobs['dependency_scanning_pipeline'] = estimate_batch_distinct_count(relation, :pipeline_id, batch_size: 1000, start: start_id, finish: finish_id) do |result| - ::Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll - .save_aggregated_metrics(metric_name: 'dependency_scanning_pipeline', recorded_at_timestamp: recorded_at, time_period: time_period, data: result) - end - end -end -``` - -#### Add new aggregated metric definition - -After all metrics are persisted, you can add an aggregated metric definition following [Aggregated metric instrumentation guide](metrics_instrumentation.md#aggregated-metrics). -To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), -you must fulfill the following requirements: - -- Metrics names listed in the `events:` attribute, have to use the same names you passed in the `metric_name` argument while persisting metrics in previous step. -- Every metric listed in the `events:` attribute, has to be persisted for **every** selected `time_frame:` value. +<!-- This redirect file can be deleted after <2024-01-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/index.md b/doc/development/internal_analytics/service_ping/index.md index f532bb1ac31..bae4e35149d 100644 --- a/doc/development/internal_analytics/service_ping/index.md +++ b/doc/development/internal_analytics/service_ping/index.md @@ -31,7 +31,7 @@ We use the following terminology to describe the Service Ping components: - **Service Ping**: the process that collects and generates a JSON payload. - **Service Data**: the contents of the Service Ping JSON payload. This includes metrics. - **Metrics**: primarily made up of row counts for different tables in an instance's database. Each - metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) + metric has a corresponding [metric definition](../metrics/metrics_dictionary.md#metrics-definition-and-validation) in a YAML file. - **MAU**: monthly active users. - **WAU**: weekly active users. @@ -148,10 +148,6 @@ We also collect metrics specific to [Geo](../../../administration/geo/index.md) ```json [ { - "repository_verification_enabled"=>true, - "repositories_replication_enabled"=>true, - "repositories_synced_count"=>24, - "repositories_failed_count"=>0, "git_fetch_event_count_weekly"=>nil, "git_push_event_count_weekly"=>nil, ... other geo node status fields @@ -159,10 +155,6 @@ We also collect metrics specific to [Geo](../../../administration/geo/index.md) ] ``` -## Implementing Service Ping - -See the [implement Service Ping](implement.md) guide. - ## Example Service Ping payload The following is example content of the Service Ping payload. @@ -237,7 +229,8 @@ The following is example content of the Service Ping payload. }, "container_registry_server": { "vendor": "gitlab", - "version": "2.9.1-gitlab" + "version": "2.9.1-gitlab", + "db_enabled": false }, "database": { "adapter": "postgresql", @@ -495,15 +488,24 @@ skip_db_write: GitlabServicePingWorker.new.perform('triggered_from_cron' => false, 'skip_db_write' => true) ``` +### Fallback values for Service Ping + +We return fallback values in these cases: + +| Case | Value | +|-----------------------------|-------| +| Deprecated Metric ([Removed with version 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/335894)) | -1000 | +| Timeouts, general failures | -1 | +| Standard errors in counters | -2 | +| Histogram metrics failure | { '-1' => -1 } | + ## Monitoring Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). ## Related topics -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Snowplow Guide](../snowplow/index.md) -- [Product Intelligence Direction](https://about.gitlab.com/direction/analytics/product-intelligence/) +- [Analytics Instrumentation Direction](https://about.gitlab.com/direction/analytics/analytics-instrumentation/) - [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) - [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) - [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/internal_analytics/service_ping/metrics_dictionary.md b/doc/development/internal_analytics/service_ping/metrics_dictionary.md index e677118fff6..9adcaaf431d 100644 --- a/doc/development/internal_analytics/service_ping/metrics_dictionary.md +++ b/doc/development/internal_analytics/service_ping/metrics_dictionary.md @@ -1,230 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../metrics/metrics_dictionary.md' +remove_date: '2023-09-25' --- -# Metrics Dictionary Guide +This document was moved to [another location](../metrics/metrics_dictionary.md). -[Service Ping](index.md) metrics are defined in individual YAML files definitions from which the -[Metrics Dictionary](https://metrics.gitlab.com/) is built. Currently, the metrics dictionary is built automatically once a day. When a change to a metric is made in a YAML file, you can see the change in the dictionary within 24 hours. -This guide describes the dictionary and how it's implemented. - -## Metrics Definition and validation - -We are using [JSON Schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json) to validate the metrics definition. - -This process is meant to ensure consistent and valid metrics defined for Service Ping. All metrics *must*: - -- Comply with the defined [JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json). -- Have a unique `key_path` . -- Have an owner. - -All metrics are stored in YAML files: - -- [`config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics) - -WARNING: -Only metrics with a metric definition YAML and whose status is not `removed` are added to the Service Ping JSON payload. - -Each metric is defined in a separate YAML file consisting of a number of fields: - -| Field | Required | Additional information | -|---------------------|----------|----------------------------------------------------------------| -| `key_path` | yes | JSON key path for the metric, location in Service Ping payload. | -| `description` | yes | | -| `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | -| `product_stage` | yes | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the metric. | -| `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | -| `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | -| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `removed`, `broken`. | -| `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | -| `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`, `license`, `internal_events`. | -| `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `operational`, `optional`, `subscription`, `standard`. The default value is `optional`.| -| `instrumentation_class` | yes | `string`; [the class that implements the metric](metrics_instrumentation.md). | -| `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. | -| `performance_indicator_type` | no | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, `umau` or `customer_health_score`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). | -| `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | -| `milestone` | yes | The milestone when the metric is introduced and when it's available to self-managed instances with the official GitLab release. | -| `milestone_removed` | no | The milestone when the metric is removed. Required for removed metrics. | -| `introduced_by_url` | no | The URL to the merge request that introduced the metric to be available for self-managed instances. | -| `removed_by_url` | no | The URL to the merge request that removed the metric. Required for removed metrics. | -| `repair_issue_url` | no | The URL of the issue that was created to repair a metric with a `broken` status. | -| `options` | no | `object`: options information needed to calculate the metric value. | -| `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | - -### Metric `key_path` - -The `key_path` of the metric is the location in the JSON Service Ping payload. - -The `key_path` could be composed from multiple parts separated by `.` and it must be unique. - -We recommend to add the metric in one of the top-level keys: - -- `settings`: for settings related metrics. -- `counts_weekly`: for counters that have data for the most recent 7 days. -- `counts_monthly`: for counters that have data for the most recent 28 days. -- `counts`: for counters that have data for all time. - -NOTE: -We can't control what the metric's `key_path` is, because some of them are generated dynamically in `usage_data.rb`. -For example, see [Redis HLL metrics](implement.md#redis-hll-counters). - -### Metric statuses - -Metric definitions can have one of the following statuses: - -- `active`: Metric is used and reports data. -- `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute. -- `removed`: Metric was removed, but it may appear in Service Ping payloads sent from instances running on older versions of GitLab. - -### Metric `value_type` - -Metric definitions can have one of the following values for `value_type`: - -- `boolean` -- `number` -- `string` -- `object`: A metric with `value_type: object` must have `value_json_schema` with a link to the JSON schema for the object. -In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. -An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), -which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. - -### Metric `time_frame` - -A metric's time frame is calculated based on the `time_frame` field and the `data_source` of the metric. - -| data_source | time_frame | Description | -|------------------------|------------|-------------------------------------------------| -| any | `none` | A type of data that's not tracked over time, such as settings and configuration information | -| `database` | `all` | The whole time the metric has been active (all-time interval) | -| `database` | `7d` | 9 days ago to 2 days ago | -| `database` | `28d` | 30 days ago to 2 days ago | -| `redis` | `all` | The whole time the metric has been active (all-time interval) | -| `redis_hll` | `7d` | Most recent complete week | -| `redis_hll` | `28d` | Most recent 4 complete weeks | - -### Data category - -We use the following categories to classify a metric: - -- `operational`: Required data for operational purposes. -- `optional`: Default value for a metric. Data that is optional to collect. This can be [enabled or disabled](../../../administration/settings/usage_statistics.md#enable-or-disable-usage-statistics) in the Admin Area. -- `subscription`: Data related to licensing. -- `standard`: Standard set of identifiers that are included when collecting data. - -An aggregate metric is a metric that is the sum of two or more child metrics. Service Ping uses the data category of -the aggregate metric to determine whether or not the data is included in the reported Service Ping payload. - -### Example YAML metric definition - -The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/uuid.yml) -YAML file includes an example metric definition, where the `uuid` metric is the GitLab -instance unique identifier. - -```yaml -key_path: uuid -description: GitLab instance unique identifier -product_section: analytics -product_stage: analytics -product_group: analytics_instrumentation -value_type: string -status: active -milestone: 9.1 -instrumentation_class: UuidMetric -introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521 -time_frame: none -data_source: database -distribution: -- ce -- ee -tier: -- free -- premium -- ultimate -``` - -### Create a new metric definition - -The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions. - -For uniqueness, the generated files include a timestamp prefix in ISO 8601 format. - -The generator takes a list of key paths and 3 options as arguments. It creates metric YAML definitions in the corresponding location: - -- `--ee`, `--no-ee` Indicates if metric is for EE. -- `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. -- `--class_name=CLASS_NAME` Indicates the instrumentation class. For example `UsersCreatingIssuesMetric`, `UuidMetric` - -**Single metric example** - -```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d --class_name=CountIssues -// Creates 1 file -// create config/metrics/counts_7d/issues.yml -``` - -**Multiple metrics example** - -```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d --class_name=CountUsersCreatingIssues -// Creates 2 files -// create config/metrics/counts_7d/issues.yml -// create config/metrics/counts_7d/users.yml -``` - -NOTE: -To create a metric definition used in EE, add the `--ee` flag. - -```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d --class_name=CountUsersCreatingIssues -// Creates 1 file -// create ee/config/metrics/counts_7d/issues.yml -``` - -### Metrics added dynamic to Service Ping payload - -The [Redis HLL metrics](implement.md#known-events-are-added-automatically-in-service-data-payload) are added automatically to Service Ping payload. - -A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events. - -The generator takes `category` and `events` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for each of the events (for weekly and monthly time frames): - -**Single metric example** - -```shell -bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues -// Creates 2 files -// create config/metrics/counts_7d/count_users_closing_issues_weekly.yml -// create config/metrics/counts_28d/count_users_closing_issues_monthly.yml -``` - -**Multiple metrics example** - -```shell -bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues count_users_reopening_issues -// Creates 4 files -// create config/metrics/counts_7d/count_users_closing_issues_weekly.yml -// create config/metrics/counts_28d/count_users_closing_issues_monthly.yml -// create config/metrics/counts_7d/count_users_reopening_issues_weekly.yml -// create config/metrics/counts_28d/count_users_reopening_issues_monthly.yml -``` - -To create a metric definition used in EE, add the `--ee` flag. - -```shell -bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues users_closing_issues --ee -// Creates 2 files -// create config/metrics/counts_7d/i_closed_weekly.yml -// create config/metrics/counts_28d/i_closed_monthly.yml -``` - -## Metrics Dictionary - -[Metrics Dictionary is a separate application](https://gitlab.com/gitlab-org/analytics-section/analytics-instrumentation/metric-dictionary). - -All metrics available in Service Ping are in the [Metrics Dictionary](https://metrics.gitlab.com/). - -### Copy query to clipboard - -To check if a metric has data in Sisense, use the copy query to clipboard feature. This copies a query that's ready to use in Sisense. The query gets the last five service ping data for GitLab.com for a given metric. For information about how to check if a Service Ping metric has data in Sisense, see this [demo](https://www.youtube.com/watch?v=n4o65ivta48). +<!-- This redirect file can be deleted after <2023-12-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/metrics_instrumentation.md b/doc/development/internal_analytics/service_ping/metrics_instrumentation.md index ada7cc566a1..8a85a310a9e 100644 --- a/doc/development/internal_analytics/service_ping/metrics_instrumentation.md +++ b/doc/development/internal_analytics/service_ping/metrics_instrumentation.md @@ -1,508 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../metrics/metrics_instrumentation.md' +remove_date: '2024-01-13' --- -# Metrics instrumentation guide +This document was moved to [another location](../metrics/metrics_instrumentation.md). -This guide describes how to develop Service Ping metrics using metrics instrumentation. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a video tutorial, see the [Adding Service Ping metric via instrumentation class](https://youtu.be/p2ivXhNxUoY). - -## Nomenclature - -- **Instrumentation class**: - - Inherits one of the metric classes: `DatabaseMetric`, `RedisMetric`, `RedisHLLMetric`, `NumbersMetric` or `GenericMetric`. - - Implements the logic that calculates the value for a Service Ping metric. - -- **Metric definition** - The Service Data metric YAML definition. - -- **Hardening**: - Hardening a method is the process that ensures the method fails safe, returning a fallback value like -1. - -## How it works - -A metric definition has the [`instrumentation_class`](metrics_dictionary.md) field, which can be set to a class. - -The defined instrumentation class should inherit one of the existing metric classes: `DatabaseMetric`, `RedisMetric`, `RedisHLLMetric`, `NumbersMetric` or `GenericMetric`. - -The current convention is that a single instrumentation class corresponds to a single metric. On rare occasions, there are exceptions to that convention like [Redis metrics](#redis-metrics). To use a single instrumentation class for more than one metric, please reach out to one of the `@gitlab-org/analytics-section/analytics-instrumentation/engineers` members to consult about your case. - -Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire - process of Service Ping generation. - -We have built a domain-specific language (DSL) to define the metrics instrumentation. - -## Database metrics - -You can use database metrics to track data kept in the database, for example, a count of issues that exist on a given instance. - -- `operation`: Operations for the given `relation`, one of `count`, `distinct_count`, `sum`, and `average`. -- `relation`: Assigns lambda that returns the `ActiveRecord::Relation` for the objects we want to perform the `operation`. The assigned lambda can accept up to one parameter. The parameter is hashed and stored under the `options` key in the metric definition. -- `start`: Specifies the start value of the batch counting, by default is `relation.minimum(:id)`. -- `finish`: Specifies the end value of the batch counting, by default is `relation.maximum(:id)`. -- `cache_start_and_finish_as`: Specifies the cache key for `start` and `finish` values and sets up caching them. Use this call when `start` and `finish` are expensive queries that should be reused between different metric calculations. -- `available?`: Specifies whether the metric should be reported. The default is `true`. -- `timestamp_column`: Optionally specifies timestamp column for metric used to filter records for time constrained metrics. The default is `created_at`. - -[Example of a merge request that adds a database metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60022). - -```ruby -module Gitlab - module Usage - module Metrics - module Instrumentations - class CountIssuesMetric < DatabaseMetric - operation :count - - relation ->(options) { Issue.where(confidential: options[:confidential]) } - end - end - end - end -end -``` - -### Ordinary batch counters Example - -```ruby -module Gitlab - module Usage - module Metrics - module Instrumentations - class CountIssuesMetric < DatabaseMetric - operation :count - - start { Issue.minimum(:id) } - finish { Issue.maximum(:id) } - - relation { Issue } - end - end - end - end -end -``` - -### Distinct batch counters Example - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class CountUsersAssociatingMilestonesToReleasesMetric < DatabaseMetric - operation :distinct_count, column: :author_id - - relation { Release.with_milestones } - - start { Release.minimum(:author_id) } - finish { Release.maximum(:author_id) } - end - end - end - end -end -``` - -### Sum Example - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class JiraImportsTotalImportedIssuesCountMetric < DatabaseMetric - operation :sum, column: :imported_issues_count - - relation { JiraImportState.finished } - end - end - end - end -end -``` - -### Average Example - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class CountIssuesWeightAverageMetric < DatabaseMetric - operation :average, column: :weight - - relation { Issue } - end - end - end - end -end -``` - -## Redis metrics - -You can use Redis metrics to track events not kept in the database, for example, a count of how many times the search bar has been used. - -[Example of a merge request that adds `Redis` metrics](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103455). - -The `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class. - -Required options: - -- `event`: the event name. -- `prefix`: the value of the `PREFIX` constant used in the counter classes from the `Gitlab::UsageDataCounters` namespace. - -Count unique values for `source_code_pushes` event. - -```yaml -time_frame: all -data_source: redis -instrumentation_class: RedisMetric -options: - event: pushes - prefix: source_code -``` - -### Availability-restrained Redis metrics - -If the Redis metric should only be available in the report under some conditions, then you must specify these conditions in a new class that is a child of the `RedisMetric` class. - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class MergeUsageCountRedisMetric < RedisMetric - available? { Feature.enabled?(:merge_usage_data_missing_key_paths) } - end - end - end - end -end -``` - -You must also use the class's name in the YAML setup. - -```yaml -time_frame: all -data_source: redis -instrumentation_class: MergeUsageCountRedisMetric -options: - event: pushes - prefix: source_code -``` - -## Redis HyperLogLog metrics - -You can use Redis HyperLogLog metrics to track events not kept in the database and incremented for unique values such as unique users, -for example, a count of how many different users used the search bar. - -[Example of a merge request that adds a `RedisHLL` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61685). - -Count unique values for `i_quickactions_approve` event. - -```yaml -time_frame: 28d -data_source: redis_hll -instrumentation_class: RedisHLLMetric -options: - events: - - i_quickactions_approve -``` - -### Availability-restrained Redis HyperLogLog metrics - -If the Redis HyperLogLog metric should only be available in the report under some conditions, then you must specify these conditions in a new class that is a child of the `RedisHLLMetric` class. - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class MergeUsageCountRedisHLLMetric < RedisHLLMetric - available? { Feature.enabled?(:merge_usage_data_missing_key_paths) } - end - end - end - end -end -``` - -You must also use the class's name in the YAML setup. - -```yaml -time_frame: 28d -data_source: redis_hll -instrumentation_class: MergeUsageCountRedisHLLMetric -options: - events: - - i_quickactions_approve -``` - -## Aggregated metrics - -<div class="video-fallback"> - See the video from: <a href="https://www.youtube.com/watch?v=22LbYqHwtUQ">Product Intelligence Office Hours Oct 6th</a> for an aggregated metrics walk-through. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/22LbYqHwtUQ" frameborder="0" allowfullscreen> </iframe> -</figure> - -The aggregated metrics feature provides insight into the number of data attributes, for example `pseudonymized_user_ids`, that occurred in a collection of events. For example, you can aggregate the number of users who perform multiple actions such as creating a new issue and opening -a new merge request. - -You can use a YAML file to define your aggregated metrics. The following arguments are required: - -- `options.events`: List of event names to aggregate into metric data. All events in this list must - use the same data source. Additional data source requirements are described in - [Database sourced aggregated metrics](implement.md#database-sourced-aggregated-metrics) and - [Redis sourced aggregated metrics](implement.md#redis-sourced-aggregated-metrics). -- `options.aggregate.operator`: Operator that defines how the aggregated metric data is counted. Available operators are: - - `OR`: Removes duplicates and counts all entries that triggered any of the listed events. - - `AND`: Removes duplicates and counts all elements that were observed triggering all of the following events. -- `options.aggregate.attribute`: Information pointing to the attribute that is being aggregated across events. -- `time_frame`: One or more valid time frames. Use these to limit the data included in aggregated metrics to events within a specific date-range. Valid time frames are: - - `7d`: The last 7 days of data. - - `28d`: The last 28 days of data. - - `all`: All historical data, only available for `database` sourced aggregated metrics. -- `data_source`: Data source used to collect all events data included in the aggregated metrics. Valid data sources are: - - [`database`](implement.md#database-sourced-aggregated-metrics) - - [`redis_hll`](implement.md#redis-sourced-aggregated-metrics) - -Refer to merge request [98206](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98206) for an example of a merge request that adds an `AggregatedMetric` metric. - -Count unique `user_ids` that occurred in at least one of the events: `incident_management_alert_status_changed`, -`incident_management_alert_assigned`, `incident_management_alert_todo`, `incident_management_alert_create_incident`. - -```yaml -time_frame: 28d -instrumentation_class: AggregatedMetric -data_source: redis_hll -options: - aggregate: - operator: OR - attribute: user_id - events: - - `incident_management_alert_status_changed` - - `incident_management_alert_assigned` - - `incident_management_alert_todo` - - `incident_management_alert_create_incident` -``` - -### Availability-restrained Aggregated metrics - -If the Aggregated metric should only be available in the report under specific conditions, then you must specify these conditions in a new class that is a child of the `AggregatedMetric` class. - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class MergeUsageCountAggregatedMetric < AggregatedMetric - available? { Feature.enabled?(:merge_usage_data_missing_key_paths) } - end - end - end - end -end -``` - -You must also use the class's name in the YAML setup. - -```yaml -time_frame: 28d -instrumentation_class: MergeUsageCountAggregatedMetric -data_source: redis_hll -options: - aggregate: - operator: OR - attribute: user_id - events: - - `incident_management_alert_status_changed` - - `incident_management_alert_assigned` - - `incident_management_alert_todo` - - `incident_management_alert_create_incident` -``` - -## Numbers metrics - -- `operation`: Operations for the given `data` block. Currently we only support `add` operation. -- `data`: a `block` which contains an array of numbers. -- `available?`: Specifies whether the metric should be reported. The default is `true`. - -```ruby -# frozen_string_literal: true - -module Gitlab - module Usage - module Metrics - module Instrumentations - class IssuesBoardsCountMetric < NumbersMetric - operation :add - - data do |time_frame| - [ - CountIssuesMetric.new(time_frame: time_frame).value, - CountBoardsMetric.new(time_frame: time_frame).value - ] - end - end - end - end - end - end -end -``` - -You must also include the instrumentation class name in the YAML setup. - -```yaml -time_frame: 28d -instrumentation_class: IssuesBoardsCountMetric -``` - -## Generic metrics - -You can use generic metrics for other metrics, for example, an instance's database version. Observations type of data will always have a Generic metric counter type. - -- `value`: Specifies the value of the metric. -- `available?`: Specifies whether the metric should be reported. The default is `true`. - -[Example of a merge request that adds a generic metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60256). - -```ruby -module Gitlab - module Usage - module Metrics - module Instrumentations - class UuidMetric < GenericMetric - value do - Gitlab::CurrentSettings.uuid - end - end - end - end - end -end -``` - -## Prometheus metrics - -This instrumentation class lets you handle Prometheus queries by passing a Prometheus client object as an argument to the `value` block. -Any Prometheus error handling should be done in the block itself. - -- `value`: Specifies the value of the metric. A Prometheus client object is passed as the first argument. -- `available?`: Specifies whether the metric should be reported. The default is `true`. - -[Example of a merge request that adds a Prometheus metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122400). - -```ruby -module Gitlab - module Usage - module Metrics - module Instrumentations - class GitalyApdexMetric < PrometheusMetric - value do |client| - result = client.query('avg_over_time(gitlab_usage_ping:gitaly_apdex:ratio_avg_over_time_5m[1w])').first - - break FALLBACK unless result - - result['value'].last.to_f - end - end - end - end - end -end -``` - -## Support for instrumentation classes - -There is support for: - -- `count`, `distinct_count`, `estimate_batch_distinct_count`, `sum`, and `average` for [database metrics](#database-metrics). -- [Redis metrics](#redis-metrics). -- [Redis HLL metrics](#redis-hyperloglog-metrics). -- `add` for [numbers metrics](#numbers-metrics). -- [Generic metrics](#generic-metrics), which are metrics based on settings or configurations. - -There is no support for: - -- `add`, `histogram` for database metrics. - -You can [track the progress to support these](https://gitlab.com/groups/gitlab-org/-/epics/6118). - -## Create a new metric instrumentation class - -To create a stub instrumentation for a Service Ping metric, you can use a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_generator.rb): - -The generator takes the class name as an argument and the following options: - -- `--type=TYPE` Required. Indicates the metric type. It must be one of: `database`, `generic`, `redis`, `numbers`. -- `--operation` Required for `database` & `numbers` type. - - For `database` it must be one of: `count`, `distinct_count`, `estimate_batch_distinct_count`, `sum`, `average`. - - For `numbers` it must be: `add`. -- `--ee` Indicates if the metric is for EE. - -```shell -rails generate gitlab:usage_metric CountIssues --type database --operation distinct_count - create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb - create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb -``` - -## Migrate Service Ping metrics to instrumentation classes - -This guide describes how to migrate a Service Ping metric from [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) or [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb) to instrumentation classes. - -1. Choose the metric type: - -- [Database metric](#database-metrics) -- [Redis HyperLogLog metrics](#redis-hyperloglog-metrics) -- [Redis metric](#redis-metrics) -- [Numbers metric](#numbers-metrics) -- [Generic metric](#generic-metrics) - -1. Determine the location of instrumentation class: either under `ee` or outside `ee`. - -1. [Generate the instrumentation class file](#create-a-new-metric-instrumentation-class). - -1. Fill the instrumentation class body: - - - Add code logic for the metric. This might be similar to the metric implementation in `usage_data.rb`. - - Add tests for the individual metric [`spec/lib/gitlab/usage/metrics/instrumentations/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage/metrics/instrumentations). - - Add tests for Service Ping. - -1. [Generate the metric definition file](metrics_dictionary.md#create-a-new-metric-definition). - -1. Remove the code from [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) or [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb). - -1. Remove the tests from [`spec/lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/usage_data_spec.rb) or [`ee/spec/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/ee/gitlab/usage_data_spec.rb). - -## Troubleshoot metrics - -Sometimes metrics fail for reasons that are not immediately clear. The failures can be related to performance issues or other problems. -The following pairing session video gives you an example of an investigation in to a real-world failing metric. - -<div class="video-fallback"> - See the video from: <a href="https://www.youtube.com/watch?v=y_6m2POx2ug">Product Intelligence Office Hours Oct 27th</a> to learn more about the metrics troubleshooting process. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube-nocookie.com/embed/y_6m2POx2ug" frameborder="0" allowfullscreen> </iframe> -</figure> +<!-- This redirect file can be deleted after <2024-01-13>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md index 4980a8cf63d..126bc3988ed 100644 --- a/doc/development/internal_analytics/service_ping/metrics_lifecycle.md +++ b/doc/development/internal_analytics/service_ping/metrics_lifecycle.md @@ -1,105 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../metrics/metrics_lifecycle.md' +remove_date: '2023-09-25' --- -# Service Ping metric lifecycle +This document was moved to [another location](../metrics/metrics_lifecycle.md). -The following guidelines explain the steps to follow at each stage of a metric's lifecycle. - -## Add a new metric - -Follow the [Implement Service Ping](implement.md) guide. - -## Change an existing metric - -WARNING: -We want to **PREVENT** changes to the calculation logic or important attributes on any metric as this invalidates comparisons of the same metric across different versions of GitLab. - -If you change a metric, you have to consider that not all instances of GitLab are running on the newest version. Old instances will still report the old version of the metric. -Additionally, a metric's reported numbers are primarily interesting compared to previously reported numbers. -As a result, if you need to change one of the following parts of a metric, you need to add a new metric instead. It's your choice whether to keep the old metric alongside the new one or [remove it](#remove-a-metric). - -- **calculation logic**: This means any changes that can produce a different value than the previous implementation -- **YAML attributes**: The following attributes are directly used for analysis or calculation: `key_path`, `time_frame`, `value_type`, `data_source`. - -If you change the `performance_indicator_type` attribute of a metric or think your case needs an exception from the outlined rules then please notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) teams by `@` mentioning those groups in a comment on the merge request or issue. - -You can change any other attributes without impact to the calculation or analysis. See [this video tutorial](https://youtu.be/bYf3c01KCls) for help updating metric attributes. - -Currently, the [Metrics Dictionary](https://metrics.gitlab.com/) is built automatically once a day. You can see the change in the dictionary within 24 hours when you change the metric's YAML file. - -## Remove a metric - -WARNING: -If a metric is not used in Sisense or any other system after 6 months, the -Analytics Instrumentation team marks it as inactive and assigns it to the group owner for review. - -We are working on automating this process. See [this epic](https://gitlab.com/groups/gitlab-org/-/epics/8988) for details. - -Analytics Instrumentation removes metrics from Service Ping if they are not used in any Sisense dashboard. - -For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/388236). - -To remove a metric: - -1. Create an issue for removing the metric if none exists yet. The issue needs to outline why the metric should be deleted. You can use this issue to document the removal process. - -1. Verify the metric is not used to calculate the conversational index. The - conversational index is a measure that reports back to self-managed instances - to inform administrators of the progress of DevOps adoption for the instance. - - You can check - [`CalculateConvIndexService`](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/app/services/calculate_conv_index_service.rb) - to view the metrics that are used. The metrics are represented - as the keys that are passed as a field argument into the `get_value` method. - -1. Verify that removing the metric from the Service Ping payload does not cause - errors in [Version App](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com) - when the updated payload is collected and processed. Version App collects - and persists all Service Ping reports. To verify Service Ping processing in your local development environment, follow this [guide](https://www.youtube.com/watch?v=FS5emplabRU). - Alternatively, you can modify [fixtures](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/spec/support/usage_data_helpers.rb) - used to test the [`UsageDataController#create`](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com/-/blob/main/spec/controllers/usage_data_controller_spec.rb) - endpoint, and assure that test suite does not fail when metric that you wish to remove is not included into test payload. - -1. Remove data from Redis - - For [Ordinary Redis](implement.md#ordinary-redis-counters) counters remove data stored in Redis. - - - Add a migration to remove the data from Redis for the related Redis keys. For more details, see [this MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82604/diffs). - -1. Create an issue in the - [GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues). - Ask for confirmation that the metric is not referred to in any SiSense dashboards and - can be safely removed from Service Ping. Use this - [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/15266) for guidance. - -1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment in the issue from step 1 regarding the deletion of the metric. - Many Service Ping metrics are relied upon for health score and XMAU reporting and unexpected changes to those metrics could break reporting. - -1. After you verify the metric can be safely removed, - update the attributes of the metric's YAML definition: - - - Set the `status:` to `removed`. - - Set `removed_by_url:` to the URL of the MR removing the metric - - Set `milestone_removed:` to the number of the - milestone in which the metric was removed. - - Do not remove the metric's YAML definition altogether. Some self-managed - instances might not immediately update to the latest version of GitLab, and - therefore continue to report the removed metric. The Analytics Instrumentation team - requires a record of all removed metrics to identify and filter them. - - For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#b01f429a54843feb22265100c0e4fec1b7da1240_10_10). - -1. After you verify the metric can be safely removed, - remove the metric's instrumentation from - [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) - or - [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb). - - For example please take a look at this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60149/diffs#6335dc533bd21df26db9de90a02dd66278c2390d_167_167). - -1. Remove any other records related to the metric: - - The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags). +<!-- This redirect file can be deleted after <2023-12-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md b/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md index 63177f093e2..7388415f384 100644 --- a/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md +++ b/doc/development/internal_analytics/service_ping/performance_indicator_metrics.md @@ -1,16 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../metrics/metrics_dictionary.md#performance-indicator-metrics' +remove_date: '2023-09-25' --- -# Performance Indicator Metrics guide +This document was moved to [another location](../metrics/metrics_lifecycle.md). -This guide describes how to use metrics definitions to define [performance indicator](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/#implementing-product-performance-indicators) metrics. - -To use a metric definition to manage a performance indicator: - -1. Create a merge request that includes related changes. -1. Use labels `~"analytics instrumentation"`, `"~Data Warehouse::Impact Check"`. -1. Update the metric definition `performance_indicator_type` [field](metrics_dictionary.md#metrics-definition-and-validation). -1. Create an issue in GitLab Product Data Insights project with the [PI Chart Help template](https://gitlab.com/gitlab-data/product-analytics/-/issues/new?issuable_template=PI%20Chart%20Help) to have the new metric visualized. +<!-- This redirect file can be deleted after <2023-12-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/review_guidelines.md b/doc/development/internal_analytics/service_ping/review_guidelines.md index c816c905097..0ca7b084fc4 100644 --- a/doc/development/internal_analytics/service_ping/review_guidelines.md +++ b/doc/development/internal_analytics/service_ping/review_guidelines.md @@ -1,81 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../review_guidelines.md' +remove_date: '2023-12-29' --- -# Service Ping review guidelines +This document was moved to [another location](../review_guidelines.md). -This page includes introductory material for a -[Analytics Instrumentation](https://about.gitlab.com/handbook/engineering/development/analytics/analytics-instrumentation/) -review, and is specific to Service Ping related reviews. For broader advice and -general best practices for code reviews, refer to our [code review guide](../../code_review.md). - -## Resources for reviewers - -- [Service Ping Guide](index.md) -- [Metrics Dictionary](https://metrics.gitlab.com/) - -## Review process - -We recommend a Analytics Instrumentation review when a merge request (MR) touches -any of the following Service Ping files: - -- `usage_data*` files. -- The Metrics Dictionary, including files in: - - [`config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics). - - [`ee/config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/config/metrics). - - [`schema.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/schema.json). -- Analytics Instrumentation tooling. For example, - [`Gitlab::UsageMetricDefinitionGenerator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) - -### Roles and process - -#### The merge request **author** should - -- Decide whether a Analytics Instrumentation review is needed. You can skip the Analytics Instrumentation -review and remove the labels if the changes are not related to the Analytics Instrumentation domain and -are regular backend changes. -- If a Analytics Instrumentation review is needed, add the labels - `~analytics instrumentation` and `~analytics instrumentation::review pending`. -- For merge requests authored by Analytics Instrumentation team members: - - Assign both the `~backend` and `~analytics instrumentation` reviews to another Analytics Instrumentation team member. - - Assign the maintainer review to someone outside of the Analytics Instrumentation group. -- Assign an - [engineer](https://gitlab.com/groups/gitlab-org/analytics-section/analytics-instrumentation/engineers/-/group_members?with_inherited_permissions=exclude) from the Analytics Instrumentation team for a review. -- Set the correct attributes in the metric's YAML definition: - - `product_section`, `product_stage`, `product_group` - - Provide a clear description of the metric. -- Add a changelog [according to guidelines](../../changelog.md). - -#### The Analytics Instrumentation **reviewer** should - -- Perform a first-pass review on the merge request and suggest improvements to the author. -- Check the [metric's location](metrics_dictionary.md#metric-key_path) in - the Service Ping JSON payload. -- Add the `~database` label and ask for a [database review](../../database_review.md) for - metrics that are based on Database. -- Add `~Data Warehouse::Impact Check` for any database metric that has a query change. Changes in queries can affect [data operations](https://about.gitlab.com/handbook/business-technology/data-team/how-we-work/triage/#gitlabcom-db-structure-changes). -- For tracking using Redis HLL (HyperLogLog): - - Check if a [feature flag is needed](implement.md#recommendations). -- For a metric's YAML definition: - - Check the metric's `description`. - - Check the metric's `key_path`. - - Check the `product_section`, `product_stage`, and `product_group` fields. - Read the [stages file](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml). - - Check the file location. Consider the time frame, and if the file should be under `ee`. - - Check the tiers. -- If a metric was changed or removed: Make sure the MR author notified the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the issue for the MR and all of these groups have acknowledged the removal. -- Make sure that the new metric is available in Service Ping payload, by running: `Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values).dig(*'key_path'.split('.'))` with `key_path` substituted by the new metric's key_path. -- Metrics instrumentations - - Recommend using metrics instrumentation for new metrics, [if possible](metrics_instrumentation.md#support-for-instrumentation-classes). -- Approve the MR, and relabel the MR with `~"analytics instrumentation::approved"`. - -## Review workload distribution - -[Danger bot](../../dangerbot.md) adds the list of changed Analytics Instrumentation files -and pings the -[`@gitlab-org/analytics-section/analytics-instrumentation/engineers`](https://gitlab.com/groups/gitlab-org/analytics-section/analytics-instrumentation/engineers/-/group_members?with_inherited_permissions=exclude) group for merge requests -that are not drafts. - -Any of the Analytics Instrumentation engineers can be assigned for the Analytics Instrumentation review. +<!-- This redirect file can be deleted after <2023-12-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/service_ping/troubleshooting.md b/doc/development/internal_analytics/service_ping/troubleshooting.md index e685635c5f7..ba56ee1e223 100644 --- a/doc/development/internal_analytics/service_ping/troubleshooting.md +++ b/doc/development/internal_analytics/service_ping/troubleshooting.md @@ -6,6 +6,90 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting Service Ping +## Set up and test Service Ping locally + +To set up Service Ping locally, you must: + +1. [Set up local repositories](#set-up-local-repositories). +1. [Test local setup](#test-local-setup). +1. Optional. [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). + +### Set up local repositories + +1. Clone and start [GitLab](https://gitlab.com/gitlab-org/gitlab-development-kit). +1. Clone and start [Versions Application](https://gitlab.com/gitlab-org/gitlab-services/version.gitlab.com). + Make sure you run `docker-compose up` to start a PostgreSQL and Redis instance. +1. Point GitLab to the Versions Application endpoint instead of the default endpoint: + 1. Open [service_ping/submit_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L5) locally and modify `STAGING_BASE_URL`. + 1. Set it to the local Versions Application URL: `http://localhost:3000`. + +### Test local setup + +1. Using the `gitlab` Rails console, manually trigger Service Ping: + + ```ruby + GitlabServicePingWorker.new.perform('triggered_from_cron' => false) + ``` + +1. Use the `versions` Rails console to check the Service Ping was successfully received, + parsed, and stored in the Versions database: + + ```ruby + UsageData.last + ``` + +## Test Prometheus-based Service Ping + +If the data submitted includes metrics [queried from Prometheus](../metrics/metrics_instrumentation.md#prometheus-metrics) +you want to inspect and verify, you must: + +- Ensure that a Prometheus server is running locally. +- Ensure the respective GitLab components are exporting metrics to the Prometheus server. + +If you do not need to test data coming from Prometheus, no further action +is necessary. Service Ping should degrade gracefully in the absence of a running Prometheus server. + +Three kinds of components may export data to Prometheus, and are included in Service Ping: + +- [`node_exporter`](https://github.com/prometheus/node_exporter): Exports node metrics + from the host machine. +- [`gitlab-exporter`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-exporter): Exports process metrics + from various GitLab components. +- Other various GitLab services, such as Sidekiq and the Rails server, which export their own metrics. + +### Test with an Omnibus container + +This is the recommended approach to test Prometheus-based Service Ping. + +To verify your change, build a new Omnibus image from your code branch using CI/CD, download the image, +and run a local container instance: + +1. From your merge request, select the `qa` stage, then trigger the `e2e:package-and-test` job. This job triggers an Omnibus + build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). +1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. +1. Open the job logs and locate the full container name including the version. It takes the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. +1. On your local machine, make sure you are signed in to the GitLab Docker registry. You can find the instructions for this in + [Authenticate to the GitLab Container Registry](../../../user/packages/container_registry/authenticate_with_container_registry.md). +1. Once signed in, download the new image by using `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` +1. For more information about working with and running Omnibus GitLab containers in Docker, refer to [GitLab Docker images](../../../install/docker.md) documentation. + +### Test with GitLab development toolkits + +This is the less recommended approach, because it comes with a number of difficulties when emulating a real GitLab deployment. + +The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would +like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start. + +The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Service Ping. +By default, it comes with a fully configured Prometheus service that is set up to scrape a number of components. +However, it has the following limitations: + +- It does not run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing. +- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it usually reports itself as not associated + with any of the other running services. That is not how node metrics are reported in a production setup, where `node_exporter` + always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore + appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping. + ## Service Ping Payload drop ### Symptoms diff --git a/doc/development/internal_analytics/service_ping/usage_data.md b/doc/development/internal_analytics/service_ping/usage_data.md deleted file mode 100644 index 8742bc03fbb..00000000000 --- a/doc/development/internal_analytics/service_ping/usage_data.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Usage Data Metrics guide - -This guide describes deprecated usage for metrics in `usage_data.rb`. - -NOTE: -Implementing metrics direct in `usage_data.rb` is deprecated, We recommend you use [instrumentation classes](metrics_instrumentation.md). - -## Ordinary batch counters - -Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors. -Handles the `ActiveRecord::StatementInvalid` error. - -Method: - -```ruby -count(relation, column = nil, batch: true, start: nil, finish: nil) -``` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the count on, by default is the primary key -- `batch`: default `true` to use batch counting -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - -Examples: - -```ruby -count(User.active) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) -``` - -## Distinct batch counters - -Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors. -Handles the `ActiveRecord::StatementInvalid` error. - -Method: - -```ruby -distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil) -``` - -Arguments: - -- `relation`: the ActiveRecord_Relation to perform the count -- `column`: the column to perform the distinct count, by default is the primary key -- `batch`: default `true` to use batch counting -- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - -WARNING: -Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../../database/iterating_tables_in_batches.md) guide. - -Examples: - -```ruby -distinct_count(::Project, :creator_id) -distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -``` |