diff options
Diffstat (limited to 'doc/development/service_ping')
| -rw-r--r-- | doc/development/service_ping/implement.md | 15 | ||||
| -rw-r--r-- | doc/development/service_ping/index.md | 17 | ||||
| -rw-r--r-- | doc/development/service_ping/metrics_instrumentation.md | 26 | ||||
| -rw-r--r-- | doc/development/service_ping/troubleshooting.md | 6 |
4 files changed, 33 insertions, 31 deletions
diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 0ebc58dd669..5448bbb4293 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -127,7 +127,7 @@ Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recomme #### Grouping and batch operations -The `count`, `distinct_count`, `sum`, and `average` batch counters can accept an `ActiveRecord::Relation` +The `count`, `distinct_count` and `sum` batch counters can accept an `ActiveRecord::Relation` object, which groups by a specified column. With a grouped relation, the methods do batch counting, handle errors, and returns a hash table of key-value pairs. @@ -142,9 +142,6 @@ distinct_count(Project.group(:visibility_level), :creator_id) sum(Issue.group(:state_id), :weight)) # returns => {1=>3542, 2=>6820} - -average(Issue.group(:state_id), :weight)) -# returns => {1=>3.5, 2=>2.5} ``` #### Add operation @@ -275,7 +272,7 @@ Events are handled by counter classes in the `Gitlab::UsageDataCounters` namespa 1. Listed in [`Gitlab::UsageDataCounters::COUNTERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters.rb#L5) to be then included in `Gitlab::UsageData`. -1. Specified in the metric definition using the `RedisMetric` instrumentation class as a `counter_class` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation. +1. Specified in the metric definition using the `RedisMetric` instrumentation class by their `prefix` option to be picked up using the [metric instrumentation](metrics_instrumentation.md) framework. Refer to the [Redis metrics](metrics_instrumentation.md#redis-metrics) documentation for an example implementation. Inheriting classes are expected to override `KNOWN_EVENTS` and `PREFIX` constants to build event names and associated metrics. For example, for prefix `issues` and events array `%w[create, update, delete]`, three metrics will be added to the Service Ping payload: `counts.issues_create`, `counts.issues_update` and `counts.issues_delete`. @@ -385,7 +382,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd/) and [P - `controller_actions`: the controller actions to track. - `name`: the event name. - `conditions`: optional custom conditions. Uses the same format as Rails callbacks. - - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: [:redis_hll]. + - `destinations`: optional list of destinations. Currently supports `:redis_hll` and `:snowplow`. Default: `:redis_hll`. - `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`. Example: @@ -623,7 +620,7 @@ alt_usage_data(999) ### Add counters to build new metrics When adding the results of two counters, use the `add` Service Data method that -handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-sql-queries-and-definitions). +handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-data). Example: @@ -773,7 +770,7 @@ To set up Service Ping locally, you must: 1. Using the `gitlab` Rails console, manually trigger Service Ping: ```ruby - ServicePing::SubmitService.new.execute + GitlabServicePingWorker.new.perform('triggered_from_cron' => false) ``` 1. Use the `versions` Rails console to check the Service Ping was successfully received, @@ -809,7 +806,7 @@ This is the recommended approach to test Prometheus-based Service Ping. To verify your change, build a new Omnibus image from your code branch using CI/CD, download the image, and run a local container instance: -1. From your merge request, select the `qa` stage, then trigger the `package-and-qa` job. This job triggers an Omnibus +1. From your merge request, select the `qa` stage, then trigger the `e2e:package-and-test` job. This job triggers an Omnibus build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). 1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. 1. Open the job logs and locate the full container name including the version. It takes the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 4481fe33bda..f252eb967aa 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -374,9 +374,9 @@ Possible values are "Amazon Aurora PostgreSQL", "PostgreSQL on Amazon RDS", "Clo In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/). -## Export Service Ping SQL queries and definitions +## Export Service Ping data -Two Rake tasks exist to export Service Ping definitions. +Rake tasks exist to export Service Ping data in different formats. - The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. - The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. @@ -385,12 +385,15 @@ Two Rake tasks exist to export Service Ping definitions. In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: ```shell -# for YAML export +# for YAML export of SQL queries bin/rake gitlab:usage_data:dump_sql_in_yaml -# for JSON export +# for JSON export of SQL queries bin/rake gitlab:usage_data:dump_sql_in_json +# for JSON export of Non SQL data +bin/rake gitlab:usage_data:dump_non_sql_in_json + # You may pipe the output into a file bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml ``` @@ -405,7 +408,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment. 1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval). -1. Run `ServicePing::SubmitService.new.execute`. +1. Run `GitlabServicePingWorker.new.perform('triggered_from_cron' => false)`. #### Trigger Service Ping with a detached screen session @@ -430,7 +433,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Run: ```shell - ServicePing::SubmitService.new.execute + GitlabServicePingWorker.new.perform('triggered_from_cron' => false) ``` 1. To detach from screen, press `ctrl + A`, `ctrl + D`. @@ -490,7 +493,7 @@ To skip database write operations, DevOps report creation, and storage of usage ```shell skip_db_write: -ServicePing::SubmitService.new(skip_db_write: true).execute +GitlabServicePingWorker.new.perform('triggered_from_cron' => false, 'skip_db_write' => true) ``` ## Monitoring diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 9dc37386111..debb3a68c04 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -154,22 +154,24 @@ end You can use Redis metrics to track events not kept in the database, for example, a count of how many times the search bar has been used. -[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66582). +[Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97009). + +Please note that `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class. Count unique values for `source_code_pushes` event. Required options: - `event`: the event name. -- `counter_class`: one of the counter classes from the `Gitlab::UsageDataCounters` namespace; it should implement `read` method or inherit it from `BaseCounter`. +- `prefix`: the value of the `PREFIX` constant used in the counter classes from the `Gitlab::UsageDataCounters` namespace. ```yaml time_frame: all data_source: redis -instrumentation_class: 'RedisMetric' +instrumentation_class: RedisMetric options: event: pushes - counter_class: SourceCodeCounter + prefix: source_code ``` ### Availability-restrained Redis metrics @@ -197,14 +199,14 @@ You must also use the class's name in the YAML setup. ```yaml time_frame: all data_source: redis -instrumentation_class: 'MergeUsageCountRedisMetric' +instrumentation_class: MergeUsageCountRedisMetric options: event: pushes - counter_class: SourceCodeCounter + prefix: source_code ``` ## Redis HyperLogLog metrics - + You can use Redis HyperLogLog metrics to track events not kept in the database and incremented for unique values such as unique users, for example, a count of how many different users used the search bar. @@ -215,7 +217,7 @@ Count unique values for `i_quickactions_approve` event. ```yaml time_frame: 28d data_source: redis_hll -instrumentation_class: 'RedisHLLMetric' +instrumentation_class: RedisHLLMetric options: events: - i_quickactions_approve @@ -246,7 +248,7 @@ You must also use the class's name in the YAML setup. ```yaml time_frame: 28d data_source: redis_hll -instrumentation_class: 'MergeUsageCountRedisHLLMetric' +instrumentation_class: MergeUsageCountRedisHLLMetric options: events: - i_quickactions_approve @@ -286,7 +288,7 @@ You must also include the instrumentation class name in the YAML setup. ```yaml time_frame: 28d -instrumentation_class: 'IssuesBoardsCountMetric' +instrumentation_class: IssuesBoardsCountMetric ``` ## Generic metrics @@ -337,13 +339,13 @@ To create a stub instrumentation for a Service Ping metric, you can use a dedica The generator takes the class name as an argument and the following options: - `--type=TYPE` Required. Indicates the metric type. It must be one of: `database`, `generic`, `redis`, `numbers`. -- `--operation` Required for `database` & `numebers` type. +- `--operation` Required for `database` & `numbers` type. - For `database` it must be one of: `count`, `distinct_count`, `estimate_batch_distinct_count`, `sum`, `average`. - For `numbers` it must be: `add`. - `--ee` Indicates if the metric is for EE. ```shell -rails generate gitlab:usage_metric CountIssues --type database +rails generate gitlab:usage_metric CountIssues --type database --operation distinct_count create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb ``` diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 29ab334f867..4431d5df3ff 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -4,7 +4,7 @@ group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting +# Troubleshooting Service Ping ## Service Ping Payload drop @@ -58,7 +58,7 @@ checking the configuration file of your GitLab instance: - Using the Admin Area: - 1. On the top bar, select **Menu > Admin**. + 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage Statistics**. 1. Are you able to check or uncheck the checkbox to disable Service Ping? @@ -115,7 +115,7 @@ To work around this bug, you have two options: sudo gitlab-ctl reconfigure ``` - 1. In GitLab, on the top bar, select **Menu > Admin**. + 1. In GitLab, on the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage Statistics**. 1. Clear the **Enable Service Ping** checkbox. |