diff options
Diffstat (limited to 'doc/development/service_ping/index.md')
| -rw-r--r-- | doc/development/service_ping/index.md | 512 |
1 files changed, 7 insertions, 505 deletions
diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 01772b194a4..d0806ed375b 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -1,509 +1,11 @@ --- -stage: Analytics -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../internal_analytics/service_ping/index.md' +remove_date: '2023-08-20' --- -# Service Ping development guidelines +This document was moved to [another location](../internal_analytics/service_ping/index.md). -> - Introduced in GitLab Ultimate 11.2, more statistics. -> - In GitLab 14.1, [renamed from Usage Ping to Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/5990). In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version. - -Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. -The payload provides important high-level data that helps our product, support, -and sales teams understand how GitLab is used. The data helps to: - -- Compare counts month over month (or week over week) to get a rough sense for how an instance uses - different product features. -- Collect other facts that help us classify and understand GitLab installations. -- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages - and features. - -Service Ping information is not anonymous. It's linked to the instance's hostname, but does -not contain project names, usernames, or any other specific data. - -Service Ping is enabled by default. However, you can [disable](../../user/admin_area/settings/usage_statistics.md#enable-or-disable-usage-statistics) it on any self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance's usage statistics to your users. - -## Service Ping terminology - -We use the following terminology to describe the Service Ping components: - -- **Service Ping**: the process that collects and generates a JSON payload. -- **Service Data**: the contents of the Service Ping JSON payload. This includes metrics. -- **Metrics**: primarily made up of row counts for different tables in an instance's database. Each - metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) - in a YAML file. -- **MAU**: monthly active users. -- **WAU**: weekly active users. - -### Limitations - -- Service Ping does not track frontend events things like page views, link clicks, or user sessions. -- Service Ping focuses only on aggregated backend events. - -Because of these limitations we recommend you: - -- Instrument your products with Snowplow for more detailed analytics on GitLab.com. -- Use Service Ping to track aggregated backend events on self-managed instances. - -## Service Ping request flow - -The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, the GitLab S3 Bucket, the GitLab Snowflake Data Warehouse, and Sisense: - -```mermaid -sequenceDiagram - participant GitLab Instance - participant Versions Application - participant Licenses Application - participant Salesforce - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - GitLab Instance->>Versions Application: Send Service Ping - loop Process usage data - Versions Application->>Versions Application: Parse usage data - Versions Application->>Versions Application: Write to database - Versions Application->>Versions Application: Update license ping time - end - loop Process data for Salesforce - Versions Application-xLicenses Application: Request Zuora subscription id - Licenses Application-xVersions Application: Zuora subscription id - Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id - Salesforce-xVersions Application: Zuora account id - Versions Application-xSalesforce: Usage data for the Zuora account - end - Versions Application->>S3 Bucket: Export Versions database - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying - Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) -``` - -## How Service Ping works - -1. The Service Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_service_ping_worker.rb#L24) is set in Sidekiq to run weekly. -1. When the cron job runs, it calls [`Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb). -1. `Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) to ~400+ other counter method calls. -1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L68) into a single JSON payload. -1. The JSON payload is then [posted to the Versions application](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L20) - If a firewall exception is needed, the required URL depends on several things. If - the hostname is `version.gitlab.com`, the protocol is `TCP`, and the port number is `443`, - the required URL is <https://version.gitlab.com/>. -1. In case of an error, it will be reported to the Version application along with following pieces of information: - - - `uuid` - GitLab instance unique identifier - - `hostname` - GitLab instance hostname - - `version` - GitLab instance current versions - - `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence - - `message` - Error message - - <pre> - <code> - { - "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", - "hostname"=>"127.0.0.1", - "version"=>"14.7.0-pre", - "elapsed"=>0.006946, - "message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...' - } - </code> - </pre> - -1. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics: - - > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag](../../user/feature_flags.md), enabled by default. - > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/295289) in GitLab 15.2. [Feature flag `measure_service_ping_metric_collection`](https://gitlab.com/gitlab-org/gitlab/-/issues/358128) removed. - -```ruby - { - "metadata"=> - { - "uuid"=>"0000000-0000-0000-0000-000000000000", - "metrics"=> - [{"name"=>"version", "time_elapsed"=>1.1811964213848114e-05}, - {"name"=>"installation_type", "time_elapsed"=>0.00017242692410945892}, - {"name"=>"license_billable_users", "time_elapsed"=>0.009520471096038818}, - .... - {"name"=>"counts.clusters_platforms_eks", - "time_elapsed"=>0.05638605775311589}, - {"name"=>"counts.clusters_platforms_gke", - "time_elapsed"=>0.40995341585949063}, - {"name"=>"counts.clusters_platforms_user", - "time_elapsed"=>0.06410990096628666}, - {"name"=>"counts.clusters_management_project", - "time_elapsed"=>0.24020783510059118} - ] - } - } -``` - -### On a Geo secondary site - -We also collect metrics specific to [Geo](../../administration/geo/index.md) secondary sites to send with Service Ping. - -1. The [Geo secondary service ping cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/geo/secondary_usage_data_cron_worker.rb) is set in Sidekiq to run weekly. -1. When the cron job runs, it calls [`SecondaryUsageData.update_metrics!`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo/secondary_usage_data.rb#L33). This collects the relevant metrics from Prometheus and stores the data in the Geo secondary tracking database for transmission to the primary site during a [Geo node status update](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node_status.rb#L105). -1. Geo node status data is sent with the JSON payload in the process described above. The following is an example of the payload where each object in the array represents a Geo node: - - ```json - [ - { - "repository_verification_enabled"=>true, - "repositories_replication_enabled"=>true, - "repositories_synced_count"=>24, - "repositories_failed_count"=>0, - "git_fetch_event_count_weekly"=>nil, - "git_push_event_count_weekly"=>nil, - ... other geo node status fields - } - ] - ``` - -## Implementing Service Ping - -See the [implement Service Ping](implement.md) guide. - -## Example Service Ping payload - -The following is example content of the Service Ping payload. - -```json -{ - "uuid": "0000000-0000-0000-0000-000000000000", - "hostname": "example.com", - "version": "12.10.0-pre", - "installation_type": "omnibus-gitlab", - "active_user_count": 999, - "recorded_at": "2020-04-17T07:43:54.162+00:00", - "edition": "EEU", - "license_md5": "00000000000000000000000000000000", - "license_sha256": "0000000000000000000000000000000000000000000000000000000000000000", - "license_id": null, - "historical_max_users": 999, - "licensee": { - "Name": "ABC, Inc.", - "Email": "email@example.com", - "Company": "ABC, Inc." - }, - "license_user_count": 999, - "license_starts_at": "2020-01-01", - "license_expires_at": "2021-01-01", - "license_plan": "ultimate", - "license_add_ons": { - }, - "license_trial": false, - "counts": { - "assignee_lists": 999, - "boards": 999, - "ci_builds": 999, - ... - }, - "container_registry_enabled": true, - "dependency_proxy_enabled": false, - "gitlab_shared_runners_enabled": true, - "gravatar_enabled": true, - "influxdb_metrics_enabled": true, - "ldap_enabled": false, - "mattermost_enabled": false, - "omniauth_enabled": true, - "prometheus_enabled": false, - "prometheus_metrics_enabled": false, - "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", - "signup_enabled": true, - "projects_with_expiration_policy_disabled": 999, - "projects_with_expiration_policy_enabled": 999, - ... - "elasticsearch_enabled": true, - "license_trial_ends_on": null, - "geo_enabled": false, - "git": { - "version": { - "major": 2, - "minor": 26, - "patch": 1 - } - }, - "gitaly": { - "version": "12.10.0-rc1-93-g40980d40", - "servers": 56, - "clusters": 14, - "filesystems": [ - "EXT_2_3_4" - ] - }, - "gitlab_pages": { - "enabled": true, - "version": "1.17.0" - }, - "container_registry_server": { - "vendor": "gitlab", - "version": "2.9.1-gitlab" - }, - "database": { - "adapter": "postgresql", - "version": "9.6.15", - "pg_system_id": 6842684531675334351, - "flavor": "Cloud SQL for PostgreSQL" - }, - "analytics_unique_visits": { - "g_analytics_contribution": 999, - ... - }, - "usage_activity_by_stage": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - }, - "usage_activity_by_stage_monthly": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - }, - "topology": { - "duration_s": 0.013836685999194742, - "application_requests_per_hour": 4224, - "query_apdex_weekly_average": 0.996, - "failures": [], - "nodes": [ - { - "node_memory_total_bytes": 33269903360, - "node_memory_utilization": 0.35, - "node_cpus": 16, - "node_cpu_utilization": 0.2, - "node_uname_info": { - "machine": "x86_64", - "sysname": "Linux", - "release": "4.19.76-linuxkit" - }, - "node_services": [ - { - "name": "web", - "process_count": 16, - "process_memory_pss": 233349888, - "process_memory_rss": 788220927, - "process_memory_uss": 195295487, - "server": "puma" - }, - { - "name": "sidekiq", - "process_count": 1, - "process_memory_pss": 734080000, - "process_memory_rss": 750051328, - "process_memory_uss": 731533312 - }, - ... - ], - ... - }, - ... - ] - } -} -``` - -## Notable changes - -In GitLab 14.6, [`flavor`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75587) was added to try to detect the underlying managed database variant. -Possible values are "Amazon Aurora PostgreSQL", "PostgreSQL on Amazon RDS", "Cloud SQL for PostgreSQL", -"Azure Database for PostgreSQL - Flexible Server", or "null". - -In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/). - -## Export Service Ping data - -Rake tasks exist to export Service Ping data in different formats. - -- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. -- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. -- The Rake tasks calculate the `alt_usage_data` metrics. - -In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: - -```shell -# for YAML export of SQL queries -bin/rake gitlab:usage_data:dump_sql_in_yaml - -# for JSON export of SQL queries -bin/rake gitlab:usage_data:dump_sql_in_json - -# for JSON export of Non SQL data -bin/rake gitlab:usage_data:dump_non_sql_in_json - -# You may pipe the output into a file -bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml -``` - -## Generate Service Ping - -To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server. - -### Triggering - -#### Trigger Service Ping with Teleport - -1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment. -1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval). -1. Run `GitlabServicePingWorker.new.perform('triggered_from_cron' => false)`. - -#### Trigger Service Ping with a detached screen session - -1. Connect to bastion with agent forwarding: - - ```shell - ssh -A lb-bastion.gprd.gitlab.com - ``` - -1. Create named screen: - - ```shell - screen -S <username>_usage_ping_<date> - ``` - -1. Connect to console host: - - ```shell - ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal - ``` - -1. Run: - - ```shell - GitlabServicePingWorker.new.perform('triggered_from_cron' => false) - ``` - -1. To detach from screen, press `ctrl + A`, `ctrl + D`. -1. Exit from bastion: - - ```shell - exit - ``` - -1. Get the metrics duration from logs: - -Search in Google Console logs for `time_elapsed`. [Query example](https://cloudlogging.app.goo.gl/nWheZvD8D3nWazNe6). - -### Verification (After approx 30 hours) - -#### Verify with Teleport - -1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console -1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` -1. Check the when the payload was sent: `RawUsageData.last.sent_at` - -#### Verify using detached screen session - -1. Reconnect to bastion: - - ```shell - ssh -A lb-bastion.gprd.gitlab.com - ``` - -1. Find your screen session: - - ```shell - screen -ls - ``` - -1. Attach to your screen session: - - ```shell - screen -x 14226.mwawrzyniak_usage_ping_2021_01_22 - ``` - -1. Check the last payload in `raw_usage_data` table: - - ```shell - RawUsageData.last.payload - ``` - -1. Check the when the payload was sent: - - ```shell - RawUsageData.last.sent_at - ``` - -### Skip database write operations - -To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument: - -```shell -skip_db_write: -GitlabServicePingWorker.new.perform('triggered_from_cron' => false, 'skip_db_write' => true) -``` - -## Monitoring - -Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489). - -## Related topics - -- [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) -- [Snowplow Guide](../snowplow/index.md) -- [Analytics Instrumentation Direction](https://about.gitlab.com/direction/analytics/analytics-instrumentation/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) +<!-- This redirect file can be deleted after <2023-08-20>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html |