diff options
Diffstat (limited to 'doc/development/internal_analytics/snowplow')
7 files changed, 7 insertions, 1228 deletions
diff --git a/doc/development/internal_analytics/snowplow/event_dictionary_guide.md b/doc/development/internal_analytics/snowplow/event_dictionary_guide.md deleted file mode 100644 index c0d5e3efdfa..00000000000 --- a/doc/development/internal_analytics/snowplow/event_dictionary_guide.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Event dictionary guide - -NOTE: -The event dictionary is a work in progress, and this process is subject to change. - -This guide describes the event dictionary and how it's implemented. - -## Event definition and validation - -This process is meant to document all Snowplow events and ensure consistency. Every Snowplow event needs to have such a definition. Event definitions must comply with the [JSON Schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/events/schema.json). - -All event definitions are stored in the following directories: - -- [`config/events`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/events) -- [`ee/config/events`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/config/events) - -Each event is defined in a separate YAML file consisting of the following fields: - -| Field | Required | Additional information | -|------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `description` | yes | A description of the event. | -| `category` | yes | The event category (see [Event schema](index.md#event-schema)). | -| `action` | yes | The event action (see [Event schema](index.md#event-schema)). | -| `label_description` | no | A description of the event label (see [Event schema](index.md#event-schema)). | -| `property_description` | no | A description of the event property (see [Event schema](index.md#event-schema)). | -| `value_description` | no | A description of the event value (see [Event schema](index.md#event-schema)). | -| `extra_properties` | no | The type and description of each extra property sent with the event. | -| `identifiers` | no | A list of identifiers sent with the event. Can be set to one or more of `project`, `user`, or `namespace`. | -| `iglu_schema_url` | no | The URL to the custom schema sent with the event, for example, `iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0`. | -| `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | -| `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the event. | -| `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the event. | -| `milestone` | no | The milestone when the event is introduced. | -| `introduced_by_url` | no | The URL to the merge request that introduced the event. | -| `distributions` | yes | The [distributions](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/#definitions) where the tracked feature is available. Can be set to one or more of `ce` or `ee`. | -| `tiers` | yes | The [tiers](https://about.gitlab.com/handbook/marketing/brand-and-product-marketing/product-and-solution-marketing/tiers/) where the tracked feature is available. Can be set to one or more of `free`, `premium`, or `ultimate`. | - -### Example event definition - -The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/events/epics_promote.yml) -YAML file includes an example event definition. - -```yaml -description: Issue promoted to epic -category: epics -action: promote -property_description: The string "issue_id" -value_description: ID of the issue -extra_properties: - weight: - type: integer - description: Weight of the issue -identifiers: -- project -- user -- namespace -product_section: dev -product_stage: plan -product_group: group::product planning -milestone: "11.10" -introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/10537 -distributions: -- ee -tiers: -- premium -- ultimate -``` - -## Create a new event definition - -Use the dedicated [event definition generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/snowplow_event_definition_generator.rb) -to create new event definitions. - -The `category` and `action` of each event are included in the filename to standardize file naming. - -The generator takes three options: - -- `--ee`: Indicates if the event is for EE. -- `--category=CATEGORY`: Indicates the `category` of the event. -- `--action=ACTION`: Indicates the `action` of the event. - -```shell -bundle exec rails generate gitlab:snowplow_event_definition --category Groups::EmailCampaignsController --action click -create create config/events/groups__email_campaigns_controller_click.yml -``` diff --git a/doc/development/internal_analytics/snowplow/implementation.md b/doc/development/internal_analytics/snowplow/implementation.md deleted file mode 100644 index 5d328f22ca5..00000000000 --- a/doc/development/internal_analytics/snowplow/implementation.md +++ /dev/null @@ -1,523 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Implement Snowplow tracking - -This page describes how to: - -- Implement Snowplow frontend and backend tracking -- Test Snowplow events - -## Event definitions - -Every Snowplow event, regardless of frontend or backend, requires a corresponding event definition. These definitions document the event and its properties to make it easier to maintain and analyze. -These definitions can be browsed in the [event dictionary](https://metrics.gitlab.com/snowplow/). The [event dictionary guide](event_dictionary_guide.md) provides instructions for setting up an event definition. - -## Snowplow JavaScript frontend tracking - -GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) -to track custom events. - -For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). - -Structured events and page views include the [`gitlab_standard`](schemas.md#gitlab_standard) -context, using the `window.gl.snowplowStandardContext` object which includes -[default data](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/views/layouts/_snowplow.html.haml) -as base: - -| Property | Example | -| -------- | ------- | -| `context_generated_at` | `"2022-01-01T01:00:00.000Z"` | -| `environment` | `"production"` | -| `extra` | `{}` | -| `namespace_id` | `123` | -| `plan` | `"gold"` | -| `project_id` | `456` | -| `source` | `"gitlab-rails"` | -| `user_id` | `789`* | -| `is_gitlab_team_member` | `true`| - -_\* Undergoes a pseudonymization process at the collector level._ - -These properties [are overridden](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/get_standard_context.js) -with frontend-specific values, like `source` (`gitlab-javascript`), `google_analytics_id` -and the custom `extra` object. You can modify this object for any subsequent -structured event that fires, although this is not recommended. - -Tracking implementations must have an `action` and a `category`. You can provide additional -properties from the [event schema](index.md#event-schema), in -addition to an `extra` object that accepts key-value pairs. - -| Property | Type | Default value | Description | -|:-----------|:-------|:---------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured. | -| `action` | string | `'generic'` | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value` as described in [Event schema](index.md#event-schema), `context` for custom contexts, and `extra` (key-value pairs object). | - -### Usage recommendations - -- Use [data attributes](#implement-data-attribute-tracking) on HTML elements that emit `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. -- Use the [Vue mixin](#implement-vue-component-tracking) for tracking custom events, or if the supported events for data attributes are not propagating. For example, clickable components that don't emit `click`. -- Use the [tracking class](#implement-raw-javascript-tracking) when tracking in vanilla JavaScript files. - -### Implement data attribute tracking - -To implement tracking for HAML or Vue templates, add a [`data-track` attribute](#data-track-attributes) to the element. - -The following example shows `data-track-*` attributes assigned to a button: - -```haml -%button.btn{ data: { track_action: "click_button", track_label: "template_preview", track_property: "my-template" } } -``` - -```html -<button class="btn" - data-track-action="click_button" - data-track-label="template_preview" - data-track-property="my-template" - data-track-extra='{ "template_variant": "primary" }' -/> -``` - -#### `data-track` attributes - -| Attribute | Required | Description | -|:----------------------|:---------|:------------| -| `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | -| `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list; or the name or title attribute of a record being created. | -| `data-track-property` | false | Any additional property of the element, or object being acted on. | -| `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. The value is parsed as numeric before sending the event. | -| `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | -| `data-track-context` | false | To append a custom context object, passed as a valid JSON string. | - -#### Event listeners - -Event listeners bind at the document level to handle click events in elements with data attributes. -This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces -the likelihood that click events stop propagating up the DOM tree. - -If click events stop propagating, you must implement listeners and [Vue component tracking](#implement-vue-component-tracking) or [raw JavaScript tracking](#implement-raw-javascript-tracking). - -#### Helper methods - -You can use the following Ruby helpers: - -```ruby -tracking_attrs(label, action, property) # { data: { track_label... } } - -tracking_attrs_data(label, action, property) # { track_label... } -``` - -You can also use it on HAML templates: - -```haml -%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } - -// When merging with additional data -// %button{ data: { platform: "...", **tracking_attrs_data('main_navigation', 'click_button', 'navigation') } } -``` - -If you use the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76), you must wrap `html_options` under the `html_options` keyword argument. If you -use the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to), you don't need to wrap `html_options`. - -```ruby -# Bad -= nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", -track_action: "click_button" }) - -# Good -= nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: -"explore_groups", track_action: "click_button" } }) - -# Good (other helpers) -= link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: -"click_button" } -``` - -### Implement Vue component tracking - -For custom event tracking, use the [Vue mixin](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/tracking.js#L207). It exposes `Tracking.event` as the `track` method. -You can specify tracking options by creating a `tracking` data object or -computed property, and as a second parameter: `this.track('click_button', opts)`. -These options override any defaults and allow the values to be dynamic from props or based on state: - -| Property | Type | Default | Example | -| -- | -- | -- | -- | -| `category` | string | `document.body.dataset.page` | `'code_quality_walkthrough'` | -| `label` | string | `''` | `'process_start_button'` | -| `property` | string | `''` | `'asc'` or `'desc'` | -| `value` | integer | `undefined` | `0`, `1`, `500` | -| `extra` | object | `{}` | `{ selectedVariant: this.variant }` | - -To implement Vue component tracking: - -1. Import the `Tracking` library and call the `mixin` method: - - ```javascript - import Tracking from '~/tracking'; - - const trackingMixin = Tracking.mixin(); - - // Optionally provide default properties - // const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); - ``` - -1. Use the mixin in the component: - - ```javascript - export default { - mixins: [trackingMixin], - // Or - // mixins: [Tracking.mixin()], - // mixins: [Tracking.mixin({ label: 'right_sidebar' })], - - data() { - return { - expanded: false, - }; - }, - }; - ``` - -1. You can specify tracking options in by creating a `tracking` data object -or computed property: - - ```javascript - export default { - name: 'RightSidebar', - - mixins: [Tracking.mixin()], - - data() { - return { - expanded: false, - variant: '', - tracking: { - label: 'right_sidebar', - // property: '', - // value: '', - // experiment: '', - // extra: {}, - }, - }; - }, - - // Or - // computed: { - // tracking() { - // return { - // property: this.variant, - // extra: { expanded: this.expanded }, - // }; - // }, - // }, - }; - ``` - -1. Call the `track` method. Tracking options can be passed as the second parameter: - - ```javascript - this.track('click_button', { - label: 'right_sidebar', - }); - ``` - - Or use the `track` method in the template: - - ```html - <template> - <div> - <button data-testid="toggle" @click="toggle">Toggle</button> - - <div v-if="expanded"> - <p>Hello world!</p> - <button @click="track('click_button')">Track another event</button> - </div> - </div> - </template> - ``` - -#### Testing example - -```javascript -export default { - name: 'CountDropdown', - - mixins: [Tracking.mixin({ label: 'count_dropdown' })], - - data() { - return { - variant: 'counter', - count: 0, - }; - }, - - methods: { - handleChange({ target }) { - const { variant } = this; - - this.count = Number(target.value); - - this.track('change_value', { - value: this.count, - extra: { variant } - }); - }, - }, -}; -``` - -```javascript -import { mockTracking } from 'helpers/tracking_helper'; -// mockTracking(category, documentOverride, spyMethod) - -describe('CountDropdown.vue', () => { - let trackingSpy; - let wrapper; - - ... - - beforeEach(() => { - trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); - }); - - const findDropdown = () => wrapper.find('[data-testid="dropdown"]'); - - it('tracks change event', () => { - const dropdown = findDropdown(); - dropdown.element.value = 30; - dropdown.trigger('change'); - - expect(trackingSpy).toHaveBeenCalledWith(undefined, 'change_value', { - value: 30, - label: 'count_dropdown', - extra: { variant: 'counter' }, - }); - }); -}); -``` - -### Implement raw JavaScript tracking - -To track from a vanilla JavaScript file, use the `Tracking.event` static function -(calls [`dispatchSnowplowEvent`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/dispatch_snowplow_event.js)). - -The following example demonstrates tracking a click on a button by manually calling `Tracking.event`. - -```javascript -import Tracking from '~/tracking'; - -const button = document.getElementById('create_from_template_button'); - -button.addEventListener('click', () => { - Tracking.event(undefined, 'click_button', { - label: 'create_from_template', - property: 'template_preview', - extra: { - templateVariant: 'primary', - valid: 1, - }, - }); -}); -``` - -#### Testing example - -```javascript -import Tracking from '~/tracking'; - -describe('MyTracking', () => { - let wrapper; - - beforeEach(() => { - jest.spyOn(Tracking, 'event'); - }); - - const findButton = () => wrapper.find('[data-testid="create_from_template"]'); - - it('tracks event', () => { - findButton().trigger('click'); - - expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { - label: 'create_from_template', - property: 'template_preview', - extra: { - templateVariant: 'primary', - valid: true, - }, - }); - }); -}); -``` - -### Form tracking - -To enable Snowplow automatic [form tracking](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): - -1. Call `Tracking.enableFormTracking` when the DOM is ready. -1. Provide a `config` object that includes at least one of the following elements: - - `forms` determines the forms to track. Identified by the CSS class name. - - `fields` determines the fields inside the tracked forms to track. Identified by the field `name`. -1. Optional. Provide a list of contexts as the second argument. The [`gitlab_standard`](schemas.md#gitlab_standard) schema is excluded from these events. - -```javascript -Tracking.enableFormTracking({ - forms: { allow: ['sign-in-form', 'password-recovery-form'] }, - fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, -}); -``` - -#### Testing example - -```javascript -import Tracking from '~/tracking'; - -describe('MyFormTracking', () => { - let formTrackingSpy; - - beforeEach(() => { - formTrackingSpy = jest - .spyOn(Tracking, 'enableFormTracking') - .mockImplementation(() => null); - }); - - it('initialized with the correct configuration', () => { - expect(formTrackingSpy).toHaveBeenCalledWith({ - forms: { allow: ['sign-in-form', 'password-recovery-form'] }, - fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, - }); - }); -}); -``` - -## Implement Ruby backend tracking - -`Gitlab::Tracking` is an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. -Backend tracking provides: - -- User behavior tracking -- Instrumentation to monitor and visualize performance over time in a section or aspect of code. - -To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. -For example: - -```ruby -class Projects::CreateService < BaseService - def execute - project = Project.create(params) - - Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, - property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) - end -end -``` - -Use the following arguments: - -| Argument | Type | Default value | Description | -|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| -| `category` | String | | Area or aspect of the application. For example, `HealthCheckController` or `Lfs::FileTransformer`. | -| `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | -| `label` | String | `nil` | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list; or the name or title attribute of a record being created. | -| `property` | String | `nil` | Any additional property of the element, or object being acted on. | -| `value` | Numeric | `nil` | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | -| `context` | Array\[SelfDescribingJSON\] | `nil` | An array of custom contexts to send with this event. Most events should not have any custom contexts. | -| `project` | Project | `nil` | The project associated with the event. | -| `user` | User | `nil` | The user associated with the event. This value undergoes a pseudonymization process at the collector level. | -| `namespace` | Namespace | `nil` | The namespace associated with the event. | -| `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | - -### Unit testing - -To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see -[testing best practices](../../testing_guide/best_practices.md#test-snowplow-events). - -### Performance - -We use the [AsyncEmitter](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/AsyncEmitter.html) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. - -## Develop and test Snowplow - -To develop and test a Snowplow event, there are several tools to test frontend and backend events: - -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | Yes | No | Yes | Yes | Yes | -| Snowplow Inspector Chrome Extension | Yes | No | Yes | Yes | Yes | -| Snowplow Micro | Yes | Yes | Yes | No | No | - -### Test frontend events - -Before you test frontend events in development, you must: - -1. [Enable Snowplow tracking in the Admin Area](index.md#enable-snowplow-tracking). -1. Turn off ad blockers that could prevent Snowplow JavaScript from loading in your environment. -1. Turn off "Do Not Track" (DNT) in your browser. - -All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#setting-a-custom-page-url-and-referrer-url) personally identifiable -information (PII). PII includes usernames, group, and project names. -Page titles are hardcoded as `GitLab` for the same reason. - -#### Snowplow Analytics Debugger Chrome Extension - -[Snowplow Analytics Debugger](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html) is a browser extension for testing frontend events. It works in production, staging, and local development environments. - -1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. - -#### Snowplow Inspector Chrome Extension - -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works in production, staging, and local development environments. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a video tutorial, see the [Snowplow plugin walk through](https://www.youtube.com/watch?v=g4rqnIZ1Mb4). - -1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). -1. To open the extension, select the Snowplow Inspector icon beside the address bar. -1. Click around on a webpage with Snowplow to see JavaScript events firing in the inspector window. - -### Test backend events with Snowplow Micro - -[Snowplow Micro](https://snowplow.io/blog/introducing-snowplow-micro/) is a -Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro -records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. - -It can be set up automatically using [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit). -See the [how-to docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/snowplow_micro.md) for more details. - -1. Set the environment variable to tell the GDK to use Snowplow Micro in development. This overrides two `application_settings` options: - - `snowplow_enabled` setting will instead return `true` from `Gitlab::Tracking.enabled?` - - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever port is set for `snowplow_micro.port` GDK setting) from `Gitlab::Tracking.collector_hostname`. -With Snowplow Micro set up you can now manually test backend Snowplow events: - -1. Send a test Snowplow event from the Rails console: - - ```ruby - Gitlab::Tracking.event('category', 'action') - ``` - -1. Navigate to `localhost:9090/micro/good` to see the event. - -#### Useful links - -- [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- [Installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) - -### Troubleshoot - -To control content security policy warnings when using an external host, modify `config/gitlab.yml` -to allow or prevent them. To allow them, add the relevant host for `connect_src`. For example, for -`https://snowplow.trx.gitlab.net`: - -```yaml -development: - <<: *base - gitlab: - content_security_policy: - enabled: true - directives: - connect_src: "'self' http://localhost:* http://127.0.0.1:* ws://localhost:* wss://localhost:* ws://127.0.0.1:* https://snowplow.trx.gitlab.net/" -``` diff --git a/doc/development/internal_analytics/snowplow/index.md b/doc/development/internal_analytics/snowplow/index.md deleted file mode 100644 index 17d3f3f2cfc..00000000000 --- a/doc/development/internal_analytics/snowplow/index.md +++ /dev/null @@ -1,201 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Snowplow development guidelines - -Snowplow is an enterprise-grade marketing and Analytics Instrumentation platform that tracks how users engage with our website and application. - -[Snowplow](https://snowplow.io/) consists of several loosely-coupled sub-systems: - -- **Trackers** fire Snowplow events. Snowplow has twelve trackers that cover web, mobile, desktop, server, and IoT. -- **Collectors** receive Snowplow events from trackers. We use different event collectors that synchronize events to Amazon S3, Apache Kafka, or Amazon Kinesis. -- **Enrich** cleans raw Snowplow events, enriches them, and puts them into storage. There is a Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. -- **Storage** stores Snowplow events. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. -- **Data modeling** joins event-level data with other data sets, aggregates them into smaller data sets, and applies business logic. This produces a clean set of tables for data analysis. We use data models for Redshift and Looker. -- **Analytics** are performed on Snowplow events or on aggregate tables. - - - -## Enable Snowplow tracking - -Tracking can be enabled at: - -- The instance level, which enables tracking on both the frontend and backend layers. -- The user level. User tracking can be disabled on a per user basis. - GitLab respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. - -Snowplow tracking is configured to send data for GitLab.com to a collector configured by GitLab. By default, self-managed -instances do not have a collector configured and do not collect data via Snowplow. - -You can configure your self-managed GitLab instance to use a custom Snowplow collector. - -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Settings > General**. -1. Expand **Snowplow**. -1. Select **Enable Snowplow tracking** and enter your Snowplow configuration information. For example: - - | Name | Value | - |--------------------|-------------------------------| - | Collector hostname | `your-snowplow-collector.net` | - | App ID | `gitlab` | - | Cookie domain | `.your-gitlab-instance.com` | - -1. Select **Save changes**. - -## Snowplow request flow - -The following example shows a basic request/response flow between the following components: - -- Snowplow JS / Ruby Trackers on GitLab.com -- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) -- The GitLab S3 Bucket -- The GitLab Snowflake Data Warehouse -- Sisense: - -```mermaid -sequenceDiagram - participant Snowplow JS (Frontend) - participant Snowplow Ruby (Backend) - participant GitLab.com Snowplow Collector - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event - Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event - loop Process using Kinesis Stream - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk - end - GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose - Note over GitLab.com Snowplow Collector, S3 Bucket: Pseudonymization - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying -``` - -For more details about the architecture, see [Snowplow infrastructure](infrastructure.md). - -## Event schema - -All the events must be consistent. If each feature captures events differently, it can be difficult -to perform analysis. - -Each event provides attributes that describe the event. - -| Attribute | Type | Required | Description | -| --------- | ------- | -------- | ----------- | -| category | text | true | The page or backend section of the application. Unless infeasible, use the Rails page attribute by default in the frontend, and namespace + class name on the backend, for example, `Notes::CreateService`. | -| action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown list is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | -| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list; or the name or title attribute of a record being created. For Service Ping metrics adapted to Snowplow events, this should be the full metric [key path](../service_ping/metrics_dictionary.md#metric-key_path) taken from its definition file. | -| property | text | false | Any additional property of the element, or object being acted on. For Service Ping metrics adapted to Snowplow events, this should be additional information or context that can help analyze the event. For example, in the case of `usage_activity_by_stage_monthly.create.merge_requests_users`, there are four different possible merge request actions: "create", "merge", "comment", and "close". Each of these would be a possible property value. | -| value | decimal | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | -| context | vector | false | Additional data in the form of a [self-describing JSON](https://docs.snowplow.io/docs/pipeline-components-and-applications/iglu/common-architecture/self-describing-json-schemas/) to describe the event if the attributes are not sufficient. Each context must have its schema defined to assure data integrity. Refer to the list of GitLab-defined contexts for more details. | - -### Examples - -| Category* | Label | Action | Property** | Value | -|-------------|------------------|-----------------------|----------|:-----:| -| `[root:index]` | `main_navigation` | `click_navigation_link` | `[link_label]` | - | -| `[groups:boards:show]` | `toggle_swimlanes` | `click_toggle_button` | - | `[is_active]` | -| `[projects:registry:index]` | `registry_delete` | `click_button` | - | - | -| `[projects:registry:index]` | `registry_delete` | `confirm_deletion` | - | - | -| `[projects:blob:show]` | `congratulate_first_pipeline` | `click_button` | `[human_access]` | - | -| `[projects:clusters:new]` | `chart_options` | `generate_link` | `[chart_link]` | - | -| `[projects:clusters:new]` | `chart_options` | `click_add_label_button` | `[label_id]` | - | -| `API::NpmPackages` | `counts.package_events_i_package_push_package_by_deploy_token` | `push_package` | `npm` | - | - -_* If you choose to omit the category you can use the default._<br> -_** Use property for variable strings._ - -### Reference SQL - -#### Last 20 `reply_comment_button` events - -```sql -SELECT - session_id, - event_id, - event_label, - event_action, - event_property, - event_value, - event_category, - contexts -FROM legacy.snowplow_structured_events_all -WHERE - event_label = 'reply_comment_button' - AND event_action = 'click_button' - -- AND event_category = 'projects:issues:show' - -- AND event_value = 1 -ORDER BY collector_tstamp DESC -LIMIT 20 -``` - -#### Last 100 page view events - -```sql -SELECT - -- page_url, - -- page_title, - -- referer_url, - -- marketing_medium, - -- marketing_source, - -- marketing_campaign, - -- browser_window_width, - -- device_is_mobile - * -FROM legacy.snowplow_page_views_30 -ORDER BY page_view_start DESC -LIMIT 100 -``` - -#### Top 20 users who fired `reply_comment_button` in the last 30 days - -```sql -SELECT - count(*) as hits, - se_action, - se_category, - gsc_pseudonymized_user_id -FROM legacy.snowplow_gitlab_events_30 -WHERE - se_label = 'reply_comment_button' - AND gsc_pseudonymized_user_id IS NOT NULL -GROUP BY gsc_pseudonymized_user_id, se_category, se_action -ORDER BY count(*) DESC -LIMIT 20 -``` - -#### Query JSON formatted data - -```sql -SELECT - derived_tstamp, - contexts:data[0]:data:extra:old_format as CURRENT_FORMAT, - contexts:data[0]:data:extra:value as UPDATED_FORMAT -FROM legacy.snowplow_structured_events_all -WHERE event_action in ('wiki_format_updated') -ORDER BY derived_tstamp DESC -LIMIT 100 -``` - -### Web-specific parameters - -Snowplow JavaScript adds [web-specific parameters](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. - -## Related topics - -- [Snowplow data structure](https://docs.snowplow.io/docs/understanding-your-pipeline/canonical-event/) -- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) -- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow/) -- [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) -- [Service Ping Guide](../service_ping/index.md) -- [Analytics Instrumentation Direction](https://about.gitlab.com/direction/analytics/analytics-instrumentation/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/internal_analytics/snowplow/infrastructure.md b/doc/development/internal_analytics/snowplow/infrastructure.md deleted file mode 100644 index b856ccec78e..00000000000 --- a/doc/development/internal_analytics/snowplow/infrastructure.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Snowplow infrastructure - -Snowplow events on GitLab SaaS fired by a [tracker](implementation.md) go through an AWS pipeline, managed by GitLab. - -## Event flow in the AWS pipeline - -Every event goes through a collector, enricher, and pseudonymization lambda. The event is then dumped to S3 storage where it can be picked up by the Snowflake data warehouse. - -Deploying and managing the infrastructure is automated using Terraform in the current [Terraform repository](https://gitlab.com/gitlab-com/gl-infra/config-mgmt/-/tree/master/environments/aws-snowplow). - -```mermaid -graph LR - GL[GitLab.com]-->COL - - subgraph aws-cloud[AWS] - COL[Collector]-->|snowplow-raw-good|ENR - COL[Collector]-->|snowplow-raw-bad|FRBE - subgraph firehoserbe[Firehose] - FRBE[AWS Lambda] - end - FRBE-->S3RBE - - ENR[Enricher]-->|snowplow-enriched-bad|FEBE - subgraph firehoseebe[Firehose] - FEBE[AWS Lambda] - end - FEBE-->S3EBE - - ENR[Enricher]-->|snowplow-enriched-good|FRGE - subgraph firehosege[Firehose] - FRGE[AWS Lambda] - end - FRGE-->S3GE - end - - subgraph snowflake[Data warehouse] - S3RBE[S3 raw-bad]-->BE[gitlab_bad_events] - S3EBE[S3 enriched-bad]-->BE[gitlab_bad_events] - S3GE[S3 output]-->GE[gitlab_events] - end -``` - -See [Snowplow technology 101](https://github.com/snowplow/snowplow/#snowplow-technology-101) for Snowplow's own documentation and an overview how collectors and enrichers work. - -### Pseudonymization - -In contrast to a typical Snowplow pipeline, after enrichment, GitLab Snowplow events go through a [pseudonymization service](https://gitlab.com/gitlab-org/analytics-section/analytics-instrumentation/snowplow-pseudonymization) in the form of an AWS Lambda service before they are stored in S3 storage. - -#### Why events need to be pseudonymized - -GitLab is bound by its [obligations to community](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/service-usage-data-commitment/) -and by [legal regulations](https://about.gitlab.com/handbook/legal/privacy/customer-product-usage-information/) to protect the privacy of its users. - -GitLab must provide valuable insights for business decisions, and there is a need -for a better understanding of different users' behavior patterns. The -pseudonymization process helps you find a compromise between these two requirements. - -Pseudonymization processes personally identifiable information inside a Snowplow event in an irreversible fashion -maintaining deterministic output for given input, while masking any relation to that input. - -#### How events are pseudonymized - -Pseudonymization uses an allowlist that provides privacy by default. Therefore, each -attribute received as part of a Snowplow event is pseudonymized unless the attribute -is an allowed exception. - -Pseudonymization is done using the HMAC-SHA256 keyed hash algorithm. -Attributes are combined with a secret salt to replace each identifiable information with a pseudonym. - -### S3 bucket data lake to Snowflake - -See [Data team's Snowplow Overview](https://about.gitlab.com/handbook/business-technology/data-team/platform/snowplow/) for further details how data is ingested into our Snowflake data warehouse. - -## Monitoring - -There are several tools that monitor Snowplow events tracking in different stages of the processing pipeline: - -- [Analytics Instrumentation Grafana dashboard](https://dashboards.gitlab.net/d/product-intelligence-main/product-intelligence-product-intelligence?orgId=1) monitors backend events sent from a GitLab.com instance to a collectors fleet. This dashboard provides information about: - - The number of events that successfully reach Snowplow collectors. - - The number of events that failed to reach Snowplow collectors. - - The number of backend events that were sent. -- [AWS CloudWatch dashboard](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=SnowPlow;start=P3D) monitors the state of the events in a processing pipeline. The pipeline starts from Snowplow collectors, goes through to enrichers and pseudonymization, and then up to persistence in an S3 bucket. From S3, the events are imported into the Snowflake Data Warehouse. You must have AWS access rights to view this dashboard. For more information, see [monitoring](https://gitlab.com/gitlab-org/analytics-section/analytics-instrumentation/snowplow-pseudonymization#monitoring) in the Snowplow Events pseudonymization service documentation. -- [Sisense dashboard](https://app.periscopedata.com/app/gitlab/417669/Snowplow-Summary-Dashboard) provides information about the number of good and bad events imported into the Data Warehouse, in addition to the total number of imported Snowplow events. - -For more information, see this [video walk-through](https://www.youtube.com/watch?v=NxPS0aKa_oU). - -## Related topics - -- [Snowplow technology 101](https://github.com/snowplow/snowplow/#snowplow-technology-101) -- [Snowplow pseudonymization AWS Lambda project](https://gitlab.com/gitlab-org/analytics-section/analytics-instrumentation/snowplow-pseudonymization) -- [Analytics Instrumentation Guide](https://about.gitlab.com/handbook/product/analytics-instrumentation-guide/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) -- [Snowplow architecture overview (internal)](https://www.youtube.com/watch?v=eVYJjzspsLU) -- [Snowplow architecture overview slide deck (internal)](https://docs.google.com/presentation/d/16gQEO5CAg8Tx4NBtfnZj-GF4juFI6HfEPWcZgH4Rn14/edit?usp=sharing) -- [AWS Lambda implementation (internal)](https://youtu.be/cQd0mdMhkQA) diff --git a/doc/development/internal_analytics/snowplow/review_guidelines.md b/doc/development/internal_analytics/snowplow/review_guidelines.md index a0bdad8fafb..0ca7b084fc4 100644 --- a/doc/development/internal_analytics/snowplow/review_guidelines.md +++ b/doc/development/internal_analytics/snowplow/review_guidelines.md @@ -1,44 +1,11 @@ --- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../review_guidelines.md' +remove_date: '2023-12-29' --- -# Snowplow review guidelines +This document was moved to [another location](../review_guidelines.md). -This page includes introductory material for an -[Analytics Instrumentation](https://about.gitlab.com/handbook/engineering/development/analytics/analytics-instrumentation/) -review, and is specific to Snowplow related reviews. For broader advice and -general best practices for code reviews, refer to our [code review guide](../../code_review.md). - -## Resources for reviewers - -- [Snowplow Guide](index.md) -- [Event Dictionary](https://metrics.gitlab.com/snowplow/) - -## Review process - -We recommend an Analytics Instrumentation review when a merge request (MR) involves changes in -events or touches Snowplow related files. - -### Roles and process - -#### The merge request **author** should - -- For frontend events, when relevant, add a screenshot of the event in - the [testing tool](implementation.md#develop-and-test-snowplow) used. -- For backend events, when relevant, add the output of the - [Snowplow Micro](implementation.md#test-backend-events-with-snowplow-micro) good events - `GET http://localhost:9090/micro/good` (it might be a good idea - to reset with `GET http://localhost:9090/micro/reset` first). -- Add or update the event definition file according to the [Event Dictionary Guide](event_dictionary_guide.md). - -#### The Analytics Instrumentation **reviewer** should - -- Check that the [event schema](index.md#event-schema) is correct. -- Check the [usage recommendations](implementation.md#usage-recommendations). -- Check that an event definition file was created or updated in accordance with the [Event Dictionary Guide](event_dictionary_guide.md). -- If needed, check that the events are firing locally using one of the -[testing tools](implementation.md#develop-and-test-snowplow) available. -- Approve the MR, and relabel the MR with `~"analytics instrumentation::approved"`. -- If the snowplow event mirrors a RedisHLL event, then tag @mdrussell to review if the payload is usable for this purpose. +<!-- This redirect file can be deleted after <2023-12-29>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/internal_analytics/snowplow/schemas.md b/doc/development/internal_analytics/snowplow/schemas.md deleted file mode 100644 index 2cc09500c36..00000000000 --- a/doc/development/internal_analytics/snowplow/schemas.md +++ /dev/null @@ -1,190 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Snowplow schemas - -This page provides Snowplow schema reference for GitLab events. - -## `gitlab_standard` - -We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) for structured events and page views. - -The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) -class represents this schema in the application. Some properties are -[automatically populated for frontend events](implementation.md#snowplow-javascript-frontend-tracking), -and can be [provided manually for backend events](implementation.md#implement-ruby-backend-tracking). - -| Field Name | Required | Default value | Type | Description | -|-------------------------|:-------------------:|------------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------| -| `project_id` | **{dotted-circle}** | Current project ID * | integer | | -| `namespace_id` | **{dotted-circle}** | Current group/namespace ID * | integer | | -| `user_id` | **{dotted-circle}** | Current user ID * | integer | User database record ID attribute. This value undergoes a pseudonymization process at the collector level. | -| `context_generated_at` | **{dotted-circle}** | Current timestamp | string (date time format) | Timestamp indicating when context was generated. | -| `environment` | **{check-circle}** | Current environment | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | -| `source` | **{check-circle}** | Event source | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | -| `plan` | **{dotted-circle}** | Current namespace plan * | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | -| `google_analytics_id` | **{dotted-circle}** | GA ID value * | string (max 32 chars) | Google Analytics ID, present when set from our marketing sites. | -| `is_gitlab_team_member` | **{dotted-circle}** | | boolean | Indicates if the events is triggered by a GitLab team member | -| `extra` | **{dotted-circle}** | | JSON | Any additional data associated with the event, in the form of key-value pairs | - -_\* Default value present for frontend events only_ - -## Default Schema - -Frontend events include a [web-specific schema](https://docs.snowplow.io/docs/understanding-your-pipeline/canonical-event/#web-specific-fields) provided by Snowplow. -All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#setting-a-custom-page-url-and-referrer-url) personally identifiable -information (PII). PII includes usernames, group, and project names. -Page titles are hardcoded as `GitLab` for the same reason. - -| Field Name | Required | Type | Description | -|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| -| `app_id` | **{check-circle}** | string | Unique identifier for website / application | -| `base_currency` | **{dotted-circle}** | string | Reporting currency | -| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | -| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | -| `br_family` | **{dotted-circle}** | string | Browser family | -| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | -| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | -| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | -| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | -| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | -| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | -| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | -| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | -| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | -| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | -| `br_name` | **{dotted-circle}** | string | Browser name | -| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | -| `br_type` | **{dotted-circle}** | string | Browser type | -| `br_version` | **{dotted-circle}** | string | Browser version | -| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | -| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | -| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | -| `contexts` | **{dotted-circle}** | | | -| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | -| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | -| `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | -| `doc_height` | **{dotted-circle}** | string | Web page height | -| `doc_width` | **{dotted-circle}** | string | Web page width | -| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this `user_id` to this domain | -| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this `user_id` has made to this domain (The first visit is `1`) | -| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | -| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | -| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | -| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | -| `dvce_type` | **{dotted-circle}** | string | Type of device | -| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | -| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | -| `event` | **{dotted-circle}** | string | Event type | -| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | -| `event_format` | **{dotted-circle}** | string | Format for event | -| `event_id` | **{dotted-circle}** | string | Event UUID | -| `event_name` | **{dotted-circle}** | string | Event name | -| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | -| `event_version` | **{dotted-circle}** | string | Version of event schema | -| `geo_city` | **{dotted-circle}** | string | City of IP origin | -| `geo_country` | **{dotted-circle}** | string | Country of IP origin | -| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | -| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | -| `geo_region` | **{dotted-circle}** | string | Region of IP origin | -| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | -| `geo_timezone` | **{dotted-circle}** | string | Time zone of IP origin | -| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | -| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor's IP address | -| `ip_isp` | **{dotted-circle}** | string | Visitor's ISP | -| `ip_netspeed` | **{dotted-circle}** | string | Visitor's connection type | -| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor's IP address – defaults to ISP name if none is found | -| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | -| `mkt_clickid` | **{dotted-circle}** | string | The click ID | -| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | -| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | -| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | -| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | -| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | -| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | -| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn't be set by a tracker) | -| `os_family` | **{dotted-circle}** | string | Operating system family | -| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | -| `os_name` | **{dotted-circle}** | string | Name of operating system | -| `os_timezone` | **{dotted-circle}** | string | Client operating system time zone | -| `page_referrer` | **{dotted-circle}** | string | Referrer URL | -| `page_title` | **{dotted-circle}** | string | To not expose personal identifying information, the page title is hardcoded as `GitLab` | -| `page_url` | **{dotted-circle}** | string | Page URL | -| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | -| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | -| `page_urlpath` | **{dotted-circle}** | string | Path to page | -| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | -| `page_urlquery` | **{dotted-circle}** | string | Query string | -| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | -| `platform` | **{dotted-circle}** | string | The platform the app runs on | -| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | -| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | -| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | -| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | -| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | -| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | -| `refr_medium` | **{dotted-circle}** | string | Type of referer | -| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | -| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | -| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | -| `refr_urlhost` | **{dotted-circle}** | string | Referer host | -| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | -| `refr_urlport` | **{dotted-circle}** | integer | Referer port | -| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | -| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | -| `se_action` | **{dotted-circle}** | string | The action / event itself | -| `se_category` | **{dotted-circle}** | string | The category of event | -| `se_label` | **{dotted-circle}** | string | A label often used to refer to the 'object' the action is performed on | -| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | -| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | -| `ti_category` | **{dotted-circle}** | string | Item category | -| `ti_currency` | **{dotted-circle}** | string | Currency | -| `ti_name` | **{dotted-circle}** | string | Item name | -| `ti_orderid` | **{dotted-circle}** | string | Order ID | -| `ti_price` | **{dotted-circle}** | decimal | Item price | -| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | -| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | -| `ti_sku` | **{dotted-circle}** | string | Item SKU | -| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | -| `tr_city` | **{dotted-circle}** | string | Delivery address: city | -| `tr_country` | **{dotted-circle}** | string | Delivery address: country | -| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | -| `tr_orderid` | **{dotted-circle}** | string | Order ID | -| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | -| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | -| `tr_state` | **{dotted-circle}** | string | Delivery address: state | -| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | -| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | -| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | -| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | -| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | -| `txn_id` | **{dotted-circle}** | string | Transaction ID | -| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | -| `uploaded_at` | **{dotted-circle}** | | | -| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | -| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | -| `user_ipaddress` | **{dotted-circle}** | string | IP address | -| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | -| `v_collector` | **{dotted-circle}** | string | Collector version | -| `v_etl` | **{dotted-circle}** | string | ETL version | -| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | - -## `gitlab_service_ping` - -Backend events converted from ServicePing (`redis` and `redis_hll`) must include [ServicePing context](https://gitlab.com/gitlab-org/iglu/-/tree/master/public/schemas/com.gitlab/gitlab_service_ping/jsonschema) -using the [helper class](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/tracking/service_ping_context.rb). - -An example of converted `redis_hll` [event with context](https://gitlab.com/gitlab-org/gitlab/-/edit/master/app/controllers/concerns/product_analytics_tracking.rb#L58). - -| Field Name | Required | Type | Description | -|---------------|:-------------------:|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `data_source` | **{check-circle}** | string (max 64 chars) | The `data_source` attribute from the metrics YAML definition. | -| `event_name`* | **{dotted-circle}** | string (max 128 chars) | When there is a many-to-many relationship between events and metrics, this field contains the name of a Redis event that can be used for aggregations in downstream systems | -| `key_path`* | **{dotted-circle}** | string (max 256 chars) | The `key_path` attribute from the metrics YAML definition | - -_\* Either `event_name` or `key_path` is required_ diff --git a/doc/development/internal_analytics/snowplow/troubleshooting.md b/doc/development/internal_analytics/snowplow/troubleshooting.md deleted file mode 100644 index 5eabba04792..00000000000 --- a/doc/development/internal_analytics/snowplow/troubleshooting.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -stage: Analyze -group: Analytics Instrumentation -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments ---- - -# Troubleshooting Snowplow - -## Monitoring - -This page covers dashboards and alerts coming from a number of internal tools. - -For a brief video overview of the tools used to monitor Snowplow usage, please check out [this internal video](https://www.youtube.com/watch?v=NxPS0aKa_oU) (you must be logged into GitLab Unfiltered to view). - -## Good events drop - -### Symptoms - -You will be alarmed via a [Sisense alert](https://app.periscopedata.com/app/gitlab/alert/Volume-of-Snowplow-Good-events/5a5f80ef34fe450da5ebb84eaa84067f/edit) that is sent to `#g_product_intelligence` Slack channel - -### Locating the problem - -First you need to identify at which stage in Snowplow the data pipeline the drop is occurring. -Start at [Snowplow dashboard](https://console.aws.amazon.com/systems-manager/resource-groups/cloudwatch?dashboard=SnowPlow®ion=us-east-1#) on CloudWatch. -If you do not have access to CloudWatch, GitLab team members can create an access request issue, similar to this one: `https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/9730`. -While on CloudWatch dashboard set time range to last 4 weeks, to get better picture of system characteristics over time. Than visit following charts: - -1. `ELB New Flow Count` and `Collector Auto Scaling Group Network In/Out` - they show in order: number of connections to collectors via load balancers and data volume (in bytes) processed by collectors. If there is drop visible there, it means less events were fired from the GitLab application. Proceed to [application layer guide](#troubleshooting-gitlab-application-layer) for more details -1. `Firehose Records to S3` - it shows how many event records were saved to S3 bucket, if there was drop on this chart but not on the charts from 1. it means that problem is located at AWS infrastructure layer, please refer to [AWS layer guide](#troubleshooting-aws-layer) -1. If drop wasn't visible on any of previous charts it means that problem is at data warehouse layer, please refer to [data warehouse layer guide](#troubleshooting-data-warehouse-layer) - -### Troubleshooting GitLab application layer - -Drop occurring at application layer can be symptom of some issue, but it might be also a result of typical application lifecycle, intended changes done to analytics instrumentation or experiments tracking -or even a result of a public holiday in some regions of the world with a larger user-base. To verify if there is an underlying problem to solve, you can check following things: - -1. Check `about.gitlab.com` website traffic on [Google Analytics](https://analytics.google.com/analytics/web/) to verify if some public holiday might impact overall use of GitLab system - 1. You may require to open an access request for Google Analytics access first, for example: [access request internal issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/1772) -1. Plot `select date(dvce_created_tstamp) , event , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-10' group by 1 , 2 order by 1 , 2` in SiSense to see what type of events was responsible for drop -1. Plot `select date(dvce_created_tstamp) ,se_category , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-31' and event = 'struct' group by 1 , 2 order by 1, 2` what events recorded the biggest drops in suspected category -1. Check if there was any MR merged that might cause reduction in reported events, pay an attention to ~"analytics instrumentation" and ~"growth experiment" labeled MRs -1. Check (via [Grafana explore tab](https://dashboards.gitlab.net/explore) ) following Prometheus counters `gitlab_snowplow_events_total`, `gitlab_snowplow_failed_events_total` and `gitlab_snowplow_successful_events_total` to see how many events were fired correctly from GitLab.com. Example query to use `sum(rate(gitlab_snowplow_successful_events_total{env="gprd"}[5m])) / sum(rate(gitlab_snowplow_events_total{env="gprd"}[5m]))` would chart rate at which number of good events rose in comparison to events sent in total. If number drops from 1 it means that problem might be in communication between GitLab and AWS collectors fleet. -1. Check [logs in Kibana](https://log.gprd.gitlab.net/app/discover#) and filter with `{ "query": { "match_phrase": { "json.message": "failed to be reported to collector at" } } }` if there are some failed events logged - -We conducted an investigation into an unexpected drop in snowplow events volume. - -GitLab team members can view more information in this confidential issue: `https://gitlab.com/gitlab-org/gitlab/-/issues/335206` - -### Troubleshooting AWS layer - -Already conducted investigations: - -- [Steep decrease of Snowplow page views](https://gitlab.com/gitlab-org/gitlab/-/issues/268009) -- [`snowplow.trx.gitlab.net` unreachable](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5073) - -### Troubleshooting data warehouse layer - -Reach out to [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us) - -## Delay in Snowplow Enrichers - -If there is an alert for **Snowplow Raw Good Stream Backing Up**, we receive an email notification. This sometimes happens because Snowplow Enrichers don't scale well enough for the amount of Snowplow events. - -If the delay goes over 48 hours, we lose data. - -### Contact SRE on-call - -Send a message in the [#infrastructure_lounge](https://gitlab.slack.com/archives/CB3LSMEJV) Slack channel using the following template: - -```markdown -Hello team! - -We received an alert for [Snowplow Raw Good Stream Backing Up](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#alarmsV2:alarm/SnowPlow+Raw+Good+Stream+Backing+Up?). - -Enrichers are not scalling well for the amount of events we receive. - -See the [dashboard](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=SnowPlow). - -Could we get assistance to fix the delay? - -Thank you! -``` |