diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-03-16 21:18:33 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-03-16 21:18:33 +0300 |
commit | f64a639bcfa1fc2bc89ca7db268f594306edfd7c (patch) | |
tree | a2c3c2ebcc3b45e596949db485d6ed18ffaacfa1 /doc/development/experiment_guide | |
parent | bfbc3e0d6583ea1a91f627528bedc3d65ba4b10f (diff) |
Add latest changes from gitlab-org/gitlab@13-10-stable-eev13.10.0-rc40
Diffstat (limited to 'doc/development/experiment_guide')
-rw-r--r-- | doc/development/experiment_guide/experimentation.md | 399 | ||||
-rw-r--r-- | doc/development/experiment_guide/gitlab_experiment.md | 547 | ||||
-rw-r--r-- | doc/development/experiment_guide/index.md | 397 |
3 files changed, 965 insertions, 378 deletions
diff --git a/doc/development/experiment_guide/experimentation.md b/doc/development/experiment_guide/experimentation.md new file mode 100644 index 00000000000..7135f8acd9b --- /dev/null +++ b/doc/development/experiment_guide/experimentation.md @@ -0,0 +1,399 @@ +--- +stage: Growth +group: Activation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Create an A/B test with `Experimentation Module` + +## Implement the experiment + +1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in + [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): + + ```ruby + EXPERIMENTS = { + other_experiment: { + #... + }, + # Add your experiment here: + signup_flow: { + tracking_category: 'Growth::Activation::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data + } + }.freeze + ``` + +1. Use the experiment in the code. + + Experiments can be performed on a `subject`. The provided `subject` should + respond to `to_global_id` or `to_s`. + The resulting string is bucketed and assigned to either the control or the + experimental group, so you must always provide the same `subject` + for an experiment to have the same experience. + + 1. Use this standard for the experiment in a controller: + + - Experiment run for a user: + + ```ruby + class ProjectController < ApplicationController + def show + # experiment_enabled?(:experiment_key) is also available in views and helpers + if experiment_enabled?(:signup_flow, subject: current_user) + # render the experiment + else + # render the original version + end + end + end + ``` + + - Experiment run for a namespace: + + ```ruby + if experiment_enabled?(:signup_flow, subject: namespace) + # experiment code + else + # control code + end + ``` + + When no subject is given, it falls back to a cookie that gets set and is consistent until + the cookie gets deleted. + + ```ruby + class RegistrationController < ApplicationController + def show + # falls back to a cookie + if experiment_enabled?(:signup_flow) + # render the experiment + else + # render the original version + end + end + end + ``` + + 1. Make the experiment available to the frontend in a controller. This example + checks whether the experiment is enabled and pushes the result to the frontend: + + ```ruby + before_action do + push_frontend_experiment(:signup_flow, subject: current_user) + end + ``` + + You can check the state of the feature flag in JavaScript: + + ```javascript + import { isExperimentEnabled } from '~/experimentation'; + + if ( isExperimentEnabled('signupFlow') ) { + // ... + } + ``` + +You can also run an experiment outside of the controller scope, such as in a worker: + +```ruby +class SomeWorker + def perform + # Check if the experiment is active at all (the percentage_of_time_value > 0) + return unless Gitlab::Experimentation.active?(:experiment_key) + + # Since we cannot access cookies in a worker, we need to bucket models + # based on a unique, unchanging attribute instead. + # It is therefore necessery to always provide the same subject. + if Gitlab::Experimentation.in_experiment_group?(:experiment_key, subject: user) + # execute experimental code + else + # execute control code + end + end +end +``` + +## Implement tracking events + +To determine whether the experiment is a success or not, we must implement tracking events +to acquire data for analyzing. We can send events to Snowplow via either the backend or frontend. +Read the [product intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for more details. + +### Track backend events + +The framework provides a helper method that is available in controllers: + +```ruby +before_action do + track_experiment_event(:signup_flow, 'action', 'value', subject: current_user) +end +``` + +To test it: + +```ruby +context 'when the experiment is active and the user is in the experimental group' do + before do + stub_experiment(signup_flow: true) + stub_experiment_for_subject(signup_flow: true) + end + + it 'tracks an event', :snowplow do + subject + + expect_snowplow_event( + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + ) + end +end +``` + +### Track frontend events + +The framework provides a helper method that is available in controllers: + +```ruby +before_action do + push_frontend_experiment(:signup_flow, subject: current_user) + frontend_experimentation_tracking_data(:signup_flow, 'action', 'value', subject: current_user) +end +``` + +This pushes tracking data to `gon.experiments` and `gon.tracking_data`. + +```ruby +expect(Gon.experiments['signupFlow']).to eq(true) + +expect(Gon.tracking_data).to eq( + { + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + } +) +``` + +To track it: + +```javascript +import { isExperimentEnabled } from '~/lib/utils/experimentation'; +import Tracking from '~/tracking'; + +document.addEventListener('DOMContentLoaded', () => { + const signupFlowExperimentEnabled = isExperimentEnabled('signupFlow'); + + if (signupFlowExperimentEnabled && gon.tracking_data) { + const { category, action, ...data } = gon.tracking_data; + + Tracking.event(category, action, data); + } +} +``` + +To test it in Jest: + +```javascript +import { withGonExperiment } from 'helpers/experimentation_helper'; +import Tracking from '~/tracking'; + +describe('event tracking', () => { + describe('with tracking data', () => { + withGonExperiment('signupFlow'); + + beforeEach(() => { + jest.spyOn(Tracking, 'event').mockImplementation(() => {}); + + gon.tracking_data = { + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + }; + }); + + it('should track data', () => { + performAction() + + expect(Tracking.event).toHaveBeenCalledWith( + 'Growth::Activation::Experiment::SignUpFlow', + 'action', + { + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + }, + ); + }); + }); +}); +``` + +## Record experiment user + +In addition to the anonymous tracking of events, we can also record which users +have participated in which experiments, and whether they were given the control +experience or the experimental experience. + +The `record_experiment_user` helper method is available to all controllers, and it +enables you to record these experiment participants (the current user) and which +experience they were given: + +```ruby +before_action do + record_experiment_user(:signup_flow) +end +``` + +Subsequent calls to this method for the same experiment and the same user have no +effect unless the user is then enrolled into a different experience. This happens +when we roll out the experimental experience to a greater percentage of users. + +This data is completely separate from the [events tracking data](#implement-tracking-events). +They are not linked together in any way. + +### Add context + +You can add arbitrary context data in a hash which gets stored as part of the experiment +user record. New calls to the `record_experiment_user` with newer contexts are merged +deeply into the existing context. + +This data can then be used by data analytics dashboards. + +```ruby +before_action do + record_experiment_user(:signup_flow, foo: 42, bar: { a: 22}) + # context is { "foo" => 42, "bar" => { "a" => 22 }} +end + +# Additional contexts for newer record calls are merged deeply +record_experiment_user(:signup_flow, foo: 40, bar: { b: 2 }, thor: 3) +# context becomes { "foo" => 40, "bar" => { "a" => 22, "b" => 2 }, "thor" => 3} +``` + +## Record experiment conversion event + +Along with the tracking of backend and frontend events and the +[recording of experiment participants](#record-experiment-user), we can also record +when a user performs the desired conversion event action. For example: + +- **Experimental experience:** Show an in-product nudge to test if the change causes more + people to sign up for trials. +- **Conversion event:** The user starts a trial. + +The `record_experiment_conversion_event` helper method is available to all controllers. +Use it to record the conversion event for the current user, regardless of whether +the user is in the control or experimental group: + +```ruby +before_action do + record_experiment_conversion_event(:signup_flow) +end +``` + +Note that the use of this method requires that we have first +[recorded the user](#record-experiment-user) as being part of the experiment. + +## Enable the experiment + +After all merge requests have been merged, use [ChatOps](../../ci/chatops/index.md) in the +[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. +The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. +For visibility, share any commands run against production in the `#s_growth` channel: + + ```shell + /chatops run feature set signup_flow_experiment_percentage 10 + ``` + + If you notice issues with the experiment, you can disable the experiment by removing the feature flag: + + ```shell + /chatops run feature delete signup_flow_experiment_percentage + ``` + +## Add user to experiment group manually + +To force the application to add your current user into the experiment group, +add a query string parameter to the path where the experiment runs. If you add the +query string parameter, the experiment works only for this request, and doesn't work +after following links or submitting forms. + +For example, to forcibly enable the `EXPERIMENT_KEY` experiment, add `force_experiment=EXPERIMENT_KEY` +to the URL: + +```shell +https://gitlab.com/<EXPERIMENT_ENTRY_URL>?force_experiment=<EXPERIMENT_KEY> +``` + +## Add user to experiment group with a cookie + +You can force the current user into the experiment group for `<EXPERIMENT_KEY>` +during the browser session by using your browser's developer tools: + +```javascript +document.cookie = "force_experiment=<EXPERIMENT_KEY>; path=/"; +``` + +Use a comma to list more than one experiment to be forced: + +```javascript +document.cookie = "force_experiment=<EXPERIMENT_KEY>,<ANOTHER_EXPERIMENT_KEY>; path=/"; +``` + +To clear the experiments, unset the `force_experiment` cookie: + +```javascript +document.cookie = "force_experiment=; path=/"; +``` + +## Testing and test helpers + +### RSpec + +Use the following in RSpec to mock the experiment: + +```ruby +context 'when the experiment is active' do + before do + stub_experiment(signup_flow: true) + end + + context 'when the user is in the experimental group' do + before do + stub_experiment_for_subject(signup_flow: true) + end + + it { is_expected.to do_experimental_thing } + end + + context 'when the user is in the control group' do + before do + stub_experiment_for_subject(signup_flow: false) + end + + it { is_expected.to do_control_thing } + end +end +``` + +### Jest + +Use the following in Jest to mock the experiment: + +```javascript +import { withGonExperiment } from 'helpers/experimentation_helper'; + +describe('given experiment is enabled', () => { + withGonExperiment('signupFlow'); + + it('should do the experimental thing', () => { + expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element)); + }); +}); +``` diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md new file mode 100644 index 00000000000..6b15449b812 --- /dev/null +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -0,0 +1,547 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Implementing an A/B/n experiment using GLEX + +## Introduction + +`Gitlab::Experiment` (GLEX) is tightly coupled with the concepts provided by +[Feature flags in development of GitLab](../feature_flags/index.md). Here, we refer +to this layer as feature flags, and may also use the term Flipper, because we +built our development and experiment feature flags atop it. + +You're strongly encouraged to read and understand the +[Feature flags in development of GitLab](../feature_flags/index.md) portion of the +documentation before considering running experiments. Experiments add additional +concepts which may seem confusing or advanced without understanding the underpinnings +of how GitLab uses feature flags in development. One concept: GLEX supports multivariate +experiments, which are sometimes referred to as A/B/n tests. + +The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment) +exists in a separate repository, so it can be shared across any GitLab property that uses +Ruby. You should feel comfortable reading the documentation on that project as well +if you want to dig into more advanced topics. + +## Glossary of terms + +To ensure a shared language, you should understand these fundamental terms we use +when communicating about experiments: + +- `experiment`: Any deviation of code paths we want to run at some times, but not others. +- `context`: A consistent experience we provide in an experiment. +- `control`: The default, or "original" code path. +- `candidate`: Defines an experiment with only one code path. +- `variant(s)`: Defines an experiment with multiple code paths. + +### How it works + +Use this decision tree diagram to understand how GLEX works. When an experiment runs, +the following logic is executed to determine what variant should be provided, +given how the experiment has been defined and using the provided context: + +```mermaid +graph TD + GP[General Pool/Population] --> Running? + Running? -->|Yes| Cached?[Cached? / Pre-segmented?] + Running? -->|No| Excluded[Control / No Tracking] + Cached? -->|No| Excluded? + Cached? -->|Yes| Cached[Cached Value] + Excluded? -->|Yes / Cached| Excluded + Excluded? -->|No| Segmented? + Segmented? -->|Yes / Cached| VariantA + Segmented? -->|No| Included?[Experiment Group?] + Included? -->|Yes| Rollout + Included? -->|No| Control + Rollout -->|Cached| VariantA + Rollout -->|Cached| VariantB + Rollout -->|Cached| VariantC + +classDef included fill:#380d75,color:#ffffff,stroke:none +classDef excluded fill:#fca121,stroke:none +classDef cached fill:#2e2e2e,color:#ffffff,stroke:none +classDef default fill:#fff,stroke:#6e49cb + +class VariantA,VariantB,VariantC included +class Control,Excluded excluded +class Cached cached +``` + +## Implement an experiment + +Start by generating a feature flag using the `bin/feature-flag` command as you +normally would for a development feature flag, making sure to use `experiment` for +the type. For the sake of documentation let's name our feature flag (and experiment) +"pill_color". + +```shell +bin/feature-flag pill_color -t experiment +``` + +After you generate the desired feature flag, you can immediately implement an +experiment in code. An experiment implementation can be as simple as: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.use { 'control' } + e.try(:red) { 'red' } + e.try(:blue) { 'blue' } +end +``` + +When this code executes, the experiment is run, a variant is assigned, and (if within a +controller or view) a `window.gon.experiment.pillColor` object will be available in the +client layer, with details like: + +- The assigned variant. +- The context key for client tracking events. + +In addition, when an experiment runs, an event is tracked for +the experiment `:assignment`. We cover more about events, tracking, and +the client layer later. + +In local development, you can make the experiment active by using the feature flag +interface. You can also target specific cases by providing the relevant experiment +to the call to enable the feature flag: + +```ruby +# Enable for everyone +Feature.enable(:pill_color) + +# Get the `experiment` method -- already available in controllers, views, and mailers. +include Gitlab::Experiment::Dsl +# Enable for only the first user +Feature.enable(:pill_color, experiment(:pill_color, actor: User.first)) +``` + +To roll out your experiment feature flag on an environment, run +the following command using ChatOps (which is covered in more depth in the +[Feature flags in development of GitLab](../feature_flags/index.md) documentation). +This command creates a scenario where half of everyone who encounters +the experiment would be assigned the _control_, 25% would be assigned the _red_ +variant, and 25% would be assigned the _blue_ variant: + +```slack +/chatops run feature set pill_color 50 --actors +``` + +For an even distribution in this example, change the command to set it to 66% instead +of 50. + +NOTE: +To immediately stop running an experiment, use the +`/chatops run feature set pill_color false` command. + +WARNING: +We strongly recommend using the `--actors` flag when using the ChatOps commands, +as anything else may give odd behaviors due to how the caching of variant assignment is +handled. + +We can also implement this experiment in a HAML file with HTML wrappings: + +```haml +#cta-interface + - experiment(:pill_color, actor: current_user) do |e| + - e.use do + .pill-button control + - e.try(:red) do + .pill-button.red red + - e.try(:blue) do + .pill-button.blue blue +``` + +### The importance of context + +In our previous example experiment, our context (this is an important term) is a hash +that's set to `{ actor: current_user }`. Context must be unique based on how you +want to run your experiment, and should be understood at a lower level. + +It's expected, and recommended, that you use some of these +contexts to simplify reporting: + +- `{ actor: current_user }`: Assigns a variant and is "sticky" to each user + (or "client" if `current_user` is nil) who enters the experiment. +- `{ project: project }`: Assigns a variant and is "sticky" to the project currently + being viewed. If running your experiment is more useful when viewing a project, + rather than when a specific user is viewing any project, consider this approach. +- `{ group: group }`: Similar to the project example, but applies to a wider + scope of projects and users. +- `{ actor: current_user, project: project }`: Assigns a variant and is "sticky" + to the user who is viewing the given project. This creates a different variant + assignment possibility for every project that `current_user` views. Understand this + can create a large cache size if an experiment like this in a highly trafficked part + of the application. +- `{ wday: Time.current.wday }`: Assigns a variant based on the current day of the + week. In this example, it would consistently assign one variant on Friday, and a + potentially different variant on Saturday. + +Context is critical to how you define and report on your experiment. It's usually +the most important aspect of how you choose to implement your experiment, so consider +it carefully, and discuss it with the wider team if needed. Also, take into account +that the context you choose affects our cache size. + +After the above examples, we can state the general case: *given a specific +and consistent context, we can provide a consistent experience and track events for +that experience.* To dive a bit deeper into the implementation details: a context key +is generated from the context that's provided. Use this context key to: + +- Determine the assigned variant. +- Identify events tracked against that context key. + +We can think about this as the experience that we've rendered, which is both dictated +and tracked by the context key. The context key is used to track the interaction and +results of the experience we've rendered to that context key. These concepts are +somewhat abstract and hard to understand initially, but this approach enables us to +communicate about experiments as something that's wider than just user behavior. + +NOTE: +Using `actor:` utilizes cookies if the `current_user` is nil. If you don't need +cookies though - meaning that the exposed functionality would only be visible to +signed in users - `{ user: current_user }` would be just as effective. + +WARNING: +The caching of variant assignment is done by using this context, and so consider +your impact on the cache size when defining your experiment. If you use +`{ time: Time.current }` you would be inflating the cache size every time the +experiment is run. Not only that, your experiment would not be "sticky" and events +wouldn't be resolvable. + +### Advanced experimentation + +GLEX allows for two general implementation styles: + +1. The simple experiment style described previously. +1. A more advanced style where an experiment class can be provided. + +The advanced style is handled by naming convention, and works similar to what you +would expect in Rails. + +To generate a custom experiment class that can override the defaults in +`ApplicationExperiment` (our base GLEX implementation), use the rails generator: + +```shell +rails generate gitlab:experiment pill_color control red blue +``` + +This generates an experiment class in `app/experiments/pill_color_experiment.rb` +with the variants (or _behaviors_) we've provided to the generator. Here's an example +of how that class would look after migrating the previous example into it: + +```ruby +class PillColorExperiment < ApplicationExperiment + def control_behavior + 'control' + end + + def red_behavior + 'red' + end + + def blue_behavior + 'blue' + end +end +``` + +We can now simplify where we run our experiment to the following call, instead of +providing the block we were initially providing, by explicitly calling `run`: + +```ruby +experiment(:pill_color, actor: current_user).run +``` + +The _behavior_ methods we defined in our experiment class represent the default +implementation. You can still use the block syntax to override these _behavior_ +methods however, so the following would also be valid: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.use { '<strong>control</strong>' } +end +``` + +NOTE: +When passing a block to the `experiment` method, it is implicitly invoked as +if `run` has been called. + +#### Segmentation rules + +You can use runtime segmentation rules to, for instance, segment contexts into a specific +variant. The `segment` method is a callback (like `before_action`) and so allows providing +a block or method name. + +In this example, any user named `'Richard'` would always be assigned the _red_ +variant, and any account older than 2 weeks old would be assigned the _blue_ variant: + +```ruby +class PillColorExperiment < ApplicationExperiment + segment(variant: :red) { context.actor.first_name == 'Richard' } + segment :old_account?, variant: :blue + + # ...behaviors + + private + + def old_account? + context.actor.created_at < 2.weeks.ago + end +end +``` + +When an experiment runs, the segmentation rules are executed in the order they're +defined. The first segmentation rule to produce a truthy result assigns the variant. + +In our example, any user named `'Richard'`, regardless of account age, will always +be assigned the _red_ variant. If you want the opposite logic, flip the order. + +NOTE: +Keep in mind when defining segmentation rules: after a truthy result, the remaining +segmentation rules are skipped to achieve optimal performance. + +#### Exclusion rules + +Exclusion rules are similar to segmentation rules, but are intended to determine +if a context should even be considered as something we should include in the experiment +and track events toward. Exclusion means we don't care about the events in relation +to the given context. + +These examples exclude all users named `'Richard'`, *and* any account +older than 2 weeks old. Not only are they given the control behavior - which could +be nothing - but no events are tracked in these cases as well. + +```ruby +class PillColorExperiment < ApplicationExperiment + exclude :old_account?, ->{ context.actor.first_name == 'Richard' } + + # ...behaviors + + private + + def old_account? + context.actor.created_at < 2.weeks.ago + end +end +``` + +We can also do exclusion when we run the experiment. For instance, +if we wanted to prevent the inclusion of non-administrators in an experiment, consider +the following experiment. This type of logic enables us to do complex experiments +while preventing us from passing things into our experiments, because +we want to minimize passing things into our experiments: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.exclude! unless can?(current_user, :admin_project, project) +end +``` + +You may also need to check exclusion in custom tracking logic by calling `should_track?`: + +```ruby +class PillColorExperiment < ApplicationExperiment + # ...behaviors + + def expensive_tracking_logic + return unless should_track? + + track(:my_event, value: expensive_method_call) + end +end +``` + +Exclusion rules aren't the best way to determine if an experiment is active. Override +the `enabled?` method for a high-level way of determining if an experiment should +run and track. Make the `enabled?` check as efficient as possible because it's the +first early opt-out path an experiment can implement. + +### Tracking events + +One of the most important aspects of experiments is gathering data and reporting on +it. GLEX provides an interface that allows tracking events across an experiment. +You can implement it consistently if you provide the same context between +calls to your experiment. If you do not yet understand context, you should read +about contexts now. + +We can assume we run the experiment in one or a few places, but +track events potentially in many places. The tracking call remains the same, with +the arguments you would normally use when +[tracking events using snowplow](../snowplow.md). The easiest example +of tracking an event in Ruby would be: + +```ruby +experiment(:pill_color, actor: current_user).track(:created) +``` + +When you run an experiment with any of these examples, an `:assigned` event +is tracked automatically by default. All events that are tracked from an +experiment have a special +[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) +added to the event. This can be used - typically by the data team - to create a connection +between the events on a given experiment. + +If our current user hasn't encountered the experiment yet (meaning where the experiment +is run), and we track an event for them, they are assigned a variant and see +that variant if they ever encountered the experiment later, when an `:assignment` +event would be tracked at that time for them. + +NOTE: +GitLab tries to be sensitive and respectful of our customers regarding tracking, +so GLEX allows us to implement an experiment without ever tracking identifying +IDs. It's not always possible, though, based on experiment reporting requirements. +You may be asked from time to time to track a specific record ID in experiments. +The approach is largely up to the PM and engineer creating the implementation. +No recommendations are provided here at this time. + +## Test with RSpec + +This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10. + +First, require the RSpec support file to mix in some of the basics: + +```ruby +require 'gitlab/experiment/rspec' +``` + +You still need to include matchers and other aspects, which happens +automatically for files in `spec/experiments`, but for other files and specs +you want to include it in, you can specify the `:experiment` type: + +```ruby +it "tests", :experiment do +end +``` + +### Stub helpers + +You can stub experiments using `stub_experiments`. Pass it a hash using experiment +names as the keys, and the variants you want each to resolve to, as the values: + +```ruby +# Ensures the experiments named `:example` & `:example2` are both +# "enabled" and that each will resolve to the given variant +# (`:my_variant` & `:control` respectively). +stub_experiments(example: :my_variant, example2: :control) + +experiment(:example) do |e| + e.enabled? # => true + e.variant.name # => 'my_variant' +end + +experiment(:example2) do |e| + e.enabled? # => true + e.variant.name # => 'control' +end +``` + +### Exclusion and segmentation matchers + +You can also test the exclusion and segmentation matchers. + +```ruby +class ExampleExperiment < ApplicationExperiment + exclude { context.actor.first_name == 'Richard' } + segment(variant: :candidate) { context.actor.username == 'jejacks0n' } +end + +excluded = double(username: 'rdiggitty', first_name: 'Richard') +segmented = double(username: 'jejacks0n', first_name: 'Jeremy') + +# exclude matcher +expect(experiment(:example)).to exclude(actor: excluded) +expect(experiment(:example)).not_to exclude(actor: segmented) + +# segment matcher +expect(experiment(:example)).to segment(actor: segmented).into(:candidate) +expect(experiment(:example)).not_to segment(actor: excluded) +``` + +### Tracking matcher + +Tracking events is a major aspect of experimentation. We try +to provide a flexible way to ensure your tracking calls are covered. + +You can do this on the instance level or at an "any instance" level: + +```ruby +subject = experiment(:example) + +expect(subject).to track(:my_event) + +subject.track(:my_event) +``` + +You can use the `on_any_instance` chain method to specify that it could happen on +any instance of the experiment. This helps you if you're calling +`experiment(:example).track` downstream: + +```ruby +expect(experiment(:example)).to track(:my_event).on_any_instance + +experiment(:example).track(:my_event) +``` + +A full example of the methods you can chain onto the `track` matcher: + +```ruby +expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_') + .on_any_instance + .with_context(foo: :bar) + .for(:variant_name) + +experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_') +``` + +## Experiments in the client layer + +This is in flux as of GitLab 13.10, and can't be documented just yet. + +Any experiment that's been run in the request lifecycle surfaces in `window.gon.experiment`, +and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) +so you can use it when resolving some concepts around experimentation in the client layer. + +## Notes on feature flags + +NOTE: +We use the terms "enabled" and "disabled" here, even though it's against our +[documentation style guide recommendations](../documentation/styleguide/index.md#avoid-ableist-language) +because these are the terms that the feature flag documentation uses. + +You may already be familiar with the concept of feature flags in GitLab, but using +feature flags in experiments is a bit different. While in general terms, a feature flag +is viewed as being either `on` or `off`, this isn't accurate for experiments. + +Generally, `off` means that when we ask if a feature flag is enabled, it will always +return `false`, and `on` means that it will always return `true`. An interim state, +considered `conditional`, also exists. GLEX takes advantage of this trinary state of +feature flags. To understand this `conditional` aspect: consider that either of these +settings puts a feature flag into this state: + +- Setting a `percentage_of_actors` of any percent greater than 0%. +- Enabling it for a single user or group. + +Conditional means that it returns `true` in some situations, but not all situations. + +When a feature flag is disabled (meaning the state is `off`), the experiment is +considered _inactive_. You can visualize this in the [decision tree diagram](#how-it-works) +as reaching the first [Running?] node, and traversing the negative path. + +When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the +state is `conditional`) the experiment is considered to be _running_ +where sometimes the control is assigned, and sometimes the candidate is assigned. +We don't refer to this as being enabled, because that's a confusing and overloaded +term here. In the experiment terms, our experiment is _running_, and the feature flag is +`conditional`. + +When a feature flag is enabled (meaning the state is `on`), the candidate will always be +assigned. + +We should try to be consistent with our terms, and so for experiments, we have an +_inactive_ experiment until we set the feature flag to `conditional`. After which, +our experiment is then considered _running_. If you choose to "enable" your feature flag, +you should consider the experiment to be _resolved_, because everyone is assigned +the candidate unless they've opted out of experimentation. + +As of GitLab 13.10, work is being done to improve this process and how we communicate +about it. diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 21c61324dc1..15430831f4a 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they primarily target GitLab.com. -Experiments are run as an A/B test and are behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default or rolled back. +Experiments are run as an A/B/n test, and are behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back. ## Experiment tracking issue @@ -36,386 +36,27 @@ and link to the issue that resolves the experiment. If the experiment is successful and becomes part of the product, any follow up issues should be addressed. -## Experiments using `gitlab-experiment` +## Implementing an experiment -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300383) in GitLab 13.7. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. -> - It is not yet intended for use in GitLab self-managed instances. +There are currently two options when implementing an experiment. -[GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment/) is a gem included -in GitLab that can be used for running experiments. +One is built into GitLab directly and has been around for a while (this is called +`Exerimentation Module`), and the other is provided by +[`gitlab-experiment`](https://gitlab.com/gitlab-org/gitlab-experiment) and is referred +to as `Gitlab::Experiment` -- GLEX for short. -## How to create an A/B test using `experimentation.rb` +Both approaches use [experiment](../feature_flags/index.md#experiment-type) +feature flags, and there is currently no strong suggestion to use one over the other. -### Implement the experiment +| Feature | `Experimentation Module` | GLEX | +| -------------------- |------------------------- | ---- | +| Record user grouping | Yes | No | +| Uses feature flags | Yes | Yes | +| Multivariate (A/B/n) | No | Yes | -1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): +- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md) +- [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md) - ```ruby - EXPERIMENTS = { - other_experiment: { - #... - }, - # Add your experiment here: - signup_flow: { - tracking_category: 'Growth::Activation::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data - } - }.freeze - ``` - -1. Use the experiment in the code. - - Experiments can be performed on a `subject`. The `subject` that gets provided needs to respond to `to_global_id` or `to_s`. - The resulting string is bucketed and assigned to either the control or the experimental group. It's therefore necessary to always provide the same `subject` for an experiment to have the same experience. - - - Use this standard for the experiment in a controller: - - Experiment run for a user: - - ```ruby - class ProjectController < ApplicationController - def show - # experiment_enabled?(:experiment_key) is also available in views and helpers - if experiment_enabled?(:signup_flow, subject: current_user) - # render the experiment - else - # render the original version - end - end - end - ``` - - or experiment run for a namespace: - - ```ruby - if experiment_enabled?(:signup_flow, subject: namespace) - # experiment code - else - # control code - end - ``` - - When no subject is given, it falls back to a cookie that gets set and is consistent until - the cookie gets deleted. - - ```ruby - class RegistrationController < ApplicationController - def show - # falls back to a cookie - if experiment_enabled?(:signup_flow) - # render the experiment - else - # render the original version - end - end - end - ``` - - - Make the experiment available to the frontend in a controller: - - ```ruby - before_action do - push_frontend_experiment(:signup_flow, subject: current_user) - end - ``` - - The above checks whether the experiment is enabled and pushes the result to the frontend. - - You can check the state of the feature flag in JavaScript: - - ```javascript - import { isExperimentEnabled } from '~/experimentation'; - - if ( isExperimentEnabled('signupFlow') ) { - // ... - } - ``` - - - It is also possible to run an experiment outside of the controller scope, for example in a worker: - - ```ruby - class SomeWorker - def perform - # Check if the experiment is active at all (the percentage_of_time_value > 0) - return unless Gitlab::Experimentation.active?(:experiment_key) - - # Since we cannot access cookies in a worker, we need to bucket models based on a unique, unchanging attribute instead. - # It is therefore necessery to always provide the same subject. - if Gitlab::Experimentation.in_experiment_group?(:experiment_key, subject: user) - # execute experimental code - else - # execute control code - end - end - end - ``` - -### Implement the tracking events - -To determine whether the experiment is a success or not, we must implement tracking events -to acquire data for analyzing. We can send events to Snowplow via either the backend or frontend. -Read the [product intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for more details. - -#### Track backend events - -The framework provides the following helper method that is available in controllers: - -```ruby -before_action do - track_experiment_event(:signup_flow, 'action', 'value', subject: current_user) -end -``` - -Which can be tested as follows: - -```ruby -context 'when the experiment is active and the user is in the experimental group' do - before do - stub_experiment(signup_flow: true) - stub_experiment_for_subject(signup_flow: true) - end - - it 'tracks an event', :snowplow do - subject - - expect_snowplow_event( - category: 'Growth::Activation::Experiment::SignUpFlow', - action: 'action', - value: 'value', - label: 'experimentation_subject_id', - property: 'experimental_group' - ) - end -end -``` - -#### Track frontend events - -The framework provides the following helper method that is available in controllers: - -```ruby -before_action do - push_frontend_experiment(:signup_flow, subject: current_user) - frontend_experimentation_tracking_data(:signup_flow, 'action', 'value', subject: current_user) -end -``` - -This pushes tracking data to `gon.experiments` and `gon.tracking_data`. - -```ruby -expect(Gon.experiments['signupFlow']).to eq(true) - -expect(Gon.tracking_data).to eq( - { - category: 'Growth::Activation::Experiment::SignUpFlow', - action: 'action', - value: 'value', - label: 'experimentation_subject_id', - property: 'experimental_group' - } -) -``` - -Which can then be used for tracking as follows: - -```javascript -import { isExperimentEnabled } from '~/lib/utils/experimentation'; -import Tracking from '~/tracking'; - -document.addEventListener('DOMContentLoaded', () => { - const signupFlowExperimentEnabled = isExperimentEnabled('signupFlow'); - - if (signupFlowExperimentEnabled && gon.tracking_data) { - const { category, action, ...data } = gon.tracking_data; - - Tracking.event(category, action, data); - } -} -``` - -Which can be tested in Jest as follows: - -```javascript -import { withGonExperiment } from 'helpers/experimentation_helper'; -import Tracking from '~/tracking'; - -describe('event tracking', () => { - describe('with tracking data', () => { - withGonExperiment('signupFlow'); - - beforeEach(() => { - jest.spyOn(Tracking, 'event').mockImplementation(() => {}); - - gon.tracking_data = { - category: 'Growth::Activation::Experiment::SignUpFlow', - action: 'action', - value: 'value', - label: 'experimentation_subject_id', - property: 'experimental_group' - }; - }); - - it('should track data', () => { - performAction() - - expect(Tracking.event).toHaveBeenCalledWith( - 'Growth::Activation::Experiment::SignUpFlow', - 'action', - { - value: 'value', - label: 'experimentation_subject_id', - property: 'experimental_group' - }, - ); - }); - }); -}); -``` - -### Record experiment user - -In addition to the anonymous tracking of events, we can also record which users have participated in which experiments and whether they were given the control experience or the experimental experience. - -The `record_experiment_user` helper method is available to all controllers, and it enables you to record these experiment participants (the current user) and which experience they were given: - -```ruby -before_action do - record_experiment_user(:signup_flow) -end -``` - -Subsequent calls to this method for the same experiment and the same user have no effect unless the user has gets enrolled into a different experience. This happens when we roll out the experimental experience to a greater percentage of users. - -Note that this data is completely separate from the [events tracking data](#implement-the-tracking-events). They are not linked together in any way. - -#### Add context - -You can add arbitrary context data in a hash which gets stored as part of the experiment user record. New calls to the `record_experiment_user` with newer contexts get merged deeply into the existing context. - -This data can then be used by data analytics dashboards. - -```ruby -before_action do - record_experiment_user(:signup_flow, foo: 42, bar: { a: 22}) - # context is { "foo" => 42, "bar" => { "a" => 22 }} -end - -# Additional contexts for newer record calls are merged deeply -record_experiment_user(:signup_flow, foo: 40, bar: { b: 2 }, thor: 3) -# context becomes { "foo" => 40, "bar" => { "a" => 22, "b" => 2 }, "thor" => 3} -``` - -### Record experiment conversion event - -Along with the tracking of backend and frontend events and the [recording of experiment participants](#record-experiment-user), we can also record when a user performs the desired conversion event action. For example: - -- **Experimental experience:** Show an in-product nudge to see if it causes more people to sign up for trials. -- **Conversion event:** The user starts a trial. - -The `record_experiment_conversion_event` helper method is available to all controllers. It enables us to record the conversion event for the current user, regardless of whether they are in the control or experimental group: - -```ruby -before_action do - record_experiment_conversion_event(:signup_flow) -end -``` - -Note that the use of this method requires that we have first [recorded the user as being part of the experiment](#record-experiment-user). - -### Enable the experiment - -After all merge requests have been merged, use [`chatops`](../../ci/chatops/index.md) in the -[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. -The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. -For visibility, please also share any commands run against production in the `#s_growth` channel: - - ```shell - /chatops run feature set signup_flow_experiment_percentage 10 - ``` - - If you notice issues with the experiment, you can disable the experiment by removing the feature flag: - - ```shell - /chatops run feature delete signup_flow_experiment_percentage - ``` - -### Manually force the current user to be in the experiment group - -You may force the application to put your current user in the experiment group. To do so -add a query string parameter to the path where the experiment runs. If you do so, -the experiment will work only for this request and won't work after following links or submitting forms. - -For example, to forcibly enable the `EXPERIMENT_KEY` experiment, add `force_experiment=EXPERIMENT_KEY` -to the URL: - -```shell -https://gitlab.com/<EXPERIMENT_ENTRY_URL>?force_experiment=<EXPERIMENT_KEY> -``` - -### A cookie-based approach to force an experiment - -It's possible to force the current user to be in the experiment group for `<EXPERIMENT_KEY>` -during the browser session by using your browser's developer tools: - -```javascript -document.cookie = "force_experiment=<EXPERIMENT_KEY>; path=/"; -``` - -Use a comma to list more than one experiment to be forced: - -```javascript -document.cookie = "force_experiment=<EXPERIMENT_KEY>,<ANOTHER_EXPERIMENT_KEY>; path=/"; -``` - -To clear the experiments, unset the `force_experiment` cookie: - -```javascript -document.cookie = "force_experiment=; path=/"; -``` - -### Testing and test helpers - -#### RSpec - -Use the following in RSpec to mock the experiment: - -```ruby -context 'when the experiment is active' do - before do - stub_experiment(signup_flow: true) - end - - context 'when the user is in the experimental group' do - before do - stub_experiment_for_subject(signup_flow: true) - end - - it { is_expected.to do_experimental_thing } - end - - context 'when the user is in the control group' do - before do - stub_experiment_for_subject(signup_flow: false) - end - - it { is_expected.to do_control_thing } - end -end -``` - -#### Jest - -Use the following in Jest to mock the experiment: - -```javascript -import { withGonExperiment } from 'helpers/experimentation_helper'; - -describe('given experiment is enabled', () => { - withGonExperiment('signupFlow'); - - it('should do the experimental thing', () => { - expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element)); - }); -}); -``` +Historical Context: `Experimentation Module` was built iteratively with the needs that +appeared while implementing Growth sub-department experiments, while GLEX was built +with the learnings of the team and an easier to use API. |