Skip to content

Latest commit

 

History

History
451 lines (329 loc) · 19.5 KB

README.md

File metadata and controls

451 lines (329 loc) · 19.5 KB
Raw

TestTrack Rails Client

Build Status

This is the Rails client library for the TestTrack system.

It provides server-side split-testing and feature-toggling through a simple API.

If you're looking to do client-side assignment, then check out our JS client.

Installation

Install the gem:

# Gemfile
gem 'test_track_rails_client'

Create an app in the TestTrack server

In every environment (local included) cut an App record via the TestTrack server rails console:

> App.create!(name: "[myapp]", auth_secret: SecureRandom.urlsafe_base64(32)).auth_secret
=> "[your new app password]"

Note: see the TestTrack README for additional information on configuring seed apps for local development.

Set up ENV vars

Set up ENV vars in every environment:

  • MIXPANEL_TOKEN - By default, TestTrack reports to Mixpanel. If you're using a custom analytics provider you can omit this.

  • TEST_TRACK_API_URL - Set this to the URL of your TestTrack instance with your app credentials, e.g. http://[myapp]:[your new app password]@[your-app-domain]/

  • TEST_TRACK_PUBLIC_API_URL - (optional) If public traffic to TestTrack should use a different host name, set this variable. By default this will use TEST_TRACK_API_URL without any credentials

    [your-app-domain] can be

    • testtrack.test
    • localhost:PORT
    • example.org
    • etc

Prepare your controllers

Mix TestTrack::Controller into any controllers needing access to TestTrack and configure it with the name of your :current_user method.

class MyController < ApplicationController
  include TestTrack::Controller

  self.test_track_identity = :current_user
end

If your app doesn't support authentication, set self.test_track_identity to :none.

Prepare your identity models (optional)

If your app supports authentication, you'll need to configure your User model as a TestTrack Identity.

Set up the Chrome extension (optional)

If you'd like to be able to use the TestTrack Chrome Extension which makes it easy for you and your team to change assignments via your browser, you must set up the TestTrack JS client.

  1. testTrack.bundle.min in your application.js file after your reference to jQuery
//= require jquery
//= require testTrack.bundle.min
  1. Then, follow the instructions for configuring the JS client.

Concepts

  • Visitor - a person using your application. test_track_rails_client manages visitors for you and ensures that test_track_visitor is available in any controller that mixes in TestTrack::Controller
  • Split - A feature for which TestTrack will be assigning different behavior for different visitors. Split names must be strings and should be expressed in snake_case. E.g. homepage_redesign_late_2015 or signup_button_color.
  • Variant - one the values that a given visitor will be assigned for a split, e.g. true or false for a classic A/B test or e.g. red, blue, and green for a multi-way split. Variants may be strings or booleans, and they should be expressed in snake_case.
  • Weighting - Variants are assigned pseudo-randomly to visitors based on their visitor IDs and the weightings for the variants. Weightings describe the probability of a visitor being assigned to a given variant in integer percentages. All the variant weightings for a given split must sum to 100, though variants may have a weighting of 0.
  • IdentifierType - A name for a customer identifier that is meaningful in your application, typically things that people sign up as, log in as. They should be expressed in snake_case and conventionally are prefixed with the application name that the identifier is for, e.g. myapp_user_id, myapp_lead_id.

Configuring the TestTrack server from your app

TestTrack leans on ActiveRecord migrations to run idempotent configuration changes. There are two things an app can configure about TestTrack. It can define identifier_types and configure splits.

Defining identifier types:

class AddIdentifierType < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.identifier_type :myapp_user_id
    end
  end
end

Configuring splits

Splits can be created or reconfigured using the config DSL. Variants can be added to an existing split, and weightings can be reassigned, but note that once a variant is added to a split, it doesn't ever completely disappear. Attempts to remove it will simply result in it having a 0 weighting moving forward. People who were already assigned to a given variant will continue to see the experience associated with that split.

class ConfigureMySplit < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.split :signup_button_color, red: 34, green: 33, blue: 33, indigo: 0
    end
  end
end

Cleaning Up Old Splits

In order to avoid clutter in the Test Track server's split registry as well as the Test Track Chrome Extension, a split can be dropped. This will remove the split from the split registry, dropping it from Test Track clients' perspectives. Thus, like a non-additive DDL migration (e.g. DROP COLUMN, RENAME COLUMN), it should be released in a subsequent deployment, after all code paths referencing the split have been removed. Otherwise those code paths will raise and potentially break the user experience.

class RemoveMyOldSplit < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.drop_split :signup_button_color
    end
  end
end

Note: drop_split (a.k.a. finish_split) does not physically delete split data from mixpanel or Test Track's database.

Generating Split Migrations

Split configuration changes can be generated using Rails generators that are included with the TestTrack Rails client.

rails generate test_track:migration add_name_of_split

will generate a timestamped migration file with the content

class AddNameOfSplit < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.split :name_of_split, control: 50, treatment: 50
    end
  end
end

The generator infers the type of split from the migration name.

Adding Drop to the migration name will create a migration to drop a split.

rails generate test_track:migration drop_name_of_split

class DropNameOfSplit < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.drop_split :name_of_split
    end
  end
end

Adding Enabled or FeatureFlag to the end of the migration name will create sensible defaults for a feature flag split.

rails generate test_track:migration add_name_of_split_enabled

class AddNameOfSplitEnabled < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.split :name_of_split_enabled, true: 0, false: 100
    end
  end
end

Adding Experiment to the end of the migration name will create sensible defaults for an experiment.

rails generate test_track:migration add_name_of_split_experiment

class AddNameOfSplitExperiment < ActiveRecord::Migration
  def change
    TestTrack.update_config do |c|
      c.split :name_of_split_experiment, control: 50, treatment: 50
    end
  end
end

Varying app behavior based on assigned variant

Varying app behavior in a web context

The test_track_visitor, which is accessible from all controllers and views that mix in TestTrack::Controller provides a vary DSL.

You must provide at least one call to when and only one call to default. when can take multiple variant names if you'd like to map multiple variants to one user experience.

If the user is assigned to a variant that is not represented in your vary configuration, Test Track will execute the default handler and re-assign the user to the variant specified in the default call. You should not rely on this defaulting behavior, it is merely provided to ensure we don't break the customer experience. You should instead make sure that you represent all variants of the split and if variants are added to the split on the backend, update your code to reflect the new variants. Because default re-assigns the user to the default variant, no data will be recorded for the variant that is not represented. This will impede our abiltiy to collect meaningful data for the split.

You must also provide a context at each vary and ab call. Context is a string value which represents the high-level user action in which the assignment takes place. For example, if a split can be assigned when viewing the home page and when going through sign up, the assignment calls in each of those paths should tagged with 'home_page' and 'signup' respectively. This will allow the test results to be filtered by what the user was doing when the split was assigned.

test_track_visitor.vary :name_of_split, context: 'home_page' do |v|
  v.when :variant_1, :variant_2 do
    # Do something
  end
  v.when :variant_3 do
    # Do another thing
  end
  v.default :variant_4 do
    # Do something else
  end
end

The test_track_visitor's ab method provides a convenient way to do two-way splits. The true_variant option is used to tell ab which variant is the "true" variant. If no true_variant option is provided, the "true" variant is assumed to be true, which is convenient for splits that have variants of true and false. ab can be easily used in an if statement.

# "button_color" split with "blue" and "red" variants
if test_track_visitor.ab :button_color, true_variant: :blue, context: 'signup'
  # Color the button blue
else
  # Color the button red
end
# "dark_deployed_feature" split with "true" and "false" variants
if test_track_visitor.ab :dark_deployed_feature, context: 'signup'
  # Show the dark deployed feature
end

Varying app behavior from within a model

The TestTrack::Identity concern can be included in a model and it will add two methods to the model: test_track_vary and test_track_ab.

class User
  include TestTrack::Identity

  test_track_identifier :myapp_user_id, :id # `id` is a column on User model which is what we're using as the identifier value in this example.
end

Varying app behavior globally

The TestTrack.app_ab method uses an "app" identifier type to be able to globally access "Feature Gate" splits. This is useful when a visitor context is not handy, and when you don't care about visitor specific assignments.

In order to use this feature, you need to set TestTrack.app_name, preferably in an app initializer.

# config/initializers/test_track.rb
TestTrack.app_name = "MyApp"
class BackgroundWorkJob
  def perform
    if TestTrack.app_ab(:fancy_new_api_enabled, context: 'BackgroundWorkJob')
      FancyNewApi.do_thing
    else
      CruftyOldApi.do_thing
    end
  end
end

Tracking visitor logins

The test_track_visitor.log_in! is used to ensure a consistent experience across devices. For instance, when a user logs in to your app on their mobile device we can log in to Test Track in order to grab their existing split assignments instead of treating them like a new visitor.

test_track_visitor.log_in!(:myapp_user_id, 1234)

When we call log_in! we merge assignments between the visitor prior to login (i.e. the current visitor) and the visitor we retrieve from the test track server (i.e. the canonical visitor). This means that any assignments for splits that the current visitor has which the canonical visitor does not have are copied from the prior visitor to the canonical visitor. While this merging behavior is preferrable there may be a case where we do not want to merge. In that case, we can pass the forget_current_visitor option to forget the current visitor before retrieving the canonical visitor.

test_track_visitor.log_in!(:myapp_user_id, 1234, forget_current_visitor: true)

Tracking signups

The test_track_visitor.sign_up! method tells TestTrack when a new identifier has been created and assigned to a visitor. It works a lot like the log_in! method, but should only be used once per customer signup.

test_track_visitor.sign_up!(:myapp_user_id, 2345)

Testing your split-dependent application code with RSpec

Add this line to your rails_helper.rb:

# spec/rails_helper.rb
require 'test_track_rails_client/rspec_helpers'

Force TestTrack to return a specific set of splits during a spec:

it "shows the right info" do
  stub_test_track_assignments(button_color: :red)
  # All `vary` calls for `button_color` will  run the `red` codepath until the mocks are reset (after each `it` block)
end

Analytics

TestTrack does not offer built-in functionality for analyzing the results of split tests. TestTrack does provide hooks to easily integrate with your preferred analytics tool. By default, TestTrack will use Mixpanel as an analytics backend. If you wish to use another tool, you can set the analytics attribute on TestTrack with your custom client. You should do this in a Rails initializer.

# config/initializers/test_track.rb
TestTrack.analytics_class_name = 'MyCustomAnalyticsClient'

Your client must be a singleton or require no initializer arguments, implement the following methods:

# Called when a new Split has been Assigned
#
# @param analytics_event [TestTrack::AnalyticsEvent] An object
# representing an analytics event providing name and properties
# values you can send to your analytics backend
def track(analytics_event)

# Called after TestTrack.sign_up!
#
# @param visitor_id [String] TestTrack's unique visitor identification key
def sign_up!(visitor_id)

Using TestTrack with a new analytics tool

TestTrack manages its own visitor identifier which is different from the identifier of your analytics tool. We recommend using TestTrack's visitor identifier as your analytics identifier when possible. Within TestTrack Rails Client, assignment events will trigger a call to TestTrack.analytics.track with a TestTrack visitor identifier. To ensure that analytics events coming from within the browser have the right identifier, you must set the identifier when your analytics javascript library is loaded.

Here's an example for how to do it with Mixpanel:

mixpanel.init('YOUR MIXPANEL TOKEN', {
    loaded: function(mixpanel) {
        mixpanel.identify('<%= test_track_visitor.id %>');
    }
});

Misconfiguration notifications

TestTrack provides hooks to easily integrate with your preferred error catching tool to receive notifications when we detect a misconfiguration in split usage. TestTrack has built-in support for Airbrake. By default, if you've included the airbrake gem, TestTrack will use Airbrake as a misconfiguration notifier backend. If you wish to use another tool, you can set the misconfiguration_notifier attribute on TestTrack with your custom client. You should do this in a Rails initializer.

# config/initializers/test_track.rb
TestTrack.misconfiguration_notifier_class_name = 'MyCustomMisconfigurationNotifier'

Your client must be a singleton or require no initializer arguments, implement the following methods:

# Called when a Split misconfiguration is detected
#
# @param string message describing the misconfiguration
def notify(message)

Upgrading

From 5.0 to 6.0

  • The offline split registry (used in tests, etc) will now favor variants with weights of 100% when generating deterministic weightings. If no variant has a weight of 100%, then it will fall back on the previous behavior of favoring the variant with the lowest alphanumerically-sorted name (conventionally "false" and "control").

    This means that, when deciding a split via a migration (and/or updating a new variant to 100% in the schema), tests that assumed the previous variant may begin to fail and will require additional code cleanup (or must be explicitly stubbed to use the previous variant).

From 4.0 to 5.0

  • The direct dependency on the delayed_job_active_record gem has been removed, and background jobs now depend on the ActiveJob framework. When upgrading, ensure that your application specifies a backend (e.g. config.active_job.queue_adapter = :delayed_job).

From 3.0 to 4.0

  • The contract of custom analytics plugins has changed. Instead of implementing track_assignment you now must implement track. It's easier and more conventional, though, and takes care of differentiating between experiment assignments and feature gate experiences, which are no longer recorded server-side.

  • You must add self.test_track_identity = :current_user (or whatever your controller uses as a sign-in identity) to your TestTrack-enabled controllers, or set it to :none if your app doesn't support authentication. If your app supports authentication, you'll need to configure your user model as a TestTrack Identity.

  • TestTrack server introduced a new endpoint for fetching the split registry that requires a timestamp of when the application was built. This prevents dropped splits from breaking a running application. If you use rake assets:precompile in your build pipeline, you're all set. If you don't, you'll need to call rake test_track:generate_build_timestamp and ensure that the file testtrack/build_timestamp is deployed along with your app.

From 2.0 to 3.0

  • TestTrack Rails Client no longer manages your Mixpanel cookie. The analytics plugin now provides a callback on sign_up! that will allow you to implement this functionality within your application. Please see the analytics documentation for more details.

  • TestTrack.analytics#track_assignment no longer accepts a properties hash as an argument as mixpanel_distinct_id is no longer relevant.

From 1.x to 1.3

  • TestTrack::Session#log_in! and TestTrack:Session#sign_up! now take a TestTrack::Identity instance argument instead of an identity type and identity value.

How to Contribute

We would love for you to contribute! Anything that benefits the majority of test_track users—from a documentation fix to an entirely new feature—is encouraged.

Before diving in, check our issue tracker and consider creating a new issue to get early feedback on your proposed change.

Suggested Workflow

  • Fork the project and create a new branch for your contribution.
  • Write your contribution (and any applicable test coverage).
  • Make sure all tests pass (bundle exec rake).
  • Submit a pull request.