Multivariate (A/B) Testing

December 9, 2013

In web application development, testing the source code to verify it correctly produces the features required is a common practice. Developers will often write tests to validate that the code written satisfies the requirement originally set during the initial design of the feature. This same mindset should exist not just during development, but early in the design of the web application itself, to verify the product is something people actually want. And it doesn't have to stop there — after the product is launched, testing can continue to gather information on how users interact with the application, what works, what doesn't, and what should be redesigned.

The process of testing the design of the product is known as multivariate testing. Multivariate testing (sometimes referred to as A/B testing, or split testing), gives the product designers a way of testing certain ideas on the users while providing analytics on those actions. Those results help determine if any changes are necessary to improve the product and increase conversion rates. Changes can be anything, from a simple change of color, to a complete layout redesign.

When a user visits a site under multivariate testing, the user is put into either a control or test group. Within the control group, the user will see the normal content originally designed. Users in the test group, however, will see a new design intended to test out a new look or feel. Users proceed as normal from there but actions they take are collected and analyzed to understand which design performed better: the control or the test.

Before we take a look at how to setup a page for multivariate testing, let's first discuss how to take advantage of Google Analytics to provide both page and user interaction tracking.

Google Analytics: Universal Analytics

There are many ways to perform analytics on a web application, but Google Analytics certainly is at the top of the list. Google is (at the time of this writing) currently rolling out a change to the way they handle analytics with the introduction of Universal Analytics. Custom dimensions and metrics have replaced custom variables while new features (such as support for platforms other than websites) are available on the universal platform. For more information on Universal Analytics, please see this overview.

To begin using Google Analytics, you must create an account within the platform. The account is not your Google account, but instead an account to reference the analytics that will be tracked. This can be either a website or mobile application, and Google recommends the use of Universal Analytics rather than the Classic Analytics tracking method. Within the creation of a new account, a property is necessary. For websites, a property would be the website that will be tracked. A name and URL is required, along with some additional information. (Note that for local testing on localhost, the URL must still end in .com/.org/etc. Specifying http://localhost.com for the default URL will suffice).

With the account and property setup, a Tracking ID will be provided which can be used within the analytics JavaScript code to reference your account. From here you can add custom domains and metrics which help manage custom attributes tracked on your application. Google Analytics also includes events which can be tracked in real-time giving you an immediate indication of what actions your users are taking. Other data such as page views can also be tracked real-time. Custom reports can be created to give a personalized look at the collected data. More information can be found on the Universal Analytics developers guide site.

With analytics in place, we can turn our attention to multivariate testing.

The Multivariate Library

When I first set out to build a multivariate test, I looked through the open source community to find one that was JavaScript based (for client side usage) and supported the new Universal Analytics version of Google Analytics. And at that time, one didn't seem to exist that met those requirements. I began to build a solution around it myself and found it could be generalized and potentially used by others. The multivariate project can be found on GitHub, and includes documentation, a demo site, along with instructions on how to install via Bower.

Quick Start with Multivariate

A/B web tests are intended to display one page to the control group and a separate, though similar, page to the test group. To begin using multivariate to create an A/B test, first decide which element(s) you wish to toggle between the control and test groups. Then assign the mv-control or mv-test class to these elements. For example:

<!-- This will be displayed to the control group -->
<button class="mv-control">Submit</button>

<!-- This will be displayed to the test group -->
<button class="mv-test">Press Here To Complete</button>

When users in the control group are shown the page, they will see the elements with the mv-control class displayed (and not test elements). Similarly, when users in the test group view the page, they will see elements with the mv-test class displayed (and not control elements).

To configure multivariate so this behavior takes place, include the multivariate library on your page, and configure multivariate:

// create a new multivariate test
var mv = new Multivariate("submit-button-test");

// execute the A/B test
mv.runTest();

Instantiate a new multivariate object for each test you wish to run. Each test requires a unique name which is used when remembering which user is in which test. Once instantiated, execute runTest() to run the A/B test on the user. Calling this function will toggle either the control or test content based on which cohort the user falls within.

Further Information

To learn more about multivariate, please refer to the multivariate GitHub page. The project includes documentation on the product, a demo site, and an API overview. Pull requests are welcome!