Testing & Optimization Part 1

In this issue:

This issue will begin a series of two articles taking an in-depth look at online testing and optimization. The second issue we will continue the discussion with a detailed examination of some of the underlying technologies that are required to implement a successful testing program. We will also address the intricacies of experimental design and assist in the identification of what variables to test.

What is Testing & Optimization?  Scientifically and statistically rigorous tests designed to compare results obtained from an experimental sample group versus a control group.

Overview

Experimentation can be found everywhere: from 8th grade science projects, to NASA, to prescription drugs, to direct mail. Yet despite these far reaching applications, testing and optimization seems to be largely missing from the practices of most web analytics departments. The reasons for performing testing are obvious but implementation and interpretation can be complicated and may require additional expertise not often available in an IT or marketing department. However, in a field where 1% increases to conversion rates can mean millions in increased revenue and profits, testing and optimization is an essential tool. For example, we have seen results such as the following:
  •  36.4% in landing page conversion.
  •  22% increase in average order size.
  •  43% increase in overall site conversion.

Clearly, it is no wonder that the testing buzz among web analytics practitioners has increased during recent months. The concept of single variable, or A/B, testing has become a topic of several forums and is a frequent topic at web analytics conferences and tradeshows. However, testing is a difficult topic to cover in an educational setting as so much of it directly relates to the specific type of test and client's needs or expectations. The problem lies in that the answer to so many questions around testing and optimization is "it depends." It depends on everything from the number of variables, the ability to implement complex versus simple tests, the willingness to accept potential errors as well as so many other factors. However, it is my belief that all serious ecommerce players need to implement testing strategies in order to optimize their websites and continue to compete.  

As in any testing situation, the test can be structured as single or multivariable. Obviously, a single variable test offers a simpler experimental design, implementation and interpretation and is often a good place to get started with controlled experimentation. However, because single variable testing offers a limited amount of information, it can be time consuming and expensive to answer all of the important questions.

Single Variable Vs Multi variable
 

Single Variable Testing

The commonly referred to A/B testing is a very typical form of single variable testing with a control page and a test page. Another commonly referenced example of single variable testing is "A/B/A testing". A/B/A testing involves running two identical versions of the control group. Users are routed to the 3 pages where the two "A"s are identical controls and the "B" is the variable. Once the performance of the two A's converge the test sample size should in theory be statistically significant. Obviously, this will minimize the statistical significance calculations but can also have the potential for misleading results. More specifically, it may be difficult to determine when the control groups have converged as opposed to just intersected and how long the control groups results need to stay converged.
 
Advantages
  • Easier to design experiments
  • Easier to explain results
  • Easier to implement
Disadvantages
  • Only able to test one variable at a time
  • Need the maximum number of test subjects to gain complete information on more than one variable (multiple tests)
  • Perhaps elements of the "losing" page actually perform better than some of the elements in the "winning" page

It is important to clarify that there may be more than one difference between the control page and the variable page in a single variable test. For example, there could be several differences between the 2 pages. But, because it is set-up as a single variable test, each page is being tested as a whole, rather than testing each difference individually. This makes it impossible to identify which of the individual differences between the pages attributed to the success or failure of that specific page.

Multivariable Testing

On the other hand, multivariable testing allows the experiment to attribute the effects of the variables on the results. Multivariable testing can be a much more efficient means to gather information albeit more complicated to design and execute. Multivariable experiments can run from a simple testing of 3 different variables with 6 possible combinations of variables to more complex testing of 100’s of variables with billions of possible combinations. Additionally, multivariable testing requires much more complicated statistical models and the interpretation of the results can be complex in some situations.
 
Advantages
  • Allows you to test many potential combinations of variables on one page and understand the effects of each variable.
  • Accelerates the learning process.
Disadvantages
  • Complex to design experiments.
  • Complex to interpret the results.
In the Next Issue:
We will continue our discussion of testing and optimization and focus specifically on experimental design and the identification of what variables to test. View Testing and Optimization Part 2.
 
Josh Manion
Josh Manion
Chief Executive Officer
Stratigent, LLC
 
For more information please call 877-427-2900 or email info@stratigent.com.