When and How to Run A/B and Multivariate Testing for your website

This guest article was written by Rebecca Q Jones from the multivariate and A/B testing website Maxymiser, enabling SEO`s to increase the effectiveness of websites by using multivariate testing.
You can follow Rebecca on Twitter: @maxymiser


A/B and multivariate testing (MVT) are used to help website owners determine the most effective marketing strategies for their products. Changes to business websites are made based on quantitative testing results, rather than gut-feelings or design bias. Hypotheses can be tested on real customers resulting in actual numbers to prove or disprove an idea.

What is A/B testing?

The simplest of the tests, A/B testing sends half of the website traffic to a “control” version and half to a test version. Any number of elements can be tested for efficacy; a landing page, a site process or a call-to-action. The name A/B is misleading in that the control version can be tested against as many as nine different versions.


What is multivariate testing?

When a number of variables are tested at one time, such as testing the color of the cart button, the thumbnail image and size of each, it is called a multivariate test. The number of variables and the number of versions of each variable determines the number of test versions. If every conceivable combination is utilized the test is called “full-factorial” testing. If not, the design of the test is referred to as “fractional factorial”. Although this form of multivariate testing saves time, it is also less reliable because you may be excluding the best performing factor.


Deciding which testing method to apply can be determined by using the following examples.

When should you use A/B split testing?

You should use A/B split testing when:

  • You are inexperienced with testing. Because A/B testing is the simplest form, tests can be completed quickly. After having a few successes, you can focus on the more involved experiments.
  • You are only interested in the impact of one variable. For example, you may want to determine if placing testimonials on your landing page helps with conversion. This can be tested easily by testing with the testimonials on the page against the absence of the testimonials. You may believe that visitors will be more likely to convert if there is a security seal on your page. In addition, you could test the positioning of the security seal.
  • You need design freedom. This could mean comparing a new website design to the older control version of the website. Testing big changes before moving to multivariate testing saves time in the end.
  • You do not have access to a multivariate tool. Some e-commerce platforms do not provide a full testing suite.
  • You do not have enough results to reach statistical significance. You may have the traffic, but if there are not enough conversions, you cannot get an accurate test result. The test will have to run too long which will likely skew the results.
  • You have limited understanding of coding. The JavaScript coding necessary to do A/B testing is simple and easily added to a website.
  • You need fast results. Because A/B tests measure extreme changes between fewer combinations, the time required to run the test is much shorter than MVT.

When should you use MVT testing?

You should use MVT testing when:

  • You are fine-tuning. You have already performed extensive A/B testing and have found your best performing version. Then it is time to fine-tune for maximum effectiveness.
  • You need to know a combination of variations for each test variable. You could use an A/B split test tool to test two variables with two variations, but it is ultimately a multivariate test. By using a multivariate tool, you will be able to access an aggregate report of the best performing variations.
  • You want to know what variables are influencing each other. If you need to test more than what a single A/B test can handle (more than 10 variables at once), use the multivariate test. Testing all versions during the same time period with the same traffic will produce less testing bias and provide faster results.
  • You have enough conversions and traffic. For multivariate testing, there is not a specific number to qualify. However, online test duration estimators may help. Although they are helpful, they are not perfect.
  • You have technological or branding and design restrictions. In this case, you are only able to test specific variables. MVT will be a much more efficient strategy than trying to use the A/B testing method.

A/B testing vs MVT testing


Each testing technique has its own unique advantages and disadvantages. Essentially, the techniques go hand-in-hand. The A/B test finds the optimal website design and MVT would be used to optimize the elements of the site. Any organization must keep in mind, testing is not a one-time effort. Ongoing testing must be performed since markets and trends change rapidly.

Don`t forget to like & share this post if you like it. :)

Image by BlueFountainMedia

  • Pingback: When and How to Run A/B and Multivariate … – Web SEO Analytics :PC & Internet Lifestyle

  • Lake Elsinore SEO

    Thank you for the great post! I often use A/B testing for client websites when I test small changes. Sometimes just the smallest of changes generates spikes in traffic. A/B testing is a great way to test these small changes. You did a great job of nailing down this fact in your article.

    A/B also yields quick results as you have noted here as well. This is another reason that I use the A/B method. Thanks again for the great read and education post!

  • Chicago SEO Comapny

    Have you seen the A/B testing case study done on Barack Obama website during his campaign? They tested different variations of buttons and photos. They raised an extra 60 million dollars. READ THE CASE STUDY> It’s awesome. (I promise this is not spam web address)


  • Pingback: When and How to Run A/B and Multivariate Testing for your website | Web SEO Analytics | Succesful B2B Marketing Tactics and Strategies | Scoop.it

  • Antioxidantes

    thanks, going to put this up to the IT people to see if we can test our webpage. I am thinking in introducing a video & would like to see its impact. Cheers

  • Burglar Alarms Maidenhead

    Make sure that you have a good and thorough understanding of the people who use and go on your website. This will achieve for you the best results.

  • Dr RWP

    One problem with this kind of testing (and especially multivariate testing) could in some cases be acquiring enough data for the results to be statistically significant. This means the methodology could be more reliable for high traffic sites than e.g. SMEs with smaller data volumes.

  • Dennis van der Heijden

    Good article, just an update on this one. Multivariate testing in Google is not available anymore. You would need tools like ours Convert.com to get that done. Regular A/B testing is still possible go for that to Google Analytics – Content – Experiments (they have some basic split testing there for free)


Leave a Reply