AB and multivariate testing

Often site owners and marketers reviewing the effectiveness of a site will disagree and the only method to be certain of the best-performing design or creative alternatives is through designing and running experiments to evaluate the best to use.

AB testing and multivariate testing are two measurement techniques that can be used to review design effectiveness to improve results.

AB testing

In its simplest form, A/B testing refers to testing two different versions of a page or a page element such as heading, image or button. Some members of the site are served alternatively, with the visitors to the page randomly split between the two pages. Hence, it is sometimes called ‘live split testing’. The goal is to increase page or site effectiveness against key performance indicators including click-through rate, conversion rates and revenue per visit.

When completing AB testing it is important to identify a realistic baseline or control page (or audience sample) to compare against. This will typically be an existing landing page. Two new alternatives can be compared to previous control which is known as an ABC test. Different variables are then applied. For example: Test 1: original page and new headlines, existing button, existing body copy. Test 2: original page and existing headlines, new button and existing body copy. Test 3: original page and existing headline, existing button, new body copy.

An example of the power of AB testing is an experiment Skype performed on their main top bar navigation, where they found that changing the main menu options ‘Call Phones’ to ‘Skype Credit’ and ‘Shop’ to ‘Accessories’ gave an increase of 18.75% revenue per visit. That’s significant when you have hundreds of millions of visitors! It also shows the importance of being direct with navigation and simply describing the offer available rethan than the activity.

Multivariate testing

Multivariate testing is a more sophisticated form of AB testing which enables simultaneous testing of pages for different combinations of page elements that are being tested. This enables selection of the most effective combination of design elements to achieve the desired goal.


  • Structured experiments to review influence of on page variables (e.g. messaging and buttons) to improve conversion from a website


  • Often requires cost of separate tools or module from standard web analytics package
  • Content management systems or page templates may not support AB/multivariate testing

Chaffey, D. and Ellis-Chadwick, F., 2012. Digital marketing: strategy, implementation and practice (Vol. 5). Harlow: Pearson.

Related posts