The Optimization Long Tail: Are You Testing What Really Matters?

By Adam Figueira

January 3, 2012

Ten variations of 10 variables. That’s 1,000 possible combinations (10 x 10 x 10). Even the highest traffic web pages would have difficulty testing this many combinations in a short enough time frame to achieve statistical significance and make a valid conclusion.

But you don’t have to. Of all the combinations you can test, only a few really matter. Call it the “Long Tail of Website Optimization.” Essentially, if you were to graph this as a Power Law Distribution (a frequency distribution with a long tail):

  • A very small number of combinations would wield significant influence on conversion rate
  • A very large number of combinations would matter much less

Doesn’t everyone get this? Apparently not, as a team at Google once quibbled over two shades of blue, and ended up testing the 41 hues in between. Mathematically, there’s no denying that testing a large number of variables can lead to incremental improvements in conversion rate. The point is that diminishing returns set in, costing you valuable time that could have been better invested in other tests.

As online marketing expert Bryan Eisenberg has written, there are 10 variables and five dimensions to landing page success. But if that sounds large, and therefore, contradictory to the points I’ve made above, it’s not. The key takeaway isn't that you should avoid certain tests. On the contrary, you should test as much as possible. Rather, the point is to avoid an endless number of test combinations, such as Red x Square x Large x Top x Text: “Buy Now.”

Starting Off on the Right Foot

This last point cannot be overstated. Avoiding unnecessary effort is essential to achieving a high ROI (remember, motion and forward progress aren't synonymous), and that’s really where the value of full-service management comes into play. With expert resources behind the wheel, you can eliminate many combinations right off the bat—they’ve been tried and tested in the past, and they don’t work.

In addition, your core digital strategy plays a larger role than many believe. For example, testing whether your checkout button should read “Buy Now” versus “Buy Today” matters little if you haven’t answered the more fundamental question of why people should buy from you? A smart way to start testing is to begin with a SWOT Analysis, answering four questions about your business and website:

  1. S: What are our strengths?
  2. W: What are our weaknesses?
  3. O: What are our opportunities?
  4. T: What are our threats?

Once you’ve answered (and addressed) these, you’re ready to lay the framework of a strategy that tests what matters.

Adam Figueira is a former product marketing director at Monetate.

Experience the future of ecommerce personalization

Book your personalized demo today.