Reevaluating A/B Testing as a Strategy: Change is Normal

By ​David Moore

February 17, 2017

This is the first in a series of blog posts I'm writing to bust some myths that exist about a/b testing as a strategy. The first of these posts addresses the hypothesis that test results are absolute. Ummm, no.

Actually, unexpected changes happen. Change is normal. 

Let me illustrate.

On October 22nd, the New York Times and Five Thirty Eight had probabilistic models that predicted the presidential election, based on polling data and additional data (historical and economic data). The media and many statisticians, leveraging the data they had, believed a single outcome would prevail.


This obviously was not the case. The reason why? In retrospect, it is obvious that not all of the data was present. The polls proved themselves to be not reliable, primarily because individuals weren’t always comfortable sharing their choice. Furthermore, as new data arrived in realtime, the state of "knowledge" for predicting the outcome changed.

It’s a fact, change is common. Translating this kind of change to the world of ecommerce, brands may see a large change in performance quickly. This can be due to ad traffic, an email promotion, weather, or even viral social media content. The world changes often and so does ecommerce behavior - at an even faster rate.

The US Election in 2016 is a perfect example where polling data didn’t sufficiently represent true action. And, it is not unlike the practice we employ of using a test to determine the rules for a larger population. Situations like this lead to misconceptions that behavior for some reason will remain consistent over time.

This leads us to a new question: 

If behavior is changing all of the time, are rules the best way to determine the right experience for each individual?

I know from my day to day job in product management that the answer to that question is no. I also know what Monetate has done and is doing to solve it. Are you using legacy approaches in the new era of automation and machine learning? Get your copy of the Closing the insights loop (perhaps for good) whitepaper to find out.


David Moore is a Senior Product Manager at Monetate. Formerly a product eBusiness lead at Gordon Food Service, Dave designed, developed and launched ecommerce and social community tools for large enterprises in the restaurant, healthcare and education sectors. Dave is an award-winning photographer, an avid traveler and a keen user advocate.

Experience the future of ecommerce personalization

Book your personalized demo today.