February 17, 2017
This is the first in a series of blog posts I'm writing to bust some myths that exist about a/b testing as a strategy. The first of these posts addresses the hypothesis that
test results are absolute. Ummm, no.
Let me illustrate.
On October 22nd, the New York Times and Five Thirty Eight had probabilistic models that predicted the presidential election, based on polling data and additional data (historical and economic data). The media and many statisticians, leveraging the data they had, believed a single outcome would prevail.
This obviously was not the case. The reason why? In retrospect, it is obvious that not all of the data was present. The polls proved themselves to be not reliable, primarily because individuals weren’t always comfortable sharing their choice. Furthermore, as new data arrived in realtime, the state of "knowledge" for predicting the outcome changed.
It’s a fact, change is common. Translating this kind of change to the world of ecommerce, brands may see a large change in performance quickly. This can be due to ad traffic, an email promotion, weather, or even viral social media content. The world changes often and so does ecommerce behavior - at an even faster rate.
The US Election in 2016 is a perfect example where polling data didn’t sufficiently represent true action. And, it is not unlike the practice we employ of using a test to determine the rules for a larger population. Situations like this lead to misconceptions that behavior for some reason will remain consistent over time.
This leads us to a new question:
I know from my day to day job in product management that the answer to that question is no. I also know what Monetate has done and is doing to solve for that, but I will spare you the pitch and encourage you to read more about our products.