Reevaluating A/B Testing as a Strategy: Ending Revenue Regret

By ​David Moore

March 29, 2017

This is the third, and final, post in a blog series I’m writing about how to move beyond some of the underlying myths of a/b testing as a strategy. In parts one and two, I talked about how using test results and customer segments that rely on too few data points can sometimes hold us back from creating a dynamic personalized experience.

Once we recognize that our audience is a heterogeneous group, what is the best and most efficient way to build that knowledge into our customer experience?

Post-hoc segmentation is the best way to discover differences in your audience.Delayed insight causes regret

Regret is when we learn a test result that, if we had known about it sooner, would have saved us time and money. This is a perspective shift, since marketers are often trained to look at what was gained, not what was lost, particularly when evaluating testing results.

How does that play out in testing? A/b testing can help us discover nuances in our audience by allowing us to slice and dice data after test results come in, and that added knowledge will then prompt us to re-initiate the testing process based on the new hypothesis. For example, a cart page a/b test may not reach significance, but if you segment the test results you may find that test A was actually beneficial for a mobile audience and B was best for a desktop audience. This post-hoc segmentation is in frequent use for a reason--it can help us generate new and useful insights.

The rub is that if we only discovered that Device Type mattered earlier, we could have provided a better customer experience and increased our goal metric of driving revenue for the business.

This is why regret can exist even in the best case testing scenarios, and over the course of weeks in some testing experiences it can create hundreds of thousands of dollars in revenue regret. A/b testing might help us improve the experience for some, or even many of our customers, but it involves repeatedly creating scenarios of lost opportunity and reacting after the fact.  And, even when we identify a winner that can help us boost revenue overall, there are losses associated with the portion of the audience that wasn’t best served by that option.

These posts have laid out some of the problems that we see our clients grappling with, and why we think it’s worth reevaluating assumptions about who your customers are and how you can better meet their needs and expectations. If you want to dive deeper into what we’ve learned, I’ve just published a white paper that builds on these ideas and explains how you can improve the experience of each customer, and save yourself all of that regret in the form of lost time and revenue in the process.

Click here to download the free white paper “Closing the Insights Loop (Perhaps for Good).”

David Moore is a Senior Product Manager at Monetate. Formerly a product eBusiness lead at Gordon Food Service, Dave designed, developed and launched ecommerce and social community tools for large enterprises in the restaurant, healthcare and education sectors. Dave is an award-winning photographer, an avid traveler and a keen user advocate.

Experience the future of ecommerce personalization

Book your personalized demo today.

Thanks for requesting a demo

We'll be in touch soon.