October 24, 2011
So, you want to try A/B and multivariate testing on your website. Should you use a free website testing platform or a paid tool?
Why, some people wonder, would I ever use a paid tool when I can use something else for free? The answer—and I know it's a cliche—is that NOTHING is truly free. Many things in life are worth what you pay for them, and website testing platforms are no exception.
10 Reasons Why Free Website Testing Tools Aren't Free:
10. Free tools may be free, but your employees are not. At the very least, you're going to need a full-time or part-time employee dedicated to making the testing tool work. That could cost at least $30,000 a year. The other extreme will require a full team of people to manage a complicated tool. I've heard a story about one company that had a team of 15 people managing its testing efforts. Fifteen people—that's crazy!
9. QA isn't included. Every single test you design will have to go through some quality assurance. In other words, all the code required to run campaigns is going to have to be tested—on different browsers, on different operating systems—to make sure that it doesn't break your site or lead to bugs. Another way of making this point would be to say: Free tools require the most IT involvement.
8. No targeting. With a lot of free tools, it's relatively difficult—impossible even—to segment specific visitor segments. Why is segmenting necessary? Because some visitor segments perform very different than others (e.g., new vs. returning visitors). What works for one segment might not work for another. You have to target.
7. Expertise isn't included. A/B and multivariate testing are relatively new. Sure, direct marketers have been doing A/B testing forever, but website testing is new. Ecommerce is new. And in two disciplines that are barely 20 years old, there just aren't that many experts. So, what do you test first? What best practices will lead to an immediate impact for your business? With a free tool, you're thrown into the deep end of the pool and told to swim. Is that a good idea for your business?
6. Full service isn't included. Because of the resources and expertise needed to do testing effectively, it helps to have a team on your side—a team that's been there before, and that's committed to your success. When it comes to testing, this isn't a luxury—it's essential. In a way, it's as if you're standing inside a plane having never flown before. Would you feel comfortable sitting down in the cockpit and trying to take off? Wouldn't it be better to have a pilot and crew on board who know exactly what to do?
5. Consistency campaigns aren't possible. A good example of a consistency campaign is when a visitor clicks on a display ad for your website. Echoing that ad's creative and messaging on your landing page, and then across your site when they browse, is a best practice and can easily increase your conversion by double-digits.
4. Product badging isn't possible. Product badging, like consistency campaigns, has been shown to have a dramatic effect on retailers' bottom lines. Free tools don't support it.
3. No multi-user capacity. If you're an organization of any size at all, you'll have a couple people managing your testing efforts. Perhaps a marketing manager using the tool on a daily basis, with your head of marketing checking in from time to time. Yet, one of the biggest limitations of every free tool on the market is that they're designed for one user at best, while anything more becomes clunky.
2. Total cost of ownership. You have to ask yourself: How much is your time worth? Is there a huge opportunity cost to using platforms that require massive amounts of time and quality assurance before you can successfully test that great idea of yours? How much will your team cost? What other time-saving features are included in the package? What about full service, is it included? All these are valid questions. You should not make a decision on which platform to use based on that platform's sticker price, but instead, consider its total cost of ownership.
And the #1 reason why free website testing tools aren't free...
1. The wrong tool might totally turn you off of testing. This is exactly what happened to me a few years ago, and it's a significant problem.
I had heard of this new thing called "website testing." I had seen the case studies, the promises of how one simple change could double my conversion rate or dramatically reduce shopping cart abandonment. So, I signed up for a free tool and spent weeks brainstorming what variants to test.
"I'll try a bunch of different background colors, a couple different buttons, and a few different testimonials," I thought to myself. It seemed like a good idea at the time, but it wasn't.
After the two weeks it took to code all the variants into my first "perfect test," I quickly found out that I had decided to test too many variants. It would take something like 64 years to even possibly reach statistical significance.
The whole thing was a wash. And, in the end, I was totally turned off of testing. It just wasn't worth the effort. It took too long. Required too much coding. Too much QA. Little did I know, I was just using the wrong tool.