Adam Figueira, Product Marketing Manager, Monetate
There’s a theory in website testing that when two or more tests run concurrently that the results of each test are invalid. But it’s not actually true. In a world where each person is exactly like everyone else in every way, we’d only need a sample size of two to conclusively measure the impact of a test. But visitors to your website are all different.
Two people may live in Chicago and have come to your website via AdWords, but they still earn different salaries, drive different cars, and will take different vacations this year. In a clinical trial, we’d be able to ask hundreds of questions to weed out these differences. But online markers can’t actually do that. For all the control that markers want to have in website testing, there’s still many factors that we neither know about nor can control.
And many of these factors can wield significantly more influence on our results than what we’re actually testing on the page. And that’s exactly why concurrent independent tests are still equally valid. The net effect is exactly the same as there being just one more difference between the visitors and your test.