Landing Page Optimization Pitfall -- Not Collecting Enough Data
SEW EXPERTS: BY THE NUMBERS By Tim Ash, Search Engine Watch, Sep 2, 2009
Columns | Contact Tim | Biography
One of the advantages of online marketing in general (and landing page optimization in particular) is the ability to measure everything. All online marketing campaigns and programs should be run "by the numbers." The difficult part is in knowing which numbers to use and when.
Many online marketers like to watch the pot boil. They come to resemble stock market day traders who get hooked on the action and make frequent changes to their programs. Some even use automated tools (e.g., for PPC bid management) to make decisions more frequently than is humanly possible.
There is nothing inherently wrong with automated tools or frequent changes per se, but each change must be made after collecting an appropriate amount of data. Unfortunately, this is often not the case in practice.
Let's consider the dangers of small sample sizes. Let's assume that you have had four visitors to your e-commerce website. One of the visitors has bought something. So what is your conversion rate of visitors to sales?
If you answered 25% (one sale out of four visitors), you are probably way off. Most e-commerce sites we have seen range from 1% to 5%. Unless your product is unique, indispensable, and available for sale only from your site, the 25% conversion rate is highly unlikely.
Similarly, if you had no sales after the first four visits, you would probably be wrong to conclude that your conversion rate was really zero, and that you would never get a sale. This example may seem a bit extreme, but too many landing page tests are decided with inappropriately small data samples. And remember that your sample size should be expressed in the number of conversion actions, and not in the number of unique visitors.
Once you start the data collection in your test, resist the temptation to monitor the results frequently, especially early in the test. Otherwise, you run the risk of getting on an emotional roller-coaster caused by the early streaks in the data. One moment you may be euphoric about excellent results, and the next you may be despondent as the indicated improvement vanishes like a mirage.
So pick a statistical confidence level in your answer and wait until you have collected enough data to reach it. You need to have the self-discipline not to even look at the early results.
Remember, the statistics Law Of Large Numbers tells us that our measured conversion rate average will eventually stabilize at the actual value. But it does not tell us exactly how long this will take. It is possible to have large deviations from the actual mean early in the data collection.
Even if you are confident in the fact that your new landing page converts better than the original, you still do not have a clear indication of exactly how much better it is if you haven't collected enough data.
Do not simply use the observed difference in lift as your reported answer. Always present it with the correct error bars (also called "confidence intervals"). I strongly urge you to review a primer on basic statistics to solidify your understanding.
» Print this article » E-mail a colleague » Post a comment
Biography
Tim Ash is the president of SiteTuners.com, a performance-based landing page optimization company. During his Internet career, Tim has worked with companies like American Express, Sony Music, Black&Decker, eBags, American Honda, and McAfee (HackerSafe). He has chaired Internet conferences and spoken internationally at such industry events as Search Engine Strategies, PC Expo, the Affiliate Summit, eComXpo, and Internet World. Tim is a frequent columnist and writer on conversion improvement, and is the author of Amazon's e-commerce bestseller book Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions (John Wiley & Sons Press, 2008).
Article Archives by Tim Ash:
» Landing Page Optimization Pitfall -- Not Collecting Enough Data - September 2, 2009
» The Art of the Landing Page: 7 Tips For Increasing Conversions - August 19, 2009
» Lies, Damned Lies, and Statistics in Landing Page Optimization - August 5, 2009
» Your Baby Is Ugly - July 22, 2009
» Meaning-Making in Landing Page Optimization - July 8, 2009
» Applying Probability to Landing Page Optimization - June 24, 2009
» More Articles by Tim Ash
No comments:
Post a Comment