A/B Testing: Trail Testing for Your Website

I recently decided to relive my youth a bit by (re)picking up camping, and being that a good backpack is paramount to a successful trip I’ve been doing a ton of research as to which pack to purchase. One thing I know from experience is that most packs look great and feel great on your shoulders in the store, but trail testing your gear is truly the only way you’re going to find out what is, and isn’t going to work for you. The same can be said about any e-commerce site. Once the site has launched the “trail testing” period begins and regardless of how much testing you did during development… you’re about to discover a multitude of opportunities to increase conversions and improve the site.

So how do you go about choosing and implementing these new improvements? It really is as simple as ABC… well maybe not so much C. A/B split testing is a great method for testing out new ideas to find out which ones work and which ones do not work. You simply divide your audience into two groups and expose one group to your original site (control group) and the second group to a version of the site where your new “improvement” has been implemented (test group). Then you track the results over a period of time and compare the statistics to see which has performed better.

A/B testing can be an informative and affordable solution for figuring out what does and does not convert, but there are some general guidelines to be aware of:

1. Have a clear goal in mind. A/B testing measures actual behavior of users, so be sure you know exactly what you’re testing for before you begin.
2. Your test group should be of a similar size to the control group. This should go without saying but the closer your groups sizes are the more accurately the results can be measured against one another.
3. Test only one change at a time. While it’s tempting to make numerous changes at once it’s impossible to measure what was and was not successful accurately. For instance you may make 3 changes and see a 5% net increase. However change #1 could measure a 15% increase in the conversion rate while changes #2 and #3 actually decrease the rate by 5% each.

It is important to note that A/B testing can only be used when there is a clear goal that can be measured by a computer. You can measure banner ad campaigns, the amount of newsletters read and shopping cart conversions, but you can’t measure things like the users experience or how they view your brand after visiting your site. So when making decisions based on your testing you will also need to consider the things that can’t be measured… or you may risk undermining other parts of your business. While I’m pretty sure he wasn’t speaking about statistics, conversions, and online shopping… I still think Ben Parker said it best “with great power comes great responsibility”.

So much like my testing of backpacks to see which works best for me in the woods, you should consider testing various elements of your site to see what works best for your business online. Is featuring a product on a landing page taking away from the sale of other products in your store? Possibly…but who really knows? Does having a field for a promo/coupon code during checkout lead to a higher abandoned cart ratio? Very likely… but are you sure about that? So the next time it comes to making that potentially expensive decision you’ll be able to remove the gut instinct and make the correct decisions based on proven statistics.