5 Tips for Running an A/B Test on a Budget
A/B tests are like eating your vegetables. You know they’re good for you, and they’re something you’re supposed to do, right? The trouble is that they can be intensely time- and money-consuming. You’re probably already envisioning a process filled with negotiations with graphic designers and engineers, time-consuming integrations, maybe even difficulty convincing your manager that this is the right move. Never fear! With the right tools and some preparation, you’ll be running A/B tests with ease.
A recent A/B test that we ran on the WordStream.com homepage
I recently joined WordStream as a Senior Optimization Specialist and have been running lots of A/B tests, from home page tests to mobile testing, to increase conversion rates at key points in our user journeys. Our website gets a ton of traffic, but since we’re still in startup mode, we make a point to run our tests as frugally and efficiently as possible.
Based on our personal experiences, here are our top 5 tips to guide you through the A/B testing process if you’re also a small business or working on a budget.
Tip #1: Find Your Biggest Pain Points & Get Real Feedback
You probably already have a good idea of what needs improvement on your website. Does a particular page have an insanely high bounce rate? Are your conversion rates on a specific section of your site falling behind? That's your pain point!
When deciding where to run a test, make sure that you’re running on pages with enough traffic to actually get real, actionable results. Sure, changing the imagery on your “About Us” page may slightly improve your conversion rates – but you’ll likely have to wait months to see any real results. Make sure that your test is running on a page that has a high traffic volume and is integral to the conversion process.
Your product page likely has the traffic volume and impact on the conversion funnel that makes it perfect for an A/B test. For example, you could test moving the CTA above the fold on this page.
Once you’ve identified the page or section that you’d like to test, it’s time to get some real, honest feedback. It’s hard for many marketers to look objectively at their own websites – that’s like calling your own baby ugly! You could reward some of your customers for completing a survey about their experience and asking them to identify specific points in the conversion path where they felt confused or couldn’t find the information they were looking for. In a survey by the Rockefeller Corporation (PDF), 68% of users reported abandoning a website experience because “they think it doesn’t care about them.” What’s worse, you’ll only ever hear from 4% of unsatisfied customers – the other 96% just abandon without submitting any feedback to your business. If you don’t have the resources to devote to user testing, you could call in a favor from a fellow marketer and ask for their unbiased evaluation.
Tip #2: Think Big!
Big problems require big solutions. Changing the color or size of your button may be an easy fix, but it’s not going help you achieve the long-term improvement you’re looking for. Instead, think about the call to action (CTA) featured in the button or the offer itself. Is your CTA confusing, or is it not presenting enough value? Revise the copy with clear, action-driven language. Is your page not converting because the offer isn’t presented clearly enough? Think about adding a graphic or a video to demonstrate the value of the offer. Perhaps the offer itself isn’t appealing to your current audience. Instead of urging your user to buy your product now, you could test a free trial offer, or offer different contract lengths.
Think bigger! This likely won’t have the big impact on conversion rate that you’re looking for.
Tip #3: Form a Realistic, Actionable Hypothesis, and Identify KPIs
Once you’ve identified what your test will look like and where it will run, set some time aside to develop a hypothesis. Your hypothesis should take the form of a simple if/then statement (remember these from 7th grade science?): If I change the text of the CTA on my homepage, then my click-through rate will improve significantly. Refer back to this hypothesis when you’re analyzing the results of your test!
A recent test we ran on our blog sidebar.
The hypothesis was: “If we include an image of the AdWords Grader Report, then users will be more likely to complete the Grader.”
After you’ve developed a hypothesis, choose some key performance indicators (KPIs) to monitor your test. In the example above, the main KPI would be click-through rate. You’d also want to pay attention to conversion rate. Develop a daily or weekly tracker where you can keep an eye on these metrics. Many A/B testing services will monitor them for you, and if you’re running a test using two unique URLs, you can isolate these two pages in your reporting software. Also, if you’re presenting multiple offers on the same page, make sure that you’re tracking all of them so that you’re able to hone in on the results of your test! You may find that you’ve cannibalized one offer by optimizing the other.
Examples of tracker formatting with fictional data.
Tip #4: Be Patient!
Doing nothing may be the most difficult part of your test! Make sure that you run your test for at least 7 days and that your results achieve statistical significance – at least 95% is recommended. Most A/B testing services will compute the significance of your results for you, but you can also find a calculator online (here is a good one from Optimizely). Be aware that if you’re testing a low-traffic page, or if your website doesn’t have a high number of conversions, you’ll need to run your test for a few weeks or more.
Sample from Optimizely’s free sample size calculator. Take advantage of this even if you’re not an Optimizely customer!
Tip #5: Choose the Right Testing Partner & Invest in Your Own Training
There are several good A/B testing platforms on the market right now! You’ll find that almost all platforms offer the same basic A/B testing features, including a visual WYSIWYG editor so that you can move, resize, and edit different elements of your page without calling in development resources (a big timesaver!). They also offer basic reporting features, visitor segmentation (would you like to only show your test to new visitors? Desktop visitors?), the ability to set custom goals, and run tests across multiple pages.
An example of Optimizely’s visual editor in action.
Optimizely has been around for a while and has set the standard for A/B testing tools. They also offer a suite of training lessons geared towards specialized users. However, their pricing may be prohibitive for many small businesses. Some other options to consider are Convert, Marketizator, and Visual Website Optimizer (VWO).
Heatmap tracking may be a value-add for your A/B test.
Convert has the most competitive pricing and integrates goals, segmentation, and tracking easily with Google Analytics, Crazy Egg, and other CRO software. Marketizator is also cost-competitive, especially when signing a longer-term contract. They include additional survey and personalization features to deliver banner ads when your users load, scroll, or exit a page. Visual Website Optimizer aims to serve as an all-in-one conversion optimization platform, offering heatmaps and advanced targeting and personalization features on top of standard A/B testing.
Finally, after you’ve chosen a testing partner, take the time to master the tool and make sure that you’re fully trained on how to set up and run tests. Any good software partner will take the time to offer you training resources and help you get your first test set up.
You’re all ready to start setting up your first (or your best) A/B tests! Let us know how it goes.