How to Develop a Plan for A/B Testing Your eCommerce Site
A/B testing offers a host of benefits for your eCommerce website, from improved conversion rates, traffic, and SEO to sales, revenue, and your business’s overall bottom line.
Knowing what, how, and when to test your eCommerce site might seem overwhelming, but with a data-driven A/B testing plan, you can take the guesswork out of improving your site’s structure, functionality, and content.
In this post we’ll help you develop a plan for testing your website and show you how the results can make a significant impact on your site’s usability and conversion rate. But before we do that, let’s make sure we’re all talking about the same thing.
What Do We Mean by the Term A/B?
First, a confession: while this article talks about A/B testing, we’re focusing our discussion on test planning, so the tactics we cover can be applied to any optimization test you perform. Examples of those tactics include multivariate and controlled sequential tests—more on those later!
A/B testing means you create two versions of a web experience, such as two email subject lines or two versions of a landing page. You show version A to one group of web viewers and version B to a second group, then determine which version performs best based on predefined goals. This is called a split test.
Other Types of A/B Tests
You can easily grow your A/B tests to include more than two variants to become a larger split test, or you can test multiple variables within each variant (two email subject lines and two versions of a landing page), called a multivariate test.
To add another layer to testing options, some eCommerce platforms apply algorithms to the winning test and then send more people to that version. These tests are also great because they’re awesomely referred to as multi-armed bandits. They allow you to run tests for longer periods of time, which gains statistical significance, and expose more people to the winning version.
Regardless of the test type, they all work by exposing similar groups of people to different web experiences to help you identify what’s most effective. The more test variations you have, however, the longer you’ll have to run the test if you want to see meaningful results.
What’s the First Step in Planning an A/B Test?
The first step is to analyze and identify the areas of your website that need the most improvement. Analysis helps you improve your conversion rate by revealing how visitors like to use your site, where they have problems, and what elements contribute to successful use of your site.
Data to Gather When Building a Testing Plan
There’s an absolute wealth of data you can use to develop your testing plans. We recommend using both quantitative and qualitative methods because maximizing your eCommerce experience is both art and science.
- Web Analytics: Your digital analytics data, collected by Google Analytics, SiteCatalyst, Coremetrics, or whatever platform you use, is one of your biggest optimization assets. It gives you a strong understanding of where visitors enter your site and how they engage with your site. Web analytics also helps you validate the importance of other data sources. For example, if your colleague believes there’s a problem with a specific page on your site, analytics will reveal how many people visited that page and whether it caused any negative activity. You might discover only a small number of people viewed the page and that their conversion rate is the same as those who never viewed it. This means the page in question is likely not as big a problem as you think.
- Testing Data: The results from your web analytics tests will tell you what future tests you should conduct. There will always be aspects of your site that elicit bounces or abandonment, such as customer service and checkout pages, so it’s important to know if you should focus on improving these seemingly weak areas, or if there are better opportunities to make the strong parts of your site even stronger.
- Competitor Data: Your competitors’ activities are a great resource for new test ideas. Elements of your competitors’ websites to monitor include traffic acquisition tactics, website navigation, content, and the use of integrations. Watching how your competitors transition visitors who click on a CPC advertisement or an email might affect how you test your own landing pages. Or, the way your competitors design their navigation and product filtering experiences could inform how you optimize your own taxonomy. You can take this even a step further and monitor how these elements evolve over time. Chances are your competitors are also performing A/B tests, and if you keep a close enough eye on them, you can benefit from their successes.
- Social Listening: Observing what people say about you on social media helps inform your content strategy, so it’s something you’ll want to A/B test on your website and social media accounts. Sequential testing is a good method here, which involves comparing two variations given to the same group of people at different times. Social listening will help you uncover what best resonates with your audience, thus provoking the highest engagement.
- Online Surveys: Posting a survey on your website is a great way to add qualitative data to the quantitative data your analytics platform provides. You can ask users if they had a specific problem with your site or request demographic and personal-interest information. You’ll be able to build stronger site personas and tie them to your testing and analytics data. For example, you can ascertain whether a site problem or strength is related to a certain group of users.
- Usability Studies: Another great qualitative resource is to ask users to complete websites tasks and then observe and record their actions. You might discover that something on your website isn’t working as intended or a task could be made easier. Not all site problems are immediately obvious; if a user is aware of a site problem they might tell you during a survey, but if they’re unaware a usability study could help reveal what those problems are.
- User Feedback: User feedback from customer service channels provides another qualitative resource to uncover site problems. Unlike other qualitative sources, you likely already have this feedback on hand. User feedback can tell you if information is missing on your website or if it’s hard to find or use.
Creating a Testing Plan Focus
The abovementioned data sources will give you a long list of elements to improve, so you’ll need to know what elements are key to improving the customer journey. There are three primary focuses to guide this prioritization:
Site pathing tells you where visitors entered your site and how they prefer to use it. When looking at the landing pages through which visitors entered your site, note the volume of entry traffic to each page and how they found that specific page. You’ll want to target the biggest entry points because you might be able to move the conversion rate more for certain traffic types.
Check how many visitors arrive a page from your internal site search and on what page they conduct those searches. If people repeatedly cease activity on a certain page in favor or using internal site search, it could mean that problems exist on that particular page. People who use internal site search are three to five times more likely to convert as well as make a purchase, so it’s a worthwhile area to optimize.
Site bottlenecks are an extension of site pathing. Chances are you’ll have somewhere around 40 percent of visits entering through your homepage, and after another couple of pages, nothing will own more than one percent of entry traffic. Look beyond that initial entry for any primary arteries that affect a significant amount of visits. Are there category pages or content that most visits path through? Or are there pages that users have to path through to accomplish a success goal, e.g., the cart.
Behavior benchmarks help you understand if those bottlenecks are problems or not. Remember when I said users are three to five times more likely to convert after conducting an internal search? Benchmarks like this help identify if you have a problem or not. The more specific the benchmarks are to your business category (for example, apparel) and business type (retailer), the better. Forrester and Econsultancy can provide some high-level benchmarking data while competitive platforms and databases like Internet Retailer can give you more specific benchmarks. As a starter, these resources can act as a guide for high-level site behavior benchmarks. Your site might outperform or underperform all of the benchmarks, but the variance between each of your steps and the benchmark steps might reveal areas of opportunity.
The Power of Data
You now know what areas of your website to test and the types of data to use when building your A/B testing plan. If qualitative feedback from surveys or customer service channels is available, you might even know what content to add to your site. Before moving forward with testing, you should identify other gaps to ensure you’re not excluding additional tests that could prove to be useful.
