The real keys to A/B testing
A/B testing, a numbers game.
In Digital Marketing we are blessed with close to unlimited data to play with. At Pacho y Nacho we say that marketers are now like a Zorro en el gallinero.
We have access to so much data and so many different key performance indicators that it is easy to get lost and to miss the real impact of an A/B test. This is why it is important to set the test and the reporting properly in order not only to understand the results but also to be able to apply those findings in the future.
Why should I A/B test?
It is very easy to say “Hey! This CTA looks retarded, let’s change the color, shape and it’s going to be way better”.
Let’s imagine now that you do not A/B test your CTA revamp on your landing page and that the sales numbers go way up. It is easy to assume that the uplift in sales was a direct consequence of your CTA update. But what if it isn’t? What if someone mentioned your brand on last night’s telenovela?
Had you run your old CTA in parallel to the new CTA, both would have been impacted by the mention and you could actually be comparing manzanas to manzanas rather than second guessing if your CTA update had a positive impact, no impact or even a negative impact on your conversions.
How to prepare an A/B test?
The first step in setting up an A/B test is to sit down and think.
You need to think at the issue you are trying to address or the thing that you want to improve.
Then, once you identify the issue at hand, think about how you hypothesize you can fix it. Assuming you have a taco stand in one of the busiest mercados of Tijuana and your sales of soft drinks are not satisfactory, the hypothesis could be something like “If I add more salsa picante in my tacos, people will be thirsty and order a drink”.
Then I need to think about how to measure my impact. Going back to the example above, no matter how many tacos I sell through the day, the metric that I am trying to influence is the percentage of taco buyers that order a drink after eating the taco.
I can now design my test: One out of two taco order will get the double amount of salsa picante. I will then compare the percentage of customers in each group that buy soft drinks.
My reporting should then focus on that metric. In some cases you can add a secondary metric that will ensure that you are not having a negative impact on something else. In the example above, I could check what percentage of each customer group bought a second taco after the first one. In other words: are people receiving the extra spicy taco as willing as the others to come back and order a second one. I want to sell more Jarritos, but not if it is costing me clients.
What next with my A/B test?
Once you have done all this, you are pretty much ready to run your test. All you need is to make sure you track your customers and their subsequent actions properly. You also need to make sure that the population and the uplift are sufficient to draw conclusions (Confidence). For example, if during your day you sell 6 tacos (3A and 3B), regardless of the number of Jarritos soft drinks you sell, there’s no way you can be confident this is representative of hundreds of potential customers.
In the same way, if you serve 1,000 customers (500 each) and you sell 100 Jarritos to group A and 102 Jarritos to group B, it is too tight to draw confident conclusions.
In both cases you will need to extend your test until you have enough data to safely draw conclusions.
A/B testing optimization is one of Pacho y Nacho Performance marketing services.
Tags: AB testing, Optimization