Why You Shouldn’t Always Listen to the A/B Testing Enthusiasts
Listen on the go by downloading the audio of today’s video.
Join us on the journey to predictable growth. Each week we tackle a different level of your funnel, from traffic and email list growth to sales conversions and upsells. Sign up for our newsletter here to get daily 10-minute #MattHacks videos.
- Who’s giving you advice that you MUST be A/B testing?
- Remember that A/B testing requires that you already have a big following
- Try using chat widgets or surveys instead
You see it all the time. People screaming at the top of their lungs, saying, "You must be A/B testing!"
My name is Matt Ackerson. I'm the founder of AutoGrow.co, as you may already know. And in this video, I'm talking about the great lie of A/B testing for more conversions...
So, let me explain what I mean. I came across a Youtube video talk recently from one of the growth managers over at Mixpanel. They provide analytics for tracking of apps and websites across desktop and mobile. They've been around for about five, six, seven years now.
They're doing pretty well. And the growth manager gave a talk called, "The Five F- Ups of Growth,” about growth mistakes that you want to avoid if you're a small business or a start-up that's getting off the ground. One of her top five things was that you must be A/B testing.
It reminded me that when you want to get reliable results with A/B testing, you need a certain amount of views or traffic actually coming through and flowing through your marketing funnel. Look at the people who are giving this advice, too, because everyone and their grandmother is saying, "You must be A/B testing."
Why I’m skeptical of A/B Testing
I'm kind of skeptical about the ideological belief. It's like an ideological belief in the power of A/B testing because "it's statistics." But another thing that it doesn't take into account is when you have a new design or a new ad format, for example. This is a documented fact and it makes sense when you think about it.
So for example, Google rolled out a new ad format several years ago, and I remember reading about it and how they made note of the fact that the click-through rate for this new ad format was some sort of new design. Imagine it was a square, and now it's like a circle or rectangle; something new.
The click-through rate was significantly higher in the beginning, and so they either discarded those results or made a specific note of it because after a certain amount of time, they're going to settle down to be whatever the average is going to be, right?
This is the thing to beware of with A/B testing, because if you are testing a significantly different version of a pop-up or a landing page, it might convert better, at least in the beginning, just because of the new format. It's sort of a new style. But over time, things might settle down to a lower average that you didn't perhaps anticipate.
A/B testing usually requires a lot of traffic
You want to look at who's giving the advice, like an established company like Mixpanel, or other bigger names where people have tens of thousands of subscribers on their email list and tens of thousands of visits to their website.
And this is part of why I don't necessarily recommend A/B testing in some of my previous videos, because we have a fair amount of traffic where we can do it and we can get results in a reasonable amount of time.
But if you have less than 10,000 visits coming to your website per month, or you're testing a key page and that page doesn't have a minimum amount of traffic, I would say again, 2,000 to 10,000 visits or page views, then it's going to take you awhile to see results. That’s the other thing about A/B testing--you need to get results within a reasonable amount of time.
And further, a lot of the people that champion this idea of A/B testing, if you read some of the many case studies out there, won't even tell you what their confidence interval is. And what that means is they can say statistically that within a 98% chance that this number is accurate and that one version will outperform the control version, the original version.
But many of those people are not testing an adequate sample size, meaning they may only be testing to a 90% confidence level or a 95% confidence level, which means that there is a 5% chance that one version will actually not be beating the other.
We have a nice, long skyscraper blog post talking about different A/B test case studies, and we only selected the ones that were 98% or better. And that's what you want to be testing to as well because otherwise, you're just gambling, like, "90% is good enough because I don't have a lot of traffic, so let me just go with that."
Alternatives to A/B Testing
What I recommend is a better way. When you're a young business and/or don’t have a lot of traffic, invest more of your time into looking at and understanding what individuals’ and potential customers’ objections are. What are their objections on your sales page?
And then you can kind of vet this information out by investing in surveys. You can email them a survey. You can use a chat widget. Listen and ask questions, especially in the beginning or especially if you find that a certain page or a step in your sales funnel is a bottleneck.
And put some belief (because people don't talk about this) in sequence testing, because with A/B testing, part of why it takes such a long time is that you're testing version A versus version B, and one is showing 50% randomly, then the other.
But if you do a sequence test, you're showing version A is your current version, and then boom--you put up version B, so that one gets all of the traffic. Then you can start to notice a pattern of success, or not, because it is getting more data points instead of just 50% of the data points. Make sense?
Next steps
So that's it for today's video. Feel free to leave me a comment. What did you think of this video? How are you going to apply this tip to your business and to your sales funnel? If you like this video, give it a thumbs up and give me some feedback in the comments as well.
If you want to know more about how to set up your own successful sales funnel in record time, how to do it fast, how to do it right and how to uncover gaps that are leaking money in your existing sales funnel and marketing online, I encourage you to check out our top two selling products in terms of volume.
They are our Sales Funnel Diagram Pack and our Sales Funnel Blueprint product. You can access them by going to the products link in the navigation. And if you want, you also get on the waiting list for our top-tier training program, The Six Figure Sales Funnel, which we open the doors on once every three months. So if you're interested in that, get on the waiting list for it.
Until next time, as always, my name is Matt Ack in the Matt hat delivering you your daily dose of Matt hacks. Keep hustling, stay focused. I’ll see you in the next video.
11-Point Perfect Sales Funnel Checklist or invest in our new Rocket Content course, which is currently discounted.
