March 6, 2013
A/B-testing is something that is quite frequently discussed as something you should do. Yet, most marketing managers and online communicators still don’t. In fact. They dread the mere notion of having to test and change their digital campaigns.
I will get to the solution and how to easily do an A/B-test. But in order to fully understand the notion that there is a problem to be solved here. I first need to explain the essence of the challenge of starting using A/B-testing as a methodology for constant improvements.
The challenge is threefold.
First off there is a problem with the relationship between marketing and IT. Marketing wants things to move along fast, whilst IT wants things to be done the right way. This has been a problem ever since IT became a part of the communication process.
There is a vast difference between building solid systems and supporting quick communication. This has bearing on the split testing as any such functionality has to be planned when building the architecture of the system according to IT, whilst Marketing gets frustrated by slow development causing revenue, participation and results to diminish.
This problem can now be solved.
The second problem is that the wish to test something has been seen as weakness. “What, you don’t believe in your idea enough… why test it if you believe in it..“.
This is something that has come out of traditional marketing structures where your idea went of to print or went out on air. There was no way that you could split test your campaigns during a campaign. The mere fabric of digital gives you an opportunity to change and fine tune your campaign, website and online presence so that it improves over time.
I like to view the A/B-split testing as a Formula 1 race, where I am the engineer standing in the pit stop ready to adjust any small detail to make perfection out of good. No one can say that any Formula 1 car (read creative) is slow (bad). But the conditions might be such for the day, that things need to be adjusted for it to work at full capacity. (Love that metaphor… WhOOt… you don’t like it.. well.. then.. then… pfff…)
Companies love to discuss 360 approaches to advertising. And yes. Different medias can support each other. But without using them properly, if applying the same methodology to digital as you do print, you are doomed to end up with less than the maximum effect of the campaign. With A/B-testing you can increase conversion rates, e-mail opening rates, click through rates etc, constantly giving you more for every dollar you spend.
And as I promised above. The challenge of time to market – from idea to implementation is solved. It doesn’t exist anymore.
Thus. This second problem can be overcome.
The third problem is that most marketing managers – clients – do not consider growth when considering the creative of an advertising campaign. They do not see it as a part of building a brand over time, but they rather see it as an instance to push a new product or meet a seasonal increase/decrease in demand. Advertising campaigns can have great success whilst the product they are meant to support doesn’t grow its default revenue streams. Meaning, in order to maintain your sales volumes at the hight of the top of the campaign, you have to campaign all the time which, in many cases, is a very expensive affair.
This is a much more difficult thing to overcome as it has to do with behavior and moreover habits. It is not a problem of lack of knowledge. It is quite easy to rack up the numbers. But still this seems to be a very difficult task for some (would really like to write most) marketing managers to wrap their arms around. Creativity is a lot more fun to purchase than growth, as you actually have to think and not only feel. However, as I will come to, A/B-testing is ALL about feeling, however, it doesn’t give you the power to choose as the data does that for you. Ie. you can still feel, but you need to be able to disregard taste.
To illustrate this I will show you the difference in growth and in simply having campaign success by studying two graphs of search queries made in Google. I use this measure as it is a great tool to estimate brand demand amongst consumers.
This is the search graph for Kodak – Successful campaigns, yet no gained growth. You can see that regardless of their marketing efforts. Their growth is negative.
This is the search graph for “Mitt Romney” and as you can see… no one cares about Mitt Romney if he doesn’t buy a shit load of media to get elected
Now. In comparison to growth which looks like this. First off is iPhone.
Secondly we’ll have a look at McDonalds.
There is a very strong difference in the way these brands have worked with campaigning. One uses the campaign to promote innovation in the product whilst the other use it as a way to promote more of the same.
By focusing on creating campaigns that allow for A/B-testing, we also allow users to respond to the way we market our products better. I believe that this is one great way to overcome the hurdle of taste, and “internal protectionists” that seem blinded by previous success. It is also a methodology to use data to implement a process of change and improvement in an organization without threatening anyones position. It is the data that says so. Not anyone smarter. Just the data.
Another way to overcome this challenge, is if we as analysts, developers and optimizers become as cool as the advertisers. That’s why I love the emergence of the infographic hype, growth hackers and the concept that everyone is a nerd. This actually helps us get through as they want to be like us, dress like us yet, they are too socially skilled to ever be like us.
So. Finally. What is A/B-split testing
Well, basically it is looking at some data and seeing that traffic doesn’t flow the way you want it to flow. It can be that they do not click on a website the way we want them too. It can be that they do not sign up to the newsletter, or simply that they do not fill in a specific form or find a specific document. As a result you get either lower conversion rates or more calls to your customer service.
STEP1 – The Hypothesis Gut
In order to improve this, you start out with listening to your own to gut.
First off. You try to do whatever it is that you want your users to do. Ie. Sign up, buy, click or find. Somewhere along the way you may realize that there might be a few other ways in which a user complete a task a lot faster. This gut feeling is what we optimizers like to call – Hypothesis. Not only does it sound a lot more valuable than “gut feeling”, but it also gives the client a honest hint that it is simply an educated guess.
The more experience the optimizer – or you – has, the better the hypothesis and probably the fewer tests needed.
STEP 2 – Setting up your test
Once you have your hypothesis in your head. You should start testing the changes. It might be a larger text, a variation in navigation or a change in a background color.
In essence a change to the copy, page or design in order to change the way the user uses it. Ok. So once you have that ready in your head, you should get to the nitty gritty of creating these variations.
Now. This has previously been extremely time consuming and problematic as you have had to create all these variations in code – meaning you have had to contact IT to do these variations for you.
This problem is now a segment of the past.
Several great tools such as Optimizely – which I can honestly say that I love (video embedded below this section) and Visual Website optimizer have given us simple drag and drop possibilities to make variations quick.
This means you do not have to implement 6 different versions of a design. No more waiting for coding. No more waiting for design templates. YOU can do this.
Step 3 – The implementation
You run the test. Get the numbers and see which one of your hypotheses actually created an improvement.
Now you have all the data. You can call IT and say with 100% accuracy (perhaps 95% or some confidence interval) that sales will increase with 40% by making these changes. It is thus a priority project. NO IT-person… no matter how big, rough skinned or flanell shirt pansared, can object to that. You won’t have to ask for resources as you can do the testing yourself. You are not there with a problem. You are there with a solution!
Regardless if you run A/B-testing on a website or if you do it on a campaign, or some e-mail copy or whatever you are testing, it is worth it. There are tools simple enough for you to be able to do this on your own. You can relax and go with your gut whilst the data takes care of the decision making.
You then send the solution rather than another problem to the IT department. You become the true hero of your organization as you just increased the bottom line result with 40%. Congratulations! You’re it!