How to Market Scientifically: AB Split Testing

Marketing is mostly trial and error.

You are selling your product or someone’s product.

You identify the target customers, their desires and needs, where they hang out physically or online, and how you can attract their attention to your products.

You will develop a campaign plan, and advertisements, and engage your target customers.

If your campaign fails to achieve the target revenue, you must find out what went wrong.

Then you go back to the drawing board and redesign the campaign.

Then you see if the changed campaign has worked or not.

This is called AB split testing.

Let us discuss what AB testing looks like.

Make major changes:

In AB testing, you have to make major changes to consider the impact of the changes.

If you only change the font of a billboard campaign, that won’t qualify as a major impactful change.

The difficult part is at times you will make a minor change and get some results. 

This will make you think the minor change causes the improved result.

But that is not the case.

In one direct mail campaign, some customers were sent a sales letter in a purple envelope. The other customers were sent the letter in a standard white envelope.

The purple envelope receivers bought more products for some reason.

The campaign manager was super excited that he cracked the code of customer response.

In the next campaign, he sent out all the sales letters in purple envelopes.

The campaign flopped.

So the purple envelope was not a factor in the high response in the test campaign.

CTA above the fold:

When you open a sales email, you see part of the email on the screen. 

You have to scroll down to see the rest of the email.

The line where the initial view ends is called the fold.

You must have a call to action above the fold.

A customer may not scroll down most of the time.

So if you don’t have a CTA above the fold, you may miss some sales.

The same thing applies to billboards or printed ads.

The CTA must be written clearly for the customer to see.

The sample size has to be big enough:

If your sample customer size is too small, you can not consider the results of that test.

For email campaigns, if your email list size is less than a thousand people, you can’t take the results seriously.

In a smaller sample, a few customer's behavior will impact the results.

The advertisement spend has to be large:

In one ad campaign, a newbie marketer ran to his boss saying he had a fifty percent uplift in sales using a different message.

The boss asked what the ad spend is for that test campaign.

The newbie said the spend is fifty dollars only.

The boss said come to me when you spend five thousand dollars on the campaign.

Again, a few customers can skew the results in a small ad campaign.

So if you want to test something, test it in a big campaign.

Conclusion:

In the marketing world, you get results that do not make sense at times.

All the results have to be analyzed meticulously and with a grain of salt.

If you understand your target customers and you know the test results make sense, then you can accept the changes as impactful and roll them out for future campaigns.